For contributors running quality and performance evaluations against the shipped engine. If you're using ternlight in an app, see
packages/ternlight/. For the latest published numbers, seequality/RESULTS.md.
Cross-cutting evaluation of the shipped engine (engine + bundled model + JS API). Distinct from:
engine/tests/— element-level parity tests (does the Rust math match Python?)packages/ternlight/tests/— JS API integration tests
eval/
├── quality/ Quality scorecard — Spearman, retrieval, charts
├── regression/ Engine vs baselines on real eval tasks
├── benchmarks/ Latency (cold + warm), throughput, memory
├── compatibility/ Target-runtime matrix (Node, Browser, CF Workers, Deno, Bun)
├── results/ Committed JSON outputs per release version
└── REPORT.md Human-readable scorecard, regenerated per release
bash scripts/run-eval.sh # populates results/v<X.Y.Z>.json + regenerates REPORT.mdSix dimensions per release. See ../docs/eval-methodology.md for what each metric measures and why.
| Dimension | What gets measured |
|---|---|
| Quality | Teacher alignment, STS-B AUC + Spearman, Recall@K |
| Quantization gap | fp32 baseline vs ternary, per component (embedding / BitLinear / activations) |
| Runtime performance | Latency (cold + warm), throughput, memory peak |
| Size | .wasm bytes, .bin bytes, gzipped over the wire |
| Compatibility | Required Wasm features, min Node/browser, per-target pass/fail |
| Honest comparison | Side-by-side with transformers.js + MiniLM, ONNX, embedding APIs |
- Always publish the gap, not just the headline. Numbers without context mislead.
- Version-anchor everything.
results/v<X.Y.Z>.jsonis committed at release time. - Same harness, every release.
scripts/run-eval.shregenerates everything; manual scorecard updates are a smell. - Distinguish engine from model. Both engine bugs and model weaknesses show up as metric drops — separate them by reporting parity vs absolute quality side-by-side.