Lightning-fast semantic embeddings in a 5–7 MB WebAssembly bundle. Engine + model + tokenizer in one file. Embedding search on CPU — no API calls, no GPU. Try the live demo - search 2k docs entirely on-device.
Two tiers, same API — pick by size/quality trade (full comparison):
npm install @ternlight/base # quality tier — 7 MB wire, ~5 ms/embed
npm install @ternlight/mini # small tier — 5 MB wire, ~2.5 ms/embedimport { embed, cosineSim, similar } from '@ternlight/base';
// One primitive: string → 384-dim L2-normalized Float32Array
cosineSim(embed('reset my password'), embed('I forgot my password')); // 0.88
// Nearest-neighbor search over a corpus
similar('I want my money back', [
'Refunds: how to get your money back',
'Track the status of your delivery',
'Update your billing address',
], { topK: 2 });
// → [{ text: 'Refunds: how to get your money back', sim: 0.70 },
// { text: 'Update your billing address', sim: 0.24 }]Works in Node ≥ 18, browsers (via any bundler), Cloudflare Workers, Vercel Edge, Deno, and Bun — the package routes each environment to the right loader. Package docs: @ternlight/base · @ternlight/mini.
Distilled from all-MiniLM-L6 with BitNet b1.58-style quantization-aware training. Three design choices stack to fit an embedding model in a few MB:
- Ternary weights — every weight is
-1,0, or+1; inference is adds and subtracts. Quality holds because the model trains as a ternary model from the start. - One bundle — model + BERT tokenizer + engine in a single
.wasm. No postinstall step, no runtime fetch. - SIMD inference engine — hand-written Rust compiled to WASM SIMD; the add/subtract math rides CPU vector instructions.
All numbers measured on the shipped int4 builds (M-series Mac, Node/V8):
| @ternlight/mini | @ternlight/base | |
|---|---|---|
| Wire size (gzipped wasm) | 5.0 MB | 7.2 MB |
| Latency (p50 per embed) | 2.5 ms | 5.1 ms |
| Throughput (single-thread) | ~400 emb/s | ~195 emb/s |
| Spearman vs MiniLM-L6 teacher | 0.820 | 0.844 |
| Retrieval (SciFact NDCG@10) | 0.439 | 0.465 |
| Architecture | 2-layer · d_model=256 · 4 heads | 2-layer · d_model=384 · 6 heads |
| Parameters | ~9.5M | ~15.4M |
| Output | 384-dim L2-normalized | 384-dim L2-normalized |
| Max input | 128 tokens (~95 words) | 128 tokens (~95 words) |
| Quantization | ternary weights · int4 embeddings | ternary weights · int4 embeddings |
On-device embedding unlocks:
- Search-as-you-type. Results appear before the user finishes typing. Faster than any network round-trip can be.
- Privacy-sensitive apps. Queries and documents never leave the device — no data-handling agreements, no leak risk.
- Offline-first apps. Browser extensions, Obsidian plugins, desktop apps.
- Edge-runtime apps. Cloudflare Workers, Deno Deploy, Vercel Edge. Embeddings co-locate with your request handler — no separate inference service to call.
- Edge devices and IoT hardware. Raspberry Pi, single-board computers, industrial gateways, kiosks. Add/subtract math runs efficiently on ARM cores — no GPU or NPU required.
- Static sites. Jekyll, Hugo, Astro. Ship the model with the bundle; semantic search works without a backend.
ternlight/
├── packages/ Published npm packages (pnpm workspace)
│ ├── base/ @ternlight/base — quality tier (d384)
│ └── mini/ @ternlight/mini — small/fast tier (d256)
├── engine/ Rust → Wasm inference engine
├── training/ Python distillation + QAT pipeline, packer
├── eval/ Engine quality + perf benchmarks
├── docs/ Design docs
├── models/ Model release registry pointers
├── scripts/ Build + release orchestration
└── .github/ CI workflows
Deeper reading: project overview · architecture · inference engine · model internals · eval methodology.
There's still tons of headroom for perf and quality improvements. Beyond stacked encoders, I'm curious about other modalities, specially generative use cases that fit the same tight constraints. JS, Rust, and ML contributors all welcome.
ternlight builds on three open-source efforts:
- BitNet b1.58 (Ma et al., Microsoft Research, 2024) — the architectural research underlying ternary weight training.
bitlinearby @schneiderkamplab — the reference PyTorch implementation of BitLinear. We use it directly during training (bitlinear==2.4.6) and the Rust inference engine mirrors its forward-pass math byte-for-byte.sentence-transformers/all-MiniLM-L6-v2— the teacher model the student is distilled from.
The Rust engine in engine/src/kernels.rs is an independent reimplementation of bitlinear's BitLinear.forward() for the WASM target; parity tests guard against drift.
MIT — see LICENSE.
Banner photo via Macaulay Library, Cornell Lab of Ornithology.
In memory of Alex Movsessian - who held software to the highest standard, and treated everyone around him with kindness.