BGE Small English v1.5
16 configurations testedQuality criteria
Finite normalized distinct 384d vectors; relevant passage cosine exceeds unrelated passage by >=0.1.
| Precision | Device | Result | |
|---|---|---|---|
Q4F16 | GPU · WebGPUAccelerated | Passed | Very fast |
Q4 | GPU · WebGPUAccelerated | Passed | Very fast |
FP16 | GPU · WebGPUAccelerated | Passed | Very fast |
FP32 | GPU · WebGPUAccelerated | Passed | Very fast |
Q8 | GPU · WebGPUAccelerated | Passed | Fast |
UINT8 | GPU · WebGPUAccelerated | Passed | Fast |
BNB4 | GPU · WebGPUAccelerated | Passed | Fast |
INT8 | GPU · WebGPUAccelerated | Passed | Fast |
FP32 | CPU · WASM | Passed | Very fast |
UINT8 | CPU · WASM | Passed | Very fast |
Q8 | CPU · WASM | Passed | Very fast |
INT8 | CPU · WASM | Passed | Very fast |
BNB4 | CPU · WASM | Passed | Very fast |
Q4 | CPU · WASM | Passed | Very fast |
FP16 | CPU · WASM | Failed | — |
Q4F16 | CPU · WASM | Failed | — |
Test setup & sample details
- Quality gate
- Finite normalized distinct 384d vectors; relevant passage cosine exceeds unrelated passage by >=0.1.
- Tested
- 2026-09-15T20:15:36.758Z
- Environment
- Isolated headless Chrome 151 on macOS; Transformers.js 4.2.0
- Runtime
- Transformers.js 4.2.0
- Artifact revision
- ea104dacec62c0de699686887e3f920caeb4f3e3
- Fixture
{ "file": "bge-small-en-v1.5.json", "url": "https://f005.backblazeb2.com/file/hubters-webai-public/assets/benchmarks/bge-small-en-v1.5.json", "size": 606, "sha1": "2f3cc9dccba6004ab0061011297a862c5bda1634", "source": "WebAI-authored English/Chinese retrieval examples with upstream BGE query instruction", "license": "MIT", "preprocessing": "Caller supplies query instruction; tokenizer truncation at 512 tokens; normalized CLS pooling." }- Rerun when
- artifact revision change; runtime version change; fixture or quality-gate change; browser/backend behavior change
Per-case quality & failure details
[
{
"precision": "fp32",
"device": "wasm",
"status": "pass",
"quality": {
"shape": true,
"finite": true,
"normalized": true,
"distinct": true,
"positive": 0.7934246311375007,
"negative": 0.43608810141044896,
"margin": 0.35733652972705177,
"minimumMargin": 0.1,
"grade": "semantic-retrieval-pass"
},
"runtime": {
"loadMs": 20520.30000001192,
"inferenceMs": 72.09999999403954,
"totalMs": 21847.09999999404,
"grade": "excellent",
"embeddingsPerSecond": 41.608876563772654
}
},
{
"precision": "fp32",
"device": "webgpu",
"status": "pass",
"quality": {
"shape": true,
"finite": true,
"normalized": true,
"distinct": true,
"positive": 0.7934244779234982,
"negative": 0.43608809101734064,
"margin": 0.3573363869061576,
"minimumMargin": 0.1,
"grade": "semantic-retrieval-pass"
},
"runtime": {
"loadMs": 19949.69999998808,
"inferenceMs": 69.5,
"totalMs": 21249.69999998808,
"grade": "excellent",
"embeddingsPerSecond": 43.165467625899275
}
},
{
"precision": "fp16",
"device": "wasm",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": null,
"inferenceMs": null,
"totalMs": 15786.300000011921,
"grade": "failed"
},
"failure": "Error: Can't create a session. ERROR_CODE: 1, ERROR_MESSAGE: /mnt/vss/_work/1/s/onnxruntime/core/graph/graph_utils.cc:30 int onnxruntime::graph_utils::GetIndexFromName(const Node &, const std::string &, bool) itr != node_args.end() was false. Attempting to get index by a name which does not exist:InsertedPrecisionFreeCast_/encoder/layer.11/output/LayerNorm/Constant_output_0for node: /embeddings/LayerNorm/Mul/SimplifiedLayerNormFusion/\n"
},
{
"precision": "fp16",
"device": "webgpu",
"status": "pass",
"quality": {
"shape": true,
"finite": true,
"normalized": true,
"distinct": true,
"positive": 0.598378441238317,
"negative": 0.44285379270737396,
"margin": 0.155524648530943,
"minimumMargin": 0.1,
"grade": "semantic-retrieval-pass"
},
"runtime": {
"loadMs": 12284.800000011921,
"inferenceMs": 68.09999999403954,
"totalMs": 13708.40000000596,
"grade": "excellent",
"embeddingsPerSecond": 44.05286343997908
}
},
{
"precision": "int8",
"device": "wasm",
"status": "pass",
"quality": {
"shape": true,
"finite": true,
"normalized": true,
"distinct": true,
"positive": 0.7895325092454644,
"negative": 0.43330974144109247,
"margin": 0.3562227678043719,
"minimumMargin": 0.1,
"grade": "semantic-retrieval-pass"
},
"runtime": {
"loadMs": 9452.09999999404,
"inferenceMs": 81.30000001192093,
"totalMs": 10858.699999988079,
"grade": "excellent",
"embeddingsPerSecond": 36.900368998279376
}
},
{
"precision": "int8",
"device": "webgpu",
"status": "pass",
"quality": {
"shape": true,
"finite": true,
"normalized": true,
"distinct": true,
"positive": 0.7905615544798418,
"negative": 0.4338808018805735,
"margin": 0.3566807525992683,
"minimumMargin": 0.1,
"grade": "semantic-retrieval-pass"
},
"runtime": {
"loadMs": 9614.800000011921,
"inferenceMs": 158.59999999403954,
"totalMs": 11132.699999988079,
"grade": "good",
"embeddingsPerSecond": 18.915510719500286
}
},
{
"precision": "uint8",
"device": "wasm",
"status": "pass",
"quality": {
"shape": true,
"finite": true,
"normalized": true,
"distinct": true,
"positive": 0.7959000266393531,
"negative": 0.42816350597093267,
"margin": 0.36773652066842044,
"minimumMargin": 0.1,
"grade": "semantic-retrieval-pass"
},
"runtime": {
"loadMs": 8721.300000011921,
"inferenceMs": 78.7999999821186,
"totalMs": 10110.5,
"grade": "excellent",
"embeddingsPerSecond": 38.07106599848685
}
},
{
"precision": "uint8",
"device": "webgpu",
"status": "pass",
"quality": {
"shape": true,
"finite": true,
"normalized": true,
"distinct": true,
"positive": 0.7951920390125116,
"negative": 0.42894769632935664,
"margin": 0.3662443426831549,
"minimumMargin": 0.1,
"grade": "semantic-retrieval-pass"
},
"runtime": {
"loadMs": 8747,
"inferenceMs": 157,
"totalMs": 10133.299999982119,
"grade": "good",
"embeddingsPerSecond": 19.10828025477707
}
},
{
"precision": "q8",
"device": "wasm",
"status": "pass",
"quality": {
"shape": true,
"finite": true,
"normalized": true,
"distinct": true,
"positive": 0.789386296060994,
"negative": 0.44365669471073066,
"margin": 0.3457296013502633,
"minimumMargin": 0.1,
"grade": "semantic-retrieval-pass"
},
"runtime": {
"loadMs": 9662.40000000596,
"inferenceMs": 79,
"totalMs": 11021.700000017881,
"grade": "excellent",
"embeddingsPerSecond": 37.974683544303794
}
},
{
"precision": "q8",
"device": "webgpu",
"status": "pass",
"quality": {
"shape": true,
"finite": true,
"normalized": true,
"distinct": true,
"positive": 0.7941647763078531,
"negative": 0.44616234955631195,
"margin": 0.34800242675154114,
"minimumMargin": 0.1,
"grade": "semantic-retrieval-pass"
},
"runtime": {
"loadMs": 9407.899999976158,
"inferenceMs": 150.30000001192093,
"totalMs": 10891.200000017881,
"grade": "good",
"embeddingsPerSecond": 19.96007983873624
}
},
{
"precision": "q4",
"device": "wasm",
"status": "pass",
"quality": {
"shape": true,
"finite": true,
"normalized": true,
"distinct": true,
"positive": 0.8025921206884732,
"negative": 0.4634487855493754,
"margin": 0.3391433351390978,
"minimumMargin": 0.1,
"grade": "semantic-retrieval-pass"
},
"runtime": {
"loadMs": 13145.59999999404,
"inferenceMs": 98.40000000596046,
"totalMs": 14592.199999988079,
"grade": "excellent",
"embeddingsPerSecond": 30.487804876202016
}
},
{
"precision": "q4",
"device": "webgpu",
"status": "pass",
"quality": {
"shape": true,
"finite": true,
"normalized": true,
"distinct": true,
"positive": 0.802592032564747,
"negative": 0.46344865109644134,
"margin": 0.33914338146830564,
"minimumMargin": 0.1,
"grade": "semantic-retrieval-pass"
},
"runtime": {
"loadMs": 12616.600000023842,
"inferenceMs": 66.69999998807907,
"totalMs": 14001.09999999404,
"grade": "excellent",
"embeddingsPerSecond": 44.9775112524164
}
},
{
"precision": "q4f16",
"device": "wasm",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": null,
"inferenceMs": null,
"totalMs": 11522.399999976158,
"grade": "failed"
},
"failure": "Error: Can't create a session. ERROR_CODE: 1, ERROR_MESSAGE: /mnt/vss/_work/1/s/onnxruntime/core/graph/graph_utils.cc:30 int onnxruntime::graph_utils::GetIndexFromName(const Node &, const std::string &, bool) itr != node_args.end() was false. Attempting to get index by a name which does not exist:InsertedPrecisionFreeCast_/embeddings/LayerNorm/Constant_output_0for node: /embeddings/LayerNorm/Mul/SimplifiedLayerNormFusion/\n"
},
{
"precision": "q4f16",
"device": "webgpu",
"status": "pass",
"quality": {
"shape": true,
"finite": true,
"normalized": true,
"distinct": true,
"positive": 0.6466405016170046,
"negative": 0.477640892871366,
"margin": 0.16899960874563857,
"minimumMargin": 0.1,
"grade": "semantic-retrieval-pass"
},
"runtime": {
"loadMs": 9663.59999999404,
"inferenceMs": 60.900000005960464,
"totalMs": 10985.699999988079,
"grade": "excellent",
"embeddingsPerSecond": 49.26108373902104
}
},
{
"precision": "bnb4",
"device": "wasm",
"status": "pass",
"quality": {
"shape": true,
"finite": true,
"normalized": true,
"distinct": true,
"positive": 0.8010671572276266,
"negative": 0.4567326450450716,
"margin": 0.34433451218255495,
"minimumMargin": 0.1,
"grade": "semantic-retrieval-pass"
},
"runtime": {
"loadMs": 12497.700000017881,
"inferenceMs": 86.7999999821186,
"totalMs": 13940.90000000596,
"grade": "excellent",
"embeddingsPerSecond": 34.56221198868687
}
},
{
"precision": "bnb4",
"device": "webgpu",
"status": "pass",
"quality": {
"shape": true,
"finite": true,
"normalized": true,
"distinct": true,
"positive": 0.8010672071515009,
"negative": 0.4567325524965178,
"margin": 0.34433465465498314,
"minimumMargin": 0.1,
"grade": "semantic-retrieval-pass"
},
"runtime": {
"loadMs": 12225.59999999404,
"inferenceMs": 157.7999999821186,
"totalMs": 13678.40000000596,
"grade": "good",
"embeddingsPerSecond": 19.011406846260776
}
}
]