SmolVLM2 256M — Image + Text
28 configurations testedQuality criteria
Nonempty, non-repetitive caption with at least five words and a person label matching the independently specified portrait reference.
| Precision | Device | Result | |
|---|---|---|---|
Q4 | GPU · WebGPUAccelerated | Passed | Fast |
FP32 | GPU · WebGPUAccelerated | Passed | Moderate |
FP32 | GPU · WebGPUAccelerated | Passed | Moderate |
Q4 | GPU · WebGPUAccelerated | Passed | Moderate |
Q4 | GPU · WebGPUAccelerated | Passed | Moderate |
FP32 | GPU · WebGPUAccelerated | Passed | Moderate |
FP32 | GPU · WebGPUAccelerated | Passed | Slow |
FP32 | GPU · WebGPUAccelerated | Passed | Slow |
Q4 | GPU · WebGPUAccelerated | Passed | Slow |
Q4 | GPU · WebGPUAccelerated | Passed | Slow |
Q8 | CPU · WASM | Failed | — |
Q8 | GPU · WebGPU | Failed | — |
Q4 | CPU · WASM | Failed | — |
FP32 | CPU · WASM | Failed | — |
FP16 | CPU · WASM | Failed | — |
FP16 | GPU · WebGPU | Failed | — |
Q4F16 | CPU · WASM | Failed | — |
Q4F16 | GPU · WebGPU | Failed | — |
INT8 | CPU · WASM | Failed | — |
INT8 | GPU · WebGPU | Failed | — |
UINT8 | CPU · WASM | Failed | — |
UINT8 | GPU · WebGPU | Failed | — |
BNB4 | CPU · WASM | Failed | — |
BNB4 | GPU · WebGPU | Failed | — |
FP32 | GPU · WebGPU | Failed | — |
FP32 | GPU · WebGPU | Failed | — |
Q4 | GPU · WebGPU | Failed | — |
Q4 | GPU · WebGPU | Failed | — |
Test setup & sample details
- Quality gate
- Nonempty, non-repetitive caption with at least five words and a person label matching the independently specified portrait reference.
- Tested
- 2026-11-04T17:45:47.085Z
- Environment
- Isolated Chrome 156.0.8078.79 module worker per case on macOS arm64; Transformers.js 4.2.0; cold model cache
- Runtime
- Transformers.js 4.2.0
- Artifact revision
- 067788b187b95ebe7b2e040b3e4299e342e5b8fd
- Fixture
{ "file": "model-samples/lfm2-vl-450m.json", "url": "https://f005.backblazeb2.com/file/hubters-webai-public/assets/benchmarks/lfm2-vl-450m.json", "size": 417, "sha1": "2c1a441093498eea1887511f171471d1bfc608a3" }- Rerun when
- artifact revision change; runtime version change; fixture or quality-gate change; browser/backend behavior change
Per-case quality & failure details
[
{
"precision": "q8",
"device": "wasm",
"status": "fail",
"quality": {
"personMention": true,
"wordCount": 54,
"uniqueRatio": 0.7407407407407407,
"passed": true,
"visualization": true
},
"runtime": {
"loadMs": 1877.9000000059605,
"inferenceMs": 77445.59999999404,
"totalMs": 80712.59999999404,
"grade": "very-slow",
"imagesPerSecond": 0.012912289400560871,
"artifactRevision": "067788b187b95ebe7b2e040b3e4299e342e5b8fd",
"engine": {
"name": "Transformers.js",
"version": "4.2.0"
}
},
"failure": "Quality or 15-second interactive usability gate failed"
},
{
"precision": "q8",
"device": "webgpu",
"status": "fail",
"quality": {
"personMention": true,
"wordCount": 6,
"uniqueRatio": 1,
"passed": true,
"visualization": true
},
"runtime": {
"loadMs": 1975.4000000059605,
"inferenceMs": 64148.19999998808,
"totalMs": 67664.89999997616,
"grade": "very-slow",
"imagesPerSecond": 0.015588901948927418,
"artifactRevision": "067788b187b95ebe7b2e040b3e4299e342e5b8fd",
"engine": {
"name": "Transformers.js",
"version": "4.2.0"
}
},
"failure": "Quality or 15-second interactive usability gate failed"
},
{
"precision": "q4",
"device": "wasm",
"status": "fail",
"quality": {
"personMention": true,
"wordCount": 12,
"uniqueRatio": 0.8333333333333334,
"passed": true,
"visualization": true
},
"runtime": {
"loadMs": 6261.9000000059605,
"inferenceMs": 79937.5,
"totalMs": 88018.69999998808,
"grade": "very-slow",
"imagesPerSecond": 0.012509773260359656,
"artifactRevision": "067788b187b95ebe7b2e040b3e4299e342e5b8fd",
"engine": {
"name": "Transformers.js",
"version": "4.2.0"
}
},
"failure": "Quality or 15-second interactive usability gate failed"
},
{
"precision": "q4",
"device": "webgpu",
"status": "pass",
"quality": {
"personMention": true,
"wordCount": 12,
"uniqueRatio": 0.8333333333333334,
"passed": true,
"visualization": true
},
"runtime": {
"loadMs": 2221.0999999940395,
"inferenceMs": 3561.800000011921,
"totalMs": 7279.800000011921,
"grade": "good",
"imagesPerSecond": 0.2807569206571546,
"artifactRevision": "067788b187b95ebe7b2e040b3e4299e342e5b8fd",
"engine": {
"name": "Transformers.js",
"version": "4.2.0"
}
}
},
{
"precision": "fp32",
"device": "wasm",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": 2378.800000011921,
"inferenceMs": null,
"totalMs": 51758.79999998212,
"grade": "failed"
},
"failure": "generate timeout"
},
{
"precision": "fp32",
"device": "webgpu",
"status": "pass",
"quality": {
"personMention": true,
"wordCount": 16,
"uniqueRatio": 0.8125,
"passed": true,
"visualization": true
},
"runtime": {
"loadMs": 2400.199999988079,
"inferenceMs": 2918.800000011921,
"totalMs": 6831.5,
"grade": "good",
"imagesPerSecond": 0.34260655063584894,
"artifactRevision": "067788b187b95ebe7b2e040b3e4299e342e5b8fd",
"engine": {
"name": "Transformers.js",
"version": "4.2.0"
}
}
},
{
"precision": "fp16",
"device": "wasm",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": null,
"inferenceMs": null,
"totalMs": 3233.9000000059605,
"grade": "failed"
},
"failure": "Can't create a session. ERROR_CODE: 1, ERROR_MESSAGE: /mnt/vss/_work/1/s/onnxruntime/core/graph/graph_utils.cc:30 int onnxruntime::graph_utils::GetIndexFromName(const Node &, const std::string &, bool) itr != node_args.end() was false. Attempting to get index by a name which does not exist:InsertedPrecisionFreeCast_/vision_model/encoder/layers.0/layer_norm1/Constant_output_0for node: /vision_model/encoder/layers.0/layer_norm1/Mul/SimplifiedLayerNormFusion/\n"
},
{
"precision": "fp16",
"device": "webgpu",
"status": "fail",
"quality": {
"personMention": false,
"wordCount": 1,
"uniqueRatio": 1,
"passed": false,
"visualization": true
},
"runtime": {
"loadMs": 2274.2999999821186,
"inferenceMs": 3691.9000000059605,
"totalMs": 7308.199999988079,
"grade": "good",
"imagesPerSecond": 0.2708632411491063,
"artifactRevision": "067788b187b95ebe7b2e040b3e4299e342e5b8fd",
"engine": {
"name": "Transformers.js",
"version": "4.2.0"
}
},
"failure": "Quality or 15-second interactive usability gate failed"
},
{
"precision": "q4f16",
"device": "wasm",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": null,
"inferenceMs": null,
"totalMs": 3380.300000011921,
"grade": "failed"
},
"failure": "Can't create a session. ERROR_CODE: 1, ERROR_MESSAGE: /mnt/vss/_work/1/s/onnxruntime/core/graph/graph_utils.cc:30 int onnxruntime::graph_utils::GetIndexFromName(const Node &, const std::string &, bool) itr != node_args.end() was false. Attempting to get index by a name which does not exist:InsertedPrecisionFreeCast_/vision_model/encoder/layers.0/layer_norm1/Constant_output_0for node: /vision_model/encoder/layers.0/layer_norm1/Mul/SimplifiedLayerNormFusion/\n"
},
{
"precision": "q4f16",
"device": "webgpu",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": 2172.2999999821186,
"inferenceMs": null,
"totalMs": 5247.5999999940395,
"grade": "failed"
},
"failure": "Model returned empty answer"
},
{
"precision": "int8",
"device": "wasm",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": 1975.199999988079,
"inferenceMs": null,
"totalMs": 51337.90000000596,
"grade": "failed"
},
"failure": "generate timeout"
},
{
"precision": "int8",
"device": "webgpu",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": 2039.5999999940395,
"inferenceMs": null,
"totalMs": 48817.69999998808,
"grade": "failed"
},
"failure": "generate timeout"
},
{
"precision": "uint8",
"device": "wasm",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": 1870.5,
"inferenceMs": null,
"totalMs": 51209,
"grade": "failed"
},
"failure": "generate timeout"
},
{
"precision": "uint8",
"device": "webgpu",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": 2088.5,
"inferenceMs": null,
"totalMs": 49201.70000001788,
"grade": "failed"
},
"failure": "generate timeout"
},
{
"precision": "bnb4",
"device": "wasm",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": 2230.7999999821186,
"inferenceMs": null,
"totalMs": 51683.19999998808,
"grade": "failed"
},
"failure": "generate timeout"
},
{
"precision": "bnb4",
"device": "webgpu",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": 2031.2999999821186,
"inferenceMs": null,
"totalMs": 49187.19999998808,
"grade": "failed"
},
"failure": "generate timeout"
},
{
"precision": "fp32",
"device": "webgpu",
"status": "pass",
"quality": {
"passed": true,
"nonEmpty": true,
"gibberish": false,
"grade": "fixture-pass",
"score": 1,
"scope": "Fixed English object/OCR/count, color-order or capital-city fixture; not broad model quality certification."
},
"runtime": {
"loadMs": 42457.889999985695,
"inferenceMs": 3401.4700000286102,
"totalMs": 45859.564999997616,
"wallMs": 3401.675000011921,
"charactersPerSecond": 1.7639432362918188,
"inputModality": "image",
"grade": "slow",
"imagesPerSecond": 0.29399053938196984
}
},
{
"precision": "fp32",
"device": "webgpu",
"status": "pass",
"quality": {
"passed": true,
"nonEmpty": true,
"gibberish": false,
"grade": "fixture-pass",
"score": 1,
"scope": "Fixed English object/OCR/count, color-order or capital-city fixture; not broad model quality certification."
},
"runtime": {
"loadMs": 42457.889999985695,
"inferenceMs": 1709.8950000405312,
"totalMs": 47569.54500001669,
"wallMs": 1709.9700000286102,
"charactersPerSecond": 8.187637252385759,
"inputModality": "image",
"grade": "good",
"imagesPerSecond": 0.5848312323132684
}
},
{
"precision": "fp32",
"device": "webgpu",
"status": "pass",
"quality": {
"passed": true,
"nonEmpty": true,
"gibberish": false,
"grade": "fixture-pass",
"score": 1,
"scope": "Fixed English object/OCR/count, color-order or capital-city fixture; not broad model quality certification."
},
"runtime": {
"loadMs": 42457.889999985695,
"inferenceMs": 1549.5799999833107,
"totalMs": 49119.18000000715,
"wallMs": 1549.625,
"charactersPerSecond": 1.2906723112208085,
"inputModality": "image",
"grade": "slow",
"imagesPerSecond": 0.6453361556104042
}
},
{
"precision": "fp32",
"device": "webgpu",
"status": "fail",
"quality": {
"passed": false,
"nonEmpty": true,
"gibberish": false,
"grade": "failed",
"score": 0,
"scope": "Fixed English object/OCR/count, color-order or capital-city fixture; not broad model quality certification."
},
"runtime": {
"loadMs": 42457.889999985695,
"inferenceMs": 1245.7699999809265,
"totalMs": 50365,
"wallMs": 1245.8100000023842,
"charactersPerSecond": 4.013581961418684,
"inputModality": "video",
"grade": "slow",
"framesPerSecond": 3.2108655691349464
}
},
{
"precision": "fp32",
"device": "webgpu",
"status": "fail",
"quality": {
"passed": false,
"nonEmpty": true,
"gibberish": false,
"grade": "failed",
"score": 0,
"scope": "Fixed English object/OCR/count, color-order or capital-city fixture; not broad model quality certification."
},
"runtime": {
"loadMs": 42457.889999985695,
"inferenceMs": 1147.8550000190735,
"totalMs": 51512.87000000477,
"wallMs": 1147.8650000095367,
"charactersPerSecond": 4.35595088222547,
"inputModality": "video",
"grade": "slow",
"framesPerSecond": 3.484760705780376
}
},
{
"precision": "fp32",
"device": "webgpu",
"status": "pass",
"quality": {
"passed": true,
"nonEmpty": true,
"gibberish": false,
"grade": "fixture-pass",
"score": 1,
"scope": "Fixed English object/OCR/count, color-order or capital-city fixture; not broad model quality certification."
},
"runtime": {
"loadMs": 42457.889999985695,
"inferenceMs": 1257.5649999976158,
"totalMs": 52770.485000014305,
"wallMs": 1257.6150000095367,
"charactersPerSecond": 205.1583814757004,
"inputModality": "text",
"grade": "fast"
}
},
{
"precision": "q4",
"device": "webgpu",
"status": "pass",
"quality": {
"passed": true,
"nonEmpty": true,
"gibberish": false,
"grade": "fixture-pass",
"score": 1,
"scope": "Fixed English object/OCR/count, color-order or capital-city fixture; not broad model quality certification."
},
"runtime": {
"loadMs": 43917.89500004053,
"inferenceMs": 3472.9900000095367,
"totalMs": 47391.11500000954,
"wallMs": 3473.2199999690056,
"charactersPerSecond": 5.75872663035168,
"inputModality": "image",
"grade": "good",
"imagesPerSecond": 0.28793633151758397
}
},
{
"precision": "q4",
"device": "webgpu",
"status": "pass",
"quality": {
"passed": true,
"nonEmpty": true,
"gibberish": false,
"grade": "fixture-pass",
"score": 1,
"scope": "Fixed English object/OCR/count, color-order or capital-city fixture; not broad model quality certification."
},
"runtime": {
"loadMs": 43917.89500004053,
"inferenceMs": 1658.010000050068,
"totalMs": 49049.205000042915,
"wallMs": 1658.085000038147,
"charactersPerSecond": 8.443857395056263,
"inputModality": "image",
"grade": "good",
"imagesPerSecond": 0.6031326710754473
}
},
{
"precision": "q4",
"device": "webgpu",
"status": "pass",
"quality": {
"passed": true,
"nonEmpty": true,
"gibberish": false,
"grade": "fixture-pass",
"score": 1,
"scope": "Fixed English object/OCR/count, color-order or capital-city fixture; not broad model quality certification."
},
"runtime": {
"loadMs": 43917.89500004053,
"inferenceMs": 1580.0400000214577,
"totalMs": 50629.30500000715,
"wallMs": 1580.089999973774,
"charactersPerSecond": 1.26579073945776,
"inputModality": "image",
"grade": "slow",
"imagesPerSecond": 0.6328953697288799
}
},
{
"precision": "q4",
"device": "webgpu",
"status": "fail",
"quality": {
"passed": false,
"nonEmpty": true,
"gibberish": false,
"grade": "failed",
"score": 0,
"scope": "Fixed English object/OCR/count, color-order or capital-city fixture; not broad model quality certification."
},
"runtime": {
"loadMs": 43917.89500004053,
"inferenceMs": 1333.9400000572205,
"totalMs": 51963.29500001669,
"wallMs": 1333.9900000095367,
"charactersPerSecond": 2.9986356206639107,
"inputModality": "video",
"grade": "slow",
"framesPerSecond": 2.9986356206639107
}
},
{
"precision": "q4",
"device": "webgpu",
"status": "fail",
"quality": {
"passed": false,
"nonEmpty": true,
"gibberish": false,
"grade": "failed",
"score": 0,
"scope": "Fixed English object/OCR/count, color-order or capital-city fixture; not broad model quality certification."
},
"runtime": {
"loadMs": 43917.89500004053,
"inferenceMs": 1175.7699999809265,
"totalMs": 53139.080000042915,
"wallMs": 1175.785000026226,
"charactersPerSecond": 3.402025906482465,
"inputModality": "video",
"grade": "slow",
"framesPerSecond": 3.4020259064824656
}
},
{
"precision": "q4",
"device": "webgpu",
"status": "pass",
"quality": {
"passed": true,
"nonEmpty": true,
"gibberish": false,
"grade": "fixture-pass",
"score": 1,
"scope": "Fixed English object/OCR/count, color-order or capital-city fixture; not broad model quality certification."
},
"runtime": {
"loadMs": 43917.89500004053,
"inferenceMs": 104.23500001430511,
"totalMs": 53243.37999999523,
"wallMs": 104.29999995231628,
"charactersPerSecond": 306.99860887042115,
"inputModality": "text",
"grade": "fast"
}
}
]