Sweep Next-Edit 1.5B — ONNX
10 configurations testedQuality criteria
No explicit quality gate recorded. Review per-case evidence; full certification is pending.
| Precision | Device | Result | |
|---|---|---|---|
Q4 | GPU · WebGPUAccelerated | Passed | Moderate |
FP32 | CPU · WASM | Failed | — |
FP32 | GPU · WebGPU | Failed | — |
FP16 | CPU · WASM | Failed | — |
FP16 | GPU · WebGPU | Failed | — |
Q8 | CPU · WASM | Failed | — |
Q8 | GPU · WebGPU | Failed | — |
Q4F16 | CPU · WASM | Failed | — |
Q4F16 | GPU · WebGPU | Failed | — |
Q4 | CPU · WASM | Failed | — |
Test setup & sample details
- Quality gate
- No explicit quality gate recorded. Review the per-case quality evidence; full certification is pending.
- Tested
- 2026-11-04T19:18:16.527Z
- Environment
- Chrome 151.0.7922.176 on macOS arm64; Transformers.js 4.2.0
- Runtime
- Transformers.js 4.2.0
- Artifact revision
- 402c437f67cee450c00d3add669b43ae79e55e50
- Fixture
{ "file": "sweep-next-edit-1.5b.json", "url": "https://f005.backblazeb2.com/file/hubters-webai-public/assets/benchmarks/sweep-next-edit-1.5b.json", "size": 1291, "sha1": "98a29ff43b6bdcd2faba07f9ba47bb9d9ed36192" }- Rerun when
- model artifact revision change; Transformers.js or ONNX Runtime version change; fixture or quality-gate change; browser/backend behavior change
Per-case quality & failure details
[
{
"precision": "fp32",
"device": "wasm",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": null,
"inferenceMs": null,
"totalMs": 7676,
"grade": "failed",
"firstChunkMs": null,
"outputTokens": null,
"tokensPerSecond": null
},
"failure": "Error: Can't create a session. ERROR_CODE: 1, ERROR_MESSAGE: Deserialize tensor model.layers.7.input_layernorm.weight failed.Failed to load external data file \"\"model.onnx_data\"\", error: Module.MountedFiles is not available."
},
{
"precision": "fp32",
"device": "webgpu",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": null,
"inferenceMs": null,
"totalMs": 7366.100000023842,
"grade": "failed",
"firstChunkMs": null,
"outputTokens": null,
"tokensPerSecond": null
},
"failure": "Error: Can't create a session. ERROR_CODE: 1, ERROR_MESSAGE: Deserialize tensor model.layers.11.mlp.up_proj.MatMul.weight failed.Failed to load external data file \"\"model.onnx_data_1\"\", error: Module.MountedFiles is not available."
},
{
"precision": "fp16",
"device": "wasm",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": null,
"inferenceMs": null,
"totalMs": 7499.699999928474,
"grade": "failed",
"firstChunkMs": null,
"outputTokens": null,
"tokensPerSecond": null
},
"failure": "Error: Can't create a session. ERROR_CODE: 1, ERROR_MESSAGE: /mnt/vss/_work/1/s/onnxruntime/core/optimizer/initializer.cc:51 onnxruntime::Initializer::Initializer(const onnx::TensorProto &, const std::filesystem::path &) [ONNXRuntimeError] : 1 : FAIL : Failed to load external data file \"\"model_fp16.onnx_data\"\", error: Module.MountedFiles is not available.\n"
},
{
"precision": "fp16",
"device": "webgpu",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": null,
"inferenceMs": null,
"totalMs": 7566.5,
"grade": "failed",
"firstChunkMs": null,
"outputTokens": null,
"tokensPerSecond": null
},
"failure": "Error: Can't create a session. ERROR_CODE: 1, ERROR_MESSAGE: Deserialize tensor model.layers.14.mlp.up_proj.MatMul.weight failed.Failed to load external data file \"\"model_fp16.onnx_data\"\", error: Module.MountedFiles is not available."
},
{
"precision": "q8",
"device": "wasm",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": null,
"inferenceMs": null,
"totalMs": 7987.799999952316,
"grade": "failed",
"firstChunkMs": null,
"outputTokens": null,
"tokensPerSecond": null
},
"failure": "Error: Can't create a session. ERROR_CODE: 9, ERROR_MESSAGE: Could not find an implementation for GatherBlockQuantized(1) node with name '/model/embed_tokens/Gather_Quant'"
},
{
"precision": "q8",
"device": "webgpu",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": null,
"inferenceMs": null,
"totalMs": 7678.300000071526,
"grade": "failed",
"firstChunkMs": null,
"outputTokens": null,
"tokensPerSecond": null
},
"failure": "Error: Can't create a session. ERROR_CODE: 1, ERROR_MESSAGE: Deserialize tensor model_layers_14_attn_q_proj_MatMul_weight_zp failed.Failed to load external data file \"\"model_quantized.onnx_data\"\", error: Module.MountedFiles is not available."
},
{
"precision": "q4f16",
"device": "wasm",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": null,
"inferenceMs": null,
"totalMs": 7555.699999928474,
"grade": "failed",
"firstChunkMs": null,
"outputTokens": null,
"tokensPerSecond": null
},
"failure": "Error: Can't create a session. ERROR_CODE: 1, ERROR_MESSAGE: /mnt/vss/_work/1/s/onnxruntime/core/optimizer/initializer.cc:51 onnxruntime::Initializer::Initializer(const onnx::TensorProto &, const std::filesystem::path &) [ONNXRuntimeError] : 1 : FAIL : Failed to load external data file \"\"model_q4f16.onnx_data\"\", error: Module.MountedFiles is not available.\n"
},
{
"precision": "q4f16",
"device": "webgpu",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": null,
"inferenceMs": null,
"totalMs": 7826.200000047684,
"grade": "failed",
"firstChunkMs": null,
"outputTokens": null,
"tokensPerSecond": null
},
"failure": "Error: Can't create a session. ERROR_CODE: 1, ERROR_MESSAGE: Deserialize tensor model_layers_27_mlp_down_proj_MatMul_weight_zp failed.Failed to load external data file \"\"model_q4f16.onnx_data\"\", error: Module.MountedFiles is not available."
},
{
"precision": "q4",
"device": "wasm",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": null,
"inferenceMs": null,
"totalMs": 153972.90000000596,
"grade": "failed"
},
"failure": "Error: [WorkerError:runtime_error] init: Can't create a session. ERROR_CODE: 9, ERROR_MESSAGE: Could not find an implementation for GatherBlockQuantized(1) node with name '/model/embed_tokens/Gather_Quant'"
},
{
"precision": "q4",
"device": "webgpu",
"status": "pass",
"quality": {
"passed": true,
"requiredCodeRetained": true,
"expectedEdit": true,
"repeatedLineRatio": 0
},
"runtime": {
"loadMs": 96958.09999999404,
"inferenceMs": 836.5,
"totalMs": 97794.59999999404,
"charactersPerSecond": 141.06395696353854,
"grade": "good"
}
}
]