Open Jev DeBERTa V3 Large — Typed Decisions
8 configurations testedQuality criteria
All checked-in typed questions must return finite normalized distributions, the expected top choice/yes-no decision, and an ordered-score value within the fixture tolerance; outputs must be non-degenerate.
| Precision | Device | Result | |
|---|---|---|---|
Q4F16 | GPU · WebGPUAccelerated | Passed | Fast |
FP32 | GPU · WebGPUAccelerated | Passed | Fast |
Q4 | GPU · WebGPUAccelerated | Passed | Moderate |
FP16 | GPU · WebGPUAccelerated | Passed | Moderate |
FP32 | CPU · WASM | Passed | Moderate |
FP16 | CPU · WASM | Passed | Slow |
Q4 | CPU · WASM | Failed | — |
Q4F16 | CPU · WASM | Failed | — |
Test setup & sample details
- Quality gate
- All checked-in typed questions must return finite normalized distributions, the expected top choice/yes-no decision, and an ordered-score value within the fixture tolerance; outputs must be non-degenerate.
- Tested
- 2026-09-22T14:47:49.157Z
- Environment
- Isolated headless Google Chrome 151 on macOS; Transformers.js 4.2.0
- Runtime
- Transformers.js 4.2.0
- Artifact revision
- 7c79f25b5ac496089f448a969c801872ad59d31c
- Fixture
{ "file": "open-jev-deberta-v3-large.json", "url": "https://f005.backblazeb2.com/file/hubters-webai-public/assets/benchmarks/open-jev-deberta-v3-large.json", "size": 819, "sha1": "bf1e5a03646050ac2c9b0fa51bf8f01b72f95ece", "source": "Adapted from the pinned source model card usage example.", "sourceUrl": "https://huggingface.co/com-kotobalabs/open-jev-deberta-v3-large/blob/19bf9a64815add579fbf6c907bef584d9277a8e4/README.md#use", "license": "Apache-2.0", "licenseUrl": "https://www.apache.org/licenses/LICENSE-2.0.txt", "preprocessing": "Pinned SentencePiece tokenizer; state truncates to 256 tokens; questions and options use the artifact's marker-token span interface." }- Rerun when
- artifact revision change; runtime version change; fixture or quality-gate change; browser/backend behavior change
Per-case quality & failure details
[
{
"precision": "fp32",
"device": "wasm",
"status": "pass",
"quality": {
"normalized": true,
"finite": true,
"expected": true,
"nondegenerate": true,
"grade": "typed-decision-pass"
},
"runtime": {
"loadMs": 182152.3,
"inferenceMs": 1475.1,
"totalMs": 185097.3,
"grade": "slow",
"questionsPerSecond": 2.0338
}
},
{
"precision": "fp32",
"device": "webgpu",
"status": "pass",
"quality": {
"normalized": true,
"finite": true,
"expected": true,
"nondegenerate": true,
"grade": "typed-decision-pass"
},
"runtime": {
"loadMs": 181271.3,
"inferenceMs": 445.4,
"totalMs": 183191.6,
"grade": "good",
"questionsPerSecond": 6.7355
}
},
{
"precision": "fp16",
"device": "wasm",
"status": "pass",
"quality": {
"normalized": true,
"finite": true,
"expected": true,
"nondegenerate": true,
"grade": "typed-decision-pass"
},
"runtime": {
"loadMs": 110955.2,
"inferenceMs": 2716.9,
"totalMs": 115036.4,
"grade": "slow",
"questionsPerSecond": 1.1042
}
},
{
"precision": "fp16",
"device": "webgpu",
"status": "pass",
"quality": {
"normalized": true,
"finite": true,
"expected": true,
"nondegenerate": true,
"grade": "typed-decision-pass"
},
"runtime": {
"loadMs": 97316.4,
"inferenceMs": 965.6,
"totalMs": 99586.3,
"grade": "good",
"questionsPerSecond": 3.1069
}
},
{
"precision": "q4",
"device": "wasm",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": null,
"inferenceMs": null,
"totalMs": 64089.3,
"grade": "failed"
},
"failure": "Could not find a WASM implementation for GatherBlockQuantized(1)."
},
{
"precision": "q4",
"device": "webgpu",
"status": "pass",
"quality": {
"normalized": true,
"finite": true,
"expected": true,
"nondegenerate": true,
"grade": "typed-decision-pass"
},
"runtime": {
"loadMs": 58865.6,
"inferenceMs": 745.3,
"totalMs": 61101.1,
"grade": "good",
"questionsPerSecond": 4.0252
}
},
{
"precision": "q4f16",
"device": "wasm",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": null,
"inferenceMs": null,
"totalMs": 43624.3,
"grade": "failed"
},
"failure": "No CPUExecutionProvider kernel was found for GatherBlockQuantized(1)."
},
{
"precision": "q4f16",
"device": "webgpu",
"status": "pass",
"quality": {
"normalized": true,
"finite": true,
"expected": true,
"nondegenerate": true,
"grade": "typed-decision-pass"
},
"runtime": {
"loadMs": 45219.2,
"inferenceMs": 387.9,
"totalMs": 47054.6,
"grade": "good",
"questionsPerSecond": 7.734
}
}
]