Chatterbox English
8 configurations testedQuality criteria
All three English utterances: ASR word error rate <= 0.35, no repetitive output, finite audible PCM, clipping <= 2%; each real-time factor <= 4.
| Precision | Device | Result | |
|---|---|---|---|
Q4 | GPU · WebGPUAccelerated | Passed | Slow |
FP16 | GPU · WebGPUAccelerated | Passed | Slow |
FP32 | GPU · WebGPUAccelerated | Passed | Slow |
Q4F16 | GPU · WebGPUAccelerated | Passed | Slow |
FP32 | CPU · WASM | Failed | — |
FP16 | CPU · WASM | Failed | — |
Q4 | CPU · WASM | Failed | — |
Q4F16 | CPU · WASM | Failed | — |
Test setup & sample details
- Quality gate
- All three English utterances: ASR word error rate <= 0.35, no repetitive output, finite audible PCM, clipping <= 2%; each real-time factor <= 4.
- Tested
- 2026-11-03T23:11:44.198Z
- Environment
- Apple M3 Max, 96 GiB unified memory, macOS 27.2 (26B118), Chrome 156.0.8078.80; Transformers.js 4.3.2 / ONNX Runtime Web 1.31.0-dev.20260914-8d85527a0; isolated browser per pair; checksum-verified local artifact mirror (not Internet download timing).
- Runtime
- transformers.js 4.3.2
- Artifact revision
- 3cab09af388d3f02bba43443fce88c1f4525ac43
- Fixture
{ "file": "model-samples/desktop/chatterbox-original-text.json", "size": 431, "sha1": "3db95482e0caeec4043332d1496ee162bf9ee399", "sha256": "4337922b782ce58d3e2f35e0d09cd48cc0cded76c09f728d5de4310b207cb7c6", "url": "https://assets.localai.tools/samples/v1/files/4337922b782ce58d3e2f35e0d09cd48cc0cded76c09f728d5de4310b207cb7c6/chatterbox-original-text.json" }- Rerun when
- artifact revision; Transformers.js/ONNX Runtime version; fixture or quality gate; browser/backend behavior
Per-case quality & failure details
[
{
"precision": "fp32",
"device": "wasm",
"status": "fail",
"quality": {
"passed": true,
"scope": "1 completed short English utterances; not voice similarity or multilingual certification.",
"worstWordErrorRate": 0
},
"runtime": {
"loadMs": 52522.80000001192,
"inferenceMs": 52639.59999999404,
"totalMs": 105219.90000000596,
"audioSeconds": 3.04,
"realTimeFactor": 17.31565789473488,
"rms": 0.027834147081959984,
"clippedFraction": 0,
"grade": "slow"
},
"failure": {
"message": "Performance gate: one or more utterances exceed real-time factor 4."
}
},
{
"precision": "fp32",
"device": "webgpu",
"status": "pass",
"quality": {
"passed": true,
"scope": "3 completed short English utterances; not voice similarity or multilingual certification.",
"worstWordErrorRate": 0
},
"runtime": {
"loadMs": 57162.5,
"inferenceMs": 4057.300000011921,
"totalMs": 69304.59999999404,
"audioSeconds": 3.04,
"realTimeFactor": 1.3346381578986581,
"rms": 0.029418089527818762,
"clippedFraction": 0,
"grade": "usable"
}
},
{
"precision": "fp16",
"device": "wasm",
"status": "fail",
"quality": {
"passed": false,
"scope": "0 completed short English utterances; not voice similarity or multilingual certification."
},
"runtime": "Not embedded",
"failure": {
"message": "Error: Can't create a session. ERROR_CODE: 6, ERROR_MESSAGE: std::bad_alloc",
"stack": "Error: Can't create a session. ERROR_CODE: 6, ERROR_MESSAGE: std::bad_alloc\n at ze (https://cdn.jsdelivr.net/npm/@huggingface/transformers@4.3.2:15:31333)\n at pm (https://cdn.jsdelivr.net/npm/@huggingface/transformers@4.3.2:16:22185)\n at async uS.loadModel (https://cdn.jsdelivr.net/npm/@huggingface/transformers@4.3.2:16:35074)\n at async Jd.createInferenceSessionHandler (https://cdn.jsdelivr.net/npm/@huggingface/transformers@4.3.2:16:36795)\n at async RT.create (https://cdn.jsdelivr.net/npm/@huggingface/transformers@4.3.2:11:49080)\n at async iu (https://cdn.jsdelivr.net/npm/@huggingface/transformers@4.3.2:16:53598)\n at async https://cdn.jsdelivr.net/npm/@huggingface/transformers@4.3.2:40:18082\n at async Promise.all (index 2)\n at async cM (https://cdn.jsdelivr.net/npm/@huggingface/transformers@4.3.2:40:17939)\n at async Promise.all (index 0)"
}
},
{
"precision": "fp16",
"device": "webgpu",
"status": "pass",
"quality": {
"passed": true,
"scope": "3 completed short English utterances; not voice similarity or multilingual certification.",
"worstWordErrorRate": 0
},
"runtime": {
"loadMs": 57081.90000000596,
"inferenceMs": 3537.7999999821186,
"totalMs": 68469.20000001788,
"audioSeconds": 2.96,
"realTimeFactor": 1.1952027026966616,
"rms": 0.032839201590761535,
"clippedFraction": 0,
"grade": "usable"
}
},
{
"precision": "q4",
"device": "wasm",
"status": "fail",
"quality": {
"passed": true,
"scope": "3 completed short English utterances; not voice similarity or multilingual certification.",
"worstWordErrorRate": 0
},
"runtime": {
"loadMs": 50207.59999999404,
"inferenceMs": 95549,
"totalMs": 362366.90000000596,
"audioSeconds": 3.08,
"realTimeFactor": 31.0224025974026,
"rms": 0.032904547910073474,
"clippedFraction": 0,
"grade": "slow"
},
"failure": {
"message": "Performance gate: one or more utterances exceed real-time factor 4."
}
},
{
"precision": "q4",
"device": "webgpu",
"status": "pass",
"quality": {
"passed": true,
"scope": "3 completed short English utterances; not voice similarity or multilingual certification.",
"worstWordErrorRate": 0
},
"runtime": {
"loadMs": 57547.90000000596,
"inferenceMs": 3179.0999999940395,
"totalMs": 66384.90000000596,
"audioSeconds": 3.08,
"realTimeFactor": 1.0321753246733893,
"rms": 0.033996771549202796,
"clippedFraction": 0,
"grade": "usable"
}
},
{
"precision": "q4f16",
"device": "wasm",
"status": "fail",
"quality": {
"passed": true,
"scope": "3 completed short English utterances; not voice similarity or multilingual certification.",
"worstWordErrorRate": 0
},
"runtime": {
"loadMs": 50380.80000001192,
"inferenceMs": 95294.5,
"totalMs": 359206,
"audioSeconds": 3.08,
"realTimeFactor": 30.939772727272725,
"rms": 0.03133795468971013,
"clippedFraction": 0,
"grade": "slow"
},
"failure": {
"message": "Performance gate: one or more utterances exceed real-time factor 4."
}
},
{
"precision": "q4f16",
"device": "webgpu",
"status": "pass",
"quality": {
"passed": true,
"scope": "3 completed short English utterances; not voice similarity or multilingual certification.",
"worstWordErrorRate": 0
},
"runtime": {
"loadMs": 59663.09999999404,
"inferenceMs": 5944.200000017881,
"totalMs": 71237,
"audioSeconds": 3.08,
"realTimeFactor": 1.9299350649408704,
"rms": 0.02900927959267544,
"clippedFraction": 0,
"grade": "usable"
}
}
]