HY-MT1.5 1.8B — Translation
8 configurations testedQuality criteria
{ "metric": "best-reference character-ngram F2 orders1–6", "minimumScore": 0.55, "requireSemanticAnchors": true, "rejectRepetitive": true, "scope": "All five authored cases in four accepted directions; Hebrew is rejected by validation after failing the retained broader diagnostic." }
| Precision | Device | Result | |
|---|---|---|---|
Q4F16 | GPU · WebGPUAccelerated | Passed | Moderate |
Q4 | GPU · WebGPUAccelerated | Passed | Moderate |
Q4 | CPU · WASM | Failed | — |
Q4F16 | CPU · WASM | Failed | — |
FP16 | GPU · WebGPU | Failed | — |
FP16 | CPU · WASM | Failed | — |
FP32 | GPU · WebGPU | Failed | — |
FP32 | CPU · WASM | Failed | — |
Test setup & sample details
- Quality gate
- { "metric": "best-reference character-ngram F2 orders1–6", "minimumScore": 0.55, "requireSemanticAnchors": true, "rejectRepetitive": true, "scope": "All five authored cases in four accepted directions; Hebrew is rejected by validation after failing the retained broader diagnostic." }
- Tested
- 2026-11-02T15:19:01.046Z
- Environment
- { "browser": "156.0.8078.79", "os": "Darwin", "release": "27.2.0", "arch": "arm64", "cpu": "Apple M3 Max", "runtime": { "name": "Transformers.js", "version": "4.2.0" }, "workerType": "module", "cache": "fresh disk-backed profile per case", "artifactTransport": "checksum-verified pinned local mirror", "wasmThreads": 1 }
- Runtime
- Transformers.js 4.2.0
- Artifact revision
- 2f11819b25de08cecd344735cdfa5136ade41a67
- Fixture
{ "file": "model-samples/hunyuan-translation-browser.json", "url": "https://assets.localai.tools/samples/v1/files/858fa49a0d90bb76dcec9f86febec9536ff304d28321361f78239a153b011d8c/hunyuan-translation-browser.json", "size": 2810, "sha1": "b2b4c977b0cf11da74177cb838e7aa9258092bad", "sha256": "858fa49a0d90bb76dcec9f86febec9536ff304d28321361f78239a153b011d8c" }- Rerun when
- model artifact revision change; runtime version change; fixture or quality-gate change; backend/platform behavior change
Per-case quality & failure details
[
{
"precision": "q4",
"device": "webgpu",
"status": "pass",
"quality": {
"passed": true,
"meanCharacterF2": 1,
"cases": [
{
"fixtureId": "standard",
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
{
"fixtureId": "negation",
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
{
"fixtureId": "reverse",
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
{
"fixtureId": "french",
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
{
"fixtureId": "chinese",
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
}
]
},
"runtime": {
"loadMs": 3554.7999999970198,
"inferenceMs": 1367.5999999940395,
"totalMs": 6446.70000000298,
"grade": "excellent",
"charactersPerSecond": 135.2734717759625
}
},
{
"precision": "q4",
"device": "wasm",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": null,
"inferenceMs": null,
"totalMs": 5751.70000000298,
"grade": "failed"
},
"failure": "Can't create a session. ERROR_CODE: 9, ERROR_MESSAGE: Could not find an implementation for GatherBlockQuantized(1) node with name '/model/embed_tokens/Gather_Quant'"
},
{
"precision": "q4f16",
"device": "webgpu",
"status": "pass",
"quality": {
"passed": true,
"meanCharacterF2": 1,
"cases": [
{
"fixtureId": "standard",
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
{
"fixtureId": "negation",
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
{
"fixtureId": "reverse",
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
{
"fixtureId": "french",
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
{
"fixtureId": "chinese",
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
}
]
},
"runtime": {
"loadMs": 5026.20000000298,
"inferenceMs": 1328.3999999910593,
"totalMs": 7622.600000008941,
"grade": "excellent",
"charactersPerSecond": 139.26528154264162
}
},
{
"precision": "q4f16",
"device": "wasm",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": null,
"inferenceMs": null,
"totalMs": 5566.4000000059605,
"grade": "failed"
},
"failure": "Can't create a session. ERROR_CODE: 9, ERROR_MESSAGE: Failed to find kernel for com.microsoft.GatherBlockQuantized(1) (node:'/model/embed_tokens/Gather_Quant' ep:'CPUExecutionProvider'). Kernel not found"
},
{
"precision": "fp16",
"device": "webgpu",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": null,
"inferenceMs": null,
"totalMs": 6888.399999991059,
"grade": "failed"
},
"failure": "Can't create a session. ERROR_CODE: 1, ERROR_MESSAGE: Deserialize tensor model.layers.22.attn.o_proj.MatMul.weight failed.Failed to load external data file \"\"model_fp16.onnx_data_1\"\", error: File not found in preloaded files."
},
{
"precision": "fp16",
"device": "wasm",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": null,
"inferenceMs": null,
"totalMs": 7292.699999988079,
"grade": "failed"
},
"failure": "Can't create a session. ERROR_CODE: 1, ERROR_MESSAGE: /mnt/vss/_work/1/s/onnxruntime/core/optimizer/initializer.cc:51 onnxruntime::Initializer::Initializer(const onnx::TensorProto &, const std::filesystem::path &) [ONNXRuntimeError] : 1 : FAIL : Failed to load external data file \"\"model_fp16.onnx_data_1\"\", error: File not found in preloaded files.\n"
},
{
"precision": "fp32",
"device": "webgpu",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": null,
"inferenceMs": null,
"totalMs": 7060.300000011921,
"grade": "failed"
},
"failure": "Can't create a session. ERROR_CODE: 1, ERROR_MESSAGE: Deserialize tensor model.layers.6.mlp.down_proj.MatMul.weight failed.Failed to load external data file \"\"model.onnx_data_1\"\", error: File not found in preloaded files."
},
{
"precision": "fp32",
"device": "wasm",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": null,
"inferenceMs": null,
"totalMs": 6775.9000000059605,
"grade": "failed"
},
"failure": "Can't create a session. ERROR_CODE: 1, ERROR_MESSAGE: Deserialize tensor model.layers.11.mlp.gate_proj.MatMul.weight failed.Failed to load external data file \"\"model.onnx_data_1\"\", error: File not found in preloaded files."
}
]