OPUS-MT · English to Spanish
16 configurations testedQuality criteria
All four translations nonempty and non-repetitive; per-case character n-gram F2 (orders 1–6, Unicode-normalized) >=0.55 against independent references, mean >=0.70. Preserve travel time eight/ocho and negation-free sentence meaning through reference overlap.
| Precision | Device | Result | |
|---|---|---|---|
Q4 | GPU · WebGPUAccelerated | Passed | Moderate |
FP32 | GPU · WebGPUAccelerated | Passed | Moderate |
FP16 | GPU · WebGPUAccelerated | Passed | Moderate |
INT8 | GPU · WebGPUAccelerated | Passed | Moderate |
UINT8 | GPU · WebGPUAccelerated | Passed | Moderate |
Q8 | GPU · WebGPUAccelerated | Passed | Moderate |
BNB4 | GPU · WebGPUAccelerated | Passed | Moderate |
FP32 | CPU · WASM | Passed | Fast |
BNB4 | CPU · WASM | Passed | Moderate |
Q4 | CPU · WASM | Passed | Moderate |
FP16 | CPU · WASM | Failed | — |
INT8 | CPU · WASM | Failed | — |
UINT8 | CPU · WASM | Failed | — |
Q8 | CPU · WASM | Failed | — |
Q4F16 | CPU · WASM | Failed | — |
Q4F16 | GPU · WebGPU | Failed | — |
Test setup & sample details
- Quality gate
- All four translations nonempty and non-repetitive; per-case character n-gram F2 (orders 1–6, Unicode-normalized) >=0.55 against independent references, mean >=0.70. Preserve travel time eight/ocho and negation-free sentence meaning through reference overlap.
- Tested
- 2026-10-31T16:05:37.902Z
- Environment
- Isolated Chrome 156.0.8078.79 module Web Worker per case on macOS arm64; Transformers.js 4.2.0; cold model cache; single-case inference
- Runtime
- Transformers.js 4.2.0
- Artifact revision
- 4b002a4c7edd54a7ced58877258b87f7efd3f892
- Fixture
{ "file": "model-samples/opus-mt-en-es.json", "url": "https://f005.backblazeb2.com/file/hubters-webai-public/assets/benchmarks/opus-mt-en-es.json", "size": 788, "sha1": "6d7e2fff178ede20ba545d0414ad9bc19c93b19f" }- Rerun when
- artifact revision change; runtime version change; fixture or quality-gate change; browser/backend behavior change
Per-case quality & failure details
[
{
"precision": "fp32",
"device": "wasm",
"status": "pass",
"quality": {
"cases": [
{
"source": "Hello, how are you?",
"references": [
"Hola, ¿cómo estás?",
"Hola, ¿cómo está usted?"
],
"output": {
"translation_text": "Hola, ¿cómo estás?",
"source_language": "en",
"target_language": "es"
},
"characterF2": 1,
"nonempty": true,
"repetitive": false,
"passed": true
},
{
"source": "The train leaves at eight in the morning.",
"references": [
"El tren sale a las ocho de la mañana."
],
"output": {
"translation_text": "El tren sale a las ocho de la mañana.",
"source_language": "en",
"target_language": "es"
},
"characterF2": 1,
"nonempty": true,
"repetitive": false,
"passed": true
},
{
"source": "Please close the door.",
"references": [
"Por favor, cierra la puerta.",
"Por favor, cierre la puerta."
],
"output": {
"translation_text": "Por favor, cierra la puerta.",
"source_language": "en",
"target_language": "es"
},
"characterF2": 1,
"nonempty": true,
"repetitive": false,
"passed": true
},
{
"source": "I would like a glass of water.",
"references": [
"Me gustaría un vaso de agua.",
"Quisiera un vaso de agua."
],
"output": {
"translation_text": "Me gustaría un vaso de agua.",
"source_language": "en",
"target_language": "es"
},
"characterF2": 1,
"nonempty": true,
"repetitive": false,
"passed": true
}
],
"meanCharacterF2": 1,
"passed": true
},
"runtime": {
"loadMs": 53798.59999999404,
"inferenceMs": 244.7000000178814,
"totalMs": 55476.40000000596,
"grade": "excellent",
"charactersPerSecond": 457.70331014227884
}
},
{
"precision": "fp32",
"device": "webgpu",
"status": "pass",
"quality": {
"cases": [
{
"source": "Hello, how are you?",
"references": [
"Hola, ¿cómo estás?",
"Hola, ¿cómo está usted?"
],
"output": {
"translation_text": "Hola, ¿cómo estás?",
"source_language": "en",
"target_language": "es"
},
"characterF2": 1,
"nonempty": true,
"repetitive": false,
"passed": true
},
{
"source": "The train leaves at eight in the morning.",
"references": [
"El tren sale a las ocho de la mañana."
],
"output": {
"translation_text": "El tren sale a las ocho de la mañana.",
"source_language": "en",
"target_language": "es"
},
"characterF2": 1,
"nonempty": true,
"repetitive": false,
"passed": true
},
{
"source": "Please close the door.",
"references": [
"Por favor, cierra la puerta.",
"Por favor, cierre la puerta."
],
"output": {
"translation_text": "Por favor, cierra la puerta.",
"source_language": "en",
"target_language": "es"
},
"characterF2": 1,
"nonempty": true,
"repetitive": false,
"passed": true
},
{
"source": "I would like a glass of water.",
"references": [
"Me gustaría un vaso de agua.",
"Quisiera un vaso de agua."
],
"output": {
"translation_text": "Me gustaría un vaso de agua.",
"source_language": "en",
"target_language": "es"
},
"characterF2": 1,
"nonempty": true,
"repetitive": false,
"passed": true
}
],
"meanCharacterF2": 1,
"passed": true
},
"runtime": {
"loadMs": 66004.5,
"inferenceMs": 814.0999999940395,
"totalMs": 68272.80000001192,
"grade": "excellent",
"charactersPerSecond": 137.57523645844492
}
},
{
"precision": "fp16",
"device": "wasm",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": null,
"inferenceMs": null,
"totalMs": 30863,
"grade": "failed"
},
"failure": "Can't create a session. ERROR_CODE: 1, ERROR_MESSAGE: /mnt/vss/_work/1/s/onnxruntime/core/graph/graph_utils.cc:30 int onnxruntime::graph_utils::GetIndexFromName(const Node &, const std::string &, bool) itr != node_args.end() was false. Attempting to get index by a name which does not exist:InsertedPrecisionFreeCast_/layers.5/self_attn_layer_norm/Constant_output_0for node: /layers.0/self_attn_layer_norm/Mul/SimplifiedLayerNormFusion/\n"
},
{
"precision": "fp16",
"device": "webgpu",
"status": "pass",
"quality": {
"cases": [
{
"source": "Hello, how are you?",
"references": [
"Hola, ¿cómo estás?",
"Hola, ¿cómo está usted?"
],
"output": {
"translation_text": "Hola, ¿cómo estás?",
"source_language": "en",
"target_language": "es"
},
"characterF2": 1,
"nonempty": true,
"repetitive": false,
"passed": true
},
{
"source": "The train leaves at eight in the morning.",
"references": [
"El tren sale a las ocho de la mañana."
],
"output": {
"translation_text": "El tren sale a las ocho de la mañana.",
"source_language": "en",
"target_language": "es"
},
"characterF2": 1,
"nonempty": true,
"repetitive": false,
"passed": true
},
{
"source": "Please close the door.",
"references": [
"Por favor, cierra la puerta.",
"Por favor, cierre la puerta."
],
"output": {
"translation_text": "Cierre la puerta. Cierre la puerta.",
"source_language": "en",
"target_language": "es"
},
"characterF2": 0.5745801503805409,
"nonempty": true,
"repetitive": false,
"passed": true
},
{
"source": "I would like a glass of water.",
"references": [
"Me gustaría un vaso de agua.",
"Quisiera un vaso de agua."
],
"output": {
"translation_text": "Me gustaría un vaso de agua.",
"source_language": "en",
"target_language": "es"
},
"characterF2": 1,
"nonempty": true,
"repetitive": false,
"passed": true
}
],
"meanCharacterF2": 0.8936450375951352,
"passed": true
},
"runtime": {
"loadMs": 31592.19999998808,
"inferenceMs": 921.5,
"totalMs": 34364,
"grade": "excellent",
"charactersPerSecond": 121.54096581660336
}
},
{
"precision": "int8",
"device": "wasm",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": null,
"inferenceMs": null,
"totalMs": 19172.5,
"grade": "failed"
},
"failure": "Can't create a session. ERROR_CODE: 1, ERROR_MESSAGE: qdq_actions.cc:137 TransposeDQWeightsForMatMulNBits Missing required scale: model.shared.weight_merged_0_scale for node: model.shared.weight_transposed_DequantizeLinear"
},
{
"precision": "int8",
"device": "webgpu",
"status": "pass",
"quality": {
"cases": [
{
"source": "Hello, how are you?",
"references": [
"Hola, ¿cómo estás?",
"Hola, ¿cómo está usted?"
],
"output": {
"translation_text": "Hola, ¿cómo estás?",
"source_language": "en",
"target_language": "es"
},
"characterF2": 1,
"nonempty": true,
"repetitive": false,
"passed": true
},
{
"source": "The train leaves at eight in the morning.",
"references": [
"El tren sale a las ocho de la mañana."
],
"output": {
"translation_text": "El tren sale a las ocho de la mañana.",
"source_language": "en",
"target_language": "es"
},
"characterF2": 1,
"nonempty": true,
"repetitive": false,
"passed": true
},
{
"source": "Please close the door.",
"references": [
"Por favor, cierra la puerta.",
"Por favor, cierre la puerta."
],
"output": {
"translation_text": "Por favor, cierra la puerta.",
"source_language": "en",
"target_language": "es"
},
"characterF2": 1,
"nonempty": true,
"repetitive": false,
"passed": true
},
{
"source": "I would like a glass of water.",
"references": [
"Me gustaría un vaso de agua.",
"Quisiera un vaso de agua."
],
"output": {
"translation_text": "Me gustaría un vaso de agua.",
"source_language": "en",
"target_language": "es"
},
"characterF2": 1,
"nonempty": true,
"repetitive": false,
"passed": true
}
],
"meanCharacterF2": 1,
"passed": true
},
"runtime": {
"loadMs": 21196.69999998808,
"inferenceMs": 1362.199999988079,
"totalMs": 24136.59999999404,
"grade": "good",
"charactersPerSecond": 82.21993833576578
}
},
{
"precision": "uint8",
"device": "wasm",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": null,
"inferenceMs": null,
"totalMs": 21459.69999998808,
"grade": "failed"
},
"failure": "Can't create a session. ERROR_CODE: 1, ERROR_MESSAGE: qdq_actions.cc:137 TransposeDQWeightsForMatMulNBits Missing required scale: model.shared.weight_merged_0_scale for node: model.shared.weight_transposed_DequantizeLinear"
},
{
"precision": "uint8",
"device": "webgpu",
"status": "pass",
"quality": {
"cases": [
{
"source": "Hello, how are you?",
"references": [
"Hola, ¿cómo estás?",
"Hola, ¿cómo está usted?"
],
"output": {
"translation_text": "Hola, ¿cómo estás?",
"source_language": "en",
"target_language": "es"
},
"characterF2": 1,
"nonempty": true,
"repetitive": false,
"passed": true
},
{
"source": "The train leaves at eight in the morning.",
"references": [
"El tren sale a las ocho de la mañana."
],
"output": {
"translation_text": "El tren sale a las ocho de la mañana.",
"source_language": "en",
"target_language": "es"
},
"characterF2": 1,
"nonempty": true,
"repetitive": false,
"passed": true
},
{
"source": "Please close the door.",
"references": [
"Por favor, cierra la puerta.",
"Por favor, cierre la puerta."
],
"output": {
"translation_text": "Por favor, cierra la puerta.",
"source_language": "en",
"target_language": "es"
},
"characterF2": 1,
"nonempty": true,
"repetitive": false,
"passed": true
},
{
"source": "I would like a glass of water.",
"references": [
"Me gustaría un vaso de agua.",
"Quisiera un vaso de agua."
],
"output": {
"translation_text": "Me gustaría un vaso de agua.",
"source_language": "en",
"target_language": "es"
},
"characterF2": 1,
"nonempty": true,
"repetitive": false,
"passed": true
}
],
"meanCharacterF2": 1,
"passed": true
},
"runtime": {
"loadMs": 17090.59999999404,
"inferenceMs": 1375.5,
"totalMs": 19849.19999998808,
"grade": "good",
"charactersPerSecond": 81.42493638676845
}
},
{
"precision": "q8",
"device": "wasm",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": null,
"inferenceMs": null,
"totalMs": 19074.59999999404,
"grade": "failed"
},
"failure": "Can't create a session. ERROR_CODE: 1, ERROR_MESSAGE: qdq_actions.cc:137 TransposeDQWeightsForMatMulNBits Missing required scale: model.shared.weight_merged_0_scale for node: model.shared.weight_transposed_DequantizeLinear"
},
{
"precision": "q8",
"device": "webgpu",
"status": "pass",
"quality": {
"cases": [
{
"source": "Hello, how are you?",
"references": [
"Hola, ¿cómo estás?",
"Hola, ¿cómo está usted?"
],
"output": {
"translation_text": "Hola, ¿cómo estás?",
"source_language": "en",
"target_language": "es"
},
"characterF2": 1,
"nonempty": true,
"repetitive": false,
"passed": true
},
{
"source": "The train leaves at eight in the morning.",
"references": [
"El tren sale a las ocho de la mañana."
],
"output": {
"translation_text": "El tren sale a las ocho de la mañana.",
"source_language": "en",
"target_language": "es"
},
"characterF2": 1,
"nonempty": true,
"repetitive": false,
"passed": true
},
{
"source": "Please close the door.",
"references": [
"Por favor, cierra la puerta.",
"Por favor, cierre la puerta."
],
"output": {
"translation_text": "Por favor, cierra la puerta.",
"source_language": "en",
"target_language": "es"
},
"characterF2": 1,
"nonempty": true,
"repetitive": false,
"passed": true
},
{
"source": "I would like a glass of water.",
"references": [
"Me gustaría un vaso de agua.",
"Quisiera un vaso de agua."
],
"output": {
"translation_text": "Me gustaría un vaso de agua.",
"source_language": "en",
"target_language": "es"
},
"characterF2": 1,
"nonempty": true,
"repetitive": false,
"passed": true
}
],
"meanCharacterF2": 1,
"passed": true
},
"runtime": {
"loadMs": 19168.100000023842,
"inferenceMs": 1402.0999999940395,
"totalMs": 21996.09999999404,
"grade": "good",
"charactersPerSecond": 79.88017973074398
}
},
{
"precision": "q4",
"device": "wasm",
"status": "pass",
"quality": {
"cases": [
{
"source": "Hello, how are you?",
"references": [
"Hola, ¿cómo estás?",
"Hola, ¿cómo está usted?"
],
"output": {
"translation_text": "Hola, ¿cómo estás?",
"source_language": "en",
"target_language": "es"
},
"characterF2": 1,
"nonempty": true,
"repetitive": false,
"passed": true
},
{
"source": "The train leaves at eight in the morning.",
"references": [
"El tren sale a las ocho de la mañana."
],
"output": {
"translation_text": "El tren sale a las ocho de la mañana.",
"source_language": "en",
"target_language": "es"
},
"characterF2": 1,
"nonempty": true,
"repetitive": false,
"passed": true
},
{
"source": "Please close the door.",
"references": [
"Por favor, cierra la puerta.",
"Por favor, cierre la puerta."
],
"output": {
"translation_text": "Por favor, cierra la puerta.",
"source_language": "en",
"target_language": "es"
},
"characterF2": 1,
"nonempty": true,
"repetitive": false,
"passed": true
},
{
"source": "I would like a glass of water.",
"references": [
"Me gustaría un vaso de agua.",
"Quisiera un vaso de agua."
],
"output": {
"translation_text": "Me gustaría un vaso de agua.",
"source_language": "en",
"target_language": "es"
},
"characterF2": 1,
"nonempty": true,
"repetitive": false,
"passed": true
}
],
"meanCharacterF2": 1,
"passed": true
},
"runtime": {
"loadMs": 37645.90000000596,
"inferenceMs": 1022.7000000178814,
"totalMs": 39966.79999998212,
"grade": "excellent",
"charactersPerSecond": 109.51403148336925
}
},
{
"precision": "q4",
"device": "webgpu",
"status": "pass",
"quality": {
"cases": [
{
"source": "Hello, how are you?",
"references": [
"Hola, ¿cómo estás?",
"Hola, ¿cómo está usted?"
],
"output": {
"translation_text": "Hola, ¿cómo estás?",
"source_language": "en",
"target_language": "es"
},
"characterF2": 1,
"nonempty": true,
"repetitive": false,
"passed": true
},
{
"source": "The train leaves at eight in the morning.",
"references": [
"El tren sale a las ocho de la mañana."
],
"output": {
"translation_text": "El tren sale a las ocho de la mañana.",
"source_language": "en",
"target_language": "es"
},
"characterF2": 1,
"nonempty": true,
"repetitive": false,
"passed": true
},
{
"source": "Please close the door.",
"references": [
"Por favor, cierra la puerta.",
"Por favor, cierre la puerta."
],
"output": {
"translation_text": "Por favor, cierra la puerta.",
"source_language": "en",
"target_language": "es"
},
"characterF2": 1,
"nonempty": true,
"repetitive": false,
"passed": true
},
{
"source": "I would like a glass of water.",
"references": [
"Me gustaría un vaso de agua.",
"Quisiera un vaso de agua."
],
"output": {
"translation_text": "Me gustaría un vaso de agua.",
"source_language": "en",
"target_language": "es"
},
"characterF2": 1,
"nonempty": true,
"repetitive": false,
"passed": true
}
],
"meanCharacterF2": 1,
"passed": true
},
"runtime": {
"loadMs": 37181.69999998808,
"inferenceMs": 540.9000000059605,
"totalMs": 39288.69999998808,
"grade": "excellent",
"charactersPerSecond": 207.06230356584544
}
},
{
"precision": "q4f16",
"device": "wasm",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": null,
"inferenceMs": null,
"totalMs": 25393.20000001788,
"grade": "failed"
},
"failure": "Can't create a session. ERROR_CODE: 1, ERROR_MESSAGE: /mnt/vss/_work/1/s/onnxruntime/core/graph/graph_utils.cc:30 int onnxruntime::graph_utils::GetIndexFromName(const Node &, const std::string &, bool) itr != node_args.end() was false. Attempting to get index by a name which does not exist:InsertedPrecisionFreeCast_/layers.0/self_attn_layer_norm/Constant_output_0for node: /layers.0/self_attn_layer_norm/Mul/SimplifiedLayerNormFusion/\n"
},
{
"precision": "q4f16",
"device": "webgpu",
"status": "fail",
"quality": {
"cases": [
{
"source": "Hello, how are you?",
"references": [
"Hola, ¿cómo estás?",
"Hola, ¿cómo está usted?"
],
"output": {
"translation_text": "Hola, ¿cómo estás?",
"source_language": "en",
"target_language": "es"
},
"characterF2": 1,
"nonempty": true,
"repetitive": false,
"passed": true
},
{
"source": "The train leaves at eight in the morning.",
"references": [
"El tren sale a las ocho de la mañana."
],
"output": {
"translation_text": "El tren sale a las ocho de la mañana.",
"source_language": "en",
"target_language": "es"
},
"characterF2": 1,
"nonempty": true,
"repetitive": false,
"passed": true
},
{
"source": "Please close the door.",
"references": [
"Por favor, cierra la puerta.",
"Por favor, cierre la puerta."
],
"output": {
"translation_text": "Cierre la puerta. Cierre la puerta. Cierre la puerta.",
"source_language": "en",
"target_language": "es"
},
"characterF2": 0.5079151291702035,
"nonempty": true,
"repetitive": false,
"passed": false
},
{
"source": "I would like a glass of water.",
"references": [
"Me gustaría un vaso de agua.",
"Quisiera un vaso de agua."
],
"output": {
"translation_text": "Me gustaría un vaso de agua.",
"source_language": "en",
"target_language": "es"
},
"characterF2": 1,
"nonempty": true,
"repetitive": false,
"passed": true
}
],
"meanCharacterF2": 0.8769787822925509,
"passed": false
},
"runtime": {
"loadMs": 23015.70000001788,
"inferenceMs": 516.9000000059605,
"totalMs": 25057.5,
"grade": "excellent",
"charactersPerSecond": 216.67633971504839
},
"failure": "Translation reference quality gate failed"
},
{
"precision": "bnb4",
"device": "wasm",
"status": "pass",
"quality": {
"cases": [
{
"source": "Hello, how are you?",
"references": [
"Hola, ¿cómo estás?",
"Hola, ¿cómo está usted?"
],
"output": {
"translation_text": "Hola, ¿cómo estás?",
"source_language": "en",
"target_language": "es"
},
"characterF2": 1,
"nonempty": true,
"repetitive": false,
"passed": true
},
{
"source": "The train leaves at eight in the morning.",
"references": [
"El tren sale a las ocho de la mañana."
],
"output": {
"translation_text": "El tren sale a las ocho de la mañana.",
"source_language": "en",
"target_language": "es"
},
"characterF2": 1,
"nonempty": true,
"repetitive": false,
"passed": true
},
{
"source": "Please close the door.",
"references": [
"Por favor, cierra la puerta.",
"Por favor, cierre la puerta."
],
"output": {
"translation_text": "Por favor, cierra la puerta.",
"source_language": "en",
"target_language": "es"
},
"characterF2": 1,
"nonempty": true,
"repetitive": false,
"passed": true
},
{
"source": "I would like a glass of water.",
"references": [
"Me gustaría un vaso de agua.",
"Quisiera un vaso de agua."
],
"output": {
"translation_text": "Me gustaría un vaso de agua.",
"source_language": "en",
"target_language": "es"
},
"characterF2": 1,
"nonempty": true,
"repetitive": false,
"passed": true
}
],
"meanCharacterF2": 1,
"passed": true
},
"runtime": {
"loadMs": 37726.69999998808,
"inferenceMs": 629.2000000178814,
"totalMs": 39748.59999999404,
"grade": "excellent",
"charactersPerSecond": 178.003814362392
}
},
{
"precision": "bnb4",
"device": "webgpu",
"status": "pass",
"quality": {
"cases": [
{
"source": "Hello, how are you?",
"references": [
"Hola, ¿cómo estás?",
"Hola, ¿cómo está usted?"
],
"output": {
"translation_text": "Hola, ¿cómo estás?",
"source_language": "en",
"target_language": "es"
},
"characterF2": 1,
"nonempty": true,
"repetitive": false,
"passed": true
},
{
"source": "The train leaves at eight in the morning.",
"references": [
"El tren sale a las ocho de la mañana."
],
"output": {
"translation_text": "El tren sale a las ocho de la mañana.",
"source_language": "en",
"target_language": "es"
},
"characterF2": 1,
"nonempty": true,
"repetitive": false,
"passed": true
},
{
"source": "Please close the door.",
"references": [
"Por favor, cierra la puerta.",
"Por favor, cierre la puerta."
],
"output": {
"translation_text": "Por favor, cierra la puerta.",
"source_language": "en",
"target_language": "es"
},
"characterF2": 1,
"nonempty": true,
"repetitive": false,
"passed": true
},
{
"source": "I would like a glass of water.",
"references": [
"Me gustaría un vaso de agua.",
"Quisiera un vaso de agua."
],
"output": {
"translation_text": "Me gustaría un vaso de agua.",
"source_language": "en",
"target_language": "es"
},
"characterF2": 1,
"nonempty": true,
"repetitive": false,
"passed": true
}
],
"meanCharacterF2": 1,
"passed": true
},
"runtime": {
"loadMs": 36271.80000001192,
"inferenceMs": 1513.5,
"totalMs": 39266,
"grade": "good",
"charactersPerSecond": 74.000660720185
}
}
]