Hy-MT 2
Translate text locally with Tencent Hy-MT2 1.8B using explicit language selections and the official translation prompt.
Desktop app
Run Hy-MT 2 on your computer
Download the desktop app to load this model, provide input and view results locally.
Public desktop downloads are coming soon.Device requirements
If your device falls short
MemoryAdvisory only: close other apps, reduce image/tile/token/batch settings or choose a smaller verified model/precision. CPU can need more RAM than GPU.
GPUIf the selected GPU runtime fails, existing CPU fallback may recover when enabled, but CPU has its own memory/runtime needs.
PlatformUse a compatible OS/architecture/runtime or another model. These estimates do not add hardware detection or block runs.
Benchmark evidence
Saved desktop measurements apply to the recorded hardware and runtime. Fallback cases report the device that actually ran; they do not prove GPU support on the requested device.
Hy-MT2 1.8B — Translation
56 configurations testedQuality criteria
{ "minimumCharacterF2": 0.55, "requireSemanticAnchors": true, "rejectRepetitionAndTruncation": true }
| Precision | Device | Result | Platform | Inference / load | |
|---|---|---|---|---|---|
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 238 ms / 21948 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 256 ms / 24701 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 310 ms / 21948 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 343 ms / 20614 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 344 ms / 20389 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 355 ms / 24701 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 362 ms / 21948 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 390 ms / 24701 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 392 ms / 21948 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 397 ms / 21948 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 413 ms / 24701 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 447 ms / 24701 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 489 ms / 20389 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 551 ms / 20389 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 589 ms / 21948 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 592 ms / 20389 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 601 ms / 20614 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 621 ms / 20614 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 622 ms / 20389 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 632 ms / 20614 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 649 ms / 20614 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 652 ms / 24701 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 653 ms / 21948 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 656 ms / 20614 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 677 ms / 20389 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 733 ms / 20614 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 757 ms / 20389 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 770 ms / 24701 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 739 ms / 19427 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 742 ms / 21287 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 800 ms / 19837 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 850 ms / 19481 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1185 ms / 19427 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1212 ms / 21287 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1225 ms / 19427 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1244 ms / 19837 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1246 ms / 21287 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1248 ms / 19427 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1270 ms / 19837 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1334 ms / 21287 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1353 ms / 19837 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1362 ms / 19481 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1372 ms / 21287 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1384 ms / 19837 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1495 ms / 19427 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1553 ms / 19427 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1567 ms / 19837 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1570 ms / 21287 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1761 ms / 19481 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1775 ms / 19427 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1801 ms / 19481 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1834 ms / 19481 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1849 ms / 19481 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1898 ms / 19837 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1932 ms / 19481 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1968 ms / 21287 ms |
Test setup & sample details
- Quality gate
- { "minimumCharacterF2": 0.55, "requireSemanticAnchors": true, "rejectRepetitionAndTruncation": true }
- Tested
- 2026-11-02T15:21:17.573Z
- Environment
- { "os": "macos", "arch": "aarch64", "hardware": "Apple M3 Max", "osVersion": "27.2", "runtime": "App-owned CPython3.13.18; Transformers5.22.1/Torch2.10.0; MPS op fallback disabled" }
- Runtime
- transformers 5.22.1
- Artifact revision
- 9a341cd1b679d3efd23b46e847b01745a71ed792
- Fixture
{ "file": "model-samples/hunyuan-translation.json", "url": "https://assets.localai.tools/samples/v1/files/b63cba0b5f61f8423d4234d4d28e9cdde538b8345b4ccdc8b2044b1bc88e2fa8/hunyuan-translation.json", "size": 3099, "sha1": "dd6e22e1c2966dd5a249b11eb2fb48b076505088", "sha256": "b63cba0b5f61f8423d4234d4d28e9cdde538b8345b4ccdc8b2044b1bc88e2fa8" }- Rerun when
- model artifact revision change; runtime version change; fixture or quality-gate change; backend/platform behavior change
Per-case quality & failure details
[
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "partial",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 4,
"decoding": "greedy",
"device": "mps",
"driverAllocatedMemoryBytes": 4374102016,
"durationMs": 659.7776250000001,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 653.4228340024129,
"inputTokens": 33,
"loadMs": 21948.349125000004,
"maxNewTokens": 128,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"outputLimitReached": false,
"outputTokens": 14,
"parameterDtype": "torch.bfloat16",
"placement": {
"mps:0/torch.bfloat16": 354
},
"precision": "bf16",
"provider": "Metal/MPS",
"requestedDevice": "mps",
"requestedPrecision": "bf16",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 21.425636313083455,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 22608.612959000002,
"grade": "excellent"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "partial",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 4,
"decoding": "greedy",
"device": "mps",
"driverAllocatedMemoryBytes": 4374790144,
"durationMs": 591.922042,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 589.3725420028204,
"inputTokens": 39,
"loadMs": 21948.349125000004,
"maxNewTokens": 128,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"outputLimitReached": false,
"outputTokens": 21,
"parameterDtype": "torch.bfloat16",
"placement": {
"mps:0/torch.bfloat16": 354
},
"precision": "bf16",
"provider": "Metal/MPS",
"requestedDevice": "mps",
"requestedPrecision": "bf16",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 35.63111360539003,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 22540.609583,
"grade": "excellent"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "partial",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 4,
"decoding": "greedy",
"device": "mps",
"driverAllocatedMemoryBytes": 4375019520,
"durationMs": 240.208417,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 237.9574579972541,
"inputTokens": 36,
"loadMs": 21948.349125000004,
"maxNewTokens": 128,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"outputLimitReached": false,
"outputTokens": 8,
"parameterDtype": "torch.bfloat16",
"placement": {
"mps:0/torch.bfloat16": 354
},
"precision": "bf16",
"provider": "Metal/MPS",
"requestedDevice": "mps",
"requestedPrecision": "bf16",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 33.61945478545294,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 22188.885625000006,
"grade": "excellent"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "partial",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 4,
"decoding": "greedy",
"device": "mps",
"driverAllocatedMemoryBytes": 4375248896,
"durationMs": 364.464292,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 362.15649999212474,
"inputTokens": 32,
"loadMs": 21948.349125000004,
"maxNewTokens": 128,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"outputLimitReached": false,
"outputTokens": 14,
"parameterDtype": "torch.bfloat16",
"placement": {
"mps:0/torch.bfloat16": 354
},
"precision": "bf16",
"provider": "Metal/MPS",
"requestedDevice": "mps",
"requestedPrecision": "bf16",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 38.65732079999789,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 22313.242542000004,
"grade": "excellent"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.769566037344403,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "partial",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 4,
"decoding": "greedy",
"device": "mps",
"driverAllocatedMemoryBytes": 4375248896,
"durationMs": 399.539042,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 397.1101670031203,
"inputTokens": 33,
"loadMs": 21948.349125000004,
"maxNewTokens": 128,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"outputLimitReached": false,
"outputTokens": 18,
"parameterDtype": "torch.bfloat16",
"placement": {
"mps:0/torch.bfloat16": 354
},
"precision": "bf16",
"provider": "Metal/MPS",
"requestedDevice": "mps",
"requestedPrecision": "bf16",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 45.32747206106804,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 22348.222792000004,
"grade": "excellent"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "partial",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 4,
"decoding": "greedy",
"device": "mps",
"driverAllocatedMemoryBytes": 4375461888,
"durationMs": 394.006292,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 391.5271250007208,
"inputTokens": 31,
"loadMs": 21948.349125000004,
"maxNewTokens": 128,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"outputLimitReached": false,
"outputTokens": 16,
"parameterDtype": "torch.bfloat16",
"placement": {
"mps:0/torch.bfloat16": 354
},
"precision": "bf16",
"provider": "Metal/MPS",
"requestedDevice": "mps",
"requestedPrecision": "bf16",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 40.8656232948625,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 22342.842375000004,
"grade": "excellent"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "partial",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 4,
"decoding": "greedy",
"device": "mps",
"driverAllocatedMemoryBytes": 4375461888,
"durationMs": 312.324875,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 309.8432499973569,
"inputTokens": 33,
"loadMs": 21948.349125000004,
"maxNewTokens": 128,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"outputLimitReached": false,
"outputTokens": 14,
"parameterDtype": "torch.bfloat16",
"placement": {
"mps:0/torch.bfloat16": 354
},
"precision": "bf16",
"provider": "Metal/MPS",
"requestedDevice": "mps",
"requestedPrecision": "bf16",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 45.18413746344135,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 22261.096458000004,
"grade": "excellent"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "partial",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 4,
"decoding": "greedy",
"device": "mps",
"driverAllocatedMemoryBytes": 4374102016,
"durationMs": 778.082833,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 769.989375010482,
"inputTokens": 33,
"loadMs": 24700.573875000002,
"maxNewTokens": 128,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"outputLimitReached": false,
"outputTokens": 14,
"parameterDtype": "torch.bfloat16",
"placement": {
"mps:0/torch.bfloat16": 354
},
"precision": "bf16",
"provider": "Metal/MPS",
"requestedDevice": "mps",
"requestedPrecision": "bf16",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 18.182069070510764,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 25479.062416999994,
"grade": "good"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "partial",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 4,
"decoding": "greedy",
"device": "mps",
"driverAllocatedMemoryBytes": 4374790144,
"durationMs": 654.535584,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 651.9678749900777,
"inputTokens": 39,
"loadMs": 24700.573875000002,
"maxNewTokens": 128,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"outputLimitReached": false,
"outputTokens": 21,
"parameterDtype": "torch.bfloat16",
"placement": {
"mps:0/torch.bfloat16": 354
},
"precision": "bf16",
"provider": "Metal/MPS",
"requestedDevice": "mps",
"requestedPrecision": "bf16",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 32.21017600034296,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 25355.421124999993,
"grade": "excellent"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "partial",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 4,
"decoding": "greedy",
"device": "mps",
"driverAllocatedMemoryBytes": 4375019520,
"durationMs": 258.640458,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 256.1951669922564,
"inputTokens": 36,
"loadMs": 24700.573875000002,
"maxNewTokens": 128,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"outputLimitReached": false,
"outputTokens": 8,
"parameterDtype": "torch.bfloat16",
"placement": {
"mps:0/torch.bfloat16": 354
},
"precision": "bf16",
"provider": "Metal/MPS",
"requestedDevice": "mps",
"requestedPrecision": "bf16",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 31.22619405323053,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 24959.517000000007,
"grade": "excellent"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "partial",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 4,
"decoding": "greedy",
"device": "mps",
"driverAllocatedMemoryBytes": 4375248896,
"durationMs": 392.977625,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 390.1868749962887,
"inputTokens": 32,
"loadMs": 24700.573875000002,
"maxNewTokens": 128,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"outputLimitReached": false,
"outputTokens": 14,
"parameterDtype": "torch.bfloat16",
"placement": {
"mps:0/torch.bfloat16": 354
},
"precision": "bf16",
"provider": "Metal/MPS",
"requestedDevice": "mps",
"requestedPrecision": "bf16",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 35.8802432812051,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 25093.88958300001,
"grade": "excellent"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.769566037344403,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "partial",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 4,
"decoding": "greedy",
"device": "mps",
"driverAllocatedMemoryBytes": 4375248896,
"durationMs": 416.077792,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 413.22800000489224,
"inputTokens": 33,
"loadMs": 24700.573875000002,
"maxNewTokens": 128,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"outputLimitReached": false,
"outputTokens": 18,
"parameterDtype": "torch.bfloat16",
"placement": {
"mps:0/torch.bfloat16": 354
},
"precision": "bf16",
"provider": "Metal/MPS",
"requestedDevice": "mps",
"requestedPrecision": "bf16",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 43.55948773990847,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 25116.964791000006,
"grade": "excellent"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "partial",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 4,
"decoding": "greedy",
"device": "mps",
"driverAllocatedMemoryBytes": 4375461888,
"durationMs": 450.004625,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 447.3837499972433,
"inputTokens": 31,
"loadMs": 24700.573875000002,
"maxNewTokens": 128,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"outputLimitReached": false,
"outputTokens": 16,
"parameterDtype": "torch.bfloat16",
"placement": {
"mps:0/torch.bfloat16": 354
},
"precision": "bf16",
"provider": "Metal/MPS",
"requestedDevice": "mps",
"requestedPrecision": "bf16",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 35.76348045743411,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 25150.878375,
"grade": "excellent"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "partial",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 4,
"decoding": "greedy",
"device": "mps",
"driverAllocatedMemoryBytes": 4375461888,
"durationMs": 357.838541,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 354.92241600877605,
"inputTokens": 33,
"loadMs": 24700.573875000002,
"maxNewTokens": 128,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"outputLimitReached": false,
"outputTokens": 14,
"parameterDtype": "torch.bfloat16",
"placement": {
"mps:0/torch.bfloat16": 354
},
"precision": "bf16",
"provider": "Metal/MPS",
"requestedDevice": "mps",
"requestedPrecision": "bf16",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 39.4452403357184,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 25058.674874999997,
"grade": "excellent"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "cpu",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 4,
"decoding": "greedy",
"device": "cpu",
"durationMs": 1358.75125,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 1353.4542080014944,
"inputTokens": 33,
"loadMs": 19836.568667000014,
"maxNewTokens": 128,
"outputLimitReached": false,
"outputTokens": 14,
"parameterDtype": "torch.float32",
"placement": {
"cpu/torch.float32": 354
},
"precision": "fp32",
"provider": "PyTorch CPU",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 10.343903707442273,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 21195.607707999996,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "cpu",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 4,
"decoding": "greedy",
"device": "cpu",
"durationMs": 1899.234667,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 1898.1381250050615,
"inputTokens": 39,
"loadMs": 19836.568667000014,
"maxNewTokens": 128,
"outputLimitReached": false,
"outputTokens": 21,
"parameterDtype": "torch.float32",
"placement": {
"cpu/torch.float32": 354
},
"precision": "fp32",
"provider": "PyTorch CPU",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 11.063473054651384,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 21736.07412600002,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "cpu",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 4,
"decoding": "greedy",
"device": "cpu",
"durationMs": 801.300792,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 800.2311659947736,
"inputTokens": 36,
"loadMs": 19836.568667000014,
"maxNewTokens": 128,
"outputLimitReached": false,
"outputTokens": 8,
"parameterDtype": "torch.float32",
"placement": {
"cpu/torch.float32": 354
},
"precision": "fp32",
"provider": "PyTorch CPU",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 9.99711125978846,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 20638.127792000014,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "cpu",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 4,
"decoding": "greedy",
"device": "cpu",
"durationMs": 1244.796834,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 1243.75899998995,
"inputTokens": 32,
"loadMs": 19836.568667000014,
"maxNewTokens": 128,
"outputLimitReached": false,
"outputTokens": 14,
"parameterDtype": "torch.float32",
"placement": {
"cpu/torch.float32": 354
},
"precision": "fp32",
"provider": "PyTorch CPU",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 11.256199955226956,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 21081.65066700001,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.769566037344403,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "cpu",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 4,
"decoding": "greedy",
"device": "cpu",
"durationMs": 1568.0272909999999,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 1566.976958012674,
"inputTokens": 33,
"loadMs": 19836.568667000014,
"maxNewTokens": 128,
"outputLimitReached": false,
"outputTokens": 18,
"parameterDtype": "torch.float32",
"placement": {
"cpu/torch.float32": 354
},
"precision": "fp32",
"provider": "PyTorch CPU",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 11.48708658921736,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 21404.911542000016,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "cpu",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 4,
"decoding": "greedy",
"device": "cpu",
"durationMs": 1384.9958339999998,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 1383.80125000549,
"inputTokens": 31,
"loadMs": 19836.568667000014,
"maxNewTokens": 128,
"outputLimitReached": false,
"outputTokens": 16,
"parameterDtype": "torch.float32",
"placement": {
"cpu/torch.float32": 354
},
"precision": "fp32",
"provider": "PyTorch CPU",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 11.562354059108214,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 21221.874708999996,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "cpu",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 4,
"decoding": "greedy",
"device": "cpu",
"durationMs": 1270.922625,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 1269.897124991985,
"inputTokens": 33,
"loadMs": 19836.568667000014,
"maxNewTokens": 128,
"outputLimitReached": false,
"outputTokens": 14,
"parameterDtype": "torch.float32",
"placement": {
"cpu/torch.float32": 354
},
"precision": "fp32",
"provider": "PyTorch CPU",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 11.024515076438464,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 21107.77583400003,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "cpu",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 4,
"decoding": "greedy",
"device": "cpu",
"durationMs": 1806.117833,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 1800.6467079976574,
"inputTokens": 33,
"loadMs": 19480.776207999996,
"maxNewTokens": 128,
"outputLimitReached": false,
"outputTokens": 14,
"parameterDtype": "torch.float32",
"placement": {
"cpu/torch.float32": 354
},
"precision": "fp32",
"provider": "PyTorch CPU",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 7.77498436412781,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 21287.190040999994,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "cpu",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 4,
"decoding": "greedy",
"device": "cpu",
"durationMs": 1932.746791,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 1931.6712919971903,
"inputTokens": 39,
"loadMs": 19480.776207999996,
"maxNewTokens": 128,
"outputLimitReached": false,
"outputTokens": 21,
"parameterDtype": "torch.float32",
"placement": {
"cpu/torch.float32": 354
},
"precision": "fp32",
"provider": "PyTorch CPU",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 10.871414865977384,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 21413.791582999984,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "cpu",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 4,
"decoding": "greedy",
"device": "cpu",
"durationMs": 851.316958,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 850.2449160005199,
"inputTokens": 36,
"loadMs": 19480.776207999996,
"maxNewTokens": 128,
"outputLimitReached": false,
"outputTokens": 8,
"parameterDtype": "torch.float32",
"placement": {
"cpu/torch.float32": 354
},
"precision": "fp32",
"provider": "PyTorch CPU",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 9.409053614376576,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 20332.353166999994,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "cpu",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 4,
"decoding": "greedy",
"device": "cpu",
"durationMs": 1762.56175,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 1761.337041010847,
"inputTokens": 32,
"loadMs": 19480.776207999996,
"maxNewTokens": 128,
"outputLimitReached": false,
"outputTokens": 14,
"parameterDtype": "torch.float32",
"placement": {
"cpu/torch.float32": 354
},
"precision": "fp32",
"provider": "PyTorch CPU",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 7.948507113644344,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 21243.611458,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.769566037344403,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "cpu",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 4,
"decoding": "greedy",
"device": "cpu",
"durationMs": 1850.378667,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 1849.1938750084955,
"inputTokens": 33,
"loadMs": 19480.776207999996,
"maxNewTokens": 128,
"outputLimitReached": false,
"outputTokens": 18,
"parameterDtype": "torch.float32",
"placement": {
"cpu/torch.float32": 354
},
"precision": "fp32",
"provider": "PyTorch CPU",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 9.733971241883603,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 21331.400832999992,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "cpu",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 4,
"decoding": "greedy",
"device": "cpu",
"durationMs": 1835.0864169999998,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 1833.836332996725,
"inputTokens": 31,
"loadMs": 19480.776207999996,
"maxNewTokens": 128,
"outputLimitReached": false,
"outputTokens": 16,
"parameterDtype": "torch.float32",
"placement": {
"cpu/torch.float32": 354
},
"precision": "fp32",
"provider": "PyTorch CPU",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 8.724878939362018,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 21316.205707999994,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "cpu",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 4,
"decoding": "greedy",
"device": "cpu",
"durationMs": 1363.349125,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 1362.1994999994058,
"inputTokens": 33,
"loadMs": 19480.776207999996,
"maxNewTokens": 128,
"outputLimitReached": false,
"outputTokens": 14,
"parameterDtype": "torch.float32",
"placement": {
"cpu/torch.float32": 354
},
"precision": "fp32",
"provider": "PyTorch CPU",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 10.27749606427407,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 20844.401082999975,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "cpu",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 2,
"decoding": "greedy",
"device": "cpu",
"durationMs": 1377.025792,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 1371.6296659986256,
"inputTokens": 33,
"loadMs": 21287.11720799998,
"maxNewTokens": 128,
"outputLimitReached": false,
"outputTokens": 14,
"parameterDtype": "torch.float32",
"placement": {
"cpu/torch.float32": 354
},
"precision": "fp32",
"provider": "PyTorch CPU",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 10.206836689994738,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 22664.436623999965,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "cpu",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 2,
"decoding": "greedy",
"device": "cpu",
"durationMs": 1969.116708,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 1968.03429198917,
"inputTokens": 39,
"loadMs": 21287.11720799998,
"maxNewTokens": 128,
"outputLimitReached": false,
"outputTokens": 21,
"parameterDtype": "torch.float32",
"placement": {
"cpu/torch.float32": 354
},
"precision": "fp32",
"provider": "PyTorch CPU",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 10.67054577528447,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 23256.509373999987,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "cpu",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 2,
"decoding": "greedy",
"device": "cpu",
"durationMs": 743.0032090000001,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 741.9097079982748,
"inputTokens": 36,
"loadMs": 21287.11720799998,
"maxNewTokens": 128,
"outputLimitReached": false,
"outputTokens": 8,
"parameterDtype": "torch.float32",
"placement": {
"cpu/torch.float32": 354
},
"precision": "fp32",
"provider": "PyTorch CPU",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 10.7829833115199,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 22030.44070799998,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "cpu",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 2,
"decoding": "greedy",
"device": "cpu",
"durationMs": 1213.374583,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 1212.3512910038698,
"inputTokens": 32,
"loadMs": 21287.11720799998,
"maxNewTokens": 128,
"outputLimitReached": false,
"outputTokens": 14,
"parameterDtype": "torch.float32",
"placement": {
"cpu/torch.float32": 354
},
"precision": "fp32",
"provider": "PyTorch CPU",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 11.547808051911671,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 22500.799207999982,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.769566037344403,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "cpu",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 2,
"decoding": "greedy",
"device": "cpu",
"durationMs": 1570.908875,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 1569.8936249973483,
"inputTokens": 33,
"loadMs": 21287.11720799998,
"maxNewTokens": 128,
"outputLimitReached": false,
"outputTokens": 18,
"parameterDtype": "torch.float32",
"placement": {
"cpu/torch.float32": 354
},
"precision": "fp32",
"provider": "PyTorch CPU",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 11.465745011882827,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 22858.326,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "cpu",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 2,
"decoding": "greedy",
"device": "cpu",
"durationMs": 1334.775958,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 1333.7626660068054,
"inputTokens": 31,
"loadMs": 21287.11720799998,
"maxNewTokens": 128,
"outputLimitReached": false,
"outputTokens": 16,
"parameterDtype": "torch.float32",
"placement": {
"cpu/torch.float32": 354
},
"precision": "fp32",
"provider": "PyTorch CPU",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 11.996137249742423,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 22622.18037399996,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "cpu",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 2,
"decoding": "greedy",
"device": "cpu",
"durationMs": 1247.4299170000002,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 1246.3203750085086,
"inputTokens": 33,
"loadMs": 21287.11720799998,
"maxNewTokens": 128,
"outputLimitReached": false,
"outputTokens": 14,
"parameterDtype": "torch.float32",
"placement": {
"cpu/torch.float32": 354
},
"precision": "fp32",
"provider": "PyTorch CPU",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 11.233066778599705,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 22534.846290999965,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "cpu",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 2,
"decoding": "greedy",
"device": "cpu",
"durationMs": 1252.09325,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 1247.739915997954,
"inputTokens": 33,
"loadMs": 19426.93958300003,
"maxNewTokens": 128,
"outputLimitReached": false,
"outputTokens": 14,
"parameterDtype": "torch.float32",
"placement": {
"cpu/torch.float32": 354
},
"precision": "fp32",
"provider": "PyTorch CPU",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 11.220287032977277,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 20679.309458000003,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "cpu",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 2,
"decoding": "greedy",
"device": "cpu",
"durationMs": 1776.27025,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 1775.1904169999762,
"inputTokens": 39,
"loadMs": 19426.93958300003,
"maxNewTokens": 128,
"outputLimitReached": false,
"outputTokens": 21,
"parameterDtype": "torch.float32",
"placement": {
"cpu/torch.float32": 354
},
"precision": "fp32",
"provider": "PyTorch CPU",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 11.829716856791864,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 21203.49979100004,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "cpu",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 2,
"decoding": "greedy",
"device": "cpu",
"durationMs": 740.449292,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 739.4447080005193,
"inputTokens": 36,
"loadMs": 19426.93958300003,
"maxNewTokens": 128,
"outputLimitReached": false,
"outputTokens": 8,
"parameterDtype": "torch.float32",
"placement": {
"cpu/torch.float32": 354
},
"precision": "fp32",
"provider": "PyTorch CPU",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 10.818929276851868,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 20167.71320800006,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "cpu",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 2,
"decoding": "greedy",
"device": "cpu",
"durationMs": 1185.9571660000001,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 1184.974791991408,
"inputTokens": 32,
"loadMs": 19426.93958300003,
"maxNewTokens": 128,
"outputLimitReached": false,
"outputTokens": 14,
"parameterDtype": "torch.float32",
"placement": {
"cpu/torch.float32": 354
},
"precision": "fp32",
"provider": "PyTorch CPU",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 11.81459731854069,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 20613.20424900006,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.769566037344403,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "cpu",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 2,
"decoding": "greedy",
"device": "cpu",
"durationMs": 1553.7456670000001,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 1552.6519999984885,
"inputTokens": 33,
"loadMs": 19426.93958300003,
"maxNewTokens": 128,
"outputLimitReached": false,
"outputTokens": 18,
"parameterDtype": "torch.float32",
"placement": {
"cpu/torch.float32": 354
},
"precision": "fp32",
"provider": "PyTorch CPU",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 11.593067860678069,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 20981.01787500002,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "cpu",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 2,
"decoding": "greedy",
"device": "cpu",
"durationMs": 1496.346084,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 1495.246207996388,
"inputTokens": 31,
"loadMs": 19426.93958300003,
"maxNewTokens": 128,
"outputLimitReached": false,
"outputTokens": 16,
"parameterDtype": "torch.float32",
"placement": {
"cpu/torch.float32": 354
},
"precision": "fp32",
"provider": "PyTorch CPU",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 10.700578884222558,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 20923.616791000008,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "cpu",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 2,
"decoding": "greedy",
"device": "cpu",
"durationMs": 1226.307041,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 1225.226875001681,
"inputTokens": 33,
"loadMs": 19426.93958300003,
"maxNewTokens": 128,
"outputLimitReached": false,
"outputTokens": 14,
"parameterDtype": "torch.float32",
"placement": {
"cpu/torch.float32": 354
},
"precision": "fp32",
"provider": "PyTorch CPU",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 11.426455202413669,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 20653.54458300001,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "partial",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 4,
"decoding": "greedy",
"device": "mps",
"driverAllocatedMemoryBytes": 7486275584,
"durationMs": 662.381208,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 655.7098339981167,
"inputTokens": 33,
"loadMs": 20613.526916000003,
"maxNewTokens": 128,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"outputLimitReached": false,
"outputTokens": 14,
"parameterDtype": "torch.float32",
"placement": {
"mps:0/torch.float32": 354
},
"precision": "fp32",
"provider": "Metal/MPS",
"requestedDevice": "mps",
"requestedPrecision": "fp32",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 21.35090748088462,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 21276.195083,
"grade": "excellent"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "partial",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 4,
"decoding": "greedy",
"device": "mps",
"driverAllocatedMemoryBytes": 7486963712,
"durationMs": 735.567875,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 733.0782920034835,
"inputTokens": 39,
"loadMs": 20613.526916000003,
"maxNewTokens": 128,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"outputLimitReached": false,
"outputTokens": 21,
"parameterDtype": "torch.float32",
"placement": {
"mps:0/torch.float32": 354
},
"precision": "fp32",
"provider": "Metal/MPS",
"requestedDevice": "mps",
"requestedPrecision": "fp32",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 28.646326359777422,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 21349.34904100001,
"grade": "excellent"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "partial",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 4,
"decoding": "greedy",
"device": "mps",
"driverAllocatedMemoryBytes": 7487193088,
"durationMs": 345.366542,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 342.9617910005618,
"inputTokens": 36,
"loadMs": 20613.526916000003,
"maxNewTokens": 128,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"outputLimitReached": false,
"outputTokens": 8,
"parameterDtype": "torch.float32",
"placement": {
"mps:0/torch.float32": 354
},
"precision": "fp32",
"provider": "Metal/MPS",
"requestedDevice": "mps",
"requestedPrecision": "fp32",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 23.32621361890105,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 20959.145500000042,
"grade": "excellent"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "partial",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 4,
"decoding": "greedy",
"device": "mps",
"driverAllocatedMemoryBytes": 7487422464,
"durationMs": 635.143834,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 632.427499993355,
"inputTokens": 32,
"loadMs": 20613.526916000003,
"maxNewTokens": 128,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"outputLimitReached": false,
"outputTokens": 14,
"parameterDtype": "torch.float32",
"placement": {
"mps:0/torch.float32": 354
},
"precision": "fp32",
"provider": "Metal/MPS",
"requestedDevice": "mps",
"requestedPrecision": "fp32",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 22.136924786077618,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 21248.94958200003,
"grade": "excellent"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.769566037344403,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "partial",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 4,
"decoding": "greedy",
"device": "mps",
"driverAllocatedMemoryBytes": 7487422464,
"durationMs": 623.886709,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 621.4268329931656,
"inputTokens": 33,
"loadMs": 20613.526916000003,
"maxNewTokens": 128,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"outputLimitReached": false,
"outputTokens": 18,
"parameterDtype": "torch.float32",
"placement": {
"mps:0/torch.float32": 354
},
"precision": "fp32",
"provider": "Metal/MPS",
"requestedDevice": "mps",
"requestedPrecision": "fp32",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 28.96559827212669,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 21237.697083000035,
"grade": "excellent"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "partial",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 4,
"decoding": "greedy",
"device": "mps",
"driverAllocatedMemoryBytes": 7487635456,
"durationMs": 651.96175,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 649.4119169947226,
"inputTokens": 31,
"loadMs": 20613.526916000003,
"maxNewTokens": 128,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"outputLimitReached": false,
"outputTokens": 16,
"parameterDtype": "torch.float32",
"placement": {
"mps:0/torch.float32": 354
},
"precision": "fp32",
"provider": "Metal/MPS",
"requestedDevice": "mps",
"requestedPrecision": "fp32",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 24.63767538181783,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 21265.76658299996,
"grade": "excellent"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "partial",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 4,
"decoding": "greedy",
"device": "mps",
"driverAllocatedMemoryBytes": 7487635456,
"durationMs": 604.573,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 600.6057920021703,
"inputTokens": 33,
"loadMs": 20613.526916000003,
"maxNewTokens": 128,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"outputLimitReached": false,
"outputTokens": 14,
"parameterDtype": "torch.float32",
"placement": {
"mps:0/torch.float32": 354
},
"precision": "fp32",
"provider": "Metal/MPS",
"requestedDevice": "mps",
"requestedPrecision": "fp32",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 23.30979851747652,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 21218.51187399996,
"grade": "excellent"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "partial",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 4,
"decoding": "greedy",
"device": "mps",
"driverAllocatedMemoryBytes": 7486275584,
"durationMs": 683.779083,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 676.9712079985766,
"inputTokens": 33,
"loadMs": 20389.204541000014,
"maxNewTokens": 128,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"outputLimitReached": false,
"outputTokens": 14,
"parameterDtype": "torch.float32",
"placement": {
"mps:0/torch.float32": 354
},
"precision": "fp32",
"provider": "Metal/MPS",
"requestedDevice": "mps",
"requestedPrecision": "fp32",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 20.6803477527355,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 21073.237916000013,
"grade": "excellent"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "partial",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 4,
"decoding": "greedy",
"device": "mps",
"driverAllocatedMemoryBytes": 7486963712,
"durationMs": 759.264584,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 756.7442080035107,
"inputTokens": 39,
"loadMs": 20389.204541000014,
"maxNewTokens": 128,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"outputLimitReached": false,
"outputTokens": 21,
"parameterDtype": "torch.float32",
"placement": {
"mps:0/torch.float32": 354
},
"precision": "fp32",
"provider": "Metal/MPS",
"requestedDevice": "mps",
"requestedPrecision": "fp32",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 27.75046016593043,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 21148.765041000035,
"grade": "excellent"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "partial",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 4,
"decoding": "greedy",
"device": "mps",
"driverAllocatedMemoryBytes": 7487193088,
"durationMs": 347.28325,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 344.3750000005821,
"inputTokens": 36,
"loadMs": 20389.204541000014,
"maxNewTokens": 128,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"outputLimitReached": false,
"outputTokens": 8,
"parameterDtype": "torch.float32",
"placement": {
"mps:0/torch.float32": 354
},
"precision": "fp32",
"provider": "Metal/MPS",
"requestedDevice": "mps",
"requestedPrecision": "fp32",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 23.230490018109556,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 20736.79116600001,
"grade": "excellent"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "partial",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 4,
"decoding": "greedy",
"device": "mps",
"driverAllocatedMemoryBytes": 7487422464,
"durationMs": 554.4377499999999,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 551.4502919977531,
"inputTokens": 32,
"loadMs": 20389.204541000014,
"maxNewTokens": 128,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"outputLimitReached": false,
"outputTokens": 14,
"parameterDtype": "torch.float32",
"placement": {
"mps:0/torch.float32": 354
},
"precision": "fp32",
"provider": "Metal/MPS",
"requestedDevice": "mps",
"requestedPrecision": "fp32",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 25.387601027976324,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 20943.933958000038,
"grade": "excellent"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.769566037344403,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "partial",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 4,
"decoding": "greedy",
"device": "mps",
"driverAllocatedMemoryBytes": 7487422464,
"durationMs": 624.1876669999999,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 622.2563329938566,
"inputTokens": 33,
"loadMs": 20389.204541000014,
"maxNewTokens": 128,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"outputLimitReached": false,
"outputTokens": 18,
"parameterDtype": "torch.float32",
"placement": {
"mps:0/torch.float32": 354
},
"precision": "fp32",
"provider": "Metal/MPS",
"requestedDevice": "mps",
"requestedPrecision": "fp32",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 28.926985625677368,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 21013.697082999977,
"grade": "excellent"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "partial",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 4,
"decoding": "greedy",
"device": "mps",
"driverAllocatedMemoryBytes": 7487635456,
"durationMs": 594.683584,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 591.7933330056258,
"inputTokens": 31,
"loadMs": 20389.204541000014,
"maxNewTokens": 128,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"outputLimitReached": false,
"outputTokens": 16,
"parameterDtype": "torch.float32",
"placement": {
"mps:0/torch.float32": 354
},
"precision": "fp32",
"provider": "Metal/MPS",
"requestedDevice": "mps",
"requestedPrecision": "fp32",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 27.03646544772396,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 20984.17712399998,
"grade": "excellent"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6"
},
"runtime": {
"acceleration": "partial",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"artifactRevision": "9a341cd1b679d3efd23b46e847b01745a71ed792",
"cpuThreads": 4,
"decoding": "greedy",
"device": "mps",
"driverAllocatedMemoryBytes": 7487635456,
"durationMs": 491.29566600000004,
"engine": "transformers",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 488.5365410009399,
"inputTokens": 33,
"loadMs": 20389.204541000014,
"maxNewTokens": 128,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"outputLimitReached": false,
"outputTokens": 14,
"parameterDtype": "torch.float32",
"placement": {
"mps:0/torch.float32": 354
},
"precision": "fp32",
"provider": "Metal/MPS",
"requestedDevice": "mps",
"requestedPrecision": "fp32",
"reused": false,
"selectionReason": "Explicit device selection; actual placement is reported after inference",
"tokensPerSecond": 28.657017080679463,
"torchVersion": "2.10.0",
"version": "5.22.1",
"totalMs": 20880.79920800001,
"grade": "excellent"
}
}
]Hy-MT2 7B — Text Translation
83 configurations testedQuality criteria
{ "minimumCharacterF2": 0.55, "requireSemanticAnchors": true, "rejectRepetitionAndTruncation": true }
| Precision | Device | Result | Platform | Inference / load | |
|---|---|---|---|---|---|
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 659 ms / 4566 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 662 ms / 4574 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 667 ms / 4667 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 669 ms / 5109 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 846 ms / 4566 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 855 ms / 4667 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 902 ms / 4574 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 903 ms / 4574 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 913 ms / 4566 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 914 ms / 4667 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 1020 ms / 4667 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 1050 ms / 4574 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 1056 ms / 4566 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 1103 ms / 5109 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 1151 ms / 4566 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 1167 ms / 4667 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 1170 ms / 4574 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 1183 ms / 5109 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 1459 ms / 4667 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 1460 ms / 4566 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 1468 ms / 4574 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 1550 ms / 5109 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 1688 ms / 4566 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 1693 ms / 5109 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 1709 ms / 4667 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 1806 ms / 4574 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 1849 ms / 5109 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 2985 ms / 3911 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 3035 ms / 4022 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 3047 ms / 3945 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 3080 ms / 3910 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 3086 ms / 12265 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 3128 ms / 4066 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 3234 ms / 3895 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 3842 ms / 4907 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 4022 ms / 3911 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 4085 ms / 4066 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 4165 ms / 4022 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 4181 ms / 3910 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 4190 ms / 4022 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 4196 ms / 3911 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 4201 ms / 3910 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 4213 ms / 3945 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 4232 ms / 3945 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 4238 ms / 12265 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 4252 ms / 4066 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 4344 ms / 3895 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 4364 ms / 3895 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 4539 ms / 4066 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 4617 ms / 3945 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 4804 ms / 4907 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 4841 ms / 3911 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 4929 ms / 3895 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 4971 ms / 3910 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 5608 ms / 3911 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 5795 ms / 3945 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 5818 ms / 12265 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 5826 ms / 4022 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 5876 ms / 3910 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 5994 ms / 4066 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 6270 ms / 3895 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 6487 ms / 4022 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 7108 ms / 4066 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 7239 ms / 3911 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 7282 ms / 12265 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 7286 ms / 3910 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 7335 ms / 3945 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 7364 ms / 4022 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 7403 ms / 4907 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 7847 ms / 3895 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 8218 ms / 3911 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 8310 ms / 4066 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 8373 ms / 3945 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 8399 ms / 3910 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 8407 ms / 4022 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 8443 ms / 12265 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 8783 ms / 3895 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 8898 ms / 12265 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 10471 ms / 4907 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 16483 ms / 4907 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 17719 ms / 4907 ms |
BF16 | mps | Failed | — | macos · aarch64 | 3042 ms / 23206 ms |
FP32 | mps | Failed | — | macos · aarch64 | 3610 ms / 22668 ms |
Test setup & sample details
- Quality gate
- { "minimumCharacterF2": 0.55, "requireSemanticAnchors": true, "rejectRepetitionAndTruncation": true }
- Tested
- 2026-11-06T13:34:21.657206+00:00
- Environment
- { "os": "macos", "arch": "aarch64", "hardware": "Apple M3 Max", "osVersion": "27.2", "runtime": "App-owned CPython3.13.18; Transformers5.22.1/Torch2.10.0; built-in HunYuanDenseV1ForCausalLM, SDPA; MPS CPU op fallback disabled" }
- Runtime
- transformers 5.22.1
- Artifact revision
- 9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223
- Fixture
{ "file": "model-samples/hunyuan-translation.json", "url": "https://assets.localai.tools/samples/v1/files/b63cba0b5f61f8423d4234d4d28e9cdde538b8345b4ccdc8b2044b1bc88e2fa8/hunyuan-translation.json", "size": 3099, "sha1": "dd6e22e1c2966dd5a249b11eb2fb48b076505088", "sha256": "b63cba0b5f61f8423d4234d4d28e9cdde538b8345b4ccdc8b2044b1bc88e2fa8" }- Rerun when
- model artifact revision change; runtime version change; fixture or quality-gate change; backend/platform behavior change
Per-case quality & failure details
[
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4022.0228329999372,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 6494.175249999898,
"inferenceMs": 6487.1073750000505,
"tokensPerSecond": 1.849822934370638,
"inputTokens": 32,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 10517.494333000286,
"residentMemoryBytes": 30606082048,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4022.0228329999372,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 7364.277582999421,
"inferenceMs": 7363.7570410010085,
"tokensPerSecond": 2.851805115659454,
"inputTokens": 39,
"outputTokens": 21,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 11387.184748999061,
"residentMemoryBytes": 30612570112,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4022.0228329999372,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 3035.7154169996647,
"inferenceMs": 3035.2572909996525,
"tokensPerSecond": 2.6356908930660126,
"inputTokens": 35,
"outputTokens": 8,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 7058.587040999555,
"residentMemoryBytes": 30616649728,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4022.0228329999372,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 4189.968167000188,
"inferenceMs": 4189.6531250004045,
"tokensPerSecond": 2.8641989305496125,
"inputTokens": 31,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 8212.78966599857,
"residentMemoryBytes": 30619959296,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.8772489985666848,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4022.0228329999372,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 5826.339375000316,
"inferenceMs": 5825.831374999325,
"tokensPerSecond": 2.9180384576445055,
"inputTokens": 32,
"outputTokens": 17,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 9849.2066660001,
"residentMemoryBytes": 30619959296,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4022.0228329999372,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 8407.206874999247,
"inferenceMs": 8406.83908399842,
"tokensPerSecond": 2.9737693026127983,
"inputTokens": 30,
"outputTokens": 25,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 12430.087041000661,
"residentMemoryBytes": 30619959296,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4022.0228329999372,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 4165.448792000461,
"inferenceMs": 4165.125915998942,
"tokensPerSecond": 2.8810653608108225,
"inputTokens": 32,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 8188.344999998662,
"residentMemoryBytes": 30619959296,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3945.2491249994637,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 4620.3236670007755,
"inferenceMs": 4616.5452919995005,
"tokensPerSecond": 2.599346316561882,
"inputTokens": 32,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 8566.323208999165,
"residentMemoryBytes": 30603968512,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3945.2491249994637,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 7335.508834001303,
"inferenceMs": 7335.092417000851,
"tokensPerSecond": 2.8629496134673666,
"inputTokens": 39,
"outputTokens": 21,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 11281.639124999856,
"residentMemoryBytes": 30611259392,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3945.2491249994637,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 3047.0905830006814,
"inferenceMs": 3046.679834000315,
"tokensPerSecond": 2.625809220490338,
"inputTokens": 35,
"outputTokens": 8,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 6993.1555000002845,
"residentMemoryBytes": 30620844032,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3945.2491249994637,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 4213.6778330004745,
"inferenceMs": 4213.337375000265,
"tokensPerSecond": 2.8480985337660614,
"inputTokens": 31,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 8159.799958000804,
"residentMemoryBytes": 30621679616,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.8772489985666848,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3945.2491249994637,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 5795.280541999091,
"inferenceMs": 5794.827292000264,
"tokensPerSecond": 2.9336508481397594,
"inputTokens": 32,
"outputTokens": 17,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 9741.396333000012,
"residentMemoryBytes": 30621990912,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3945.2491249994637,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 8373.782374999792,
"inferenceMs": 8373.424624998734,
"tokensPerSecond": 2.9856362384110886,
"inputTokens": 30,
"outputTokens": 25,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 12319.861582998783,
"residentMemoryBytes": 30621990912,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3945.2491249994637,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 4231.950459001382,
"inferenceMs": 4231.638958000985,
"tokensPerSecond": 2.835780679566474,
"inputTokens": 32,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 8177.99579199891,
"residentMemoryBytes": 30621990912,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3894.951875001425,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 4933.41379100093,
"inferenceMs": 4929.271834000247,
"tokensPerSecond": 2.4344366478692763,
"inputTokens": 32,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 8829.159250000885,
"residentMemoryBytes": 30605721600,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3894.951875001425,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 7847.231791998638,
"inferenceMs": 7846.855708001385,
"tokensPerSecond": 2.6762311913785344,
"inputTokens": 39,
"outputTokens": 21,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 11742.96379200132,
"residentMemoryBytes": 30610014208,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3894.951875001425,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 3234.80387500058,
"inferenceMs": 3234.3485829987912,
"tokensPerSecond": 2.4734501537810867,
"inputTokens": 35,
"outputTokens": 8,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 7130.638041000566,
"residentMemoryBytes": 30614192128,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3894.951875001425,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 4344.249707999552,
"inferenceMs": 4343.851790999906,
"tokensPerSecond": 2.76252519132052,
"inputTokens": 31,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 8240.01033300192,
"residentMemoryBytes": 30620499968,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.8772489985666848,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3894.951875001425,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 6270.701542000097,
"inferenceMs": 6270.2846660013165,
"tokensPerSecond": 2.711200671984998,
"inputTokens": 32,
"outputTokens": 17,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 10166.503417000058,
"residentMemoryBytes": 30620745728,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3894.951875001425,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 8782.950540999082,
"inferenceMs": 8782.525125001484,
"tokensPerSecond": 2.8465617398385494,
"inputTokens": 30,
"outputTokens": 25,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 12678.93712500154,
"residentMemoryBytes": 30620745728,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3894.951875001425,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 4364.486624999699,
"inferenceMs": 4364.040583999667,
"tokensPerSecond": 2.7497452805541815,
"inputTokens": 32,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 8260.30216700019,
"residentMemoryBytes": 30620909568,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3909.921165999549,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 4975.298750001457,
"inferenceMs": 4970.807874999082,
"tokensPerSecond": 2.414094509738463,
"inputTokens": 32,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 2,
"totalMs": 8886.006124999767,
"residentMemoryBytes": 30604869632,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3909.921165999549,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 7286.183709000397,
"inferenceMs": 7285.776500000793,
"tokensPerSecond": 2.8823283283528824,
"inputTokens": 39,
"outputTokens": 21,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 2,
"totalMs": 11196.97558199914,
"residentMemoryBytes": 30612930560,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3909.921165999549,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 3080.272249999325,
"inferenceMs": 3079.848833000142,
"tokensPerSecond": 2.5975300846850464,
"inputTokens": 35,
"outputTokens": 8,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 2,
"totalMs": 6991.044791000604,
"residentMemoryBytes": 30616076288,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3909.921165999549,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 4201.54999999977,
"inferenceMs": 4201.203041999179,
"tokensPerSecond": 2.856324695578078,
"inputTokens": 31,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 2,
"totalMs": 8112.319666000985,
"residentMemoryBytes": 30618222592,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.8772489985666848,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3909.921165999549,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 5877.0284579986765,
"inferenceMs": 5876.462333000745,
"tokensPerSecond": 2.8928969568191127,
"inputTokens": 32,
"outputTokens": 17,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 2,
"totalMs": 9787.875207999605,
"residentMemoryBytes": 30618255360,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3909.921165999549,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 8399.129334000463,
"inferenceMs": 8398.718917000224,
"tokensPerSecond": 2.9766444438801707,
"inputTokens": 30,
"outputTokens": 25,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 2,
"totalMs": 12309.84024999998,
"residentMemoryBytes": 30618419200,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3909.921165999549,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 4181.007582999882,
"inferenceMs": 4180.699792001178,
"tokensPerSecond": 2.8703328622062942,
"inputTokens": 32,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 2,
"totalMs": 8091.7292489994,
"residentMemoryBytes": 30618419200,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4066.322250000667,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 4542.732040999908,
"inferenceMs": 4538.781625000411,
"tokensPerSecond": 2.6438813301573885,
"inputTokens": 32,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 2,
"totalMs": 8609.964458000832,
"residentMemoryBytes": 30606409728,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4066.322250000667,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 7108.121750001374,
"inferenceMs": 7107.7397919998475,
"tokensPerSecond": 2.9545257162673084,
"inputTokens": 39,
"outputTokens": 21,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 2,
"totalMs": 11175.250416999916,
"residentMemoryBytes": 30608703488,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4066.322250000667,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 3128.2166670007427,
"inferenceMs": 3127.816915999574,
"tokensPerSecond": 2.5576944606565615,
"inputTokens": 35,
"outputTokens": 8,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 2,
"totalMs": 7195.364000001064,
"residentMemoryBytes": 30612783104,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4066.322250000667,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 4251.979165999728,
"inferenceMs": 4251.642457998969,
"tokensPerSecond": 2.822438650132351,
"inputTokens": 31,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 2,
"totalMs": 8319.148915999904,
"residentMemoryBytes": 30616961024,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.8772489985666848,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4066.322250000667,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 5993.992457999411,
"inferenceMs": 5993.595500000083,
"tokensPerSecond": 2.836360912243705,
"inputTokens": 32,
"outputTokens": 17,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 2,
"totalMs": 10061.203500001284,
"residentMemoryBytes": 30617059328,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4066.322250000667,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 8310.341374999552,
"inferenceMs": 8309.993082999426,
"tokensPerSecond": 3.0084260901666657,
"inputTokens": 30,
"outputTokens": 25,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 2,
"totalMs": 12377.470583000104,
"residentMemoryBytes": 30617911296,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4066.322250000667,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 4085.2622920010617,
"inferenceMs": 4084.950750000644,
"tokensPerSecond": 2.9376119161285135,
"inputTokens": 32,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 2,
"totalMs": 8152.549792001082,
"residentMemoryBytes": 30617911296,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3910.641000000396,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 4844.899125000666,
"inferenceMs": 4840.623124999183,
"tokensPerSecond": 2.4790196819964216,
"inputTokens": 32,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 2,
"totalMs": 8756.358333001117,
"residentMemoryBytes": 30602002432,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3910.641000000396,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 7239.205292000406,
"inferenceMs": 7238.761124999655,
"tokensPerSecond": 2.9010489001322033,
"inputTokens": 39,
"outputTokens": 21,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 2,
"totalMs": 11150.688458001241,
"residentMemoryBytes": 30612013056,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3910.641000000396,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 2985.626042000149,
"inferenceMs": 2985.2354169997852,
"tokensPerSecond": 2.6798556503929403,
"inputTokens": 35,
"outputTokens": 8,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 2,
"totalMs": 6897.072833999118,
"residentMemoryBytes": 30612062208,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3910.641000000396,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 4196.046208000553,
"inferenceMs": 4195.742042000347,
"tokensPerSecond": 2.8600423667320887,
"inputTokens": 31,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 2,
"totalMs": 8107.486667000558,
"residentMemoryBytes": 30616518656,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.8772489985666848,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3910.641000000396,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 5608.752417001597,
"inferenceMs": 5608.377624999775,
"tokensPerSecond": 3.0311796274596756,
"inputTokens": 32,
"outputTokens": 17,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 2,
"totalMs": 9520.176083000479,
"residentMemoryBytes": 30616518656,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3910.641000000396,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 8218.601124999623,
"inferenceMs": 8218.29975000037,
"tokensPerSecond": 3.041991745311903,
"inputTokens": 30,
"outputTokens": 25,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 2,
"totalMs": 12130.04070900024,
"residentMemoryBytes": 30616911872,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3910.641000000396,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 4022.1556659998896,
"inferenceMs": 4021.8333749999147,
"tokensPerSecond": 2.9837138640782834,
"inputTokens": 32,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 2,
"totalMs": 7933.63516700083,
"residentMemoryBytes": 30616928256,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4574.295749998782,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1055.83895899872,
"inferenceMs": 1050.4177920010989,
"tokensPerSecond": 11.424025841317286,
"inputTokens": 32,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15088517120,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 5631.066916997952,
"residentMemoryBytes": 704987136,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "good",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "mps",
"requestedPrecision": "fp16",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4574.295749998782,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1470.30787499898,
"inferenceMs": 1467.8658329994505,
"tokensPerSecond": 14.306484644504877,
"inputTokens": 39,
"outputTokens": 21,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15097888768,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 6045.597374997669,
"residentMemoryBytes": 773537792,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "good",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "mps",
"requestedPrecision": "fp16",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4574.295749998782,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 663.8540000003559,
"inferenceMs": 661.5540000002511,
"tokensPerSecond": 12.09273921705101,
"inputTokens": 35,
"outputTokens": 8,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15098265600,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 5238.965957998516,
"residentMemoryBytes": 786661376,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "good",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "mps",
"requestedPrecision": "fp16",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4574.295749998782,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 905.468375000055,
"inferenceMs": 903.4549170009996,
"tokensPerSecond": 13.282345111180266,
"inputTokens": 31,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15098609664,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 5480.917040998975,
"residentMemoryBytes": 803717120,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "good",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "mps",
"requestedPrecision": "fp16",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.8772489985666848,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4574.295749998782,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1171.9027500002994,
"inferenceMs": 1169.818624999607,
"tokensPerSecond": 14.532167326371395,
"inputTokens": 32,
"outputTokens": 17,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15098609664,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 5747.296540999741,
"residentMemoryBytes": 804962304,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "good",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "mps",
"requestedPrecision": "fp16",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4574.295749998782,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1807.6829590008856,
"inferenceMs": 1805.7072499996139,
"tokensPerSecond": 13.84499065394202,
"inputTokens": 30,
"outputTokens": 25,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15098937344,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 6383.049208998273,
"residentMemoryBytes": 822607872,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "good",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "mps",
"requestedPrecision": "fp16",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4574.295749998782,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 904.1937920010241,
"inferenceMs": 902.283333000014,
"tokensPerSecond": 13.29959178133219,
"inputTokens": 32,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15098937344,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 5479.58270899835,
"residentMemoryBytes": 823541760,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "good",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "mps",
"requestedPrecision": "fp16",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4667.247833998772,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1026.13966699937,
"inferenceMs": 1020.0371249993623,
"tokensPerSecond": 11.764277697252933,
"inputTokens": 32,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15088517120,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 5694.2606669981615,
"residentMemoryBytes": 706691072,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "good",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "mps",
"requestedPrecision": "fp16",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4667.247833998772,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1461.0373329996946,
"inferenceMs": 1459.3446250000852,
"tokensPerSecond": 14.390021137055802,
"inputTokens": 39,
"outputTokens": 21,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15097888768,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 6129.278291999071,
"residentMemoryBytes": 776830976,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "good",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "mps",
"requestedPrecision": "fp16",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4667.247833998772,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 668.9497909992497,
"inferenceMs": 666.9504170004075,
"tokensPerSecond": 11.9948946669526,
"inputTokens": 35,
"outputTokens": 8,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15098265600,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 5336.987749999025,
"residentMemoryBytes": 790085632,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "good",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "mps",
"requestedPrecision": "fp16",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4667.247833998772,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 915.9534579994215,
"inferenceMs": 913.7914590010041,
"tokensPerSecond": 13.13209910401101,
"inputTokens": 31,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15098609664,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 5584.252583998023,
"residentMemoryBytes": 806993920,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "good",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "mps",
"requestedPrecision": "fp16",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.8772489985666848,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4667.247833998772,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1169.325166998533,
"inferenceMs": 1167.1897910000553,
"tokensPerSecond": 14.56489778361949,
"inputTokens": 32,
"outputTokens": 17,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15098609664,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 5837.45987599832,
"residentMemoryBytes": 808435712,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "good",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "mps",
"requestedPrecision": "fp16",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4667.247833998772,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1710.9502919993247,
"inferenceMs": 1708.9125829988916,
"tokensPerSecond": 14.62918597985197,
"inputTokens": 30,
"outputTokens": 25,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15098937344,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 6379.165791999185,
"residentMemoryBytes": 826671104,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "good",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "mps",
"requestedPrecision": "fp16",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4667.247833998772,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 856.6502920002677,
"inferenceMs": 855.3774170013639,
"tokensPerSecond": 14.028895036845315,
"inputTokens": 32,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15098937344,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 5524.878291998903,
"residentMemoryBytes": 827637760,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "good",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "mps",
"requestedPrecision": "fp16",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4565.782250001575,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1061.2369589998707,
"inferenceMs": 1056.0281660000328,
"tokensPerSecond": 11.3633332768509,
"inputTokens": 32,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15088517120,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 5627.958292001495,
"residentMemoryBytes": 702185472,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "good",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "mps",
"requestedPrecision": "fp16",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4565.782250001575,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1461.458624999068,
"inferenceMs": 1459.5982920000097,
"tokensPerSecond": 14.387520261636386,
"inputTokens": 39,
"outputTokens": 21,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15097888768,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 6028.138041001512,
"residentMemoryBytes": 770736128,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "good",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "mps",
"requestedPrecision": "fp16",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4565.782250001575,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 660.0942080003733,
"inferenceMs": 658.8478329995269,
"tokensPerSecond": 12.142409216978853,
"inputTokens": 35,
"outputTokens": 8,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15098265600,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 5226.789625001402,
"residentMemoryBytes": 783794176,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "good",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "mps",
"requestedPrecision": "fp16",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4565.782250001575,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 914.5162089989753,
"inferenceMs": 912.7977909993206,
"tokensPerSecond": 13.146394654244876,
"inputTokens": 31,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15098609664,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 5481.202334001864,
"residentMemoryBytes": 800702464,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "good",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "mps",
"requestedPrecision": "fp16",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.8772489985666848,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4565.782250001575,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1153.0442090006545,
"inferenceMs": 1151.0140419995878,
"tokensPerSecond": 14.769585235004534,
"inputTokens": 32,
"outputTokens": 17,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15098609664,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 5719.807541001501,
"residentMemoryBytes": 802062336,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "good",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "mps",
"requestedPrecision": "fp16",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4565.782250001575,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1689.92791600067,
"inferenceMs": 1688.0604160014627,
"tokensPerSecond": 14.809896472318167,
"inputTokens": 30,
"outputTokens": 25,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15098937344,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 6256.759125000826,
"residentMemoryBytes": 819494912,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "good",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "mps",
"requestedPrecision": "fp16",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4565.782250001575,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 847.2154589999263,
"inferenceMs": 845.6187919982767,
"tokensPerSecond": 14.19079154052723,
"inputTokens": 32,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15098937344,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 5414.094375002605,
"residentMemoryBytes": 820412416,
"resourceMeasurementScope": "Post-inference process RSS; MPS driver allocation where available, not peak or a validated minimum",
"grade": "good",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "mps",
"requestedPrecision": "fp16",
"fallback": false
}
},
{
"precision": "bf16",
"device": "mps",
"status": "fail",
"quality": {
"passed": false,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": false
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 23205.868708000708,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 3050.0502079994476,
"inferenceMs": 3041.5640420014824,
"tokensPerSecond": 3.9453385936610013,
"inputTokens": 32,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15088517120,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"precision": "bf16",
"cpuThreads": 4,
"totalMs": 26256.862499001727,
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "mps",
"requestedPrecision": "bf16",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 12264.60891700117,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 8450.321665999581,
"inferenceMs": 8442.504000000554,
"tokensPerSecond": 1.4213792495685182,
"inputTokens": 32,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 20715.750749999643,
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 12264.60891700117,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 7282.779500001197,
"inferenceMs": 7282.349166000131,
"tokensPerSecond": 2.883684855162522,
"inputTokens": 39,
"outputTokens": 21,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 19548.181125001065,
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 12264.60891700117,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 3086.745249998785,
"inferenceMs": 3086.4015410006687,
"tokensPerSecond": 2.5920152947455604,
"inputTokens": 35,
"outputTokens": 8,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 15352.081042001373,
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 12264.60891700117,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 4238.076084000568,
"inferenceMs": 4237.744000000021,
"tokensPerSecond": 2.831695354886926,
"inputTokens": 31,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 16503.47645900183,
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.8772489985666848,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 12264.60891700117,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 5818.490167001073,
"inferenceMs": 5818.08970900056,
"tokensPerSecond": 2.921921257711285,
"inputTokens": 32,
"outputTokens": 17,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 18083.896167001512,
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 12264.60891700117,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 8898.271042000488,
"inferenceMs": 8897.999082999377,
"tokensPerSecond": 2.8096204289080338,
"inputTokens": 30,
"outputTokens": 25,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 21163.626667001154,
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4907.014500000514,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 4808.753625000463,
"inferenceMs": 4804.160792000403,
"tokensPerSecond": 2.497834797699043,
"inputTokens": 32,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 2,
"totalMs": 9716.658749999624,
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4907.014500000514,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 7402.894999999262,
"inferenceMs": 7402.5348750001285,
"tokensPerSecond": 2.8368660674495825,
"inputTokens": 39,
"outputTokens": 21,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 2,
"totalMs": 12310.727000001862,
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4907.014500000514,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 3842.158749999726,
"inferenceMs": 3841.842165998969,
"tokensPerSecond": 2.082334373546502,
"inputTokens": 35,
"outputTokens": 8,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 2,
"totalMs": 8750.225834000958,
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4907.014500000514,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 10471.883041000183,
"inferenceMs": 10471.499625000433,
"tokensPerSecond": 1.1459676674533141,
"inputTokens": 31,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 2,
"totalMs": 15380.878500000108,
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.8772489985666848,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4907.014500000514,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 16483.510458998353,
"inferenceMs": 16482.942583001204,
"tokensPerSecond": 1.03136924213593,
"inputTokens": 32,
"outputTokens": 17,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 2,
"totalMs": 21391.742499999964,
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4907.014500000514,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 17719.418915999995,
"inferenceMs": 17719.05975000118,
"tokensPerSecond": 1.4109100794695575,
"inputTokens": 30,
"outputTokens": 25,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 2,
"totalMs": 22627.242042000944,
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "fail",
"quality": {
"passed": false,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": false
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 22668.143791999682,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 3616.6300829991087,
"inferenceMs": 3609.568040999875,
"tokensPerSecond": 3.324497519840606,
"inputTokens": 32,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 30131388416,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 26285.963624999567,
"grade": "acceptable",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "mps",
"requestedPrecision": "fp32",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5109.491499999422,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1555.3694169993832,
"inferenceMs": 1550.3912500007573,
"tokensPerSecond": 7.739981762664192,
"inputTokens": 32,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15088517120,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 6665.8748329991795,
"grade": "good",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "mps",
"requestedPrecision": "fp16",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5109.491499999422,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1694.610124999599,
"inferenceMs": 1693.4167920007894,
"tokensPerSecond": 12.400963601635414,
"inputTokens": 39,
"outputTokens": 21,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15097888768,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 6805.068833000405,
"grade": "good",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "mps",
"requestedPrecision": "fp16",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5109.491499999422,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 672.4714169995423,
"inferenceMs": 669.121916000222,
"tokensPerSecond": 11.955967677491744,
"inputTokens": 35,
"outputTokens": 8,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15098265600,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 5782.86295900034,
"grade": "good",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "mps",
"requestedPrecision": "fp16",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5109.491499999422,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1104.9929999990127,
"inferenceMs": 1103.0618340009823,
"tokensPerSecond": 10.878809899961885,
"inputTokens": 31,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15098609664,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 6215.443333998337,
"grade": "good",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "mps",
"requestedPrecision": "fp16",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.8772489985666848,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5109.491499999422,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1184.4537500001024,
"inferenceMs": 1182.711792000191,
"tokensPerSecond": 14.37374693901526,
"inputTokens": 32,
"outputTokens": 17,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15098609664,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 6294.840291999208,
"grade": "good",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "mps",
"requestedPrecision": "fp16",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5109.491499999422,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1850.5546249998588,
"inferenceMs": 1849.0752080015227,
"tokensPerSecond": 13.520272129450028,
"inputTokens": 30,
"outputTokens": 25,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15098937344,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 6960.866916999294,
"grade": "good",
"artifactRevision": "9b0eb4e8f001def3e5ff6469a0ac96fdb39ec223",
"modelId": "hy-mt2-7b-native",
"requestedDevice": "mps",
"requestedPrecision": "fp16",
"fallback": false
}
}
]