Granite 4.0
Local chat, drafting and text extraction using IBM Granite 4.0 Micro.
Desktop app
Run Granite 4.0 on your computer
Download the desktop app to load this model, provide input and view results locally.
Public desktop downloads are coming soon.Device requirements
If your device falls short
MemoryAdvisory only: close other apps, reduce image/tile/token/batch settings or choose a smaller verified model/precision. CPU can need more RAM than GPU.
GPUIf the selected GPU runtime fails, existing CPU fallback may recover when enabled, but CPU has its own memory/runtime needs.
PlatformUse a compatible OS/architecture/runtime or another model. These estimates do not add hardware detection or block runs.
Benchmark evidence
Saved desktop measurements apply to the recorded hardware and runtime. Fallback cases report the device that actually ran; they do not prove GPU support on the requested device.
Granite 4.0 Micro Desktop
77 configurations testedQuality criteria
No explicit quality gate recorded. Review per-case evidence; full certification is pending.
| Precision | Device | Result | Platform | Inference / load | |
|---|---|---|---|---|---|
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 317 ms / 1678 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 318 ms / 1635 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 323 ms / 1698 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 346 ms / 1698 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 353 ms / 1678 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 357 ms / 1635 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 143 ms / 1698 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 143 ms / 1635 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 143 ms / 1698 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 143 ms / 1678 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 144 ms / 1698 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 145 ms / 1678 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 145 ms / 1635 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 147 ms / 1635 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 148 ms / 1678 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 167 ms / 1698 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 167 ms / 1698 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 167 ms / 1635 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 168 ms / 1635 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 168 ms / 1678 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 168 ms / 1678 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 186 ms / 1698 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 189 ms / 1635 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 191 ms / 1678 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 197 ms / 1698 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 198 ms / 1698 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 200 ms / 1678 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 200 ms / 1635 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 201 ms / 1635 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 202 ms / 1698 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 205 ms / 1635 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 211 ms / 1678 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 237 ms / 1678 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 1309 ms / 1354 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 1319 ms / 1300 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 1351 ms / 1311 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 1369 ms / 4598 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 1376 ms / 1354 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 1396 ms / 4598 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 1417 ms / 1311 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 1418 ms / 1300 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 534 ms / 1300 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 535 ms / 1311 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 539 ms / 1300 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 540 ms / 1311 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 542 ms / 1354 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 544 ms / 1354 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 548 ms / 1300 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 549 ms / 1354 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 553 ms / 1300 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 556 ms / 4598 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 563 ms / 4598 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 574 ms / 1311 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 576 ms / 1354 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 595 ms / 4598 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 598 ms / 4598 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 599 ms / 1311 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 599 ms / 1300 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 601 ms / 1311 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 605 ms / 1354 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 608 ms / 1300 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 610 ms / 1300 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 611 ms / 1354 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 612 ms / 1300 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 612 ms / 1354 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 614 ms / 1311 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 614 ms / 1354 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 614 ms / 1311 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 619 ms / 1354 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 624 ms / 1300 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 625 ms / 1311 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 627 ms / 1311 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 633 ms / 4598 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 641 ms / 4598 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 646 ms / 4598 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 673 ms / 4598 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 688 ms / 4598 ms |
Test setup & sample details
- Quality gate
- No explicit quality gate recorded. Review the per-case quality evidence; full certification is pending.
- Tested
- 2026-11-04T10:38:04.717566+00:00
- Environment
- { "os": "macos", "arch": "aarch64", "hardware": "Apple M3 Max", "osVersion": "27.2" }
- Runtime
- transformers 5.22.1
- Artifact revision
- 56111ae135df9c53a78c99028e7bc24035a9e979
- Fixture
{ "file": "model-samples/desktop/granite4/granite-4.0-micro-3b-native-capital.json", "size": 244, "sha1": "9013b4f5a71cd1280d78b3dfa751ead1186d071f", "sha256": "8ff34789fa2a0f5bbef2080057f4b36376d3c73ea64baccefb6caec96eafbd68", "url": "https://assets.localai.tools/samples/v1/files/8ff34789fa2a0f5bbef2080057f4b36376d3c73ea64baccefb6caec96eafbd68/granite-4.0-micro-3b-native-capital.json" }- Rerun when
- Artifact/runtime revision; Input/generation processing; Fixture/quality gates; OS/backend behavior
Per-case quality & failure details
[
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "Paris"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4597.526416997425,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 604.9998329835944,
"inferenceMs": 594.7437499999069,
"tokensPerSecond": 3.3627927994204447,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp32",
"cpuThreads": 1,
"rssBytes": 14061158400,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 5202.52624998102,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "56"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4597.526416997425,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 598.7748749903403,
"inferenceMs": 598.3119160227943,
"tokensPerSecond": 3.3427380375352653,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp32",
"cpuThreads": 1,
"rssBytes": 14065467392,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 5196.301291987766,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4597.526416997425,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 673.2560409873258,
"inferenceMs": 672.8133749857079,
"tokensPerSecond": 2.9725925113223033,
"inputTokens": 68,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp32",
"cpuThreads": 1,
"rssBytes": 14072266752,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 5270.782457984751,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "hola"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4597.526416997425,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 688.9472909970209,
"inferenceMs": 688.3237499860115,
"tokensPerSecond": 2.9056094607234533,
"inputTokens": 69,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp32",
"cpuThreads": 1,
"rssBytes": 14080163840,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 5286.473707994446,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "2026-11-04"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4597.526416997425,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 1369.7359159996267,
"inferenceMs": 1369.301375001669,
"tokensPerSecond": 5.112095940158877,
"inputTokens": 72,
"outputTokens": 7,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp32",
"cpuThreads": 1,
"rssBytes": 14084915200,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 5967.262332997052,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "Unknown"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4597.526416997425,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 641.4512920018751,
"inferenceMs": 641.05129201198,
"tokensPerSecond": 3.1198751565149703,
"inputTokens": 74,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp32",
"cpuThreads": 1,
"rssBytes": 14096252928,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 5238.9777089993,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "Paris"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4597.526416997425,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 563.3985000022221,
"inferenceMs": 563.0373330204748,
"tokensPerSecond": 3.5521623215831593,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp32",
"cpuThreads": 1,
"rssBytes": 14096269312,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 5160.924916999647,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "56"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4597.526416997425,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 556.2327920051757,
"inferenceMs": 555.8991660072934,
"tokensPerSecond": 3.5977747805683165,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp32",
"cpuThreads": 1,
"rssBytes": 14096269312,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 5153.759209002601,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4597.526416997425,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 646.3567079917993,
"inferenceMs": 645.9092920122202,
"tokensPerSecond": 3.0964100141203126,
"inputTokens": 68,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp32",
"cpuThreads": 1,
"rssBytes": 14096269312,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 5243.883124989225,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "hola"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4597.526416997425,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 632.995584019227,
"inferenceMs": 632.617041002959,
"tokensPerSecond": 3.16147032149051,
"inputTokens": 69,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp32",
"cpuThreads": 1,
"rssBytes": 14098956288,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 5230.522001016652,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "2026-11-04"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4597.526416997425,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 1396.2768329947721,
"inferenceMs": 1395.8155839936808,
"tokensPerSecond": 5.014989143459578,
"inputTokens": 72,
"outputTokens": 7,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp32",
"cpuThreads": 1,
"rssBytes": 14098956288,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 5993.803249992197,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "Paris"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1310.5549169995356,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 583.8928329758346,
"inferenceMs": 574.2461670015473,
"tokensPerSecond": 3.4828269040838213,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp32",
"cpuThreads": 4,
"rssBytes": 14055866368,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1894.4477499753702,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "56"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1310.5549169995356,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 614.3457500147633,
"inferenceMs": 613.9265839883592,
"tokensPerSecond": 3.2577185157988247,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp32",
"cpuThreads": 4,
"rssBytes": 14069399552,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1924.900667014299,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1310.5549169995356,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 626.9493330037221,
"inferenceMs": 626.5720420051366,
"tokensPerSecond": 3.191971339161035,
"inputTokens": 68,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp32",
"cpuThreads": 4,
"rssBytes": 14077362176,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1937.5042500032578,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "hola"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1310.5549169995356,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 625.2870000025723,
"inferenceMs": 624.8886669927742,
"tokensPerSecond": 3.200570126555242,
"inputTokens": 69,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp32",
"cpuThreads": 4,
"rssBytes": 14083457024,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1935.841917002108,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "2026-11-04"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1310.5549169995356,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 1351.646916999016,
"inferenceMs": 1351.2178749952,
"tokensPerSecond": 5.18051169210951,
"inputTokens": 72,
"outputTokens": 7,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp32",
"cpuThreads": 4,
"rssBytes": 14090059776,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 2662.2018339985516,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "Unknown"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1310.5549169995356,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 614.6077919984236,
"inferenceMs": 614.2186670040246,
"tokensPerSecond": 3.256169353750519,
"inputTokens": 74,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp32",
"cpuThreads": 4,
"rssBytes": 14096384000,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1925.1627089979593,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "Paris"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1310.5549169995356,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 539.8697500058915,
"inferenceMs": 539.5069590013009,
"tokensPerSecond": 3.7070884195864053,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp32",
"cpuThreads": 4,
"rssBytes": 14096384000,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1850.4246670054272,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "56"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1310.5549169995356,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 535.4864169785287,
"inferenceMs": 535.1558749971446,
"tokensPerSecond": 3.7372288961803353,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp32",
"cpuThreads": 4,
"rssBytes": 14096384000,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1846.0413339780644,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1310.5549169995356,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 601.2195000075735,
"inferenceMs": 600.8753750065807,
"tokensPerSecond": 3.3284772237139792,
"inputTokens": 68,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp32",
"cpuThreads": 4,
"rssBytes": 14096416768,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1911.7744170071092,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "hola"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1310.5549169995356,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 599.3902499903925,
"inferenceMs": 599.0102089999709,
"tokensPerSecond": 3.3388412583801172,
"inputTokens": 69,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp32",
"cpuThreads": 4,
"rssBytes": 14096695296,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1909.945166989928,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "2026-11-04"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1310.5549169995356,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 1416.9614580168854,
"inferenceMs": 1416.6262499929871,
"tokensPerSecond": 4.941317443492702,
"inputTokens": 72,
"outputTokens": 7,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp32",
"cpuThreads": 4,
"rssBytes": 14096695296,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 2727.516375016421,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "Paris"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1299.7478340112139,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 548.2074579922482,
"inferenceMs": 538.9182500075549,
"tokensPerSecond": 3.7111380065009167,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp32",
"cpuThreads": 4,
"rssBytes": 14053965824,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1847.955292003462,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "56"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1299.7478340112139,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 548.6775409954134,
"inferenceMs": 548.1935419957153,
"tokensPerSecond": 3.648346517762575,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp32",
"cpuThreads": 4,
"rssBytes": 14058192896,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1848.4253750066273,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1299.7478340112139,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 624.5060410001315,
"inferenceMs": 624.1044170164969,
"tokensPerSecond": 3.2045919648524683,
"inputTokens": 68,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp32",
"cpuThreads": 4,
"rssBytes": 14068514816,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1924.2538750113454,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "hola"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1299.7478340112139,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 612.2970000142232,
"inferenceMs": 611.8591249978635,
"tokensPerSecond": 3.268726277485824,
"inputTokens": 69,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp32",
"cpuThreads": 4,
"rssBytes": 14074576896,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1912.044834025437,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "2026-11-04"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1299.7478340112139,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 1319.8480420105625,
"inferenceMs": 1319.473832991207,
"tokensPerSecond": 5.305144994146048,
"inputTokens": 72,
"outputTokens": 7,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp32",
"cpuThreads": 4,
"rssBytes": 14084325376,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 2619.5958760217763,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "Unknown"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1299.7478340112139,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 608.1316669879016,
"inferenceMs": 607.7278750017285,
"tokensPerSecond": 3.290946626389832,
"inputTokens": 74,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp32",
"cpuThreads": 4,
"rssBytes": 14092369920,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1907.8795009991154,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "Paris"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1299.7478340112139,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 553.8077499950305,
"inferenceMs": 553.4598340163939,
"tokensPerSecond": 3.6136316984129304,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp32",
"cpuThreads": 4,
"rssBytes": 14092337152,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1853.5555840062443,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "56"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1299.7478340112139,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 534.8343749938067,
"inferenceMs": 534.4896249880549,
"tokensPerSecond": 3.7418874127719457,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp32",
"cpuThreads": 4,
"rssBytes": 14092337152,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1834.5822090050206,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1299.7478340112139,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 599.478292017011,
"inferenceMs": 599.0963749936782,
"tokensPerSecond": 3.338361044199449,
"inputTokens": 68,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp32",
"cpuThreads": 4,
"rssBytes": 14092337152,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1899.2261260282248,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "hola"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1299.7478340112139,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 610.735583002679,
"inferenceMs": 610.3197500051465,
"tokensPerSecond": 3.2769708009336664,
"inputTokens": 69,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp32",
"cpuThreads": 4,
"rssBytes": 14092337152,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1910.483417013893,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "2026-11-04"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1299.7478340112139,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 1418.831582996063,
"inferenceMs": 1418.4050830081105,
"tokensPerSecond": 4.9351204982674,
"inputTokens": 72,
"outputTokens": 7,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp32",
"cpuThreads": 4,
"rssBytes": 14092353536,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 2718.579417007277,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "Paris"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1354.042249993654,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 551.7477910034359,
"inferenceMs": 542.1407080139033,
"tokensPerSecond": 3.6890791826477445,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp32",
"cpuThreads": 4,
"rssBytes": 14053277696,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1905.7900409970898,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "56"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1354.042249993654,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 576.3487500080373,
"inferenceMs": 575.9893330105115,
"tokensPerSecond": 3.47228652577061,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp32",
"cpuThreads": 4,
"rssBytes": 14055391232,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1930.3910000016913,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1354.042249993654,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 619.2984159861226,
"inferenceMs": 618.8890000048559,
"tokensPerSecond": 3.2315972653970384,
"inputTokens": 68,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp32",
"cpuThreads": 4,
"rssBytes": 14076067840,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1973.3406659797765,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "hola"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1354.042249993654,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 605.3586670022923,
"inferenceMs": 604.9653340014629,
"tokensPerSecond": 3.305974553568657,
"inputTokens": 69,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp32",
"cpuThreads": 4,
"rssBytes": 14081310720,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1959.4009169959463,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "2026-11-04"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1354.042249993654,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 1309.3345830275211,
"inferenceMs": 1308.965125004761,
"tokensPerSecond": 5.34773605979345,
"inputTokens": 72,
"outputTokens": 7,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp32",
"cpuThreads": 4,
"rssBytes": 14086062080,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 2663.376833021175,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "Unknown"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1354.042249993654,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 611.0214999935124,
"inferenceMs": 610.6169160048012,
"tokensPerSecond": 3.2753760133043452,
"inputTokens": 74,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp32",
"cpuThreads": 4,
"rssBytes": 14093942784,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1965.0637499871664,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "Paris"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1354.042249993654,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 548.9256670116447,
"inferenceMs": 548.5876670281868,
"tokensPerSecond": 3.645725414926688,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp32",
"cpuThreads": 4,
"rssBytes": 14093991936,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1902.9679170052987,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "56"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1354.042249993654,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 544.4394580263179,
"inferenceMs": 544.0798329946119,
"tokensPerSecond": 3.675931138619884,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp32",
"cpuThreads": 4,
"rssBytes": 14093991936,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1898.4817080199718,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1354.042249993654,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 612.6240420271643,
"inferenceMs": 612.267916003475,
"tokensPerSecond": 3.2665438572297307,
"inputTokens": 68,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp32",
"cpuThreads": 4,
"rssBytes": 14093991936,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1966.6662920208182,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "hola"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1354.042249993654,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 614.4819160108455,
"inferenceMs": 614.1169159964193,
"tokensPerSecond": 3.256708857718001,
"inputTokens": 69,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp32",
"cpuThreads": 4,
"rssBytes": 14095450112,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1968.5241660044994,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "2026-11-04"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1354.042249993654,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 1376.1902500118595,
"inferenceMs": 1375.8320409979206,
"tokensPerSecond": 5.087830339321615,
"inputTokens": 72,
"outputTokens": 7,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp32",
"cpuThreads": 4,
"rssBytes": 14095450112,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 2730.2325000055134,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "Paris"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1697.501624992583,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 322
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 196.6235419968143,
"inferenceMs": 185.52470800932497,
"tokensPerSecond": 10.780235266018986,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7570833408,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp16",
"cpuThreads": 4,
"rssBytes": 2113503232,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1894.1251669893973,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "56"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1697.501624992583,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 322
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 145.5222080112435,
"inferenceMs": 144.35595902614295,
"tokensPerSecond": 13.854641079540047,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7570817024,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp16",
"cpuThreads": 4,
"rssBytes": 2114371584,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1843.0238330038264,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1697.501624992583,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 322
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 203.40866700280458,
"inferenceMs": 201.74183300696313,
"tokensPerSecond": 9.91366029637973,
"inputTokens": 68,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7580745728,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp16",
"cpuThreads": 4,
"rssBytes": 2125643776,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1900.9102919953875,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "hola"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1697.501624992583,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 322
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 199.2749590135645,
"inferenceMs": 198.04854100220837,
"tokensPerSecond": 10.098534378890974,
"inputTokens": 69,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7582334976,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp16",
"cpuThreads": 4,
"rssBytes": 2135425024,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1896.7765840061475,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "2026-11-04"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1697.501624992583,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 322
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 347.57487499155104,
"inferenceMs": 345.96283300197683,
"tokensPerSecond": 20.233387324470204,
"inputTokens": 72,
"outputTokens": 7,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7584055296,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp16",
"cpuThreads": 4,
"rssBytes": 2145140736,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 2045.076499984134,
"grade": "fast",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "Unknown"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1697.501624992583,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 322
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 198.3077499899082,
"inferenceMs": 197.13629101170227,
"tokensPerSecond": 10.145265439133565,
"inputTokens": 74,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7585873920,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp16",
"cpuThreads": 4,
"rssBytes": 2154627072,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1895.8093749824911,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "Paris"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1697.501624992583,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 322
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 144.55950001138262,
"inferenceMs": 143.3369159931317,
"tokensPerSecond": 13.95313960916973,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7585873920,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp16",
"cpuThreads": 4,
"rssBytes": 2154971136,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1842.0611250039656,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "56"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1697.501624992583,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 322
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 144.5252090052236,
"inferenceMs": 142.8537920000963,
"tokensPerSecond": 14.00032839169332,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7585873920,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp16",
"cpuThreads": 4,
"rssBytes": 2155298816,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1842.0268339978065,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1697.501624992583,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 322
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 167.80179098714143,
"inferenceMs": 166.64208300062455,
"tokensPerSecond": 12.001770285075615,
"inputTokens": 68,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7585873920,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp16",
"cpuThreads": 4,
"rssBytes": 2155511808,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1865.3034159797244,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "hola"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1697.501624992583,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 322
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 168.29345800215378,
"inferenceMs": 166.95145799894817,
"tokensPerSecond": 11.9795300021435,
"inputTokens": 69,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7585873920,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp16",
"cpuThreads": 4,
"rssBytes": 2155855872,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1865.7950829947367,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "2026-11-04"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1697.501624992583,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 322
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 324.51579201733693,
"inferenceMs": 322.628750000149,
"tokensPerSecond": 21.69676446998839,
"inputTokens": 72,
"outputTokens": 7,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7585873920,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp16",
"cpuThreads": 4,
"rssBytes": 2156544000,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 2022.0174170099199,
"grade": "fast",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "Paris"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1635.3245840000454,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 322
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 199.96629198431037,
"inferenceMs": 188.96491700434126,
"tokensPerSecond": 10.583975225168661,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7570833408,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp16",
"cpuThreads": 4,
"rssBytes": 2110996480,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1835.2908759843558,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "56"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1635.3245840000454,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 322
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 144.64216699707322,
"inferenceMs": 143.0024580040481,
"tokensPerSecond": 13.98577358679656,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7570833408,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp16",
"cpuThreads": 4,
"rssBytes": 2111913984,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1779.9667509971187,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1635.3245840000454,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 322
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 202.43075001053512,
"inferenceMs": 200.85512500372715,
"tokensPerSecond": 9.957425781208656,
"inputTokens": 68,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7580745728,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp16",
"cpuThreads": 4,
"rssBytes": 2122743808,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1837.7553340105806,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "hola"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1635.3245840000454,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 322
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 201.52783297817223,
"inferenceMs": 199.88599998760037,
"tokensPerSecond": 10.005703251473674,
"inputTokens": 69,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7582334976,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp16",
"cpuThreads": 4,
"rssBytes": 2132148224,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1836.8524169782177,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "2026-11-04"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1635.3245840000454,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 322
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 358.4991250245366,
"inferenceMs": 356.81395800202154,
"tokensPerSecond": 19.61806662271979,
"inputTokens": 72,
"outputTokens": 7,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7584055296,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp16",
"cpuThreads": 4,
"rssBytes": 2141831168,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1993.823709024582,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "Unknown"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1635.3245840000454,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 322
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 206.46887499606237,
"inferenceMs": 204.67158398241736,
"tokensPerSecond": 9.771752194832347,
"inputTokens": 74,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7585873920,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp16",
"cpuThreads": 4,
"rssBytes": 2151071744,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1841.7934589961078,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "Paris"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1635.3245840000454,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 322
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 148.67225001216866,
"inferenceMs": 146.939416998066,
"tokensPerSecond": 13.611051689597513,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7585873920,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp16",
"cpuThreads": 4,
"rssBytes": 2151366656,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1783.996834012214,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "56"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1635.3245840000454,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 322
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 146.70850001857616,
"inferenceMs": 145.05137500236742,
"tokensPerSecond": 13.788218139727096,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7585873920,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp16",
"cpuThreads": 4,
"rssBytes": 2151596032,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1782.0330840186216,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1635.3245840000454,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 322
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 170.00762501265854,
"inferenceMs": 167.72308299550787,
"tokensPerSecond": 11.924417106341684,
"inputTokens": 68,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7585873920,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp16",
"cpuThreads": 4,
"rssBytes": 2151874560,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1805.332209012704,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "hola"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1635.3245840000454,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 322
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 169.15341702406295,
"inferenceMs": 167.4354170099832,
"tokensPerSecond": 11.944904105209423,
"inputTokens": 69,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7585873920,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp16",
"cpuThreads": 4,
"rssBytes": 2152202240,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1804.4780010241084,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "2026-11-04"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1635.3245840000454,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 322
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 319.94579199817963,
"inferenceMs": 318.2375839969609,
"tokensPerSecond": 21.996144867875973,
"inputTokens": 72,
"outputTokens": 7,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7585873920,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp16",
"cpuThreads": 4,
"rssBytes": 2152792064,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1955.270375998225,
"grade": "fast",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "Paris"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1677.6497910032049,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 322
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 202.12183400872163,
"inferenceMs": 191.03404198540375,
"tokensPerSecond": 10.469338235291138,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7570833408,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp16",
"cpuThreads": 4,
"rssBytes": 2112946176,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1879.7716250119265,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "56"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1677.6497910032049,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 322
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 150.36466700257733,
"inferenceMs": 148.35137501358986,
"tokensPerSecond": 13.481506321170183,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7570817024,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp16",
"cpuThreads": 4,
"rssBytes": 2114093056,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1828.0144580057822,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1677.6497910032049,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 322
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 239.3982089997735,
"inferenceMs": 237.19783299020492,
"tokensPerSecond": 8.43178023503524,
"inputTokens": 68,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7580745728,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp16",
"cpuThreads": 4,
"rssBytes": 2124644352,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1917.0480000029784,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "hola"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1677.6497910032049,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 322
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 212.91716600535437,
"inferenceMs": 211.3567090127617,
"tokensPerSecond": 9.462675726462225,
"inputTokens": 69,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7582334976,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp16",
"cpuThreads": 4,
"rssBytes": 2134589440,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1890.5669570085593,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "2026-11-04"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1677.6497910032049,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 322
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 354.79137499351054,
"inferenceMs": 352.5634169927798,
"tokensPerSecond": 19.854584062371263,
"inputTokens": 72,
"outputTokens": 7,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7584055296,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp16",
"cpuThreads": 4,
"rssBytes": 2144796672,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 2032.4411659967154,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "Unknown"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1677.6497910032049,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 322
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 200.72204098687507,
"inferenceMs": 199.5894580031745,
"tokensPerSecond": 10.020569322695337,
"inputTokens": 74,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7585873920,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp16",
"cpuThreads": 4,
"rssBytes": 2154266624,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1878.37183199008,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "Paris"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1677.6497910032049,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 322
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 145.79770900309086,
"inferenceMs": 144.51187499798834,
"tokensPerSecond": 13.839693104998055,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7585873920,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp16",
"cpuThreads": 4,
"rssBytes": 2154577920,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1823.4475000062957,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "56"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1677.6497910032049,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 322
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 144.72083299187943,
"inferenceMs": 143.38570900144987,
"tokensPerSecond": 13.9483914675191,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7585873920,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp16",
"cpuThreads": 4,
"rssBytes": 2154856448,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1822.3706239950843,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1677.6497910032049,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 322
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 169.23595900880173,
"inferenceMs": 168.0347920046188,
"tokensPerSecond": 11.902296995404534,
"inputTokens": 68,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7585873920,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp16",
"cpuThreads": 4,
"rssBytes": 2155151360,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1846.8857500120066,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "hola"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1677.6497910032049,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 322
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 170.19470798550174,
"inferenceMs": 168.23483299231157,
"tokensPerSecond": 11.888144472978443,
"inputTokens": 69,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7585873920,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp16",
"cpuThreads": 4,
"rssBytes": 2155380736,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1847.8444989887066,
"grade": "acceptable",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"passed": true,
"expected": "2026-11-04"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1677.6497910032049,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 322
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 318.5505839937832,
"inferenceMs": 316.9354590063449,
"tokensPerSecond": 22.086515727670168,
"inputTokens": 72,
"outputTokens": 7,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7585873920,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
"precision": "fp16",
"cpuThreads": 4,
"rssBytes": 2156036096,
"memoryScope": "Post-inference helper RSS, not peak/minimum",
"totalMs": 1996.200374996988,
"grade": "fast",
"loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
}
}
]Granite 4.0 Dense 1B
48 configurations testedQuality criteria
No explicit quality gate recorded. Review per-case evidence; full certification is pending.
| Precision | Device | Result | Platform | Inference / load | |
|---|---|---|---|---|---|
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 228 ms / 1547 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 229 ms / 1528 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 232 ms / 1602 ms |
BF16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 101 ms / 1528 ms |
BF16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 101 ms / 1547 ms |
BF16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 101 ms / 1602 ms |
BF16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 140 ms / 1547 ms |
BF16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 142 ms / 1528 ms |
BF16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 142 ms / 1547 ms |
BF16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 143 ms / 1547 ms |
BF16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 144 ms / 1602 ms |
BF16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 145 ms / 1528 ms |
BF16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 146 ms / 1547 ms |
BF16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 147 ms / 1602 ms |
BF16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 147 ms / 1528 ms |
BF16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 148 ms / 1528 ms |
BF16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 165 ms / 1602 ms |
BF16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 240 ms / 1602 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 701 ms / 1069 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 738 ms / 1013 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 742 ms / 1008 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 807 ms / 1038 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 279 ms / 1069 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 286 ms / 1013 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 288 ms / 1069 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 290 ms / 1008 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 292 ms / 1013 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 296 ms / 1008 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 301 ms / 1038 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 301 ms / 1038 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 308 ms / 1069 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 311 ms / 1069 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 312 ms / 1008 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 325 ms / 1013 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 325 ms / 1013 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 327 ms / 1008 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 331 ms / 1038 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 331 ms / 1069 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 332 ms / 1013 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 332 ms / 1008 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 340 ms / 1038 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 363 ms / 1038 ms |
FP16 | mps | Failed | — | macos · aarch64 | 789 ms / 1373 ms |
FP16 | mps | Failed | — | macos · aarch64 | 599 ms / 1373 ms |
FP16 | mps | Failed | — | macos · aarch64 | 666 ms / 1373 ms |
FP16 | mps | Failed | — | macos · aarch64 | 649 ms / 1373 ms |
FP16 | mps | Failed | — | macos · aarch64 | 653 ms / 1373 ms |
FP16 | mps | Failed | — | macos · aarch64 | 645 ms / 1373 ms |
Test setup & sample details
- Quality gate
- No explicit quality gate recorded. Review the per-case quality evidence; full certification is pending.
- Tested
- 2026-11-07T12:18:59.941540+00:00
- Environment
- { "os": "macos", "arch": "aarch64", "hardware": "Apple M3 Max", "osVersion": "27.2" }
- Runtime
- transformers 5.22.1
- Artifact revision
- 6a7381ba1f54d684ff508d991aeb7dc580157103
- Fixture
{ "file": "model-samples/desktop/granite4/granite-4.0-1b-native-capital.json", "size": 244, "sha1": "9013b4f5a71cd1280d78b3dfa751ead1186d071f", "sha256": "8ff34789fa2a0f5bbef2080057f4b36376d3c73ea64baccefb6caec96eafbd68", "url": "https://assets.localai.tools/samples/v1/files/8ff34789fa2a0f5bbef2080057f4b36376d3c73ea64baccefb6caec96eafbd68/granite-4.0-1b-native-capital.json" }- Rerun when
- Artifact/runtime revision; Input/generation processing; Fixture/quality gates; OS/backend behavior
Per-case quality & failure details
[
{
"precision": "fp16",
"device": "mps",
"status": "fail",
"quality": {
"exactNormalized": false,
"streamParity": true,
"withinTokenBudget": false,
"plainParity": true,
"passed": false,
"expected": "Paris"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
"torchVersion": "2.10.0",
"loadMs": 1372.7315830037696,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 322
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 800.7019579963526,
"inferenceMs": 789.4715830043424,
"tokensPerSecond": 40.533441214215294,
"inputTokens": 63,
"outputTokens": 32,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": true,
"driverAllocatedMemoryBytes": 4348821504,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 1745797120,
"cpuThreads": 4,
"totalMs": 2173.433541000122,
"grade": "fast",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "fail",
"quality": {
"exactNormalized": false,
"streamParity": true,
"withinTokenBudget": false,
"passed": false,
"expected": "56"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
"torchVersion": "2.10.0",
"loadMs": 1372.7315830037696,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 322
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 600.4489580009249,
"inferenceMs": 598.7156659975881,
"tokensPerSecond": 53.4477412524043,
"inputTokens": 63,
"outputTokens": 32,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": true,
"driverAllocatedMemoryBytes": 4348821504,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 1747779584,
"cpuThreads": 4,
"totalMs": 1973.1805410046945,
"grade": "fast",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "fail",
"quality": {
"exactNormalized": false,
"streamParity": true,
"withinTokenBudget": false,
"passed": false,
"expected": "chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
"torchVersion": "2.10.0",
"loadMs": 1372.7315830037696,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 322
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 667.8376249983557,
"inferenceMs": 666.1485830045422,
"tokensPerSecond": 48.037331034571615,
"inputTokens": 68,
"outputTokens": 32,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": true,
"driverAllocatedMemoryBytes": 4349476864,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 1758953472,
"cpuThreads": 4,
"totalMs": 2040.5692080021254,
"grade": "fast",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "fail",
"quality": {
"exactNormalized": false,
"streamParity": true,
"withinTokenBudget": false,
"passed": false,
"expected": "hola"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
"torchVersion": "2.10.0",
"loadMs": 1372.7315830037696,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 322
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 650.5502920044819,
"inferenceMs": 648.7943329993868,
"tokensPerSecond": 49.32225571709832,
"inputTokens": 69,
"outputTokens": 32,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": true,
"driverAllocatedMemoryBytes": 4350148608,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 1769439232,
"cpuThreads": 4,
"totalMs": 2023.2818750082515,
"grade": "fast",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "fail",
"quality": {
"exactNormalized": false,
"streamParity": true,
"withinTokenBudget": false,
"passed": false,
"expected": "2026-11-04"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
"torchVersion": "2.10.0",
"loadMs": 1372.7315830037696,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 322
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 654.2875000013737,
"inferenceMs": 653.0187089956598,
"tokensPerSecond": 49.00319019835722,
"inputTokens": 72,
"outputTokens": 32,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": true,
"driverAllocatedMemoryBytes": 4350869504,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 1779712000,
"cpuThreads": 4,
"totalMs": 2027.0190830051433,
"grade": "fast",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "fail",
"quality": {
"exactNormalized": false,
"streamParity": true,
"withinTokenBudget": false,
"passed": false,
"expected": "Unknown"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
"torchVersion": "2.10.0",
"loadMs": 1372.7315830037696,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 322
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 646.8709999971907,
"inferenceMs": 645.0455410013092,
"tokensPerSecond": 49.60890040465384,
"inputTokens": 74,
"outputTokens": 32,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": true,
"driverAllocatedMemoryBytes": 4351639552,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 1790148608,
"cpuThreads": 4,
"totalMs": 2019.6025830009603,
"grade": "fast",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"plainParity": true,
"passed": true,
"expected": "Paris"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
"torchVersion": "2.10.0",
"loadMs": 1037.506749999011,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 311.1241250007879,
"inferenceMs": 301.29562500223983,
"tokensPerSecond": 6.637998809259616,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 6971244544,
"cpuThreads": 4,
"totalMs": 1348.6308749997988,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "56"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
"torchVersion": "2.10.0",
"loadMs": 1037.506749999011,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 301.17291700298665,
"inferenceMs": 300.80766700120876,
"tokensPerSecond": 6.6487667017874355,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 6972325888,
"cpuThreads": 4,
"totalMs": 1338.6796670019976,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
"torchVersion": "2.10.0",
"loadMs": 1037.506749999011,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 340.6294579981477,
"inferenceMs": 340.2599159962847,
"tokensPerSecond": 5.877859559636869,
"inputTokens": 68,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 6984630272,
"cpuThreads": 4,
"totalMs": 1378.1362079971586,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "hola"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
"torchVersion": "2.10.0",
"loadMs": 1037.506749999011,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 331.48708299995633,
"inferenceMs": 331.09712499572197,
"tokensPerSecond": 6.040523607765672,
"inputTokens": 69,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 6989004800,
"cpuThreads": 4,
"totalMs": 1368.9938329989673,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "2026-11-04"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
"torchVersion": "2.10.0",
"loadMs": 1037.506749999011,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 806.9329999998445,
"inferenceMs": 806.5121250037919,
"tokensPerSecond": 8.679348745026108,
"inputTokens": 72,
"outputTokens": 7,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 6991609856,
"cpuThreads": 4,
"totalMs": 1844.4397499988554,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "Unknown"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
"torchVersion": "2.10.0",
"loadMs": 1037.506749999011,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 363.110708000022,
"inferenceMs": 362.6732080010697,
"tokensPerSecond": 5.514606416678292,
"inputTokens": 74,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 6994100224,
"cpuThreads": 4,
"totalMs": 1400.617457999033,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "Paris"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
"torchVersion": "2.10.0",
"loadMs": 1008.2986670022365,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 300.26425000687595,
"inferenceMs": 290.22729199641617,
"tokensPerSecond": 6.891150677947603,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 6966853632,
"cpuThreads": 4,
"totalMs": 1308.5629170091124,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "56"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
"torchVersion": "2.10.0",
"loadMs": 1008.2986670022365,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 296.5291669970611,
"inferenceMs": 296.16458300006343,
"tokensPerSecond": 6.753001927983981,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 6969163776,
"cpuThreads": 4,
"totalMs": 1304.8278339992976,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
"torchVersion": "2.10.0",
"loadMs": 1008.2986670022365,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 312.8949579986511,
"inferenceMs": 312.4881670009927,
"tokensPerSecond": 6.400242348996359,
"inputTokens": 68,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 6973718528,
"cpuThreads": 4,
"totalMs": 1321.1936250008876,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "hola"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
"torchVersion": "2.10.0",
"loadMs": 1008.2986670022365,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 327.1032920019934,
"inferenceMs": 326.75804199971026,
"tokensPerSecond": 6.120736884577652,
"inputTokens": 69,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 6976356352,
"cpuThreads": 4,
"totalMs": 1335.4019590042299,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "2026-11-04"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
"torchVersion": "2.10.0",
"loadMs": 1008.2986670022365,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 742.6865419984097,
"inferenceMs": 742.2947910017683,
"tokensPerSecond": 9.430215710598087,
"inputTokens": 72,
"outputTokens": 7,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 6978781184,
"cpuThreads": 4,
"totalMs": 1750.9852090006461,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "Unknown"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
"torchVersion": "2.10.0",
"loadMs": 1008.2986670022365,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 332.26579100301024,
"inferenceMs": 331.8535409998731,
"tokensPerSecond": 6.0267550377013,
"inputTokens": 74,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 6987300864,
"cpuThreads": 4,
"totalMs": 1340.5644580052467,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "Paris"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
"torchVersion": "2.10.0",
"loadMs": 1013.4504999950877,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 301.7093750022468,
"inferenceMs": 292.00954199768603,
"tokensPerSecond": 6.84909125338051,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 6964969472,
"cpuThreads": 4,
"totalMs": 1315.1598749973346,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "56"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
"torchVersion": "2.10.0",
"loadMs": 1013.4504999950877,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 286.80916700250236,
"inferenceMs": 286.4351660027751,
"tokensPerSecond": 6.982382882347006,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 6969245696,
"cpuThreads": 4,
"totalMs": 1300.25966699759,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
"torchVersion": "2.10.0",
"loadMs": 1013.4504999950877,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 325.0256250030361,
"inferenceMs": 324.6042920000036,
"tokensPerSecond": 6.161347983655058,
"inputTokens": 68,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 6971654144,
"cpuThreads": 4,
"totalMs": 1338.4761249981239,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "hola"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
"torchVersion": "2.10.0",
"loadMs": 1013.4504999950877,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 332.0834170008311,
"inferenceMs": 331.6714999964461,
"tokensPerSecond": 6.030062878545278,
"inputTokens": 69,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 6975389696,
"cpuThreads": 4,
"totalMs": 1345.5339169959188,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "2026-11-04"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
"torchVersion": "2.10.0",
"loadMs": 1013.4504999950877,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 737.9900000014459,
"inferenceMs": 737.5955410025199,
"tokensPerSecond": 9.490295983196685,
"inputTokens": 72,
"outputTokens": 7,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 6984810496,
"cpuThreads": 4,
"totalMs": 1751.4404999965336,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "Unknown"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
"torchVersion": "2.10.0",
"loadMs": 1013.4504999950877,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 325.5090000020573,
"inferenceMs": 325.0664169972879,
"tokensPerSecond": 6.152588810848112,
"inputTokens": 74,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 6993477632,
"cpuThreads": 4,
"totalMs": 1338.959499997145,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"plainParity": true,
"passed": true,
"expected": "Paris"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
"torchVersion": "2.10.0",
"loadMs": 1068.5277080046944,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 297.60641699976986,
"inferenceMs": 288.0704999988666,
"tokensPerSecond": 6.942744918371957,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 6971277312,
"cpuThreads": 1,
"totalMs": 1366.1341250044643,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "56"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
"torchVersion": "2.10.0",
"loadMs": 1068.5277080046944,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 279.09091600304237,
"inferenceMs": 278.7209580055787,
"tokensPerSecond": 7.175635496918639,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 6971277312,
"cpuThreads": 1,
"totalMs": 1347.6186240077368,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
"torchVersion": "2.10.0",
"loadMs": 1068.5277080046944,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 307.975791998615,
"inferenceMs": 307.6092920018709,
"tokensPerSecond": 6.501754179739914,
"inputTokens": 68,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 6973521920,
"cpuThreads": 1,
"totalMs": 1376.5035000033095,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "hola"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
"torchVersion": "2.10.0",
"loadMs": 1068.5277080046944,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 311.829124999349,
"inferenceMs": 311.4636669997708,
"tokensPerSecond": 6.421294718788088,
"inputTokens": 69,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 6976798720,
"cpuThreads": 1,
"totalMs": 1380.3568330040434,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "2026-11-04"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
"torchVersion": "2.10.0",
"loadMs": 1068.5277080046944,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 701.3185000032536,
"inferenceMs": 700.8254159954959,
"tokensPerSecond": 9.988222230862968,
"inputTokens": 72,
"outputTokens": 7,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 6982631424,
"cpuThreads": 1,
"totalMs": 1769.846208007948,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "Unknown"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
"torchVersion": "2.10.0",
"loadMs": 1068.5277080046944,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 322
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 331.74608400440775,
"inferenceMs": 331.3417079989449,
"tokensPerSecond": 6.036064738358772,
"inputTokens": 74,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 6987595776,
"cpuThreads": 1,
"totalMs": 1400.2737920091022,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"plainParity": true,
"passed": true,
"expected": "Paris"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
"torchVersion": "2.10.0",
"loadMs": 1601.937542000087,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 322
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 250.708334002411,
"inferenceMs": 239.65508399851387,
"tokensPerSecond": 8.34532681982412,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4348821504,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 510246912,
"cpuThreads": 4,
"totalMs": 1852.645876002498,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "56"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
"torchVersion": "2.10.0",
"loadMs": 1601.937542000087,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 322
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 102.42675000336021,
"inferenceMs": 101.24408300180221,
"tokensPerSecond": 19.754240847481416,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4348821504,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 510623744,
"cpuThreads": 4,
"totalMs": 1704.3642920034472,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
"torchVersion": "2.10.0",
"loadMs": 1601.937542000087,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 322
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 166.7393749958137,
"inferenceMs": 164.99674999795388,
"tokensPerSecond": 12.121450877213048,
"inputTokens": 68,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4349476864,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 520667136,
"cpuThreads": 4,
"totalMs": 1768.6769169959007,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "hola"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
"torchVersion": "2.10.0",
"loadMs": 1601.937542000087,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 322
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 148.7805419965298,
"inferenceMs": 147.075249995396,
"tokensPerSecond": 13.598481050092435,
"inputTokens": 69,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4350148608,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 530038784,
"cpuThreads": 4,
"totalMs": 1750.7180839966168,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "2026-11-04"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
"torchVersion": "2.10.0",
"loadMs": 1601.937542000087,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 322
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 233.8637499997276,
"inferenceMs": 232.1823750025942,
"tokensPerSecond": 30.148713914748214,
"inputTokens": 72,
"outputTokens": 7,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4350869504,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 539426816,
"cpuThreads": 4,
"totalMs": 1835.8012919998146,
"grade": "fast",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "Unknown"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
"torchVersion": "2.10.0",
"loadMs": 1601.937542000087,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 322
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 146.2107500046841,
"inferenceMs": 144.44766600354342,
"tokensPerSecond": 13.845845040867177,
"inputTokens": 74,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4351639552,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 548716544,
"cpuThreads": 4,
"totalMs": 1748.148292004771,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "Paris"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
"torchVersion": "2.10.0",
"loadMs": 1546.7671250007697,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 322
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 156.74033400136977,
"inferenceMs": 145.57950000016717,
"tokensPerSecond": 13.738198029239717,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4348837888,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 505151488,
"cpuThreads": 4,
"totalMs": 1703.5074590021395,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "56"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
"torchVersion": "2.10.0",
"loadMs": 1546.7671250007697,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 322
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 102.72541700396687,
"inferenceMs": 101.03929099568632,
"tokensPerSecond": 19.79427983204461,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4348837888,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 505610240,
"cpuThreads": 4,
"totalMs": 1649.4925420047366,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
"torchVersion": "2.10.0",
"loadMs": 1546.7671250007697,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 322
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 144.80779199948302,
"inferenceMs": 143.03274999838322,
"tokensPerSecond": 13.982811628963347,
"inputTokens": 68,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4349476864,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 515817472,
"cpuThreads": 4,
"totalMs": 1691.5749170002528,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "hola"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
"torchVersion": "2.10.0",
"loadMs": 1546.7671250007697,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 322
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 141.60166699730325,
"inferenceMs": 140.4084590030834,
"tokensPerSecond": 14.244156044445154,
"inputTokens": 69,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4350148608,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 525156352,
"cpuThreads": 4,
"totalMs": 1688.368791998073,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "2026-11-04"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
"torchVersion": "2.10.0",
"loadMs": 1546.7671250007697,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 322
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 229.66729199833935,
"inferenceMs": 227.94625000096858,
"tokensPerSecond": 30.708993896456978,
"inputTokens": 72,
"outputTokens": 7,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4350869504,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 534528000,
"cpuThreads": 4,
"totalMs": 1776.434416999109,
"grade": "fast",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "Unknown"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
"torchVersion": "2.10.0",
"loadMs": 1546.7671250007697,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 322
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 144.36045799811836,
"inferenceMs": 142.40566599619342,
"tokensPerSecond": 14.044385003988964,
"inputTokens": 74,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4351639552,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 543686656,
"cpuThreads": 4,
"totalMs": 1691.127582998888,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "Paris"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
"torchVersion": "2.10.0",
"loadMs": 1527.7355420039385,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 322
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 159.09820899832994,
"inferenceMs": 147.9200000030687,
"tokensPerSecond": 13.520822065701113,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4348837888,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 509001728,
"cpuThreads": 4,
"totalMs": 1686.8337510022684,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "56"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
"torchVersion": "2.10.0",
"loadMs": 1527.7355420039385,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 322
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 101.8580829986604,
"inferenceMs": 100.7070410050801,
"tokensPerSecond": 19.859584593485486,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4348837888,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 509509632,
"cpuThreads": 4,
"totalMs": 1629.5936250025989,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
"torchVersion": "2.10.0",
"loadMs": 1527.7355420039385,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 322
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 146.13899999676505,
"inferenceMs": 144.56670799700078,
"tokensPerSecond": 13.83444381981426,
"inputTokens": 68,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4349476864,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 519356416,
"cpuThreads": 4,
"totalMs": 1673.8745420007035,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "hola"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
"torchVersion": "2.10.0",
"loadMs": 1527.7355420039385,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 322
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 144.12462500331458,
"inferenceMs": 142.28900000307476,
"tokensPerSecond": 14.055900315251224,
"inputTokens": 69,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4350148608,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 528515072,
"cpuThreads": 4,
"totalMs": 1671.860167007253,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "2026-11-04"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
"torchVersion": "2.10.0",
"loadMs": 1527.7355420039385,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 322
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 230.75229200185277,
"inferenceMs": 228.98787499434547,
"tokensPerSecond": 30.5693041615101,
"inputTokens": 72,
"outputTokens": 7,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4350869504,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 537837568,
"cpuThreads": 4,
"totalMs": 1758.4878340057912,
"grade": "fast",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "Unknown"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
"torchVersion": "2.10.0",
"loadMs": 1527.7355420039385,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 322
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 151.71829099563183,
"inferenceMs": 147.4896660001832,
"tokensPerSecond": 13.560272080333517,
"inputTokens": 74,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4351639552,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 547045376,
"cpuThreads": 4,
"totalMs": 1679.4538329995703,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
}
]Granite 4.0 Dense 350M
28 configurations testedQuality criteria
No explicit quality gate recorded. Review per-case evidence; full certification is pending.
| Precision | Device | Result | Platform | Inference / load | |
|---|---|---|---|---|---|
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 48 ms / 1027 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 52 ms / 1154 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 52 ms / 1107 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 79 ms / 1027 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 79 ms / 1027 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 82 ms / 1107 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 85 ms / 1107 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 92 ms / 1154 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 93 ms / 1154 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 104 ms / 1027 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 119 ms / 1107 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 120 ms / 1154 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 82 ms / 790 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 84 ms / 790 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 89 ms / 809 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 90 ms / 790 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 91 ms / 855 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 91 ms / 855 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 95 ms / 809 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 98 ms / 855 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 99 ms / 790 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 100 ms / 855 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 100 ms / 819 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 104 ms / 819 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 104 ms / 809 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 109 ms / 809 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 114 ms / 819 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 117 ms / 819 ms |
Test setup & sample details
- Quality gate
- No explicit quality gate recorded. Review the per-case quality evidence; full certification is pending.
- Tested
- 2026-11-07T12:16:29.639275+00:00
- Environment
- { "os": "macos", "arch": "aarch64", "hardware": "Apple M3 Max", "osVersion": "27.2" }
- Runtime
- transformers 5.22.1
- Artifact revision
- bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2
- Fixture
{ "file": "model-samples/desktop/granite4/granite-4.0-350m-native-capital.json", "size": 244, "sha1": "9013b4f5a71cd1280d78b3dfa751ead1186d071f", "sha256": "8ff34789fa2a0f5bbef2080057f4b36376d3c73ea64baccefb6caec96eafbd68", "url": "https://assets.localai.tools/samples/v1/files/8ff34789fa2a0f5bbef2080057f4b36376d3c73ea64baccefb6caec96eafbd68/granite-4.0-350m-native-capital.json" }- Rerun when
- Artifact/runtime revision; Input/generation processing; Fixture/quality gates; OS/backend behavior
Per-case quality & failure details
[
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"plainParity": true,
"passed": true,
"expected": "Paris"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
"torchVersion": "2.10.0",
"loadMs": 1153.7135419930564,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 226
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 131.51083400589414,
"inferenceMs": 119.536458994844,
"tokensPerSecond": 16.731297018646558,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 1136001024,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 1037860864,
"cpuThreads": 4,
"totalMs": 1285.2243759989506,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "56"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
"torchVersion": "2.10.0",
"loadMs": 1153.7135419930564,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 226
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 53.727541999251116,
"inferenceMs": 51.9242089940235,
"tokensPerSecond": 38.51767872342939,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 1136001024,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 1038106624,
"cpuThreads": 4,
"totalMs": 1207.4410839923075,
"grade": "fast",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "summer"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
"torchVersion": "2.10.0",
"loadMs": 1153.7135419930564,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 226
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 93.75641599763185,
"inferenceMs": 91.96091699413955,
"tokensPerSecond": 21.748369474474195,
"inputTokens": 67,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 1136640000,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 1047920640,
"cpuThreads": 4,
"totalMs": 1247.4699579906883,
"grade": "fast",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "7"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
"torchVersion": "2.10.0",
"loadMs": 1153.7135419930564,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 226
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 94.54145900235744,
"inferenceMs": 92.6774170002318,
"tokensPerSecond": 21.580230273303773,
"inputTokens": 75,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 1137426432,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 1057226752,
"cpuThreads": 4,
"totalMs": 1248.2550009954139,
"grade": "fast",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "Paris"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
"torchVersion": "2.10.0",
"loadMs": 1106.7504169986933,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 226
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 130.84404199616984,
"inferenceMs": 118.98054199991748,
"tokensPerSecond": 16.809471249520676,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 1136001024,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 1201274880,
"cpuThreads": 4,
"totalMs": 1237.594458994863,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "56"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
"torchVersion": "2.10.0",
"loadMs": 1106.7504169986933,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 226
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 53.620083002897445,
"inferenceMs": 52.36283299745992,
"tokensPerSecond": 38.19503043498465,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 1136001024,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 1201700864,
"cpuThreads": 4,
"totalMs": 1160.3705000015907,
"grade": "fast",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "summer"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
"torchVersion": "2.10.0",
"loadMs": 1106.7504169986933,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 226
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 86.38733300176682,
"inferenceMs": 84.71229099814082,
"tokensPerSecond": 23.60932488585268,
"inputTokens": 67,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 1136640000,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 1211514880,
"cpuThreads": 4,
"totalMs": 1193.13775000046,
"grade": "fast",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "7"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
"torchVersion": "2.10.0",
"loadMs": 1106.7504169986933,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 226
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 83.82970799721079,
"inferenceMs": 82.03133400093066,
"tokensPerSecond": 24.380927414606102,
"inputTokens": 75,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 1137426432,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 1220919296,
"cpuThreads": 4,
"totalMs": 1190.580124995904,
"grade": "fast",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "Paris"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
"torchVersion": "2.10.0",
"loadMs": 1026.9828749951557,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 226
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 115.35945899959188,
"inferenceMs": 103.9500830011093,
"tokensPerSecond": 19.24000387742506,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 1136001024,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 1044250624,
"cpuThreads": 4,
"totalMs": 1142.3423339947476,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "56"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
"torchVersion": "2.10.0",
"loadMs": 1026.9828749951557,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 226
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 49.815040998510085,
"inferenceMs": 48.15483299898915,
"tokensPerSecond": 41.532695171884065,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 1136001024,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 1044676608,
"cpuThreads": 4,
"totalMs": 1076.7979159936658,
"grade": "fast",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "summer"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
"torchVersion": "2.10.0",
"loadMs": 1026.9828749951557,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 226
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 80.7666250038892,
"inferenceMs": 78.96066600369522,
"tokensPerSecond": 25.329067005417652,
"inputTokens": 67,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 1136640000,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 1054539776,
"cpuThreads": 4,
"totalMs": 1107.749499999045,
"grade": "fast",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "7"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
"torchVersion": "2.10.0",
"loadMs": 1026.9828749951557,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 226
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 81.00491600634996,
"inferenceMs": 79.22775000042748,
"tokensPerSecond": 25.243680402248064,
"inputTokens": 75,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 1137426432,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 1063911424,
"cpuThreads": 4,
"totalMs": 1107.9877910015057,
"grade": "fast",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"plainParity": true,
"passed": true,
"expected": "Paris"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
"torchVersion": "2.10.0",
"loadMs": 818.9316249990952,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 226
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 123.5065000000759,
"inferenceMs": 113.90570900402963,
"tokensPerSecond": 17.55838243304597,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 1834057728,
"cpuThreads": 4,
"totalMs": 942.4381249991711,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "56"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
"torchVersion": "2.10.0",
"loadMs": 818.9316249990952,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 226
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 104.3223750020843,
"inferenceMs": 103.98008400079561,
"tokensPerSecond": 19.234452628300403,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 1836793856,
"cpuThreads": 4,
"totalMs": 923.2540000011795,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "summer"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
"torchVersion": "2.10.0",
"loadMs": 818.9316249990952,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 226
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 117.16191600135062,
"inferenceMs": 116.7806669982383,
"tokensPerSecond": 17.126122425984867,
"inputTokens": 67,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 1838317568,
"cpuThreads": 4,
"totalMs": 936.0935410004458,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "7"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
"torchVersion": "2.10.0",
"loadMs": 818.9316249990952,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 226
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 100.62195800128393,
"inferenceMs": 100.16212499613175,
"tokensPerSecond": 19.967627484712807,
"inputTokens": 75,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 1839104000,
"cpuThreads": 4,
"totalMs": 919.5535830003791,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "Paris"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
"torchVersion": "2.10.0",
"loadMs": 855.4552500063437,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 226
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 102.5817499976256,
"inferenceMs": 91.46650000184309,
"tokensPerSecond": 21.86592905555257,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 1833025536,
"cpuThreads": 4,
"totalMs": 958.0370000039693,
"grade": "fast",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "56"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
"torchVersion": "2.10.0",
"loadMs": 855.4552500063437,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 226
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 91.43366699572653,
"inferenceMs": 91.08829199976753,
"tokensPerSecond": 21.956718652767186,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 1834205184,
"cpuThreads": 4,
"totalMs": 946.8889170020702,
"grade": "fast",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "summer"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
"torchVersion": "2.10.0",
"loadMs": 855.4552500063437,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 226
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 98.27200000290759,
"inferenceMs": 97.87729099480202,
"tokensPerSecond": 20.433749030775832,
"inputTokens": 67,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 1836040192,
"cpuThreads": 4,
"totalMs": 953.7272500092513,
"grade": "fast",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "7"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
"torchVersion": "2.10.0",
"loadMs": 855.4552500063437,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 226
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 100.29762499470962,
"inferenceMs": 99.89679099817295,
"tokensPerSecond": 20.020663126572092,
"inputTokens": 75,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 1838088192,
"cpuThreads": 4,
"totalMs": 955.7528750010533,
"grade": "fast",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "Paris"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
"torchVersion": "2.10.0",
"loadMs": 809.3695409988868,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 226
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 113.83824999938952,
"inferenceMs": 104.44516599818598,
"tokensPerSecond": 19.148803880830027,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 1836826624,
"cpuThreads": 4,
"totalMs": 923.2077909982763,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "56"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
"torchVersion": "2.10.0",
"loadMs": 809.3695409988868,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 226
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 95.27779200288933,
"inferenceMs": 94.86583399848314,
"tokensPerSecond": 21.082405706062513,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 1840807936,
"cpuThreads": 4,
"totalMs": 904.6473330017761,
"grade": "fast",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "summer"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
"torchVersion": "2.10.0",
"loadMs": 809.3695409988868,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 226
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 109.13429200445535,
"inferenceMs": 108.78875000344124,
"tokensPerSecond": 18.384253885964636,
"inputTokens": 67,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 1842069504,
"cpuThreads": 4,
"totalMs": 918.5038330033422,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "7"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
"torchVersion": "2.10.0",
"loadMs": 809.3695409988868,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 226
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 89.68825000192737,
"inferenceMs": 89.30945800238987,
"tokensPerSecond": 22.394044760035168,
"inputTokens": 75,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 1844150272,
"cpuThreads": 4,
"totalMs": 899.0577910008142,
"grade": "fast",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"plainParity": true,
"passed": true,
"expected": "Paris"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
"torchVersion": "2.10.0",
"loadMs": 790.2471249981318,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 226
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 91.50879200024065,
"inferenceMs": 81.92720799706876,
"tokensPerSecond": 24.411914538471237,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 1827979264,
"cpuThreads": 1,
"totalMs": 881.7559169983724,
"grade": "fast",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "56"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
"torchVersion": "2.10.0",
"loadMs": 790.2471249981318,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 226
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 83.9100840021274,
"inferenceMs": 83.55279199895449,
"tokensPerSecond": 23.936961915348398,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 1828405248,
"cpuThreads": 1,
"totalMs": 874.1572090002592,
"grade": "fast",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "summer"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
"torchVersion": "2.10.0",
"loadMs": 790.2471249981318,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 226
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 90.5117500005872,
"inferenceMs": 90.14249999745516,
"tokensPerSecond": 22.187092659472086,
"inputTokens": 67,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 1832271872,
"cpuThreads": 1,
"totalMs": 880.758874998719,
"grade": "fast",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "7"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
"torchVersion": "2.10.0",
"loadMs": 790.2471249981318,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 226
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 99.82495800068136,
"inferenceMs": 99.42108300310792,
"tokensPerSecond": 20.11645759217368,
"inputTokens": 75,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 1836056576,
"cpuThreads": 1,
"totalMs": 890.0720829988131,
"grade": "fast",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
}
]Granite 4.0 Hybrid 1B
30 configurations testedQuality criteria
No explicit quality gate recorded. Review per-case evidence; full certification is pending.
| Precision | Device | Result | Platform | Inference / load | |
|---|---|---|---|---|---|
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 373 ms / 1419 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 384 ms / 1616 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 384 ms / 1446 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 175 ms / 1446 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 263 ms / 1446 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 265 ms / 1616 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 268 ms / 1419 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 180 ms / 1419 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 180 ms / 1616 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 233 ms / 1446 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 234 ms / 1446 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 235 ms / 1419 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 241 ms / 1419 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 241 ms / 1446 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 242 ms / 1419 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 243 ms / 1616 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 274 ms / 1616 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 361 ms / 1616 ms |
FP32 | cpu | Failed | — | macos · aarch64 | 5659 ms / 4055 ms |
FP32 | cpu | Failed | — | macos · aarch64 | 5433 ms / 4055 ms |
FP32 | cpu | Failed | — | macos · aarch64 | 5294 ms / 4055 ms |
FP32 | cpu | Failed | — | macos · aarch64 | 7330 ms / 4055 ms |
FP32 | cpu | Failed | — | macos · aarch64 | 14982 ms / 4055 ms |
FP32 | cpu | Failed | — | macos · aarch64 | 4431 ms / 4055 ms |
FP32 | cpu | Failed | — | macos · aarch64 | 1606 ms / 3678 ms |
FP32 | cpu | Failed | — | macos · aarch64 | 1606 ms / 3678 ms |
FP32 | cpu | Failed | — | macos · aarch64 | 1643 ms / 3678 ms |
FP32 | cpu | Failed | — | macos · aarch64 | 1999 ms / 3678 ms |
FP32 | cpu | Failed | — | macos · aarch64 | 3420 ms / 3678 ms |
FP32 | cpu | Failed | — | macos · aarch64 | 1651 ms / 3678 ms |
Test setup & sample details
- Quality gate
- No explicit quality gate recorded. Review the per-case quality evidence; full certification is pending.
- Tested
- 2026-11-07T12:20:59.105634+00:00
- Environment
- { "os": "macos", "arch": "aarch64", "hardware": "Apple M3 Max", "osVersion": "27.2" }
- Runtime
- transformers 5.22.1
- Artifact revision
- d18cca4c121edb87d022116d281ce212c9136f57
- Fixture
{ "file": "model-samples/desktop/granite4/granite-4.0-h-1b-native-capital.json", "size": 244, "sha1": "9013b4f5a71cd1280d78b3dfa751ead1186d071f", "sha256": "8ff34789fa2a0f5bbef2080057f4b36376d3c73ea64baccefb6caec96eafbd68", "url": "https://assets.localai.tools/samples/v1/files/8ff34789fa2a0f5bbef2080057f4b36376d3c73ea64baccefb6caec96eafbd68/granite-4.0-h-1b-native-capital.json" }- Rerun when
- Artifact/runtime revision; Input/generation processing; Fixture/quality gates; OS/backend behavior
Per-case quality & failure details
[
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"plainParity": true,
"passed": true,
"expected": "Paris"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
"torchVersion": "2.10.0",
"loadMs": 1615.5484580012853,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 466
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 372.524541002349,
"inferenceMs": 361.1726669987547,
"tokensPerSecond": 5.537517599599794,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 3258384384,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 1599078400,
"cpuThreads": 4,
"totalMs": 1988.0729990036343,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "56"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
"torchVersion": "2.10.0",
"loadMs": 1615.5484580012853,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 466
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 181.17112499749055,
"inferenceMs": 179.51862500194693,
"tokensPerSecond": 11.140905295917399,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 3258384384,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 1600225280,
"cpuThreads": 4,
"totalMs": 1796.7195829987759,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
"torchVersion": "2.10.0",
"loadMs": 1615.5484580012853,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 466
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 275.88537499832455,
"inferenceMs": 274.11974999995437,
"tokensPerSecond": 7.296081365900607,
"inputTokens": 68,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 3267346432,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 1615806464,
"cpuThreads": 4,
"totalMs": 1891.4338329996099,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "hola"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
"torchVersion": "2.10.0",
"loadMs": 1615.5484580012853,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 466
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 266.7036250059027,
"inferenceMs": 264.7699160006596,
"tokensPerSecond": 11.330592407607693,
"inputTokens": 69,
"outputTokens": 3,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 3267936256,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 1630928896,
"cpuThreads": 4,
"totalMs": 1882.252083007188,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "2026-11-04"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
"torchVersion": "2.10.0",
"loadMs": 1615.5484580012853,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 466
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 385.33874999848194,
"inferenceMs": 383.50454199826345,
"tokensPerSecond": 18.252717330350933,
"inputTokens": 72,
"outputTokens": 7,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 3268558848,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 1647083520,
"cpuThreads": 4,
"totalMs": 2000.8872079997673,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "Unknown"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
"torchVersion": "2.10.0",
"loadMs": 1615.5484580012853,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 466
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 244.55133300216403,
"inferenceMs": 242.83783300052164,
"tokensPerSecond": 8.235948967621136,
"inputTokens": 74,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 3277602816,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 1661960192,
"cpuThreads": 4,
"totalMs": 1860.0997910034494,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "Paris"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
"torchVersion": "2.10.0",
"loadMs": 1446.4713749985094,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 466
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 252.28354099817807,
"inferenceMs": 241.0489169997163,
"tokensPerSecond": 8.297071087867007,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 3258400768,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 1592852480,
"cpuThreads": 4,
"totalMs": 1698.7549159966875,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "56"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
"torchVersion": "2.10.0",
"loadMs": 1446.4713749985094,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 466
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 176.63241700211074,
"inferenceMs": 174.99975000100676,
"tokensPerSecond": 11.428587755059617,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 3258400768,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 1594228736,
"cpuThreads": 4,
"totalMs": 1623.1037920006202,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
"torchVersion": "2.10.0",
"loadMs": 1446.4713749985094,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 466
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 235.52391699922737,
"inferenceMs": 233.8634589978028,
"tokensPerSecond": 8.551998711430974,
"inputTokens": 68,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 3267346432,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 1609695232,
"cpuThreads": 4,
"totalMs": 1681.9952919977368,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "hola"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
"torchVersion": "2.10.0",
"loadMs": 1446.4713749985094,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 466
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 265.2217080030823,
"inferenceMs": 263.47412500035716,
"tokensPerSecond": 11.386317347086486,
"inputTokens": 69,
"outputTokens": 3,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 3267936256,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 1626259456,
"cpuThreads": 4,
"totalMs": 1711.6930830015917,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "2026-11-04"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
"torchVersion": "2.10.0",
"loadMs": 1446.4713749985094,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 466
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 385.5130830052076,
"inferenceMs": 383.80429199605715,
"tokensPerSecond": 18.238462013009254,
"inputTokens": 72,
"outputTokens": 7,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 3268558848,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 1642430464,
"cpuThreads": 4,
"totalMs": 1831.984458003717,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "Unknown"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
"torchVersion": "2.10.0",
"loadMs": 1446.4713749985094,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 466
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 234.76033299812116,
"inferenceMs": 232.5196249948931,
"tokensPerSecond": 8.601424503604488,
"inputTokens": 74,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 3277602816,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 1657372672,
"cpuThreads": 4,
"totalMs": 1681.2317079966306,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "Paris"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
"torchVersion": "2.10.0",
"loadMs": 1418.5662919990136,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 466
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 245.96404100157088,
"inferenceMs": 235.08370800118428,
"tokensPerSecond": 8.5076078517101,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 3258400768,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 1596850176,
"cpuThreads": 4,
"totalMs": 1664.5303330005845,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "56"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
"torchVersion": "2.10.0",
"loadMs": 1418.5662919990136,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 466
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 181.15433299681172,
"inferenceMs": 179.51779199938755,
"tokensPerSecond": 11.140956992200657,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 3258400768,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 1598242816,
"cpuThreads": 4,
"totalMs": 1599.7206249958253,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
"torchVersion": "2.10.0",
"loadMs": 1418.5662919990136,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 466
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 242.39862499962328,
"inferenceMs": 240.71654200088233,
"tokensPerSecond": 8.308527462947142,
"inputTokens": 68,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 3267346432,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 1614168064,
"cpuThreads": 4,
"totalMs": 1660.964916998637,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "hola"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
"torchVersion": "2.10.0",
"loadMs": 1418.5662919990136,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 466
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 269.2904169962276,
"inferenceMs": 267.83616700413404,
"tokensPerSecond": 11.200877139022435,
"inputTokens": 69,
"outputTokens": 3,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 3267936256,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 1630846976,
"cpuThreads": 4,
"totalMs": 1687.8567089952412,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "2026-11-04"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
"torchVersion": "2.10.0",
"loadMs": 1418.5662919990136,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 466
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 374.7637910055346,
"inferenceMs": 373.08862499776296,
"tokensPerSecond": 18.762298100195288,
"inputTokens": 72,
"outputTokens": 7,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 3268558848,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 1647329280,
"cpuThreads": 4,
"totalMs": 1793.3300830045482,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "Unknown"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
"torchVersion": "2.10.0",
"loadMs": 1418.5662919990136,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 466
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 243.1170830022893,
"inferenceMs": 241.72537500271574,
"tokensPerSecond": 8.273852093424326,
"inputTokens": 74,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 3277602816,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 1662746624,
"cpuThreads": 4,
"totalMs": 1661.683375001303,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "fail",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"plainParity": true,
"passed": false,
"expected": "Paris"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
"torchVersion": "2.10.0",
"loadMs": 4054.5797909944667,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 466
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 5669.2278330010595,
"inferenceMs": 5659.269166993909,
"tokensPerSecond": 0.35340252265512234,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 6458064896,
"cpuThreads": 4,
"totalMs": 9723.807623995526,
"grade": "slow",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "fail",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": false,
"expected": "56"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
"torchVersion": "2.10.0",
"loadMs": 4054.5797909944667,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 466
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 5433.794791999389,
"inferenceMs": 5433.386249998875,
"tokensPerSecond": 0.3680945745391125,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 6462586880,
"cpuThreads": 4,
"totalMs": 9488.374582993856,
"grade": "slow",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "fail",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": false,
"expected": "chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
"torchVersion": "2.10.0",
"loadMs": 4054.5797909944667,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 466
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 5294.674124997982,
"inferenceMs": 5294.235000001208,
"tokensPerSecond": 0.3777694038892387,
"inputTokens": 68,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 6462980096,
"cpuThreads": 4,
"totalMs": 9349.253915992449,
"grade": "slow",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "fail",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": false,
"expected": "hola"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
"torchVersion": "2.10.0",
"loadMs": 4054.5797909944667,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 466
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 7330.640082996979,
"inferenceMs": 7330.113250005525,
"tokensPerSecond": 0.4092706207503327,
"inputTokens": 69,
"outputTokens": 3,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 6468009984,
"cpuThreads": 4,
"totalMs": 11385.219873991446,
"grade": "slow",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "fail",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": false,
"expected": "2026-11-04"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
"torchVersion": "2.10.0",
"loadMs": 4054.5797909944667,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 466
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 14983.009625000705,
"inferenceMs": 14982.482708001044,
"tokensPerSecond": 0.46721228626960565,
"inputTokens": 72,
"outputTokens": 7,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 6468059136,
"cpuThreads": 4,
"totalMs": 19037.58941599517,
"grade": "slow",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "fail",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": false,
"expected": "Unknown"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
"torchVersion": "2.10.0",
"loadMs": 4054.5797909944667,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 466
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 4431.094125000527,
"inferenceMs": 4430.652041999565,
"tokensPerSecond": 0.4514008279236017,
"inputTokens": 74,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 6468108288,
"cpuThreads": 4,
"totalMs": 8485.673915994994,
"grade": "slow",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "fail",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"plainParity": true,
"passed": false,
"expected": "Paris"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
"torchVersion": "2.10.0",
"loadMs": 3678.4359160010354,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 466
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 1616.2001249977038,
"inferenceMs": 1606.1470419954276,
"tokensPerSecond": 1.2452160030847872,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 6462767104,
"cpuThreads": 1,
"totalMs": 5294.636040998739,
"grade": "slow",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "fail",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": false,
"expected": "56"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
"torchVersion": "2.10.0",
"loadMs": 3678.4359160010354,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 466
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 1606.2157499982277,
"inferenceMs": 1605.7623329979833,
"tokensPerSecond": 1.245514332289741,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 6462783488,
"cpuThreads": 1,
"totalMs": 5284.651665999263,
"grade": "slow",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "fail",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": false,
"expected": "chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
"torchVersion": "2.10.0",
"loadMs": 3678.4359160010354,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 466
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 1643.326707999222,
"inferenceMs": 1642.9632090002997,
"tokensPerSecond": 1.2173127122042786,
"inputTokens": 68,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 6463127552,
"cpuThreads": 1,
"totalMs": 5321.762624000257,
"grade": "slow",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "fail",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": false,
"expected": "hola"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
"torchVersion": "2.10.0",
"loadMs": 3678.4359160010354,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 466
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 1999.6779580033035,
"inferenceMs": 1999.2964999983087,
"tokensPerSecond": 1.5005278106586681,
"inputTokens": 69,
"outputTokens": 3,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 6476218368,
"cpuThreads": 1,
"totalMs": 5678.113874004339,
"grade": "slow",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "fail",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": false,
"expected": "2026-11-04",
"fixturePassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
"torchVersion": "2.10.0",
"loadMs": 3678.4359160010354,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 466
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 3420.493332996557,
"inferenceMs": 3420.0012920045992,
"tokensPerSecond": 2.046782852499164,
"inputTokens": 72,
"outputTokens": 7,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 6476218368,
"cpuThreads": 1,
"totalMs": 7098.929248997592,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
},
"failure": "This fixture passed but the backend did not pass the complete repeated profile."
},
{
"precision": "fp32",
"device": "cpu",
"status": "fail",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": false,
"expected": "Unknown"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
"torchVersion": "2.10.0",
"loadMs": 3678.4359160010354,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 466
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 1651.6368750017136,
"inferenceMs": 1651.1553329983144,
"tokensPerSecond": 1.2112730765119613,
"inputTokens": 74,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 6476251136,
"cpuThreads": 1,
"totalMs": 5330.072791002749,
"grade": "slow",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
}
]Granite 4.0 Hybrid 350M
20 configurations testedQuality criteria
No explicit quality gate recorded. Review per-case evidence; full certification is pending.
| Precision | Device | Result | Platform | Inference / load | |
|---|---|---|---|---|---|
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 284 ms / 1212 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 301 ms / 1159 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 103 ms / 3600 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 105 ms / 1159 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 109 ms / 1212 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 399 ms / 3600 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 185 ms / 1212 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 210 ms / 1159 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 230 ms / 1212 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 284 ms / 1159 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 382 ms / 3600 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 1104 ms / 3600 ms |
FP32 | cpu | Failed | — | macos · aarch64 | 3470 ms / 2002 ms |
FP32 | cpu | Failed | — | macos · aarch64 | 8195 ms / 2002 ms |
FP32 | cpu | Failed | — | macos · aarch64 | 2988 ms / 2002 ms |
FP32 | cpu | Failed | — | macos · aarch64 | 4133 ms / 2002 ms |
FP32 | cpu | Failed | — | macos · aarch64 | 1157 ms / 1625 ms |
FP32 | cpu | Failed | — | macos · aarch64 | 1761 ms / 1625 ms |
FP32 | cpu | Failed | — | macos · aarch64 | 1007 ms / 1625 ms |
FP32 | cpu | Failed | — | macos · aarch64 | 1139 ms / 1625 ms |
Test setup & sample details
- Quality gate
- No explicit quality gate recorded. Review the per-case quality evidence; full certification is pending.
- Tested
- 2026-11-07T12:16:16.883639+00:00
- Environment
- { "os": "macos", "arch": "aarch64", "hardware": "Apple M3 Max", "osVersion": "27.2" }
- Runtime
- transformers 5.22.1
- Artifact revision
- 3b17b717b8f2f5d305b0a92c1491e239aeda19c8
- Fixture
{ "file": "model-samples/desktop/granite4/granite-4.0-h-350m-native-document-place.json", "size": 304, "sha1": "7dcdd80b3065bd555172453dbf8def706517421c", "sha256": "09d6cface3de3505ba7aceeabaaff63d9207d7e0596302661e2e987262d30589", "url": "https://assets.localai.tools/samples/v1/files/09d6cface3de3505ba7aceeabaaff63d9207d7e0596302661e2e987262d30589/granite-4.0-h-350m-native-document-place.json" }- Rerun when
- Artifact/runtime revision; Input/generation processing; Fixture/quality gates; OS/backend behavior
Per-case quality & failure details
[
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"plainParity": true,
"passed": true,
"expected": "Cedar"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "3b17b717b8f2f5d305b0a92c1491e239aeda19c8",
"torchVersion": "2.10.0",
"loadMs": 1158.8636669985135,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 370
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 295.71645799296675,
"inferenceMs": 284.3197499969392,
"tokensPerSecond": 10.551500555386307,
"inputTokens": 76,
"outputTokens": 3,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 1127825408,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 828424192,
"cpuThreads": 4,
"totalMs": 1454.5801249914803,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "2027-03-14"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "3b17b717b8f2f5d305b0a92c1491e239aeda19c8",
"torchVersion": "2.10.0",
"loadMs": 1158.8636669985135,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 370
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 303.1012919964269,
"inferenceMs": 301.3171670027077,
"tokensPerSecond": 23.231334841061663,
"inputTokens": 79,
"outputTokens": 7,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 1128513536,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 845545472,
"cpuThreads": 4,
"totalMs": 1461.9649589949404,
"grade": "fast",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "Unknown"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "3b17b717b8f2f5d305b0a92c1491e239aeda19c8",
"torchVersion": "2.10.0",
"loadMs": 1158.8636669985135,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 370
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 107.04383299889741,
"inferenceMs": 105.24533400166547,
"tokensPerSecond": 19.003217757552566,
"inputTokens": 76,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 1128513536,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 846020608,
"cpuThreads": 4,
"totalMs": 1265.907499997411,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "Dana"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "3b17b717b8f2f5d305b0a92c1491e239aeda19c8",
"torchVersion": "2.10.0",
"loadMs": 1158.8636669985135,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 370
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 211.8947089984431,
"inferenceMs": 209.56008299981477,
"tokensPerSecond": 14.315703434812304,
"inputTokens": 71,
"outputTokens": 3,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 1129103360,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 861863936,
"cpuThreads": 4,
"totalMs": 1370.7583759969566,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "Cedar"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "3b17b717b8f2f5d305b0a92c1491e239aeda19c8",
"torchVersion": "2.10.0",
"loadMs": 3599.5609579986194,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 370
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 1115.9589170056279,
"inferenceMs": 1104.3106249999255,
"tokensPerSecond": 2.7166269454305056,
"inputTokens": 76,
"outputTokens": 3,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 1127809024,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 821100544,
"cpuThreads": 4,
"totalMs": 4715.519875004247,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "2027-03-14"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "3b17b717b8f2f5d305b0a92c1491e239aeda19c8",
"torchVersion": "2.10.0",
"loadMs": 3599.5609579986194,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 370
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 401.51600000535836,
"inferenceMs": 399.0823330022977,
"tokensPerSecond": 17.54024024902074,
"inputTokens": 79,
"outputTokens": 7,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 1128513536,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 837550080,
"cpuThreads": 4,
"totalMs": 4001.076958003978,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "Unknown"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "3b17b717b8f2f5d305b0a92c1491e239aeda19c8",
"torchVersion": "2.10.0",
"loadMs": 3599.5609579986194,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 370
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 105.00237499945797,
"inferenceMs": 103.29120800452074,
"tokensPerSecond": 19.3627322076867,
"inputTokens": 76,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 1128513536,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 837992448,
"cpuThreads": 4,
"totalMs": 3704.5633329980774,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "Dana"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "3b17b717b8f2f5d305b0a92c1491e239aeda19c8",
"torchVersion": "2.10.0",
"loadMs": 3599.5609579986194,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 370
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 383.9327499954379,
"inferenceMs": 382.22833400504896,
"tokensPerSecond": 7.848711707385806,
"inputTokens": 71,
"outputTokens": 3,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 1129103360,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 853852160,
"cpuThreads": 4,
"totalMs": 3983.4937079940573,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "Cedar"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "3b17b717b8f2f5d305b0a92c1491e239aeda19c8",
"torchVersion": "2.10.0",
"loadMs": 1211.986207999871,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 370
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 242.2459999943385,
"inferenceMs": 230.48787500010803,
"tokensPerSecond": 13.015869056012573,
"inputTokens": 76,
"outputTokens": 3,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 1127825408,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 824459264,
"cpuThreads": 4,
"totalMs": 1454.2322079942096,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "2027-03-14"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "3b17b717b8f2f5d305b0a92c1491e239aeda19c8",
"torchVersion": "2.10.0",
"loadMs": 1211.986207999871,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 370
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 285.60583300713915,
"inferenceMs": 283.85195799637586,
"tokensPerSecond": 24.66074234404039,
"inputTokens": 79,
"outputTokens": 7,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 1128529920,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 841728000,
"cpuThreads": 4,
"totalMs": 1497.5920410070103,
"grade": "fast",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "Unknown"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "3b17b717b8f2f5d305b0a92c1491e239aeda19c8",
"torchVersion": "2.10.0",
"loadMs": 1211.986207999871,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 370
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 110.52754100091988,
"inferenceMs": 108.57416599901626,
"tokensPerSecond": 18.42058819054729,
"inputTokens": 76,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 1128513536,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 842317824,
"cpuThreads": 4,
"totalMs": 1322.513749000791,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "Dana"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "3b17b717b8f2f5d305b0a92c1491e239aeda19c8",
"torchVersion": "2.10.0",
"loadMs": 1211.986207999871,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 370
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 186.4249159989413,
"inferenceMs": 184.65708299481776,
"tokensPerSecond": 16.246330502709135,
"inputTokens": 71,
"outputTokens": 3,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 1129103360,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 858308608,
"cpuThreads": 4,
"totalMs": 1398.4111239988124,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "fail",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"plainParity": true,
"passed": false,
"expected": "Cedar"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "3b17b717b8f2f5d305b0a92c1491e239aeda19c8",
"torchVersion": "2.10.0",
"loadMs": 2002.2069170008763,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 370
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 3479.801708002924,
"inferenceMs": 3470.2159160005976,
"tokensPerSecond": 0.8644995218215359,
"inputTokens": 76,
"outputTokens": 3,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 1897349120,
"cpuThreads": 4,
"totalMs": 5482.0086250038,
"grade": "slow",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "fail",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": false,
"expected": "2027-03-14"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "3b17b717b8f2f5d305b0a92c1491e239aeda19c8",
"torchVersion": "2.10.0",
"loadMs": 2002.2069170008763,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 370
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 8195.009707997087,
"inferenceMs": 8194.530834000034,
"tokensPerSecond": 0.8542282824730135,
"inputTokens": 79,
"outputTokens": 7,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 1902346240,
"cpuThreads": 4,
"totalMs": 10197.216624997964,
"grade": "slow",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "fail",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": false,
"expected": "Unknown"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "3b17b717b8f2f5d305b0a92c1491e239aeda19c8",
"torchVersion": "2.10.0",
"loadMs": 2002.2069170008763,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 370
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 2988.5040419976576,
"inferenceMs": 2988.031792003312,
"tokensPerSecond": 0.6693369211641182,
"inputTokens": 76,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 1902362624,
"cpuThreads": 4,
"totalMs": 4990.710958998534,
"grade": "slow",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "fail",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": false,
"expected": "Dana"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "3b17b717b8f2f5d305b0a92c1491e239aeda19c8",
"torchVersion": "2.10.0",
"loadMs": 2002.2069170008763,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 370
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 4134.1128749991185,
"inferenceMs": 4132.940999996208,
"tokensPerSecond": 0.7258753512336016,
"inputTokens": 71,
"outputTokens": 3,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 1902395392,
"cpuThreads": 4,
"totalMs": 6136.319791999995,
"grade": "slow",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "fail",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"plainParity": true,
"passed": false,
"expected": "Cedar",
"fixturePassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "3b17b717b8f2f5d305b0a92c1491e239aeda19c8",
"torchVersion": "2.10.0",
"loadMs": 1625.3247919958085,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 370
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 1167.05683300097,
"inferenceMs": 1157.2270000033313,
"tokensPerSecond": 2.5924040832017954,
"inputTokens": 76,
"outputTokens": 3,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 1897660416,
"cpuThreads": 1,
"totalMs": 2792.3816249967786,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
},
"failure": "This fixture passed but the backend did not pass the complete repeated profile."
},
{
"precision": "fp32",
"device": "cpu",
"status": "fail",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": false,
"expected": "2027-03-14",
"fixturePassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "3b17b717b8f2f5d305b0a92c1491e239aeda19c8",
"torchVersion": "2.10.0",
"loadMs": 1625.3247919958085,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 370
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 1761.850166003569,
"inferenceMs": 1761.3827090026462,
"tokensPerSecond": 3.974150514946087,
"inputTokens": 79,
"outputTokens": 7,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 1902592000,
"cpuThreads": 1,
"totalMs": 3387.1749579993775,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
},
"failure": "This fixture passed but the backend did not pass the complete repeated profile."
},
{
"precision": "fp32",
"device": "cpu",
"status": "fail",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": false,
"expected": "Unknown"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "3b17b717b8f2f5d305b0a92c1491e239aeda19c8",
"torchVersion": "2.10.0",
"loadMs": 1625.3247919958085,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 370
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 1007.1682919951854,
"inferenceMs": 1006.7262499942444,
"tokensPerSecond": 1.9866373803319763,
"inputTokens": 76,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 1902592000,
"cpuThreads": 1,
"totalMs": 2632.493083990994,
"grade": "slow",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "fail",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": false,
"expected": "Dana",
"fixturePassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "3b17b717b8f2f5d305b0a92c1491e239aeda19c8",
"torchVersion": "2.10.0",
"loadMs": 1625.3247919958085,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 370
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 1139.8297499981709,
"inferenceMs": 1139.3076250024023,
"tokensPerSecond": 2.633178198902754,
"inputTokens": 71,
"outputTokens": 3,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 1902592000,
"cpuThreads": 1,
"totalMs": 2765.1545419939794,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
},
"failure": "This fixture passed but the backend did not pass the complete repeated profile."
}
]Granite 4.0 Hybrid Micro
30 configurations testedQuality criteria
No explicit quality gate recorded. Review per-case evidence; full certification is pending.
| Precision | Device | Result | Platform | Inference / load | |
|---|---|---|---|---|---|
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 489 ms / 4238 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 492 ms / 1925 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 501 ms / 1962 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 344 ms / 1925 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 346 ms / 1962 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 234 ms / 1925 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 237 ms / 1962 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 357 ms / 4238 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 238 ms / 4238 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 293 ms / 1925 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 305 ms / 1925 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 307 ms / 1962 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 307 ms / 1925 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 311 ms / 4238 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 311 ms / 1962 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 312 ms / 1962 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 362 ms / 4238 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 369 ms / 4238 ms |
FP32 | cpu | Failed | — | macos · aarch64 | 6014 ms / 4618 ms |
FP32 | cpu | Failed | — | macos · aarch64 | 5414 ms / 4618 ms |
FP32 | cpu | Failed | — | macos · aarch64 | 5755 ms / 4618 ms |
FP32 | cpu | Failed | — | macos · aarch64 | 8517 ms / 4618 ms |
FP32 | cpu | Failed | — | macos · aarch64 | 17597 ms / 4618 ms |
FP32 | cpu | Failed | — | macos · aarch64 | 5506 ms / 4618 ms |
FP32 | cpu | Failed | — | macos · aarch64 | 2417 ms / 6964 ms |
FP32 | cpu | Failed | — | macos · aarch64 | 2600 ms / 6964 ms |
FP32 | cpu | Failed | — | macos · aarch64 | 2958 ms / 6964 ms |
FP32 | cpu | Failed | — | macos · aarch64 | 3573 ms / 6964 ms |
FP32 | cpu | Failed | — | macos · aarch64 | 5862 ms / 6964 ms |
FP32 | cpu | Failed | — | macos · aarch64 | 2611 ms / 6964 ms |
Test setup & sample details
- Quality gate
- No explicit quality gate recorded. Review the per-case quality evidence; full certification is pending.
- Tested
- 2026-11-07T12:22:53.450203+00:00
- Environment
- { "os": "macos", "arch": "aarch64", "hardware": "Apple M3 Max", "osVersion": "27.2" }
- Runtime
- transformers 5.22.1
- Artifact revision
- d5f01a3ea75f088947be3aae039f4ad52837dfde
- Fixture
{ "file": "model-samples/desktop/granite4/granite-4.0-h-micro-native-capital.json", "size": 244, "sha1": "9013b4f5a71cd1280d78b3dfa751ead1186d071f", "sha256": "8ff34789fa2a0f5bbef2080057f4b36376d3c73ea64baccefb6caec96eafbd68", "url": "https://assets.localai.tools/samples/v1/files/8ff34789fa2a0f5bbef2080057f4b36376d3c73ea64baccefb6caec96eafbd68/granite-4.0-h-micro-native-capital.json" }- Rerun when
- Artifact/runtime revision; Input/generation processing; Fixture/quality gates; OS/backend behavior
Per-case quality & failure details
[
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"plainParity": true,
"passed": true,
"expected": "Paris"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
"torchVersion": "2.10.0",
"loadMs": 4238.254416995915,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 466
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 380.27199999487493,
"inferenceMs": 369.1712500003632,
"tokensPerSecond": 5.417539962816802,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7554039808,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 2933342208,
"cpuThreads": 4,
"totalMs": 4618.52641699079,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "56"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
"torchVersion": "2.10.0",
"loadMs": 4238.254416995915,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 466
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 239.99333300162107,
"inferenceMs": 238.32175000279676,
"tokensPerSecond": 8.392016255237005,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7554039808,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 2934128640,
"cpuThreads": 4,
"totalMs": 4478.247749997536,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
"torchVersion": "2.10.0",
"loadMs": 4238.254416995915,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 466
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 363.3048750052694,
"inferenceMs": 361.5772499979357,
"tokensPerSecond": 5.5313214534692605,
"inputTokens": 68,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7563755520,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 2951036928,
"cpuThreads": 4,
"totalMs": 4601.559292001184,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "hola"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
"torchVersion": "2.10.0",
"loadMs": 4238.254416995915,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 466
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 358.55141700449167,
"inferenceMs": 357.23649999999907,
"tokensPerSecond": 8.3977980973389,
"inputTokens": 69,
"outputTokens": 3,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7565115392,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 2966192128,
"cpuThreads": 4,
"totalMs": 4596.805834000406,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "2026-11-04"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
"torchVersion": "2.10.0",
"loadMs": 4238.254416995915,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 466
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 491.55462499766145,
"inferenceMs": 489.41774999548215,
"tokensPerSecond": 14.302709699565693,
"inputTokens": 72,
"outputTokens": 7,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7566589952,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 2981249024,
"cpuThreads": 4,
"totalMs": 4729.809041993576,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "Unknown"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
"torchVersion": "2.10.0",
"loadMs": 4238.254416995915,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 466
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 312.47787499887636,
"inferenceMs": 310.7872909968137,
"tokensPerSecond": 6.435269581279322,
"inputTokens": 74,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7568146432,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 2996502528,
"cpuThreads": 4,
"totalMs": 4550.732291994791,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "Paris"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
"torchVersion": "2.10.0",
"loadMs": 1925.2609999966808,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 466
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 303.6869579955237,
"inferenceMs": 292.9782499995781,
"tokensPerSecond": 6.8264453078099825,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7554039808,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 2935439360,
"cpuThreads": 4,
"totalMs": 2228.9479579922045,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "56"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
"torchVersion": "2.10.0",
"loadMs": 1925.2609999966808,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 466
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 235.4229580014362,
"inferenceMs": 233.82262499944773,
"tokensPerSecond": 8.553492203779356,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7554039808,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 2936766464,
"cpuThreads": 4,
"totalMs": 2160.683957998117,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
"torchVersion": "2.10.0",
"loadMs": 1925.2609999966808,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 466
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 306.73191699315794,
"inferenceMs": 305.0386670001899,
"tokensPerSecond": 6.556545829643148,
"inputTokens": 68,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7563755520,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 2954297344,
"cpuThreads": 4,
"totalMs": 2231.9929169898387,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "hola"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
"torchVersion": "2.10.0",
"loadMs": 1925.2609999966808,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 466
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 345.6560839986196,
"inferenceMs": 343.80783300002804,
"tokensPerSecond": 8.725804685199698,
"inputTokens": 69,
"outputTokens": 3,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7565115392,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 2970222592,
"cpuThreads": 4,
"totalMs": 2270.9170839953003,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "2026-11-04"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
"torchVersion": "2.10.0",
"loadMs": 1925.2609999966808,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 466
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 493.6798339986126,
"inferenceMs": 491.99379199853865,
"tokensPerSecond": 14.227821801501088,
"inputTokens": 72,
"outputTokens": 7,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7566589952,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 2985934848,
"cpuThreads": 4,
"totalMs": 2418.9408339952934,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "Unknown"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
"torchVersion": "2.10.0",
"loadMs": 1925.2609999966808,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 466
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 308.8843329969677,
"inferenceMs": 307.2142500022892,
"tokensPerSecond": 6.510114683759289,
"inputTokens": 74,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7568146432,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 3000975360,
"cpuThreads": 4,
"totalMs": 2234.1453329936485,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "Paris"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
"torchVersion": "2.10.0",
"loadMs": 1961.5091249943362,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 466
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 318.30379200255265,
"inferenceMs": 306.7352090001805,
"tokensPerSecond": 6.5202817978382885,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7554039808,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 2934865920,
"cpuThreads": 4,
"totalMs": 2279.812916996889,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "56"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
"torchVersion": "2.10.0",
"loadMs": 1961.5091249943362,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 466
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 238.54745899734553,
"inferenceMs": 237.10237499471987,
"tokensPerSecond": 8.435174890359233,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7554039808,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 2936815616,
"cpuThreads": 4,
"totalMs": 2200.0565839916817,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
"torchVersion": "2.10.0",
"loadMs": 1961.5091249943362,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 466
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 312.9267920012353,
"inferenceMs": 311.21004200394964,
"tokensPerSecond": 6.4265278431298745,
"inputTokens": 68,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7563755520,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 2954395648,
"cpuThreads": 4,
"totalMs": 2274.4359169955715,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "hola"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
"torchVersion": "2.10.0",
"loadMs": 1961.5091249943362,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 466
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 347.84408300038194,
"inferenceMs": 346.2073339978815,
"tokensPerSecond": 8.66532769643279,
"inputTokens": 69,
"outputTokens": 3,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7565115392,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 2970320896,
"cpuThreads": 4,
"totalMs": 2309.353207994718,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "2026-11-04"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
"torchVersion": "2.10.0",
"loadMs": 1961.5091249943362,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 466
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 502.90058300015517,
"inferenceMs": 500.97775000176625,
"tokensPerSecond": 13.97267643118945,
"inputTokens": 72,
"outputTokens": 7,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7566589952,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 2985754624,
"cpuThreads": 4,
"totalMs": 2464.4097079944913,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "Unknown"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
"torchVersion": "2.10.0",
"loadMs": 1961.5091249943362,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 466
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 314.43504100025166,
"inferenceMs": 312.2585410019383,
"tokensPerSecond": 6.404948904144099,
"inputTokens": 74,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7568146432,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"rssBytes": 3000975360,
"cpuThreads": 4,
"totalMs": 2275.944165994588,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "fail",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"plainParity": true,
"passed": false,
"expected": "Paris"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
"torchVersion": "2.10.0",
"loadMs": 4617.936540998926,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 466
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 6023.984749997908,
"inferenceMs": 6013.853958000254,
"tokensPerSecond": 0.33256544205557104,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 13423312896,
"cpuThreads": 4,
"totalMs": 10641.921290996834,
"grade": "slow",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "fail",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": false,
"expected": "56"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
"torchVersion": "2.10.0",
"loadMs": 4617.936540998926,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 466
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 5414.4182500022,
"inferenceMs": 5413.9613330044085,
"tokensPerSecond": 0.36941527229011917,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 13427589120,
"cpuThreads": 4,
"totalMs": 10032.354791001126,
"grade": "slow",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "fail",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": false,
"expected": "chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
"torchVersion": "2.10.0",
"loadMs": 4617.936540998926,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 466
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 5755.799167003715,
"inferenceMs": 5755.414541999926,
"tokensPerSecond": 0.34749886136004166,
"inputTokens": 68,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 13427818496,
"cpuThreads": 4,
"totalMs": 10373.735708002641,
"grade": "slow",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "fail",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": false,
"expected": "hola"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
"torchVersion": "2.10.0",
"loadMs": 4617.936540998926,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 466
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 8517.614333002712,
"inferenceMs": 8517.201750000822,
"tokensPerSecond": 0.352228359507829,
"inputTokens": 69,
"outputTokens": 3,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 13432111104,
"cpuThreads": 4,
"totalMs": 13135.550874001638,
"grade": "slow",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "fail",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": false,
"expected": "2026-11-04"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
"torchVersion": "2.10.0",
"loadMs": 4617.936540998926,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 466
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 17596.97354200034,
"inferenceMs": 17596.55304199987,
"tokensPerSecond": 0.39780518282712723,
"inputTokens": 72,
"outputTokens": 7,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 13432127488,
"cpuThreads": 4,
"totalMs": 22214.910082999268,
"grade": "slow",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "fail",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": false,
"expected": "Unknown"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
"torchVersion": "2.10.0",
"loadMs": 4617.936540998926,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 466
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 5506.182166995131,
"inferenceMs": 5505.780542000139,
"tokensPerSecond": 0.36325458029851665,
"inputTokens": 74,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 13432143872,
"cpuThreads": 4,
"totalMs": 10124.118707994057,
"grade": "slow",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "fail",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"plainParity": true,
"passed": false,
"expected": "Paris"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
"torchVersion": "2.10.0",
"loadMs": 6963.629167003091,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 466
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 2427.8841669947724,
"inferenceMs": 2417.44362499594,
"tokensPerSecond": 0.8273202234461038,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 13416546304,
"cpuThreads": 1,
"totalMs": 9391.513333997864,
"grade": "slow",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "fail",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": false,
"expected": "56"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
"torchVersion": "2.10.0",
"loadMs": 6963.629167003091,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 466
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 2600.4014999998617,
"inferenceMs": 2599.905208000564,
"tokensPerSecond": 0.7692588152235303,
"inputTokens": 63,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 13420773376,
"cpuThreads": 1,
"totalMs": 9564.030667002953,
"grade": "slow",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "fail",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": false,
"expected": "chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
"torchVersion": "2.10.0",
"loadMs": 6963.629167003091,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 466
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 2958.3785840004566,
"inferenceMs": 2957.889666999108,
"tokensPerSecond": 0.6761577425668741,
"inputTokens": 68,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 13420806144,
"cpuThreads": 1,
"totalMs": 9922.007751003548,
"grade": "slow",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "fail",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": false,
"expected": "hola"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
"torchVersion": "2.10.0",
"loadMs": 6963.629167003091,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 466
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 3573.3351669987314,
"inferenceMs": 3572.786709002685,
"tokensPerSecond": 0.839680687470265,
"inputTokens": 69,
"outputTokens": 3,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 13433454592,
"cpuThreads": 1,
"totalMs": 10536.964334001823,
"grade": "slow",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "fail",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": false,
"expected": "2026-11-04"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
"torchVersion": "2.10.0",
"loadMs": 6963.629167003091,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 466
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 5862.599625004805,
"inferenceMs": 5862.039291001565,
"tokensPerSecond": 1.1941236918601423,
"inputTokens": 72,
"outputTokens": 7,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 13458685952,
"cpuThreads": 1,
"totalMs": 12826.228792007896,
"grade": "slow",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "fail",
"quality": {
"exactNormalized": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": false,
"expected": "Unknown"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
"torchVersion": "2.10.0",
"loadMs": 6963.629167003091,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 466
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 2611.811167000269,
"inferenceMs": 2611.232666997239,
"tokensPerSecond": 0.7659217906077592,
"inputTokens": 74,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"outputLimitReached": false,
"rssBytes": 13458685952,
"cpuThreads": 1,
"totalMs": 9575.44033400336,
"grade": "slow",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
}
]Granite 4.0 Hybrid Small
30 configurations testedQuality criteria
No explicit quality gate recorded. Review per-case evidence; full certification is pending.
| Precision | Device | Result | Platform | Inference / load | |
|---|---|---|---|---|---|
Q4 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 552 ms / 1374 ms |
Q4 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 558 ms / 1278 ms |
Q4 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 564 ms / 1559 ms |
Q4 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 402 ms / 1559 ms |
Q4 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 403 ms / 1278 ms |
Q4 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 405 ms / 1374 ms |
Q4 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 330 ms / 1278 ms |
Q4 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 333 ms / 1559 ms |
Q4 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 334 ms / 1374 ms |
Q4 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 357 ms / 1278 ms |
Q4 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 358 ms / 1374 ms |
Q4 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 365 ms / 1374 ms |
Q4 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 366 ms / 1559 ms |
Q4 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 366 ms / 1278 ms |
Q4 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 378 ms / 1374 ms |
Q4 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 379 ms / 1559 ms |
Q4 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 385 ms / 1278 ms |
Q4 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 412 ms / 1559 ms |
Q4 | cpu | Failed | — | macos · aarch64 | 3833 ms / 34820 ms |
Q4 | cpu | Failed | — | macos · aarch64 | 2350 ms / 34820 ms |
Q4 | cpu | Failed | — | macos · aarch64 | 2191 ms / 34820 ms |
Q4 | cpu | Failed | — | macos · aarch64 | 2424 ms / 34820 ms |
Q4 | cpu | Failed | — | macos · aarch64 | 2686 ms / 34820 ms |
Q4 | cpu | Failed | — | macos · aarch64 | 2411 ms / 34820 ms |
Q4 | cpu | Failed | — | macos · aarch64 | 7620 ms / 19608 ms |
Q4 | cpu | Failed | — | macos · aarch64 | 7652 ms / 19608 ms |
Q4 | cpu | Failed | — | macos · aarch64 | 8072 ms / 19608 ms |
Q4 | cpu | Failed | — | macos · aarch64 | 8438 ms / 19608 ms |
Q4 | cpu | Failed | — | macos · aarch64 | 9486 ms / 19608 ms |
Q4 | cpu | Failed | — | macos · aarch64 | 8769 ms / 19608 ms |
Test setup & sample details
- Quality gate
- No explicit quality gate recorded. Review the per-case quality evidence; full certification is pending.
- Tested
- 2026-11-07T12:55:47.397494+00:00
- Environment
- { "os": "macos", "arch": "aarch64", "hardware": "Apple M3 Max", "osVersion": "27.2" }
- Runtime
- llama.cpp 0.3.36
- Artifact revision
- 6522095069a86fc3186b8632850e4373dd850cac
- Fixture
{ "file": "model-samples/desktop/granite4/granite-4.0-h-small-native-capital.json", "size": 244, "sha1": "9013b4f5a71cd1280d78b3dfa751ead1186d071f", "sha256": "8ff34789fa2a0f5bbef2080057f4b36376d3c73ea64baccefb6caec96eafbd68", "url": "https://assets.localai.tools/samples/v1/files/8ff34789fa2a0f5bbef2080057f4b36376d3c73ea64baccefb6caec96eafbd68/granite-4.0-h-small-native-capital.json" }- Rerun when
- Artifact/runtime revision; Input/generation processing; Fixture/quality gates; OS/backend behavior
Per-case quality & failure details
[
{
"precision": "q4",
"device": "mps",
"status": "pass",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"plainParity": true,
"passed": true,
"expected": "Paris"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
"device": "mps",
"precision": "q4",
"provider": "llama.cpp Metal",
"loadMs": 1558.7679159943946,
"acceleration": "partial",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer 0 assigned to device MTL0, is_swa = 0\nload_tensors: layer 1 assigned to device MTL0, is_swa = 0\nload_tensors: layer 2 assigned to device MTL0, is_swa = 0\nload_tensors: layer 3 assigned to device MTL0, is_swa = 0\nload_tensors: layer 4 assigned to device MTL0, is_swa = 0\nload_tensors: layer 5 assigned to device MTL0, is_swa = 0\nload_tensors: layer 6 assigned to device MTL0, is_swa = 0\nload_tensors: layer 7 assigned to device MTL0, is_swa = 0\nload_tensors: layer 8 assigned to device MTL0, is_swa = 0\nload_tensors: layer 9 assigned to device MTL0, is_swa = 0\nload_tensors: layer 10 assigned to device MTL0, is_swa = 0\nload_tensors: layer 11 assigned to device MTL0, is_swa = 0\nload_tensors: layer 12 assigned to device MTL0, is_swa = 0\nload_tensors: layer 13 assigned to device MTL0, is_swa = 0\nload_tensors: layer 14 assigned to device MTL0, is_swa = 0\nload_tensors: layer 15 assigned to device MTL0, is_swa = 0\nload_tensors: layer 16 assigned to device MTL0, is_swa = 0\nload_tensors: layer 17 assigned to device MTL0, is_swa = 0\nload_tensors: layer 18 assigned to device MTL0, is_swa = 0\nload_tensors: layer 19 assigned to device MTL0, is_swa = 0\nload_tensors: layer 20 assigned to device MTL0, is_swa = 0\nload_tensors: layer 21 assigned to device MTL0, is_swa = 0\nload_tensors: layer 22 assigned to device MTL0, is_swa = 0\nload_tensors: layer 23 assigned to device MTL0, is_swa = 0\nload_tensors: layer 24 assigned to device MTL0, is_swa = 0\nload_tensors: layer 25 assigned to device MTL0, is_swa = 0\nload_tensors: layer 26 assigned to device MTL0, is_swa = 0\nload_tensors: layer 27 assigned to device MTL0, is_swa = 0\nload_tensors: layer 28 assigned to device MTL0, is_swa = 0\nload_tensors: layer 29 assigned to device MTL0, is_swa = 0\nload_tensors: layer 30 assigned to device MTL0, is_swa = 0\nload_tensors: layer 31 assigned to device MTL0, is_swa = 0\nload_tensors: layer 32 assigned to device MTL0, is_swa = 0\nload_tensors: layer 33 assigned to device MTL0, is_swa = 0\nload_tensors: layer 34 assigned to device MTL0, is_swa = 0\nload_tensors: layer 35 assigned to device MTL0, is_swa = 0\nload_tensors: layer 36 assigned to device MTL0, is_swa = 0\nload_tensors: layer 37 assigned to device MTL0, is_swa = 0\nload_tensors: layer 38 assigned to device MTL0, is_swa = 0\nload_tensors: layer 39 assigned to device MTL0, is_swa = 0\nload_tensors: layer 40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 321.56 MiB\nload_tensors: MTL0_Mapped model buffer size = 18570.93 MiB\nllama_kv_cache: layer 5: dev = MTL0\nllama_kv_cache: layer 15: dev = MTL0\nllama_kv_cache: layer 25: dev = MTL0\nllama_kv_cache: layer 35: dev = MTL0\nllama_kv_cache: MTL0 KV buffer size = 128.00 MiB\nllama_memory_recurrent, layer 0: dev = MTL0\nllama_memory_recurrent, layer 1: dev = MTL0\nllama_memory_recurrent, layer 2: dev = MTL0\nllama_memory_recurrent, layer 3: dev = MTL0\nllama_memory_recurrent, layer 4: dev = MTL0\nllama_memory_recurrent, layer 6: dev = MTL0\nllama_memory_recurrent, layer 7: dev = MTL0\nllama_memory_recurrent, layer 8: dev = MTL0\nllama_memory_recurrent, layer 9: dev = MTL0\nllama_memory_recurrent, layer 10: dev = MTL0\nllama_memory_recurrent, layer 11: dev = MTL0\nllama_memory_recurrent, layer 12: dev = MTL0\nllama_memory_recurrent, layer 13: dev = MTL0\nllama_memory_recurrent, layer 14: dev = MTL0\nllama_memory_recurrent, layer 16: dev = MTL0\nllama_memory_recurrent, layer 17: dev = MTL0\nllama_memory_recurrent, layer 18: dev = MTL0\nllama_memory_recurrent, layer 19: dev = MTL0\nllama_memory_recurrent, layer 20: dev = MTL0\nllama_memory_recurrent, layer 21: dev = MTL0\nllama_memory_recurrent, layer 22: dev = MTL0\nllama_memory_recurrent, layer 23: dev = MTL0\nllama_memory_recurrent, layer 24: dev = MTL0\nllama_memory_recurrent, layer 26: dev = MTL0\nllama_memory_recurrent, layer 27: dev = MTL0\nllama_memory_recurrent, layer 28: dev = MTL0\nllama_memory_recurrent, layer 29: dev = MTL0\nllama_memory_recurrent, layer 30: dev = MTL0\nllama_memory_recurrent, layer 31: dev = MTL0\nllama_memory_recurrent, layer 32: dev = MTL0\nllama_memory_recurrent, layer 33: dev = MTL0\nllama_memory_recurrent, layer 34: dev = MTL0\nllama_memory_recurrent, layer 36: dev = MTL0\nllama_memory_recurrent, layer 37: dev = MTL0\nllama_memory_recurrent, layer 38: dev = MTL0\nllama_memory_recurrent, layer 39: dev = MTL0\nllama_memory_recurrent: MTL0 RS buffer size = 147.48 MiB\nsched_reserve: MTL0 compute buffer size = 282.28 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 41,
"totalLayers": 41,
"inferenceMs": 411.71433399722446,
"durationMs": 421.66816600365564,
"tokensPerSecond": 4.857737112484121,
"inputTokens": 63,
"outputTokens": 2,
"outputLimitReached": false,
"rssBytes": 19944783872,
"cpuThreads": 4,
"totalMs": 1980.4360819980502,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "q4",
"device": "mps",
"status": "pass",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "56"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
"device": "mps",
"precision": "q4",
"provider": "llama.cpp Metal",
"loadMs": 1558.7679159943946,
"acceleration": "partial",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer 0 assigned to device MTL0, is_swa = 0\nload_tensors: layer 1 assigned to device MTL0, is_swa = 0\nload_tensors: layer 2 assigned to device MTL0, is_swa = 0\nload_tensors: layer 3 assigned to device MTL0, is_swa = 0\nload_tensors: layer 4 assigned to device MTL0, is_swa = 0\nload_tensors: layer 5 assigned to device MTL0, is_swa = 0\nload_tensors: layer 6 assigned to device MTL0, is_swa = 0\nload_tensors: layer 7 assigned to device MTL0, is_swa = 0\nload_tensors: layer 8 assigned to device MTL0, is_swa = 0\nload_tensors: layer 9 assigned to device MTL0, is_swa = 0\nload_tensors: layer 10 assigned to device MTL0, is_swa = 0\nload_tensors: layer 11 assigned to device MTL0, is_swa = 0\nload_tensors: layer 12 assigned to device MTL0, is_swa = 0\nload_tensors: layer 13 assigned to device MTL0, is_swa = 0\nload_tensors: layer 14 assigned to device MTL0, is_swa = 0\nload_tensors: layer 15 assigned to device MTL0, is_swa = 0\nload_tensors: layer 16 assigned to device MTL0, is_swa = 0\nload_tensors: layer 17 assigned to device MTL0, is_swa = 0\nload_tensors: layer 18 assigned to device MTL0, is_swa = 0\nload_tensors: layer 19 assigned to device MTL0, is_swa = 0\nload_tensors: layer 20 assigned to device MTL0, is_swa = 0\nload_tensors: layer 21 assigned to device MTL0, is_swa = 0\nload_tensors: layer 22 assigned to device MTL0, is_swa = 0\nload_tensors: layer 23 assigned to device MTL0, is_swa = 0\nload_tensors: layer 24 assigned to device MTL0, is_swa = 0\nload_tensors: layer 25 assigned to device MTL0, is_swa = 0\nload_tensors: layer 26 assigned to device MTL0, is_swa = 0\nload_tensors: layer 27 assigned to device MTL0, is_swa = 0\nload_tensors: layer 28 assigned to device MTL0, is_swa = 0\nload_tensors: layer 29 assigned to device MTL0, is_swa = 0\nload_tensors: layer 30 assigned to device MTL0, is_swa = 0\nload_tensors: layer 31 assigned to device MTL0, is_swa = 0\nload_tensors: layer 32 assigned to device MTL0, is_swa = 0\nload_tensors: layer 33 assigned to device MTL0, is_swa = 0\nload_tensors: layer 34 assigned to device MTL0, is_swa = 0\nload_tensors: layer 35 assigned to device MTL0, is_swa = 0\nload_tensors: layer 36 assigned to device MTL0, is_swa = 0\nload_tensors: layer 37 assigned to device MTL0, is_swa = 0\nload_tensors: layer 38 assigned to device MTL0, is_swa = 0\nload_tensors: layer 39 assigned to device MTL0, is_swa = 0\nload_tensors: layer 40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 321.56 MiB\nload_tensors: MTL0_Mapped model buffer size = 18570.93 MiB\nllama_kv_cache: layer 5: dev = MTL0\nllama_kv_cache: layer 15: dev = MTL0\nllama_kv_cache: layer 25: dev = MTL0\nllama_kv_cache: layer 35: dev = MTL0\nllama_kv_cache: MTL0 KV buffer size = 128.00 MiB\nllama_memory_recurrent, layer 0: dev = MTL0\nllama_memory_recurrent, layer 1: dev = MTL0\nllama_memory_recurrent, layer 2: dev = MTL0\nllama_memory_recurrent, layer 3: dev = MTL0\nllama_memory_recurrent, layer 4: dev = MTL0\nllama_memory_recurrent, layer 6: dev = MTL0\nllama_memory_recurrent, layer 7: dev = MTL0\nllama_memory_recurrent, layer 8: dev = MTL0\nllama_memory_recurrent, layer 9: dev = MTL0\nllama_memory_recurrent, layer 10: dev = MTL0\nllama_memory_recurrent, layer 11: dev = MTL0\nllama_memory_recurrent, layer 12: dev = MTL0\nllama_memory_recurrent, layer 13: dev = MTL0\nllama_memory_recurrent, layer 14: dev = MTL0\nllama_memory_recurrent, layer 16: dev = MTL0\nllama_memory_recurrent, layer 17: dev = MTL0\nllama_memory_recurrent, layer 18: dev = MTL0\nllama_memory_recurrent, layer 19: dev = MTL0\nllama_memory_recurrent, layer 20: dev = MTL0\nllama_memory_recurrent, layer 21: dev = MTL0\nllama_memory_recurrent, layer 22: dev = MTL0\nllama_memory_recurrent, layer 23: dev = MTL0\nllama_memory_recurrent, layer 24: dev = MTL0\nllama_memory_recurrent, layer 26: dev = MTL0\nllama_memory_recurrent, layer 27: dev = MTL0\nllama_memory_recurrent, layer 28: dev = MTL0\nllama_memory_recurrent, layer 29: dev = MTL0\nllama_memory_recurrent, layer 30: dev = MTL0\nllama_memory_recurrent, layer 31: dev = MTL0\nllama_memory_recurrent, layer 32: dev = MTL0\nllama_memory_recurrent, layer 33: dev = MTL0\nllama_memory_recurrent, layer 34: dev = MTL0\nllama_memory_recurrent, layer 36: dev = MTL0\nllama_memory_recurrent, layer 37: dev = MTL0\nllama_memory_recurrent, layer 38: dev = MTL0\nllama_memory_recurrent, layer 39: dev = MTL0\nllama_memory_recurrent: MTL0 RS buffer size = 147.48 MiB\nsched_reserve: MTL0 compute buffer size = 282.28 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 41,
"totalLayers": 41,
"inferenceMs": 333.4497090036166,
"durationMs": 343.2479159964714,
"tokensPerSecond": 5.997905969017679,
"inputTokens": 63,
"outputTokens": 2,
"outputLimitReached": false,
"rssBytes": 19946078208,
"cpuThreads": 4,
"totalMs": 1902.015831990866,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "q4",
"device": "mps",
"status": "pass",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "chat"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
"device": "mps",
"precision": "q4",
"provider": "llama.cpp Metal",
"loadMs": 1558.7679159943946,
"acceleration": "partial",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer 0 assigned to device MTL0, is_swa = 0\nload_tensors: layer 1 assigned to device MTL0, is_swa = 0\nload_tensors: layer 2 assigned to device MTL0, is_swa = 0\nload_tensors: layer 3 assigned to device MTL0, is_swa = 0\nload_tensors: layer 4 assigned to device MTL0, is_swa = 0\nload_tensors: layer 5 assigned to device MTL0, is_swa = 0\nload_tensors: layer 6 assigned to device MTL0, is_swa = 0\nload_tensors: layer 7 assigned to device MTL0, is_swa = 0\nload_tensors: layer 8 assigned to device MTL0, is_swa = 0\nload_tensors: layer 9 assigned to device MTL0, is_swa = 0\nload_tensors: layer 10 assigned to device MTL0, is_swa = 0\nload_tensors: layer 11 assigned to device MTL0, is_swa = 0\nload_tensors: layer 12 assigned to device MTL0, is_swa = 0\nload_tensors: layer 13 assigned to device MTL0, is_swa = 0\nload_tensors: layer 14 assigned to device MTL0, is_swa = 0\nload_tensors: layer 15 assigned to device MTL0, is_swa = 0\nload_tensors: layer 16 assigned to device MTL0, is_swa = 0\nload_tensors: layer 17 assigned to device MTL0, is_swa = 0\nload_tensors: layer 18 assigned to device MTL0, is_swa = 0\nload_tensors: layer 19 assigned to device MTL0, is_swa = 0\nload_tensors: layer 20 assigned to device MTL0, is_swa = 0\nload_tensors: layer 21 assigned to device MTL0, is_swa = 0\nload_tensors: layer 22 assigned to device MTL0, is_swa = 0\nload_tensors: layer 23 assigned to device MTL0, is_swa = 0\nload_tensors: layer 24 assigned to device MTL0, is_swa = 0\nload_tensors: layer 25 assigned to device MTL0, is_swa = 0\nload_tensors: layer 26 assigned to device MTL0, is_swa = 0\nload_tensors: layer 27 assigned to device MTL0, is_swa = 0\nload_tensors: layer 28 assigned to device MTL0, is_swa = 0\nload_tensors: layer 29 assigned to device MTL0, is_swa = 0\nload_tensors: layer 30 assigned to device MTL0, is_swa = 0\nload_tensors: layer 31 assigned to device MTL0, is_swa = 0\nload_tensors: layer 32 assigned to device MTL0, is_swa = 0\nload_tensors: layer 33 assigned to device MTL0, is_swa = 0\nload_tensors: layer 34 assigned to device MTL0, is_swa = 0\nload_tensors: layer 35 assigned to device MTL0, is_swa = 0\nload_tensors: layer 36 assigned to device MTL0, is_swa = 0\nload_tensors: layer 37 assigned to device MTL0, is_swa = 0\nload_tensors: layer 38 assigned to device MTL0, is_swa = 0\nload_tensors: layer 39 assigned to device MTL0, is_swa = 0\nload_tensors: layer 40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 321.56 MiB\nload_tensors: MTL0_Mapped model buffer size = 18570.93 MiB\nllama_kv_cache: layer 5: dev = MTL0\nllama_kv_cache: layer 15: dev = MTL0\nllama_kv_cache: layer 25: dev = MTL0\nllama_kv_cache: layer 35: dev = MTL0\nllama_kv_cache: MTL0 KV buffer size = 128.00 MiB\nllama_memory_recurrent, layer 0: dev = MTL0\nllama_memory_recurrent, layer 1: dev = MTL0\nllama_memory_recurrent, layer 2: dev = MTL0\nllama_memory_recurrent, layer 3: dev = MTL0\nllama_memory_recurrent, layer 4: dev = MTL0\nllama_memory_recurrent, layer 6: dev = MTL0\nllama_memory_recurrent, layer 7: dev = MTL0\nllama_memory_recurrent, layer 8: dev = MTL0\nllama_memory_recurrent, layer 9: dev = MTL0\nllama_memory_recurrent, layer 10: dev = MTL0\nllama_memory_recurrent, layer 11: dev = MTL0\nllama_memory_recurrent, layer 12: dev = MTL0\nllama_memory_recurrent, layer 13: dev = MTL0\nllama_memory_recurrent, layer 14: dev = MTL0\nllama_memory_recurrent, layer 16: dev = MTL0\nllama_memory_recurrent, layer 17: dev = MTL0\nllama_memory_recurrent, layer 18: dev = MTL0\nllama_memory_recurrent, layer 19: dev = MTL0\nllama_memory_recurrent, layer 20: dev = MTL0\nllama_memory_recurrent, layer 21: dev = MTL0\nllama_memory_recurrent, layer 22: dev = MTL0\nllama_memory_recurrent, layer 23: dev = MTL0\nllama_memory_recurrent, layer 24: dev = MTL0\nllama_memory_recurrent, layer 26: dev = MTL0\nllama_memory_recurrent, layer 27: dev = MTL0\nllama_memory_recurrent, layer 28: dev = MTL0\nllama_memory_recurrent, layer 29: dev = MTL0\nllama_memory_recurrent, layer 30: dev = MTL0\nllama_memory_recurrent, layer 31: dev = MTL0\nllama_memory_recurrent, layer 32: dev = MTL0\nllama_memory_recurrent, layer 33: dev = MTL0\nllama_memory_recurrent, layer 34: dev = MTL0\nllama_memory_recurrent, layer 36: dev = MTL0\nllama_memory_recurrent, layer 37: dev = MTL0\nllama_memory_recurrent, layer 38: dev = MTL0\nllama_memory_recurrent, layer 39: dev = MTL0\nllama_memory_recurrent: MTL0 RS buffer size = 147.48 MiB\nsched_reserve: MTL0 compute buffer size = 282.28 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 41,
"totalLayers": 41,
"inferenceMs": 366.17212500277674,
"durationMs": 375.0527919983142,
"tokensPerSecond": 5.461912208595435,
"inputTokens": 68,
"outputTokens": 2,
"outputLimitReached": false,
"rssBytes": 19948191744,
"cpuThreads": 4,
"totalMs": 1933.8207079927088,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "q4",
"device": "mps",
"status": "pass",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "hola"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
"device": "mps",
"precision": "q4",
"provider": "llama.cpp Metal",
"loadMs": 1558.7679159943946,
"acceleration": "partial",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer 0 assigned to device MTL0, is_swa = 0\nload_tensors: layer 1 assigned to device MTL0, is_swa = 0\nload_tensors: layer 2 assigned to device MTL0, is_swa = 0\nload_tensors: layer 3 assigned to device MTL0, is_swa = 0\nload_tensors: layer 4 assigned to device MTL0, is_swa = 0\nload_tensors: layer 5 assigned to device MTL0, is_swa = 0\nload_tensors: layer 6 assigned to device MTL0, is_swa = 0\nload_tensors: layer 7 assigned to device MTL0, is_swa = 0\nload_tensors: layer 8 assigned to device MTL0, is_swa = 0\nload_tensors: layer 9 assigned to device MTL0, is_swa = 0\nload_tensors: layer 10 assigned to device MTL0, is_swa = 0\nload_tensors: layer 11 assigned to device MTL0, is_swa = 0\nload_tensors: layer 12 assigned to device MTL0, is_swa = 0\nload_tensors: layer 13 assigned to device MTL0, is_swa = 0\nload_tensors: layer 14 assigned to device MTL0, is_swa = 0\nload_tensors: layer 15 assigned to device MTL0, is_swa = 0\nload_tensors: layer 16 assigned to device MTL0, is_swa = 0\nload_tensors: layer 17 assigned to device MTL0, is_swa = 0\nload_tensors: layer 18 assigned to device MTL0, is_swa = 0\nload_tensors: layer 19 assigned to device MTL0, is_swa = 0\nload_tensors: layer 20 assigned to device MTL0, is_swa = 0\nload_tensors: layer 21 assigned to device MTL0, is_swa = 0\nload_tensors: layer 22 assigned to device MTL0, is_swa = 0\nload_tensors: layer 23 assigned to device MTL0, is_swa = 0\nload_tensors: layer 24 assigned to device MTL0, is_swa = 0\nload_tensors: layer 25 assigned to device MTL0, is_swa = 0\nload_tensors: layer 26 assigned to device MTL0, is_swa = 0\nload_tensors: layer 27 assigned to device MTL0, is_swa = 0\nload_tensors: layer 28 assigned to device MTL0, is_swa = 0\nload_tensors: layer 29 assigned to device MTL0, is_swa = 0\nload_tensors: layer 30 assigned to device MTL0, is_swa = 0\nload_tensors: layer 31 assigned to device MTL0, is_swa = 0\nload_tensors: layer 32 assigned to device MTL0, is_swa = 0\nload_tensors: layer 33 assigned to device MTL0, is_swa = 0\nload_tensors: layer 34 assigned to device MTL0, is_swa = 0\nload_tensors: layer 35 assigned to device MTL0, is_swa = 0\nload_tensors: layer 36 assigned to device MTL0, is_swa = 0\nload_tensors: layer 37 assigned to device MTL0, is_swa = 0\nload_tensors: layer 38 assigned to device MTL0, is_swa = 0\nload_tensors: layer 39 assigned to device MTL0, is_swa = 0\nload_tensors: layer 40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 321.56 MiB\nload_tensors: MTL0_Mapped model buffer size = 18570.93 MiB\nllama_kv_cache: layer 5: dev = MTL0\nllama_kv_cache: layer 15: dev = MTL0\nllama_kv_cache: layer 25: dev = MTL0\nllama_kv_cache: layer 35: dev = MTL0\nllama_kv_cache: MTL0 KV buffer size = 128.00 MiB\nllama_memory_recurrent, layer 0: dev = MTL0\nllama_memory_recurrent, layer 1: dev = MTL0\nllama_memory_recurrent, layer 2: dev = MTL0\nllama_memory_recurrent, layer 3: dev = MTL0\nllama_memory_recurrent, layer 4: dev = MTL0\nllama_memory_recurrent, layer 6: dev = MTL0\nllama_memory_recurrent, layer 7: dev = MTL0\nllama_memory_recurrent, layer 8: dev = MTL0\nllama_memory_recurrent, layer 9: dev = MTL0\nllama_memory_recurrent, layer 10: dev = MTL0\nllama_memory_recurrent, layer 11: dev = MTL0\nllama_memory_recurrent, layer 12: dev = MTL0\nllama_memory_recurrent, layer 13: dev = MTL0\nllama_memory_recurrent, layer 14: dev = MTL0\nllama_memory_recurrent, layer 16: dev = MTL0\nllama_memory_recurrent, layer 17: dev = MTL0\nllama_memory_recurrent, layer 18: dev = MTL0\nllama_memory_recurrent, layer 19: dev = MTL0\nllama_memory_recurrent, layer 20: dev = MTL0\nllama_memory_recurrent, layer 21: dev = MTL0\nllama_memory_recurrent, layer 22: dev = MTL0\nllama_memory_recurrent, layer 23: dev = MTL0\nllama_memory_recurrent, layer 24: dev = MTL0\nllama_memory_recurrent, layer 26: dev = MTL0\nllama_memory_recurrent, layer 27: dev = MTL0\nllama_memory_recurrent, layer 28: dev = MTL0\nllama_memory_recurrent, layer 29: dev = MTL0\nllama_memory_recurrent, layer 30: dev = MTL0\nllama_memory_recurrent, layer 31: dev = MTL0\nllama_memory_recurrent, layer 32: dev = MTL0\nllama_memory_recurrent, layer 33: dev = MTL0\nllama_memory_recurrent, layer 34: dev = MTL0\nllama_memory_recurrent, layer 36: dev = MTL0\nllama_memory_recurrent, layer 37: dev = MTL0\nllama_memory_recurrent, layer 38: dev = MTL0\nllama_memory_recurrent, layer 39: dev = MTL0\nllama_memory_recurrent: MTL0 RS buffer size = 147.48 MiB\nsched_reserve: MTL0 compute buffer size = 282.28 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 41,
"totalLayers": 41,
"inferenceMs": 401.8288749939529,
"durationMs": 410.65070900367573,
"tokensPerSecond": 7.465864667005691,
"inputTokens": 69,
"outputTokens": 3,
"outputLimitReached": false,
"rssBytes": 19948191744,
"cpuThreads": 4,
"totalMs": 1969.4186249980703,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "q4",
"device": "mps",
"status": "pass",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "2026-11-04"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
"device": "mps",
"precision": "q4",
"provider": "llama.cpp Metal",
"loadMs": 1558.7679159943946,
"acceleration": "partial",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer 0 assigned to device MTL0, is_swa = 0\nload_tensors: layer 1 assigned to device MTL0, is_swa = 0\nload_tensors: layer 2 assigned to device MTL0, is_swa = 0\nload_tensors: layer 3 assigned to device MTL0, is_swa = 0\nload_tensors: layer 4 assigned to device MTL0, is_swa = 0\nload_tensors: layer 5 assigned to device MTL0, is_swa = 0\nload_tensors: layer 6 assigned to device MTL0, is_swa = 0\nload_tensors: layer 7 assigned to device MTL0, is_swa = 0\nload_tensors: layer 8 assigned to device MTL0, is_swa = 0\nload_tensors: layer 9 assigned to device MTL0, is_swa = 0\nload_tensors: layer 10 assigned to device MTL0, is_swa = 0\nload_tensors: layer 11 assigned to device MTL0, is_swa = 0\nload_tensors: layer 12 assigned to device MTL0, is_swa = 0\nload_tensors: layer 13 assigned to device MTL0, is_swa = 0\nload_tensors: layer 14 assigned to device MTL0, is_swa = 0\nload_tensors: layer 15 assigned to device MTL0, is_swa = 0\nload_tensors: layer 16 assigned to device MTL0, is_swa = 0\nload_tensors: layer 17 assigned to device MTL0, is_swa = 0\nload_tensors: layer 18 assigned to device MTL0, is_swa = 0\nload_tensors: layer 19 assigned to device MTL0, is_swa = 0\nload_tensors: layer 20 assigned to device MTL0, is_swa = 0\nload_tensors: layer 21 assigned to device MTL0, is_swa = 0\nload_tensors: layer 22 assigned to device MTL0, is_swa = 0\nload_tensors: layer 23 assigned to device MTL0, is_swa = 0\nload_tensors: layer 24 assigned to device MTL0, is_swa = 0\nload_tensors: layer 25 assigned to device MTL0, is_swa = 0\nload_tensors: layer 26 assigned to device MTL0, is_swa = 0\nload_tensors: layer 27 assigned to device MTL0, is_swa = 0\nload_tensors: layer 28 assigned to device MTL0, is_swa = 0\nload_tensors: layer 29 assigned to device MTL0, is_swa = 0\nload_tensors: layer 30 assigned to device MTL0, is_swa = 0\nload_tensors: layer 31 assigned to device MTL0, is_swa = 0\nload_tensors: layer 32 assigned to device MTL0, is_swa = 0\nload_tensors: layer 33 assigned to device MTL0, is_swa = 0\nload_tensors: layer 34 assigned to device MTL0, is_swa = 0\nload_tensors: layer 35 assigned to device MTL0, is_swa = 0\nload_tensors: layer 36 assigned to device MTL0, is_swa = 0\nload_tensors: layer 37 assigned to device MTL0, is_swa = 0\nload_tensors: layer 38 assigned to device MTL0, is_swa = 0\nload_tensors: layer 39 assigned to device MTL0, is_swa = 0\nload_tensors: layer 40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 321.56 MiB\nload_tensors: MTL0_Mapped model buffer size = 18570.93 MiB\nllama_kv_cache: layer 5: dev = MTL0\nllama_kv_cache: layer 15: dev = MTL0\nllama_kv_cache: layer 25: dev = MTL0\nllama_kv_cache: layer 35: dev = MTL0\nllama_kv_cache: MTL0 KV buffer size = 128.00 MiB\nllama_memory_recurrent, layer 0: dev = MTL0\nllama_memory_recurrent, layer 1: dev = MTL0\nllama_memory_recurrent, layer 2: dev = MTL0\nllama_memory_recurrent, layer 3: dev = MTL0\nllama_memory_recurrent, layer 4: dev = MTL0\nllama_memory_recurrent, layer 6: dev = MTL0\nllama_memory_recurrent, layer 7: dev = MTL0\nllama_memory_recurrent, layer 8: dev = MTL0\nllama_memory_recurrent, layer 9: dev = MTL0\nllama_memory_recurrent, layer 10: dev = MTL0\nllama_memory_recurrent, layer 11: dev = MTL0\nllama_memory_recurrent, layer 12: dev = MTL0\nllama_memory_recurrent, layer 13: dev = MTL0\nllama_memory_recurrent, layer 14: dev = MTL0\nllama_memory_recurrent, layer 16: dev = MTL0\nllama_memory_recurrent, layer 17: dev = MTL0\nllama_memory_recurrent, layer 18: dev = MTL0\nllama_memory_recurrent, layer 19: dev = MTL0\nllama_memory_recurrent, layer 20: dev = MTL0\nllama_memory_recurrent, layer 21: dev = MTL0\nllama_memory_recurrent, layer 22: dev = MTL0\nllama_memory_recurrent, layer 23: dev = MTL0\nllama_memory_recurrent, layer 24: dev = MTL0\nllama_memory_recurrent, layer 26: dev = MTL0\nllama_memory_recurrent, layer 27: dev = MTL0\nllama_memory_recurrent, layer 28: dev = MTL0\nllama_memory_recurrent, layer 29: dev = MTL0\nllama_memory_recurrent, layer 30: dev = MTL0\nllama_memory_recurrent, layer 31: dev = MTL0\nllama_memory_recurrent, layer 32: dev = MTL0\nllama_memory_recurrent, layer 33: dev = MTL0\nllama_memory_recurrent, layer 34: dev = MTL0\nllama_memory_recurrent, layer 36: dev = MTL0\nllama_memory_recurrent, layer 37: dev = MTL0\nllama_memory_recurrent, layer 38: dev = MTL0\nllama_memory_recurrent, layer 39: dev = MTL0\nllama_memory_recurrent: MTL0 RS buffer size = 147.48 MiB\nsched_reserve: MTL0 compute buffer size = 282.28 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 41,
"totalLayers": 41,
"inferenceMs": 563.7482079982874,
"durationMs": 571.8166250007926,
"tokensPerSecond": 12.416890911023996,
"inputTokens": 72,
"outputTokens": 7,
"outputLimitReached": false,
"rssBytes": 19949109248,
"cpuThreads": 4,
"totalMs": 2130.584540995187,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "q4",
"device": "mps",
"status": "pass",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "Unknown"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
"device": "mps",
"precision": "q4",
"provider": "llama.cpp Metal",
"loadMs": 1558.7679159943946,
"acceleration": "partial",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer 0 assigned to device MTL0, is_swa = 0\nload_tensors: layer 1 assigned to device MTL0, is_swa = 0\nload_tensors: layer 2 assigned to device MTL0, is_swa = 0\nload_tensors: layer 3 assigned to device MTL0, is_swa = 0\nload_tensors: layer 4 assigned to device MTL0, is_swa = 0\nload_tensors: layer 5 assigned to device MTL0, is_swa = 0\nload_tensors: layer 6 assigned to device MTL0, is_swa = 0\nload_tensors: layer 7 assigned to device MTL0, is_swa = 0\nload_tensors: layer 8 assigned to device MTL0, is_swa = 0\nload_tensors: layer 9 assigned to device MTL0, is_swa = 0\nload_tensors: layer 10 assigned to device MTL0, is_swa = 0\nload_tensors: layer 11 assigned to device MTL0, is_swa = 0\nload_tensors: layer 12 assigned to device MTL0, is_swa = 0\nload_tensors: layer 13 assigned to device MTL0, is_swa = 0\nload_tensors: layer 14 assigned to device MTL0, is_swa = 0\nload_tensors: layer 15 assigned to device MTL0, is_swa = 0\nload_tensors: layer 16 assigned to device MTL0, is_swa = 0\nload_tensors: layer 17 assigned to device MTL0, is_swa = 0\nload_tensors: layer 18 assigned to device MTL0, is_swa = 0\nload_tensors: layer 19 assigned to device MTL0, is_swa = 0\nload_tensors: layer 20 assigned to device MTL0, is_swa = 0\nload_tensors: layer 21 assigned to device MTL0, is_swa = 0\nload_tensors: layer 22 assigned to device MTL0, is_swa = 0\nload_tensors: layer 23 assigned to device MTL0, is_swa = 0\nload_tensors: layer 24 assigned to device MTL0, is_swa = 0\nload_tensors: layer 25 assigned to device MTL0, is_swa = 0\nload_tensors: layer 26 assigned to device MTL0, is_swa = 0\nload_tensors: layer 27 assigned to device MTL0, is_swa = 0\nload_tensors: layer 28 assigned to device MTL0, is_swa = 0\nload_tensors: layer 29 assigned to device MTL0, is_swa = 0\nload_tensors: layer 30 assigned to device MTL0, is_swa = 0\nload_tensors: layer 31 assigned to device MTL0, is_swa = 0\nload_tensors: layer 32 assigned to device MTL0, is_swa = 0\nload_tensors: layer 33 assigned to device MTL0, is_swa = 0\nload_tensors: layer 34 assigned to device MTL0, is_swa = 0\nload_tensors: layer 35 assigned to device MTL0, is_swa = 0\nload_tensors: layer 36 assigned to device MTL0, is_swa = 0\nload_tensors: layer 37 assigned to device MTL0, is_swa = 0\nload_tensors: layer 38 assigned to device MTL0, is_swa = 0\nload_tensors: layer 39 assigned to device MTL0, is_swa = 0\nload_tensors: layer 40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 321.56 MiB\nload_tensors: MTL0_Mapped model buffer size = 18570.93 MiB\nllama_kv_cache: layer 5: dev = MTL0\nllama_kv_cache: layer 15: dev = MTL0\nllama_kv_cache: layer 25: dev = MTL0\nllama_kv_cache: layer 35: dev = MTL0\nllama_kv_cache: MTL0 KV buffer size = 128.00 MiB\nllama_memory_recurrent, layer 0: dev = MTL0\nllama_memory_recurrent, layer 1: dev = MTL0\nllama_memory_recurrent, layer 2: dev = MTL0\nllama_memory_recurrent, layer 3: dev = MTL0\nllama_memory_recurrent, layer 4: dev = MTL0\nllama_memory_recurrent, layer 6: dev = MTL0\nllama_memory_recurrent, layer 7: dev = MTL0\nllama_memory_recurrent, layer 8: dev = MTL0\nllama_memory_recurrent, layer 9: dev = MTL0\nllama_memory_recurrent, layer 10: dev = MTL0\nllama_memory_recurrent, layer 11: dev = MTL0\nllama_memory_recurrent, layer 12: dev = MTL0\nllama_memory_recurrent, layer 13: dev = MTL0\nllama_memory_recurrent, layer 14: dev = MTL0\nllama_memory_recurrent, layer 16: dev = MTL0\nllama_memory_recurrent, layer 17: dev = MTL0\nllama_memory_recurrent, layer 18: dev = MTL0\nllama_memory_recurrent, layer 19: dev = MTL0\nllama_memory_recurrent, layer 20: dev = MTL0\nllama_memory_recurrent, layer 21: dev = MTL0\nllama_memory_recurrent, layer 22: dev = MTL0\nllama_memory_recurrent, layer 23: dev = MTL0\nllama_memory_recurrent, layer 24: dev = MTL0\nllama_memory_recurrent, layer 26: dev = MTL0\nllama_memory_recurrent, layer 27: dev = MTL0\nllama_memory_recurrent, layer 28: dev = MTL0\nllama_memory_recurrent, layer 29: dev = MTL0\nllama_memory_recurrent, layer 30: dev = MTL0\nllama_memory_recurrent, layer 31: dev = MTL0\nllama_memory_recurrent, layer 32: dev = MTL0\nllama_memory_recurrent, layer 33: dev = MTL0\nllama_memory_recurrent, layer 34: dev = MTL0\nllama_memory_recurrent, layer 36: dev = MTL0\nllama_memory_recurrent, layer 37: dev = MTL0\nllama_memory_recurrent, layer 38: dev = MTL0\nllama_memory_recurrent, layer 39: dev = MTL0\nllama_memory_recurrent: MTL0 RS buffer size = 147.48 MiB\nsched_reserve: MTL0 compute buffer size = 282.28 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 41,
"totalLayers": 41,
"inferenceMs": 379.13874999503605,
"durationMs": 396.48783300071955,
"tokensPerSecond": 5.275113662283756,
"inputTokens": 74,
"outputTokens": 2,
"outputLimitReached": false,
"rssBytes": 19949142016,
"cpuThreads": 4,
"totalMs": 1955.2557489951141,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "q4",
"device": "mps",
"status": "pass",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "Paris"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
"device": "mps",
"precision": "q4",
"provider": "llama.cpp Metal",
"loadMs": 1277.7895419931156,
"acceleration": "partial",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer 0 assigned to device MTL0, is_swa = 0\nload_tensors: layer 1 assigned to device MTL0, is_swa = 0\nload_tensors: layer 2 assigned to device MTL0, is_swa = 0\nload_tensors: layer 3 assigned to device MTL0, is_swa = 0\nload_tensors: layer 4 assigned to device MTL0, is_swa = 0\nload_tensors: layer 5 assigned to device MTL0, is_swa = 0\nload_tensors: layer 6 assigned to device MTL0, is_swa = 0\nload_tensors: layer 7 assigned to device MTL0, is_swa = 0\nload_tensors: layer 8 assigned to device MTL0, is_swa = 0\nload_tensors: layer 9 assigned to device MTL0, is_swa = 0\nload_tensors: layer 10 assigned to device MTL0, is_swa = 0\nload_tensors: layer 11 assigned to device MTL0, is_swa = 0\nload_tensors: layer 12 assigned to device MTL0, is_swa = 0\nload_tensors: layer 13 assigned to device MTL0, is_swa = 0\nload_tensors: layer 14 assigned to device MTL0, is_swa = 0\nload_tensors: layer 15 assigned to device MTL0, is_swa = 0\nload_tensors: layer 16 assigned to device MTL0, is_swa = 0\nload_tensors: layer 17 assigned to device MTL0, is_swa = 0\nload_tensors: layer 18 assigned to device MTL0, is_swa = 0\nload_tensors: layer 19 assigned to device MTL0, is_swa = 0\nload_tensors: layer 20 assigned to device MTL0, is_swa = 0\nload_tensors: layer 21 assigned to device MTL0, is_swa = 0\nload_tensors: layer 22 assigned to device MTL0, is_swa = 0\nload_tensors: layer 23 assigned to device MTL0, is_swa = 0\nload_tensors: layer 24 assigned to device MTL0, is_swa = 0\nload_tensors: layer 25 assigned to device MTL0, is_swa = 0\nload_tensors: layer 26 assigned to device MTL0, is_swa = 0\nload_tensors: layer 27 assigned to device MTL0, is_swa = 0\nload_tensors: layer 28 assigned to device MTL0, is_swa = 0\nload_tensors: layer 29 assigned to device MTL0, is_swa = 0\nload_tensors: layer 30 assigned to device MTL0, is_swa = 0\nload_tensors: layer 31 assigned to device MTL0, is_swa = 0\nload_tensors: layer 32 assigned to device MTL0, is_swa = 0\nload_tensors: layer 33 assigned to device MTL0, is_swa = 0\nload_tensors: layer 34 assigned to device MTL0, is_swa = 0\nload_tensors: layer 35 assigned to device MTL0, is_swa = 0\nload_tensors: layer 36 assigned to device MTL0, is_swa = 0\nload_tensors: layer 37 assigned to device MTL0, is_swa = 0\nload_tensors: layer 38 assigned to device MTL0, is_swa = 0\nload_tensors: layer 39 assigned to device MTL0, is_swa = 0\nload_tensors: layer 40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 321.56 MiB\nload_tensors: MTL0_Mapped model buffer size = 18570.93 MiB\nllama_kv_cache: layer 5: dev = MTL0\nllama_kv_cache: layer 15: dev = MTL0\nllama_kv_cache: layer 25: dev = MTL0\nllama_kv_cache: layer 35: dev = MTL0\nllama_kv_cache: MTL0 KV buffer size = 128.00 MiB\nllama_memory_recurrent, layer 0: dev = MTL0\nllama_memory_recurrent, layer 1: dev = MTL0\nllama_memory_recurrent, layer 2: dev = MTL0\nllama_memory_recurrent, layer 3: dev = MTL0\nllama_memory_recurrent, layer 4: dev = MTL0\nllama_memory_recurrent, layer 6: dev = MTL0\nllama_memory_recurrent, layer 7: dev = MTL0\nllama_memory_recurrent, layer 8: dev = MTL0\nllama_memory_recurrent, layer 9: dev = MTL0\nllama_memory_recurrent, layer 10: dev = MTL0\nllama_memory_recurrent, layer 11: dev = MTL0\nllama_memory_recurrent, layer 12: dev = MTL0\nllama_memory_recurrent, layer 13: dev = MTL0\nllama_memory_recurrent, layer 14: dev = MTL0\nllama_memory_recurrent, layer 16: dev = MTL0\nllama_memory_recurrent, layer 17: dev = MTL0\nllama_memory_recurrent, layer 18: dev = MTL0\nllama_memory_recurrent, layer 19: dev = MTL0\nllama_memory_recurrent, layer 20: dev = MTL0\nllama_memory_recurrent, layer 21: dev = MTL0\nllama_memory_recurrent, layer 22: dev = MTL0\nllama_memory_recurrent, layer 23: dev = MTL0\nllama_memory_recurrent, layer 24: dev = MTL0\nllama_memory_recurrent, layer 26: dev = MTL0\nllama_memory_recurrent, layer 27: dev = MTL0\nllama_memory_recurrent, layer 28: dev = MTL0\nllama_memory_recurrent, layer 29: dev = MTL0\nllama_memory_recurrent, layer 30: dev = MTL0\nllama_memory_recurrent, layer 31: dev = MTL0\nllama_memory_recurrent, layer 32: dev = MTL0\nllama_memory_recurrent, layer 33: dev = MTL0\nllama_memory_recurrent, layer 34: dev = MTL0\nllama_memory_recurrent, layer 36: dev = MTL0\nllama_memory_recurrent, layer 37: dev = MTL0\nllama_memory_recurrent, layer 38: dev = MTL0\nllama_memory_recurrent, layer 39: dev = MTL0\nllama_memory_recurrent: MTL0 RS buffer size = 147.48 MiB\nsched_reserve: MTL0 compute buffer size = 282.28 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 41,
"totalLayers": 41,
"inferenceMs": 357.1684999988065,
"durationMs": 365.3875829986646,
"tokensPerSecond": 5.599597948885982,
"inputTokens": 63,
"outputTokens": 2,
"outputLimitReached": false,
"rssBytes": 19940950016,
"cpuThreads": 4,
"totalMs": 1643.1771249917801,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "q4",
"device": "mps",
"status": "pass",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "56"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
"device": "mps",
"precision": "q4",
"provider": "llama.cpp Metal",
"loadMs": 1277.7895419931156,
"acceleration": "partial",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer 0 assigned to device MTL0, is_swa = 0\nload_tensors: layer 1 assigned to device MTL0, is_swa = 0\nload_tensors: layer 2 assigned to device MTL0, is_swa = 0\nload_tensors: layer 3 assigned to device MTL0, is_swa = 0\nload_tensors: layer 4 assigned to device MTL0, is_swa = 0\nload_tensors: layer 5 assigned to device MTL0, is_swa = 0\nload_tensors: layer 6 assigned to device MTL0, is_swa = 0\nload_tensors: layer 7 assigned to device MTL0, is_swa = 0\nload_tensors: layer 8 assigned to device MTL0, is_swa = 0\nload_tensors: layer 9 assigned to device MTL0, is_swa = 0\nload_tensors: layer 10 assigned to device MTL0, is_swa = 0\nload_tensors: layer 11 assigned to device MTL0, is_swa = 0\nload_tensors: layer 12 assigned to device MTL0, is_swa = 0\nload_tensors: layer 13 assigned to device MTL0, is_swa = 0\nload_tensors: layer 14 assigned to device MTL0, is_swa = 0\nload_tensors: layer 15 assigned to device MTL0, is_swa = 0\nload_tensors: layer 16 assigned to device MTL0, is_swa = 0\nload_tensors: layer 17 assigned to device MTL0, is_swa = 0\nload_tensors: layer 18 assigned to device MTL0, is_swa = 0\nload_tensors: layer 19 assigned to device MTL0, is_swa = 0\nload_tensors: layer 20 assigned to device MTL0, is_swa = 0\nload_tensors: layer 21 assigned to device MTL0, is_swa = 0\nload_tensors: layer 22 assigned to device MTL0, is_swa = 0\nload_tensors: layer 23 assigned to device MTL0, is_swa = 0\nload_tensors: layer 24 assigned to device MTL0, is_swa = 0\nload_tensors: layer 25 assigned to device MTL0, is_swa = 0\nload_tensors: layer 26 assigned to device MTL0, is_swa = 0\nload_tensors: layer 27 assigned to device MTL0, is_swa = 0\nload_tensors: layer 28 assigned to device MTL0, is_swa = 0\nload_tensors: layer 29 assigned to device MTL0, is_swa = 0\nload_tensors: layer 30 assigned to device MTL0, is_swa = 0\nload_tensors: layer 31 assigned to device MTL0, is_swa = 0\nload_tensors: layer 32 assigned to device MTL0, is_swa = 0\nload_tensors: layer 33 assigned to device MTL0, is_swa = 0\nload_tensors: layer 34 assigned to device MTL0, is_swa = 0\nload_tensors: layer 35 assigned to device MTL0, is_swa = 0\nload_tensors: layer 36 assigned to device MTL0, is_swa = 0\nload_tensors: layer 37 assigned to device MTL0, is_swa = 0\nload_tensors: layer 38 assigned to device MTL0, is_swa = 0\nload_tensors: layer 39 assigned to device MTL0, is_swa = 0\nload_tensors: layer 40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 321.56 MiB\nload_tensors: MTL0_Mapped model buffer size = 18570.93 MiB\nllama_kv_cache: layer 5: dev = MTL0\nllama_kv_cache: layer 15: dev = MTL0\nllama_kv_cache: layer 25: dev = MTL0\nllama_kv_cache: layer 35: dev = MTL0\nllama_kv_cache: MTL0 KV buffer size = 128.00 MiB\nllama_memory_recurrent, layer 0: dev = MTL0\nllama_memory_recurrent, layer 1: dev = MTL0\nllama_memory_recurrent, layer 2: dev = MTL0\nllama_memory_recurrent, layer 3: dev = MTL0\nllama_memory_recurrent, layer 4: dev = MTL0\nllama_memory_recurrent, layer 6: dev = MTL0\nllama_memory_recurrent, layer 7: dev = MTL0\nllama_memory_recurrent, layer 8: dev = MTL0\nllama_memory_recurrent, layer 9: dev = MTL0\nllama_memory_recurrent, layer 10: dev = MTL0\nllama_memory_recurrent, layer 11: dev = MTL0\nllama_memory_recurrent, layer 12: dev = MTL0\nllama_memory_recurrent, layer 13: dev = MTL0\nllama_memory_recurrent, layer 14: dev = MTL0\nllama_memory_recurrent, layer 16: dev = MTL0\nllama_memory_recurrent, layer 17: dev = MTL0\nllama_memory_recurrent, layer 18: dev = MTL0\nllama_memory_recurrent, layer 19: dev = MTL0\nllama_memory_recurrent, layer 20: dev = MTL0\nllama_memory_recurrent, layer 21: dev = MTL0\nllama_memory_recurrent, layer 22: dev = MTL0\nllama_memory_recurrent, layer 23: dev = MTL0\nllama_memory_recurrent, layer 24: dev = MTL0\nllama_memory_recurrent, layer 26: dev = MTL0\nllama_memory_recurrent, layer 27: dev = MTL0\nllama_memory_recurrent, layer 28: dev = MTL0\nllama_memory_recurrent, layer 29: dev = MTL0\nllama_memory_recurrent, layer 30: dev = MTL0\nllama_memory_recurrent, layer 31: dev = MTL0\nllama_memory_recurrent, layer 32: dev = MTL0\nllama_memory_recurrent, layer 33: dev = MTL0\nllama_memory_recurrent, layer 34: dev = MTL0\nllama_memory_recurrent, layer 36: dev = MTL0\nllama_memory_recurrent, layer 37: dev = MTL0\nllama_memory_recurrent, layer 38: dev = MTL0\nllama_memory_recurrent, layer 39: dev = MTL0\nllama_memory_recurrent: MTL0 RS buffer size = 147.48 MiB\nsched_reserve: MTL0 compute buffer size = 282.28 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 41,
"totalLayers": 41,
"inferenceMs": 329.7040420002304,
"durationMs": 337.9441669967491,
"tokensPerSecond": 6.066046348314415,
"inputTokens": 63,
"outputTokens": 2,
"outputLimitReached": false,
"rssBytes": 19942162432,
"cpuThreads": 4,
"totalMs": 1615.7337089898647,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "q4",
"device": "mps",
"status": "pass",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "chat"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
"device": "mps",
"precision": "q4",
"provider": "llama.cpp Metal",
"loadMs": 1277.7895419931156,
"acceleration": "partial",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer 0 assigned to device MTL0, is_swa = 0\nload_tensors: layer 1 assigned to device MTL0, is_swa = 0\nload_tensors: layer 2 assigned to device MTL0, is_swa = 0\nload_tensors: layer 3 assigned to device MTL0, is_swa = 0\nload_tensors: layer 4 assigned to device MTL0, is_swa = 0\nload_tensors: layer 5 assigned to device MTL0, is_swa = 0\nload_tensors: layer 6 assigned to device MTL0, is_swa = 0\nload_tensors: layer 7 assigned to device MTL0, is_swa = 0\nload_tensors: layer 8 assigned to device MTL0, is_swa = 0\nload_tensors: layer 9 assigned to device MTL0, is_swa = 0\nload_tensors: layer 10 assigned to device MTL0, is_swa = 0\nload_tensors: layer 11 assigned to device MTL0, is_swa = 0\nload_tensors: layer 12 assigned to device MTL0, is_swa = 0\nload_tensors: layer 13 assigned to device MTL0, is_swa = 0\nload_tensors: layer 14 assigned to device MTL0, is_swa = 0\nload_tensors: layer 15 assigned to device MTL0, is_swa = 0\nload_tensors: layer 16 assigned to device MTL0, is_swa = 0\nload_tensors: layer 17 assigned to device MTL0, is_swa = 0\nload_tensors: layer 18 assigned to device MTL0, is_swa = 0\nload_tensors: layer 19 assigned to device MTL0, is_swa = 0\nload_tensors: layer 20 assigned to device MTL0, is_swa = 0\nload_tensors: layer 21 assigned to device MTL0, is_swa = 0\nload_tensors: layer 22 assigned to device MTL0, is_swa = 0\nload_tensors: layer 23 assigned to device MTL0, is_swa = 0\nload_tensors: layer 24 assigned to device MTL0, is_swa = 0\nload_tensors: layer 25 assigned to device MTL0, is_swa = 0\nload_tensors: layer 26 assigned to device MTL0, is_swa = 0\nload_tensors: layer 27 assigned to device MTL0, is_swa = 0\nload_tensors: layer 28 assigned to device MTL0, is_swa = 0\nload_tensors: layer 29 assigned to device MTL0, is_swa = 0\nload_tensors: layer 30 assigned to device MTL0, is_swa = 0\nload_tensors: layer 31 assigned to device MTL0, is_swa = 0\nload_tensors: layer 32 assigned to device MTL0, is_swa = 0\nload_tensors: layer 33 assigned to device MTL0, is_swa = 0\nload_tensors: layer 34 assigned to device MTL0, is_swa = 0\nload_tensors: layer 35 assigned to device MTL0, is_swa = 0\nload_tensors: layer 36 assigned to device MTL0, is_swa = 0\nload_tensors: layer 37 assigned to device MTL0, is_swa = 0\nload_tensors: layer 38 assigned to device MTL0, is_swa = 0\nload_tensors: layer 39 assigned to device MTL0, is_swa = 0\nload_tensors: layer 40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 321.56 MiB\nload_tensors: MTL0_Mapped model buffer size = 18570.93 MiB\nllama_kv_cache: layer 5: dev = MTL0\nllama_kv_cache: layer 15: dev = MTL0\nllama_kv_cache: layer 25: dev = MTL0\nllama_kv_cache: layer 35: dev = MTL0\nllama_kv_cache: MTL0 KV buffer size = 128.00 MiB\nllama_memory_recurrent, layer 0: dev = MTL0\nllama_memory_recurrent, layer 1: dev = MTL0\nllama_memory_recurrent, layer 2: dev = MTL0\nllama_memory_recurrent, layer 3: dev = MTL0\nllama_memory_recurrent, layer 4: dev = MTL0\nllama_memory_recurrent, layer 6: dev = MTL0\nllama_memory_recurrent, layer 7: dev = MTL0\nllama_memory_recurrent, layer 8: dev = MTL0\nllama_memory_recurrent, layer 9: dev = MTL0\nllama_memory_recurrent, layer 10: dev = MTL0\nllama_memory_recurrent, layer 11: dev = MTL0\nllama_memory_recurrent, layer 12: dev = MTL0\nllama_memory_recurrent, layer 13: dev = MTL0\nllama_memory_recurrent, layer 14: dev = MTL0\nllama_memory_recurrent, layer 16: dev = MTL0\nllama_memory_recurrent, layer 17: dev = MTL0\nllama_memory_recurrent, layer 18: dev = MTL0\nllama_memory_recurrent, layer 19: dev = MTL0\nllama_memory_recurrent, layer 20: dev = MTL0\nllama_memory_recurrent, layer 21: dev = MTL0\nllama_memory_recurrent, layer 22: dev = MTL0\nllama_memory_recurrent, layer 23: dev = MTL0\nllama_memory_recurrent, layer 24: dev = MTL0\nllama_memory_recurrent, layer 26: dev = MTL0\nllama_memory_recurrent, layer 27: dev = MTL0\nllama_memory_recurrent, layer 28: dev = MTL0\nllama_memory_recurrent, layer 29: dev = MTL0\nllama_memory_recurrent, layer 30: dev = MTL0\nllama_memory_recurrent, layer 31: dev = MTL0\nllama_memory_recurrent, layer 32: dev = MTL0\nllama_memory_recurrent, layer 33: dev = MTL0\nllama_memory_recurrent, layer 34: dev = MTL0\nllama_memory_recurrent, layer 36: dev = MTL0\nllama_memory_recurrent, layer 37: dev = MTL0\nllama_memory_recurrent, layer 38: dev = MTL0\nllama_memory_recurrent, layer 39: dev = MTL0\nllama_memory_recurrent: MTL0 RS buffer size = 147.48 MiB\nsched_reserve: MTL0 compute buffer size = 282.28 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 41,
"totalLayers": 41,
"inferenceMs": 366.49279199627927,
"durationMs": 376.4192090020515,
"tokensPerSecond": 5.4571332470306935,
"inputTokens": 68,
"outputTokens": 2,
"outputLimitReached": false,
"rssBytes": 19944161280,
"cpuThreads": 4,
"totalMs": 1654.208750995167,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "q4",
"device": "mps",
"status": "pass",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "hola"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
"device": "mps",
"precision": "q4",
"provider": "llama.cpp Metal",
"loadMs": 1277.7895419931156,
"acceleration": "partial",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer 0 assigned to device MTL0, is_swa = 0\nload_tensors: layer 1 assigned to device MTL0, is_swa = 0\nload_tensors: layer 2 assigned to device MTL0, is_swa = 0\nload_tensors: layer 3 assigned to device MTL0, is_swa = 0\nload_tensors: layer 4 assigned to device MTL0, is_swa = 0\nload_tensors: layer 5 assigned to device MTL0, is_swa = 0\nload_tensors: layer 6 assigned to device MTL0, is_swa = 0\nload_tensors: layer 7 assigned to device MTL0, is_swa = 0\nload_tensors: layer 8 assigned to device MTL0, is_swa = 0\nload_tensors: layer 9 assigned to device MTL0, is_swa = 0\nload_tensors: layer 10 assigned to device MTL0, is_swa = 0\nload_tensors: layer 11 assigned to device MTL0, is_swa = 0\nload_tensors: layer 12 assigned to device MTL0, is_swa = 0\nload_tensors: layer 13 assigned to device MTL0, is_swa = 0\nload_tensors: layer 14 assigned to device MTL0, is_swa = 0\nload_tensors: layer 15 assigned to device MTL0, is_swa = 0\nload_tensors: layer 16 assigned to device MTL0, is_swa = 0\nload_tensors: layer 17 assigned to device MTL0, is_swa = 0\nload_tensors: layer 18 assigned to device MTL0, is_swa = 0\nload_tensors: layer 19 assigned to device MTL0, is_swa = 0\nload_tensors: layer 20 assigned to device MTL0, is_swa = 0\nload_tensors: layer 21 assigned to device MTL0, is_swa = 0\nload_tensors: layer 22 assigned to device MTL0, is_swa = 0\nload_tensors: layer 23 assigned to device MTL0, is_swa = 0\nload_tensors: layer 24 assigned to device MTL0, is_swa = 0\nload_tensors: layer 25 assigned to device MTL0, is_swa = 0\nload_tensors: layer 26 assigned to device MTL0, is_swa = 0\nload_tensors: layer 27 assigned to device MTL0, is_swa = 0\nload_tensors: layer 28 assigned to device MTL0, is_swa = 0\nload_tensors: layer 29 assigned to device MTL0, is_swa = 0\nload_tensors: layer 30 assigned to device MTL0, is_swa = 0\nload_tensors: layer 31 assigned to device MTL0, is_swa = 0\nload_tensors: layer 32 assigned to device MTL0, is_swa = 0\nload_tensors: layer 33 assigned to device MTL0, is_swa = 0\nload_tensors: layer 34 assigned to device MTL0, is_swa = 0\nload_tensors: layer 35 assigned to device MTL0, is_swa = 0\nload_tensors: layer 36 assigned to device MTL0, is_swa = 0\nload_tensors: layer 37 assigned to device MTL0, is_swa = 0\nload_tensors: layer 38 assigned to device MTL0, is_swa = 0\nload_tensors: layer 39 assigned to device MTL0, is_swa = 0\nload_tensors: layer 40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 321.56 MiB\nload_tensors: MTL0_Mapped model buffer size = 18570.93 MiB\nllama_kv_cache: layer 5: dev = MTL0\nllama_kv_cache: layer 15: dev = MTL0\nllama_kv_cache: layer 25: dev = MTL0\nllama_kv_cache: layer 35: dev = MTL0\nllama_kv_cache: MTL0 KV buffer size = 128.00 MiB\nllama_memory_recurrent, layer 0: dev = MTL0\nllama_memory_recurrent, layer 1: dev = MTL0\nllama_memory_recurrent, layer 2: dev = MTL0\nllama_memory_recurrent, layer 3: dev = MTL0\nllama_memory_recurrent, layer 4: dev = MTL0\nllama_memory_recurrent, layer 6: dev = MTL0\nllama_memory_recurrent, layer 7: dev = MTL0\nllama_memory_recurrent, layer 8: dev = MTL0\nllama_memory_recurrent, layer 9: dev = MTL0\nllama_memory_recurrent, layer 10: dev = MTL0\nllama_memory_recurrent, layer 11: dev = MTL0\nllama_memory_recurrent, layer 12: dev = MTL0\nllama_memory_recurrent, layer 13: dev = MTL0\nllama_memory_recurrent, layer 14: dev = MTL0\nllama_memory_recurrent, layer 16: dev = MTL0\nllama_memory_recurrent, layer 17: dev = MTL0\nllama_memory_recurrent, layer 18: dev = MTL0\nllama_memory_recurrent, layer 19: dev = MTL0\nllama_memory_recurrent, layer 20: dev = MTL0\nllama_memory_recurrent, layer 21: dev = MTL0\nllama_memory_recurrent, layer 22: dev = MTL0\nllama_memory_recurrent, layer 23: dev = MTL0\nllama_memory_recurrent, layer 24: dev = MTL0\nllama_memory_recurrent, layer 26: dev = MTL0\nllama_memory_recurrent, layer 27: dev = MTL0\nllama_memory_recurrent, layer 28: dev = MTL0\nllama_memory_recurrent, layer 29: dev = MTL0\nllama_memory_recurrent, layer 30: dev = MTL0\nllama_memory_recurrent, layer 31: dev = MTL0\nllama_memory_recurrent, layer 32: dev = MTL0\nllama_memory_recurrent, layer 33: dev = MTL0\nllama_memory_recurrent, layer 34: dev = MTL0\nllama_memory_recurrent, layer 36: dev = MTL0\nllama_memory_recurrent, layer 37: dev = MTL0\nllama_memory_recurrent, layer 38: dev = MTL0\nllama_memory_recurrent, layer 39: dev = MTL0\nllama_memory_recurrent: MTL0 RS buffer size = 147.48 MiB\nsched_reserve: MTL0 compute buffer size = 282.28 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 41,
"totalLayers": 41,
"inferenceMs": 403.24220799811883,
"durationMs": 411.4744590042392,
"tokensPerSecond": 7.439697384094265,
"inputTokens": 69,
"outputTokens": 3,
"outputLimitReached": false,
"rssBytes": 19945586688,
"cpuThreads": 4,
"totalMs": 1689.2640009973547,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "q4",
"device": "mps",
"status": "pass",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "2026-11-04"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
"device": "mps",
"precision": "q4",
"provider": "llama.cpp Metal",
"loadMs": 1277.7895419931156,
"acceleration": "partial",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer 0 assigned to device MTL0, is_swa = 0\nload_tensors: layer 1 assigned to device MTL0, is_swa = 0\nload_tensors: layer 2 assigned to device MTL0, is_swa = 0\nload_tensors: layer 3 assigned to device MTL0, is_swa = 0\nload_tensors: layer 4 assigned to device MTL0, is_swa = 0\nload_tensors: layer 5 assigned to device MTL0, is_swa = 0\nload_tensors: layer 6 assigned to device MTL0, is_swa = 0\nload_tensors: layer 7 assigned to device MTL0, is_swa = 0\nload_tensors: layer 8 assigned to device MTL0, is_swa = 0\nload_tensors: layer 9 assigned to device MTL0, is_swa = 0\nload_tensors: layer 10 assigned to device MTL0, is_swa = 0\nload_tensors: layer 11 assigned to device MTL0, is_swa = 0\nload_tensors: layer 12 assigned to device MTL0, is_swa = 0\nload_tensors: layer 13 assigned to device MTL0, is_swa = 0\nload_tensors: layer 14 assigned to device MTL0, is_swa = 0\nload_tensors: layer 15 assigned to device MTL0, is_swa = 0\nload_tensors: layer 16 assigned to device MTL0, is_swa = 0\nload_tensors: layer 17 assigned to device MTL0, is_swa = 0\nload_tensors: layer 18 assigned to device MTL0, is_swa = 0\nload_tensors: layer 19 assigned to device MTL0, is_swa = 0\nload_tensors: layer 20 assigned to device MTL0, is_swa = 0\nload_tensors: layer 21 assigned to device MTL0, is_swa = 0\nload_tensors: layer 22 assigned to device MTL0, is_swa = 0\nload_tensors: layer 23 assigned to device MTL0, is_swa = 0\nload_tensors: layer 24 assigned to device MTL0, is_swa = 0\nload_tensors: layer 25 assigned to device MTL0, is_swa = 0\nload_tensors: layer 26 assigned to device MTL0, is_swa = 0\nload_tensors: layer 27 assigned to device MTL0, is_swa = 0\nload_tensors: layer 28 assigned to device MTL0, is_swa = 0\nload_tensors: layer 29 assigned to device MTL0, is_swa = 0\nload_tensors: layer 30 assigned to device MTL0, is_swa = 0\nload_tensors: layer 31 assigned to device MTL0, is_swa = 0\nload_tensors: layer 32 assigned to device MTL0, is_swa = 0\nload_tensors: layer 33 assigned to device MTL0, is_swa = 0\nload_tensors: layer 34 assigned to device MTL0, is_swa = 0\nload_tensors: layer 35 assigned to device MTL0, is_swa = 0\nload_tensors: layer 36 assigned to device MTL0, is_swa = 0\nload_tensors: layer 37 assigned to device MTL0, is_swa = 0\nload_tensors: layer 38 assigned to device MTL0, is_swa = 0\nload_tensors: layer 39 assigned to device MTL0, is_swa = 0\nload_tensors: layer 40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 321.56 MiB\nload_tensors: MTL0_Mapped model buffer size = 18570.93 MiB\nllama_kv_cache: layer 5: dev = MTL0\nllama_kv_cache: layer 15: dev = MTL0\nllama_kv_cache: layer 25: dev = MTL0\nllama_kv_cache: layer 35: dev = MTL0\nllama_kv_cache: MTL0 KV buffer size = 128.00 MiB\nllama_memory_recurrent, layer 0: dev = MTL0\nllama_memory_recurrent, layer 1: dev = MTL0\nllama_memory_recurrent, layer 2: dev = MTL0\nllama_memory_recurrent, layer 3: dev = MTL0\nllama_memory_recurrent, layer 4: dev = MTL0\nllama_memory_recurrent, layer 6: dev = MTL0\nllama_memory_recurrent, layer 7: dev = MTL0\nllama_memory_recurrent, layer 8: dev = MTL0\nllama_memory_recurrent, layer 9: dev = MTL0\nllama_memory_recurrent, layer 10: dev = MTL0\nllama_memory_recurrent, layer 11: dev = MTL0\nllama_memory_recurrent, layer 12: dev = MTL0\nllama_memory_recurrent, layer 13: dev = MTL0\nllama_memory_recurrent, layer 14: dev = MTL0\nllama_memory_recurrent, layer 16: dev = MTL0\nllama_memory_recurrent, layer 17: dev = MTL0\nllama_memory_recurrent, layer 18: dev = MTL0\nllama_memory_recurrent, layer 19: dev = MTL0\nllama_memory_recurrent, layer 20: dev = MTL0\nllama_memory_recurrent, layer 21: dev = MTL0\nllama_memory_recurrent, layer 22: dev = MTL0\nllama_memory_recurrent, layer 23: dev = MTL0\nllama_memory_recurrent, layer 24: dev = MTL0\nllama_memory_recurrent, layer 26: dev = MTL0\nllama_memory_recurrent, layer 27: dev = MTL0\nllama_memory_recurrent, layer 28: dev = MTL0\nllama_memory_recurrent, layer 29: dev = MTL0\nllama_memory_recurrent, layer 30: dev = MTL0\nllama_memory_recurrent, layer 31: dev = MTL0\nllama_memory_recurrent, layer 32: dev = MTL0\nllama_memory_recurrent, layer 33: dev = MTL0\nllama_memory_recurrent, layer 34: dev = MTL0\nllama_memory_recurrent, layer 36: dev = MTL0\nllama_memory_recurrent, layer 37: dev = MTL0\nllama_memory_recurrent, layer 38: dev = MTL0\nllama_memory_recurrent, layer 39: dev = MTL0\nllama_memory_recurrent: MTL0 RS buffer size = 147.48 MiB\nsched_reserve: MTL0 compute buffer size = 282.28 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 41,
"totalLayers": 41,
"inferenceMs": 557.7706249969197,
"durationMs": 566.5681669997866,
"tokensPerSecond": 12.549961733891344,
"inputTokens": 72,
"outputTokens": 7,
"outputLimitReached": false,
"rssBytes": 19945783296,
"cpuThreads": 4,
"totalMs": 1844.3577089929022,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "q4",
"device": "mps",
"status": "pass",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "Unknown"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
"device": "mps",
"precision": "q4",
"provider": "llama.cpp Metal",
"loadMs": 1277.7895419931156,
"acceleration": "partial",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer 0 assigned to device MTL0, is_swa = 0\nload_tensors: layer 1 assigned to device MTL0, is_swa = 0\nload_tensors: layer 2 assigned to device MTL0, is_swa = 0\nload_tensors: layer 3 assigned to device MTL0, is_swa = 0\nload_tensors: layer 4 assigned to device MTL0, is_swa = 0\nload_tensors: layer 5 assigned to device MTL0, is_swa = 0\nload_tensors: layer 6 assigned to device MTL0, is_swa = 0\nload_tensors: layer 7 assigned to device MTL0, is_swa = 0\nload_tensors: layer 8 assigned to device MTL0, is_swa = 0\nload_tensors: layer 9 assigned to device MTL0, is_swa = 0\nload_tensors: layer 10 assigned to device MTL0, is_swa = 0\nload_tensors: layer 11 assigned to device MTL0, is_swa = 0\nload_tensors: layer 12 assigned to device MTL0, is_swa = 0\nload_tensors: layer 13 assigned to device MTL0, is_swa = 0\nload_tensors: layer 14 assigned to device MTL0, is_swa = 0\nload_tensors: layer 15 assigned to device MTL0, is_swa = 0\nload_tensors: layer 16 assigned to device MTL0, is_swa = 0\nload_tensors: layer 17 assigned to device MTL0, is_swa = 0\nload_tensors: layer 18 assigned to device MTL0, is_swa = 0\nload_tensors: layer 19 assigned to device MTL0, is_swa = 0\nload_tensors: layer 20 assigned to device MTL0, is_swa = 0\nload_tensors: layer 21 assigned to device MTL0, is_swa = 0\nload_tensors: layer 22 assigned to device MTL0, is_swa = 0\nload_tensors: layer 23 assigned to device MTL0, is_swa = 0\nload_tensors: layer 24 assigned to device MTL0, is_swa = 0\nload_tensors: layer 25 assigned to device MTL0, is_swa = 0\nload_tensors: layer 26 assigned to device MTL0, is_swa = 0\nload_tensors: layer 27 assigned to device MTL0, is_swa = 0\nload_tensors: layer 28 assigned to device MTL0, is_swa = 0\nload_tensors: layer 29 assigned to device MTL0, is_swa = 0\nload_tensors: layer 30 assigned to device MTL0, is_swa = 0\nload_tensors: layer 31 assigned to device MTL0, is_swa = 0\nload_tensors: layer 32 assigned to device MTL0, is_swa = 0\nload_tensors: layer 33 assigned to device MTL0, is_swa = 0\nload_tensors: layer 34 assigned to device MTL0, is_swa = 0\nload_tensors: layer 35 assigned to device MTL0, is_swa = 0\nload_tensors: layer 36 assigned to device MTL0, is_swa = 0\nload_tensors: layer 37 assigned to device MTL0, is_swa = 0\nload_tensors: layer 38 assigned to device MTL0, is_swa = 0\nload_tensors: layer 39 assigned to device MTL0, is_swa = 0\nload_tensors: layer 40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 321.56 MiB\nload_tensors: MTL0_Mapped model buffer size = 18570.93 MiB\nllama_kv_cache: layer 5: dev = MTL0\nllama_kv_cache: layer 15: dev = MTL0\nllama_kv_cache: layer 25: dev = MTL0\nllama_kv_cache: layer 35: dev = MTL0\nllama_kv_cache: MTL0 KV buffer size = 128.00 MiB\nllama_memory_recurrent, layer 0: dev = MTL0\nllama_memory_recurrent, layer 1: dev = MTL0\nllama_memory_recurrent, layer 2: dev = MTL0\nllama_memory_recurrent, layer 3: dev = MTL0\nllama_memory_recurrent, layer 4: dev = MTL0\nllama_memory_recurrent, layer 6: dev = MTL0\nllama_memory_recurrent, layer 7: dev = MTL0\nllama_memory_recurrent, layer 8: dev = MTL0\nllama_memory_recurrent, layer 9: dev = MTL0\nllama_memory_recurrent, layer 10: dev = MTL0\nllama_memory_recurrent, layer 11: dev = MTL0\nllama_memory_recurrent, layer 12: dev = MTL0\nllama_memory_recurrent, layer 13: dev = MTL0\nllama_memory_recurrent, layer 14: dev = MTL0\nllama_memory_recurrent, layer 16: dev = MTL0\nllama_memory_recurrent, layer 17: dev = MTL0\nllama_memory_recurrent, layer 18: dev = MTL0\nllama_memory_recurrent, layer 19: dev = MTL0\nllama_memory_recurrent, layer 20: dev = MTL0\nllama_memory_recurrent, layer 21: dev = MTL0\nllama_memory_recurrent, layer 22: dev = MTL0\nllama_memory_recurrent, layer 23: dev = MTL0\nllama_memory_recurrent, layer 24: dev = MTL0\nllama_memory_recurrent, layer 26: dev = MTL0\nllama_memory_recurrent, layer 27: dev = MTL0\nllama_memory_recurrent, layer 28: dev = MTL0\nllama_memory_recurrent, layer 29: dev = MTL0\nllama_memory_recurrent, layer 30: dev = MTL0\nllama_memory_recurrent, layer 31: dev = MTL0\nllama_memory_recurrent, layer 32: dev = MTL0\nllama_memory_recurrent, layer 33: dev = MTL0\nllama_memory_recurrent, layer 34: dev = MTL0\nllama_memory_recurrent, layer 36: dev = MTL0\nllama_memory_recurrent, layer 37: dev = MTL0\nllama_memory_recurrent, layer 38: dev = MTL0\nllama_memory_recurrent, layer 39: dev = MTL0\nllama_memory_recurrent: MTL0 RS buffer size = 147.48 MiB\nsched_reserve: MTL0 compute buffer size = 282.28 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 41,
"totalLayers": 41,
"inferenceMs": 385.34866699774284,
"durationMs": 393.4925000066869,
"tokensPerSecond": 5.190104887560737,
"inputTokens": 74,
"outputTokens": 2,
"outputLimitReached": false,
"rssBytes": 19945848832,
"cpuThreads": 4,
"totalMs": 1671.2820419998025,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "q4",
"device": "mps",
"status": "pass",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "Paris"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
"device": "mps",
"precision": "q4",
"provider": "llama.cpp Metal",
"loadMs": 1374.4882920000236,
"acceleration": "partial",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer 0 assigned to device MTL0, is_swa = 0\nload_tensors: layer 1 assigned to device MTL0, is_swa = 0\nload_tensors: layer 2 assigned to device MTL0, is_swa = 0\nload_tensors: layer 3 assigned to device MTL0, is_swa = 0\nload_tensors: layer 4 assigned to device MTL0, is_swa = 0\nload_tensors: layer 5 assigned to device MTL0, is_swa = 0\nload_tensors: layer 6 assigned to device MTL0, is_swa = 0\nload_tensors: layer 7 assigned to device MTL0, is_swa = 0\nload_tensors: layer 8 assigned to device MTL0, is_swa = 0\nload_tensors: layer 9 assigned to device MTL0, is_swa = 0\nload_tensors: layer 10 assigned to device MTL0, is_swa = 0\nload_tensors: layer 11 assigned to device MTL0, is_swa = 0\nload_tensors: layer 12 assigned to device MTL0, is_swa = 0\nload_tensors: layer 13 assigned to device MTL0, is_swa = 0\nload_tensors: layer 14 assigned to device MTL0, is_swa = 0\nload_tensors: layer 15 assigned to device MTL0, is_swa = 0\nload_tensors: layer 16 assigned to device MTL0, is_swa = 0\nload_tensors: layer 17 assigned to device MTL0, is_swa = 0\nload_tensors: layer 18 assigned to device MTL0, is_swa = 0\nload_tensors: layer 19 assigned to device MTL0, is_swa = 0\nload_tensors: layer 20 assigned to device MTL0, is_swa = 0\nload_tensors: layer 21 assigned to device MTL0, is_swa = 0\nload_tensors: layer 22 assigned to device MTL0, is_swa = 0\nload_tensors: layer 23 assigned to device MTL0, is_swa = 0\nload_tensors: layer 24 assigned to device MTL0, is_swa = 0\nload_tensors: layer 25 assigned to device MTL0, is_swa = 0\nload_tensors: layer 26 assigned to device MTL0, is_swa = 0\nload_tensors: layer 27 assigned to device MTL0, is_swa = 0\nload_tensors: layer 28 assigned to device MTL0, is_swa = 0\nload_tensors: layer 29 assigned to device MTL0, is_swa = 0\nload_tensors: layer 30 assigned to device MTL0, is_swa = 0\nload_tensors: layer 31 assigned to device MTL0, is_swa = 0\nload_tensors: layer 32 assigned to device MTL0, is_swa = 0\nload_tensors: layer 33 assigned to device MTL0, is_swa = 0\nload_tensors: layer 34 assigned to device MTL0, is_swa = 0\nload_tensors: layer 35 assigned to device MTL0, is_swa = 0\nload_tensors: layer 36 assigned to device MTL0, is_swa = 0\nload_tensors: layer 37 assigned to device MTL0, is_swa = 0\nload_tensors: layer 38 assigned to device MTL0, is_swa = 0\nload_tensors: layer 39 assigned to device MTL0, is_swa = 0\nload_tensors: layer 40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 321.56 MiB\nload_tensors: MTL0_Mapped model buffer size = 18570.93 MiB\nllama_kv_cache: layer 5: dev = MTL0\nllama_kv_cache: layer 15: dev = MTL0\nllama_kv_cache: layer 25: dev = MTL0\nllama_kv_cache: layer 35: dev = MTL0\nllama_kv_cache: MTL0 KV buffer size = 128.00 MiB\nllama_memory_recurrent, layer 0: dev = MTL0\nllama_memory_recurrent, layer 1: dev = MTL0\nllama_memory_recurrent, layer 2: dev = MTL0\nllama_memory_recurrent, layer 3: dev = MTL0\nllama_memory_recurrent, layer 4: dev = MTL0\nllama_memory_recurrent, layer 6: dev = MTL0\nllama_memory_recurrent, layer 7: dev = MTL0\nllama_memory_recurrent, layer 8: dev = MTL0\nllama_memory_recurrent, layer 9: dev = MTL0\nllama_memory_recurrent, layer 10: dev = MTL0\nllama_memory_recurrent, layer 11: dev = MTL0\nllama_memory_recurrent, layer 12: dev = MTL0\nllama_memory_recurrent, layer 13: dev = MTL0\nllama_memory_recurrent, layer 14: dev = MTL0\nllama_memory_recurrent, layer 16: dev = MTL0\nllama_memory_recurrent, layer 17: dev = MTL0\nllama_memory_recurrent, layer 18: dev = MTL0\nllama_memory_recurrent, layer 19: dev = MTL0\nllama_memory_recurrent, layer 20: dev = MTL0\nllama_memory_recurrent, layer 21: dev = MTL0\nllama_memory_recurrent, layer 22: dev = MTL0\nllama_memory_recurrent, layer 23: dev = MTL0\nllama_memory_recurrent, layer 24: dev = MTL0\nllama_memory_recurrent, layer 26: dev = MTL0\nllama_memory_recurrent, layer 27: dev = MTL0\nllama_memory_recurrent, layer 28: dev = MTL0\nllama_memory_recurrent, layer 29: dev = MTL0\nllama_memory_recurrent, layer 30: dev = MTL0\nllama_memory_recurrent, layer 31: dev = MTL0\nllama_memory_recurrent, layer 32: dev = MTL0\nllama_memory_recurrent, layer 33: dev = MTL0\nllama_memory_recurrent, layer 34: dev = MTL0\nllama_memory_recurrent, layer 36: dev = MTL0\nllama_memory_recurrent, layer 37: dev = MTL0\nllama_memory_recurrent, layer 38: dev = MTL0\nllama_memory_recurrent, layer 39: dev = MTL0\nllama_memory_recurrent: MTL0 RS buffer size = 147.48 MiB\nsched_reserve: MTL0 compute buffer size = 282.28 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 41,
"totalLayers": 41,
"inferenceMs": 357.9981660004705,
"durationMs": 367.01845800416777,
"tokensPerSecond": 5.586620798491385,
"inputTokens": 63,
"outputTokens": 2,
"outputLimitReached": false,
"rssBytes": 19943505920,
"cpuThreads": 4,
"totalMs": 1741.5067500041914,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "q4",
"device": "mps",
"status": "pass",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "56"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
"device": "mps",
"precision": "q4",
"provider": "llama.cpp Metal",
"loadMs": 1374.4882920000236,
"acceleration": "partial",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer 0 assigned to device MTL0, is_swa = 0\nload_tensors: layer 1 assigned to device MTL0, is_swa = 0\nload_tensors: layer 2 assigned to device MTL0, is_swa = 0\nload_tensors: layer 3 assigned to device MTL0, is_swa = 0\nload_tensors: layer 4 assigned to device MTL0, is_swa = 0\nload_tensors: layer 5 assigned to device MTL0, is_swa = 0\nload_tensors: layer 6 assigned to device MTL0, is_swa = 0\nload_tensors: layer 7 assigned to device MTL0, is_swa = 0\nload_tensors: layer 8 assigned to device MTL0, is_swa = 0\nload_tensors: layer 9 assigned to device MTL0, is_swa = 0\nload_tensors: layer 10 assigned to device MTL0, is_swa = 0\nload_tensors: layer 11 assigned to device MTL0, is_swa = 0\nload_tensors: layer 12 assigned to device MTL0, is_swa = 0\nload_tensors: layer 13 assigned to device MTL0, is_swa = 0\nload_tensors: layer 14 assigned to device MTL0, is_swa = 0\nload_tensors: layer 15 assigned to device MTL0, is_swa = 0\nload_tensors: layer 16 assigned to device MTL0, is_swa = 0\nload_tensors: layer 17 assigned to device MTL0, is_swa = 0\nload_tensors: layer 18 assigned to device MTL0, is_swa = 0\nload_tensors: layer 19 assigned to device MTL0, is_swa = 0\nload_tensors: layer 20 assigned to device MTL0, is_swa = 0\nload_tensors: layer 21 assigned to device MTL0, is_swa = 0\nload_tensors: layer 22 assigned to device MTL0, is_swa = 0\nload_tensors: layer 23 assigned to device MTL0, is_swa = 0\nload_tensors: layer 24 assigned to device MTL0, is_swa = 0\nload_tensors: layer 25 assigned to device MTL0, is_swa = 0\nload_tensors: layer 26 assigned to device MTL0, is_swa = 0\nload_tensors: layer 27 assigned to device MTL0, is_swa = 0\nload_tensors: layer 28 assigned to device MTL0, is_swa = 0\nload_tensors: layer 29 assigned to device MTL0, is_swa = 0\nload_tensors: layer 30 assigned to device MTL0, is_swa = 0\nload_tensors: layer 31 assigned to device MTL0, is_swa = 0\nload_tensors: layer 32 assigned to device MTL0, is_swa = 0\nload_tensors: layer 33 assigned to device MTL0, is_swa = 0\nload_tensors: layer 34 assigned to device MTL0, is_swa = 0\nload_tensors: layer 35 assigned to device MTL0, is_swa = 0\nload_tensors: layer 36 assigned to device MTL0, is_swa = 0\nload_tensors: layer 37 assigned to device MTL0, is_swa = 0\nload_tensors: layer 38 assigned to device MTL0, is_swa = 0\nload_tensors: layer 39 assigned to device MTL0, is_swa = 0\nload_tensors: layer 40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 321.56 MiB\nload_tensors: MTL0_Mapped model buffer size = 18570.93 MiB\nllama_kv_cache: layer 5: dev = MTL0\nllama_kv_cache: layer 15: dev = MTL0\nllama_kv_cache: layer 25: dev = MTL0\nllama_kv_cache: layer 35: dev = MTL0\nllama_kv_cache: MTL0 KV buffer size = 128.00 MiB\nllama_memory_recurrent, layer 0: dev = MTL0\nllama_memory_recurrent, layer 1: dev = MTL0\nllama_memory_recurrent, layer 2: dev = MTL0\nllama_memory_recurrent, layer 3: dev = MTL0\nllama_memory_recurrent, layer 4: dev = MTL0\nllama_memory_recurrent, layer 6: dev = MTL0\nllama_memory_recurrent, layer 7: dev = MTL0\nllama_memory_recurrent, layer 8: dev = MTL0\nllama_memory_recurrent, layer 9: dev = MTL0\nllama_memory_recurrent, layer 10: dev = MTL0\nllama_memory_recurrent, layer 11: dev = MTL0\nllama_memory_recurrent, layer 12: dev = MTL0\nllama_memory_recurrent, layer 13: dev = MTL0\nllama_memory_recurrent, layer 14: dev = MTL0\nllama_memory_recurrent, layer 16: dev = MTL0\nllama_memory_recurrent, layer 17: dev = MTL0\nllama_memory_recurrent, layer 18: dev = MTL0\nllama_memory_recurrent, layer 19: dev = MTL0\nllama_memory_recurrent, layer 20: dev = MTL0\nllama_memory_recurrent, layer 21: dev = MTL0\nllama_memory_recurrent, layer 22: dev = MTL0\nllama_memory_recurrent, layer 23: dev = MTL0\nllama_memory_recurrent, layer 24: dev = MTL0\nllama_memory_recurrent, layer 26: dev = MTL0\nllama_memory_recurrent, layer 27: dev = MTL0\nllama_memory_recurrent, layer 28: dev = MTL0\nllama_memory_recurrent, layer 29: dev = MTL0\nllama_memory_recurrent, layer 30: dev = MTL0\nllama_memory_recurrent, layer 31: dev = MTL0\nllama_memory_recurrent, layer 32: dev = MTL0\nllama_memory_recurrent, layer 33: dev = MTL0\nllama_memory_recurrent, layer 34: dev = MTL0\nllama_memory_recurrent, layer 36: dev = MTL0\nllama_memory_recurrent, layer 37: dev = MTL0\nllama_memory_recurrent, layer 38: dev = MTL0\nllama_memory_recurrent, layer 39: dev = MTL0\nllama_memory_recurrent: MTL0 RS buffer size = 147.48 MiB\nsched_reserve: MTL0 compute buffer size = 282.28 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 41,
"totalLayers": 41,
"inferenceMs": 333.6505830011447,
"durationMs": 342.3435000004247,
"tokensPerSecond": 5.994294935768592,
"inputTokens": 63,
"outputTokens": 2,
"outputLimitReached": false,
"rssBytes": 19944849408,
"cpuThreads": 4,
"totalMs": 1716.8317920004483,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "q4",
"device": "mps",
"status": "pass",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "chat"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
"device": "mps",
"precision": "q4",
"provider": "llama.cpp Metal",
"loadMs": 1374.4882920000236,
"acceleration": "partial",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer 0 assigned to device MTL0, is_swa = 0\nload_tensors: layer 1 assigned to device MTL0, is_swa = 0\nload_tensors: layer 2 assigned to device MTL0, is_swa = 0\nload_tensors: layer 3 assigned to device MTL0, is_swa = 0\nload_tensors: layer 4 assigned to device MTL0, is_swa = 0\nload_tensors: layer 5 assigned to device MTL0, is_swa = 0\nload_tensors: layer 6 assigned to device MTL0, is_swa = 0\nload_tensors: layer 7 assigned to device MTL0, is_swa = 0\nload_tensors: layer 8 assigned to device MTL0, is_swa = 0\nload_tensors: layer 9 assigned to device MTL0, is_swa = 0\nload_tensors: layer 10 assigned to device MTL0, is_swa = 0\nload_tensors: layer 11 assigned to device MTL0, is_swa = 0\nload_tensors: layer 12 assigned to device MTL0, is_swa = 0\nload_tensors: layer 13 assigned to device MTL0, is_swa = 0\nload_tensors: layer 14 assigned to device MTL0, is_swa = 0\nload_tensors: layer 15 assigned to device MTL0, is_swa = 0\nload_tensors: layer 16 assigned to device MTL0, is_swa = 0\nload_tensors: layer 17 assigned to device MTL0, is_swa = 0\nload_tensors: layer 18 assigned to device MTL0, is_swa = 0\nload_tensors: layer 19 assigned to device MTL0, is_swa = 0\nload_tensors: layer 20 assigned to device MTL0, is_swa = 0\nload_tensors: layer 21 assigned to device MTL0, is_swa = 0\nload_tensors: layer 22 assigned to device MTL0, is_swa = 0\nload_tensors: layer 23 assigned to device MTL0, is_swa = 0\nload_tensors: layer 24 assigned to device MTL0, is_swa = 0\nload_tensors: layer 25 assigned to device MTL0, is_swa = 0\nload_tensors: layer 26 assigned to device MTL0, is_swa = 0\nload_tensors: layer 27 assigned to device MTL0, is_swa = 0\nload_tensors: layer 28 assigned to device MTL0, is_swa = 0\nload_tensors: layer 29 assigned to device MTL0, is_swa = 0\nload_tensors: layer 30 assigned to device MTL0, is_swa = 0\nload_tensors: layer 31 assigned to device MTL0, is_swa = 0\nload_tensors: layer 32 assigned to device MTL0, is_swa = 0\nload_tensors: layer 33 assigned to device MTL0, is_swa = 0\nload_tensors: layer 34 assigned to device MTL0, is_swa = 0\nload_tensors: layer 35 assigned to device MTL0, is_swa = 0\nload_tensors: layer 36 assigned to device MTL0, is_swa = 0\nload_tensors: layer 37 assigned to device MTL0, is_swa = 0\nload_tensors: layer 38 assigned to device MTL0, is_swa = 0\nload_tensors: layer 39 assigned to device MTL0, is_swa = 0\nload_tensors: layer 40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 321.56 MiB\nload_tensors: MTL0_Mapped model buffer size = 18570.93 MiB\nllama_kv_cache: layer 5: dev = MTL0\nllama_kv_cache: layer 15: dev = MTL0\nllama_kv_cache: layer 25: dev = MTL0\nllama_kv_cache: layer 35: dev = MTL0\nllama_kv_cache: MTL0 KV buffer size = 128.00 MiB\nllama_memory_recurrent, layer 0: dev = MTL0\nllama_memory_recurrent, layer 1: dev = MTL0\nllama_memory_recurrent, layer 2: dev = MTL0\nllama_memory_recurrent, layer 3: dev = MTL0\nllama_memory_recurrent, layer 4: dev = MTL0\nllama_memory_recurrent, layer 6: dev = MTL0\nllama_memory_recurrent, layer 7: dev = MTL0\nllama_memory_recurrent, layer 8: dev = MTL0\nllama_memory_recurrent, layer 9: dev = MTL0\nllama_memory_recurrent, layer 10: dev = MTL0\nllama_memory_recurrent, layer 11: dev = MTL0\nllama_memory_recurrent, layer 12: dev = MTL0\nllama_memory_recurrent, layer 13: dev = MTL0\nllama_memory_recurrent, layer 14: dev = MTL0\nllama_memory_recurrent, layer 16: dev = MTL0\nllama_memory_recurrent, layer 17: dev = MTL0\nllama_memory_recurrent, layer 18: dev = MTL0\nllama_memory_recurrent, layer 19: dev = MTL0\nllama_memory_recurrent, layer 20: dev = MTL0\nllama_memory_recurrent, layer 21: dev = MTL0\nllama_memory_recurrent, layer 22: dev = MTL0\nllama_memory_recurrent, layer 23: dev = MTL0\nllama_memory_recurrent, layer 24: dev = MTL0\nllama_memory_recurrent, layer 26: dev = MTL0\nllama_memory_recurrent, layer 27: dev = MTL0\nllama_memory_recurrent, layer 28: dev = MTL0\nllama_memory_recurrent, layer 29: dev = MTL0\nllama_memory_recurrent, layer 30: dev = MTL0\nllama_memory_recurrent, layer 31: dev = MTL0\nllama_memory_recurrent, layer 32: dev = MTL0\nllama_memory_recurrent, layer 33: dev = MTL0\nllama_memory_recurrent, layer 34: dev = MTL0\nllama_memory_recurrent, layer 36: dev = MTL0\nllama_memory_recurrent, layer 37: dev = MTL0\nllama_memory_recurrent, layer 38: dev = MTL0\nllama_memory_recurrent, layer 39: dev = MTL0\nllama_memory_recurrent: MTL0 RS buffer size = 147.48 MiB\nsched_reserve: MTL0 compute buffer size = 282.28 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 41,
"totalLayers": 41,
"inferenceMs": 365.4482080019079,
"durationMs": 374.9816249983269,
"tokensPerSecond": 5.472731720139009,
"inputTokens": 68,
"outputTokens": 2,
"outputLimitReached": false,
"rssBytes": 19946733568,
"cpuThreads": 4,
"totalMs": 1749.4699169983505,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "q4",
"device": "mps",
"status": "pass",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "hola"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
"device": "mps",
"precision": "q4",
"provider": "llama.cpp Metal",
"loadMs": 1374.4882920000236,
"acceleration": "partial",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer 0 assigned to device MTL0, is_swa = 0\nload_tensors: layer 1 assigned to device MTL0, is_swa = 0\nload_tensors: layer 2 assigned to device MTL0, is_swa = 0\nload_tensors: layer 3 assigned to device MTL0, is_swa = 0\nload_tensors: layer 4 assigned to device MTL0, is_swa = 0\nload_tensors: layer 5 assigned to device MTL0, is_swa = 0\nload_tensors: layer 6 assigned to device MTL0, is_swa = 0\nload_tensors: layer 7 assigned to device MTL0, is_swa = 0\nload_tensors: layer 8 assigned to device MTL0, is_swa = 0\nload_tensors: layer 9 assigned to device MTL0, is_swa = 0\nload_tensors: layer 10 assigned to device MTL0, is_swa = 0\nload_tensors: layer 11 assigned to device MTL0, is_swa = 0\nload_tensors: layer 12 assigned to device MTL0, is_swa = 0\nload_tensors: layer 13 assigned to device MTL0, is_swa = 0\nload_tensors: layer 14 assigned to device MTL0, is_swa = 0\nload_tensors: layer 15 assigned to device MTL0, is_swa = 0\nload_tensors: layer 16 assigned to device MTL0, is_swa = 0\nload_tensors: layer 17 assigned to device MTL0, is_swa = 0\nload_tensors: layer 18 assigned to device MTL0, is_swa = 0\nload_tensors: layer 19 assigned to device MTL0, is_swa = 0\nload_tensors: layer 20 assigned to device MTL0, is_swa = 0\nload_tensors: layer 21 assigned to device MTL0, is_swa = 0\nload_tensors: layer 22 assigned to device MTL0, is_swa = 0\nload_tensors: layer 23 assigned to device MTL0, is_swa = 0\nload_tensors: layer 24 assigned to device MTL0, is_swa = 0\nload_tensors: layer 25 assigned to device MTL0, is_swa = 0\nload_tensors: layer 26 assigned to device MTL0, is_swa = 0\nload_tensors: layer 27 assigned to device MTL0, is_swa = 0\nload_tensors: layer 28 assigned to device MTL0, is_swa = 0\nload_tensors: layer 29 assigned to device MTL0, is_swa = 0\nload_tensors: layer 30 assigned to device MTL0, is_swa = 0\nload_tensors: layer 31 assigned to device MTL0, is_swa = 0\nload_tensors: layer 32 assigned to device MTL0, is_swa = 0\nload_tensors: layer 33 assigned to device MTL0, is_swa = 0\nload_tensors: layer 34 assigned to device MTL0, is_swa = 0\nload_tensors: layer 35 assigned to device MTL0, is_swa = 0\nload_tensors: layer 36 assigned to device MTL0, is_swa = 0\nload_tensors: layer 37 assigned to device MTL0, is_swa = 0\nload_tensors: layer 38 assigned to device MTL0, is_swa = 0\nload_tensors: layer 39 assigned to device MTL0, is_swa = 0\nload_tensors: layer 40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 321.56 MiB\nload_tensors: MTL0_Mapped model buffer size = 18570.93 MiB\nllama_kv_cache: layer 5: dev = MTL0\nllama_kv_cache: layer 15: dev = MTL0\nllama_kv_cache: layer 25: dev = MTL0\nllama_kv_cache: layer 35: dev = MTL0\nllama_kv_cache: MTL0 KV buffer size = 128.00 MiB\nllama_memory_recurrent, layer 0: dev = MTL0\nllama_memory_recurrent, layer 1: dev = MTL0\nllama_memory_recurrent, layer 2: dev = MTL0\nllama_memory_recurrent, layer 3: dev = MTL0\nllama_memory_recurrent, layer 4: dev = MTL0\nllama_memory_recurrent, layer 6: dev = MTL0\nllama_memory_recurrent, layer 7: dev = MTL0\nllama_memory_recurrent, layer 8: dev = MTL0\nllama_memory_recurrent, layer 9: dev = MTL0\nllama_memory_recurrent, layer 10: dev = MTL0\nllama_memory_recurrent, layer 11: dev = MTL0\nllama_memory_recurrent, layer 12: dev = MTL0\nllama_memory_recurrent, layer 13: dev = MTL0\nllama_memory_recurrent, layer 14: dev = MTL0\nllama_memory_recurrent, layer 16: dev = MTL0\nllama_memory_recurrent, layer 17: dev = MTL0\nllama_memory_recurrent, layer 18: dev = MTL0\nllama_memory_recurrent, layer 19: dev = MTL0\nllama_memory_recurrent, layer 20: dev = MTL0\nllama_memory_recurrent, layer 21: dev = MTL0\nllama_memory_recurrent, layer 22: dev = MTL0\nllama_memory_recurrent, layer 23: dev = MTL0\nllama_memory_recurrent, layer 24: dev = MTL0\nllama_memory_recurrent, layer 26: dev = MTL0\nllama_memory_recurrent, layer 27: dev = MTL0\nllama_memory_recurrent, layer 28: dev = MTL0\nllama_memory_recurrent, layer 29: dev = MTL0\nllama_memory_recurrent, layer 30: dev = MTL0\nllama_memory_recurrent, layer 31: dev = MTL0\nllama_memory_recurrent, layer 32: dev = MTL0\nllama_memory_recurrent, layer 33: dev = MTL0\nllama_memory_recurrent, layer 34: dev = MTL0\nllama_memory_recurrent, layer 36: dev = MTL0\nllama_memory_recurrent, layer 37: dev = MTL0\nllama_memory_recurrent, layer 38: dev = MTL0\nllama_memory_recurrent, layer 39: dev = MTL0\nllama_memory_recurrent: MTL0 RS buffer size = 147.48 MiB\nsched_reserve: MTL0 compute buffer size = 282.28 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 41,
"totalLayers": 41,
"inferenceMs": 405.2658340006019,
"durationMs": 413.8391250016866,
"tokensPerSecond": 7.402548520770553,
"inputTokens": 69,
"outputTokens": 3,
"outputLimitReached": false,
"rssBytes": 19948175360,
"cpuThreads": 4,
"totalMs": 1788.3274170017103,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "q4",
"device": "mps",
"status": "pass",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "2026-11-04"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
"device": "mps",
"precision": "q4",
"provider": "llama.cpp Metal",
"loadMs": 1374.4882920000236,
"acceleration": "partial",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer 0 assigned to device MTL0, is_swa = 0\nload_tensors: layer 1 assigned to device MTL0, is_swa = 0\nload_tensors: layer 2 assigned to device MTL0, is_swa = 0\nload_tensors: layer 3 assigned to device MTL0, is_swa = 0\nload_tensors: layer 4 assigned to device MTL0, is_swa = 0\nload_tensors: layer 5 assigned to device MTL0, is_swa = 0\nload_tensors: layer 6 assigned to device MTL0, is_swa = 0\nload_tensors: layer 7 assigned to device MTL0, is_swa = 0\nload_tensors: layer 8 assigned to device MTL0, is_swa = 0\nload_tensors: layer 9 assigned to device MTL0, is_swa = 0\nload_tensors: layer 10 assigned to device MTL0, is_swa = 0\nload_tensors: layer 11 assigned to device MTL0, is_swa = 0\nload_tensors: layer 12 assigned to device MTL0, is_swa = 0\nload_tensors: layer 13 assigned to device MTL0, is_swa = 0\nload_tensors: layer 14 assigned to device MTL0, is_swa = 0\nload_tensors: layer 15 assigned to device MTL0, is_swa = 0\nload_tensors: layer 16 assigned to device MTL0, is_swa = 0\nload_tensors: layer 17 assigned to device MTL0, is_swa = 0\nload_tensors: layer 18 assigned to device MTL0, is_swa = 0\nload_tensors: layer 19 assigned to device MTL0, is_swa = 0\nload_tensors: layer 20 assigned to device MTL0, is_swa = 0\nload_tensors: layer 21 assigned to device MTL0, is_swa = 0\nload_tensors: layer 22 assigned to device MTL0, is_swa = 0\nload_tensors: layer 23 assigned to device MTL0, is_swa = 0\nload_tensors: layer 24 assigned to device MTL0, is_swa = 0\nload_tensors: layer 25 assigned to device MTL0, is_swa = 0\nload_tensors: layer 26 assigned to device MTL0, is_swa = 0\nload_tensors: layer 27 assigned to device MTL0, is_swa = 0\nload_tensors: layer 28 assigned to device MTL0, is_swa = 0\nload_tensors: layer 29 assigned to device MTL0, is_swa = 0\nload_tensors: layer 30 assigned to device MTL0, is_swa = 0\nload_tensors: layer 31 assigned to device MTL0, is_swa = 0\nload_tensors: layer 32 assigned to device MTL0, is_swa = 0\nload_tensors: layer 33 assigned to device MTL0, is_swa = 0\nload_tensors: layer 34 assigned to device MTL0, is_swa = 0\nload_tensors: layer 35 assigned to device MTL0, is_swa = 0\nload_tensors: layer 36 assigned to device MTL0, is_swa = 0\nload_tensors: layer 37 assigned to device MTL0, is_swa = 0\nload_tensors: layer 38 assigned to device MTL0, is_swa = 0\nload_tensors: layer 39 assigned to device MTL0, is_swa = 0\nload_tensors: layer 40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 321.56 MiB\nload_tensors: MTL0_Mapped model buffer size = 18570.93 MiB\nllama_kv_cache: layer 5: dev = MTL0\nllama_kv_cache: layer 15: dev = MTL0\nllama_kv_cache: layer 25: dev = MTL0\nllama_kv_cache: layer 35: dev = MTL0\nllama_kv_cache: MTL0 KV buffer size = 128.00 MiB\nllama_memory_recurrent, layer 0: dev = MTL0\nllama_memory_recurrent, layer 1: dev = MTL0\nllama_memory_recurrent, layer 2: dev = MTL0\nllama_memory_recurrent, layer 3: dev = MTL0\nllama_memory_recurrent, layer 4: dev = MTL0\nllama_memory_recurrent, layer 6: dev = MTL0\nllama_memory_recurrent, layer 7: dev = MTL0\nllama_memory_recurrent, layer 8: dev = MTL0\nllama_memory_recurrent, layer 9: dev = MTL0\nllama_memory_recurrent, layer 10: dev = MTL0\nllama_memory_recurrent, layer 11: dev = MTL0\nllama_memory_recurrent, layer 12: dev = MTL0\nllama_memory_recurrent, layer 13: dev = MTL0\nllama_memory_recurrent, layer 14: dev = MTL0\nllama_memory_recurrent, layer 16: dev = MTL0\nllama_memory_recurrent, layer 17: dev = MTL0\nllama_memory_recurrent, layer 18: dev = MTL0\nllama_memory_recurrent, layer 19: dev = MTL0\nllama_memory_recurrent, layer 20: dev = MTL0\nllama_memory_recurrent, layer 21: dev = MTL0\nllama_memory_recurrent, layer 22: dev = MTL0\nllama_memory_recurrent, layer 23: dev = MTL0\nllama_memory_recurrent, layer 24: dev = MTL0\nllama_memory_recurrent, layer 26: dev = MTL0\nllama_memory_recurrent, layer 27: dev = MTL0\nllama_memory_recurrent, layer 28: dev = MTL0\nllama_memory_recurrent, layer 29: dev = MTL0\nllama_memory_recurrent, layer 30: dev = MTL0\nllama_memory_recurrent, layer 31: dev = MTL0\nllama_memory_recurrent, layer 32: dev = MTL0\nllama_memory_recurrent, layer 33: dev = MTL0\nllama_memory_recurrent, layer 34: dev = MTL0\nllama_memory_recurrent, layer 36: dev = MTL0\nllama_memory_recurrent, layer 37: dev = MTL0\nllama_memory_recurrent, layer 38: dev = MTL0\nllama_memory_recurrent, layer 39: dev = MTL0\nllama_memory_recurrent: MTL0 RS buffer size = 147.48 MiB\nsched_reserve: MTL0 compute buffer size = 282.28 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 41,
"totalLayers": 41,
"inferenceMs": 551.9109160013613,
"durationMs": 561.6470420063706,
"tokensPerSecond": 12.683206287557345,
"inputTokens": 72,
"outputTokens": 7,
"outputLimitReached": false,
"rssBytes": 19948519424,
"cpuThreads": 4,
"totalMs": 1936.1353340063943,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "q4",
"device": "mps",
"status": "pass",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "Unknown"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
"device": "mps",
"precision": "q4",
"provider": "llama.cpp Metal",
"loadMs": 1374.4882920000236,
"acceleration": "partial",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer 0 assigned to device MTL0, is_swa = 0\nload_tensors: layer 1 assigned to device MTL0, is_swa = 0\nload_tensors: layer 2 assigned to device MTL0, is_swa = 0\nload_tensors: layer 3 assigned to device MTL0, is_swa = 0\nload_tensors: layer 4 assigned to device MTL0, is_swa = 0\nload_tensors: layer 5 assigned to device MTL0, is_swa = 0\nload_tensors: layer 6 assigned to device MTL0, is_swa = 0\nload_tensors: layer 7 assigned to device MTL0, is_swa = 0\nload_tensors: layer 8 assigned to device MTL0, is_swa = 0\nload_tensors: layer 9 assigned to device MTL0, is_swa = 0\nload_tensors: layer 10 assigned to device MTL0, is_swa = 0\nload_tensors: layer 11 assigned to device MTL0, is_swa = 0\nload_tensors: layer 12 assigned to device MTL0, is_swa = 0\nload_tensors: layer 13 assigned to device MTL0, is_swa = 0\nload_tensors: layer 14 assigned to device MTL0, is_swa = 0\nload_tensors: layer 15 assigned to device MTL0, is_swa = 0\nload_tensors: layer 16 assigned to device MTL0, is_swa = 0\nload_tensors: layer 17 assigned to device MTL0, is_swa = 0\nload_tensors: layer 18 assigned to device MTL0, is_swa = 0\nload_tensors: layer 19 assigned to device MTL0, is_swa = 0\nload_tensors: layer 20 assigned to device MTL0, is_swa = 0\nload_tensors: layer 21 assigned to device MTL0, is_swa = 0\nload_tensors: layer 22 assigned to device MTL0, is_swa = 0\nload_tensors: layer 23 assigned to device MTL0, is_swa = 0\nload_tensors: layer 24 assigned to device MTL0, is_swa = 0\nload_tensors: layer 25 assigned to device MTL0, is_swa = 0\nload_tensors: layer 26 assigned to device MTL0, is_swa = 0\nload_tensors: layer 27 assigned to device MTL0, is_swa = 0\nload_tensors: layer 28 assigned to device MTL0, is_swa = 0\nload_tensors: layer 29 assigned to device MTL0, is_swa = 0\nload_tensors: layer 30 assigned to device MTL0, is_swa = 0\nload_tensors: layer 31 assigned to device MTL0, is_swa = 0\nload_tensors: layer 32 assigned to device MTL0, is_swa = 0\nload_tensors: layer 33 assigned to device MTL0, is_swa = 0\nload_tensors: layer 34 assigned to device MTL0, is_swa = 0\nload_tensors: layer 35 assigned to device MTL0, is_swa = 0\nload_tensors: layer 36 assigned to device MTL0, is_swa = 0\nload_tensors: layer 37 assigned to device MTL0, is_swa = 0\nload_tensors: layer 38 assigned to device MTL0, is_swa = 0\nload_tensors: layer 39 assigned to device MTL0, is_swa = 0\nload_tensors: layer 40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 321.56 MiB\nload_tensors: MTL0_Mapped model buffer size = 18570.93 MiB\nllama_kv_cache: layer 5: dev = MTL0\nllama_kv_cache: layer 15: dev = MTL0\nllama_kv_cache: layer 25: dev = MTL0\nllama_kv_cache: layer 35: dev = MTL0\nllama_kv_cache: MTL0 KV buffer size = 128.00 MiB\nllama_memory_recurrent, layer 0: dev = MTL0\nllama_memory_recurrent, layer 1: dev = MTL0\nllama_memory_recurrent, layer 2: dev = MTL0\nllama_memory_recurrent, layer 3: dev = MTL0\nllama_memory_recurrent, layer 4: dev = MTL0\nllama_memory_recurrent, layer 6: dev = MTL0\nllama_memory_recurrent, layer 7: dev = MTL0\nllama_memory_recurrent, layer 8: dev = MTL0\nllama_memory_recurrent, layer 9: dev = MTL0\nllama_memory_recurrent, layer 10: dev = MTL0\nllama_memory_recurrent, layer 11: dev = MTL0\nllama_memory_recurrent, layer 12: dev = MTL0\nllama_memory_recurrent, layer 13: dev = MTL0\nllama_memory_recurrent, layer 14: dev = MTL0\nllama_memory_recurrent, layer 16: dev = MTL0\nllama_memory_recurrent, layer 17: dev = MTL0\nllama_memory_recurrent, layer 18: dev = MTL0\nllama_memory_recurrent, layer 19: dev = MTL0\nllama_memory_recurrent, layer 20: dev = MTL0\nllama_memory_recurrent, layer 21: dev = MTL0\nllama_memory_recurrent, layer 22: dev = MTL0\nllama_memory_recurrent, layer 23: dev = MTL0\nllama_memory_recurrent, layer 24: dev = MTL0\nllama_memory_recurrent, layer 26: dev = MTL0\nllama_memory_recurrent, layer 27: dev = MTL0\nllama_memory_recurrent, layer 28: dev = MTL0\nllama_memory_recurrent, layer 29: dev = MTL0\nllama_memory_recurrent, layer 30: dev = MTL0\nllama_memory_recurrent, layer 31: dev = MTL0\nllama_memory_recurrent, layer 32: dev = MTL0\nllama_memory_recurrent, layer 33: dev = MTL0\nllama_memory_recurrent, layer 34: dev = MTL0\nllama_memory_recurrent, layer 36: dev = MTL0\nllama_memory_recurrent, layer 37: dev = MTL0\nllama_memory_recurrent, layer 38: dev = MTL0\nllama_memory_recurrent, layer 39: dev = MTL0\nllama_memory_recurrent: MTL0 RS buffer size = 147.48 MiB\nsched_reserve: MTL0 compute buffer size = 282.28 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 41,
"totalLayers": 41,
"inferenceMs": 377.84820800152374,
"durationMs": 386.29999999830034,
"tokensPerSecond": 5.293130833088229,
"inputTokens": 74,
"outputTokens": 2,
"outputLimitReached": false,
"rssBytes": 19948535808,
"cpuThreads": 4,
"totalMs": 1760.788291998324,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "q4",
"device": "cpu",
"status": "fail",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"plainParity": true,
"passed": false,
"expected": "Paris"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
"device": "cpu",
"precision": "q4",
"provider": "llama.cpp CPU",
"loadMs": 34819.97774999763,
"acceleration": "cpu",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 18552.93 MiB\nload_tensors: CPU_REPACK model buffer size = 18517.69 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 0,
"totalLayers": 41,
"inferenceMs": 3832.869332996779,
"durationMs": 3841.931292001391,
"tokensPerSecond": 0.521802291244892,
"inputTokens": 63,
"outputTokens": 2,
"outputLimitReached": false,
"rssBytes": 37500960768,
"cpuThreads": 4,
"totalMs": 38661.909041999024,
"grade": "slow",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "q4",
"device": "cpu",
"status": "fail",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": false,
"expected": "56"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
"device": "cpu",
"precision": "q4",
"provider": "llama.cpp CPU",
"loadMs": 34819.97774999763,
"acceleration": "cpu",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 18552.93 MiB\nload_tensors: CPU_REPACK model buffer size = 18517.69 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 0,
"totalLayers": 41,
"inferenceMs": 2350.014083000133,
"durationMs": 2358.5027090011863,
"tokensPerSecond": 0.8510587295914035,
"inputTokens": 63,
"outputTokens": 2,
"outputLimitReached": false,
"rssBytes": 37859672064,
"cpuThreads": 4,
"totalMs": 37178.48045899882,
"grade": "slow",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "q4",
"device": "cpu",
"status": "fail",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": false,
"expected": "chat"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
"device": "cpu",
"precision": "q4",
"provider": "llama.cpp CPU",
"loadMs": 34819.97774999763,
"acceleration": "cpu",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 18552.93 MiB\nload_tensors: CPU_REPACK model buffer size = 18517.69 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 0,
"totalLayers": 41,
"inferenceMs": 2191.428083999199,
"durationMs": 2199.4122079995577,
"tokensPerSecond": 0.9126468783543851,
"inputTokens": 68,
"outputTokens": 2,
"outputLimitReached": false,
"rssBytes": 38124797952,
"cpuThreads": 4,
"totalMs": 37019.38995799719,
"grade": "slow",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "q4",
"device": "cpu",
"status": "fail",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": false,
"expected": "hola"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
"device": "cpu",
"precision": "q4",
"provider": "llama.cpp CPU",
"loadMs": 34819.97774999763,
"acceleration": "cpu",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 18552.93 MiB\nload_tensors: CPU_REPACK model buffer size = 18517.69 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 0,
"totalLayers": 41,
"inferenceMs": 2424.1190839966293,
"durationMs": 2432.439500000328,
"tokensPerSecond": 1.2375629645445965,
"inputTokens": 69,
"outputTokens": 3,
"outputLimitReached": false,
"rssBytes": 38216499200,
"cpuThreads": 4,
"totalMs": 37252.41724999796,
"grade": "slow",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "q4",
"device": "cpu",
"status": "fail",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": false,
"expected": "2026-11-04",
"fixturePassed": true
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
"device": "cpu",
"precision": "q4",
"provider": "llama.cpp CPU",
"loadMs": 34819.97774999763,
"acceleration": "cpu",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 18552.93 MiB\nload_tensors: CPU_REPACK model buffer size = 18517.69 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 0,
"totalLayers": 41,
"inferenceMs": 2686.0973330039997,
"durationMs": 2694.8029580016737,
"tokensPerSecond": 2.6060112989917394,
"inputTokens": 72,
"outputTokens": 7,
"outputLimitReached": false,
"rssBytes": 38413828096,
"cpuThreads": 4,
"totalMs": 37514.78070799931,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
},
"failure": "This fixture passed but the backend did not pass the complete repeated profile."
},
{
"precision": "q4",
"device": "cpu",
"status": "fail",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": false,
"expected": "Unknown"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
"device": "cpu",
"precision": "q4",
"provider": "llama.cpp CPU",
"loadMs": 34819.97774999763,
"acceleration": "cpu",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 18552.93 MiB\nload_tensors: CPU_REPACK model buffer size = 18517.69 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 0,
"totalLayers": 41,
"inferenceMs": 2411.2892499979353,
"durationMs": 2419.3728329992155,
"tokensPerSecond": 0.8294318070723836,
"inputTokens": 74,
"outputTokens": 2,
"outputLimitReached": false,
"rssBytes": 38546784256,
"cpuThreads": 4,
"totalMs": 37239.35058299685,
"grade": "slow",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "q4",
"device": "cpu",
"status": "fail",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"plainParity": true,
"passed": false,
"expected": "Paris"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
"device": "cpu",
"precision": "q4",
"provider": "llama.cpp CPU",
"loadMs": 19607.6884590002,
"acceleration": "cpu",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 18552.93 MiB\nload_tensors: CPU_REPACK model buffer size = 18517.69 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 0,
"totalLayers": 41,
"inferenceMs": 7619.532499993511,
"durationMs": 7627.649542002473,
"tokensPerSecond": 0.2624832953992523,
"inputTokens": 63,
"outputTokens": 2,
"outputLimitReached": false,
"rssBytes": 38995722240,
"cpuThreads": 1,
"totalMs": 27235.338001002674,
"grade": "slow",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "q4",
"device": "cpu",
"status": "fail",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": false,
"expected": "56"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
"device": "cpu",
"precision": "q4",
"provider": "llama.cpp CPU",
"loadMs": 19607.6884590002,
"acceleration": "cpu",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 18552.93 MiB\nload_tensors: CPU_REPACK model buffer size = 18517.69 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 0,
"totalLayers": 41,
"inferenceMs": 7651.7220420064405,
"durationMs": 7660.638916997414,
"tokensPerSecond": 0.26137907114508285,
"inputTokens": 63,
"outputTokens": 2,
"outputLimitReached": false,
"rssBytes": 38997016576,
"cpuThreads": 1,
"totalMs": 27268.327375997615,
"grade": "slow",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "q4",
"device": "cpu",
"status": "fail",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": false,
"expected": "chat"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
"device": "cpu",
"precision": "q4",
"provider": "llama.cpp CPU",
"loadMs": 19607.6884590002,
"acceleration": "cpu",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 18552.93 MiB\nload_tensors: CPU_REPACK model buffer size = 18517.69 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 0,
"totalLayers": 41,
"inferenceMs": 8071.8802500050515,
"durationMs": 8080.053291996592,
"tokensPerSecond": 0.24777374515668124,
"inputTokens": 68,
"outputTokens": 2,
"outputLimitReached": false,
"rssBytes": 38732054528,
"cpuThreads": 1,
"totalMs": 27687.741750996793,
"grade": "slow",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "q4",
"device": "cpu",
"status": "fail",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": false,
"expected": "hola"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
"device": "cpu",
"precision": "q4",
"provider": "llama.cpp CPU",
"loadMs": 19607.6884590002,
"acceleration": "cpu",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 18552.93 MiB\nload_tensors: CPU_REPACK model buffer size = 18517.69 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 0,
"totalLayers": 41,
"inferenceMs": 8437.868083994545,
"durationMs": 8446.016834001057,
"tokensPerSecond": 0.355540045202956,
"inputTokens": 69,
"outputTokens": 3,
"outputLimitReached": false,
"rssBytes": 38664617984,
"cpuThreads": 1,
"totalMs": 28053.705293001258,
"grade": "slow",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "q4",
"device": "cpu",
"status": "fail",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": false,
"expected": "2026-11-04"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
"device": "cpu",
"precision": "q4",
"provider": "llama.cpp CPU",
"loadMs": 19607.6884590002,
"acceleration": "cpu",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 18552.93 MiB\nload_tensors: CPU_REPACK model buffer size = 18517.69 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 0,
"totalLayers": 41,
"inferenceMs": 9485.951374997967,
"durationMs": 9494.039042001532,
"tokensPerSecond": 0.7379333630625426,
"inputTokens": 72,
"outputTokens": 7,
"outputLimitReached": false,
"rssBytes": 38755663872,
"cpuThreads": 1,
"totalMs": 29101.727501001733,
"grade": "slow",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "q4",
"device": "cpu",
"status": "fail",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": false,
"expected": "Unknown"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
"device": "cpu",
"precision": "q4",
"provider": "llama.cpp CPU",
"loadMs": 19607.6884590002,
"acceleration": "cpu",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 18552.93 MiB\nload_tensors: CPU_REPACK model buffer size = 18517.69 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 0,
"totalLayers": 41,
"inferenceMs": 8769.063250001636,
"durationMs": 8776.766625000164,
"tokensPerSecond": 0.22807453236235087,
"inputTokens": 74,
"outputTokens": 2,
"outputLimitReached": false,
"rssBytes": 38781222912,
"cpuThreads": 1,
"totalMs": 28384.455084000365,
"grade": "slow",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
}
]Granite 4.0 Hybrid Tiny
36 configurations testedQuality criteria
No explicit quality gate recorded. Review per-case evidence; full certification is pending.
| Precision | Device | Result | Platform | Inference / load | |
|---|---|---|---|---|---|
Q4 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 94 ms / 1236 ms |
Q4 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 96 ms / 925 ms |
Q4 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 97 ms / 1116 ms |
Q4 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 97 ms / 1236 ms |
Q4 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 97 ms / 1236 ms |
Q4 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 99 ms / 925 ms |
Q4 | mpsAccelerated | Passed | Fast | macos · aarch64 | 165 ms / 1116 ms |
Q4 | mpsAccelerated | Passed | Fast | macos · aarch64 | 169 ms / 1236 ms |
Q4 | mpsAccelerated | Passed | Fast | macos · aarch64 | 170 ms / 925 ms |
Q4 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 100 ms / 1236 ms |
Q4 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 100 ms / 925 ms |
Q4 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 102 ms / 925 ms |
Q4 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 104 ms / 1116 ms |
Q4 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 108 ms / 1236 ms |
Q4 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 109 ms / 1116 ms |
Q4 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 126 ms / 925 ms |
Q4 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 217 ms / 1116 ms |
Q4 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 280 ms / 1116 ms |
Q4 | cpu | Failed | — | macos · aarch64 | 452 ms / 5334 ms |
Q4 | cpu | Failed | — | macos · aarch64 | 429 ms / 5334 ms |
Q4 | cpu | Failed | — | macos · aarch64 | 419 ms / 5334 ms |
Q4 | cpu | Failed | — | macos · aarch64 | 409 ms / 5334 ms |
Q4 | cpu | Failed | — | macos · aarch64 | 551 ms / 5334 ms |
Q4 | cpu | Failed | — | macos · aarch64 | 530 ms / 5334 ms |
Q4 | cpu | Failed | — | macos · aarch64 | 1444 ms / 4297 ms |
Q4 | cpu | Failed | — | macos · aarch64 | 1737 ms / 4297 ms |
Q4 | cpu | Failed | — | macos · aarch64 | 1108 ms / 4297 ms |
Q4 | cpu | Failed | — | macos · aarch64 | 915 ms / 4297 ms |
Q4 | cpu | Failed | — | macos · aarch64 | 1209 ms / 4297 ms |
Q4 | cpu | Failed | — | macos · aarch64 | 658 ms / 4297 ms |
Q4 | cpu | Failed | — | macos · aarch64 | 1374 ms / 4240 ms |
Q4 | cpu | Failed | — | macos · aarch64 | 1358 ms / 4240 ms |
Q4 | cpu | Failed | — | macos · aarch64 | 1435 ms / 4240 ms |
Q4 | cpu | Failed | — | macos · aarch64 | 1456 ms / 4240 ms |
Q4 | cpu | Failed | — | macos · aarch64 | 1682 ms / 4240 ms |
Q4 | cpu | Failed | — | macos · aarch64 | 1633 ms / 4240 ms |
Test setup & sample details
- Quality gate
- No explicit quality gate recorded. Review the per-case quality evidence; full certification is pending.
- Tested
- 2026-11-07T12:40:44.259304+00:00
- Environment
- { "os": "macos", "arch": "aarch64", "hardware": "Apple M3 Max", "osVersion": "27.2" }
- Runtime
- llama.cpp 0.3.36
- Artifact revision
- 08d5a8a9741dd5c1a95d2d39e25253226aa1464e
- Fixture
{ "file": "model-samples/desktop/granite4/granite-4.0-h-tiny-native-capital.json", "size": 244, "sha1": "9013b4f5a71cd1280d78b3dfa751ead1186d071f", "sha256": "8ff34789fa2a0f5bbef2080057f4b36376d3c73ea64baccefb6caec96eafbd68", "url": "https://assets.localai.tools/samples/v1/files/8ff34789fa2a0f5bbef2080057f4b36376d3c73ea64baccefb6caec96eafbd68/granite-4.0-h-tiny-native-capital.json" }- Rerun when
- Artifact/runtime revision; Input/generation processing; Fixture/quality gates; OS/backend behavior
Per-case quality & failure details
[
{
"precision": "q4",
"device": "mps",
"status": "pass",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"plainParity": true,
"passed": true,
"expected": "Paris"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
"device": "mps",
"precision": "q4",
"provider": "llama.cpp Metal",
"loadMs": 1116.3811659935163,
"acceleration": "partial",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer 0 assigned to device MTL0, is_swa = 0\nload_tensors: layer 1 assigned to device MTL0, is_swa = 0\nload_tensors: layer 2 assigned to device MTL0, is_swa = 0\nload_tensors: layer 3 assigned to device MTL0, is_swa = 0\nload_tensors: layer 4 assigned to device MTL0, is_swa = 0\nload_tensors: layer 5 assigned to device MTL0, is_swa = 0\nload_tensors: layer 6 assigned to device MTL0, is_swa = 0\nload_tensors: layer 7 assigned to device MTL0, is_swa = 0\nload_tensors: layer 8 assigned to device MTL0, is_swa = 0\nload_tensors: layer 9 assigned to device MTL0, is_swa = 0\nload_tensors: layer 10 assigned to device MTL0, is_swa = 0\nload_tensors: layer 11 assigned to device MTL0, is_swa = 0\nload_tensors: layer 12 assigned to device MTL0, is_swa = 0\nload_tensors: layer 13 assigned to device MTL0, is_swa = 0\nload_tensors: layer 14 assigned to device MTL0, is_swa = 0\nload_tensors: layer 15 assigned to device MTL0, is_swa = 0\nload_tensors: layer 16 assigned to device MTL0, is_swa = 0\nload_tensors: layer 17 assigned to device MTL0, is_swa = 0\nload_tensors: layer 18 assigned to device MTL0, is_swa = 0\nload_tensors: layer 19 assigned to device MTL0, is_swa = 0\nload_tensors: layer 20 assigned to device MTL0, is_swa = 0\nload_tensors: layer 21 assigned to device MTL0, is_swa = 0\nload_tensors: layer 22 assigned to device MTL0, is_swa = 0\nload_tensors: layer 23 assigned to device MTL0, is_swa = 0\nload_tensors: layer 24 assigned to device MTL0, is_swa = 0\nload_tensors: layer 25 assigned to device MTL0, is_swa = 0\nload_tensors: layer 26 assigned to device MTL0, is_swa = 0\nload_tensors: layer 27 assigned to device MTL0, is_swa = 0\nload_tensors: layer 28 assigned to device MTL0, is_swa = 0\nload_tensors: layer 29 assigned to device MTL0, is_swa = 0\nload_tensors: layer 30 assigned to device MTL0, is_swa = 0\nload_tensors: layer 31 assigned to device MTL0, is_swa = 0\nload_tensors: layer 32 assigned to device MTL0, is_swa = 0\nload_tensors: layer 33 assigned to device MTL0, is_swa = 0\nload_tensors: layer 34 assigned to device MTL0, is_swa = 0\nload_tensors: layer 35 assigned to device MTL0, is_swa = 0\nload_tensors: layer 36 assigned to device MTL0, is_swa = 0\nload_tensors: layer 37 assigned to device MTL0, is_swa = 0\nload_tensors: layer 38 assigned to device MTL0, is_swa = 0\nload_tensors: layer 39 assigned to device MTL0, is_swa = 0\nload_tensors: layer 40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 120.59 MiB\nload_tensors: MTL0_Mapped model buffer size = 4031.55 MiB\nllama_kv_cache: layer 5: dev = MTL0\nllama_kv_cache: layer 15: dev = MTL0\nllama_kv_cache: layer 25: dev = MTL0\nllama_kv_cache: layer 35: dev = MTL0\nllama_kv_cache: MTL0 KV buffer size = 64.00 MiB\nllama_memory_recurrent, layer 0: dev = MTL0\nllama_memory_recurrent, layer 1: dev = MTL0\nllama_memory_recurrent, layer 2: dev = MTL0\nllama_memory_recurrent, layer 3: dev = MTL0\nllama_memory_recurrent, layer 4: dev = MTL0\nllama_memory_recurrent, layer 6: dev = MTL0\nllama_memory_recurrent, layer 7: dev = MTL0\nllama_memory_recurrent, layer 8: dev = MTL0\nllama_memory_recurrent, layer 9: dev = MTL0\nllama_memory_recurrent, layer 10: dev = MTL0\nllama_memory_recurrent, layer 11: dev = MTL0\nllama_memory_recurrent, layer 12: dev = MTL0\nllama_memory_recurrent, layer 13: dev = MTL0\nllama_memory_recurrent, layer 14: dev = MTL0\nllama_memory_recurrent, layer 16: dev = MTL0\nllama_memory_recurrent, layer 17: dev = MTL0\nllama_memory_recurrent, layer 18: dev = MTL0\nllama_memory_recurrent, layer 19: dev = MTL0\nllama_memory_recurrent, layer 20: dev = MTL0\nllama_memory_recurrent, layer 21: dev = MTL0\nllama_memory_recurrent, layer 22: dev = MTL0\nllama_memory_recurrent, layer 23: dev = MTL0\nllama_memory_recurrent, layer 24: dev = MTL0\nllama_memory_recurrent, layer 26: dev = MTL0\nllama_memory_recurrent, layer 27: dev = MTL0\nllama_memory_recurrent, layer 28: dev = MTL0\nllama_memory_recurrent, layer 29: dev = MTL0\nllama_memory_recurrent, layer 30: dev = MTL0\nllama_memory_recurrent, layer 31: dev = MTL0\nllama_memory_recurrent, layer 32: dev = MTL0\nllama_memory_recurrent, layer 33: dev = MTL0\nllama_memory_recurrent, layer 34: dev = MTL0\nllama_memory_recurrent, layer 36: dev = MTL0\nllama_memory_recurrent, layer 37: dev = MTL0\nllama_memory_recurrent, layer 38: dev = MTL0\nllama_memory_recurrent, layer 39: dev = MTL0\nllama_memory_recurrent: MTL0 RS buffer size = 55.37 MiB\nsched_reserve: MTL0 compute buffer size = 123.63 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 41,
"totalLayers": 41,
"inferenceMs": 279.5718749985099,
"durationMs": 289.8012080040644,
"tokensPerSecond": 7.1537954238446195,
"inputTokens": 63,
"outputTokens": 2,
"outputLimitReached": false,
"rssBytes": 4529979392,
"cpuThreads": 4,
"totalMs": 1406.1823739975807,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "q4",
"device": "mps",
"status": "pass",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "56"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
"device": "mps",
"precision": "q4",
"provider": "llama.cpp Metal",
"loadMs": 1116.3811659935163,
"acceleration": "partial",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer 0 assigned to device MTL0, is_swa = 0\nload_tensors: layer 1 assigned to device MTL0, is_swa = 0\nload_tensors: layer 2 assigned to device MTL0, is_swa = 0\nload_tensors: layer 3 assigned to device MTL0, is_swa = 0\nload_tensors: layer 4 assigned to device MTL0, is_swa = 0\nload_tensors: layer 5 assigned to device MTL0, is_swa = 0\nload_tensors: layer 6 assigned to device MTL0, is_swa = 0\nload_tensors: layer 7 assigned to device MTL0, is_swa = 0\nload_tensors: layer 8 assigned to device MTL0, is_swa = 0\nload_tensors: layer 9 assigned to device MTL0, is_swa = 0\nload_tensors: layer 10 assigned to device MTL0, is_swa = 0\nload_tensors: layer 11 assigned to device MTL0, is_swa = 0\nload_tensors: layer 12 assigned to device MTL0, is_swa = 0\nload_tensors: layer 13 assigned to device MTL0, is_swa = 0\nload_tensors: layer 14 assigned to device MTL0, is_swa = 0\nload_tensors: layer 15 assigned to device MTL0, is_swa = 0\nload_tensors: layer 16 assigned to device MTL0, is_swa = 0\nload_tensors: layer 17 assigned to device MTL0, is_swa = 0\nload_tensors: layer 18 assigned to device MTL0, is_swa = 0\nload_tensors: layer 19 assigned to device MTL0, is_swa = 0\nload_tensors: layer 20 assigned to device MTL0, is_swa = 0\nload_tensors: layer 21 assigned to device MTL0, is_swa = 0\nload_tensors: layer 22 assigned to device MTL0, is_swa = 0\nload_tensors: layer 23 assigned to device MTL0, is_swa = 0\nload_tensors: layer 24 assigned to device MTL0, is_swa = 0\nload_tensors: layer 25 assigned to device MTL0, is_swa = 0\nload_tensors: layer 26 assigned to device MTL0, is_swa = 0\nload_tensors: layer 27 assigned to device MTL0, is_swa = 0\nload_tensors: layer 28 assigned to device MTL0, is_swa = 0\nload_tensors: layer 29 assigned to device MTL0, is_swa = 0\nload_tensors: layer 30 assigned to device MTL0, is_swa = 0\nload_tensors: layer 31 assigned to device MTL0, is_swa = 0\nload_tensors: layer 32 assigned to device MTL0, is_swa = 0\nload_tensors: layer 33 assigned to device MTL0, is_swa = 0\nload_tensors: layer 34 assigned to device MTL0, is_swa = 0\nload_tensors: layer 35 assigned to device MTL0, is_swa = 0\nload_tensors: layer 36 assigned to device MTL0, is_swa = 0\nload_tensors: layer 37 assigned to device MTL0, is_swa = 0\nload_tensors: layer 38 assigned to device MTL0, is_swa = 0\nload_tensors: layer 39 assigned to device MTL0, is_swa = 0\nload_tensors: layer 40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 120.59 MiB\nload_tensors: MTL0_Mapped model buffer size = 4031.55 MiB\nllama_kv_cache: layer 5: dev = MTL0\nllama_kv_cache: layer 15: dev = MTL0\nllama_kv_cache: layer 25: dev = MTL0\nllama_kv_cache: layer 35: dev = MTL0\nllama_kv_cache: MTL0 KV buffer size = 64.00 MiB\nllama_memory_recurrent, layer 0: dev = MTL0\nllama_memory_recurrent, layer 1: dev = MTL0\nllama_memory_recurrent, layer 2: dev = MTL0\nllama_memory_recurrent, layer 3: dev = MTL0\nllama_memory_recurrent, layer 4: dev = MTL0\nllama_memory_recurrent, layer 6: dev = MTL0\nllama_memory_recurrent, layer 7: dev = MTL0\nllama_memory_recurrent, layer 8: dev = MTL0\nllama_memory_recurrent, layer 9: dev = MTL0\nllama_memory_recurrent, layer 10: dev = MTL0\nllama_memory_recurrent, layer 11: dev = MTL0\nllama_memory_recurrent, layer 12: dev = MTL0\nllama_memory_recurrent, layer 13: dev = MTL0\nllama_memory_recurrent, layer 14: dev = MTL0\nllama_memory_recurrent, layer 16: dev = MTL0\nllama_memory_recurrent, layer 17: dev = MTL0\nllama_memory_recurrent, layer 18: dev = MTL0\nllama_memory_recurrent, layer 19: dev = MTL0\nllama_memory_recurrent, layer 20: dev = MTL0\nllama_memory_recurrent, layer 21: dev = MTL0\nllama_memory_recurrent, layer 22: dev = MTL0\nllama_memory_recurrent, layer 23: dev = MTL0\nllama_memory_recurrent, layer 24: dev = MTL0\nllama_memory_recurrent, layer 26: dev = MTL0\nllama_memory_recurrent, layer 27: dev = MTL0\nllama_memory_recurrent, layer 28: dev = MTL0\nllama_memory_recurrent, layer 29: dev = MTL0\nllama_memory_recurrent, layer 30: dev = MTL0\nllama_memory_recurrent, layer 31: dev = MTL0\nllama_memory_recurrent, layer 32: dev = MTL0\nllama_memory_recurrent, layer 33: dev = MTL0\nllama_memory_recurrent, layer 34: dev = MTL0\nllama_memory_recurrent, layer 36: dev = MTL0\nllama_memory_recurrent, layer 37: dev = MTL0\nllama_memory_recurrent, layer 38: dev = MTL0\nllama_memory_recurrent, layer 39: dev = MTL0\nllama_memory_recurrent: MTL0 RS buffer size = 55.37 MiB\nsched_reserve: MTL0 compute buffer size = 123.63 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 41,
"totalLayers": 41,
"inferenceMs": 109.4386660042801,
"durationMs": 134.49054199736565,
"tokensPerSecond": 18.2750765613479,
"inputTokens": 63,
"outputTokens": 2,
"outputLimitReached": false,
"rssBytes": 4531257344,
"cpuThreads": 4,
"totalMs": 1250.871707990882,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "q4",
"device": "mps",
"status": "pass",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "chat"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
"device": "mps",
"precision": "q4",
"provider": "llama.cpp Metal",
"loadMs": 1116.3811659935163,
"acceleration": "partial",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer 0 assigned to device MTL0, is_swa = 0\nload_tensors: layer 1 assigned to device MTL0, is_swa = 0\nload_tensors: layer 2 assigned to device MTL0, is_swa = 0\nload_tensors: layer 3 assigned to device MTL0, is_swa = 0\nload_tensors: layer 4 assigned to device MTL0, is_swa = 0\nload_tensors: layer 5 assigned to device MTL0, is_swa = 0\nload_tensors: layer 6 assigned to device MTL0, is_swa = 0\nload_tensors: layer 7 assigned to device MTL0, is_swa = 0\nload_tensors: layer 8 assigned to device MTL0, is_swa = 0\nload_tensors: layer 9 assigned to device MTL0, is_swa = 0\nload_tensors: layer 10 assigned to device MTL0, is_swa = 0\nload_tensors: layer 11 assigned to device MTL0, is_swa = 0\nload_tensors: layer 12 assigned to device MTL0, is_swa = 0\nload_tensors: layer 13 assigned to device MTL0, is_swa = 0\nload_tensors: layer 14 assigned to device MTL0, is_swa = 0\nload_tensors: layer 15 assigned to device MTL0, is_swa = 0\nload_tensors: layer 16 assigned to device MTL0, is_swa = 0\nload_tensors: layer 17 assigned to device MTL0, is_swa = 0\nload_tensors: layer 18 assigned to device MTL0, is_swa = 0\nload_tensors: layer 19 assigned to device MTL0, is_swa = 0\nload_tensors: layer 20 assigned to device MTL0, is_swa = 0\nload_tensors: layer 21 assigned to device MTL0, is_swa = 0\nload_tensors: layer 22 assigned to device MTL0, is_swa = 0\nload_tensors: layer 23 assigned to device MTL0, is_swa = 0\nload_tensors: layer 24 assigned to device MTL0, is_swa = 0\nload_tensors: layer 25 assigned to device MTL0, is_swa = 0\nload_tensors: layer 26 assigned to device MTL0, is_swa = 0\nload_tensors: layer 27 assigned to device MTL0, is_swa = 0\nload_tensors: layer 28 assigned to device MTL0, is_swa = 0\nload_tensors: layer 29 assigned to device MTL0, is_swa = 0\nload_tensors: layer 30 assigned to device MTL0, is_swa = 0\nload_tensors: layer 31 assigned to device MTL0, is_swa = 0\nload_tensors: layer 32 assigned to device MTL0, is_swa = 0\nload_tensors: layer 33 assigned to device MTL0, is_swa = 0\nload_tensors: layer 34 assigned to device MTL0, is_swa = 0\nload_tensors: layer 35 assigned to device MTL0, is_swa = 0\nload_tensors: layer 36 assigned to device MTL0, is_swa = 0\nload_tensors: layer 37 assigned to device MTL0, is_swa = 0\nload_tensors: layer 38 assigned to device MTL0, is_swa = 0\nload_tensors: layer 39 assigned to device MTL0, is_swa = 0\nload_tensors: layer 40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 120.59 MiB\nload_tensors: MTL0_Mapped model buffer size = 4031.55 MiB\nllama_kv_cache: layer 5: dev = MTL0\nllama_kv_cache: layer 15: dev = MTL0\nllama_kv_cache: layer 25: dev = MTL0\nllama_kv_cache: layer 35: dev = MTL0\nllama_kv_cache: MTL0 KV buffer size = 64.00 MiB\nllama_memory_recurrent, layer 0: dev = MTL0\nllama_memory_recurrent, layer 1: dev = MTL0\nllama_memory_recurrent, layer 2: dev = MTL0\nllama_memory_recurrent, layer 3: dev = MTL0\nllama_memory_recurrent, layer 4: dev = MTL0\nllama_memory_recurrent, layer 6: dev = MTL0\nllama_memory_recurrent, layer 7: dev = MTL0\nllama_memory_recurrent, layer 8: dev = MTL0\nllama_memory_recurrent, layer 9: dev = MTL0\nllama_memory_recurrent, layer 10: dev = MTL0\nllama_memory_recurrent, layer 11: dev = MTL0\nllama_memory_recurrent, layer 12: dev = MTL0\nllama_memory_recurrent, layer 13: dev = MTL0\nllama_memory_recurrent, layer 14: dev = MTL0\nllama_memory_recurrent, layer 16: dev = MTL0\nllama_memory_recurrent, layer 17: dev = MTL0\nllama_memory_recurrent, layer 18: dev = MTL0\nllama_memory_recurrent, layer 19: dev = MTL0\nllama_memory_recurrent, layer 20: dev = MTL0\nllama_memory_recurrent, layer 21: dev = MTL0\nllama_memory_recurrent, layer 22: dev = MTL0\nllama_memory_recurrent, layer 23: dev = MTL0\nllama_memory_recurrent, layer 24: dev = MTL0\nllama_memory_recurrent, layer 26: dev = MTL0\nllama_memory_recurrent, layer 27: dev = MTL0\nllama_memory_recurrent, layer 28: dev = MTL0\nllama_memory_recurrent, layer 29: dev = MTL0\nllama_memory_recurrent, layer 30: dev = MTL0\nllama_memory_recurrent, layer 31: dev = MTL0\nllama_memory_recurrent, layer 32: dev = MTL0\nllama_memory_recurrent, layer 33: dev = MTL0\nllama_memory_recurrent, layer 34: dev = MTL0\nllama_memory_recurrent, layer 36: dev = MTL0\nllama_memory_recurrent, layer 37: dev = MTL0\nllama_memory_recurrent, layer 38: dev = MTL0\nllama_memory_recurrent, layer 39: dev = MTL0\nllama_memory_recurrent: MTL0 RS buffer size = 55.37 MiB\nsched_reserve: MTL0 compute buffer size = 123.63 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 41,
"totalLayers": 41,
"inferenceMs": 216.89704199525295,
"durationMs": 233.35537499951897,
"tokensPerSecond": 9.220964848583654,
"inputTokens": 68,
"outputTokens": 2,
"outputLimitReached": false,
"rssBytes": 4533026816,
"cpuThreads": 4,
"totalMs": 1349.7365409930353,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "q4",
"device": "mps",
"status": "pass",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "hola"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
"device": "mps",
"precision": "q4",
"provider": "llama.cpp Metal",
"loadMs": 1116.3811659935163,
"acceleration": "partial",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer 0 assigned to device MTL0, is_swa = 0\nload_tensors: layer 1 assigned to device MTL0, is_swa = 0\nload_tensors: layer 2 assigned to device MTL0, is_swa = 0\nload_tensors: layer 3 assigned to device MTL0, is_swa = 0\nload_tensors: layer 4 assigned to device MTL0, is_swa = 0\nload_tensors: layer 5 assigned to device MTL0, is_swa = 0\nload_tensors: layer 6 assigned to device MTL0, is_swa = 0\nload_tensors: layer 7 assigned to device MTL0, is_swa = 0\nload_tensors: layer 8 assigned to device MTL0, is_swa = 0\nload_tensors: layer 9 assigned to device MTL0, is_swa = 0\nload_tensors: layer 10 assigned to device MTL0, is_swa = 0\nload_tensors: layer 11 assigned to device MTL0, is_swa = 0\nload_tensors: layer 12 assigned to device MTL0, is_swa = 0\nload_tensors: layer 13 assigned to device MTL0, is_swa = 0\nload_tensors: layer 14 assigned to device MTL0, is_swa = 0\nload_tensors: layer 15 assigned to device MTL0, is_swa = 0\nload_tensors: layer 16 assigned to device MTL0, is_swa = 0\nload_tensors: layer 17 assigned to device MTL0, is_swa = 0\nload_tensors: layer 18 assigned to device MTL0, is_swa = 0\nload_tensors: layer 19 assigned to device MTL0, is_swa = 0\nload_tensors: layer 20 assigned to device MTL0, is_swa = 0\nload_tensors: layer 21 assigned to device MTL0, is_swa = 0\nload_tensors: layer 22 assigned to device MTL0, is_swa = 0\nload_tensors: layer 23 assigned to device MTL0, is_swa = 0\nload_tensors: layer 24 assigned to device MTL0, is_swa = 0\nload_tensors: layer 25 assigned to device MTL0, is_swa = 0\nload_tensors: layer 26 assigned to device MTL0, is_swa = 0\nload_tensors: layer 27 assigned to device MTL0, is_swa = 0\nload_tensors: layer 28 assigned to device MTL0, is_swa = 0\nload_tensors: layer 29 assigned to device MTL0, is_swa = 0\nload_tensors: layer 30 assigned to device MTL0, is_swa = 0\nload_tensors: layer 31 assigned to device MTL0, is_swa = 0\nload_tensors: layer 32 assigned to device MTL0, is_swa = 0\nload_tensors: layer 33 assigned to device MTL0, is_swa = 0\nload_tensors: layer 34 assigned to device MTL0, is_swa = 0\nload_tensors: layer 35 assigned to device MTL0, is_swa = 0\nload_tensors: layer 36 assigned to device MTL0, is_swa = 0\nload_tensors: layer 37 assigned to device MTL0, is_swa = 0\nload_tensors: layer 38 assigned to device MTL0, is_swa = 0\nload_tensors: layer 39 assigned to device MTL0, is_swa = 0\nload_tensors: layer 40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 120.59 MiB\nload_tensors: MTL0_Mapped model buffer size = 4031.55 MiB\nllama_kv_cache: layer 5: dev = MTL0\nllama_kv_cache: layer 15: dev = MTL0\nllama_kv_cache: layer 25: dev = MTL0\nllama_kv_cache: layer 35: dev = MTL0\nllama_kv_cache: MTL0 KV buffer size = 64.00 MiB\nllama_memory_recurrent, layer 0: dev = MTL0\nllama_memory_recurrent, layer 1: dev = MTL0\nllama_memory_recurrent, layer 2: dev = MTL0\nllama_memory_recurrent, layer 3: dev = MTL0\nllama_memory_recurrent, layer 4: dev = MTL0\nllama_memory_recurrent, layer 6: dev = MTL0\nllama_memory_recurrent, layer 7: dev = MTL0\nllama_memory_recurrent, layer 8: dev = MTL0\nllama_memory_recurrent, layer 9: dev = MTL0\nllama_memory_recurrent, layer 10: dev = MTL0\nllama_memory_recurrent, layer 11: dev = MTL0\nllama_memory_recurrent, layer 12: dev = MTL0\nllama_memory_recurrent, layer 13: dev = MTL0\nllama_memory_recurrent, layer 14: dev = MTL0\nllama_memory_recurrent, layer 16: dev = MTL0\nllama_memory_recurrent, layer 17: dev = MTL0\nllama_memory_recurrent, layer 18: dev = MTL0\nllama_memory_recurrent, layer 19: dev = MTL0\nllama_memory_recurrent, layer 20: dev = MTL0\nllama_memory_recurrent, layer 21: dev = MTL0\nllama_memory_recurrent, layer 22: dev = MTL0\nllama_memory_recurrent, layer 23: dev = MTL0\nllama_memory_recurrent, layer 24: dev = MTL0\nllama_memory_recurrent, layer 26: dev = MTL0\nllama_memory_recurrent, layer 27: dev = MTL0\nllama_memory_recurrent, layer 28: dev = MTL0\nllama_memory_recurrent, layer 29: dev = MTL0\nllama_memory_recurrent, layer 30: dev = MTL0\nllama_memory_recurrent, layer 31: dev = MTL0\nllama_memory_recurrent, layer 32: dev = MTL0\nllama_memory_recurrent, layer 33: dev = MTL0\nllama_memory_recurrent, layer 34: dev = MTL0\nllama_memory_recurrent, layer 36: dev = MTL0\nllama_memory_recurrent, layer 37: dev = MTL0\nllama_memory_recurrent, layer 38: dev = MTL0\nllama_memory_recurrent, layer 39: dev = MTL0\nllama_memory_recurrent: MTL0 RS buffer size = 55.37 MiB\nsched_reserve: MTL0 compute buffer size = 123.63 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 41,
"totalLayers": 41,
"inferenceMs": 96.98383299837587,
"durationMs": 111.57320799975423,
"tokensPerSecond": 20.62199377120404,
"inputTokens": 69,
"outputTokens": 2,
"outputLimitReached": false,
"rssBytes": 4533059584,
"cpuThreads": 4,
"totalMs": 1227.9543739932706,
"grade": "fast",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "q4",
"device": "mps",
"status": "pass",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "2026-11-04"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
"device": "mps",
"precision": "q4",
"provider": "llama.cpp Metal",
"loadMs": 1116.3811659935163,
"acceleration": "partial",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer 0 assigned to device MTL0, is_swa = 0\nload_tensors: layer 1 assigned to device MTL0, is_swa = 0\nload_tensors: layer 2 assigned to device MTL0, is_swa = 0\nload_tensors: layer 3 assigned to device MTL0, is_swa = 0\nload_tensors: layer 4 assigned to device MTL0, is_swa = 0\nload_tensors: layer 5 assigned to device MTL0, is_swa = 0\nload_tensors: layer 6 assigned to device MTL0, is_swa = 0\nload_tensors: layer 7 assigned to device MTL0, is_swa = 0\nload_tensors: layer 8 assigned to device MTL0, is_swa = 0\nload_tensors: layer 9 assigned to device MTL0, is_swa = 0\nload_tensors: layer 10 assigned to device MTL0, is_swa = 0\nload_tensors: layer 11 assigned to device MTL0, is_swa = 0\nload_tensors: layer 12 assigned to device MTL0, is_swa = 0\nload_tensors: layer 13 assigned to device MTL0, is_swa = 0\nload_tensors: layer 14 assigned to device MTL0, is_swa = 0\nload_tensors: layer 15 assigned to device MTL0, is_swa = 0\nload_tensors: layer 16 assigned to device MTL0, is_swa = 0\nload_tensors: layer 17 assigned to device MTL0, is_swa = 0\nload_tensors: layer 18 assigned to device MTL0, is_swa = 0\nload_tensors: layer 19 assigned to device MTL0, is_swa = 0\nload_tensors: layer 20 assigned to device MTL0, is_swa = 0\nload_tensors: layer 21 assigned to device MTL0, is_swa = 0\nload_tensors: layer 22 assigned to device MTL0, is_swa = 0\nload_tensors: layer 23 assigned to device MTL0, is_swa = 0\nload_tensors: layer 24 assigned to device MTL0, is_swa = 0\nload_tensors: layer 25 assigned to device MTL0, is_swa = 0\nload_tensors: layer 26 assigned to device MTL0, is_swa = 0\nload_tensors: layer 27 assigned to device MTL0, is_swa = 0\nload_tensors: layer 28 assigned to device MTL0, is_swa = 0\nload_tensors: layer 29 assigned to device MTL0, is_swa = 0\nload_tensors: layer 30 assigned to device MTL0, is_swa = 0\nload_tensors: layer 31 assigned to device MTL0, is_swa = 0\nload_tensors: layer 32 assigned to device MTL0, is_swa = 0\nload_tensors: layer 33 assigned to device MTL0, is_swa = 0\nload_tensors: layer 34 assigned to device MTL0, is_swa = 0\nload_tensors: layer 35 assigned to device MTL0, is_swa = 0\nload_tensors: layer 36 assigned to device MTL0, is_swa = 0\nload_tensors: layer 37 assigned to device MTL0, is_swa = 0\nload_tensors: layer 38 assigned to device MTL0, is_swa = 0\nload_tensors: layer 39 assigned to device MTL0, is_swa = 0\nload_tensors: layer 40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 120.59 MiB\nload_tensors: MTL0_Mapped model buffer size = 4031.55 MiB\nllama_kv_cache: layer 5: dev = MTL0\nllama_kv_cache: layer 15: dev = MTL0\nllama_kv_cache: layer 25: dev = MTL0\nllama_kv_cache: layer 35: dev = MTL0\nllama_kv_cache: MTL0 KV buffer size = 64.00 MiB\nllama_memory_recurrent, layer 0: dev = MTL0\nllama_memory_recurrent, layer 1: dev = MTL0\nllama_memory_recurrent, layer 2: dev = MTL0\nllama_memory_recurrent, layer 3: dev = MTL0\nllama_memory_recurrent, layer 4: dev = MTL0\nllama_memory_recurrent, layer 6: dev = MTL0\nllama_memory_recurrent, layer 7: dev = MTL0\nllama_memory_recurrent, layer 8: dev = MTL0\nllama_memory_recurrent, layer 9: dev = MTL0\nllama_memory_recurrent, layer 10: dev = MTL0\nllama_memory_recurrent, layer 11: dev = MTL0\nllama_memory_recurrent, layer 12: dev = MTL0\nllama_memory_recurrent, layer 13: dev = MTL0\nllama_memory_recurrent, layer 14: dev = MTL0\nllama_memory_recurrent, layer 16: dev = MTL0\nllama_memory_recurrent, layer 17: dev = MTL0\nllama_memory_recurrent, layer 18: dev = MTL0\nllama_memory_recurrent, layer 19: dev = MTL0\nllama_memory_recurrent, layer 20: dev = MTL0\nllama_memory_recurrent, layer 21: dev = MTL0\nllama_memory_recurrent, layer 22: dev = MTL0\nllama_memory_recurrent, layer 23: dev = MTL0\nllama_memory_recurrent, layer 24: dev = MTL0\nllama_memory_recurrent, layer 26: dev = MTL0\nllama_memory_recurrent, layer 27: dev = MTL0\nllama_memory_recurrent, layer 28: dev = MTL0\nllama_memory_recurrent, layer 29: dev = MTL0\nllama_memory_recurrent, layer 30: dev = MTL0\nllama_memory_recurrent, layer 31: dev = MTL0\nllama_memory_recurrent, layer 32: dev = MTL0\nllama_memory_recurrent, layer 33: dev = MTL0\nllama_memory_recurrent, layer 34: dev = MTL0\nllama_memory_recurrent, layer 36: dev = MTL0\nllama_memory_recurrent, layer 37: dev = MTL0\nllama_memory_recurrent, layer 38: dev = MTL0\nllama_memory_recurrent, layer 39: dev = MTL0\nllama_memory_recurrent: MTL0 RS buffer size = 55.37 MiB\nsched_reserve: MTL0 compute buffer size = 123.63 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 41,
"totalLayers": 41,
"inferenceMs": 165.19404200516874,
"durationMs": 179.3914159934502,
"tokensPerSecond": 42.374409603591985,
"inputTokens": 72,
"outputTokens": 7,
"outputLimitReached": false,
"rssBytes": 4533174272,
"cpuThreads": 4,
"totalMs": 1295.7725819869665,
"grade": "fast",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "q4",
"device": "mps",
"status": "pass",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "Unknown"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
"device": "mps",
"precision": "q4",
"provider": "llama.cpp Metal",
"loadMs": 1116.3811659935163,
"acceleration": "partial",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer 0 assigned to device MTL0, is_swa = 0\nload_tensors: layer 1 assigned to device MTL0, is_swa = 0\nload_tensors: layer 2 assigned to device MTL0, is_swa = 0\nload_tensors: layer 3 assigned to device MTL0, is_swa = 0\nload_tensors: layer 4 assigned to device MTL0, is_swa = 0\nload_tensors: layer 5 assigned to device MTL0, is_swa = 0\nload_tensors: layer 6 assigned to device MTL0, is_swa = 0\nload_tensors: layer 7 assigned to device MTL0, is_swa = 0\nload_tensors: layer 8 assigned to device MTL0, is_swa = 0\nload_tensors: layer 9 assigned to device MTL0, is_swa = 0\nload_tensors: layer 10 assigned to device MTL0, is_swa = 0\nload_tensors: layer 11 assigned to device MTL0, is_swa = 0\nload_tensors: layer 12 assigned to device MTL0, is_swa = 0\nload_tensors: layer 13 assigned to device MTL0, is_swa = 0\nload_tensors: layer 14 assigned to device MTL0, is_swa = 0\nload_tensors: layer 15 assigned to device MTL0, is_swa = 0\nload_tensors: layer 16 assigned to device MTL0, is_swa = 0\nload_tensors: layer 17 assigned to device MTL0, is_swa = 0\nload_tensors: layer 18 assigned to device MTL0, is_swa = 0\nload_tensors: layer 19 assigned to device MTL0, is_swa = 0\nload_tensors: layer 20 assigned to device MTL0, is_swa = 0\nload_tensors: layer 21 assigned to device MTL0, is_swa = 0\nload_tensors: layer 22 assigned to device MTL0, is_swa = 0\nload_tensors: layer 23 assigned to device MTL0, is_swa = 0\nload_tensors: layer 24 assigned to device MTL0, is_swa = 0\nload_tensors: layer 25 assigned to device MTL0, is_swa = 0\nload_tensors: layer 26 assigned to device MTL0, is_swa = 0\nload_tensors: layer 27 assigned to device MTL0, is_swa = 0\nload_tensors: layer 28 assigned to device MTL0, is_swa = 0\nload_tensors: layer 29 assigned to device MTL0, is_swa = 0\nload_tensors: layer 30 assigned to device MTL0, is_swa = 0\nload_tensors: layer 31 assigned to device MTL0, is_swa = 0\nload_tensors: layer 32 assigned to device MTL0, is_swa = 0\nload_tensors: layer 33 assigned to device MTL0, is_swa = 0\nload_tensors: layer 34 assigned to device MTL0, is_swa = 0\nload_tensors: layer 35 assigned to device MTL0, is_swa = 0\nload_tensors: layer 36 assigned to device MTL0, is_swa = 0\nload_tensors: layer 37 assigned to device MTL0, is_swa = 0\nload_tensors: layer 38 assigned to device MTL0, is_swa = 0\nload_tensors: layer 39 assigned to device MTL0, is_swa = 0\nload_tensors: layer 40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 120.59 MiB\nload_tensors: MTL0_Mapped model buffer size = 4031.55 MiB\nllama_kv_cache: layer 5: dev = MTL0\nllama_kv_cache: layer 15: dev = MTL0\nllama_kv_cache: layer 25: dev = MTL0\nllama_kv_cache: layer 35: dev = MTL0\nllama_kv_cache: MTL0 KV buffer size = 64.00 MiB\nllama_memory_recurrent, layer 0: dev = MTL0\nllama_memory_recurrent, layer 1: dev = MTL0\nllama_memory_recurrent, layer 2: dev = MTL0\nllama_memory_recurrent, layer 3: dev = MTL0\nllama_memory_recurrent, layer 4: dev = MTL0\nllama_memory_recurrent, layer 6: dev = MTL0\nllama_memory_recurrent, layer 7: dev = MTL0\nllama_memory_recurrent, layer 8: dev = MTL0\nllama_memory_recurrent, layer 9: dev = MTL0\nllama_memory_recurrent, layer 10: dev = MTL0\nllama_memory_recurrent, layer 11: dev = MTL0\nllama_memory_recurrent, layer 12: dev = MTL0\nllama_memory_recurrent, layer 13: dev = MTL0\nllama_memory_recurrent, layer 14: dev = MTL0\nllama_memory_recurrent, layer 16: dev = MTL0\nllama_memory_recurrent, layer 17: dev = MTL0\nllama_memory_recurrent, layer 18: dev = MTL0\nllama_memory_recurrent, layer 19: dev = MTL0\nllama_memory_recurrent, layer 20: dev = MTL0\nllama_memory_recurrent, layer 21: dev = MTL0\nllama_memory_recurrent, layer 22: dev = MTL0\nllama_memory_recurrent, layer 23: dev = MTL0\nllama_memory_recurrent, layer 24: dev = MTL0\nllama_memory_recurrent, layer 26: dev = MTL0\nllama_memory_recurrent, layer 27: dev = MTL0\nllama_memory_recurrent, layer 28: dev = MTL0\nllama_memory_recurrent, layer 29: dev = MTL0\nllama_memory_recurrent, layer 30: dev = MTL0\nllama_memory_recurrent, layer 31: dev = MTL0\nllama_memory_recurrent, layer 32: dev = MTL0\nllama_memory_recurrent, layer 33: dev = MTL0\nllama_memory_recurrent, layer 34: dev = MTL0\nllama_memory_recurrent, layer 36: dev = MTL0\nllama_memory_recurrent, layer 37: dev = MTL0\nllama_memory_recurrent, layer 38: dev = MTL0\nllama_memory_recurrent, layer 39: dev = MTL0\nllama_memory_recurrent: MTL0 RS buffer size = 55.37 MiB\nsched_reserve: MTL0 compute buffer size = 123.63 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 41,
"totalLayers": 41,
"inferenceMs": 104.0853340018657,
"durationMs": 128.2852499934961,
"tokensPerSecond": 19.215002950983955,
"inputTokens": 74,
"outputTokens": 2,
"outputLimitReached": false,
"rssBytes": 4534435840,
"cpuThreads": 4,
"totalMs": 1244.6664159870124,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "q4",
"device": "mps",
"status": "pass",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "Paris"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
"device": "mps",
"precision": "q4",
"provider": "llama.cpp Metal",
"loadMs": 925.3593329995056,
"acceleration": "partial",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer 0 assigned to device MTL0, is_swa = 0\nload_tensors: layer 1 assigned to device MTL0, is_swa = 0\nload_tensors: layer 2 assigned to device MTL0, is_swa = 0\nload_tensors: layer 3 assigned to device MTL0, is_swa = 0\nload_tensors: layer 4 assigned to device MTL0, is_swa = 0\nload_tensors: layer 5 assigned to device MTL0, is_swa = 0\nload_tensors: layer 6 assigned to device MTL0, is_swa = 0\nload_tensors: layer 7 assigned to device MTL0, is_swa = 0\nload_tensors: layer 8 assigned to device MTL0, is_swa = 0\nload_tensors: layer 9 assigned to device MTL0, is_swa = 0\nload_tensors: layer 10 assigned to device MTL0, is_swa = 0\nload_tensors: layer 11 assigned to device MTL0, is_swa = 0\nload_tensors: layer 12 assigned to device MTL0, is_swa = 0\nload_tensors: layer 13 assigned to device MTL0, is_swa = 0\nload_tensors: layer 14 assigned to device MTL0, is_swa = 0\nload_tensors: layer 15 assigned to device MTL0, is_swa = 0\nload_tensors: layer 16 assigned to device MTL0, is_swa = 0\nload_tensors: layer 17 assigned to device MTL0, is_swa = 0\nload_tensors: layer 18 assigned to device MTL0, is_swa = 0\nload_tensors: layer 19 assigned to device MTL0, is_swa = 0\nload_tensors: layer 20 assigned to device MTL0, is_swa = 0\nload_tensors: layer 21 assigned to device MTL0, is_swa = 0\nload_tensors: layer 22 assigned to device MTL0, is_swa = 0\nload_tensors: layer 23 assigned to device MTL0, is_swa = 0\nload_tensors: layer 24 assigned to device MTL0, is_swa = 0\nload_tensors: layer 25 assigned to device MTL0, is_swa = 0\nload_tensors: layer 26 assigned to device MTL0, is_swa = 0\nload_tensors: layer 27 assigned to device MTL0, is_swa = 0\nload_tensors: layer 28 assigned to device MTL0, is_swa = 0\nload_tensors: layer 29 assigned to device MTL0, is_swa = 0\nload_tensors: layer 30 assigned to device MTL0, is_swa = 0\nload_tensors: layer 31 assigned to device MTL0, is_swa = 0\nload_tensors: layer 32 assigned to device MTL0, is_swa = 0\nload_tensors: layer 33 assigned to device MTL0, is_swa = 0\nload_tensors: layer 34 assigned to device MTL0, is_swa = 0\nload_tensors: layer 35 assigned to device MTL0, is_swa = 0\nload_tensors: layer 36 assigned to device MTL0, is_swa = 0\nload_tensors: layer 37 assigned to device MTL0, is_swa = 0\nload_tensors: layer 38 assigned to device MTL0, is_swa = 0\nload_tensors: layer 39 assigned to device MTL0, is_swa = 0\nload_tensors: layer 40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 120.59 MiB\nload_tensors: MTL0_Mapped model buffer size = 4031.55 MiB\nllama_kv_cache: layer 5: dev = MTL0\nllama_kv_cache: layer 15: dev = MTL0\nllama_kv_cache: layer 25: dev = MTL0\nllama_kv_cache: layer 35: dev = MTL0\nllama_kv_cache: MTL0 KV buffer size = 64.00 MiB\nllama_memory_recurrent, layer 0: dev = MTL0\nllama_memory_recurrent, layer 1: dev = MTL0\nllama_memory_recurrent, layer 2: dev = MTL0\nllama_memory_recurrent, layer 3: dev = MTL0\nllama_memory_recurrent, layer 4: dev = MTL0\nllama_memory_recurrent, layer 6: dev = MTL0\nllama_memory_recurrent, layer 7: dev = MTL0\nllama_memory_recurrent, layer 8: dev = MTL0\nllama_memory_recurrent, layer 9: dev = MTL0\nllama_memory_recurrent, layer 10: dev = MTL0\nllama_memory_recurrent, layer 11: dev = MTL0\nllama_memory_recurrent, layer 12: dev = MTL0\nllama_memory_recurrent, layer 13: dev = MTL0\nllama_memory_recurrent, layer 14: dev = MTL0\nllama_memory_recurrent, layer 16: dev = MTL0\nllama_memory_recurrent, layer 17: dev = MTL0\nllama_memory_recurrent, layer 18: dev = MTL0\nllama_memory_recurrent, layer 19: dev = MTL0\nllama_memory_recurrent, layer 20: dev = MTL0\nllama_memory_recurrent, layer 21: dev = MTL0\nllama_memory_recurrent, layer 22: dev = MTL0\nllama_memory_recurrent, layer 23: dev = MTL0\nllama_memory_recurrent, layer 24: dev = MTL0\nllama_memory_recurrent, layer 26: dev = MTL0\nllama_memory_recurrent, layer 27: dev = MTL0\nllama_memory_recurrent, layer 28: dev = MTL0\nllama_memory_recurrent, layer 29: dev = MTL0\nllama_memory_recurrent, layer 30: dev = MTL0\nllama_memory_recurrent, layer 31: dev = MTL0\nllama_memory_recurrent, layer 32: dev = MTL0\nllama_memory_recurrent, layer 33: dev = MTL0\nllama_memory_recurrent, layer 34: dev = MTL0\nllama_memory_recurrent, layer 36: dev = MTL0\nllama_memory_recurrent, layer 37: dev = MTL0\nllama_memory_recurrent, layer 38: dev = MTL0\nllama_memory_recurrent, layer 39: dev = MTL0\nllama_memory_recurrent: MTL0 RS buffer size = 55.37 MiB\nsched_reserve: MTL0 compute buffer size = 123.63 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 41,
"totalLayers": 41,
"inferenceMs": 126.38304200663697,
"durationMs": 171.15341699536657,
"tokensPerSecond": 15.82490790097433,
"inputTokens": 63,
"outputTokens": 2,
"outputLimitReached": false,
"rssBytes": 4530241536,
"cpuThreads": 4,
"totalMs": 1096.5127499948721,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "q4",
"device": "mps",
"status": "pass",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "56"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
"device": "mps",
"precision": "q4",
"provider": "llama.cpp Metal",
"loadMs": 925.3593329995056,
"acceleration": "partial",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer 0 assigned to device MTL0, is_swa = 0\nload_tensors: layer 1 assigned to device MTL0, is_swa = 0\nload_tensors: layer 2 assigned to device MTL0, is_swa = 0\nload_tensors: layer 3 assigned to device MTL0, is_swa = 0\nload_tensors: layer 4 assigned to device MTL0, is_swa = 0\nload_tensors: layer 5 assigned to device MTL0, is_swa = 0\nload_tensors: layer 6 assigned to device MTL0, is_swa = 0\nload_tensors: layer 7 assigned to device MTL0, is_swa = 0\nload_tensors: layer 8 assigned to device MTL0, is_swa = 0\nload_tensors: layer 9 assigned to device MTL0, is_swa = 0\nload_tensors: layer 10 assigned to device MTL0, is_swa = 0\nload_tensors: layer 11 assigned to device MTL0, is_swa = 0\nload_tensors: layer 12 assigned to device MTL0, is_swa = 0\nload_tensors: layer 13 assigned to device MTL0, is_swa = 0\nload_tensors: layer 14 assigned to device MTL0, is_swa = 0\nload_tensors: layer 15 assigned to device MTL0, is_swa = 0\nload_tensors: layer 16 assigned to device MTL0, is_swa = 0\nload_tensors: layer 17 assigned to device MTL0, is_swa = 0\nload_tensors: layer 18 assigned to device MTL0, is_swa = 0\nload_tensors: layer 19 assigned to device MTL0, is_swa = 0\nload_tensors: layer 20 assigned to device MTL0, is_swa = 0\nload_tensors: layer 21 assigned to device MTL0, is_swa = 0\nload_tensors: layer 22 assigned to device MTL0, is_swa = 0\nload_tensors: layer 23 assigned to device MTL0, is_swa = 0\nload_tensors: layer 24 assigned to device MTL0, is_swa = 0\nload_tensors: layer 25 assigned to device MTL0, is_swa = 0\nload_tensors: layer 26 assigned to device MTL0, is_swa = 0\nload_tensors: layer 27 assigned to device MTL0, is_swa = 0\nload_tensors: layer 28 assigned to device MTL0, is_swa = 0\nload_tensors: layer 29 assigned to device MTL0, is_swa = 0\nload_tensors: layer 30 assigned to device MTL0, is_swa = 0\nload_tensors: layer 31 assigned to device MTL0, is_swa = 0\nload_tensors: layer 32 assigned to device MTL0, is_swa = 0\nload_tensors: layer 33 assigned to device MTL0, is_swa = 0\nload_tensors: layer 34 assigned to device MTL0, is_swa = 0\nload_tensors: layer 35 assigned to device MTL0, is_swa = 0\nload_tensors: layer 36 assigned to device MTL0, is_swa = 0\nload_tensors: layer 37 assigned to device MTL0, is_swa = 0\nload_tensors: layer 38 assigned to device MTL0, is_swa = 0\nload_tensors: layer 39 assigned to device MTL0, is_swa = 0\nload_tensors: layer 40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 120.59 MiB\nload_tensors: MTL0_Mapped model buffer size = 4031.55 MiB\nllama_kv_cache: layer 5: dev = MTL0\nllama_kv_cache: layer 15: dev = MTL0\nllama_kv_cache: layer 25: dev = MTL0\nllama_kv_cache: layer 35: dev = MTL0\nllama_kv_cache: MTL0 KV buffer size = 64.00 MiB\nllama_memory_recurrent, layer 0: dev = MTL0\nllama_memory_recurrent, layer 1: dev = MTL0\nllama_memory_recurrent, layer 2: dev = MTL0\nllama_memory_recurrent, layer 3: dev = MTL0\nllama_memory_recurrent, layer 4: dev = MTL0\nllama_memory_recurrent, layer 6: dev = MTL0\nllama_memory_recurrent, layer 7: dev = MTL0\nllama_memory_recurrent, layer 8: dev = MTL0\nllama_memory_recurrent, layer 9: dev = MTL0\nllama_memory_recurrent, layer 10: dev = MTL0\nllama_memory_recurrent, layer 11: dev = MTL0\nllama_memory_recurrent, layer 12: dev = MTL0\nllama_memory_recurrent, layer 13: dev = MTL0\nllama_memory_recurrent, layer 14: dev = MTL0\nllama_memory_recurrent, layer 16: dev = MTL0\nllama_memory_recurrent, layer 17: dev = MTL0\nllama_memory_recurrent, layer 18: dev = MTL0\nllama_memory_recurrent, layer 19: dev = MTL0\nllama_memory_recurrent, layer 20: dev = MTL0\nllama_memory_recurrent, layer 21: dev = MTL0\nllama_memory_recurrent, layer 22: dev = MTL0\nllama_memory_recurrent, layer 23: dev = MTL0\nllama_memory_recurrent, layer 24: dev = MTL0\nllama_memory_recurrent, layer 26: dev = MTL0\nllama_memory_recurrent, layer 27: dev = MTL0\nllama_memory_recurrent, layer 28: dev = MTL0\nllama_memory_recurrent, layer 29: dev = MTL0\nllama_memory_recurrent, layer 30: dev = MTL0\nllama_memory_recurrent, layer 31: dev = MTL0\nllama_memory_recurrent, layer 32: dev = MTL0\nllama_memory_recurrent, layer 33: dev = MTL0\nllama_memory_recurrent, layer 34: dev = MTL0\nllama_memory_recurrent, layer 36: dev = MTL0\nllama_memory_recurrent, layer 37: dev = MTL0\nllama_memory_recurrent, layer 38: dev = MTL0\nllama_memory_recurrent, layer 39: dev = MTL0\nllama_memory_recurrent: MTL0 RS buffer size = 55.37 MiB\nsched_reserve: MTL0 compute buffer size = 123.63 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 41,
"totalLayers": 41,
"inferenceMs": 96.4625420019729,
"durationMs": 107.24950000440003,
"tokensPerSecond": 20.733436611685963,
"inputTokens": 63,
"outputTokens": 2,
"outputLimitReached": false,
"rssBytes": 4530307072,
"cpuThreads": 4,
"totalMs": 1032.6088330039056,
"grade": "fast",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "q4",
"device": "mps",
"status": "pass",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "chat"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
"device": "mps",
"precision": "q4",
"provider": "llama.cpp Metal",
"loadMs": 925.3593329995056,
"acceleration": "partial",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer 0 assigned to device MTL0, is_swa = 0\nload_tensors: layer 1 assigned to device MTL0, is_swa = 0\nload_tensors: layer 2 assigned to device MTL0, is_swa = 0\nload_tensors: layer 3 assigned to device MTL0, is_swa = 0\nload_tensors: layer 4 assigned to device MTL0, is_swa = 0\nload_tensors: layer 5 assigned to device MTL0, is_swa = 0\nload_tensors: layer 6 assigned to device MTL0, is_swa = 0\nload_tensors: layer 7 assigned to device MTL0, is_swa = 0\nload_tensors: layer 8 assigned to device MTL0, is_swa = 0\nload_tensors: layer 9 assigned to device MTL0, is_swa = 0\nload_tensors: layer 10 assigned to device MTL0, is_swa = 0\nload_tensors: layer 11 assigned to device MTL0, is_swa = 0\nload_tensors: layer 12 assigned to device MTL0, is_swa = 0\nload_tensors: layer 13 assigned to device MTL0, is_swa = 0\nload_tensors: layer 14 assigned to device MTL0, is_swa = 0\nload_tensors: layer 15 assigned to device MTL0, is_swa = 0\nload_tensors: layer 16 assigned to device MTL0, is_swa = 0\nload_tensors: layer 17 assigned to device MTL0, is_swa = 0\nload_tensors: layer 18 assigned to device MTL0, is_swa = 0\nload_tensors: layer 19 assigned to device MTL0, is_swa = 0\nload_tensors: layer 20 assigned to device MTL0, is_swa = 0\nload_tensors: layer 21 assigned to device MTL0, is_swa = 0\nload_tensors: layer 22 assigned to device MTL0, is_swa = 0\nload_tensors: layer 23 assigned to device MTL0, is_swa = 0\nload_tensors: layer 24 assigned to device MTL0, is_swa = 0\nload_tensors: layer 25 assigned to device MTL0, is_swa = 0\nload_tensors: layer 26 assigned to device MTL0, is_swa = 0\nload_tensors: layer 27 assigned to device MTL0, is_swa = 0\nload_tensors: layer 28 assigned to device MTL0, is_swa = 0\nload_tensors: layer 29 assigned to device MTL0, is_swa = 0\nload_tensors: layer 30 assigned to device MTL0, is_swa = 0\nload_tensors: layer 31 assigned to device MTL0, is_swa = 0\nload_tensors: layer 32 assigned to device MTL0, is_swa = 0\nload_tensors: layer 33 assigned to device MTL0, is_swa = 0\nload_tensors: layer 34 assigned to device MTL0, is_swa = 0\nload_tensors: layer 35 assigned to device MTL0, is_swa = 0\nload_tensors: layer 36 assigned to device MTL0, is_swa = 0\nload_tensors: layer 37 assigned to device MTL0, is_swa = 0\nload_tensors: layer 38 assigned to device MTL0, is_swa = 0\nload_tensors: layer 39 assigned to device MTL0, is_swa = 0\nload_tensors: layer 40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 120.59 MiB\nload_tensors: MTL0_Mapped model buffer size = 4031.55 MiB\nllama_kv_cache: layer 5: dev = MTL0\nllama_kv_cache: layer 15: dev = MTL0\nllama_kv_cache: layer 25: dev = MTL0\nllama_kv_cache: layer 35: dev = MTL0\nllama_kv_cache: MTL0 KV buffer size = 64.00 MiB\nllama_memory_recurrent, layer 0: dev = MTL0\nllama_memory_recurrent, layer 1: dev = MTL0\nllama_memory_recurrent, layer 2: dev = MTL0\nllama_memory_recurrent, layer 3: dev = MTL0\nllama_memory_recurrent, layer 4: dev = MTL0\nllama_memory_recurrent, layer 6: dev = MTL0\nllama_memory_recurrent, layer 7: dev = MTL0\nllama_memory_recurrent, layer 8: dev = MTL0\nllama_memory_recurrent, layer 9: dev = MTL0\nllama_memory_recurrent, layer 10: dev = MTL0\nllama_memory_recurrent, layer 11: dev = MTL0\nllama_memory_recurrent, layer 12: dev = MTL0\nllama_memory_recurrent, layer 13: dev = MTL0\nllama_memory_recurrent, layer 14: dev = MTL0\nllama_memory_recurrent, layer 16: dev = MTL0\nllama_memory_recurrent, layer 17: dev = MTL0\nllama_memory_recurrent, layer 18: dev = MTL0\nllama_memory_recurrent, layer 19: dev = MTL0\nllama_memory_recurrent, layer 20: dev = MTL0\nllama_memory_recurrent, layer 21: dev = MTL0\nllama_memory_recurrent, layer 22: dev = MTL0\nllama_memory_recurrent, layer 23: dev = MTL0\nllama_memory_recurrent, layer 24: dev = MTL0\nllama_memory_recurrent, layer 26: dev = MTL0\nllama_memory_recurrent, layer 27: dev = MTL0\nllama_memory_recurrent, layer 28: dev = MTL0\nllama_memory_recurrent, layer 29: dev = MTL0\nllama_memory_recurrent, layer 30: dev = MTL0\nllama_memory_recurrent, layer 31: dev = MTL0\nllama_memory_recurrent, layer 32: dev = MTL0\nllama_memory_recurrent, layer 33: dev = MTL0\nllama_memory_recurrent, layer 34: dev = MTL0\nllama_memory_recurrent, layer 36: dev = MTL0\nllama_memory_recurrent, layer 37: dev = MTL0\nllama_memory_recurrent, layer 38: dev = MTL0\nllama_memory_recurrent, layer 39: dev = MTL0\nllama_memory_recurrent: MTL0 RS buffer size = 55.37 MiB\nsched_reserve: MTL0 compute buffer size = 123.63 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 41,
"totalLayers": 41,
"inferenceMs": 98.73174999665935,
"durationMs": 112.96225000114646,
"tokensPerSecond": 20.256908239423197,
"inputTokens": 68,
"outputTokens": 2,
"outputLimitReached": false,
"rssBytes": 4532060160,
"cpuThreads": 4,
"totalMs": 1038.321583000652,
"grade": "fast",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "q4",
"device": "mps",
"status": "pass",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "hola"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
"device": "mps",
"precision": "q4",
"provider": "llama.cpp Metal",
"loadMs": 925.3593329995056,
"acceleration": "partial",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer 0 assigned to device MTL0, is_swa = 0\nload_tensors: layer 1 assigned to device MTL0, is_swa = 0\nload_tensors: layer 2 assigned to device MTL0, is_swa = 0\nload_tensors: layer 3 assigned to device MTL0, is_swa = 0\nload_tensors: layer 4 assigned to device MTL0, is_swa = 0\nload_tensors: layer 5 assigned to device MTL0, is_swa = 0\nload_tensors: layer 6 assigned to device MTL0, is_swa = 0\nload_tensors: layer 7 assigned to device MTL0, is_swa = 0\nload_tensors: layer 8 assigned to device MTL0, is_swa = 0\nload_tensors: layer 9 assigned to device MTL0, is_swa = 0\nload_tensors: layer 10 assigned to device MTL0, is_swa = 0\nload_tensors: layer 11 assigned to device MTL0, is_swa = 0\nload_tensors: layer 12 assigned to device MTL0, is_swa = 0\nload_tensors: layer 13 assigned to device MTL0, is_swa = 0\nload_tensors: layer 14 assigned to device MTL0, is_swa = 0\nload_tensors: layer 15 assigned to device MTL0, is_swa = 0\nload_tensors: layer 16 assigned to device MTL0, is_swa = 0\nload_tensors: layer 17 assigned to device MTL0, is_swa = 0\nload_tensors: layer 18 assigned to device MTL0, is_swa = 0\nload_tensors: layer 19 assigned to device MTL0, is_swa = 0\nload_tensors: layer 20 assigned to device MTL0, is_swa = 0\nload_tensors: layer 21 assigned to device MTL0, is_swa = 0\nload_tensors: layer 22 assigned to device MTL0, is_swa = 0\nload_tensors: layer 23 assigned to device MTL0, is_swa = 0\nload_tensors: layer 24 assigned to device MTL0, is_swa = 0\nload_tensors: layer 25 assigned to device MTL0, is_swa = 0\nload_tensors: layer 26 assigned to device MTL0, is_swa = 0\nload_tensors: layer 27 assigned to device MTL0, is_swa = 0\nload_tensors: layer 28 assigned to device MTL0, is_swa = 0\nload_tensors: layer 29 assigned to device MTL0, is_swa = 0\nload_tensors: layer 30 assigned to device MTL0, is_swa = 0\nload_tensors: layer 31 assigned to device MTL0, is_swa = 0\nload_tensors: layer 32 assigned to device MTL0, is_swa = 0\nload_tensors: layer 33 assigned to device MTL0, is_swa = 0\nload_tensors: layer 34 assigned to device MTL0, is_swa = 0\nload_tensors: layer 35 assigned to device MTL0, is_swa = 0\nload_tensors: layer 36 assigned to device MTL0, is_swa = 0\nload_tensors: layer 37 assigned to device MTL0, is_swa = 0\nload_tensors: layer 38 assigned to device MTL0, is_swa = 0\nload_tensors: layer 39 assigned to device MTL0, is_swa = 0\nload_tensors: layer 40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 120.59 MiB\nload_tensors: MTL0_Mapped model buffer size = 4031.55 MiB\nllama_kv_cache: layer 5: dev = MTL0\nllama_kv_cache: layer 15: dev = MTL0\nllama_kv_cache: layer 25: dev = MTL0\nllama_kv_cache: layer 35: dev = MTL0\nllama_kv_cache: MTL0 KV buffer size = 64.00 MiB\nllama_memory_recurrent, layer 0: dev = MTL0\nllama_memory_recurrent, layer 1: dev = MTL0\nllama_memory_recurrent, layer 2: dev = MTL0\nllama_memory_recurrent, layer 3: dev = MTL0\nllama_memory_recurrent, layer 4: dev = MTL0\nllama_memory_recurrent, layer 6: dev = MTL0\nllama_memory_recurrent, layer 7: dev = MTL0\nllama_memory_recurrent, layer 8: dev = MTL0\nllama_memory_recurrent, layer 9: dev = MTL0\nllama_memory_recurrent, layer 10: dev = MTL0\nllama_memory_recurrent, layer 11: dev = MTL0\nllama_memory_recurrent, layer 12: dev = MTL0\nllama_memory_recurrent, layer 13: dev = MTL0\nllama_memory_recurrent, layer 14: dev = MTL0\nllama_memory_recurrent, layer 16: dev = MTL0\nllama_memory_recurrent, layer 17: dev = MTL0\nllama_memory_recurrent, layer 18: dev = MTL0\nllama_memory_recurrent, layer 19: dev = MTL0\nllama_memory_recurrent, layer 20: dev = MTL0\nllama_memory_recurrent, layer 21: dev = MTL0\nllama_memory_recurrent, layer 22: dev = MTL0\nllama_memory_recurrent, layer 23: dev = MTL0\nllama_memory_recurrent, layer 24: dev = MTL0\nllama_memory_recurrent, layer 26: dev = MTL0\nllama_memory_recurrent, layer 27: dev = MTL0\nllama_memory_recurrent, layer 28: dev = MTL0\nllama_memory_recurrent, layer 29: dev = MTL0\nllama_memory_recurrent, layer 30: dev = MTL0\nllama_memory_recurrent, layer 31: dev = MTL0\nllama_memory_recurrent, layer 32: dev = MTL0\nllama_memory_recurrent, layer 33: dev = MTL0\nllama_memory_recurrent, layer 34: dev = MTL0\nllama_memory_recurrent, layer 36: dev = MTL0\nllama_memory_recurrent, layer 37: dev = MTL0\nllama_memory_recurrent, layer 38: dev = MTL0\nllama_memory_recurrent, layer 39: dev = MTL0\nllama_memory_recurrent: MTL0 RS buffer size = 55.37 MiB\nsched_reserve: MTL0 compute buffer size = 123.63 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 41,
"totalLayers": 41,
"inferenceMs": 101.52958300022874,
"durationMs": 119.607291999273,
"tokensPerSecond": 19.698692153551878,
"inputTokens": 69,
"outputTokens": 2,
"outputLimitReached": false,
"rssBytes": 4533436416,
"cpuThreads": 4,
"totalMs": 1044.9666249987786,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "q4",
"device": "mps",
"status": "pass",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "2026-11-04"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
"device": "mps",
"precision": "q4",
"provider": "llama.cpp Metal",
"loadMs": 925.3593329995056,
"acceleration": "partial",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer 0 assigned to device MTL0, is_swa = 0\nload_tensors: layer 1 assigned to device MTL0, is_swa = 0\nload_tensors: layer 2 assigned to device MTL0, is_swa = 0\nload_tensors: layer 3 assigned to device MTL0, is_swa = 0\nload_tensors: layer 4 assigned to device MTL0, is_swa = 0\nload_tensors: layer 5 assigned to device MTL0, is_swa = 0\nload_tensors: layer 6 assigned to device MTL0, is_swa = 0\nload_tensors: layer 7 assigned to device MTL0, is_swa = 0\nload_tensors: layer 8 assigned to device MTL0, is_swa = 0\nload_tensors: layer 9 assigned to device MTL0, is_swa = 0\nload_tensors: layer 10 assigned to device MTL0, is_swa = 0\nload_tensors: layer 11 assigned to device MTL0, is_swa = 0\nload_tensors: layer 12 assigned to device MTL0, is_swa = 0\nload_tensors: layer 13 assigned to device MTL0, is_swa = 0\nload_tensors: layer 14 assigned to device MTL0, is_swa = 0\nload_tensors: layer 15 assigned to device MTL0, is_swa = 0\nload_tensors: layer 16 assigned to device MTL0, is_swa = 0\nload_tensors: layer 17 assigned to device MTL0, is_swa = 0\nload_tensors: layer 18 assigned to device MTL0, is_swa = 0\nload_tensors: layer 19 assigned to device MTL0, is_swa = 0\nload_tensors: layer 20 assigned to device MTL0, is_swa = 0\nload_tensors: layer 21 assigned to device MTL0, is_swa = 0\nload_tensors: layer 22 assigned to device MTL0, is_swa = 0\nload_tensors: layer 23 assigned to device MTL0, is_swa = 0\nload_tensors: layer 24 assigned to device MTL0, is_swa = 0\nload_tensors: layer 25 assigned to device MTL0, is_swa = 0\nload_tensors: layer 26 assigned to device MTL0, is_swa = 0\nload_tensors: layer 27 assigned to device MTL0, is_swa = 0\nload_tensors: layer 28 assigned to device MTL0, is_swa = 0\nload_tensors: layer 29 assigned to device MTL0, is_swa = 0\nload_tensors: layer 30 assigned to device MTL0, is_swa = 0\nload_tensors: layer 31 assigned to device MTL0, is_swa = 0\nload_tensors: layer 32 assigned to device MTL0, is_swa = 0\nload_tensors: layer 33 assigned to device MTL0, is_swa = 0\nload_tensors: layer 34 assigned to device MTL0, is_swa = 0\nload_tensors: layer 35 assigned to device MTL0, is_swa = 0\nload_tensors: layer 36 assigned to device MTL0, is_swa = 0\nload_tensors: layer 37 assigned to device MTL0, is_swa = 0\nload_tensors: layer 38 assigned to device MTL0, is_swa = 0\nload_tensors: layer 39 assigned to device MTL0, is_swa = 0\nload_tensors: layer 40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 120.59 MiB\nload_tensors: MTL0_Mapped model buffer size = 4031.55 MiB\nllama_kv_cache: layer 5: dev = MTL0\nllama_kv_cache: layer 15: dev = MTL0\nllama_kv_cache: layer 25: dev = MTL0\nllama_kv_cache: layer 35: dev = MTL0\nllama_kv_cache: MTL0 KV buffer size = 64.00 MiB\nllama_memory_recurrent, layer 0: dev = MTL0\nllama_memory_recurrent, layer 1: dev = MTL0\nllama_memory_recurrent, layer 2: dev = MTL0\nllama_memory_recurrent, layer 3: dev = MTL0\nllama_memory_recurrent, layer 4: dev = MTL0\nllama_memory_recurrent, layer 6: dev = MTL0\nllama_memory_recurrent, layer 7: dev = MTL0\nllama_memory_recurrent, layer 8: dev = MTL0\nllama_memory_recurrent, layer 9: dev = MTL0\nllama_memory_recurrent, layer 10: dev = MTL0\nllama_memory_recurrent, layer 11: dev = MTL0\nllama_memory_recurrent, layer 12: dev = MTL0\nllama_memory_recurrent, layer 13: dev = MTL0\nllama_memory_recurrent, layer 14: dev = MTL0\nllama_memory_recurrent, layer 16: dev = MTL0\nllama_memory_recurrent, layer 17: dev = MTL0\nllama_memory_recurrent, layer 18: dev = MTL0\nllama_memory_recurrent, layer 19: dev = MTL0\nllama_memory_recurrent, layer 20: dev = MTL0\nllama_memory_recurrent, layer 21: dev = MTL0\nllama_memory_recurrent, layer 22: dev = MTL0\nllama_memory_recurrent, layer 23: dev = MTL0\nllama_memory_recurrent, layer 24: dev = MTL0\nllama_memory_recurrent, layer 26: dev = MTL0\nllama_memory_recurrent, layer 27: dev = MTL0\nllama_memory_recurrent, layer 28: dev = MTL0\nllama_memory_recurrent, layer 29: dev = MTL0\nllama_memory_recurrent, layer 30: dev = MTL0\nllama_memory_recurrent, layer 31: dev = MTL0\nllama_memory_recurrent, layer 32: dev = MTL0\nllama_memory_recurrent, layer 33: dev = MTL0\nllama_memory_recurrent, layer 34: dev = MTL0\nllama_memory_recurrent, layer 36: dev = MTL0\nllama_memory_recurrent, layer 37: dev = MTL0\nllama_memory_recurrent, layer 38: dev = MTL0\nllama_memory_recurrent, layer 39: dev = MTL0\nllama_memory_recurrent: MTL0 RS buffer size = 55.37 MiB\nsched_reserve: MTL0 compute buffer size = 123.63 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 41,
"totalLayers": 41,
"inferenceMs": 170.4097090041614,
"durationMs": 191.52187500003492,
"tokensPerSecond": 41.07747170572927,
"inputTokens": 72,
"outputTokens": 7,
"outputLimitReached": false,
"rssBytes": 4533567488,
"cpuThreads": 4,
"totalMs": 1116.8812079995405,
"grade": "fast",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "q4",
"device": "mps",
"status": "pass",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "Unknown"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
"device": "mps",
"precision": "q4",
"provider": "llama.cpp Metal",
"loadMs": 925.3593329995056,
"acceleration": "partial",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer 0 assigned to device MTL0, is_swa = 0\nload_tensors: layer 1 assigned to device MTL0, is_swa = 0\nload_tensors: layer 2 assigned to device MTL0, is_swa = 0\nload_tensors: layer 3 assigned to device MTL0, is_swa = 0\nload_tensors: layer 4 assigned to device MTL0, is_swa = 0\nload_tensors: layer 5 assigned to device MTL0, is_swa = 0\nload_tensors: layer 6 assigned to device MTL0, is_swa = 0\nload_tensors: layer 7 assigned to device MTL0, is_swa = 0\nload_tensors: layer 8 assigned to device MTL0, is_swa = 0\nload_tensors: layer 9 assigned to device MTL0, is_swa = 0\nload_tensors: layer 10 assigned to device MTL0, is_swa = 0\nload_tensors: layer 11 assigned to device MTL0, is_swa = 0\nload_tensors: layer 12 assigned to device MTL0, is_swa = 0\nload_tensors: layer 13 assigned to device MTL0, is_swa = 0\nload_tensors: layer 14 assigned to device MTL0, is_swa = 0\nload_tensors: layer 15 assigned to device MTL0, is_swa = 0\nload_tensors: layer 16 assigned to device MTL0, is_swa = 0\nload_tensors: layer 17 assigned to device MTL0, is_swa = 0\nload_tensors: layer 18 assigned to device MTL0, is_swa = 0\nload_tensors: layer 19 assigned to device MTL0, is_swa = 0\nload_tensors: layer 20 assigned to device MTL0, is_swa = 0\nload_tensors: layer 21 assigned to device MTL0, is_swa = 0\nload_tensors: layer 22 assigned to device MTL0, is_swa = 0\nload_tensors: layer 23 assigned to device MTL0, is_swa = 0\nload_tensors: layer 24 assigned to device MTL0, is_swa = 0\nload_tensors: layer 25 assigned to device MTL0, is_swa = 0\nload_tensors: layer 26 assigned to device MTL0, is_swa = 0\nload_tensors: layer 27 assigned to device MTL0, is_swa = 0\nload_tensors: layer 28 assigned to device MTL0, is_swa = 0\nload_tensors: layer 29 assigned to device MTL0, is_swa = 0\nload_tensors: layer 30 assigned to device MTL0, is_swa = 0\nload_tensors: layer 31 assigned to device MTL0, is_swa = 0\nload_tensors: layer 32 assigned to device MTL0, is_swa = 0\nload_tensors: layer 33 assigned to device MTL0, is_swa = 0\nload_tensors: layer 34 assigned to device MTL0, is_swa = 0\nload_tensors: layer 35 assigned to device MTL0, is_swa = 0\nload_tensors: layer 36 assigned to device MTL0, is_swa = 0\nload_tensors: layer 37 assigned to device MTL0, is_swa = 0\nload_tensors: layer 38 assigned to device MTL0, is_swa = 0\nload_tensors: layer 39 assigned to device MTL0, is_swa = 0\nload_tensors: layer 40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 120.59 MiB\nload_tensors: MTL0_Mapped model buffer size = 4031.55 MiB\nllama_kv_cache: layer 5: dev = MTL0\nllama_kv_cache: layer 15: dev = MTL0\nllama_kv_cache: layer 25: dev = MTL0\nllama_kv_cache: layer 35: dev = MTL0\nllama_kv_cache: MTL0 KV buffer size = 64.00 MiB\nllama_memory_recurrent, layer 0: dev = MTL0\nllama_memory_recurrent, layer 1: dev = MTL0\nllama_memory_recurrent, layer 2: dev = MTL0\nllama_memory_recurrent, layer 3: dev = MTL0\nllama_memory_recurrent, layer 4: dev = MTL0\nllama_memory_recurrent, layer 6: dev = MTL0\nllama_memory_recurrent, layer 7: dev = MTL0\nllama_memory_recurrent, layer 8: dev = MTL0\nllama_memory_recurrent, layer 9: dev = MTL0\nllama_memory_recurrent, layer 10: dev = MTL0\nllama_memory_recurrent, layer 11: dev = MTL0\nllama_memory_recurrent, layer 12: dev = MTL0\nllama_memory_recurrent, layer 13: dev = MTL0\nllama_memory_recurrent, layer 14: dev = MTL0\nllama_memory_recurrent, layer 16: dev = MTL0\nllama_memory_recurrent, layer 17: dev = MTL0\nllama_memory_recurrent, layer 18: dev = MTL0\nllama_memory_recurrent, layer 19: dev = MTL0\nllama_memory_recurrent, layer 20: dev = MTL0\nllama_memory_recurrent, layer 21: dev = MTL0\nllama_memory_recurrent, layer 22: dev = MTL0\nllama_memory_recurrent, layer 23: dev = MTL0\nllama_memory_recurrent, layer 24: dev = MTL0\nllama_memory_recurrent, layer 26: dev = MTL0\nllama_memory_recurrent, layer 27: dev = MTL0\nllama_memory_recurrent, layer 28: dev = MTL0\nllama_memory_recurrent, layer 29: dev = MTL0\nllama_memory_recurrent, layer 30: dev = MTL0\nllama_memory_recurrent, layer 31: dev = MTL0\nllama_memory_recurrent, layer 32: dev = MTL0\nllama_memory_recurrent, layer 33: dev = MTL0\nllama_memory_recurrent, layer 34: dev = MTL0\nllama_memory_recurrent, layer 36: dev = MTL0\nllama_memory_recurrent, layer 37: dev = MTL0\nllama_memory_recurrent, layer 38: dev = MTL0\nllama_memory_recurrent, layer 39: dev = MTL0\nllama_memory_recurrent: MTL0 RS buffer size = 55.37 MiB\nsched_reserve: MTL0 compute buffer size = 123.63 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 41,
"totalLayers": 41,
"inferenceMs": 100.23616599937668,
"durationMs": 125.43708400335163,
"tokensPerSecond": 19.952878086063638,
"inputTokens": 74,
"outputTokens": 2,
"outputLimitReached": false,
"rssBytes": 4533633024,
"cpuThreads": 4,
"totalMs": 1050.7964170028572,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "q4",
"device": "mps",
"status": "pass",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "Paris"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
"device": "mps",
"precision": "q4",
"provider": "llama.cpp Metal",
"loadMs": 1236.3420840047183,
"acceleration": "partial",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer 0 assigned to device MTL0, is_swa = 0\nload_tensors: layer 1 assigned to device MTL0, is_swa = 0\nload_tensors: layer 2 assigned to device MTL0, is_swa = 0\nload_tensors: layer 3 assigned to device MTL0, is_swa = 0\nload_tensors: layer 4 assigned to device MTL0, is_swa = 0\nload_tensors: layer 5 assigned to device MTL0, is_swa = 0\nload_tensors: layer 6 assigned to device MTL0, is_swa = 0\nload_tensors: layer 7 assigned to device MTL0, is_swa = 0\nload_tensors: layer 8 assigned to device MTL0, is_swa = 0\nload_tensors: layer 9 assigned to device MTL0, is_swa = 0\nload_tensors: layer 10 assigned to device MTL0, is_swa = 0\nload_tensors: layer 11 assigned to device MTL0, is_swa = 0\nload_tensors: layer 12 assigned to device MTL0, is_swa = 0\nload_tensors: layer 13 assigned to device MTL0, is_swa = 0\nload_tensors: layer 14 assigned to device MTL0, is_swa = 0\nload_tensors: layer 15 assigned to device MTL0, is_swa = 0\nload_tensors: layer 16 assigned to device MTL0, is_swa = 0\nload_tensors: layer 17 assigned to device MTL0, is_swa = 0\nload_tensors: layer 18 assigned to device MTL0, is_swa = 0\nload_tensors: layer 19 assigned to device MTL0, is_swa = 0\nload_tensors: layer 20 assigned to device MTL0, is_swa = 0\nload_tensors: layer 21 assigned to device MTL0, is_swa = 0\nload_tensors: layer 22 assigned to device MTL0, is_swa = 0\nload_tensors: layer 23 assigned to device MTL0, is_swa = 0\nload_tensors: layer 24 assigned to device MTL0, is_swa = 0\nload_tensors: layer 25 assigned to device MTL0, is_swa = 0\nload_tensors: layer 26 assigned to device MTL0, is_swa = 0\nload_tensors: layer 27 assigned to device MTL0, is_swa = 0\nload_tensors: layer 28 assigned to device MTL0, is_swa = 0\nload_tensors: layer 29 assigned to device MTL0, is_swa = 0\nload_tensors: layer 30 assigned to device MTL0, is_swa = 0\nload_tensors: layer 31 assigned to device MTL0, is_swa = 0\nload_tensors: layer 32 assigned to device MTL0, is_swa = 0\nload_tensors: layer 33 assigned to device MTL0, is_swa = 0\nload_tensors: layer 34 assigned to device MTL0, is_swa = 0\nload_tensors: layer 35 assigned to device MTL0, is_swa = 0\nload_tensors: layer 36 assigned to device MTL0, is_swa = 0\nload_tensors: layer 37 assigned to device MTL0, is_swa = 0\nload_tensors: layer 38 assigned to device MTL0, is_swa = 0\nload_tensors: layer 39 assigned to device MTL0, is_swa = 0\nload_tensors: layer 40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 120.59 MiB\nload_tensors: MTL0_Mapped model buffer size = 4031.55 MiB\nllama_kv_cache: layer 5: dev = MTL0\nllama_kv_cache: layer 15: dev = MTL0\nllama_kv_cache: layer 25: dev = MTL0\nllama_kv_cache: layer 35: dev = MTL0\nllama_kv_cache: MTL0 KV buffer size = 64.00 MiB\nllama_memory_recurrent, layer 0: dev = MTL0\nllama_memory_recurrent, layer 1: dev = MTL0\nllama_memory_recurrent, layer 2: dev = MTL0\nllama_memory_recurrent, layer 3: dev = MTL0\nllama_memory_recurrent, layer 4: dev = MTL0\nllama_memory_recurrent, layer 6: dev = MTL0\nllama_memory_recurrent, layer 7: dev = MTL0\nllama_memory_recurrent, layer 8: dev = MTL0\nllama_memory_recurrent, layer 9: dev = MTL0\nllama_memory_recurrent, layer 10: dev = MTL0\nllama_memory_recurrent, layer 11: dev = MTL0\nllama_memory_recurrent, layer 12: dev = MTL0\nllama_memory_recurrent, layer 13: dev = MTL0\nllama_memory_recurrent, layer 14: dev = MTL0\nllama_memory_recurrent, layer 16: dev = MTL0\nllama_memory_recurrent, layer 17: dev = MTL0\nllama_memory_recurrent, layer 18: dev = MTL0\nllama_memory_recurrent, layer 19: dev = MTL0\nllama_memory_recurrent, layer 20: dev = MTL0\nllama_memory_recurrent, layer 21: dev = MTL0\nllama_memory_recurrent, layer 22: dev = MTL0\nllama_memory_recurrent, layer 23: dev = MTL0\nllama_memory_recurrent, layer 24: dev = MTL0\nllama_memory_recurrent, layer 26: dev = MTL0\nllama_memory_recurrent, layer 27: dev = MTL0\nllama_memory_recurrent, layer 28: dev = MTL0\nllama_memory_recurrent, layer 29: dev = MTL0\nllama_memory_recurrent, layer 30: dev = MTL0\nllama_memory_recurrent, layer 31: dev = MTL0\nllama_memory_recurrent, layer 32: dev = MTL0\nllama_memory_recurrent, layer 33: dev = MTL0\nllama_memory_recurrent, layer 34: dev = MTL0\nllama_memory_recurrent, layer 36: dev = MTL0\nllama_memory_recurrent, layer 37: dev = MTL0\nllama_memory_recurrent, layer 38: dev = MTL0\nllama_memory_recurrent, layer 39: dev = MTL0\nllama_memory_recurrent: MTL0 RS buffer size = 55.37 MiB\nsched_reserve: MTL0 compute buffer size = 123.63 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 41,
"totalLayers": 41,
"inferenceMs": 97.12879200378666,
"durationMs": 107.12225000315811,
"tokensPerSecond": 20.591216659237645,
"inputTokens": 63,
"outputTokens": 2,
"outputLimitReached": false,
"rssBytes": 4531077120,
"cpuThreads": 4,
"totalMs": 1343.4643340078765,
"grade": "fast",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "q4",
"device": "mps",
"status": "pass",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "56"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
"device": "mps",
"precision": "q4",
"provider": "llama.cpp Metal",
"loadMs": 1236.3420840047183,
"acceleration": "partial",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer 0 assigned to device MTL0, is_swa = 0\nload_tensors: layer 1 assigned to device MTL0, is_swa = 0\nload_tensors: layer 2 assigned to device MTL0, is_swa = 0\nload_tensors: layer 3 assigned to device MTL0, is_swa = 0\nload_tensors: layer 4 assigned to device MTL0, is_swa = 0\nload_tensors: layer 5 assigned to device MTL0, is_swa = 0\nload_tensors: layer 6 assigned to device MTL0, is_swa = 0\nload_tensors: layer 7 assigned to device MTL0, is_swa = 0\nload_tensors: layer 8 assigned to device MTL0, is_swa = 0\nload_tensors: layer 9 assigned to device MTL0, is_swa = 0\nload_tensors: layer 10 assigned to device MTL0, is_swa = 0\nload_tensors: layer 11 assigned to device MTL0, is_swa = 0\nload_tensors: layer 12 assigned to device MTL0, is_swa = 0\nload_tensors: layer 13 assigned to device MTL0, is_swa = 0\nload_tensors: layer 14 assigned to device MTL0, is_swa = 0\nload_tensors: layer 15 assigned to device MTL0, is_swa = 0\nload_tensors: layer 16 assigned to device MTL0, is_swa = 0\nload_tensors: layer 17 assigned to device MTL0, is_swa = 0\nload_tensors: layer 18 assigned to device MTL0, is_swa = 0\nload_tensors: layer 19 assigned to device MTL0, is_swa = 0\nload_tensors: layer 20 assigned to device MTL0, is_swa = 0\nload_tensors: layer 21 assigned to device MTL0, is_swa = 0\nload_tensors: layer 22 assigned to device MTL0, is_swa = 0\nload_tensors: layer 23 assigned to device MTL0, is_swa = 0\nload_tensors: layer 24 assigned to device MTL0, is_swa = 0\nload_tensors: layer 25 assigned to device MTL0, is_swa = 0\nload_tensors: layer 26 assigned to device MTL0, is_swa = 0\nload_tensors: layer 27 assigned to device MTL0, is_swa = 0\nload_tensors: layer 28 assigned to device MTL0, is_swa = 0\nload_tensors: layer 29 assigned to device MTL0, is_swa = 0\nload_tensors: layer 30 assigned to device MTL0, is_swa = 0\nload_tensors: layer 31 assigned to device MTL0, is_swa = 0\nload_tensors: layer 32 assigned to device MTL0, is_swa = 0\nload_tensors: layer 33 assigned to device MTL0, is_swa = 0\nload_tensors: layer 34 assigned to device MTL0, is_swa = 0\nload_tensors: layer 35 assigned to device MTL0, is_swa = 0\nload_tensors: layer 36 assigned to device MTL0, is_swa = 0\nload_tensors: layer 37 assigned to device MTL0, is_swa = 0\nload_tensors: layer 38 assigned to device MTL0, is_swa = 0\nload_tensors: layer 39 assigned to device MTL0, is_swa = 0\nload_tensors: layer 40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 120.59 MiB\nload_tensors: MTL0_Mapped model buffer size = 4031.55 MiB\nllama_kv_cache: layer 5: dev = MTL0\nllama_kv_cache: layer 15: dev = MTL0\nllama_kv_cache: layer 25: dev = MTL0\nllama_kv_cache: layer 35: dev = MTL0\nllama_kv_cache: MTL0 KV buffer size = 64.00 MiB\nllama_memory_recurrent, layer 0: dev = MTL0\nllama_memory_recurrent, layer 1: dev = MTL0\nllama_memory_recurrent, layer 2: dev = MTL0\nllama_memory_recurrent, layer 3: dev = MTL0\nllama_memory_recurrent, layer 4: dev = MTL0\nllama_memory_recurrent, layer 6: dev = MTL0\nllama_memory_recurrent, layer 7: dev = MTL0\nllama_memory_recurrent, layer 8: dev = MTL0\nllama_memory_recurrent, layer 9: dev = MTL0\nllama_memory_recurrent, layer 10: dev = MTL0\nllama_memory_recurrent, layer 11: dev = MTL0\nllama_memory_recurrent, layer 12: dev = MTL0\nllama_memory_recurrent, layer 13: dev = MTL0\nllama_memory_recurrent, layer 14: dev = MTL0\nllama_memory_recurrent, layer 16: dev = MTL0\nllama_memory_recurrent, layer 17: dev = MTL0\nllama_memory_recurrent, layer 18: dev = MTL0\nllama_memory_recurrent, layer 19: dev = MTL0\nllama_memory_recurrent, layer 20: dev = MTL0\nllama_memory_recurrent, layer 21: dev = MTL0\nllama_memory_recurrent, layer 22: dev = MTL0\nllama_memory_recurrent, layer 23: dev = MTL0\nllama_memory_recurrent, layer 24: dev = MTL0\nllama_memory_recurrent, layer 26: dev = MTL0\nllama_memory_recurrent, layer 27: dev = MTL0\nllama_memory_recurrent, layer 28: dev = MTL0\nllama_memory_recurrent, layer 29: dev = MTL0\nllama_memory_recurrent, layer 30: dev = MTL0\nllama_memory_recurrent, layer 31: dev = MTL0\nllama_memory_recurrent, layer 32: dev = MTL0\nllama_memory_recurrent, layer 33: dev = MTL0\nllama_memory_recurrent, layer 34: dev = MTL0\nllama_memory_recurrent, layer 36: dev = MTL0\nllama_memory_recurrent, layer 37: dev = MTL0\nllama_memory_recurrent, layer 38: dev = MTL0\nllama_memory_recurrent, layer 39: dev = MTL0\nllama_memory_recurrent: MTL0 RS buffer size = 55.37 MiB\nsched_reserve: MTL0 compute buffer size = 123.63 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 41,
"totalLayers": 41,
"inferenceMs": 93.80783399683423,
"durationMs": 230.77516599732917,
"tokensPerSecond": 21.3201809996753,
"inputTokens": 63,
"outputTokens": 2,
"outputLimitReached": false,
"rssBytes": 4532355072,
"cpuThreads": 4,
"totalMs": 1467.1172500020475,
"grade": "fast",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "q4",
"device": "mps",
"status": "pass",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "chat"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
"device": "mps",
"precision": "q4",
"provider": "llama.cpp Metal",
"loadMs": 1236.3420840047183,
"acceleration": "partial",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer 0 assigned to device MTL0, is_swa = 0\nload_tensors: layer 1 assigned to device MTL0, is_swa = 0\nload_tensors: layer 2 assigned to device MTL0, is_swa = 0\nload_tensors: layer 3 assigned to device MTL0, is_swa = 0\nload_tensors: layer 4 assigned to device MTL0, is_swa = 0\nload_tensors: layer 5 assigned to device MTL0, is_swa = 0\nload_tensors: layer 6 assigned to device MTL0, is_swa = 0\nload_tensors: layer 7 assigned to device MTL0, is_swa = 0\nload_tensors: layer 8 assigned to device MTL0, is_swa = 0\nload_tensors: layer 9 assigned to device MTL0, is_swa = 0\nload_tensors: layer 10 assigned to device MTL0, is_swa = 0\nload_tensors: layer 11 assigned to device MTL0, is_swa = 0\nload_tensors: layer 12 assigned to device MTL0, is_swa = 0\nload_tensors: layer 13 assigned to device MTL0, is_swa = 0\nload_tensors: layer 14 assigned to device MTL0, is_swa = 0\nload_tensors: layer 15 assigned to device MTL0, is_swa = 0\nload_tensors: layer 16 assigned to device MTL0, is_swa = 0\nload_tensors: layer 17 assigned to device MTL0, is_swa = 0\nload_tensors: layer 18 assigned to device MTL0, is_swa = 0\nload_tensors: layer 19 assigned to device MTL0, is_swa = 0\nload_tensors: layer 20 assigned to device MTL0, is_swa = 0\nload_tensors: layer 21 assigned to device MTL0, is_swa = 0\nload_tensors: layer 22 assigned to device MTL0, is_swa = 0\nload_tensors: layer 23 assigned to device MTL0, is_swa = 0\nload_tensors: layer 24 assigned to device MTL0, is_swa = 0\nload_tensors: layer 25 assigned to device MTL0, is_swa = 0\nload_tensors: layer 26 assigned to device MTL0, is_swa = 0\nload_tensors: layer 27 assigned to device MTL0, is_swa = 0\nload_tensors: layer 28 assigned to device MTL0, is_swa = 0\nload_tensors: layer 29 assigned to device MTL0, is_swa = 0\nload_tensors: layer 30 assigned to device MTL0, is_swa = 0\nload_tensors: layer 31 assigned to device MTL0, is_swa = 0\nload_tensors: layer 32 assigned to device MTL0, is_swa = 0\nload_tensors: layer 33 assigned to device MTL0, is_swa = 0\nload_tensors: layer 34 assigned to device MTL0, is_swa = 0\nload_tensors: layer 35 assigned to device MTL0, is_swa = 0\nload_tensors: layer 36 assigned to device MTL0, is_swa = 0\nload_tensors: layer 37 assigned to device MTL0, is_swa = 0\nload_tensors: layer 38 assigned to device MTL0, is_swa = 0\nload_tensors: layer 39 assigned to device MTL0, is_swa = 0\nload_tensors: layer 40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 120.59 MiB\nload_tensors: MTL0_Mapped model buffer size = 4031.55 MiB\nllama_kv_cache: layer 5: dev = MTL0\nllama_kv_cache: layer 15: dev = MTL0\nllama_kv_cache: layer 25: dev = MTL0\nllama_kv_cache: layer 35: dev = MTL0\nllama_kv_cache: MTL0 KV buffer size = 64.00 MiB\nllama_memory_recurrent, layer 0: dev = MTL0\nllama_memory_recurrent, layer 1: dev = MTL0\nllama_memory_recurrent, layer 2: dev = MTL0\nllama_memory_recurrent, layer 3: dev = MTL0\nllama_memory_recurrent, layer 4: dev = MTL0\nllama_memory_recurrent, layer 6: dev = MTL0\nllama_memory_recurrent, layer 7: dev = MTL0\nllama_memory_recurrent, layer 8: dev = MTL0\nllama_memory_recurrent, layer 9: dev = MTL0\nllama_memory_recurrent, layer 10: dev = MTL0\nllama_memory_recurrent, layer 11: dev = MTL0\nllama_memory_recurrent, layer 12: dev = MTL0\nllama_memory_recurrent, layer 13: dev = MTL0\nllama_memory_recurrent, layer 14: dev = MTL0\nllama_memory_recurrent, layer 16: dev = MTL0\nllama_memory_recurrent, layer 17: dev = MTL0\nllama_memory_recurrent, layer 18: dev = MTL0\nllama_memory_recurrent, layer 19: dev = MTL0\nllama_memory_recurrent, layer 20: dev = MTL0\nllama_memory_recurrent, layer 21: dev = MTL0\nllama_memory_recurrent, layer 22: dev = MTL0\nllama_memory_recurrent, layer 23: dev = MTL0\nllama_memory_recurrent, layer 24: dev = MTL0\nllama_memory_recurrent, layer 26: dev = MTL0\nllama_memory_recurrent, layer 27: dev = MTL0\nllama_memory_recurrent, layer 28: dev = MTL0\nllama_memory_recurrent, layer 29: dev = MTL0\nllama_memory_recurrent, layer 30: dev = MTL0\nllama_memory_recurrent, layer 31: dev = MTL0\nllama_memory_recurrent, layer 32: dev = MTL0\nllama_memory_recurrent, layer 33: dev = MTL0\nllama_memory_recurrent, layer 34: dev = MTL0\nllama_memory_recurrent, layer 36: dev = MTL0\nllama_memory_recurrent, layer 37: dev = MTL0\nllama_memory_recurrent, layer 38: dev = MTL0\nllama_memory_recurrent, layer 39: dev = MTL0\nllama_memory_recurrent: MTL0 RS buffer size = 55.37 MiB\nsched_reserve: MTL0 compute buffer size = 123.63 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 41,
"totalLayers": 41,
"inferenceMs": 97.01829199912027,
"durationMs": 106.974624999566,
"tokensPerSecond": 20.61466924214802,
"inputTokens": 68,
"outputTokens": 2,
"outputLimitReached": false,
"rssBytes": 4532961280,
"cpuThreads": 4,
"totalMs": 1343.3167090042843,
"grade": "fast",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "q4",
"device": "mps",
"status": "pass",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "hola"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
"device": "mps",
"precision": "q4",
"provider": "llama.cpp Metal",
"loadMs": 1236.3420840047183,
"acceleration": "partial",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer 0 assigned to device MTL0, is_swa = 0\nload_tensors: layer 1 assigned to device MTL0, is_swa = 0\nload_tensors: layer 2 assigned to device MTL0, is_swa = 0\nload_tensors: layer 3 assigned to device MTL0, is_swa = 0\nload_tensors: layer 4 assigned to device MTL0, is_swa = 0\nload_tensors: layer 5 assigned to device MTL0, is_swa = 0\nload_tensors: layer 6 assigned to device MTL0, is_swa = 0\nload_tensors: layer 7 assigned to device MTL0, is_swa = 0\nload_tensors: layer 8 assigned to device MTL0, is_swa = 0\nload_tensors: layer 9 assigned to device MTL0, is_swa = 0\nload_tensors: layer 10 assigned to device MTL0, is_swa = 0\nload_tensors: layer 11 assigned to device MTL0, is_swa = 0\nload_tensors: layer 12 assigned to device MTL0, is_swa = 0\nload_tensors: layer 13 assigned to device MTL0, is_swa = 0\nload_tensors: layer 14 assigned to device MTL0, is_swa = 0\nload_tensors: layer 15 assigned to device MTL0, is_swa = 0\nload_tensors: layer 16 assigned to device MTL0, is_swa = 0\nload_tensors: layer 17 assigned to device MTL0, is_swa = 0\nload_tensors: layer 18 assigned to device MTL0, is_swa = 0\nload_tensors: layer 19 assigned to device MTL0, is_swa = 0\nload_tensors: layer 20 assigned to device MTL0, is_swa = 0\nload_tensors: layer 21 assigned to device MTL0, is_swa = 0\nload_tensors: layer 22 assigned to device MTL0, is_swa = 0\nload_tensors: layer 23 assigned to device MTL0, is_swa = 0\nload_tensors: layer 24 assigned to device MTL0, is_swa = 0\nload_tensors: layer 25 assigned to device MTL0, is_swa = 0\nload_tensors: layer 26 assigned to device MTL0, is_swa = 0\nload_tensors: layer 27 assigned to device MTL0, is_swa = 0\nload_tensors: layer 28 assigned to device MTL0, is_swa = 0\nload_tensors: layer 29 assigned to device MTL0, is_swa = 0\nload_tensors: layer 30 assigned to device MTL0, is_swa = 0\nload_tensors: layer 31 assigned to device MTL0, is_swa = 0\nload_tensors: layer 32 assigned to device MTL0, is_swa = 0\nload_tensors: layer 33 assigned to device MTL0, is_swa = 0\nload_tensors: layer 34 assigned to device MTL0, is_swa = 0\nload_tensors: layer 35 assigned to device MTL0, is_swa = 0\nload_tensors: layer 36 assigned to device MTL0, is_swa = 0\nload_tensors: layer 37 assigned to device MTL0, is_swa = 0\nload_tensors: layer 38 assigned to device MTL0, is_swa = 0\nload_tensors: layer 39 assigned to device MTL0, is_swa = 0\nload_tensors: layer 40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 120.59 MiB\nload_tensors: MTL0_Mapped model buffer size = 4031.55 MiB\nllama_kv_cache: layer 5: dev = MTL0\nllama_kv_cache: layer 15: dev = MTL0\nllama_kv_cache: layer 25: dev = MTL0\nllama_kv_cache: layer 35: dev = MTL0\nllama_kv_cache: MTL0 KV buffer size = 64.00 MiB\nllama_memory_recurrent, layer 0: dev = MTL0\nllama_memory_recurrent, layer 1: dev = MTL0\nllama_memory_recurrent, layer 2: dev = MTL0\nllama_memory_recurrent, layer 3: dev = MTL0\nllama_memory_recurrent, layer 4: dev = MTL0\nllama_memory_recurrent, layer 6: dev = MTL0\nllama_memory_recurrent, layer 7: dev = MTL0\nllama_memory_recurrent, layer 8: dev = MTL0\nllama_memory_recurrent, layer 9: dev = MTL0\nllama_memory_recurrent, layer 10: dev = MTL0\nllama_memory_recurrent, layer 11: dev = MTL0\nllama_memory_recurrent, layer 12: dev = MTL0\nllama_memory_recurrent, layer 13: dev = MTL0\nllama_memory_recurrent, layer 14: dev = MTL0\nllama_memory_recurrent, layer 16: dev = MTL0\nllama_memory_recurrent, layer 17: dev = MTL0\nllama_memory_recurrent, layer 18: dev = MTL0\nllama_memory_recurrent, layer 19: dev = MTL0\nllama_memory_recurrent, layer 20: dev = MTL0\nllama_memory_recurrent, layer 21: dev = MTL0\nllama_memory_recurrent, layer 22: dev = MTL0\nllama_memory_recurrent, layer 23: dev = MTL0\nllama_memory_recurrent, layer 24: dev = MTL0\nllama_memory_recurrent, layer 26: dev = MTL0\nllama_memory_recurrent, layer 27: dev = MTL0\nllama_memory_recurrent, layer 28: dev = MTL0\nllama_memory_recurrent, layer 29: dev = MTL0\nllama_memory_recurrent, layer 30: dev = MTL0\nllama_memory_recurrent, layer 31: dev = MTL0\nllama_memory_recurrent, layer 32: dev = MTL0\nllama_memory_recurrent, layer 33: dev = MTL0\nllama_memory_recurrent, layer 34: dev = MTL0\nllama_memory_recurrent, layer 36: dev = MTL0\nllama_memory_recurrent, layer 37: dev = MTL0\nllama_memory_recurrent, layer 38: dev = MTL0\nllama_memory_recurrent, layer 39: dev = MTL0\nllama_memory_recurrent: MTL0 RS buffer size = 55.37 MiB\nsched_reserve: MTL0 compute buffer size = 123.63 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 41,
"totalLayers": 41,
"inferenceMs": 100.1509170018835,
"durationMs": 109.32479199982481,
"tokensPerSecond": 19.96986208286427,
"inputTokens": 69,
"outputTokens": 2,
"outputLimitReached": false,
"rssBytes": 4534452224,
"cpuThreads": 4,
"totalMs": 1345.6668760045432,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "q4",
"device": "mps",
"status": "pass",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "2026-11-04"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
"device": "mps",
"precision": "q4",
"provider": "llama.cpp Metal",
"loadMs": 1236.3420840047183,
"acceleration": "partial",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer 0 assigned to device MTL0, is_swa = 0\nload_tensors: layer 1 assigned to device MTL0, is_swa = 0\nload_tensors: layer 2 assigned to device MTL0, is_swa = 0\nload_tensors: layer 3 assigned to device MTL0, is_swa = 0\nload_tensors: layer 4 assigned to device MTL0, is_swa = 0\nload_tensors: layer 5 assigned to device MTL0, is_swa = 0\nload_tensors: layer 6 assigned to device MTL0, is_swa = 0\nload_tensors: layer 7 assigned to device MTL0, is_swa = 0\nload_tensors: layer 8 assigned to device MTL0, is_swa = 0\nload_tensors: layer 9 assigned to device MTL0, is_swa = 0\nload_tensors: layer 10 assigned to device MTL0, is_swa = 0\nload_tensors: layer 11 assigned to device MTL0, is_swa = 0\nload_tensors: layer 12 assigned to device MTL0, is_swa = 0\nload_tensors: layer 13 assigned to device MTL0, is_swa = 0\nload_tensors: layer 14 assigned to device MTL0, is_swa = 0\nload_tensors: layer 15 assigned to device MTL0, is_swa = 0\nload_tensors: layer 16 assigned to device MTL0, is_swa = 0\nload_tensors: layer 17 assigned to device MTL0, is_swa = 0\nload_tensors: layer 18 assigned to device MTL0, is_swa = 0\nload_tensors: layer 19 assigned to device MTL0, is_swa = 0\nload_tensors: layer 20 assigned to device MTL0, is_swa = 0\nload_tensors: layer 21 assigned to device MTL0, is_swa = 0\nload_tensors: layer 22 assigned to device MTL0, is_swa = 0\nload_tensors: layer 23 assigned to device MTL0, is_swa = 0\nload_tensors: layer 24 assigned to device MTL0, is_swa = 0\nload_tensors: layer 25 assigned to device MTL0, is_swa = 0\nload_tensors: layer 26 assigned to device MTL0, is_swa = 0\nload_tensors: layer 27 assigned to device MTL0, is_swa = 0\nload_tensors: layer 28 assigned to device MTL0, is_swa = 0\nload_tensors: layer 29 assigned to device MTL0, is_swa = 0\nload_tensors: layer 30 assigned to device MTL0, is_swa = 0\nload_tensors: layer 31 assigned to device MTL0, is_swa = 0\nload_tensors: layer 32 assigned to device MTL0, is_swa = 0\nload_tensors: layer 33 assigned to device MTL0, is_swa = 0\nload_tensors: layer 34 assigned to device MTL0, is_swa = 0\nload_tensors: layer 35 assigned to device MTL0, is_swa = 0\nload_tensors: layer 36 assigned to device MTL0, is_swa = 0\nload_tensors: layer 37 assigned to device MTL0, is_swa = 0\nload_tensors: layer 38 assigned to device MTL0, is_swa = 0\nload_tensors: layer 39 assigned to device MTL0, is_swa = 0\nload_tensors: layer 40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 120.59 MiB\nload_tensors: MTL0_Mapped model buffer size = 4031.55 MiB\nllama_kv_cache: layer 5: dev = MTL0\nllama_kv_cache: layer 15: dev = MTL0\nllama_kv_cache: layer 25: dev = MTL0\nllama_kv_cache: layer 35: dev = MTL0\nllama_kv_cache: MTL0 KV buffer size = 64.00 MiB\nllama_memory_recurrent, layer 0: dev = MTL0\nllama_memory_recurrent, layer 1: dev = MTL0\nllama_memory_recurrent, layer 2: dev = MTL0\nllama_memory_recurrent, layer 3: dev = MTL0\nllama_memory_recurrent, layer 4: dev = MTL0\nllama_memory_recurrent, layer 6: dev = MTL0\nllama_memory_recurrent, layer 7: dev = MTL0\nllama_memory_recurrent, layer 8: dev = MTL0\nllama_memory_recurrent, layer 9: dev = MTL0\nllama_memory_recurrent, layer 10: dev = MTL0\nllama_memory_recurrent, layer 11: dev = MTL0\nllama_memory_recurrent, layer 12: dev = MTL0\nllama_memory_recurrent, layer 13: dev = MTL0\nllama_memory_recurrent, layer 14: dev = MTL0\nllama_memory_recurrent, layer 16: dev = MTL0\nllama_memory_recurrent, layer 17: dev = MTL0\nllama_memory_recurrent, layer 18: dev = MTL0\nllama_memory_recurrent, layer 19: dev = MTL0\nllama_memory_recurrent, layer 20: dev = MTL0\nllama_memory_recurrent, layer 21: dev = MTL0\nllama_memory_recurrent, layer 22: dev = MTL0\nllama_memory_recurrent, layer 23: dev = MTL0\nllama_memory_recurrent, layer 24: dev = MTL0\nllama_memory_recurrent, layer 26: dev = MTL0\nllama_memory_recurrent, layer 27: dev = MTL0\nllama_memory_recurrent, layer 28: dev = MTL0\nllama_memory_recurrent, layer 29: dev = MTL0\nllama_memory_recurrent, layer 30: dev = MTL0\nllama_memory_recurrent, layer 31: dev = MTL0\nllama_memory_recurrent, layer 32: dev = MTL0\nllama_memory_recurrent, layer 33: dev = MTL0\nllama_memory_recurrent, layer 34: dev = MTL0\nllama_memory_recurrent, layer 36: dev = MTL0\nllama_memory_recurrent, layer 37: dev = MTL0\nllama_memory_recurrent, layer 38: dev = MTL0\nllama_memory_recurrent, layer 39: dev = MTL0\nllama_memory_recurrent: MTL0 RS buffer size = 55.37 MiB\nsched_reserve: MTL0 compute buffer size = 123.63 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 41,
"totalLayers": 41,
"inferenceMs": 168.81433300295612,
"durationMs": 184.7548750010901,
"tokensPerSecond": 41.46567341457566,
"inputTokens": 72,
"outputTokens": 7,
"outputLimitReached": false,
"rssBytes": 4534534144,
"cpuThreads": 4,
"totalMs": 1421.0969590058085,
"grade": "fast",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "q4",
"device": "mps",
"status": "pass",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": true,
"expected": "Unknown"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
"device": "mps",
"precision": "q4",
"provider": "llama.cpp Metal",
"loadMs": 1236.3420840047183,
"acceleration": "partial",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer 0 assigned to device MTL0, is_swa = 0\nload_tensors: layer 1 assigned to device MTL0, is_swa = 0\nload_tensors: layer 2 assigned to device MTL0, is_swa = 0\nload_tensors: layer 3 assigned to device MTL0, is_swa = 0\nload_tensors: layer 4 assigned to device MTL0, is_swa = 0\nload_tensors: layer 5 assigned to device MTL0, is_swa = 0\nload_tensors: layer 6 assigned to device MTL0, is_swa = 0\nload_tensors: layer 7 assigned to device MTL0, is_swa = 0\nload_tensors: layer 8 assigned to device MTL0, is_swa = 0\nload_tensors: layer 9 assigned to device MTL0, is_swa = 0\nload_tensors: layer 10 assigned to device MTL0, is_swa = 0\nload_tensors: layer 11 assigned to device MTL0, is_swa = 0\nload_tensors: layer 12 assigned to device MTL0, is_swa = 0\nload_tensors: layer 13 assigned to device MTL0, is_swa = 0\nload_tensors: layer 14 assigned to device MTL0, is_swa = 0\nload_tensors: layer 15 assigned to device MTL0, is_swa = 0\nload_tensors: layer 16 assigned to device MTL0, is_swa = 0\nload_tensors: layer 17 assigned to device MTL0, is_swa = 0\nload_tensors: layer 18 assigned to device MTL0, is_swa = 0\nload_tensors: layer 19 assigned to device MTL0, is_swa = 0\nload_tensors: layer 20 assigned to device MTL0, is_swa = 0\nload_tensors: layer 21 assigned to device MTL0, is_swa = 0\nload_tensors: layer 22 assigned to device MTL0, is_swa = 0\nload_tensors: layer 23 assigned to device MTL0, is_swa = 0\nload_tensors: layer 24 assigned to device MTL0, is_swa = 0\nload_tensors: layer 25 assigned to device MTL0, is_swa = 0\nload_tensors: layer 26 assigned to device MTL0, is_swa = 0\nload_tensors: layer 27 assigned to device MTL0, is_swa = 0\nload_tensors: layer 28 assigned to device MTL0, is_swa = 0\nload_tensors: layer 29 assigned to device MTL0, is_swa = 0\nload_tensors: layer 30 assigned to device MTL0, is_swa = 0\nload_tensors: layer 31 assigned to device MTL0, is_swa = 0\nload_tensors: layer 32 assigned to device MTL0, is_swa = 0\nload_tensors: layer 33 assigned to device MTL0, is_swa = 0\nload_tensors: layer 34 assigned to device MTL0, is_swa = 0\nload_tensors: layer 35 assigned to device MTL0, is_swa = 0\nload_tensors: layer 36 assigned to device MTL0, is_swa = 0\nload_tensors: layer 37 assigned to device MTL0, is_swa = 0\nload_tensors: layer 38 assigned to device MTL0, is_swa = 0\nload_tensors: layer 39 assigned to device MTL0, is_swa = 0\nload_tensors: layer 40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 120.59 MiB\nload_tensors: MTL0_Mapped model buffer size = 4031.55 MiB\nllama_kv_cache: layer 5: dev = MTL0\nllama_kv_cache: layer 15: dev = MTL0\nllama_kv_cache: layer 25: dev = MTL0\nllama_kv_cache: layer 35: dev = MTL0\nllama_kv_cache: MTL0 KV buffer size = 64.00 MiB\nllama_memory_recurrent, layer 0: dev = MTL0\nllama_memory_recurrent, layer 1: dev = MTL0\nllama_memory_recurrent, layer 2: dev = MTL0\nllama_memory_recurrent, layer 3: dev = MTL0\nllama_memory_recurrent, layer 4: dev = MTL0\nllama_memory_recurrent, layer 6: dev = MTL0\nllama_memory_recurrent, layer 7: dev = MTL0\nllama_memory_recurrent, layer 8: dev = MTL0\nllama_memory_recurrent, layer 9: dev = MTL0\nllama_memory_recurrent, layer 10: dev = MTL0\nllama_memory_recurrent, layer 11: dev = MTL0\nllama_memory_recurrent, layer 12: dev = MTL0\nllama_memory_recurrent, layer 13: dev = MTL0\nllama_memory_recurrent, layer 14: dev = MTL0\nllama_memory_recurrent, layer 16: dev = MTL0\nllama_memory_recurrent, layer 17: dev = MTL0\nllama_memory_recurrent, layer 18: dev = MTL0\nllama_memory_recurrent, layer 19: dev = MTL0\nllama_memory_recurrent, layer 20: dev = MTL0\nllama_memory_recurrent, layer 21: dev = MTL0\nllama_memory_recurrent, layer 22: dev = MTL0\nllama_memory_recurrent, layer 23: dev = MTL0\nllama_memory_recurrent, layer 24: dev = MTL0\nllama_memory_recurrent, layer 26: dev = MTL0\nllama_memory_recurrent, layer 27: dev = MTL0\nllama_memory_recurrent, layer 28: dev = MTL0\nllama_memory_recurrent, layer 29: dev = MTL0\nllama_memory_recurrent, layer 30: dev = MTL0\nllama_memory_recurrent, layer 31: dev = MTL0\nllama_memory_recurrent, layer 32: dev = MTL0\nllama_memory_recurrent, layer 33: dev = MTL0\nllama_memory_recurrent, layer 34: dev = MTL0\nllama_memory_recurrent, layer 36: dev = MTL0\nllama_memory_recurrent, layer 37: dev = MTL0\nllama_memory_recurrent, layer 38: dev = MTL0\nllama_memory_recurrent, layer 39: dev = MTL0\nllama_memory_recurrent: MTL0 RS buffer size = 55.37 MiB\nsched_reserve: MTL0 compute buffer size = 123.63 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 41,
"totalLayers": 41,
"inferenceMs": 107.77562500152271,
"durationMs": 118.25495799712371,
"tokensPerSecond": 18.55707169382449,
"inputTokens": 74,
"outputTokens": 2,
"outputLimitReached": false,
"rssBytes": 4535271424,
"cpuThreads": 4,
"totalMs": 1354.597042001842,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "q4",
"device": "cpu",
"status": "fail",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"plainParity": true,
"passed": false,
"expected": "Paris",
"fixturePassed": true
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
"device": "cpu",
"precision": "q4",
"provider": "llama.cpp CPU",
"loadMs": 5334.060290995694,
"acceleration": "cpu",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 4029.02 MiB\nload_tensors: CPU_REPACK model buffer size = 4013.35 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 0,
"totalLayers": 41,
"inferenceMs": 451.78691600449383,
"durationMs": 460.1550000006682,
"tokensPerSecond": 4.426865695198899,
"inputTokens": 63,
"outputTokens": 2,
"outputLimitReached": false,
"rssBytes": 8748482560,
"cpuThreads": 4,
"totalMs": 5794.2152909963625,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
},
"failure": "This fixture passed but the backend did not pass the complete repeated profile."
},
{
"precision": "q4",
"device": "cpu",
"status": "fail",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": false,
"expected": "56",
"fixturePassed": true
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
"device": "cpu",
"precision": "q4",
"provider": "llama.cpp CPU",
"loadMs": 5334.060290995694,
"acceleration": "cpu",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 4029.02 MiB\nload_tensors: CPU_REPACK model buffer size = 4013.35 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 0,
"totalLayers": 41,
"inferenceMs": 428.8514580039191,
"durationMs": 437.12595800025156,
"tokensPerSecond": 4.6636194483492295,
"inputTokens": 63,
"outputTokens": 2,
"outputLimitReached": false,
"rssBytes": 8750235648,
"cpuThreads": 4,
"totalMs": 5771.186248995946,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
},
"failure": "This fixture passed but the backend did not pass the complete repeated profile."
},
{
"precision": "q4",
"device": "cpu",
"status": "fail",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": false,
"expected": "chat",
"fixturePassed": true
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
"device": "cpu",
"precision": "q4",
"provider": "llama.cpp CPU",
"loadMs": 5334.060290995694,
"acceleration": "cpu",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 4029.02 MiB\nload_tensors: CPU_REPACK model buffer size = 4013.35 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 0,
"totalLayers": 41,
"inferenceMs": 418.81191699940246,
"durationMs": 427.0358750000014,
"tokensPerSecond": 4.775413303253387,
"inputTokens": 68,
"outputTokens": 2,
"outputLimitReached": false,
"rssBytes": 8752988160,
"cpuThreads": 4,
"totalMs": 5761.096165995696,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
},
"failure": "This fixture passed but the backend did not pass the complete repeated profile."
},
{
"precision": "q4",
"device": "cpu",
"status": "fail",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": false,
"expected": "hola",
"fixturePassed": true
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
"device": "cpu",
"precision": "q4",
"provider": "llama.cpp CPU",
"loadMs": 5334.060290995694,
"acceleration": "cpu",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 4029.02 MiB\nload_tensors: CPU_REPACK model buffer size = 4013.35 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 0,
"totalLayers": 41,
"inferenceMs": 408.98524999647634,
"durationMs": 417.00212500290945,
"tokensPerSecond": 4.890151906498416,
"inputTokens": 69,
"outputTokens": 2,
"outputLimitReached": false,
"rssBytes": 8753217536,
"cpuThreads": 4,
"totalMs": 5751.062415998604,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
},
"failure": "This fixture passed but the backend did not pass the complete repeated profile."
},
{
"precision": "q4",
"device": "cpu",
"status": "fail",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": false,
"expected": "2026-11-04",
"fixturePassed": true
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
"device": "cpu",
"precision": "q4",
"provider": "llama.cpp CPU",
"loadMs": 5334.060290995694,
"acceleration": "cpu",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 4029.02 MiB\nload_tensors: CPU_REPACK model buffer size = 4013.35 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 0,
"totalLayers": 41,
"inferenceMs": 551.460708004015,
"durationMs": 559.5853339982568,
"tokensPerSecond": 12.69356075310634,
"inputTokens": 72,
"outputTokens": 7,
"outputLimitReached": false,
"rssBytes": 8754282496,
"cpuThreads": 4,
"totalMs": 5893.645624993951,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
},
"failure": "This fixture passed but the backend did not pass the complete repeated profile."
},
{
"precision": "q4",
"device": "cpu",
"status": "fail",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": false,
"expected": "Unknown",
"fixturePassed": true
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
"device": "cpu",
"precision": "q4",
"provider": "llama.cpp CPU",
"loadMs": 5334.060290995694,
"acceleration": "cpu",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 4029.02 MiB\nload_tensors: CPU_REPACK model buffer size = 4013.35 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 0,
"totalLayers": 41,
"inferenceMs": 530.0160419938038,
"durationMs": 538.2825420019799,
"tokensPerSecond": 3.7734706905784203,
"inputTokens": 74,
"outputTokens": 2,
"outputLimitReached": false,
"rssBytes": 8754462720,
"cpuThreads": 4,
"totalMs": 5872.342832997674,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
},
"failure": "This fixture passed but the backend did not pass the complete repeated profile."
},
{
"precision": "q4",
"device": "cpu",
"status": "fail",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": false,
"expected": "Paris"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
"device": "cpu",
"precision": "q4",
"provider": "llama.cpp CPU",
"loadMs": 4297.2847080018255,
"acceleration": "cpu",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 4029.02 MiB\nload_tensors: CPU_REPACK model buffer size = 4013.35 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 0,
"totalLayers": 41,
"inferenceMs": 1443.9783339985297,
"durationMs": 1453.518833994167,
"tokensPerSecond": 1.3850623329380485,
"inputTokens": 63,
"outputTokens": 2,
"outputLimitReached": false,
"rssBytes": 7349977088,
"cpuThreads": 4,
"totalMs": 5750.803541995992,
"grade": "slow",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "q4",
"device": "cpu",
"status": "fail",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": false,
"expected": "56"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
"device": "cpu",
"precision": "q4",
"provider": "llama.cpp CPU",
"loadMs": 4297.2847080018255,
"acceleration": "cpu",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 4029.02 MiB\nload_tensors: CPU_REPACK model buffer size = 4013.35 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 0,
"totalLayers": 41,
"inferenceMs": 1736.786374996882,
"durationMs": 1747.559250004997,
"tokensPerSecond": 1.151552101509082,
"inputTokens": 63,
"outputTokens": 2,
"outputLimitReached": false,
"rssBytes": 6324764672,
"cpuThreads": 4,
"totalMs": 6044.8439580068225,
"grade": "slow",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "q4",
"device": "cpu",
"status": "fail",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": false,
"expected": "chat"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
"device": "cpu",
"precision": "q4",
"provider": "llama.cpp CPU",
"loadMs": 4297.2847080018255,
"acceleration": "cpu",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 4029.02 MiB\nload_tensors: CPU_REPACK model buffer size = 4013.35 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 0,
"totalLayers": 41,
"inferenceMs": 1108.1044170059613,
"durationMs": 1117.1746249965508,
"tokensPerSecond": 1.8048840608395846,
"inputTokens": 68,
"outputTokens": 2,
"outputLimitReached": false,
"rssBytes": 7378649088,
"cpuThreads": 4,
"totalMs": 5414.459332998376,
"grade": "slow",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "q4",
"device": "cpu",
"status": "fail",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": false,
"expected": "hola",
"fixturePassed": true
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
"device": "cpu",
"precision": "q4",
"provider": "llama.cpp CPU",
"loadMs": 4297.2847080018255,
"acceleration": "cpu",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 4029.02 MiB\nload_tensors: CPU_REPACK model buffer size = 4013.35 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 0,
"totalLayers": 41,
"inferenceMs": 914.9198749946663,
"durationMs": 923.8642920026905,
"tokensPerSecond": 2.1859837726354554,
"inputTokens": 69,
"outputTokens": 2,
"outputLimitReached": false,
"rssBytes": 7150141440,
"cpuThreads": 4,
"totalMs": 5221.149000004516,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
},
"failure": "This fixture passed but the backend did not pass the complete repeated profile."
},
{
"precision": "q4",
"device": "cpu",
"status": "fail",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": false,
"expected": "2026-11-04",
"fixturePassed": true
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
"device": "cpu",
"precision": "q4",
"provider": "llama.cpp CPU",
"loadMs": 4297.2847080018255,
"acceleration": "cpu",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 4029.02 MiB\nload_tensors: CPU_REPACK model buffer size = 4013.35 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 0,
"totalLayers": 41,
"inferenceMs": 1209.3559999993886,
"durationMs": 1220.7180000041262,
"tokensPerSecond": 5.78820463122814,
"inputTokens": 72,
"outputTokens": 7,
"outputLimitReached": false,
"rssBytes": 7941373952,
"cpuThreads": 4,
"totalMs": 5518.002708005952,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
},
"failure": "This fixture passed but the backend did not pass the complete repeated profile."
},
{
"precision": "q4",
"device": "cpu",
"status": "fail",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": false,
"expected": "Unknown",
"fixturePassed": true
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
"device": "cpu",
"precision": "q4",
"provider": "llama.cpp CPU",
"loadMs": 4297.2847080018255,
"acceleration": "cpu",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 4029.02 MiB\nload_tensors: CPU_REPACK model buffer size = 4013.35 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 0,
"totalLayers": 41,
"inferenceMs": 658.3934589943965,
"durationMs": 672.8037499997299,
"tokensPerSecond": 3.0376972502957713,
"inputTokens": 74,
"outputTokens": 2,
"outputLimitReached": false,
"rssBytes": 8082325504,
"cpuThreads": 4,
"totalMs": 4970.088458001555,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
},
"failure": "This fixture passed but the backend did not pass the complete repeated profile."
},
{
"precision": "q4",
"device": "cpu",
"status": "fail",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"plainParity": true,
"passed": false,
"expected": "Paris"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
"device": "cpu",
"precision": "q4",
"provider": "llama.cpp CPU",
"loadMs": 4240.411375001713,
"acceleration": "cpu",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 4029.02 MiB\nload_tensors: CPU_REPACK model buffer size = 4013.35 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 0,
"totalLayers": 41,
"inferenceMs": 1374.3917920000968,
"durationMs": 1382.5569580003503,
"tokensPerSecond": 1.455189132852344,
"inputTokens": 63,
"outputTokens": 2,
"outputLimitReached": false,
"rssBytes": 8748171264,
"cpuThreads": 1,
"totalMs": 5622.968333002063,
"grade": "slow",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "q4",
"device": "cpu",
"status": "fail",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": false,
"expected": "56"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
"device": "cpu",
"precision": "q4",
"provider": "llama.cpp CPU",
"loadMs": 4240.411375001713,
"acceleration": "cpu",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 4029.02 MiB\nload_tensors: CPU_REPACK model buffer size = 4013.35 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 0,
"totalLayers": 41,
"inferenceMs": 1357.6007499941625,
"durationMs": 1365.4761249999865,
"tokensPerSecond": 1.4731871649368193,
"inputTokens": 63,
"outputTokens": 2,
"outputLimitReached": false,
"rssBytes": 8749498368,
"cpuThreads": 1,
"totalMs": 5605.8875000017,
"grade": "slow",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "q4",
"device": "cpu",
"status": "fail",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": false,
"expected": "chat"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
"device": "cpu",
"precision": "q4",
"provider": "llama.cpp CPU",
"loadMs": 4240.411375001713,
"acceleration": "cpu",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 4029.02 MiB\nload_tensors: CPU_REPACK model buffer size = 4013.35 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 0,
"totalLayers": 41,
"inferenceMs": 1434.74079099542,
"durationMs": 1443.079624994425,
"tokensPerSecond": 1.3939800224209171,
"inputTokens": 68,
"outputTokens": 2,
"outputLimitReached": false,
"rssBytes": 8752103424,
"cpuThreads": 1,
"totalMs": 5683.490999996138,
"grade": "slow",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "q4",
"device": "cpu",
"status": "fail",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": false,
"expected": "hola"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
"device": "cpu",
"precision": "q4",
"provider": "llama.cpp CPU",
"loadMs": 4240.411375001713,
"acceleration": "cpu",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 4029.02 MiB\nload_tensors: CPU_REPACK model buffer size = 4013.35 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 0,
"totalLayers": 41,
"inferenceMs": 1456.053082998551,
"durationMs": 1464.2745000019204,
"tokensPerSecond": 1.3735762956397588,
"inputTokens": 69,
"outputTokens": 2,
"outputLimitReached": false,
"rssBytes": 8752758784,
"cpuThreads": 1,
"totalMs": 5704.685875003634,
"grade": "slow",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
},
{
"precision": "q4",
"device": "cpu",
"status": "fail",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": false,
"expected": "2026-11-04",
"fixturePassed": true
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
"device": "cpu",
"precision": "q4",
"provider": "llama.cpp CPU",
"loadMs": 4240.411375001713,
"acceleration": "cpu",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 4029.02 MiB\nload_tensors: CPU_REPACK model buffer size = 4013.35 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 0,
"totalLayers": 41,
"inferenceMs": 1682.3193750024075,
"durationMs": 1690.0682080013212,
"tokensPerSecond": 4.160922179232456,
"inputTokens": 72,
"outputTokens": 7,
"outputLimitReached": false,
"rssBytes": 8754397184,
"cpuThreads": 1,
"totalMs": 5930.479583003034,
"grade": "acceptable",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
},
"failure": "This fixture passed but the backend did not pass the complete repeated profile."
},
{
"precision": "q4",
"device": "cpu",
"status": "fail",
"quality": {
"expectedText": true,
"streamParity": true,
"withinTokenBudget": true,
"passed": false,
"expected": "Unknown"
},
"runtime": {
"engine": "llama.cpp",
"version": "0.3.36",
"artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
"device": "cpu",
"precision": "q4",
"provider": "llama.cpp CPU",
"loadMs": 4240.411375001713,
"acceleration": "cpu",
"placement": "ggml_metal_device_init: GPU name: MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors: CPU_Mapped model buffer size = 4029.02 MiB\nload_tensors: CPU_REPACK model buffer size = 4013.35 MiB",
"parameterDtype": "Q4_K_M mixed tensor types",
"offloadedLayers": 0,
"totalLayers": 41,
"inferenceMs": 1633.4471669979393,
"durationMs": 1641.847124999913,
"tokensPerSecond": 1.2244044621753738,
"inputTokens": 74,
"outputTokens": 2,
"outputLimitReached": false,
"rssBytes": 8756117504,
"cpuThreads": 1,
"totalMs": 5882.258500001626,
"grade": "slow",
"loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
}
}
]