SmolLM3 Desktop
General chat, drafting and short multilingual replies locally. Strict-format output is not guaranteed.
Desktop app
Run SmolLM3 Desktop on your computer
Download the desktop app to load this model, provide input and view results locally.
Public desktop downloads are coming soon.Device requirements
If your device falls short
MemoryAdvisory only: close other apps, reduce image/tile/token/batch settings or choose a smaller verified model/precision. CPU can need more RAM than GPU.
GPUIf the selected GPU runtime fails, existing CPU fallback may recover when enabled, but CPU has its own memory/runtime needs.
PlatformUse a compatible OS/architecture/runtime or another model. These estimates do not add hardware detection or block runs.
Benchmark evidence
Saved desktop measurements apply to the recorded hardware and runtime. Fallback cases report the device that actually ran; they do not prove GPU support on the requested device.
SmolLM3 Desktop
42 configurations testedQuality criteria
No explicit quality gate recorded. Review per-case evidence; full certification is pending.
| Precision | Device | Result | Platform | Inference / load | |
|---|---|---|---|---|---|
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 151 ms / 1724 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 152 ms / 1583 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 153 ms / 1724 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 153 ms / 1664 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 155 ms / 1583 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 155 ms / 1664 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 178 ms / 1664 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 181 ms / 1664 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 186 ms / 1724 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 187 ms / 1583 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 187 ms / 1724 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 192 ms / 1583 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 195 ms / 1583 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 196 ms / 1724 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 196 ms / 1664 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 511 ms / 1284 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 515 ms / 1284 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 521 ms / 1284 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 522 ms / 1364 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 522 ms / 1449 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 531 ms / 1364 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 531 ms / 1432 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 534 ms / 1449 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 536 ms / 1364 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 537 ms / 1432 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 537 ms / 1364 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 542 ms / 1449 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 542 ms / 1284 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 544 ms / 1432 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 546 ms / 1449 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 551 ms / 1449 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 571 ms / 1432 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 582 ms / 1432 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 587 ms / 1364 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 611 ms / 1284 ms |
FP32 | cpu | Failed | — | macos · aarch64 | 2288 ms / 1449 ms |
FP32 | cpu | Failed | — | macos · aarch64 | 2748 ms / 1284 ms |
FP32 | cpu | Failed | — | macos · aarch64 | 2344 ms / 1432 ms |
FP32 | cpu | Failed | — | macos · aarch64 | 2461 ms / 1364 ms |
FP16 | mps | Failed | — | macos · aarch64 | 549 ms / 1724 ms |
FP16 | mps | Failed | — | macos · aarch64 | 558 ms / 1583 ms |
FP16 | mps | Failed | — | macos · aarch64 | 541 ms / 1664 ms |
Test setup & sample details
- Quality gate
- No explicit quality gate recorded. Review the per-case quality evidence; full certification is pending.
- Tested
- 2026-11-04T09:43:32.286323+00:00
- Environment
- { "os": "macos", "arch": "aarch64", "hardware": "Apple M3 Max", "osVersion": "27.2" }
- Runtime
- transformers 5.22.1
- Artifact revision
- a07cc9a04f16550a088caea529712d1d335b0ac1
- Fixture
{ "file": "model-samples/desktop/smollm3/smollm3-3b-native-capital.json", "size": 244, "sha1": "9013b4f5a71cd1280d78b3dfa751ead1186d071f", "sha256": "8ff34789fa2a0f5bbef2080057f4b36376d3c73ea64baccefb6caec96eafbd68", "url": "https://assets.localai.tools/samples/v1/files/8ff34789fa2a0f5bbef2080057f4b36376d3c73ea64baccefb6caec96eafbd68/smollm3-3b-native-capital.json" }- Rerun when
- Artifact/runtime revisions; Generation/preprocessing changes; Fixture/quality scope; OS/backend behavior
Per-case quality & failure details
[
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"exactNormalized": true,
"expected": "Paris",
"scope": "general-chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1448.5713749891147,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 326
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 553.6669589928351,
"inferenceMs": 546.4499579975381,
"tokensPerSecond": 3.659987471366976,
"inputTokens": 82,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 1,
"peakHelperRssBytes": 12849774592,
"memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
"artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
"totalMs": 2002.2383339819498,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"exactNormalized": true,
"expected": "56",
"scope": "general-chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1448.5713749891147,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 326
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 522.884082980454,
"inferenceMs": 522.4799169809557,
"tokensPerSecond": 3.827898326803821,
"inputTokens": 82,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 1,
"peakHelperRssBytes": 12853297152,
"memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
"artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
"totalMs": 1971.4554579695687,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"exactNormalized": true,
"expected": "chat",
"scope": "general-chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1448.5713749891147,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 326
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 541.8647920014337,
"inferenceMs": 541.5140829863958,
"tokensPerSecond": 3.693348082417729,
"inputTokens": 87,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 1,
"peakHelperRssBytes": 12870057984,
"memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
"artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
"totalMs": 1990.4361669905484,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"exactNormalized": true,
"expected": "hola",
"scope": "general-chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1448.5713749891147,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 326
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 551.3195410021581,
"inferenceMs": 550.8882080030162,
"tokensPerSecond": 3.6305006550240946,
"inputTokens": 88,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 1,
"peakHelperRssBytes": 12883214336,
"memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
"artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
"totalMs": 1999.8909159912728,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "fail",
"quality": {
"passed": false,
"exactNormalized": false,
"expected": "2026-11-04",
"scope": "diagnostic-strict-format"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1448.5713749891147,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 326
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 2288.609541981714,
"inferenceMs": 2288.246625015745,
"tokensPerSecond": 6.992253293453413,
"inputTokens": 91,
"outputTokens": 16,
"maxNewTokens": 32,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 1,
"peakHelperRssBytes": 12901138432,
"memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
"artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
"totalMs": 3737.1809169708285,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"exactNormalized": true,
"expected": "Paris",
"scope": "general-chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1448.5713749891147,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 326
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 534.2767080001067,
"inferenceMs": 533.941583998967,
"tokensPerSecond": 3.7457281094702477,
"inputTokens": 82,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 1,
"peakHelperRssBytes": 12901138432,
"memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
"artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
"totalMs": 1982.8480829892214,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"exactNormalized": true,
"expected": "Paris",
"scope": "general-chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1284.4667080207728,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 326
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 549.6564589848276,
"inferenceMs": 541.5752919798251,
"tokensPerSecond": 3.692930659167709,
"inputTokens": 82,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"peakHelperRssBytes": 12846383104,
"memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
"artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
"totalMs": 1834.1231670056004,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"exactNormalized": true,
"expected": "56",
"scope": "general-chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1284.4667080207728,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 326
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 515.1305420149583,
"inferenceMs": 514.6995830000378,
"tokensPerSecond": 3.8857618425539955,
"inputTokens": 82,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"peakHelperRssBytes": 12847448064,
"memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
"artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
"totalMs": 1799.5972500357311,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"exactNormalized": true,
"expected": "chat",
"scope": "general-chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1284.4667080207728,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 326
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 511.64462498854846,
"inferenceMs": 511.2174169917125,
"tokensPerSecond": 3.9122297745039907,
"inputTokens": 87,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"peakHelperRssBytes": 12876529664,
"memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
"artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
"totalMs": 1796.1113330093212,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"exactNormalized": true,
"expected": "hola",
"scope": "general-chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1284.4667080207728,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 326
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 521.4637909957673,
"inferenceMs": 521.062083018478,
"tokensPerSecond": 3.8383142147172427,
"inputTokens": 88,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"peakHelperRssBytes": 12888981504,
"memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
"artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
"totalMs": 1805.9304990165401,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "fail",
"quality": {
"passed": false,
"exactNormalized": false,
"expected": "2026-11-04",
"scope": "diagnostic-strict-format"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1284.4667080207728,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 326
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 2748.0667080089916,
"inferenceMs": 2747.6530830026604,
"tokensPerSecond": 5.8231514374860796,
"inputTokens": 91,
"outputTokens": 16,
"maxNewTokens": 32,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"peakHelperRssBytes": 12903923712,
"memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
"artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
"totalMs": 4032.5334160297643,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"exactNormalized": true,
"expected": "Paris",
"scope": "general-chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1284.4667080207728,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 326
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 611.310209002113,
"inferenceMs": 610.8436659851577,
"tokensPerSecond": 3.2741601679284598,
"inputTokens": 82,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"peakHelperRssBytes": 12903940096,
"memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
"artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
"totalMs": 1895.7769170228858,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"exactNormalized": true,
"expected": "Paris",
"scope": "general-chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1431.9112499943003,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 326
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 544.5574159966782,
"inferenceMs": 536.6443329839967,
"tokensPerSecond": 3.726863170023715,
"inputTokens": 82,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"peakHelperRssBytes": 12847661056,
"memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
"artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
"totalMs": 1976.4686659909785,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"exactNormalized": true,
"expected": "56",
"scope": "general-chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1431.9112499943003,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 326
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 582.4415419774596,
"inferenceMs": 582.0458329981193,
"tokensPerSecond": 3.436155516650632,
"inputTokens": 82,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"peakHelperRssBytes": 12848119808,
"memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
"artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
"totalMs": 2014.3527919717599,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"exactNormalized": true,
"expected": "chat",
"scope": "general-chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1431.9112499943003,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 326
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 571.7759580002166,
"inferenceMs": 571.3317499903496,
"tokensPerSecond": 3.5005931318078893,
"inputTokens": 87,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"peakHelperRssBytes": 12859703296,
"memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
"artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
"totalMs": 2003.687207994517,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"exactNormalized": true,
"expected": "hola",
"scope": "general-chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1431.9112499943003,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 326
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 544.6160410065204,
"inferenceMs": 544.201166019775,
"tokensPerSecond": 3.675111566973976,
"inputTokens": 88,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"peakHelperRssBytes": 12876873728,
"memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
"artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
"totalMs": 1976.5272910008207,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "fail",
"quality": {
"passed": false,
"exactNormalized": false,
"expected": "2026-11-04",
"scope": "diagnostic-strict-format"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1431.9112499943003,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 326
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 2344.899040996097,
"inferenceMs": 2344.4360840076115,
"tokensPerSecond": 6.824668886962948,
"inputTokens": 91,
"outputTokens": 16,
"maxNewTokens": 32,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"peakHelperRssBytes": 12903743488,
"memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
"artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
"totalMs": 3776.8102909903973,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"exactNormalized": true,
"expected": "Paris",
"scope": "general-chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1431.9112499943003,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 326
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 531.6927919921,
"inferenceMs": 531.2627909879666,
"tokensPerSecond": 3.7646152411327094,
"inputTokens": 82,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"peakHelperRssBytes": 12903759872,
"memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
"artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
"totalMs": 1963.6040419864003,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"exactNormalized": true,
"expected": "Paris",
"scope": "general-chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1364.149791013915,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 326
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 594.6649170073215,
"inferenceMs": 586.9804579997435,
"tokensPerSecond": 3.407268457991618,
"inputTokens": 82,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"peakHelperRssBytes": 12847251456,
"memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
"artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
"totalMs": 1958.8147080212366,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"exactNormalized": true,
"expected": "56",
"scope": "general-chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1364.149791013915,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 326
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 536.728334001964,
"inferenceMs": 536.3179999985732,
"tokensPerSecond": 3.7291308514823682,
"inputTokens": 82,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"peakHelperRssBytes": 12848742400,
"memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
"artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
"totalMs": 1900.878125015879,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"exactNormalized": true,
"expected": "chat",
"scope": "general-chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1364.149791013915,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 326
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 537.1224579866976,
"inferenceMs": 536.7711249855347,
"tokensPerSecond": 3.725982838689204,
"inputTokens": 87,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"peakHelperRssBytes": 12871221248,
"memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
"artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
"totalMs": 1901.2722490006126,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"exactNormalized": true,
"expected": "hola",
"scope": "general-chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1364.149791013915,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 326
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 522.4633339967113,
"inferenceMs": 521.9061250099912,
"tokensPerSecond": 3.8321067796660033,
"inputTokens": 88,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"peakHelperRssBytes": 12888211456,
"memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
"artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
"totalMs": 1886.6131250106264,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "fail",
"quality": {
"passed": false,
"exactNormalized": false,
"expected": "2026-11-04",
"scope": "diagnostic-strict-format"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1364.149791013915,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 326
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 2461.52837498812,
"inferenceMs": 2461.1574160226155,
"tokensPerSecond": 6.501006354098634,
"inputTokens": 91,
"outputTokens": 16,
"maxNewTokens": 32,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"peakHelperRssBytes": 12903841792,
"memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
"artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
"totalMs": 3825.678166002035,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"exactNormalized": true,
"expected": "Paris",
"scope": "general-chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1364.149791013915,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 326
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 531.0165830014739,
"inferenceMs": 530.6597090093419,
"tokensPerSecond": 3.768893635685447,
"inputTokens": 82,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"peakHelperRssBytes": 12903890944,
"memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
"artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
"totalMs": 1895.166374015389,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"exactNormalized": true,
"expected": "Paris",
"scope": "general-chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1723.9172920235433,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 326
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 205.20191700779833,
"inferenceMs": 195.98945797770284,
"tokensPerSecond": 10.20463049715426,
"inputTokens": 82,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 6530220032,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"precision": "fp16",
"cpuThreads": 4,
"peakHelperRssBytes": 2719154176,
"memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
"artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
"totalMs": 1929.1192090313416,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"exactNormalized": true,
"expected": "56",
"scope": "general-chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1723.9172920235433,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 326
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 154.83412498724647,
"inferenceMs": 153.10120797948912,
"tokensPerSecond": 13.06325421199772,
"inputTokens": 82,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 6530203648,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"precision": "fp16",
"cpuThreads": 4,
"peakHelperRssBytes": 2719891456,
"memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
"artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
"totalMs": 1878.7514170107897,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"exactNormalized": true,
"expected": "chat",
"scope": "general-chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1723.9172920235433,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 326
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 188.111374998698,
"inferenceMs": 185.74812498991378,
"tokensPerSecond": 10.767268849193504,
"inputTokens": 87,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 6539624448,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"precision": "fp16",
"cpuThreads": 4,
"peakHelperRssBytes": 2729558016,
"memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
"artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
"totalMs": 1912.0286670222413,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"exactNormalized": true,
"expected": "hola",
"scope": "general-chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1723.9172920235433,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 326
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 189.1775419935584,
"inferenceMs": 187.25770901073702,
"tokensPerSecond": 10.680468166388406,
"inputTokens": 88,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 6540673024,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"precision": "fp16",
"cpuThreads": 4,
"peakHelperRssBytes": 2739109888,
"memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
"artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
"totalMs": 1913.0948340171017,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "fail",
"quality": {
"passed": false,
"exactNormalized": false,
"expected": "2026-11-04",
"scope": "diagnostic-strict-format"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1723.9172920235433,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 326
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 551.2816250266042,
"inferenceMs": 549.4946249818895,
"tokensPerSecond": 29.117664254727398,
"inputTokens": 91,
"outputTokens": 16,
"maxNewTokens": 32,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 6541803520,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"precision": "fp16",
"cpuThreads": 4,
"peakHelperRssBytes": 2749186048,
"memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
"artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
"totalMs": 2275.1989170501474,
"grade": "fast",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"exactNormalized": true,
"expected": "Paris",
"scope": "general-chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1723.9172920235433,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 326
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 153.5433329991065,
"inferenceMs": 151.2529170140624,
"tokensPerSecond": 13.222885478724715,
"inputTokens": 82,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 6541803520,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"precision": "fp16",
"cpuThreads": 4,
"peakHelperRssBytes": 2749628416,
"memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
"artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
"totalMs": 1877.4606250226498,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"exactNormalized": true,
"expected": "Paris",
"scope": "general-chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1583.3670000138227,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 326
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 203.67754099424928,
"inferenceMs": 194.61916599539109,
"tokensPerSecond": 10.276480169724719,
"inputTokens": 82,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 6530220032,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"precision": "fp16",
"cpuThreads": 4,
"peakHelperRssBytes": 2719744000,
"memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
"artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
"totalMs": 1787.044541008072,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"exactNormalized": true,
"expected": "56",
"scope": "general-chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1583.3670000138227,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 326
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 156.57429199200124,
"inferenceMs": 154.56175000872463,
"tokensPerSecond": 12.939812080848625,
"inputTokens": 82,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 6530220032,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"precision": "fp16",
"cpuThreads": 4,
"peakHelperRssBytes": 2720595968,
"memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
"artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
"totalMs": 1739.941292005824,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"exactNormalized": true,
"expected": "chat",
"scope": "general-chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1583.3670000138227,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 326
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 188.49683398730122,
"inferenceMs": 186.79758301004767,
"tokensPerSecond": 10.706776649740815,
"inputTokens": 87,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 6539624448,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"precision": "fp16",
"cpuThreads": 4,
"peakHelperRssBytes": 2729885696,
"memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
"artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
"totalMs": 1771.863834001124,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"exactNormalized": true,
"expected": "hola",
"scope": "general-chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1583.3670000138227,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 326
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 193.88779098517261,
"inferenceMs": 192.10408398066647,
"tokensPerSecond": 10.411022808871058,
"inputTokens": 88,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 6540673024,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"precision": "fp16",
"cpuThreads": 4,
"peakHelperRssBytes": 2739077120,
"memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
"artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
"totalMs": 1777.2547909989953,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "fail",
"quality": {
"passed": false,
"exactNormalized": false,
"expected": "2026-11-04",
"scope": "diagnostic-strict-format"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1583.3670000138227,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 326
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 559.2794999829493,
"inferenceMs": 557.5622920005117,
"tokensPerSecond": 28.69634519686155,
"inputTokens": 91,
"outputTokens": 16,
"maxNewTokens": 32,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 6541803520,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"precision": "fp16",
"cpuThreads": 4,
"peakHelperRssBytes": 2748809216,
"memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
"artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
"totalMs": 2142.646499996772,
"grade": "fast",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"exactNormalized": true,
"expected": "Paris",
"scope": "general-chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1583.3670000138227,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 326
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 154.20708298915997,
"inferenceMs": 152.4789580143988,
"tokensPerSecond": 13.116563924913082,
"inputTokens": 82,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 6541803520,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"precision": "fp16",
"cpuThreads": 4,
"peakHelperRssBytes": 2749251584,
"memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
"artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
"totalMs": 1737.5740830029827,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"exactNormalized": true,
"expected": "Paris",
"scope": "general-chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1663.9372499776073,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 326
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 205.40504198288545,
"inferenceMs": 196.4990830165334,
"tokensPerSecond": 10.178164545590883,
"inputTokens": 82,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 6530220032,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"precision": "fp16",
"cpuThreads": 4,
"peakHelperRssBytes": 2718875648,
"memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
"artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
"totalMs": 1869.3422919604927,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"exactNormalized": true,
"expected": "56",
"scope": "general-chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1663.9372499776073,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 326
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 157.03620901331306,
"inferenceMs": 155.24758299579844,
"tokensPerSecond": 12.882648228115263,
"inputTokens": 82,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 6530220032,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"precision": "fp16",
"cpuThreads": 4,
"peakHelperRssBytes": 2719449088,
"memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
"artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
"totalMs": 1820.9734589909203,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"exactNormalized": true,
"expected": "chat",
"scope": "general-chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1663.9372499776073,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 326
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 182.60575001477264,
"inferenceMs": 181.0010000190232,
"tokensPerSecond": 11.049662707884487,
"inputTokens": 87,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 6539624448,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"precision": "fp16",
"cpuThreads": 4,
"peakHelperRssBytes": 2729099264,
"memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
"artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
"totalMs": 1846.54299999238,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"exactNormalized": true,
"expected": "hola",
"scope": "general-chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1663.9372499776073,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 326
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 180.093915987527,
"inferenceMs": 178.21354197803885,
"tokensPerSecond": 11.222491724262227,
"inputTokens": 88,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 6540673024,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"precision": "fp16",
"cpuThreads": 4,
"peakHelperRssBytes": 2738487296,
"memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
"artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
"totalMs": 1844.0311659651343,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "fail",
"quality": {
"passed": false,
"exactNormalized": false,
"expected": "2026-11-04",
"scope": "diagnostic-strict-format"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1663.9372499776073,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 326
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 542.4867499968968,
"inferenceMs": 540.7002919819206,
"tokensPerSecond": 29.591254595688273,
"inputTokens": 91,
"outputTokens": 16,
"maxNewTokens": 32,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 6541803520,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"precision": "fp16",
"cpuThreads": 4,
"peakHelperRssBytes": 2748497920,
"memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
"artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
"totalMs": 2206.423999974504,
"grade": "fast",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"exactNormalized": true,
"expected": "Paris",
"scope": "general-chat"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1663.9372499776073,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 326
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 155.09383301832713,
"inferenceMs": 153.16183300456032,
"tokensPerSecond": 13.058083471359677,
"inputTokens": 82,
"outputTokens": 2,
"maxNewTokens": 32,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 6541803520,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"precision": "fp16",
"cpuThreads": 4,
"peakHelperRssBytes": 2748825600,
"memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
"artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
"totalMs": 1819.0310829959344,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
}
]