Qwen3.5 Desktop
Run local text questions, still-image questions using Qwen3.5 4B Desktop. Answers can hallucinate or miss visual details.
Desktop app
Run Qwen3.5 Desktop on your computer
Download the desktop app to load this model, provide input and view results locally.
Public desktop downloads are coming soon.Device requirements
If your device falls short
MemoryAdvisory only: close other apps, reduce image/tile/token/batch settings or choose a smaller verified model/precision. CPU can need more RAM than GPU.
GPUIf the selected GPU runtime fails, existing CPU fallback may recover when enabled, but CPU has its own memory/runtime needs.
PlatformUse a compatible OS/architecture/runtime or another model. These estimates do not add hardware detection or block runs.
Benchmark evidence
Saved desktop measurements apply to the recorded hardware and runtime. Fallback cases report the device that actually ran; they do not prove GPU support on the requested device.
Qwen3.5 4B Desktop — Text, Image Desktop
47 configurations testedQuality criteria
No explicit quality gate recorded. Review per-case evidence; full certification is pending.
| Precision | Device | Result | Platform | Inference / load | |
|---|---|---|---|---|---|
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 889 ms / 2945 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 951 ms / 4310 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 966 ms / 3049 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 413 ms / 4310 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 414 ms / 2945 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 431 ms / 3049 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 527 ms / 2945 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 558 ms / 3049 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 490 ms / 2945 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 504 ms / 3327 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 1040 ms / 3327 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 1180 ms / 3327 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 613 ms / 4310 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 529 ms / 3049 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 438 ms / 2945 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 449 ms / 3049 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 555 ms / 4310 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 739 ms / 4310 ms |
FP16 | mpsAccelerated | Passed | Slow | macos · aarch64 | 4487 ms / 3327 ms |
FP16 | mpsAccelerated | Passed | Very slow | macos · aarch64 | 4489 ms / 3327 ms |
FP16 | mpsAccelerated | Passed | Very slow | macos · aarch64 | 5042 ms / 3327 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 7214 ms / 7221 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 2529 ms / 7221 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 2051 ms / 7221 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 2568 ms / 7221 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 3179 ms / 7221 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 11867 ms / 3092 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 40629 ms / 2693 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 11094 ms / 3092 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 17068 ms / 5198 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 17375 ms / 2693 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 14845 ms / 5198 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 12034 ms / 2693 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 12366 ms / 5198 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 16533 ms / 2693 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 17779 ms / 2693 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 54743 ms / 5198 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 22952 ms / 3092 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 20502 ms / 5198 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 63265 ms / 3092 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 22823 ms / 3092 ms |
FP32 | cpu | Passed | Very slow | macos · aarch64 | 51851 ms / 6714 ms |
FP32 | cpu | Passed | Very slow | macos · aarch64 | 33455 ms / 6714 ms |
FP32 | cpu | Passed | Very slow | macos · aarch64 | 23105 ms / 6714 ms |
FP32 | cpu | Passed | Very slow | macos · aarch64 | 34926 ms / 6714 ms |
FP32 | cpu | Passed | Very slow | macos · aarch64 | 23994 ms / 6714 ms |
FP32 | cpu | Passed | Very slow | macos · aarch64 | 24220 ms / 6714 ms |
Test setup & sample details
- Quality gate
- No explicit quality gate recorded. Review the per-case quality evidence; full certification is pending.
- Tested
- 2026-11-03T14:46:28.967787+00:00
- Environment
- { "platform": "macOS-27.2-arm64-arm-64bit-Mach-O", "hardware": "Apple M3 Max" }
- Runtime
- transformers 5.22.1
- Artifact revision
- 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
- Fixture
{ "file": "localai-desktop/crates/desktopai/src/models/qwen35/fixtures.json", "size": 944, "sha1": "ef3eb28d89d7289b84660b6e2db5d6768bcf6d22", "sha256": "2e2f8f9dfe0f61f0e7715a7bd2ba15f7e5c7f5ab6cb7cdb3e594b744a0d96f0b", "url": "https://assets.localai.tools/samples/v1/files/2e2f8f9dfe0f61f0e7715a7bd2ba15f7e5c7f5ab6cb7cdb3e594b744a0d96f0b/fixtures.json" }- Rerun when
- Artifact revision; Transformers/Torch/runtime lock; Adapter inference or input/output semantics; Fixture checksum or quality gate; Device/backend/OS behavior
Per-case quality & failure details
[
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "56",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4310.408624995034,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 723
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 414.86554199946113,
"inferenceMs": 413.3240829978604,
"tokensPerSecond": 12.09704492352526,
"inputTokens": 26,
"outputTokens": 5,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 9522413568,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 4726.7882080050185,
"requestMs": 416.37958300998434,
"grade": "fast",
"artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "READY",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4310.408624995034,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 723
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 557.4632080097217,
"inferenceMs": 555.3242920141201,
"tokensPerSecond": 7.202998423663939,
"inputTokens": 29,
"outputTokens": 4,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 9522757632,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 4869.358084019041,
"requestMs": 558.9494590240065,
"grade": "moderate",
"artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "Oslo",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4310.408624995034,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 723
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 740.9979999938514,
"inferenceMs": 738.9543329773005,
"tokensPerSecond": 6.766317994042525,
"inputTokens": 41,
"outputTokens": 5,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 9524363264,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 5052.681416011183,
"requestMs": 742.2727910161484,
"grade": "moderate",
"artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "ORCHID",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4310.408624995034,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 723
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 614.4294170080684,
"inferenceMs": 612.762709002709,
"tokensPerSecond": 9.791718575311466,
"inputTokens": 51,
"outputTokens": 6,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 9525067776,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 4926.251208002213,
"requestMs": 615.8425830071792,
"grade": "moderate",
"artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": {
"name": "Ada",
"count": 3
},
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4310.408624995034,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 723
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 952.905499987537,
"inferenceMs": 951.1543750122655,
"tokensPerSecond": 15.770310681488027,
"inputTokens": 40,
"outputTokens": 15,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 9525641216,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 5264.713249984197,
"requestMs": 954.3046249891631,
"grade": "fast",
"artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "56",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2944.73950000247,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 723
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 415.4665830137674,
"inferenceMs": 413.8022919942159,
"tokensPerSecond": 12.083065020988066,
"inputTokens": 26,
"outputTokens": 5,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 9522413568,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 3361.750874988502,
"requestMs": 417.011374986032,
"grade": "fast",
"artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "READY",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2944.73950000247,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 723
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 440.1672499952838,
"inferenceMs": 438.3782090153545,
"tokensPerSecond": 9.124541133065073,
"inputTokens": 29,
"outputTokens": 4,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 9522757632,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 3386.3159170141444,
"requestMs": 441.5764170116745,
"grade": "moderate",
"artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "Oslo",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2944.73950000247,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 723
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 492.0912079978734,
"inferenceMs": 490.3657920076512,
"tokensPerSecond": 10.196469822107789,
"inputTokens": 41,
"outputTokens": 5,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 9524363264,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 3438.2337080023717,
"requestMs": 493.49420799990185,
"grade": "fast",
"artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "ORCHID",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2944.73950000247,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 723
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 528.3934580220375,
"inferenceMs": 526.6213750001043,
"tokensPerSecond": 11.393384858331684,
"inputTokens": 51,
"outputTokens": 6,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 9525067776,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 3474.47524999734,
"requestMs": 529.7357499948703,
"grade": "fast",
"artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": {
"name": "Ada",
"count": 3
},
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2944.73950000247,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 723
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 890.5567080073524,
"inferenceMs": 888.9555830101017,
"tokensPerSecond": 16.873733948785546,
"inputTokens": 40,
"outputTokens": 15,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 9525641216,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 3836.76833298523,
"requestMs": 892.0288329827599,
"grade": "fast",
"artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "56",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3049.4302499864716,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 723
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 432.6420839934144,
"inferenceMs": 431.46120800520293,
"tokensPerSecond": 11.58852732813863,
"inputTokens": 26,
"outputTokens": 5,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 9522413568,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 3483.514124964131,
"requestMs": 434.08387497765943,
"grade": "fast",
"artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "READY",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3049.4302499864716,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 723
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 451.076916011516,
"inferenceMs": 449.2849580128677,
"tokensPerSecond": 8.903035654012344,
"inputTokens": 29,
"outputTokens": 4,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 9522757632,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 3501.7676669813227,
"requestMs": 452.3374169948511,
"grade": "moderate",
"artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "Oslo",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3049.4302499864716,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 723
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 531.1215000110678,
"inferenceMs": 529.4120839971583,
"tokensPerSecond": 9.44443874844919,
"inputTokens": 41,
"outputTokens": 5,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 9524363264,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 3581.7654589773156,
"requestMs": 532.335208990844,
"grade": "moderate",
"artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "ORCHID",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3049.4302499864716,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 723
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 559.5353340031579,
"inferenceMs": 557.9937919974327,
"tokensPerSecond": 10.752807801896845,
"inputTokens": 51,
"outputTokens": 6,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 9525067776,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 3610.198958980618,
"requestMs": 560.7687089941464,
"grade": "fast",
"artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": {
"name": "Ada",
"count": 3
},
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3049.4302499864716,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 723
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 967.7480840182398,
"inferenceMs": 965.7638749922626,
"tokensPerSecond": 15.531746825920752,
"inputTokens": 40,
"outputTokens": 15,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 9525641216,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 4018.417374987621,
"requestMs": 968.9871250011493,
"grade": "fast",
"artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "56",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2692.5151250034105,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 723
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 16533.240624994505,
"inferenceMs": 16532.844458008185,
"tokensPerSecond": 0.3024282973628351,
"inputTokens": 26,
"outputTokens": 5,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 19226.991541014286,
"requestMs": 16534.476416010875,
"grade": "slow",
"artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "READY",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2692.5151250034105,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 723
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 12034.509832999902,
"inferenceMs": 12034.152250009356,
"tokensPerSecond": 0.3323873520045328,
"inputTokens": 29,
"outputTokens": 4,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 14728.24170900276,
"requestMs": 12035.726583999349,
"grade": "slow",
"artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "Oslo",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2692.5151250034105,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 723
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 17779.490042012185,
"inferenceMs": 17778.98712499882,
"tokensPerSecond": 0.28123086905043987,
"inputTokens": 41,
"outputTokens": 5,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 20473.26645901194,
"requestMs": 17780.751334008528,
"grade": "slow",
"artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "ORCHID",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2692.5151250034105,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 723
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 17375.299708975945,
"inferenceMs": 17374.757292011054,
"tokensPerSecond": 0.3453285648346185,
"inputTokens": 51,
"outputTokens": 6,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 20069.065457995748,
"requestMs": 17376.550332992338,
"grade": "slow",
"artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": {
"name": "Ada",
"count": 3
},
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2692.5151250034105,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 723
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 40629.073374992004,
"inferenceMs": 40628.72970898752,
"tokensPerSecond": 0.36919687392249023,
"inputTokens": 40,
"outputTokens": 15,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 43322.79112498509,
"requestMs": 40630.27599998168,
"grade": "slow",
"artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "56",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3092.4598749843426,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 723
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 11867.062500008615,
"inferenceMs": 11866.75399998785,
"tokensPerSecond": 0.4213452136957688,
"inputTokens": 26,
"outputTokens": 5,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 14960.780082998099,
"requestMs": 11868.320208013756,
"grade": "slow",
"artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "READY",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3092.4598749843426,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 723
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 11094.28725001635,
"inferenceMs": 11093.930166010978,
"tokensPerSecond": 0.36055752471337865,
"inputTokens": 29,
"outputTokens": 4,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 14188.063290988794,
"requestMs": 11095.603416004451,
"grade": "slow",
"artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "Oslo",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3092.4598749843426,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 723
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 22823.575167014496,
"inferenceMs": 22823.029707971727,
"tokensPerSecond": 0.21907696147166553,
"inputTokens": 41,
"outputTokens": 5,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 25917.844915995374,
"requestMs": 22825.38504101103,
"grade": "slow",
"artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "ORCHID",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3092.4598749843426,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 723
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 22954.013583017513,
"inferenceMs": 22952.175249985885,
"tokensPerSecond": 0.2614131312021805,
"inputTokens": 51,
"outputTokens": 6,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 26048.110750009073,
"requestMs": 22955.65087502473,
"grade": "slow",
"artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": {
"name": "Ada",
"count": 3
},
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3092.4598749843426,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 723
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 63265.78374998644,
"inferenceMs": 63264.64683399536,
"tokensPerSecond": 0.23709924500739846,
"inputTokens": 40,
"outputTokens": 15,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 66359.52704196097,
"requestMs": 63267.06716697663,
"grade": "slow",
"artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "56",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5198.398041015025,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 723
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 20502.498666988686,
"inferenceMs": 20502.160375006497,
"tokensPerSecond": 0.24387673828243653,
"inputTokens": 26,
"outputTokens": 5,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 25702.14154102723,
"requestMs": 20503.743500012206,
"grade": "slow",
"artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "READY",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5198.398041015025,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 723
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 12366.548625024734,
"inferenceMs": 12366.199374984717,
"tokensPerSecond": 0.32346235724546885,
"inputTokens": 29,
"outputTokens": 4,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 17566.163082025014,
"requestMs": 12367.765041009989,
"grade": "slow",
"artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "Oslo",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5198.398041015025,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 723
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 14845.581332978327,
"inferenceMs": 14845.202457974665,
"tokensPerSecond": 0.3368091485551994,
"inputTokens": 41,
"outputTokens": 5,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 20045.197291008662,
"requestMs": 14846.799249993637,
"grade": "slow",
"artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "ORCHID",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5198.398041015025,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 723
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 17068.991250009276,
"inferenceMs": 17068.413042026805,
"tokensPerSecond": 0.3515265294568665,
"inputTokens": 51,
"outputTokens": 6,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 22268.679000029806,
"requestMs": 17070.28095901478,
"grade": "slow",
"artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": {
"name": "Ada",
"count": 3
},
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5198.398041015025,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 723
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 54743.846916011535,
"inferenceMs": 54743.488874984905,
"tokensPerSecond": 0.27400518871303187,
"inputTokens": 40,
"outputTokens": 15,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 59943.58316602302,
"requestMs": 54745.18512500799,
"grade": "slow",
"artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "56",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7220.5192080000415,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 723
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 2529.6247500227764,
"inferenceMs": 2529.2326670023613,
"tokensPerSecond": 1.9768841614425234,
"inputTokens": 26,
"outputTokens": 5,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 1,
"totalMs": 9751.414750004187,
"requestMs": 2530.8955420041457,
"grade": "slow",
"artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "READY",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7220.5192080000415,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 723
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 2051.846290996764,
"inferenceMs": 2051.380416000029,
"tokensPerSecond": 1.9499064965237258,
"inputTokens": 29,
"outputTokens": 4,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 1,
"totalMs": 9273.638749000384,
"requestMs": 2053.119541000342,
"grade": "slow",
"artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "Oslo",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7220.5192080000415,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 723
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 2568.103750003502,
"inferenceMs": 2567.6854589837603,
"tokensPerSecond": 1.947279010560313,
"inputTokens": 41,
"outputTokens": 5,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 1,
"totalMs": 9789.937000023201,
"requestMs": 2569.4177920231596,
"grade": "slow",
"artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "ORCHID",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7220.5192080000415,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 723
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 3179.395041981479,
"inferenceMs": 3178.9161670021713,
"tokensPerSecond": 1.887435743754831,
"inputTokens": 51,
"outputTokens": 6,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 1,
"totalMs": 10401.232458010782,
"requestMs": 3180.71325001074,
"grade": "slow",
"artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": {
"name": "Ada",
"count": 3
},
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7220.5192080000415,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 723
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 7214.029541995842,
"inferenceMs": 7213.586333004059,
"tokensPerSecond": 2.079409506942621,
"inputTokens": 40,
"outputTokens": 15,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 1,
"totalMs": 14435.75133298873,
"requestMs": 7215.232124988688,
"grade": "slow",
"artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 24039.163874986116,
"inferenceMs": 23993.61454101745,
"tokensPerSecond": 0.1667110219330205,
"inputTokens": 830,
"outputTokens": 4,
"maxNewTokens": 64,
"placement": {
"cpu/torch.float32": 723
},
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"loadMs": 6713.745333021507,
"totalMs": 30760.3046669974,
"wallMs": 24045.84541602526,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
"device": "cpu",
"precision": "fp32",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false,
"parameterDtype": "torch.float32",
"grade": "very-slow"
},
"failure": null
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 51858.392958005425,
"inferenceMs": 51851.46454203641,
"tokensPerSecond": 0.1928585834232882,
"inputTokens": 795,
"outputTokens": 10,
"maxNewTokens": 64,
"placement": {
"cpu/torch.float32": 723
},
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"loadMs": 6713.745333021507,
"totalMs": 82620.61337503837,
"wallMs": 51860.1997080259,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
"device": "cpu",
"precision": "fp32",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false,
"parameterDtype": "torch.float32",
"grade": "very-slow"
},
"failure": null
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 24226.0877090157,
"inferenceMs": 24219.806374982,
"tokensPerSecond": 0.16515408662109807,
"inputTokens": 828,
"outputTokens": 4,
"maxNewTokens": 64,
"placement": {
"cpu/torch.float32": 723
},
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"loadMs": 6713.745333021507,
"totalMs": 106848.3936249977,
"wallMs": 24227.617750002537,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
"device": "cpu",
"precision": "fp32",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false,
"parameterDtype": "torch.float32",
"grade": "very-slow"
},
"failure": null
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 34932.200959010515,
"inferenceMs": 34925.99641595734,
"tokensPerSecond": 0.17179180598147972,
"inputTokens": 154,
"outputTokens": 6,
"maxNewTokens": 64,
"placement": {
"cpu/torch.float32": 723
},
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"inputModality": "video",
"videoFrameCount": 4,
"videoFps": 1,
"loadMs": 6713.745333021507,
"totalMs": 141782.3175000376,
"wallMs": 34933.83512500441,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
"device": "cpu",
"precision": "fp32",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false,
"parameterDtype": "torch.float32",
"grade": "very-slow"
},
"failure": null
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 33461.776458017994,
"inferenceMs": 33455.48454200616,
"tokensPerSecond": 0.17934279183631305,
"inputTokens": 154,
"outputTokens": 6,
"maxNewTokens": 64,
"placement": {
"cpu/torch.float32": 723
},
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"inputModality": "video",
"videoFrameCount": 4,
"videoFps": 1,
"loadMs": 6713.745333021507,
"totalMs": 175245.62566704117,
"wallMs": 33463.22249999503,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
"device": "cpu",
"precision": "fp32",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false,
"parameterDtype": "torch.float32",
"grade": "very-slow"
},
"failure": null
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 23105.48004100565,
"inferenceMs": 23104.981457989197,
"tokensPerSecond": 0.17312283964707045,
"inputTokens": 25,
"outputTokens": 4,
"maxNewTokens": 64,
"placement": {
"cpu/torch.float32": 723
},
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"inputModality": "text",
"videoFrameCount": 0,
"videoFps": null,
"loadMs": 6713.745333021507,
"totalMs": 198352.38491703058,
"wallMs": 23106.679999968037,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
"device": "cpu",
"precision": "fp32",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false,
"parameterDtype": "torch.float32",
"grade": "very-slow"
},
"failure": null
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 5107.982166984584,
"inferenceMs": 5042.1748330118135,
"tokensPerSecond": 0.7933084695539411,
"inputTokens": 830,
"outputTokens": 4,
"maxNewTokens": 64,
"placement": {
"mps:0/torch.float16": 723
},
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"driverAllocatedMemoryBytes": 12799688704,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"loadMs": 3327.418458997272,
"totalMs": 8444.734667020384,
"wallMs": 5114.763375022449,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
"device": "mps",
"precision": "fp16",
"requestedDevice": "mps",
"requestedPrecision": "fp16",
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false,
"parameterDtype": "torch.float16",
"grade": "very-slow"
},
"failure": null
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 4499.359332956374,
"inferenceMs": 4486.562667007092,
"tokensPerSecond": 2.228877816315185,
"inputTokens": 795,
"outputTokens": 10,
"maxNewTokens": 64,
"placement": {
"mps:0/torch.float16": 723
},
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"driverAllocatedMemoryBytes": 14129397760,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"loadMs": 3327.418458997272,
"totalMs": 12946.045083983336,
"wallMs": 4501.26845802879,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
"device": "mps",
"precision": "fp16",
"requestedDevice": "mps",
"requestedPrecision": "fp16",
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false,
"parameterDtype": "torch.float16",
"grade": "slow"
},
"failure": null
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 4503.86937498115,
"inferenceMs": 4489.467957988381,
"tokensPerSecond": 0.8909741727597272,
"inputTokens": 828,
"outputTokens": 4,
"maxNewTokens": 64,
"placement": {
"mps:0/torch.float16": 723
},
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"driverAllocatedMemoryBytes": 15569272832,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"loadMs": 3327.418458997272,
"totalMs": 17451.63041696651,
"wallMs": 4505.496167053934,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
"device": "mps",
"precision": "fp16",
"requestedDevice": "mps",
"requestedPrecision": "fp16",
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false,
"parameterDtype": "torch.float16",
"grade": "very-slow"
},
"failure": null
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 1191.7708750115708,
"inferenceMs": 1179.6049160184339,
"tokensPerSecond": 5.0864487918989285,
"inputTokens": 154,
"outputTokens": 6,
"maxNewTokens": 64,
"placement": {
"mps:0/torch.float16": 723
},
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"inputModality": "video",
"videoFrameCount": 4,
"videoFps": 1,
"driverAllocatedMemoryBytes": 15598862336,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"loadMs": 3327.418458997272,
"totalMs": 18645.038958988152,
"wallMs": 1193.3839159901254,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
"device": "mps",
"precision": "fp16",
"requestedDevice": "mps",
"requestedPrecision": "fp16",
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false,
"parameterDtype": "torch.float16",
"grade": "good"
},
"failure": null
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 1048.1105000362732,
"inferenceMs": 1040.370874979999,
"tokensPerSecond": 5.767174134046524,
"inputTokens": 154,
"outputTokens": 6,
"maxNewTokens": 64,
"placement": {
"mps:0/torch.float16": 723
},
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"inputModality": "video",
"videoFrameCount": 4,
"videoFps": 1,
"driverAllocatedMemoryBytes": 14931968000,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"loadMs": 3327.418458997272,
"totalMs": 19694.801083998755,
"wallMs": 1049.7360409935936,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
"device": "mps",
"precision": "fp16",
"requestedDevice": "mps",
"requestedPrecision": "fp16",
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false,
"parameterDtype": "torch.float16",
"grade": "good"
},
"failure": null
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 506.46420900011435,
"inferenceMs": 504.0246249991469,
"tokensPerSecond": 7.936120184617508,
"inputTokens": 25,
"outputTokens": 4,
"maxNewTokens": 64,
"placement": {
"mps:0/torch.float16": 723
},
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"inputModality": "text",
"videoFrameCount": 0,
"videoFps": null,
"driverAllocatedMemoryBytes": 14940749824,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"loadMs": 3327.418458997272,
"totalMs": 20202.642416988965,
"wallMs": 507.81404197914526,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
"device": "mps",
"precision": "fp16",
"requestedDevice": "mps",
"requestedPrecision": "fp16",
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false,
"parameterDtype": "torch.float16",
"grade": "good"
},
"failure": null
}
]Qwen3.5 2B Desktop — Text, Image Desktop
47 configurations testedQuality criteria
No explicit quality gate recorded. Review per-case evidence; full certification is pending.
| Precision | Device | Result | Platform | Inference / load | |
|---|---|---|---|---|---|
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 219 ms / 2747 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 220 ms / 2845 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 224 ms / 3323 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 574 ms / 2747 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 620 ms / 3323 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 358 ms / 3323 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 365 ms / 2845 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 321 ms / 2845 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 780 ms / 2845 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 391 ms / 2747 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 335 ms / 3323 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 335 ms / 2747 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 279 ms / 2747 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 292 ms / 3323 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 297 ms / 2845 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 334 ms / 5352 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 503 ms / 5352 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 819 ms / 5352 ms |
FP16 | mpsAccelerated | Passed | Slow | macos · aarch64 | 3316 ms / 5352 ms |
FP16 | mpsAccelerated | Passed | Slow | macos · aarch64 | 3322 ms / 5352 ms |
FP16 | mpsAccelerated | Passed | Slow | macos · aarch64 | 3878 ms / 5352 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 3381 ms / 2230 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 1178 ms / 2230 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 947 ms / 2230 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 1431 ms / 2230 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 1217 ms / 2230 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 4890 ms / 2430 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 6158 ms / 2430 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 22845 ms / 2223 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 6134 ms / 2223 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 23081 ms / 2683 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 7696 ms / 2223 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 23206 ms / 2430 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 7823 ms / 2430 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 9562 ms / 2430 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 11576 ms / 2223 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 9912 ms / 2223 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 11090 ms / 2683 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 13635 ms / 2683 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 13325 ms / 2683 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 14388 ms / 2683 ms |
FP32 | cpu | Passed | Very slow | macos · aarch64 | 23946 ms / 4817 ms |
FP32 | cpu | Passed | Very slow | macos · aarch64 | 15604 ms / 4817 ms |
FP32 | cpu | Passed | Very slow | macos · aarch64 | 10831 ms / 4817 ms |
FP32 | cpu | Passed | Very slow | macos · aarch64 | 17335 ms / 4817 ms |
FP32 | cpu | Passed | Very slow | macos · aarch64 | 11917 ms / 4817 ms |
FP32 | cpu | Passed | Very slow | macos · aarch64 | 12747 ms / 4817 ms |
Test setup & sample details
- Quality gate
- No explicit quality gate recorded. Review the per-case quality evidence; full certification is pending.
- Tested
- 2026-11-03T15:08:12.825374+00:00
- Environment
- { "platform": "macOS-27.2-arm64-arm-64bit-Mach-O", "hardware": "Apple M3 Max" }
- Runtime
- transformers 5.22.1
- Artifact revision
- 15852e8c16360a2fea060d615a32b45270f8a8fc
- Fixture
{ "file": "localai-desktop/crates/desktopai/src/models/qwen35/variants/2b/qwen35-2b-chat-fixtures.json", "size": 997, "sha1": "32ddc9a166a9b21ff8d131624c9399f364b49d48", "sha256": "91faa64e06f53ff062dde09f7b5c092dec0c9ff459669c8fc8c0b29b4aa83805", "url": "https://assets.localai.tools/samples/v1/files/91faa64e06f53ff062dde09f7b5c092dec0c9ff459669c8fc8c0b29b4aa83805/qwen35-2b-chat-fixtures.json" }- Rerun when
- Artifact revision; Transformers/Torch/runtime lock; Adapter inference or input/output semantics; Fixture checksum or quality gate; Device/backend/OS behavior
Per-case quality & failure details
[
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "56",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3323.292291985126,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 617
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 225.66925000865012,
"inferenceMs": 224.43504197872244,
"tokensPerSecond": 22.27816100336963,
"inputTokens": 26,
"outputTokens": 5,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 5011357696,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 3550.083583977539,
"requestMs": 226.79129199241288,
"grade": "fast",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "READY",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3323.292291985126,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 617
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 293.8959579914808,
"inferenceMs": 292.0674580091145,
"tokensPerSecond": 13.695466202452355,
"inputTokens": 29,
"outputTokens": 4,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 5011636224,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 3618.3377919951454,
"requestMs": 295.04550001001917,
"grade": "fast",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "Oslo",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3323.292291985126,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 617
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 336.7240829975344,
"inferenceMs": 334.9137080076616,
"tokensPerSecond": 14.9292187224705,
"inputTokens": 41,
"outputTokens": 5,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 5013110784,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 3661.212374980096,
"requestMs": 337.92008299496956,
"grade": "fast",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "ORCHID",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3323.292291985126,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 617
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 359.5950000162702,
"inferenceMs": 357.8500410076231,
"tokensPerSecond": 16.76679981118735,
"inputTokens": 51,
"outputTokens": 6,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 5013635072,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 3683.980708970921,
"requestMs": 360.6884169857949,
"grade": "fast",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": {
"name": "Ada",
"count": 3
},
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3323.292291985126,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 617
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 621.7434579739347,
"inferenceMs": 619.9480420036707,
"tokensPerSecond": 19.356460843421694,
"inputTokens": 52,
"outputTokens": 12,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 5014192128,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 3946.1172509763855,
"requestMs": 622.8249589912593,
"grade": "fast",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "56",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2747.21191599383,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 617
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 220.58758299681358,
"inferenceMs": 219.44704098859802,
"tokensPerSecond": 22.784540531853374,
"inputTokens": 26,
"outputTokens": 5,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 5011357696,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 2968.845750001492,
"requestMs": 221.6338340076618,
"grade": "fast",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "READY",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2747.21191599383,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 617
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 281.4334169961512,
"inferenceMs": 279.31408301810734,
"tokensPerSecond": 14.320795989870259,
"inputTokens": 29,
"outputTokens": 4,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 5011636224,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 3029.7455829859246,
"requestMs": 282.53366699209437,
"grade": "fast",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "Oslo",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2747.21191599383,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 617
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 336.77233298658393,
"inferenceMs": 335.05329201580025,
"tokensPerSecond": 14.922999174006662,
"inputTokens": 41,
"outputTokens": 5,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 5013110784,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 3085.22854099283,
"requestMs": 338.01662499899976,
"grade": "fast",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "ORCHID",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2747.21191599383,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 617
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 393.2164160069078,
"inferenceMs": 391.37712502270006,
"tokensPerSecond": 15.330482075701275,
"inputTokens": 51,
"outputTokens": 6,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 5013635072,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 3141.5627500100527,
"requestMs": 394.3508340162225,
"grade": "fast",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": {
"name": "Ada",
"count": 3
},
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2747.21191599383,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 617
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 575.1745000015944,
"inferenceMs": 573.905916971853,
"tokensPerSecond": 20.909350548808742,
"inputTokens": 52,
"outputTokens": 12,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 5014192128,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 3323.5294990008697,
"requestMs": 576.3175830070395,
"grade": "fast",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "56",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2845.4672080115415,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 617
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 221.7684590141289,
"inferenceMs": 220.10637499624863,
"tokensPerSecond": 22.716288885704547,
"inputTokens": 26,
"outputTokens": 5,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 5011357696,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 3068.401291006012,
"requestMs": 222.93408299447037,
"grade": "fast",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "READY",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2845.4672080115415,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 617
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 298.2424590154551,
"inferenceMs": 296.60899998270907,
"tokensPerSecond": 13.485767458954992,
"inputTokens": 29,
"outputTokens": 4,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 5011636224,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 3144.784748990787,
"requestMs": 299.3175409792457,
"grade": "fast",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "Oslo",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2845.4672080115415,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 617
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 322.52808299381286,
"inferenceMs": 320.9777500014752,
"tokensPerSecond": 15.577403729626182,
"inputTokens": 41,
"outputTokens": 5,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 5013110784,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 3169.1550409886986,
"requestMs": 323.6878329771571,
"grade": "fast",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "ORCHID",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2845.4672080115415,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 617
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 367.1350420045201,
"inferenceMs": 365.40304200025275,
"tokensPerSecond": 16.420224547544542,
"inputTokens": 51,
"outputTokens": 6,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 5013635072,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 3213.7402910157107,
"requestMs": 368.27308300416917,
"grade": "fast",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": {
"name": "Ada",
"count": 3
},
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2845.4672080115415,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 617
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 781.4886249834672,
"inferenceMs": 779.6800000069197,
"tokensPerSecond": 15.390929612011979,
"inputTokens": 52,
"outputTokens": 12,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 5014192128,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"precision": "fp16",
"cpuThreads": 4,
"totalMs": 3628.161666012602,
"requestMs": 782.6944580010604,
"grade": "fast",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "56",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2683.05533300736,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 617
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 11090.146917005768,
"inferenceMs": 11089.807915996062,
"tokensPerSecond": 0.45086443677603694,
"inputTokens": 26,
"outputTokens": 5,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 13774.302750010975,
"requestMs": 11091.247417003615,
"grade": "slow",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "READY",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2683.05533300736,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 617
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 14388.207624986535,
"inferenceMs": 14387.805832986487,
"tokensPerSecond": 0.27801320412799296,
"inputTokens": 29,
"outputTokens": 4,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 17072.45350003359,
"requestMs": 14389.39816702623,
"grade": "slow",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "Oslo",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2683.05533300736,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 617
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 13325.040875002742,
"inferenceMs": 13324.686709005618,
"tokensPerSecond": 0.3752433441171042,
"inputTokens": 41,
"outputTokens": 5,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 16009.35641603428,
"requestMs": 13326.30108302692,
"grade": "slow",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "ORCHID",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2683.05533300736,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 617
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 13635.83916600328,
"inferenceMs": 13635.37383300718,
"tokensPerSecond": 0.44003193997334983,
"inputTokens": 51,
"outputTokens": 6,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 16320.005375018809,
"requestMs": 13636.95004201145,
"grade": "slow",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": {
"name": "Ada",
"count": 3
},
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2683.05533300736,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 617
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 23081.71179101919,
"inferenceMs": 23081.32212498458,
"tokensPerSecond": 0.6498761170948313,
"inputTokens": 52,
"outputTokens": 15,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 25765.78620800865,
"requestMs": 23082.73087500129,
"grade": "slow",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "56",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2222.9325420048553,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 617
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 7696.824666985776,
"inferenceMs": 7696.384582988685,
"tokensPerSecond": 0.6496556852228379,
"inputTokens": 26,
"outputTokens": 5,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 9920.840042002965,
"requestMs": 7697.907499998109,
"grade": "slow",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "READY",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2222.9325420048553,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 617
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 6134.183541988023,
"inferenceMs": 6133.81408402347,
"tokensPerSecond": 0.6521227975296251,
"inputTokens": 29,
"outputTokens": 4,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 8358.188042009715,
"requestMs": 6135.25550000486,
"grade": "slow",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "Oslo",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2222.9325420048553,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 617
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 9912.524874991504,
"inferenceMs": 9912.147750001168,
"tokensPerSecond": 0.5044315446164945,
"inputTokens": 41,
"outputTokens": 5,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 12136.538584018126,
"requestMs": 9913.60604201327,
"grade": "slow",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "ORCHID",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2222.9325420048553,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 617
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 11576.50095797726,
"inferenceMs": 11576.081584003987,
"tokensPerSecond": 0.5183100997051451,
"inputTokens": 51,
"outputTokens": 6,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 13800.527541985502,
"requestMs": 11577.594999980647,
"grade": "slow",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": {
"name": "Ada",
"count": 3
},
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2222.9325420048553,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 617
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 22845.499291986926,
"inferenceMs": 22845.097125013126,
"tokensPerSecond": 0.6565960266186166,
"inputTokens": 52,
"outputTokens": 15,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 25069.577916990966,
"requestMs": 22846.64537498611,
"grade": "slow",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "56",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2430.0800419878215,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 617
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 6158.165667002322,
"inferenceMs": 6157.832541997777,
"tokensPerSecond": 0.8119740129175154,
"inputTokens": 26,
"outputTokens": 5,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 8589.30562500609,
"requestMs": 6159.225583018269,
"grade": "slow",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "READY",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2430.0800419878215,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 617
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 4890.482459013583,
"inferenceMs": 4890.061124984641,
"tokensPerSecond": 0.8179856852019541,
"inputTokens": 29,
"outputTokens": 4,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 7321.600249968469,
"requestMs": 4891.520207980648,
"grade": "slow",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "Oslo",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2430.0800419878215,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 617
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 7823.8121250178665,
"inferenceMs": 7823.420917004114,
"tokensPerSecond": 0.6391066073324213,
"inputTokens": 41,
"outputTokens": 5,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 10254.911166994134,
"requestMs": 7824.8311250063125,
"grade": "slow",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "ORCHID",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2430.0800419878215,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 617
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 9562.146334006684,
"inferenceMs": 9561.730207991786,
"tokensPerSecond": 0.6275014949684673,
"inputTokens": 51,
"outputTokens": 6,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 11993.358459003503,
"requestMs": 9563.278417015681,
"grade": "slow",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": {
"name": "Ada",
"count": 3
},
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2430.0800419878215,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 617
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 23206.297124997946,
"inferenceMs": 23205.806417012354,
"tokensPerSecond": 0.646389947862505,
"inputTokens": 52,
"outputTokens": 15,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 25637.389124982292,
"requestMs": 23207.30908299447,
"grade": "slow",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "56",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2230.336165986955,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 617
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 1177.9553749947809,
"inferenceMs": 1177.6362920063548,
"tokensPerSecond": 4.245793063562463,
"inputTokens": 26,
"outputTokens": 5,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 1,
"totalMs": 3409.3155410082545,
"requestMs": 1178.9793750212993,
"grade": "moderate",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "READY",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2230.336165986955,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 617
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 947.3185840179212,
"inferenceMs": 946.9735419843346,
"tokensPerSecond": 4.223982849212666,
"inputTokens": 29,
"outputTokens": 4,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 1,
"totalMs": 3178.7328330101445,
"requestMs": 948.3966670231894,
"grade": "moderate",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "Oslo",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2230.336165986955,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 617
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 1217.3577500216197,
"inferenceMs": 1217.0138749934267,
"tokensPerSecond": 4.108416594697415,
"inputTokens": 41,
"outputTokens": 5,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 1,
"totalMs": 3448.7515819782857,
"requestMs": 1218.4154159913305,
"grade": "moderate",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": "ORCHID",
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2230.336165986955,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 617
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 1431.6077079856768,
"inferenceMs": 1431.2324580096174,
"tokensPerSecond": 4.192191119214879,
"inputTokens": 51,
"outputTokens": 6,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 1,
"totalMs": 3662.962915987009,
"requestMs": 1432.626750000054,
"grade": "moderate",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"expected": {
"name": "Ada",
"count": 3
},
"method": "Exact normalized string or exact parsed JSON; no truncation permitted"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2230.336165986955,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 617
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
"durationMs": 3381.6107919847127,
"inferenceMs": 3381.2420410104096,
"tokensPerSecond": 4.436239647463268,
"inputTokens": 52,
"outputTokens": 15,
"maxNewTokens": 64,
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 1,
"totalMs": 5612.975666008424,
"requestMs": 3382.639500021469,
"grade": "moderate",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 12791.747083014343,
"inferenceMs": 12747.160833969247,
"tokensPerSecond": 0.31379536605050184,
"inputTokens": 830,
"outputTokens": 4,
"maxNewTokens": 64,
"placement": {
"cpu/torch.float32": 617
},
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"loadMs": 4817.363833019044,
"totalMs": 17616.298042004928,
"wallMs": 12798.182083002757,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"device": "cpu",
"precision": "fp32",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false,
"parameterDtype": "torch.float32",
"grade": "very-slow"
},
"failure": null
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 23953.209292027168,
"inferenceMs": 23946.17183302762,
"tokensPerSecond": 0.41760328413778264,
"inputTokens": 795,
"outputTokens": 10,
"maxNewTokens": 64,
"placement": {
"cpu/torch.float32": 617
},
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"loadMs": 4817.363833019044,
"totalMs": 41571.26137497835,
"wallMs": 23954.88762500463,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"device": "cpu",
"precision": "fp32",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false,
"parameterDtype": "torch.float32",
"grade": "very-slow"
},
"failure": null
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 11922.951209009625,
"inferenceMs": 11916.825708001852,
"tokensPerSecond": 0.3356598559055957,
"inputTokens": 828,
"outputTokens": 4,
"maxNewTokens": 64,
"placement": {
"cpu/torch.float32": 617
},
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"loadMs": 4817.363833019044,
"totalMs": 53495.570999977645,
"wallMs": 11924.195749976207,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"device": "cpu",
"precision": "fp32",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false,
"parameterDtype": "torch.float32",
"grade": "very-slow"
},
"failure": null
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 15623.950292007066,
"inferenceMs": 15604.197374952491,
"tokensPerSecond": 0.3845119268762305,
"inputTokens": 154,
"outputTokens": 6,
"maxNewTokens": 64,
"placement": {
"cpu/torch.float32": 617
},
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"inputModality": "video",
"videoFrameCount": 4,
"videoFps": 1,
"loadMs": 4817.363833019044,
"totalMs": 69120.88779202895,
"wallMs": 15625.252292025834,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"device": "cpu",
"precision": "fp32",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false,
"parameterDtype": "torch.float32",
"grade": "very-slow"
},
"failure": null
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 17341.41916700173,
"inferenceMs": 17335.376416973304,
"tokensPerSecond": 0.3461130497359906,
"inputTokens": 154,
"outputTokens": 6,
"maxNewTokens": 64,
"placement": {
"cpu/torch.float32": 617
},
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"inputModality": "video",
"videoFrameCount": 4,
"videoFps": 1,
"loadMs": 4817.363833019044,
"totalMs": 86463.65883300314,
"wallMs": 17342.69400004996,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"device": "cpu",
"precision": "fp32",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false,
"parameterDtype": "torch.float32",
"grade": "very-slow"
},
"failure": null
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 10831.494499987457,
"inferenceMs": 10831.094250024762,
"tokensPerSecond": 0.36930709932570804,
"inputTokens": 25,
"outputTokens": 4,
"maxNewTokens": 64,
"placement": {
"cpu/torch.float32": 617
},
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"inputModality": "text",
"videoFrameCount": 0,
"videoFps": null,
"loadMs": 4817.363833019044,
"totalMs": 97296.33441701299,
"wallMs": 10832.603749993723,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"device": "cpu",
"precision": "fp32",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false,
"parameterDtype": "torch.float32",
"grade": "very-slow"
},
"failure": null
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 3943.6054169782437,
"inferenceMs": 3878.1595830223523,
"tokensPerSecond": 1.0314170715179014,
"inputTokens": 830,
"outputTokens": 4,
"maxNewTokens": 64,
"placement": {
"mps:0/torch.float16": 617
},
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"driverAllocatedMemoryBytes": 8224980992,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"loadMs": 5351.722374965902,
"totalMs": 9303.320834005717,
"wallMs": 3950.681500020437,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"device": "mps",
"precision": "fp16",
"requestedDevice": "mps",
"requestedPrecision": "fp16",
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false,
"parameterDtype": "torch.float16",
"grade": "slow"
},
"failure": null
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 3330.80495899776,
"inferenceMs": 3316.3356659933925,
"tokensPerSecond": 3.0153763090216468,
"inputTokens": 795,
"outputTokens": 10,
"maxNewTokens": 64,
"placement": {
"mps:0/torch.float16": 617
},
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"driverAllocatedMemoryBytes": 9507192832,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"loadMs": 5351.722374965902,
"totalMs": 12636.323624989018,
"wallMs": 3332.918916014023,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"device": "mps",
"precision": "fp16",
"requestedDevice": "mps",
"requestedPrecision": "fp16",
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false,
"parameterDtype": "torch.float16",
"grade": "slow"
},
"failure": null
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 3349.3286659941077,
"inferenceMs": 3322.10779201705,
"tokensPerSecond": 1.204054850240534,
"inputTokens": 828,
"outputTokens": 4,
"maxNewTokens": 64,
"placement": {
"mps:0/torch.float16": 617
},
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"driverAllocatedMemoryBytes": 10896261120,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"loadMs": 5351.722374965902,
"totalMs": 15987.293709011283,
"wallMs": 3350.8538330206648,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"device": "mps",
"precision": "fp16",
"requestedDevice": "mps",
"requestedPrecision": "fp16",
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false,
"parameterDtype": "torch.float16",
"grade": "slow"
},
"failure": null
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 833.0473329988308,
"inferenceMs": 818.858916987665,
"tokensPerSecond": 7.327269539998649,
"inputTokens": 154,
"outputTokens": 6,
"maxNewTokens": 64,
"placement": {
"mps:0/torch.float16": 617
},
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"inputModality": "video",
"videoFrameCount": 4,
"videoFps": 1,
"driverAllocatedMemoryBytes": 10914562048,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"loadMs": 5351.722374965902,
"totalMs": 16822.04262498999,
"wallMs": 834.6918749739416,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"device": "mps",
"precision": "fp16",
"requestedDevice": "mps",
"requestedPrecision": "fp16",
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false,
"parameterDtype": "torch.float16",
"grade": "good"
},
"failure": null
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 510.3952910285443,
"inferenceMs": 502.99841701053083,
"tokensPerSecond": 11.928466963494207,
"inputTokens": 154,
"outputTokens": 6,
"maxNewTokens": 64,
"placement": {
"mps:0/torch.float16": 617
},
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"inputModality": "video",
"videoFrameCount": 4,
"videoFps": 1,
"driverAllocatedMemoryBytes": 10914562048,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"loadMs": 5351.722374965902,
"totalMs": 17333.99700000882,
"wallMs": 511.88841700786725,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"device": "mps",
"precision": "fp16",
"requestedDevice": "mps",
"requestedPrecision": "fp16",
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false,
"parameterDtype": "torch.float16",
"grade": "good"
},
"failure": null
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 334.9757500109263,
"inferenceMs": 333.5396670154296,
"tokensPerSecond": 11.992576582547706,
"inputTokens": 25,
"outputTokens": 4,
"maxNewTokens": 64,
"placement": {
"mps:0/torch.float16": 617
},
"decoding": "greedy",
"enableThinking": false,
"outputLimitReached": false,
"inputModality": "text",
"videoFrameCount": 0,
"videoFps": null,
"driverAllocatedMemoryBytes": 10914889728,
"memoryMeasurementScope": "Post-inference driver allocation; not peak",
"loadMs": 5351.722374965902,
"totalMs": 17670.212625002023,
"wallMs": 336.16883296053857,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"device": "mps",
"precision": "fp16",
"requestedDevice": "mps",
"requestedPrecision": "fp16",
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false,
"parameterDtype": "torch.float16",
"grade": "good"
},
"failure": null
}
]