Qwen3.5 Desktop
Run local text questions, still-image questions using Qwen3.5 2B. Answers can hallucinate or miss visual details.
Desktop app
Run Qwen3.5 Desktop on your computer
Download the desktop app to load this model, provide input and view results locally.
Public desktop downloads are coming soon.Device requirements
If your device falls short
MemoryAdvisory only: close other apps, reduce image/tile/token/batch settings or choose a smaller verified model/precision. CPU can need more RAM than GPU.
GPUIf the selected GPU runtime fails, existing CPU fallback may recover when enabled, but CPU has its own memory/runtime needs.
PlatformUse a compatible OS/architecture/runtime or another model. These estimates do not add hardware detection or block runs.
Benchmark evidence
Saved desktop measurements apply to the recorded hardware and runtime. Fallback cases report the device that actually ran; they do not prove GPU support on the requested device.
Qwen3.5 2B — Text, Image Desktop
30 configurations testedQuality criteria
No explicit quality gate recorded. Review per-case evidence; full certification is pending.
| Precision | Device | Result | Platform | Inference / load | |
|---|---|---|---|---|---|
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 409 ms / 4710 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 2818 ms / 4829 ms |
FP16 | mpsAccelerated | Passed | Slow | macos · aarch64 | 1686 ms / 4710 ms |
FP16 | mpsAccelerated | Passed | Slow | macos · aarch64 | 1859 ms / 4710 ms |
FP16 | mpsAccelerated | Passed | Slow | macos · aarch64 | 2418 ms / 4829 ms |
FP16 | mpsAccelerated | Passed | Slow | macos · aarch64 | 2433 ms / 3457 ms |
FP16 | mpsAccelerated | Passed | Slow | macos · aarch64 | 2448 ms / 2781 ms |
FP16 | mpsAccelerated | Passed | Slow | macos · aarch64 | 2505 ms / 2781 ms |
FP16 | mpsAccelerated | Passed | Slow | macos · aarch64 | 2539 ms / 4829 ms |
FP16 | mpsAccelerated | Passed | Slow | macos · aarch64 | 2542 ms / 3457 ms |
FP16 | mpsAccelerated | Passed | Slow | macos · aarch64 | 2621 ms / 4829 ms |
FP16 | mpsAccelerated | Passed | Slow | macos · aarch64 | 2729 ms / 3457 ms |
FP16 | mpsAccelerated | Passed | Slow | macos · aarch64 | 2819 ms / 2781 ms |
FP16 | mpsAccelerated | Passed | Slow | macos · aarch64 | 3262 ms / 4710 ms |
FP16 | mpsAccelerated | Passed | Slow | macos · aarch64 | 3501 ms / 4829 ms |
FP16 | mpsAccelerated | Passed | Slow | macos · aarch64 | 3630 ms / 4710 ms |
FP16 | mpsAccelerated | Passed | Slow | macos · aarch64 | 3990 ms / 4710 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 10918 ms / 2248 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 11008 ms / 2248 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 11058 ms / 2248 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 11372 ms / 2248 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 11869 ms / 2515 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 13672 ms / 2515 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 21680 ms / 2248 ms |
FP32 | cpu | Passed | Very slow | macos · aarch64 | 8118 ms / 4161 ms |
FP32 | cpu | Passed | Very slow | macos · aarch64 | 11686 ms / 4161 ms |
FP32 | cpu | Passed | Very slow | macos · aarch64 | 12184 ms / 4161 ms |
FP32 | cpu | Passed | Very slow | macos · aarch64 | 14365 ms / 4161 ms |
FP32 | cpu | Passed | Very slow | macos · aarch64 | 16174 ms / 4161 ms |
FP32 | cpu | Passed | Very slow | macos · aarch64 | 23331 ms / 4161 ms |
Test setup & sample details
- Quality gate
- No explicit quality gate recorded. Review the per-case quality evidence; full certification is pending.
- Tested
- 2026-11-03T12:24:47Z
- Environment
- { "platform": "macOS-27.2-arm64-arm-64bit-Mach-O", "hardware": "Apple M3 Max" }
- Runtime
- transformers 5.22.1
- Artifact revision
- 15852e8c16360a2fea060d615a32b45270f8a8fc
- Fixture
{ "file": "model-samples/desktop/mobilenet-dog.jpg", "size": 83281, "sha1": "f1d042827c2f2154a6d8043b8e8f07c1be612941", "sha256": "d4cb12ea8a768582562e1f1c0c930ea6fc4757f5b66551b986b41498efd75ca1", "url": "https://assets.localai.tools/samples/v1/files/d4cb12ea8a768582562e1f1c0c930ea6fc4757f5b66551b986b41498efd75ca1/mobilenet-dog.jpg" }- Rerun when
- model revision/checksum; pinned runtime/processor/attention/backend/driver; fixture checksum/question/expected answers; visual pixel and token budgets; quality gates
Per-case quality & failure details
[
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4829.139582987409,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 617
},
"parameterDtype": "torch.float16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 3540.975958982017,
"inferenceMs": 3501.2301669921726,
"tokensPerSecond": 1.1424555968099377,
"inputTokens": 830,
"outputTokens": 4,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 8224980992,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"totalMs": 8395.510583010036,
"requestMs": 3566.3710000226274,
"precision": "fp16",
"cpuThreads": 4,
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"localai2026"
],
"method": "normalized exact printed text is contained in answer; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4829.139582987409,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 617
},
"parameterDtype": "torch.float16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 2825.2386249951087,
"inferenceMs": 2817.8865000081714,
"tokensPerSecond": 3.5487589723613784,
"inputTokens": 795,
"outputTokens": 10,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 768,
"inputHeight": 256,
"visionGrid": [
[
1,
32,
96
]
],
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 9507192832,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"totalMs": 7657.8495409630705,
"requestMs": 2828.7099579756614,
"precision": "fp16",
"cpuThreads": 4,
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"fallback": false,
"grade": "moderate"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"3",
"three"
],
"method": "accepted count, with word boundaries; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4829.139582987409,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 617
},
"parameterDtype": "torch.float16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 2629.1852919966914,
"inferenceMs": 2621.339125005761,
"tokensPerSecond": 1.5259376254841537,
"inputTokens": 828,
"outputTokens": 4,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 640,
"inputHeight": 320,
"visionGrid": [
[
1,
40,
80
]
],
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 10896261120,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"totalMs": 7460.689415980596,
"requestMs": 2631.549832993187,
"precision": "fp16",
"cpuThreads": 4,
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4829.139582987409,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 617
},
"parameterDtype": "torch.float16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 2547.124790988164,
"inferenceMs": 2538.6683749966323,
"tokensPerSecond": 1.5756291918219945,
"inputTokens": 830,
"outputTokens": 4,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 10896261120,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"totalMs": 7385.758457996417,
"requestMs": 2556.6188750090078,
"precision": "fp16",
"cpuThreads": 4,
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4829.139582987409,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 617
},
"parameterDtype": "torch.float16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 2426.6185000014957,
"inferenceMs": 2417.6854169927537,
"tokensPerSecond": 1.6544749667950653,
"inputTokens": 830,
"outputTokens": 4,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 10896261120,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"totalMs": 7265.1612909685355,
"requestMs": 2436.0217079811264,
"precision": "fp16",
"cpuThreads": 4,
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3456.591624999419,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 617
},
"parameterDtype": "torch.float16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 2765.446167002665,
"inferenceMs": 2729.4583750190213,
"tokensPerSecond": 1.4654922150890557,
"inputTokens": 830,
"outputTokens": 4,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 8224980992,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"totalMs": 6236.358290974749,
"requestMs": 2779.7666659753304,
"precision": "fp16",
"cpuThreads": 4,
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3456.591624999419,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 617
},
"parameterDtype": "torch.float16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 2442.4569999973755,
"inferenceMs": 2433.4362499939743,
"tokensPerSecond": 1.6437660941435819,
"inputTokens": 830,
"outputTokens": 4,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 8224980992,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"totalMs": 5908.416999998735,
"requestMs": 2451.8253749993164,
"precision": "fp16",
"cpuThreads": 4,
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3456.591624999419,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 617
},
"parameterDtype": "torch.float16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 2550.692542019533,
"inferenceMs": 2541.820583981462,
"tokensPerSecond": 1.5736751937599278,
"inputTokens": 830,
"outputTokens": 4,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 8224980992,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"totalMs": 6017.409708991181,
"requestMs": 2560.8180839917623,
"precision": "fp16",
"cpuThreads": 4,
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2780.9376250079367,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 617
},
"parameterDtype": "torch.float16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 2855.775083997287,
"inferenceMs": 2819.052875012858,
"tokensPerSecond": 1.418916273424547,
"inputTokens": 830,
"outputTokens": 4,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 8224980992,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"totalMs": 5654.870041995309,
"requestMs": 2873.9324169873726,
"precision": "fp16",
"cpuThreads": 4,
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2780.9376250079367,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 617
},
"parameterDtype": "torch.float16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 2456.1110000067856,
"inferenceMs": 2447.9581670020707,
"tokensPerSecond": 1.634014851200934,
"inputTokens": 830,
"outputTokens": 4,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 8224980992,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"totalMs": 5248.155583016342,
"requestMs": 2467.217958008405,
"precision": "fp16",
"cpuThreads": 4,
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2780.9376250079367,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 617
},
"parameterDtype": "torch.float16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 2514.555166999344,
"inferenceMs": 2505.1045839791186,
"tokensPerSecond": 1.5967397231960605,
"inputTokens": 830,
"outputTokens": 4,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 8224980992,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"totalMs": 5304.842166020535,
"requestMs": 2523.904541012598,
"precision": "fp16",
"cpuThreads": 4,
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2248.368542001117,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 617
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 11404.363916022703,
"inferenceMs": 11371.931041008793,
"tokensPerSecond": 0.35174325148257,
"inputTokens": 830,
"outputTokens": 4,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 13666.94083402399,
"requestMs": 11418.572292022873,
"precision": "fp32",
"cpuThreads": 4,
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"localai2026"
],
"method": "normalized exact printed text is contained in answer; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2248.368542001117,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 617
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 21685.598125011893,
"inferenceMs": 21680.47308298992,
"tokensPerSecond": 0.46124454764992223,
"inputTokens": 795,
"outputTokens": 10,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 768,
"inputHeight": 256,
"visionGrid": [
[
1,
32,
96
]
],
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 23936.53741700109,
"requestMs": 21688.168874999974,
"precision": "fp32",
"cpuThreads": 4,
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"3",
"three"
],
"method": "accepted count, with word boundaries; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2248.368542001117,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 617
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 10922.929207998095,
"inferenceMs": 10917.805709003005,
"tokensPerSecond": 0.36637398636811547,
"inputTokens": 828,
"outputTokens": 4,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 640,
"inputHeight": 320,
"visionGrid": [
[
1,
40,
80
]
],
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 13173.251250991598,
"requestMs": 10924.88270899048,
"precision": "fp32",
"cpuThreads": 4,
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2248.368542001117,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 617
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 11013.895374984713,
"inferenceMs": 11008.425042004092,
"tokensPerSecond": 0.36335806300515056,
"inputTokens": 830,
"outputTokens": 4,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 13271.170833992073,
"requestMs": 11022.802291990956,
"precision": "fp32",
"cpuThreads": 4,
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2248.368542001117,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 617
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 11063.421124999877,
"inferenceMs": 11057.724208018044,
"tokensPerSecond": 0.3617380868569292,
"inputTokens": 830,
"outputTokens": 4,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 13320.362375001423,
"requestMs": 11071.993833000306,
"precision": "fp32",
"cpuThreads": 4,
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2514.729874994373,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 617
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 11904.162708990043,
"inferenceMs": 11869.36270800652,
"tokensPerSecond": 0.3370020866665222,
"inputTokens": 830,
"outputTokens": 4,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 14432.75312500191,
"requestMs": 11918.023250007536,
"precision": "fp32",
"cpuThreads": 4,
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2514.729874994373,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 617
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 13678.311458003009,
"inferenceMs": 13672.118499991484,
"tokensPerSecond": 0.2925662178836799,
"inputTokens": 830,
"outputTokens": 4,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 16202.244459011126,
"requestMs": 13687.514584016753,
"precision": "fp32",
"cpuThreads": 4,
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 12227.530415984802,
"inferenceMs": 12184.311417047866,
"tokensPerSecond": 0.3282910181040956,
"inputTokens": 830,
"outputTokens": 4,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"placement": {
"cpu/torch.float32": 617
},
"decoding": "greedy",
"outputLimitReached": false,
"loadMs": 4161.168667022139,
"totalMs": 16396.11612504814,
"wallMs": 12234.045417048037,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"device": "cpu",
"precision": "fp32",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false,
"parameterDtype": "torch.float32",
"grade": "very-slow"
},
"failure": null
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 23337.12162496522,
"inferenceMs": 23330.601792025845,
"tokensPerSecond": 0.4286216056123291,
"inputTokens": 795,
"outputTokens": 10,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 768,
"inputHeight": 256,
"visionGrid": [
[
1,
32,
96
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"placement": {
"cpu/torch.float32": 617
},
"decoding": "greedy",
"outputLimitReached": false,
"loadMs": 4161.168667022139,
"totalMs": 39735.038042010274,
"wallMs": 23338.824875012506,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"device": "cpu",
"precision": "fp32",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false,
"parameterDtype": "torch.float32",
"grade": "very-slow"
},
"failure": null
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 11692.36191699747,
"inferenceMs": 11685.772624972742,
"tokensPerSecond": 0.34229657964180443,
"inputTokens": 828,
"outputTokens": 4,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 640,
"inputHeight": 320,
"visionGrid": [
[
1,
40,
80
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"placement": {
"cpu/torch.float32": 617
},
"decoding": "greedy",
"outputLimitReached": false,
"loadMs": 4161.168667022139,
"totalMs": 51428.82162501337,
"wallMs": 11693.634708004538,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"device": "cpu",
"precision": "fp32",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false,
"parameterDtype": "torch.float32",
"grade": "very-slow"
},
"failure": null
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 14373.424959019758,
"inferenceMs": 14365.009874978568,
"tokensPerSecond": 0.4176815785174635,
"inputTokens": 430,
"outputTokens": 6,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 320,
"inputHeight": 240,
"visionGrid": [
[
2,
24,
32
]
],
"inputModality": "video",
"videoFrameCount": 4,
"videoFps": 1,
"placement": {
"cpu/torch.float32": 617
},
"decoding": "greedy",
"outputLimitReached": false,
"loadMs": 4161.168667022139,
"totalMs": 65803.58358303783,
"wallMs": 14374.706832983065,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"device": "cpu",
"precision": "fp32",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false,
"parameterDtype": "torch.float32",
"grade": "very-slow"
},
"failure": null
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 16181.570166023448,
"inferenceMs": 16174.156375054736,
"tokensPerSecond": 0.37096216092319656,
"inputTokens": 430,
"outputTokens": 6,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 320,
"inputHeight": 240,
"visionGrid": [
[
2,
24,
32
]
],
"inputModality": "video",
"videoFrameCount": 4,
"videoFps": 1,
"placement": {
"cpu/torch.float32": 617
},
"decoding": "greedy",
"outputLimitReached": false,
"loadMs": 4161.168667022139,
"totalMs": 81986.44641705323,
"wallMs": 16182.79829196399,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"device": "cpu",
"precision": "fp32",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false,
"parameterDtype": "torch.float32",
"grade": "very-slow"
},
"failure": null
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 8118.9361669821665,
"inferenceMs": 8118.294834042899,
"tokensPerSecond": 0.49271430537686023,
"inputTokens": 25,
"outputTokens": 4,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": null,
"inputHeight": null,
"visionGrid": null,
"inputModality": "text",
"videoFrameCount": 0,
"videoFps": null,
"placement": {
"cpu/torch.float32": 617
},
"decoding": "greedy",
"outputLimitReached": false,
"loadMs": 4161.168667022139,
"totalMs": 90106.46545805503,
"wallMs": 8119.945833983365,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"device": "cpu",
"precision": "fp32",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false,
"parameterDtype": "torch.float32",
"grade": "very-slow"
},
"failure": null
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 3677.52179200761,
"inferenceMs": 3630.2081670146435,
"tokensPerSecond": 1.1018651867805862,
"inputTokens": 830,
"outputTokens": 4,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"placement": {
"mps:0/torch.float16": 617
},
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 8224980992,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"loadMs": 4710.468957957346,
"totalMs": 8395.532499998808,
"wallMs": 3684.2780000297353,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"device": "mps",
"precision": "fp16",
"requestedDevice": "mps",
"requestedPrecision": "fp16",
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false,
"parameterDtype": "torch.float16",
"grade": "slow"
},
"failure": null
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 4003.968542034272,
"inferenceMs": 3990.380709001329,
"tokensPerSecond": 2.506026549657889,
"inputTokens": 795,
"outputTokens": 10,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 768,
"inputHeight": 256,
"visionGrid": [
[
1,
32,
96
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"placement": {
"mps:0/torch.float16": 617
},
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 9507192832,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"loadMs": 4710.468957957346,
"totalMs": 12403.36599998409,
"wallMs": 4007.735874969512,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"device": "mps",
"precision": "fp16",
"requestedDevice": "mps",
"requestedPrecision": "fp16",
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false,
"parameterDtype": "torch.float16",
"grade": "slow"
},
"failure": null
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 3292.6492909900844,
"inferenceMs": 3261.818917002529,
"tokensPerSecond": 1.2263096455629816,
"inputTokens": 828,
"outputTokens": 4,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 640,
"inputHeight": 320,
"visionGrid": [
[
1,
40,
80
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"placement": {
"mps:0/torch.float16": 617
},
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 10896261120,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"loadMs": 4710.468957957346,
"totalMs": 15698.154749989044,
"wallMs": 3294.620875036344,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"device": "mps",
"precision": "fp16",
"requestedDevice": "mps",
"requestedPrecision": "fp16",
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false,
"parameterDtype": "torch.float16",
"grade": "slow"
},
"failure": null
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 1900.186083978042,
"inferenceMs": 1859.4473749981262,
"tokensPerSecond": 3.2267651565057314,
"inputTokens": 430,
"outputTokens": 6,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 320,
"inputHeight": 240,
"visionGrid": [
[
2,
24,
32
]
],
"inputModality": "video",
"videoFrameCount": 4,
"videoFps": 1,
"placement": {
"mps:0/torch.float16": 617
},
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 11039358976,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"loadMs": 4710.468957957346,
"totalMs": 17601.346749986988,
"wallMs": 1903.1107079936191,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"device": "mps",
"precision": "fp16",
"requestedDevice": "mps",
"requestedPrecision": "fp16",
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false,
"parameterDtype": "torch.float16",
"grade": "slow"
},
"failure": null
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 1726.7867919872515,
"inferenceMs": 1685.939832997974,
"tokensPerSecond": 3.5588458630404816,
"inputTokens": 430,
"outputTokens": 6,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 320,
"inputHeight": 240,
"visionGrid": [
[
2,
24,
32
]
],
"inputModality": "video",
"videoFrameCount": 4,
"videoFps": 1,
"placement": {
"mps:0/torch.float16": 617
},
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 11039358976,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"loadMs": 4710.468957957346,
"totalMs": 19331.72083296813,
"wallMs": 1730.173374991864,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"device": "mps",
"precision": "fp16",
"requestedDevice": "mps",
"requestedPrecision": "fp16",
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false,
"parameterDtype": "torch.float16",
"grade": "slow"
},
"failure": null
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 411.3183749723248,
"inferenceMs": 408.82558398880064,
"tokensPerSecond": 9.784123490934892,
"inputTokens": 25,
"outputTokens": 4,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": null,
"inputHeight": null,
"visionGrid": null,
"inputModality": "text",
"videoFrameCount": 0,
"videoFps": null,
"placement": {
"mps:0/torch.float16": 617
},
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 10372792320,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"loadMs": 4710.468957957346,
"totalMs": 19744.47350000264,
"wallMs": 412.6771250157617,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
"device": "mps",
"precision": "fp16",
"requestedDevice": "mps",
"requestedPrecision": "fp16",
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false,
"parameterDtype": "torch.float16",
"grade": "good"
},
"failure": null
}
]