Qwen3-VL
Run local text questions, still-image questions using Qwen3-VL-8B Instruct. Answers can hallucinate or miss visual details.
Desktop app
Run Qwen3-VL on your computer
Download the desktop app to load this model, provide input and view results locally.
Public desktop downloads are coming soon.Device requirements
If your device falls short
MemoryAdvisory only: close other apps, reduce image/tile/token/batch settings or choose a smaller verified model/precision. CPU can need more RAM than GPU.
GPUIf the selected GPU runtime fails, existing CPU fallback may recover when enabled, but CPU has its own memory/runtime needs.
PlatformUse a compatible OS/architecture/runtime or another model. These estimates do not add hardware detection or block runs.
Benchmark evidence
Saved desktop measurements apply to the recorded hardware and runtime. Fallback cases report the device that actually ran; they do not prove GPU support on the requested device.
Qwen3-VL-8B Instruct — Text, Image
61 configurations testedQuality criteria
No explicit quality gate recorded. Review per-case evidence; full certification is pending.
| Precision | Device | Result | Platform | Inference / load | |
|---|---|---|---|---|---|
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 317 ms / 8203 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 372 ms / 12035 ms |
FP32 | mpsAccelerated | Passed | Slow | macos · aarch64 | 542 ms / 19425 ms |
FP16 | mpsAccelerated | Passed | Slow | macos · aarch64 | 1672 ms / 8203 ms |
BF16 | mpsAccelerated | Passed | Slow | macos · aarch64 | 1773 ms / 12035 ms |
FP16 | mpsAccelerated | Passed | Slow | macos · aarch64 | 1871 ms / 8203 ms |
BF16 | mpsAccelerated | Passed | Slow | macos · aarch64 | 1939 ms / 12035 ms |
FP32 | mpsAccelerated | Passed | Slow | macos · aarch64 | 3074 ms / 19425 ms |
FP32 | mpsAccelerated | Passed | Slow | macos · aarch64 | 3322 ms / 19425 ms |
BF16 | mpsAccelerated | Passed | Slow | macos · aarch64 | 3539 ms / 4467 ms |
BF16 | mpsAccelerated | Passed | Slow | macos · aarch64 | 3543 ms / 4483 ms |
BF16 | mpsAccelerated | Passed | Slow | macos · aarch64 | 3556 ms / 4467 ms |
BF16 | mpsAccelerated | Passed | Slow | macos · aarch64 | 3596 ms / 4324 ms |
BF16 | mpsAccelerated | Passed | Slow | macos · aarch64 | 3599 ms / 4324 ms |
BF16 | mpsAccelerated | Passed | Slow | macos · aarch64 | 3701 ms / 4483 ms |
FP16 | mpsAccelerated | Passed | Slow | macos · aarch64 | 3733 ms / 3912 ms |
FP16 | mpsAccelerated | Passed | Slow | macos · aarch64 | 3757 ms / 3912 ms |
FP16 | mpsAccelerated | Passed | Slow | macos · aarch64 | 3819 ms / 3912 ms |
BF16 | mpsAccelerated | Passed | Slow | macos · aarch64 | 3835 ms / 4483 ms |
BF16 | mpsAccelerated | Passed | Slow | macos · aarch64 | 3835 ms / 4467 ms |
BF16 | mpsAccelerated | Passed | Slow | macos · aarch64 | 3991 ms / 4324 ms |
BF16 | mpsAccelerated | Passed | Slow | macos · aarch64 | 4022 ms / 4483 ms |
BF16 | mpsAccelerated | Passed | Slow | macos · aarch64 | 4275 ms / 4483 ms |
FP16 | mpsAccelerated | Passed | Slow | macos · aarch64 | 4296 ms / 3912 ms |
FP16 | mpsAccelerated | Passed | Slow | macos · aarch64 | 4354 ms / 8203 ms |
FP32 | mpsAccelerated | Passed | Slow | macos · aarch64 | 4455 ms / 14705 ms |
FP32 | mpsAccelerated | Passed | Slow | macos · aarch64 | 4465 ms / 14705 ms |
FP16 | mpsAccelerated | Passed | Slow | macos · aarch64 | 4492 ms / 3912 ms |
BF16 | mpsAccelerated | Passed | Slow | macos · aarch64 | 4517 ms / 12035 ms |
FP32 | mpsAccelerated | Passed | Slow | macos · aarch64 | 4528 ms / 14705 ms |
FP32 | mpsAccelerated | Passed | Slow | macos · aarch64 | 5514 ms / 14705 ms |
FP32 | mpsAccelerated | Passed | Slow | macos · aarch64 | 6360 ms / 14705 ms |
FP32 | mpsAccelerated | Passed | Slow | macos · aarch64 | 6896 ms / 19425 ms |
BF16 | mpsAccelerated | Passed | Very slow | macos · aarch64 | 3886 ms / 12035 ms |
FP16 | mpsAccelerated | Passed | Very slow | macos · aarch64 | 3896 ms / 8203 ms |
BF16 | mpsAccelerated | Passed | Very slow | macos · aarch64 | 4622 ms / 12035 ms |
FP16 | mpsAccelerated | Passed | Very slow | macos · aarch64 | 5040 ms / 8203 ms |
FP32 | mpsAccelerated | Passed | Very slow | macos · aarch64 | 6200 ms / 19425 ms |
FP32 | mpsAccelerated | Passed | Very slow | macos · aarch64 | 7158 ms / 19425 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 1040 ms / 10221 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 8718 ms / 3534 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 8783 ms / 3534 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 8991 ms / 3878 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 9002 ms / 4721 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 9024 ms / 4721 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 9106 ms / 3534 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 9197 ms / 3878 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 9585 ms / 3878 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 9769 ms / 3534 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 9800 ms / 4721 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 10359 ms / 3534 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 13715 ms / 4756 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 13763 ms / 4756 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 14163 ms / 4756 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 14816 ms / 4756 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 14948 ms / 4756 ms |
FP32 | cpu | Passed | Very slow | macos · aarch64 | 5332 ms / 10221 ms |
FP32 | cpu | Passed | Very slow | macos · aarch64 | 5603 ms / 10221 ms |
FP32 | cpu | Passed | Very slow | macos · aarch64 | 9876 ms / 10221 ms |
FP32 | cpu | Passed | Very slow | macos · aarch64 | 10974 ms / 10221 ms |
FP32 | cpu | Passed | Very slow | macos · aarch64 | 11699 ms / 10221 ms |
Test setup & sample details
- Quality gate
- No explicit quality gate recorded. Review the per-case quality evidence; full certification is pending.
- Tested
- 2026-11-01T13:27:46Z
- Environment
- { "platform": "macOS-27.2-arm64-arm-64bit-Mach-O", "hardware": "Apple M3 Max" }
- Runtime
- transformers 5.22.1
- Artifact revision
- 0c351dd01ed87e9c1b53cbc748cba10e6187ff3b
- Fixture
{ "file": "model-samples/desktop/mobilenet-dog.jpg", "url": "https://assets.localai.tools/samples/v1/files/d4cb12ea8a768582562e1f1c0c930ea6fc4757f5b66551b986b41498efd75ca1/mobilenet-dog.jpg", "size": 83281, "sha1": "f1d042827c2f2154a6d8043b8e8f07c1be612941", "sha256": "d4cb12ea8a768582562e1f1c0c930ea6fc4757f5b66551b986b41498efd75ca1" }- Rerun when
- model revision/checksum; pinned runtime/processor/attention/backend/driver; fixture checksum/question/expected answers; visual pixel and token budgets; quality gates
Per-case quality & failure details
[
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4483.0616250001185,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 750
},
"parameterDtype": "torch.bfloat16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 4304.363084000215,
"inferenceMs": 4274.956541999927,
"tokensPerSecond": 0.4678410132011195,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 20967997440,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"totalMs": 8801.348041999518,
"requestMs": 4318.286416999399,
"precision": "bf16",
"cpuThreads": 4,
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"localai2026"
],
"method": "normalized exact printed text is contained in answer; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4483.0616250001185,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 750
},
"parameterDtype": "torch.bfloat16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 4029.289042000528,
"inferenceMs": 4021.5774579992285,
"tokensPerSecond": 1.9892691570784946,
"inputTokens": 791,
"outputTokens": 8,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 768,
"inputHeight": 256,
"visionGrid": [
[
1,
32,
96
]
],
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 23469457408,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"totalMs": 8516.202999999223,
"requestMs": 4033.1413749991043,
"precision": "bf16",
"cpuThreads": 4,
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"3",
"three"
],
"method": "accepted count, with word boundaries; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4483.0616250001185,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 750
},
"parameterDtype": "torch.bfloat16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 3844.211292000182,
"inferenceMs": 3834.7400830007246,
"tokensPerSecond": 0.5215477337997257,
"inputTokens": 824,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 640,
"inputHeight": 320,
"visionGrid": [
[
1,
40,
80
]
],
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 25016041472,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"totalMs": 8330.259250000381,
"requestMs": 3847.1976250002626,
"precision": "bf16",
"cpuThreads": 4,
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4483.0616250001185,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 750
},
"parameterDtype": "torch.bfloat16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 3710.341709000204,
"inferenceMs": 3701.039707999371,
"tokensPerSecond": 0.5403886901502922,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 25016041472,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"totalMs": 8202.381042001434,
"requestMs": 3719.3194170013157,
"precision": "bf16",
"cpuThreads": 4,
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4483.0616250001185,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 750
},
"parameterDtype": "torch.bfloat16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 3552.740207998795,
"inferenceMs": 3543.2400419995247,
"tokensPerSecond": 0.5644551247708745,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 25016041472,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"totalMs": 8044.921875,
"requestMs": 3561.8602499998815,
"precision": "bf16",
"cpuThreads": 4,
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4324.244416000511,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 750
},
"parameterDtype": "torch.bfloat16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 4018.6762920002366,
"inferenceMs": 3990.6664160007495,
"tokensPerSecond": 0.5011694267355732,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 20967997440,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"totalMs": 8357.219833000272,
"requestMs": 4032.9754169997614,
"precision": "bf16",
"cpuThreads": 4,
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4324.244416000511,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 750
},
"parameterDtype": "torch.bfloat16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 3608.2952090000617,
"inferenceMs": 3598.787249999077,
"tokensPerSecond": 0.555742771401814,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 22041739264,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"totalMs": 7943.624916000772,
"requestMs": 3619.3805000002612,
"precision": "bf16",
"cpuThreads": 4,
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4324.244416000511,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 750
},
"parameterDtype": "torch.bfloat16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 3604.3845410003996,
"inferenceMs": 3596.0347079999337,
"tokensPerSecond": 0.5561681580966645,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 22041739264,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"totalMs": 7937.82445800025,
"requestMs": 3613.5800419997395,
"precision": "bf16",
"cpuThreads": 4,
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4467.237291999481,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 750
},
"parameterDtype": "torch.bfloat16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 3866.0650000001624,
"inferenceMs": 3835.346500000014,
"tokensPerSecond": 0.5214652704781674,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 20967997440,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"totalMs": 8346.760874999745,
"requestMs": 3879.523583000264,
"precision": "bf16",
"cpuThreads": 4,
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4467.237291999481,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 750
},
"parameterDtype": "torch.bfloat16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 3564.7898749994056,
"inferenceMs": 3555.5632499999774,
"tokensPerSecond": 0.5624987827174815,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 22041739264,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"totalMs": 8041.285374998552,
"requestMs": 3574.04808299907,
"precision": "bf16",
"cpuThreads": 4,
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4467.237291999481,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 750
},
"parameterDtype": "torch.bfloat16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 3547.0856669999193,
"inferenceMs": 3539.3172090007283,
"tokensPerSecond": 0.5650807435156876,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 22041739264,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"totalMs": 8023.073541999111,
"requestMs": 3555.836249999629,
"precision": "bf16",
"cpuThreads": 4,
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3911.7337499992573,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 750
},
"parameterDtype": "torch.float16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 4521.122790998561,
"inferenceMs": 4491.582208000182,
"tokensPerSecond": 0.44527738943254785,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 20967997440,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"totalMs": 8446.037707999494,
"requestMs": 4534.303958000237,
"precision": "fp16",
"cpuThreads": 4,
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"localai2026"
],
"method": "normalized exact printed text is contained in answer; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3911.7337499992573,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 750
},
"parameterDtype": "torch.float16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 4303.052041001138,
"inferenceMs": 4296.136041999489,
"tokensPerSecond": 1.8621384243401833,
"inputTokens": 791,
"outputTokens": 8,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 768,
"inputHeight": 256,
"visionGrid": [
[
1,
32,
96
]
],
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 23469457408,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"totalMs": 8218.96575000028,
"requestMs": 4307.232000001022,
"precision": "fp16",
"cpuThreads": 4,
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"3",
"three"
],
"method": "accepted count, with word boundaries; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3911.7337499992573,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 750
},
"parameterDtype": "torch.float16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 3832.534332999785,
"inferenceMs": 3819.3320829996082,
"tokensPerSecond": 0.5236517685650549,
"inputTokens": 824,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 640,
"inputHeight": 320,
"visionGrid": [
[
1,
40,
80
]
],
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 25016041472,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"totalMs": 7747.81208299828,
"requestMs": 3836.0783329990227,
"precision": "fp16",
"cpuThreads": 4,
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3911.7337499992573,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 750
},
"parameterDtype": "torch.float16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 3767.1306659995025,
"inferenceMs": 3757.2959170011018,
"tokensPerSecond": 0.5322977066965507,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 25016041472,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"totalMs": 7690.020624999306,
"requestMs": 3778.286875000049,
"precision": "fp16",
"cpuThreads": 4,
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3911.7337499992573,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 750
},
"parameterDtype": "torch.float16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 3744.0704999989975,
"inferenceMs": 3733.072582999739,
"tokensPerSecond": 0.5357517046702812,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 25016041472,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"totalMs": 7670.307541999136,
"requestMs": 3758.573791999879,
"precision": "fp16",
"cpuThreads": 4,
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3533.535499998834,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 750
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 9795.339166999838,
"inferenceMs": 9769.395958001041,
"tokensPerSecond": 0.2047209478045589,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 13345.9869169983,
"requestMs": 9812.451416999465,
"precision": "fp32",
"cpuThreads": 4,
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"localai2026"
],
"method": "normalized exact printed text is contained in answer; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3533.535499998834,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 750
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 10364.259250000032,
"inferenceMs": 10358.760166000138,
"tokensPerSecond": 0.7722931964635944,
"inputTokens": 791,
"outputTokens": 8,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 768,
"inputHeight": 256,
"visionGrid": [
[
1,
32,
96
]
],
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 13901.250082999468,
"requestMs": 10367.714583000634,
"precision": "fp32",
"cpuThreads": 4,
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"3",
"three"
],
"method": "accepted count, with word boundaries; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3533.535499998834,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 750
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 8723.67254100027,
"inferenceMs": 8718.475333998867,
"tokensPerSecond": 0.2293979077053451,
"inputTokens": 824,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 640,
"inputHeight": 320,
"visionGrid": [
[
1,
40,
80
]
],
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 12260.151790998862,
"requestMs": 8726.616291000028,
"precision": "fp32",
"cpuThreads": 4,
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3533.535499998834,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 750
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 8788.448291999885,
"inferenceMs": 8782.657707999533,
"tokensPerSecond": 0.2277215014514723,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 12330.783790997884,
"requestMs": 8797.24829099905,
"precision": "fp32",
"cpuThreads": 4,
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3533.535499998834,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 750
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 9111.44008300107,
"inferenceMs": 9105.68366699954,
"tokensPerSecond": 0.21964303539868416,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 12653.560249998918,
"requestMs": 9120.024750000084,
"precision": "fp32",
"cpuThreads": 4,
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3877.8249580009287,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 750
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 9223.932999999306,
"inferenceMs": 9197.179499999038,
"tokensPerSecond": 0.2174579717619091,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 13116.675000001123,
"requestMs": 9238.850042000195,
"precision": "fp32",
"cpuThreads": 4,
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3877.8249580009287,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 750
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 9592.003375000786,
"inferenceMs": 9585.410166000656,
"tokensPerSecond": 0.20865043491763952,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 13478.728083000533,
"requestMs": 9600.903124999604,
"precision": "fp32",
"cpuThreads": 4,
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3877.8249580009287,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 750
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 8997.418625000137,
"inferenceMs": 8990.72145900027,
"tokensPerSecond": 0.22245155843393147,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 12884.291500002291,
"requestMs": 9006.466542001363,
"precision": "fp32",
"cpuThreads": 4,
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4720.506042000125,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 750
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 9825.947249999444,
"inferenceMs": 9799.925082999835,
"tokensPerSecond": 0.20408319278577425,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 14563.516000000163,
"requestMs": 9843.009958000039,
"precision": "fp32",
"cpuThreads": 4,
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4720.506042000125,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 750
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 9007.693583000218,
"inferenceMs": 9001.534166000056,
"tokensPerSecond": 0.22218434803638867,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 13737.47241700039,
"requestMs": 9016.966375000266,
"precision": "fp32",
"cpuThreads": 4,
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4720.506042000125,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 750
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 9029.408582999167,
"inferenceMs": 9023.566374999064,
"tokensPerSecond": 0.2216418561004055,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 13758.67745799951,
"requestMs": 9038.171415999386,
"precision": "fp32",
"cpuThreads": 4,
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4755.856499999936,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 750
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 14852.60958300023,
"inferenceMs": 14816.483332999269,
"tokensPerSecond": 0.13498479734024338,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 19625.404540998716,
"requestMs": 14869.54804099878,
"precision": "fp32",
"cpuThreads": 1,
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"localai2026"
],
"method": "normalized exact printed text is contained in answer; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4755.856499999936,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 750
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 14960.14683300018,
"inferenceMs": 14948.099916000501,
"tokensPerSecond": 0.535185076695719,
"inputTokens": 791,
"outputTokens": 8,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 768,
"inputHeight": 256,
"visionGrid": [
[
1,
32,
96
]
],
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 19718.798416000936,
"requestMs": 14962.941916001,
"precision": "fp32",
"cpuThreads": 1,
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"3",
"three"
],
"method": "accepted count, with word boundaries; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4755.856499999936,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 750
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 13775.510957999359,
"inferenceMs": 13762.813374998586,
"tokensPerSecond": 0.14531912520394874,
"inputTokens": 824,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 640,
"inputHeight": 320,
"visionGrid": [
[
1,
40,
80
]
],
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 18534.327625000515,
"requestMs": 13778.471125000578,
"precision": "fp32",
"cpuThreads": 1,
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4755.856499999936,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 750
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 13728.663750000123,
"inferenceMs": 13714.888458000132,
"tokensPerSecond": 0.1458269242308978,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 18493.118291000428,
"requestMs": 13737.261791000492,
"precision": "fp32",
"cpuThreads": 1,
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4755.856499999936,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 750
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 14176.67149999943,
"inferenceMs": 14162.826333998964,
"tokensPerSecond": 0.1412147514086821,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 18941.20537500021,
"requestMs": 14185.348875000273,
"precision": "fp32",
"cpuThreads": 1,
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 14705.334291000327,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 750
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 6390.0757920000615,
"inferenceMs": 6360.303584000576,
"tokensPerSecond": 0.31445039903928884,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 38604079104,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"totalMs": 21112.886165999953,
"requestMs": 6407.551874999626,
"precision": "fp32",
"cpuThreads": 4,
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"localai2026"
],
"method": "normalized exact printed text is contained in answer; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 14705.334291000327,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 750
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 5522.364541999195,
"inferenceMs": 5513.772459000393,
"tokensPerSecond": 1.4509122491881616,
"inputTokens": 791,
"outputTokens": 8,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 768,
"inputHeight": 256,
"visionGrid": [
[
1,
32,
96
]
],
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 41112616960,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"totalMs": 20232.68512499999,
"requestMs": 5527.350833999662,
"precision": "fp32",
"cpuThreads": 4,
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"3",
"three"
],
"method": "accepted count, with word boundaries; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 14705.334291000327,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 750
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 4478.391374999774,
"inferenceMs": 4464.863459001208,
"tokensPerSecond": 0.4479420296645311,
"inputTokens": 824,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 640,
"inputHeight": 320,
"visionGrid": [
[
1,
40,
80
]
],
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 43322982400,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"totalMs": 19187.62279099974,
"requestMs": 4482.288499999413,
"precision": "fp32",
"cpuThreads": 4,
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 14705.334291000327,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 750
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 4537.940041000184,
"inferenceMs": 4527.87595799964,
"tokensPerSecond": 0.4417082134210177,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 43989876736,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"totalMs": 19252.835166000295,
"requestMs": 4547.500874999969,
"precision": "fp32",
"cpuThreads": 4,
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"accuracy": 1,
"repetitive": false,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 14705.334291000327,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 750
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 4464.924375000919,
"inferenceMs": 4455.1144580000255,
"tokensPerSecond": 0.44892224854259594,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 43989876736,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"totalMs": 19179.560041000514,
"requestMs": 4474.225750000187,
"precision": "fp32",
"cpuThreads": 4,
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 4670.471958001144,
"inferenceMs": 4621.939624950755,
"tokensPerSecond": 0.43271876361243233,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"placement": {
"mps:0/torch.bfloat16": 750
},
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 20967997440,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"loadMs": 12035.147416987456,
"totalMs": 16713.850374973845,
"wallMs": 4677.5610410259105,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"device": "mps",
"precision": "bf16",
"requestedDevice": "mps",
"requestedPrecision": "bf16",
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false,
"parameterDtype": "torch.bfloat16",
"grade": "very-slow"
},
"failure": null
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 4528.145957970992,
"inferenceMs": 4516.854416986462,
"tokensPerSecond": 1.7711440886636791,
"inputTokens": 791,
"outputTokens": 8,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 768,
"inputHeight": 256,
"visionGrid": [
[
1,
32,
96
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"placement": {
"mps:0/torch.bfloat16": 750
},
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 23469457408,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"loadMs": 12035.147416987456,
"totalMs": 21244.47691696696,
"wallMs": 4530.537542013917,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"device": "mps",
"precision": "bf16",
"requestedDevice": "mps",
"requestedPrecision": "bf16",
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false,
"parameterDtype": "torch.bfloat16",
"grade": "slow"
},
"failure": null
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 3899.396665976383,
"inferenceMs": 3886.4272499922663,
"tokensPerSecond": 0.5146114596649094,
"inputTokens": 824,
"outputTokens": 2,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 640,
"inputHeight": 320,
"visionGrid": [
[
1,
40,
80
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"placement": {
"mps:0/torch.bfloat16": 750
},
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 25005555712,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"loadMs": 12035.147416987456,
"totalMs": 25145.779124984983,
"wallMs": 3901.1726250173524,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"device": "mps",
"precision": "bf16",
"requestedDevice": "mps",
"requestedPrecision": "bf16",
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false,
"parameterDtype": "torch.bfloat16",
"grade": "very-slow"
},
"failure": null
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 1952.4582500453107,
"inferenceMs": 1938.6689999955706,
"tokensPerSecond": 2.063271244348127,
"inputTokens": 426,
"outputTokens": 4,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 320,
"inputHeight": 240,
"visionGrid": [
[
2,
24,
32
]
],
"inputModality": "video",
"videoFrameCount": 4,
"videoFps": 1,
"placement": {
"mps:0/torch.bfloat16": 750
},
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 25495011328,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"loadMs": 12035.147416987456,
"totalMs": 27100.44933401514,
"wallMs": 1954.5968750026077,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"device": "mps",
"precision": "bf16",
"requestedDevice": "mps",
"requestedPrecision": "bf16",
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false,
"parameterDtype": "torch.bfloat16",
"grade": "slow"
},
"failure": null
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 1784.460750001017,
"inferenceMs": 1773.1180000118911,
"tokensPerSecond": 2.255913030025737,
"inputTokens": 426,
"outputTokens": 4,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 320,
"inputHeight": 240,
"visionGrid": [
[
2,
24,
32
]
],
"inputModality": "video",
"videoFrameCount": 4,
"videoFps": 1,
"placement": {
"mps:0/torch.bfloat16": 750
},
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 24823922688,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"loadMs": 12035.147416987456,
"totalMs": 28886.86475000577,
"wallMs": 1786.3384159863926,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"device": "mps",
"precision": "bf16",
"requestedDevice": "mps",
"requestedPrecision": "bf16",
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false,
"parameterDtype": "torch.bfloat16",
"grade": "slow"
},
"failure": null
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 374.75775001803413,
"inferenceMs": 372.2920840373263,
"tokensPerSecond": 5.372126042302523,
"inputTokens": 21,
"outputTokens": 2,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": null,
"inputHeight": null,
"visionGrid": null,
"inputModality": "text",
"videoFrameCount": 0,
"videoFps": null,
"placement": {
"mps:0/torch.bfloat16": 750
},
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 24824102912,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"loadMs": 12035.147416987456,
"totalMs": 29263.1937499973,
"wallMs": 376.2596669839695,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"device": "mps",
"precision": "bf16",
"requestedDevice": "mps",
"requestedPrecision": "bf16",
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false,
"parameterDtype": "torch.bfloat16",
"grade": "good"
},
"failure": null
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 5093.2034580037,
"inferenceMs": 5039.896500005852,
"tokensPerSecond": 0.3968335460852574,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"placement": {
"mps:0/torch.float16": 750
},
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 20967997440,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"loadMs": 8202.622334007174,
"totalMs": 13305.446209036745,
"wallMs": 5100.997000001371,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"device": "mps",
"precision": "fp16",
"requestedDevice": "mps",
"requestedPrecision": "fp16",
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false,
"parameterDtype": "torch.float16",
"grade": "very-slow"
},
"failure": null
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 4366.2984580150805,
"inferenceMs": 4353.796207986306,
"tokensPerSecond": 1.8374769093062615,
"inputTokens": 791,
"outputTokens": 8,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 768,
"inputHeight": 256,
"visionGrid": [
[
1,
32,
96
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"placement": {
"mps:0/torch.float16": 750
},
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 23469457408,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"loadMs": 8202.622334007174,
"totalMs": 17674.725249991752,
"wallMs": 4369.10795798758,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"device": "mps",
"precision": "fp16",
"requestedDevice": "mps",
"requestedPrecision": "fp16",
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false,
"parameterDtype": "torch.float16",
"grade": "slow"
},
"failure": null
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 3909.3517089495435,
"inferenceMs": 3895.908042031806,
"tokensPerSecond": 0.5133591394926647,
"inputTokens": 824,
"outputTokens": 2,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 640,
"inputHeight": 320,
"visionGrid": [
[
1,
40,
80
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"placement": {
"mps:0/torch.float16": 750
},
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 25016041472,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"loadMs": 8202.622334007174,
"totalMs": 21586.266625032295,
"wallMs": 3911.3699159934185,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"device": "mps",
"precision": "fp16",
"requestedDevice": "mps",
"requestedPrecision": "fp16",
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false,
"parameterDtype": "torch.float16",
"grade": "very-slow"
},
"failure": null
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 1893.082750029862,
"inferenceMs": 1871.4858750463463,
"tokensPerSecond": 2.1373391342859813,
"inputTokens": 426,
"outputTokens": 4,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 320,
"inputHeight": 240,
"visionGrid": [
[
2,
24,
32
]
],
"inputModality": "video",
"videoFrameCount": 4,
"videoFps": 1,
"placement": {
"mps:0/torch.float16": 750
},
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 25505497088,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"loadMs": 8202.622334007174,
"totalMs": 23481.45541700069,
"wallMs": 1895.1128749758936,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"device": "mps",
"precision": "fp16",
"requestedDevice": "mps",
"requestedPrecision": "fp16",
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false,
"parameterDtype": "torch.float16",
"grade": "slow"
},
"failure": null
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 1682.2373750037514,
"inferenceMs": 1671.9542079954408,
"tokensPerSecond": 2.3924100198866856,
"inputTokens": 426,
"outputTokens": 4,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 320,
"inputHeight": 240,
"visionGrid": [
[
2,
24,
32
]
],
"inputModality": "video",
"videoFrameCount": 4,
"videoFps": 1,
"placement": {
"mps:0/torch.float16": 750
},
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 24823922688,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"loadMs": 8202.622334007174,
"totalMs": 25165.605584043078,
"wallMs": 1684.0610840008594,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"device": "mps",
"precision": "fp16",
"requestedDevice": "mps",
"requestedPrecision": "fp16",
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false,
"parameterDtype": "torch.float16",
"grade": "slow"
},
"failure": null
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 319.3915000301786,
"inferenceMs": 317.1830829815008,
"tokensPerSecond": 6.305506527019435,
"inputTokens": 21,
"outputTokens": 2,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": null,
"inputHeight": null,
"visionGrid": null,
"inputModality": "text",
"videoFrameCount": 0,
"videoFps": null,
"placement": {
"mps:0/torch.float16": 750
},
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 24824102912,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"loadMs": 8202.622334007174,
"totalMs": 25486.61408398766,
"wallMs": 320.93920902116224,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"device": "mps",
"precision": "fp16",
"requestedDevice": "mps",
"requestedPrecision": "fp16",
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false,
"parameterDtype": "torch.float16",
"grade": "good"
},
"failure": null
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 11013.27591703739,
"inferenceMs": 10973.914750036784,
"tokensPerSecond": 0.18225036785467066,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"placement": {
"cpu/torch.float32": 750
},
"decoding": "greedy",
"outputLimitReached": false,
"loadMs": 10221.168874995783,
"totalMs": 21242.945792037062,
"wallMs": 11019.916124991141,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"device": "cpu",
"precision": "fp32",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false,
"parameterDtype": "torch.float32",
"grade": "very-slow"
},
"failure": null
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 11706.617666990496,
"inferenceMs": 11698.730167001486,
"tokensPerSecond": 0.6838349022328539,
"inputTokens": 791,
"outputTokens": 8,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 768,
"inputHeight": 256,
"visionGrid": [
[
1,
32,
96
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"placement": {
"cpu/torch.float32": 750
},
"decoding": "greedy",
"outputLimitReached": false,
"loadMs": 10221.168874995783,
"totalMs": 32951.75000000745,
"wallMs": 11708.72087503085,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"device": "cpu",
"precision": "fp32",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false,
"parameterDtype": "torch.float32",
"grade": "very-slow"
},
"failure": null
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 9883.481499971822,
"inferenceMs": 9875.795457977802,
"tokensPerSecond": 0.2025153324114639,
"inputTokens": 824,
"outputTokens": 2,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 640,
"inputHeight": 320,
"visionGrid": [
[
1,
40,
80
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"placement": {
"cpu/torch.float32": 750
},
"decoding": "greedy",
"outputLimitReached": false,
"loadMs": 10221.168874995783,
"totalMs": 42837.194583029486,
"wallMs": 9885.31233300455,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"device": "cpu",
"precision": "fp32",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false,
"parameterDtype": "torch.float32",
"grade": "very-slow"
},
"failure": null
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 5341.767417034134,
"inferenceMs": 5331.669541017618,
"tokensPerSecond": 0.7502340438069514,
"inputTokens": 426,
"outputTokens": 4,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 320,
"inputHeight": 240,
"visionGrid": [
[
2,
24,
32
]
],
"inputModality": "video",
"videoFrameCount": 4,
"videoFps": 1,
"placement": {
"cpu/torch.float32": 750
},
"decoding": "greedy",
"outputLimitReached": false,
"loadMs": 10221.168874995783,
"totalMs": 48180.604958033655,
"wallMs": 5343.338917009532,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"device": "cpu",
"precision": "fp32",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false,
"parameterDtype": "torch.float32",
"grade": "very-slow"
},
"failure": null
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 5610.365375003312,
"inferenceMs": 5602.9090830124915,
"tokensPerSecond": 0.7139148504350418,
"inputTokens": 426,
"outputTokens": 4,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 320,
"inputHeight": 240,
"visionGrid": [
[
2,
24,
32
]
],
"inputModality": "video",
"videoFrameCount": 4,
"videoFps": 1,
"placement": {
"cpu/torch.float32": 750
},
"decoding": "greedy",
"outputLimitReached": false,
"loadMs": 10221.168874995783,
"totalMs": 53792.62308299076,
"wallMs": 5611.928957980126,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"device": "cpu",
"precision": "fp32",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false,
"parameterDtype": "torch.float32",
"grade": "very-slow"
},
"failure": null
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 1041.0531670204364,
"inferenceMs": 1040.2970829745755,
"tokensPerSecond": 1.9225277401348624,
"inputTokens": 21,
"outputTokens": 2,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": null,
"inputHeight": null,
"visionGrid": null,
"inputModality": "text",
"videoFrameCount": 0,
"videoFps": null,
"placement": {
"cpu/torch.float32": 750
},
"decoding": "greedy",
"outputLimitReached": false,
"loadMs": 10221.168874995783,
"totalMs": 54834.98145802878,
"wallMs": 1042.285958013963,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"device": "cpu",
"precision": "fp32",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false,
"parameterDtype": "torch.float32",
"grade": "slow"
},
"failure": null
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 7214.809249970131,
"inferenceMs": 7158.1280419486575,
"tokensPerSecond": 0.2794026578288952,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"placement": {
"mps:0/torch.float32": 750
},
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 38604079104,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"loadMs": 19425.220499979332,
"totalMs": 26651.330458989833,
"wallMs": 7223.546083958354,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"device": "mps",
"precision": "fp32",
"requestedDevice": "mps",
"requestedPrecision": "fp32",
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false,
"parameterDtype": "torch.float32",
"grade": "very-slow"
},
"failure": null
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 6917.717540985905,
"inferenceMs": 6895.649957994465,
"tokensPerSecond": 1.1601516969006247,
"inputTokens": 791,
"outputTokens": 8,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 768,
"inputHeight": 256,
"visionGrid": [
[
1,
32,
96
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"placement": {
"mps:0/torch.float32": 750
},
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 41104228352,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"loadMs": 19425.220499979332,
"totalMs": 33572.96566700097,
"wallMs": 6921.330375014804,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"device": "mps",
"precision": "fp32",
"requestedDevice": "mps",
"requestedPrecision": "fp32",
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false,
"parameterDtype": "torch.float32",
"grade": "slow"
},
"failure": null
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 6212.775917025283,
"inferenceMs": 6200.256666983478,
"tokensPerSecond": 0.3225672915526304,
"inputTokens": 824,
"outputTokens": 2,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 640,
"inputHeight": 320,
"visionGrid": [
[
1,
40,
80
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"placement": {
"mps:0/torch.float32": 750
},
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 43312496640,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"loadMs": 19425.220499979332,
"totalMs": 39787.781000020914,
"wallMs": 6214.658958022483,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"device": "mps",
"precision": "fp32",
"requestedDevice": "mps",
"requestedPrecision": "fp32",
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false,
"parameterDtype": "torch.float32",
"grade": "very-slow"
},
"failure": null
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 3334.3274999642745,
"inferenceMs": 3322.058250021655,
"tokensPerSecond": 1.2040728063615156,
"inputTokens": 426,
"outputTokens": 4,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 320,
"inputHeight": 240,
"visionGrid": [
[
2,
24,
32
]
],
"inputModality": "video",
"videoFrameCount": 4,
"videoFps": 1,
"placement": {
"mps:0/torch.float32": 750
},
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 42798858240,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"loadMs": 19425.220499979332,
"totalMs": 43124.384749971796,
"wallMs": 3336.5318340365775,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"device": "mps",
"precision": "fp32",
"requestedDevice": "mps",
"requestedPrecision": "fp32",
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false,
"parameterDtype": "torch.float32",
"grade": "slow"
},
"failure": null
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 3085.90337500209,
"inferenceMs": 3074.002458015457,
"tokensPerSecond": 1.3012351338789616,
"inputTokens": 426,
"outputTokens": 4,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 320,
"inputHeight": 240,
"visionGrid": [
[
2,
24,
32
]
],
"inputModality": "video",
"videoFrameCount": 4,
"videoFps": 1,
"placement": {
"mps:0/torch.float32": 750
},
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 42798858240,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"loadMs": 19425.220499979332,
"totalMs": 46212.73137500975,
"wallMs": 3088.235333969351,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"device": "mps",
"precision": "fp32",
"requestedDevice": "mps",
"requestedPrecision": "fp32",
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false,
"parameterDtype": "torch.float32",
"grade": "slow"
},
"failure": null
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 546.0952919675037,
"inferenceMs": 541.8673329986632,
"tokensPerSecond": 3.690940343150256,
"inputTokens": 21,
"outputTokens": 2,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": null,
"inputHeight": null,
"visionGrid": null,
"inputModality": "text",
"videoFrameCount": 0,
"videoFps": null,
"placement": {
"mps:0/torch.float32": 750
},
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 42824204288,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"loadMs": 19425.220499979332,
"totalMs": 46760.43570897309,
"wallMs": 547.5516250007786,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
"device": "mps",
"precision": "fp32",
"requestedDevice": "mps",
"requestedPrecision": "fp32",
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false,
"parameterDtype": "torch.float32",
"grade": "slow"
},
"failure": null
}
]Qwen3-VL 2B Instruct — Text, Image
45 configurations testedQuality criteria
No explicit quality gate recorded. Review per-case evidence; full certification is pending.
| Precision | Device | Result | Platform | Inference / load | |
|---|---|---|---|---|---|
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 162 ms / 5577 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 1415 ms / 5577 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 1505 ms / 5577 ms |
FP32 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 2034 ms / 0 ms |
FP32 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 2051 ms / 0 ms |
FP32 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 2053 ms / 0 ms |
FP32 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 2057 ms / 0 ms |
FP32 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 2057 ms / 0 ms |
FP32 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 2106 ms / 0 ms |
FP32 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 2171 ms / 0 ms |
FP32 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 2311 ms / 1164 ms |
FP32 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 2316 ms / 1171 ms |
FP32 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 2580 ms / 0 ms |
FP32 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 4522 ms / 1216 ms |
FP32 | mpsAccelerated | Passed | Slow | macos · aarch64 | 2433 ms / 5577 ms |
FP32 | mpsAccelerated | Passed | Very slow | macos · aarch64 | 2257 ms / 5577 ms |
FP32 | mpsAccelerated | Passed | Very slow | macos · aarch64 | 2565 ms / 5577 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 269 ms / 4074 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 3109 ms / 4074 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 3172 ms / 4074 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 3792 ms / 0 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 3819 ms / 0 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 3842 ms / 0 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 3874 ms / 0 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 3876 ms / 0 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 3895 ms / 0 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 3966 ms / 0 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 3971 ms / 492 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 3979 ms / 521 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 4130 ms / 0 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 4135 ms / 600 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 4440 ms / 4074 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 7702 ms / 0 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 7745 ms / 0 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 7793 ms / 0 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 7801 ms / 0 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 7834 ms / 0 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 7871 ms / 0 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 7883 ms / 0 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 7904 ms / 0 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 7993 ms / 554 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 8033 ms / 637 ms |
FP32 | cpu | Passed | Slow | macos · aarch64 | 8062 ms / 587 ms |
FP32 | cpu | Passed | Very slow | macos · aarch64 | 4109 ms / 4074 ms |
FP32 | cpu | Passed | Very slow | macos · aarch64 | 4350 ms / 4074 ms |
Test setup & sample details
- Quality gate
- No explicit quality gate recorded. Review the per-case quality evidence; full certification is pending.
- Tested
- 2026-11-01T22:56:20.265785+00:00
- Environment
- { "os": "macos", "arch": "aarch64", "hardware": "Apple M3 Max", "osVersion": "27.2", "python": "App-owned CPython3.13.18 private model probe venv" }
- Runtime
- transformers 5.22.1
- Artifact revision
- 89644892e4d85e24eaac8bacfd4f463576704203
- Fixture
{ "file": "model-samples/desktop/mobilenet-dog.jpg", "size": 83281, "sha1": "f1d042827c2f2154a6d8043b8e8f07c1be612941", "sha256": "d4cb12ea8a768582562e1f1c0c930ea6fc4757f5b66551b986b41498efd75ca1", "url": "https://assets.localai.tools/samples/v1/files/d4cb12ea8a768582562e1f1c0c930ea6fc4757f5b66551b986b41498efd75ca1/mobilenet-dog.jpg" }- Rerun when
- model revision/checksum; pinned runtime/processor/attention/backend/driver; fixture checksum/question/expected answers; visual pixel and token budgets; quality gates
Per-case quality & failure details
[
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"repetitive": false,
"eosWithinBudget": true,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
]
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 636.9023750012275,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 8075.678832996346,
"inferenceMs": 8033.141791995149,
"tokensPerSecond": 0.2489685918394913,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 1,
"totalMs": 8712.581207997573,
"artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"repetitive": false,
"eosWithinBudget": true,
"expected": [
"localai2026"
]
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 0,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 7812.195541999245,
"inferenceMs": 7801.169833997847,
"tokensPerSecond": 1.0254872243821231,
"inputTokens": 791,
"outputTokens": 8,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 768,
"inputHeight": 256,
"visionGrid": [
[
1,
32,
96
]
],
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 1,
"totalMs": 7812.195541999245,
"artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"repetitive": false,
"eosWithinBudget": true,
"expected": [
"3",
"three"
]
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 0,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 7713.636000000406,
"inferenceMs": 7702.218833997904,
"tokensPerSecond": 0.2596654344812847,
"inputTokens": 824,
"outputTokens": 2,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 640,
"inputHeight": 320,
"visionGrid": [
[
1,
40,
80
]
],
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 1,
"totalMs": 7713.636000000406,
"artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"repetitive": false,
"eosWithinBudget": true,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
]
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 0,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 7896.8052909986,
"inferenceMs": 7882.819667000149,
"tokensPerSecond": 0.2537163203634609,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 1,
"totalMs": 7896.8052909986,
"artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"repetitive": false,
"eosWithinBudget": true,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
]
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 0,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 7916.936333000194,
"inferenceMs": 7903.680291005003,
"tokensPerSecond": 0.2530466727349984,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 1,
"totalMs": 7916.936333000194,
"artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"repetitive": false,
"eosWithinBudget": true,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
]
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 554.4171250003274,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 8024.448666998069,
"inferenceMs": 7992.687541001942,
"tokensPerSecond": 0.250228723410009,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 1,
"totalMs": 8578.865791998396,
"artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"repetitive": false,
"eosWithinBudget": true,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
]
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 0,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 7885.141166996618,
"inferenceMs": 7870.676416998322,
"tokensPerSecond": 0.2541077658434279,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 1,
"totalMs": 7885.141166996618,
"artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"repetitive": false,
"eosWithinBudget": true,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
]
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 0,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 7847.544374999416,
"inferenceMs": 7834.322167000209,
"tokensPerSecond": 0.2552869230249957,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 1,
"totalMs": 7847.544374999416,
"artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"repetitive": false,
"eosWithinBudget": true,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
]
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 586.6140419966541,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 8093.579958003829,
"inferenceMs": 8061.844624993682,
"tokensPerSecond": 0.2480821813161113,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 1,
"totalMs": 8680.194000000483,
"artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"repetitive": false,
"eosWithinBudget": true,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
]
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 0,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 7806.9487499960815,
"inferenceMs": 7793.158833002963,
"tokensPerSecond": 0.25663534426249257,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 1,
"totalMs": 7806.9487499960815,
"artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"repetitive": false,
"eosWithinBudget": true,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
]
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 0,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 7758.417208002356,
"inferenceMs": 7745.325916999718,
"tokensPerSecond": 0.25822025069472265,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 1,
"totalMs": 7758.417208002356,
"artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
"fallback": false,
"grade": "slow"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"repetitive": false,
"eosWithinBudget": true,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
]
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 492.23937500210013,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 3995.2576249997946,
"inferenceMs": 3971.08383399609,
"tokensPerSecond": 0.5036408405378352,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 4487.497000001895,
"artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
"fallback": false,
"grade": "moderate"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"repetitive": false,
"eosWithinBudget": true,
"expected": [
"localai2026"
]
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 0,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 4135.194957998465,
"inferenceMs": 4130.113082996104,
"tokensPerSecond": 1.9369929682885505,
"inputTokens": 791,
"outputTokens": 8,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 768,
"inputHeight": 256,
"visionGrid": [
[
1,
32,
96
]
],
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 4135.194957998465,
"artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
"fallback": false,
"grade": "moderate"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"repetitive": false,
"eosWithinBudget": true,
"expected": [
"3",
"three"
]
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 0,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 3846.928916995239,
"inferenceMs": 3841.9710000016494,
"tokensPerSecond": 0.5205661364958615,
"inputTokens": 824,
"outputTokens": 2,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 640,
"inputHeight": 320,
"visionGrid": [
[
1,
40,
80
]
],
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 3846.928916995239,
"artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
"fallback": false,
"grade": "moderate"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"repetitive": false,
"eosWithinBudget": true,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
]
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 0,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 3798.2420839980477,
"inferenceMs": 3792.3042500042357,
"tokensPerSecond": 0.5273838458498593,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 3798.2420839980477,
"artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
"fallback": false,
"grade": "moderate"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"repetitive": false,
"eosWithinBudget": true,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
]
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 0,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 3824.5153750031022,
"inferenceMs": 3818.551166994439,
"tokensPerSecond": 0.5237588584086433,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 3824.5153750031022,
"artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
"fallback": false,
"grade": "moderate"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"repetitive": false,
"eosWithinBudget": true,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
]
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 520.9806249986286,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 4003.420166001888,
"inferenceMs": 3978.965583999525,
"tokensPerSecond": 0.502643201550305,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 4524.400791000517,
"artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
"fallback": false,
"grade": "moderate"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"repetitive": false,
"eosWithinBudget": true,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
]
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 0,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 3901.0117080033524,
"inferenceMs": 3894.970082998043,
"tokensPerSecond": 0.5134827629948204,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 3901.0117080033524,
"artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
"fallback": false,
"grade": "moderate"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"repetitive": false,
"eosWithinBudget": true,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
]
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 0,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 3881.6745840013027,
"inferenceMs": 3875.833917001728,
"tokensPerSecond": 0.5160179829240883,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 3881.6745840013027,
"artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
"fallback": false,
"grade": "moderate"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"repetitive": false,
"eosWithinBudget": true,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
]
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 600.1735420068144,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 4163.385208994441,
"inferenceMs": 4135.028584001702,
"tokensPerSecond": 0.4836725936400871,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 4763.558751001256,
"artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
"fallback": false,
"grade": "moderate"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"repetitive": false,
"eosWithinBudget": true,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
]
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 0,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 3972.2970000002533,
"inferenceMs": 3966.473833002965,
"tokensPerSecond": 0.5042261928867501,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 3972.2970000002533,
"artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
"fallback": false,
"grade": "moderate"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"repetitive": false,
"eosWithinBudget": true,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
]
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 0,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 3878.9564589969814,
"inferenceMs": 3873.626917004003,
"tokensPerSecond": 0.5163119843112999,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 3878.9564589969814,
"artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
"fallback": false,
"grade": "moderate"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"repetitive": false,
"eosWithinBudget": true,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
]
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1215.6197090007481,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 4551.082375000988,
"inferenceMs": 4522.304082995106,
"tokensPerSecond": 0.4422524366551236,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 11734597632,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 5766.702084001736,
"artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
"fallback": false,
"grade": "moderate"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"repetitive": false,
"eosWithinBudget": true,
"expected": [
"localai2026"
]
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 0,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 2587.805874994956,
"inferenceMs": 2579.6897920008632,
"tokensPerSecond": 3.1011480623780843,
"inputTokens": 791,
"outputTokens": 8,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 768,
"inputHeight": 256,
"visionGrid": [
[
1,
32,
96
]
],
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 13665746944,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 2587.805874994956,
"artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
"fallback": false,
"grade": "moderate"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"repetitive": false,
"eosWithinBudget": true,
"expected": [
"3",
"three"
]
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 0,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 2179.3269169938867,
"inferenceMs": 2170.8504580019508,
"tokensPerSecond": 0.9212979146618873,
"inputTokens": 824,
"outputTokens": 2,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 640,
"inputHeight": 320,
"visionGrid": [
[
1,
40,
80
]
],
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15760506880,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 2179.3269169938867,
"artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
"fallback": false,
"grade": "moderate"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"repetitive": false,
"eosWithinBudget": true,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
]
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 0,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 2115.038416995958,
"inferenceMs": 2106.0563749997527,
"tokensPerSecond": 0.9496421955942347,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 16427401216,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 2115.038416995958,
"artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
"fallback": false,
"grade": "moderate"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"repetitive": false,
"eosWithinBudget": true,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
]
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 0,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 2066.274583004997,
"inferenceMs": 2056.9934170052875,
"tokensPerSecond": 0.9722928539614568,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 16427401216,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 2066.274583004997,
"artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
"fallback": false,
"grade": "moderate"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"repetitive": false,
"eosWithinBudget": true,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
]
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1164.4891250034561,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 2338.0629169987515,
"inferenceMs": 2310.9830410030554,
"tokensPerSecond": 0.8654325732879127,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 11734597632,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 3502.5520420022076,
"artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
"fallback": false,
"grade": "moderate"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"repetitive": false,
"eosWithinBudget": true,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
]
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 0,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 2065.581499999098,
"inferenceMs": 2056.772625001031,
"tokensPerSecond": 0.9723972284000024,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 12401491968,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 2065.581499999098,
"artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
"fallback": false,
"grade": "moderate"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"repetitive": false,
"eosWithinBudget": true,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
]
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 0,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 2060.4178329958813,
"inferenceMs": 2050.706417001493,
"tokensPerSecond": 0.9752736829703615,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 12401491968,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 2060.4178329958813,
"artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
"fallback": false,
"grade": "moderate"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"repetitive": false,
"eosWithinBudget": true,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
]
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 1170.8507910007029,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 2343.7331249951967,
"inferenceMs": 2315.727458000765,
"tokensPerSecond": 0.8636594920054445,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 11734597632,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 3514.5839159958996,
"artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
"fallback": false,
"grade": "moderate"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"repetitive": false,
"eosWithinBudget": true,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
]
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 0,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 2063.3216659989557,
"inferenceMs": 2053.3383330039214,
"tokensPerSecond": 0.9740236023715145,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 12401491968,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 2063.3216659989557,
"artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
"fallback": false,
"grade": "moderate"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"repetitive": false,
"eosWithinBudget": true,
"expected": [
"dog",
"labrador",
"puppy",
"canine"
]
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 0,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 2042.1250840008724,
"inferenceMs": 2033.7423340024543,
"tokensPerSecond": 0.983408746802232,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 12401491968,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"precision": "fp32",
"cpuThreads": 4,
"totalMs": 2042.1250840008724,
"artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
"fallback": false,
"grade": "moderate"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 4388.4951249929145,
"inferenceMs": 4350.370042026043,
"tokensPerSecond": 0.45973100694408175,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"placement": {
"cpu/torch.float32": 625
},
"decoding": "greedy",
"outputLimitReached": false,
"loadMs": 4073.553374968469,
"totalMs": 8468.864374968689,
"wallMs": 4394.699833006598,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
"device": "cpu",
"precision": "fp32",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false,
"parameterDtype": "torch.float32",
"grade": "very-slow"
},
"failure": null
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 4447.265790950041,
"inferenceMs": 4440.3853340190835,
"tokensPerSecond": 1.8016454425046569,
"inputTokens": 791,
"outputTokens": 8,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 768,
"inputHeight": 256,
"visionGrid": [
[
1,
32,
96
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"placement": {
"cpu/torch.float32": 625
},
"decoding": "greedy",
"outputLimitReached": false,
"loadMs": 4073.553374968469,
"totalMs": 12917.885833012406,
"wallMs": 4448.937541979831,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
"device": "cpu",
"precision": "fp32",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false,
"parameterDtype": "torch.float32",
"grade": "slow"
},
"failure": null
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 4116.308333002962,
"inferenceMs": 4109.459625033196,
"tokensPerSecond": 0.4866819928870439,
"inputTokens": 824,
"outputTokens": 2,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 640,
"inputHeight": 320,
"visionGrid": [
[
1,
40,
80
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"placement": {
"cpu/torch.float32": 625
},
"decoding": "greedy",
"outputLimitReached": false,
"loadMs": 4073.553374968469,
"totalMs": 17035.700124979485,
"wallMs": 4117.6878749975,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
"device": "cpu",
"precision": "fp32",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false,
"parameterDtype": "torch.float32",
"grade": "very-slow"
},
"failure": null
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 3180.0782500067726,
"inferenceMs": 3171.657375001814,
"tokensPerSecond": 5.675266232056262,
"inputTokens": 426,
"outputTokens": 18,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 320,
"inputHeight": 240,
"visionGrid": [
[
2,
24,
32
]
],
"inputModality": "video",
"videoFrameCount": 4,
"videoFps": 1,
"placement": {
"cpu/torch.float32": 625
},
"decoding": "greedy",
"outputLimitReached": false,
"loadMs": 4073.553374968469,
"totalMs": 20217.279249976855,
"wallMs": 3181.5036250045523,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
"device": "cpu",
"precision": "fp32",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false,
"parameterDtype": "torch.float32",
"grade": "good"
},
"failure": null
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 3116.1493330146186,
"inferenceMs": 3108.579166990239,
"tokensPerSecond": 5.790426761891929,
"inputTokens": 426,
"outputTokens": 18,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 320,
"inputHeight": 240,
"visionGrid": [
[
2,
24,
32
]
],
"inputModality": "video",
"videoFrameCount": 4,
"videoFps": 1,
"placement": {
"cpu/torch.float32": 625
},
"decoding": "greedy",
"outputLimitReached": false,
"loadMs": 4073.553374968469,
"totalMs": 23334.819875017274,
"wallMs": 3117.4743750016205,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
"device": "cpu",
"precision": "fp32",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false,
"parameterDtype": "torch.float32",
"grade": "good"
},
"failure": null
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 269.33770801406354,
"inferenceMs": 268.63662496907637,
"tokensPerSecond": 7.445001217649405,
"inputTokens": 21,
"outputTokens": 2,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": null,
"inputHeight": null,
"visionGrid": null,
"inputModality": "text",
"videoFrameCount": 0,
"videoFps": null,
"placement": {
"cpu/torch.float32": 625
},
"decoding": "greedy",
"outputLimitReached": false,
"loadMs": 4073.553374968469,
"totalMs": 23605.354583007284,
"wallMs": 270.4772499855608,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
"device": "cpu",
"precision": "fp32",
"requestedDevice": "cpu",
"requestedPrecision": "fp32",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false,
"parameterDtype": "torch.float32",
"grade": "good"
},
"failure": null
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 2607.4020829983056,
"inferenceMs": 2565.0439590099268,
"tokensPerSecond": 0.77971373277048,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"placement": {
"mps:0/torch.float32": 625
},
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 11734597632,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"loadMs": 5577.300457982346,
"totalMs": 8192.509624990635,
"wallMs": 2614.388332993258,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
"device": "mps",
"precision": "fp32",
"requestedDevice": "mps",
"requestedPrecision": "fp32",
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false,
"parameterDtype": "torch.float32",
"grade": "very-slow"
},
"failure": null
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 2443.650291999802,
"inferenceMs": 2432.5532920192927,
"tokensPerSecond": 3.2887254829098116,
"inputTokens": 791,
"outputTokens": 8,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 768,
"inputHeight": 256,
"visionGrid": [
[
1,
32,
96
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"placement": {
"mps:0/torch.float32": 625
},
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 13665746944,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"loadMs": 5577.300457982346,
"totalMs": 10638.538207975216,
"wallMs": 2445.9031670121476,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
"device": "mps",
"precision": "fp32",
"requestedDevice": "mps",
"requestedPrecision": "fp32",
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false,
"parameterDtype": "torch.float32",
"grade": "slow"
},
"failure": null
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 2265.8370000426657,
"inferenceMs": 2256.543666997459,
"tokensPerSecond": 0.8863112330820462,
"inputTokens": 824,
"outputTokens": 2,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 640,
"inputHeight": 320,
"visionGrid": [
[
1,
40,
80
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"placement": {
"mps:0/torch.float32": 625
},
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15760506880,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"loadMs": 5577.300457982346,
"totalMs": 12906.167624983937,
"wallMs": 2267.510832985863,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
"device": "mps",
"precision": "fp32",
"requestedDevice": "mps",
"requestedPrecision": "fp32",
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false,
"parameterDtype": "torch.float32",
"grade": "very-slow"
},
"failure": null
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 1517.6930840243585,
"inferenceMs": 1504.8609169898555,
"tokensPerSecond": 11.961238275763753,
"inputTokens": 426,
"outputTokens": 18,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 320,
"inputHeight": 240,
"visionGrid": [
[
2,
24,
32
]
],
"inputModality": "video",
"videoFrameCount": 4,
"videoFps": 1,
"placement": {
"mps:0/torch.float32": 625
},
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15879962624,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"loadMs": 5577.300457982346,
"totalMs": 14425.565124955028,
"wallMs": 1519.3294169730507,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
"device": "mps",
"precision": "fp32",
"requestedDevice": "mps",
"requestedPrecision": "fp32",
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false,
"parameterDtype": "torch.float32",
"grade": "good"
},
"failure": null
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 1424.7224580030888,
"inferenceMs": 1414.5067500066943,
"tokensPerSecond": 12.725283919581729,
"inputTokens": 426,
"outputTokens": 18,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 320,
"inputHeight": 240,
"visionGrid": [
[
2,
24,
32
]
],
"inputModality": "video",
"videoFrameCount": 4,
"videoFps": 1,
"placement": {
"mps:0/torch.float32": 625
},
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15879962624,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"loadMs": 5577.300457982346,
"totalMs": 15852.031165966764,
"wallMs": 1426.3948749867268,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
"device": "mps",
"precision": "fp32",
"requestedDevice": "mps",
"requestedPrecision": "fp32",
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false,
"parameterDtype": "torch.float32",
"grade": "good"
},
"failure": null
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"scope": "Predeclared fixture answer check",
"nonEmpty": true,
"repetitive": false,
"grade": "fixture-pass",
"score": 1,
"gibberish": false
},
"runtime": {
"durationMs": 164.1484170104377,
"inferenceMs": 162.03762497752905,
"tokensPerSecond": 12.342812357792548,
"inputTokens": 21,
"outputTokens": 2,
"maxNewTokens": 64,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": null,
"inputHeight": null,
"visionGrid": null,
"inputModality": "text",
"videoFrameCount": 0,
"videoFps": null,
"placement": {
"mps:0/torch.float32": 625
},
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15896854528,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"loadMs": 5577.300457982346,
"totalMs": 16017.471041006502,
"wallMs": 165.37175001576543,
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
"device": "mps",
"precision": "fp32",
"requestedDevice": "mps",
"requestedPrecision": "fp32",
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false,
"parameterDtype": "torch.float32",
"grade": "good"
},
"failure": null
}
]Qwen3-VL 2B Thinking — Text, Image
42 configurations testedQuality criteria
{ "method": "Final-answer fixture semantic/normalized OCR plus completion/repetition/speed gates frozen in research/variant-onboarding/qwen3-vl-family/PLAN.md", "fixtures": [ "dog", "ocr", "circles", "arithmetic", "instruction" ] }
| Precision | Device | Result | Platform | Inference / load | |
|---|---|---|---|---|---|
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 3100 ms / 2969 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 3209 ms / 2690 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 3212 ms / 2682 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 3271 ms / 2682 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 3277 ms / 2682 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 3479 ms / 2690 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 3493 ms / 2682 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 3503 ms / 2969 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 3611 ms / 2690 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 3624 ms / 2969 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 3636 ms / 2690 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 3677 ms / 2682 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 3688 ms / 2690 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 4052 ms / 2969 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 4537 ms / 2969 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 5421 ms / 2969 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 5654 ms / 2690 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 5660 ms / 2682 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 7855 ms / 8341 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 8828 ms / 3022 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 9656 ms / 3022 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 9703 ms / 5031 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 10773 ms / 5031 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 10818 ms / 8341 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 10988 ms / 8341 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 11225 ms / 5031 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 11906 ms / 3022 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 12038 ms / 3022 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 12123 ms / 8341 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 12276 ms / 2296 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 12396 ms / 2296 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 12810 ms / 2296 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 13093 ms / 8341 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 13476 ms / 3022 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 13521 ms / 5031 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 14793 ms / 2296 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 14916 ms / 2296 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 16267 ms / 2296 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 16311 ms / 5031 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 18659 ms / 3022 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 19973 ms / 8341 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 21395 ms / 5031 ms |
Test setup & sample details
- Quality gate
- { "method": "Final-answer fixture semantic/normalized OCR plus completion/repetition/speed gates frozen in research/variant-onboarding/qwen3-vl-family/PLAN.md", "fixtures": [ "dog", "ocr", "circles", "arithmetic", "instruction" ] }
- Tested
- 2026-11-06T16:23:55.435304+00:00
- Environment
- { "os": "macos", "arch": "aarch64", "hardware": "Apple M3 Max", "osVersion": "27.2", "python": "App-owned CPython3.13.18 private model probe venv" }
- Runtime
- transformers 5.22.1
- Artifact revision
- 33e0ad94c327808ebc7d3bbc97e78e600f1eeecd
- Fixture
{ "file": "model-samples/desktop/qwen3-vl-family/Qwen3-VL-2B-Thinking-dog-input.json", "url": "https://assets.localai.tools/samples/v1/files/7b3ce826dfe3c11238f8ed74eb4c0376337fae452991b11c6e9c5b322ca981f0/Qwen3-VL-2B-Thinking-dog-input.json", "size": 609, "sha1": "2e345d8ba592df6216208fae8065b18c3a4f6e41", "sha256": "7b3ce826dfe3c11238f8ed74eb4c0376337fae452991b11c6e9c5b322ca981f0" }- Rerun when
- model revision/checksum; pinned runtime/processor/attention/backend/driver; fixture checksum/question/expected answers; visual pixel and token budgets; quality gates
Per-case quality & failure details
[
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "dog",
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 120,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3022.400458001357,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 8840.984792001109,
"inferenceMs": 8828.372374999162,
"tokensPerSecond": 5.097202302820204,
"inputTokens": 828,
"outputTokens": 45,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"requestMs": 8847.723875001975,
"totalMs": 11870.124333003332,
"artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
"precision": "fp32",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.11310957133471727,
"grade": "good",
"residentMemoryBytes": 9283813376,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "LOCAL AI 2026",
"expected": [
"localai2026"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 120,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3022.400458001357,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 12044.997207998676,
"inferenceMs": 12037.956375002977,
"tokensPerSecond": 6.313363965815269,
"inputTokens": 793,
"outputTokens": 76,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 768,
"inputHeight": 256,
"visionGrid": [
[
1,
32,
96
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"requestMs": 12046.859917001711,
"totalMs": 15069.260375003068,
"artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
"precision": "fp32",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.08302202007451971,
"grade": "moderate",
"residentMemoryBytes": 9281355776,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "3",
"expected": [
"3",
"three"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 120,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3022.400458001357,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 11912.995750000846,
"inferenceMs": 11905.521249998856,
"tokensPerSecond": 6.467587464934171,
"inputTokens": 826,
"outputTokens": 77,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 640,
"inputHeight": 320,
"visionGrid": [
[
1,
40,
80
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"requestMs": 11914.513666000857,
"totalMs": 14936.914124002215,
"artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
"precision": "fp32",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.08394194214330422,
"grade": "moderate",
"residentMemoryBytes": 9314451456,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "17 + 25 = 42\n\nAnswer: 42",
"expected": [
"42"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 120,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3022.400458001357,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 18659.51454199967,
"inferenceMs": 18658.54599999875,
"tokensPerSecond": 10.826138328249883,
"inputTokens": 24,
"outputTokens": 202,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": null,
"inputHeight": null,
"visionGrid": null,
"inputModality": "text",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"requestMs": 18660.8586250004,
"totalMs": 21683.259083001758,
"artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
"precision": "fp32",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.053591962306905425,
"grade": "moderate",
"residentMemoryBytes": 9392455680,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "READY",
"expected": [
"ready"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 120,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3022.400458001357,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 13476.740750000317,
"inferenceMs": 13475.579833000666,
"tokensPerSecond": 11.279663055964658,
"inputTokens": 17,
"outputTokens": 152,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": null,
"inputHeight": null,
"visionGrid": null,
"inputModality": "text",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"requestMs": 13477.901749996818,
"totalMs": 16500.302207998175,
"artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
"precision": "fp32",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.0742019171066993,
"grade": "moderate",
"residentMemoryBytes": 9393029120,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "dog",
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 120,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3022.400458001357,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 9667.7203750005,
"inferenceMs": 9655.712499999936,
"tokensPerSecond": 4.660453591591537,
"inputTokens": 828,
"outputTokens": 45,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"requestMs": 9674.303040999803,
"totalMs": 12696.70349900116,
"artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
"precision": "fp32",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.10343700078312912,
"grade": "good",
"residentMemoryBytes": 9398976512,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "dog",
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 120,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5031.187874999887,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 9716.61687500091,
"inferenceMs": 9703.423999999359,
"tokensPerSecond": 4.637538254538086,
"inputTokens": 828,
"outputTokens": 45,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"requestMs": 9723.536583001987,
"totalMs": 14754.724458001874,
"artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
"precision": "fp32",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.10291647935330438,
"grade": "good",
"residentMemoryBytes": 9282633728,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "LOCAL AI 2026",
"expected": [
"localai2026"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 120,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5031.187874999887,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 10780.18399999928,
"inferenceMs": 10773.396625001624,
"tokensPerSecond": 7.054414001952568,
"inputTokens": 793,
"outputTokens": 76,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 768,
"inputHeight": 256,
"visionGrid": [
[
1,
32,
96
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"requestMs": 10781.960750002327,
"totalMs": 15813.148625002214,
"artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
"precision": "fp32",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.0927627951433915,
"grade": "moderate",
"residentMemoryBytes": 9280864256,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "3",
"expected": [
"3",
"three"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 120,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5031.187874999887,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 11231.309457998577,
"inferenceMs": 11224.980125000002,
"tokensPerSecond": 6.859700341785682,
"inputTokens": 826,
"outputTokens": 77,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 640,
"inputHeight": 320,
"visionGrid": [
[
1,
40,
80
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"requestMs": 11232.843707999564,
"totalMs": 16264.03158299945,
"artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
"precision": "fp32",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.08903681300383297,
"grade": "moderate",
"residentMemoryBytes": 9313845248,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "17 + 25 = 42\n\nAnswer: 42",
"expected": [
"42"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 120,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5031.187874999887,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 21395.42891599922,
"inferenceMs": 21394.57579199734,
"tokensPerSecond": 9.441645488271764,
"inputTokens": 24,
"outputTokens": 202,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": null,
"inputHeight": null,
"visionGrid": null,
"inputModality": "text",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"requestMs": 21396.755958998256,
"totalMs": 26427.943833998143,
"artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
"precision": "fp32",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.04673895549961203,
"grade": "moderate",
"residentMemoryBytes": 9404366848,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "READY",
"expected": [
"ready"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 120,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5031.187874999887,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 16311.731042002066,
"inferenceMs": 16310.550500002137,
"tokensPerSecond": 9.319121387103403,
"inputTokens": 17,
"outputTokens": 152,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": null,
"inputHeight": null,
"visionGrid": null,
"inputModality": "text",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"requestMs": 16313.402583000425,
"totalMs": 21344.59045800031,
"artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
"precision": "fp32",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.061305571887192066,
"grade": "moderate",
"residentMemoryBytes": 9372532736,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "dog",
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 120,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5031.187874999887,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 13536.24729099829,
"inferenceMs": 13520.908292000968,
"tokensPerSecond": 3.3281787752840692,
"inputTokens": 828,
"outputTokens": 45,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"requestMs": 13544.194375001098,
"totalMs": 18575.382250000985,
"artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
"precision": "fp32",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.07387571891250892,
"grade": "moderate",
"residentMemoryBytes": 9059581952,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "dog",
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 120,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 8340.832292000414,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 12136.681249998219,
"inferenceMs": 12123.492708997219,
"tokensPerSecond": 3.7118016301196857,
"inputTokens": 828,
"outputTokens": 45,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"requestMs": 12143.483416999516,
"totalMs": 20484.31570899993,
"artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
"precision": "fp32",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.08239484743822754,
"grade": "moderate",
"residentMemoryBytes": 9071034368,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "LOCAL AI 2026",
"expected": [
"localai2026"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 120,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 8340.832292000414,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 10824.734625002748,
"inferenceMs": 10817.748624998785,
"tokensPerSecond": 7.025491406259086,
"inputTokens": 793,
"outputTokens": 76,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 768,
"inputHeight": 256,
"visionGrid": [
[
1,
32,
96
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"requestMs": 10826.52620799854,
"totalMs": 19167.358499998954,
"artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
"precision": "fp32",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.092381017608526,
"grade": "moderate",
"residentMemoryBytes": 9069838336,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "3",
"expected": [
"3",
"three"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 120,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 8340.832292000414,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 10994.872791998205,
"inferenceMs": 10988.445499999216,
"tokensPerSecond": 7.007360595273052,
"inputTokens": 826,
"outputTokens": 77,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 640,
"inputHeight": 320,
"visionGrid": [
[
1,
40,
80
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"requestMs": 10996.290833001694,
"totalMs": 19337.12312500211,
"artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
"precision": "fp32",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.09095148428891102,
"grade": "moderate",
"residentMemoryBytes": 9102917632,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "17 + 25 = 42\n\nAnswer: 42",
"expected": [
"42"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 120,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 8340.832292000414,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 19973.961916999542,
"inferenceMs": 19972.949457998766,
"tokensPerSecond": 10.113679024962588,
"inputTokens": 24,
"outputTokens": 202,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": null,
"inputHeight": null,
"visionGrid": null,
"inputModality": "text",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"requestMs": 19975.131375002093,
"totalMs": 28315.963667002507,
"artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
"precision": "fp32",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.050065180065699176,
"grade": "moderate",
"residentMemoryBytes": 9198403584,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "READY",
"expected": [
"ready"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 120,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 8340.832292000414,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 13093.458666997321,
"inferenceMs": 13092.54612499717,
"tokensPerSecond": 11.609659309108055,
"inputTokens": 17,
"outputTokens": 152,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": null,
"inputHeight": null,
"visionGrid": null,
"inputModality": "text",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"requestMs": 13095.176916998753,
"totalMs": 21436.009208999167,
"artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
"precision": "fp32",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.0763740143405002,
"grade": "moderate",
"residentMemoryBytes": 9200435200,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "dog",
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 120,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 8340.832292000414,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 7871.886917000666,
"inferenceMs": 7854.881041999761,
"tokensPerSecond": 5.7289218970200375,
"inputTokens": 828,
"outputTokens": 45,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"requestMs": 7881.953541000257,
"totalMs": 16222.785833000671,
"artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
"precision": "fp32",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.12703434520131782,
"grade": "good",
"residentMemoryBytes": 9206398976,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "dog",
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2969.3782499998633,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 625
},
"parameterDtype": "torch.bfloat16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 3642.0052079993184,
"inferenceMs": 3624.2387919992325,
"tokensPerSecond": 12.416400403676693,
"inputTokens": 828,
"outputTokens": 45,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 8272379904,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 3649.2836249999527,
"totalMs": 6618.661874999816,
"artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
"precision": "bf16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.27457401702874973,
"grade": "good",
"residentMemoryBytes": 657293312,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "LOCAL AI 2026",
"expected": [
"localai2026"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2969.3782499998633,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 625
},
"parameterDtype": "torch.bfloat16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 4546.717124998395,
"inferenceMs": 4536.961292000342,
"tokensPerSecond": 14.988005324158026,
"inputTokens": 793,
"outputTokens": 68,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 768,
"inputHeight": 256,
"visionGrid": [
[
1,
32,
96
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 9593749504,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 4549.028875000658,
"totalMs": 7518.407125000522,
"artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
"precision": "bf16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.2199389081194166,
"grade": "good",
"residentMemoryBytes": 697450496,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "3",
"expected": [
"3",
"three"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2969.3782499998633,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 625
},
"parameterDtype": "torch.bfloat16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 4064.170166999247,
"inferenceMs": 4051.7539579996082,
"tokensPerSecond": 12.587141403121924,
"inputTokens": 826,
"outputTokens": 51,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 640,
"inputHeight": 320,
"visionGrid": [
[
1,
40,
80
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 11026055168,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 4066.2290830005077,
"totalMs": 7035.607333000371,
"artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
"precision": "bf16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.24605268945673683,
"grade": "good",
"residentMemoryBytes": 734052352,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "17 + 25 = 42\n\n42",
"expected": [
"42"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2969.3782499998633,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 625
},
"parameterDtype": "torch.bfloat16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 5424.383916997613,
"inferenceMs": 5421.142041999701,
"tokensPerSecond": 44.640040442609994,
"inputTokens": 24,
"outputTokens": 242,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": null,
"inputHeight": null,
"visionGrid": null,
"inputModality": "text",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 10459955200,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 5426.033374998951,
"totalMs": 8395.411624998815,
"artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
"precision": "bf16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.18435273301110633,
"grade": "good",
"residentMemoryBytes": 763641856,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "READY",
"expected": [
"ready"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2969.3782499998633,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 625
},
"parameterDtype": "torch.bfloat16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 3103.133249998791,
"inferenceMs": 3100.471124998876,
"tokensPerSecond": 49.02480747988102,
"inputTokens": 17,
"outputTokens": 152,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": null,
"inputHeight": null,
"visionGrid": null,
"inputModality": "text",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 10460053504,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 3104.879874998005,
"totalMs": 6074.258124997868,
"artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
"precision": "bf16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.32225493378358455,
"grade": "good",
"residentMemoryBytes": 786235392,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "dog",
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2969.3782499998633,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 625
},
"parameterDtype": "torch.bfloat16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 3524.995334002597,
"inferenceMs": 3503.0560000013793,
"tokensPerSecond": 12.845926528146363,
"inputTokens": 828,
"outputTokens": 45,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 11210833920,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 3534.1916249999485,
"totalMs": 6503.569874999812,
"artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
"precision": "bf16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.28368831877700956,
"grade": "good",
"residentMemoryBytes": 789757952,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "dog",
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2681.656999997358,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 625
},
"parameterDtype": "torch.bfloat16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 3286.652040998888,
"inferenceMs": 3270.9619589986687,
"tokensPerSecond": 13.757420772259833,
"inputTokens": 828,
"outputTokens": 45,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 8272379904,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 3293.6740000004647,
"totalMs": 5975.330999997823,
"artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
"precision": "bf16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.30426098884994146,
"grade": "good",
"residentMemoryBytes": 657752064,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "LOCAL AI 2026",
"expected": [
"localai2026"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2681.656999997358,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 625
},
"parameterDtype": "torch.bfloat16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 3687.2162079998816,
"inferenceMs": 3676.653916001669,
"tokensPerSecond": 18.49507774012884,
"inputTokens": 793,
"outputTokens": 68,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 768,
"inputHeight": 256,
"visionGrid": [
[
1,
32,
96
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 9593749504,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 3689.3571250002424,
"totalMs": 6371.0141249976,
"artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
"precision": "bf16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.27120731293987416,
"grade": "good",
"residentMemoryBytes": 700317696,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "3",
"expected": [
"3",
"three"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2681.656999997358,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 625
},
"parameterDtype": "torch.bfloat16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 3502.547500000219,
"inferenceMs": 3492.5216670017107,
"tokensPerSecond": 14.602629521775567,
"inputTokens": 826,
"outputTokens": 51,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 640,
"inputHeight": 320,
"visionGrid": [
[
1,
40,
80
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 11026055168,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 3504.728667001473,
"totalMs": 6186.385666998831,
"artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
"precision": "bf16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.28550647778508,
"grade": "good",
"residentMemoryBytes": 737050624,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "17 + 25 = 42\n\n42",
"expected": [
"42"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2681.656999997358,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 625
},
"parameterDtype": "torch.bfloat16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 5662.737250000646,
"inferenceMs": 5660.2739580011985,
"tokensPerSecond": 42.75411434068767,
"inputTokens": 24,
"outputTokens": 242,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": null,
"inputHeight": null,
"visionGrid": null,
"inputModality": "text",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 10459955200,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 5664.1843329998665,
"totalMs": 8345.841332997225,
"artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
"precision": "bf16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.17659304252548289,
"grade": "good",
"residentMemoryBytes": 766574592,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "READY",
"expected": [
"ready"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2681.656999997358,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 625
},
"parameterDtype": "torch.bfloat16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 3278.5739170030865,
"inferenceMs": 3276.679750000767,
"tokensPerSecond": 46.388421083862234,
"inputTokens": 17,
"outputTokens": 152,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": null,
"inputHeight": null,
"visionGrid": null,
"inputModality": "text",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 10460053504,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 3279.8882909992244,
"totalMs": 5961.5452909965825,
"artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
"precision": "bf16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.30501066174347247,
"grade": "good",
"residentMemoryBytes": 789151744,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "dog",
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2681.656999997358,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 625
},
"parameterDtype": "torch.bfloat16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 3228.9094160005334,
"inferenceMs": 3211.6581250011222,
"tokensPerSecond": 14.011453974412134,
"inputTokens": 828,
"outputTokens": 45,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 11210833920,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 3236.58420799984,
"totalMs": 5918.241207997198,
"artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
"precision": "bf16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.3097020916860044,
"grade": "good",
"residentMemoryBytes": 793100288,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "dog",
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2689.8290419994737,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 625
},
"parameterDtype": "torch.bfloat16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 3229.67858300035,
"inferenceMs": 3208.9673330010555,
"tokensPerSecond": 14.023202896838338,
"inputTokens": 828,
"outputTokens": 45,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 8272379904,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 3237.862624999252,
"totalMs": 5927.691666998726,
"artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
"precision": "bf16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.3096283343065695,
"grade": "good",
"residentMemoryBytes": 658423808,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "LOCAL AI 2026",
"expected": [
"localai2026"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2689.8290419994737,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 625
},
"parameterDtype": "torch.bfloat16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 3695.6460000001243,
"inferenceMs": 3687.5166249992617,
"tokensPerSecond": 18.440594827152438,
"inputTokens": 793,
"outputTokens": 68,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 768,
"inputHeight": 256,
"visionGrid": [
[
1,
32,
96
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 9593749504,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 3697.675374998653,
"totalMs": 6387.5044169981265,
"artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
"precision": "bf16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.2705886873363862,
"grade": "good",
"residentMemoryBytes": 697745408,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "3",
"expected": [
"3",
"three"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2689.8290419994737,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 625
},
"parameterDtype": "torch.bfloat16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 3621.6417079995153,
"inferenceMs": 3611.137249998137,
"tokensPerSecond": 14.122974694475074,
"inputTokens": 826,
"outputTokens": 51,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 640,
"inputHeight": 320,
"visionGrid": [
[
1,
40,
80
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 11026055168,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 3624.075374998938,
"totalMs": 6313.904416998412,
"artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
"precision": "bf16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.2761178715694573,
"grade": "good",
"residentMemoryBytes": 734314496,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "17 + 25 = 42\n\n42",
"expected": [
"42"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2689.8290419994737,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 625
},
"parameterDtype": "torch.bfloat16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 5656.247999999323,
"inferenceMs": 5654.064667000057,
"tokensPerSecond": 42.801066887761785,
"inputTokens": 24,
"outputTokens": 242,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": null,
"inputHeight": null,
"visionGrid": null,
"inputModality": "text",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 10459955200,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 5657.76150000238,
"totalMs": 8347.590542001853,
"artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
"precision": "bf16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.1767956426238948,
"grade": "good",
"residentMemoryBytes": 762937344,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "READY",
"expected": [
"ready"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2689.8290419994737,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 625
},
"parameterDtype": "torch.bfloat16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 3482.4695410025015,
"inferenceMs": 3478.8234170009673,
"tokensPerSecond": 43.69293343754612,
"inputTokens": 17,
"outputTokens": 152,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": null,
"inputHeight": null,
"visionGrid": null,
"inputModality": "text",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 10460053504,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 3484.2842500002007,
"totalMs": 6174.113291999674,
"artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
"precision": "bf16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.28715254741671886,
"grade": "good",
"residentMemoryBytes": 785088512,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "dog",
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2689.8290419994737,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 625
},
"parameterDtype": "torch.bfloat16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 3656.1782080025296,
"inferenceMs": 3636.0153749992605,
"tokensPerSecond": 12.376185290473133,
"inputTokens": 828,
"outputTokens": 45,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 11210833920,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 3664.811208000174,
"totalMs": 6354.6402499996475,
"artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
"precision": "bf16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.2735096439804906,
"grade": "good",
"residentMemoryBytes": 789331968,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "dog",
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 120,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2296.0705000004964,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 12416.36437499983,
"inferenceMs": 12395.884958998067,
"tokensPerSecond": 3.63023698177635,
"inputTokens": 828,
"outputTokens": 45,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"requestMs": 12422.926582999935,
"totalMs": 14718.997083000431,
"artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
"precision": "fp32",
"cpuThreads": 1,
"fallback": false,
"requestsPerSecond": 0.08053887352190513,
"grade": "moderate",
"residentMemoryBytes": 9279471616,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "LOCAL AI 2026",
"expected": [
"localai2026"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 120,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2296.0705000004964,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 14806.454499997926,
"inferenceMs": 14793.110499998875,
"tokensPerSecond": 5.137526688522051,
"inputTokens": 793,
"outputTokens": 76,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 768,
"inputHeight": 256,
"visionGrid": [
[
1,
32,
96
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"requestMs": 14808.299083000747,
"totalMs": 17104.369583001244,
"artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
"precision": "fp32",
"cpuThreads": 1,
"fallback": false,
"requestsPerSecond": 0.06753811319246887,
"grade": "moderate",
"residentMemoryBytes": 9277063168,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "3",
"expected": [
"3",
"three"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 120,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2296.0705000004964,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 14929.877874998056,
"inferenceMs": 14916.288415999588,
"tokensPerSecond": 5.162142072649108,
"inputTokens": 826,
"outputTokens": 77,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 640,
"inputHeight": 320,
"visionGrid": [
[
1,
40,
80
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"requestMs": 14931.277916999534,
"totalMs": 17227.34841700003,
"artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
"precision": "fp32",
"cpuThreads": 1,
"fallback": false,
"requestsPerSecond": 0.06697978432058208,
"grade": "moderate",
"residentMemoryBytes": 9309601792,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "17 + 25 = 42\n\nAnswer: 42",
"expected": [
"42"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 120,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2296.0705000004964,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 16267.460209001001,
"inferenceMs": 16266.589874998317,
"tokensPerSecond": 12.418091410202281,
"inputTokens": 24,
"outputTokens": 202,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": null,
"inputHeight": null,
"visionGrid": null,
"inputModality": "text",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"requestMs": 16268.699415999436,
"totalMs": 18564.769915999932,
"artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
"precision": "fp32",
"cpuThreads": 1,
"fallback": false,
"requestsPerSecond": 0.06147241100652496,
"grade": "moderate",
"residentMemoryBytes": 9429401600,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "READY",
"expected": [
"ready"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 120,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2296.0705000004964,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 12811.984916002984,
"inferenceMs": 12810.033833000489,
"tokensPerSecond": 11.865698559548386,
"inputTokens": 17,
"outputTokens": 152,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": null,
"inputHeight": null,
"visionGrid": null,
"inputModality": "text",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"requestMs": 12813.1942910004,
"totalMs": 15109.264791000896,
"artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
"precision": "fp32",
"cpuThreads": 1,
"fallback": false,
"requestsPerSecond": 0.07805191830587752,
"grade": "moderate",
"residentMemoryBytes": 9432449024,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "dog",
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 120,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 2296.0705000004964,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 625
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 12295.713666000665,
"inferenceMs": 12275.892167002894,
"tokensPerSecond": 3.6657213494395298,
"inputTokens": 828,
"outputTokens": 45,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"requestMs": 12302.271958000347,
"totalMs": 14598.342458000843,
"artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
"precision": "fp32",
"cpuThreads": 1,
"fallback": false,
"requestsPerSecond": 0.08132915479035081,
"grade": "moderate",
"residentMemoryBytes": 9438429184,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
}
]Qwen3-VL 4B Instruct — Text, Image
42 configurations testedQuality criteria
{ "method": "Final-answer fixture semantic/normalized OCR plus completion/repetition/speed gates frozen in research/variant-onboarding/qwen3-vl-family/PLAN.md", "fixtures": [ "dog", "ocr", "circles", "arithmetic", "instruction" ] }
| Precision | Device | Result | Platform | Inference / load | |
|---|---|---|---|---|---|
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 252 ms / 6632 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 338 ms / 3696 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 345 ms / 6632 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 425 ms / 4015 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 441 ms / 4015 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 634 ms / 3696 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 3549 ms / 3696 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 3735 ms / 3696 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 3827 ms / 6632 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 3979 ms / 3696 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 3981 ms / 3696 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 4002 ms / 6632 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 4167 ms / 6632 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 4471 ms / 4015 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 4525 ms / 6632 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 4634 ms / 4015 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 4896 ms / 4015 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 4954 ms / 4015 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 498 ms / 10236 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 583 ms / 7120 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 667 ms / 10236 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 708 ms / 7120 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 710 ms / 3741 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 843 ms / 3741 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 959 ms / 7942 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1059 ms / 7942 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 6161 ms / 3741 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 6441 ms / 7942 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 6615 ms / 7120 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 6699 ms / 3741 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 6810 ms / 7120 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 7093 ms / 7120 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 7137 ms / 3741 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 7263 ms / 7942 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 7433 ms / 7942 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 7436 ms / 7120 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 8309 ms / 3741 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 10484 ms / 10236 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 10581 ms / 10236 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 11389 ms / 10236 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 11602 ms / 10236 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 16897 ms / 7942 ms |
Test setup & sample details
- Quality gate
- { "method": "Final-answer fixture semantic/normalized OCR plus completion/repetition/speed gates frozen in research/variant-onboarding/qwen3-vl-family/PLAN.md", "fixtures": [ "dog", "ocr", "circles", "arithmetic", "instruction" ] }
- Tested
- 2026-11-06T16:16:04.465768+00:00
- Environment
- { "os": "macos", "arch": "aarch64", "hardware": "Apple M3 Max", "osVersion": "27.2", "python": "App-owned CPython3.13.18 private model probe venv" }
- Runtime
- transformers 5.22.1
- Artifact revision
- ebb281ec70b05090aa6165b016eac8ec08e71b17
- Fixture
{ "file": "model-samples/desktop/qwen3-vl-family/Qwen3-VL-4B-Instruct-dog-input.json", "url": "https://assets.localai.tools/samples/v1/files/7246f398c23127a1d90d48dd4a0d56a81dd2b7b8909e5a2efa1975f1a2ed6a87/Qwen3-VL-4B-Instruct-dog-input.json", "size": 607, "sha1": "409fa36ae63ce99dbfa9da362719622bab8d19f3", "sha256": "7246f398c23127a1d90d48dd4a0d56a81dd2b7b8909e5a2efa1975f1a2ed6a87" }- Rerun when
- model revision/checksum; pinned runtime/processor/attention/backend/driver; fixture checksum/question/expected answers; visual pixel and token budgets; quality gates
Per-case quality & failure details
[
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "dog",
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3741.1324579989014,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 713
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 6174.910917001398,
"inferenceMs": 6161.06495799977,
"tokensPerSecond": 0.324619203601014,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"requestMs": 6181.928042002255,
"totalMs": 9923.060500001156,
"artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
"precision": "fp32",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.1619456561303091,
"grade": "good",
"residentMemoryBytes": 18426576896,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "LOCAL AI 2026",
"expected": [
"localai2026"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3741.1324579989014,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 713
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 7145.66441700299,
"inferenceMs": 7137.178999997559,
"tokensPerSecond": 1.1208910411246147,
"inputTokens": 791,
"outputTokens": 8,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 768,
"inputHeight": 256,
"visionGrid": [
[
1,
32,
96
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"requestMs": 7147.677042001305,
"totalMs": 10888.809500000207,
"artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
"precision": "fp32",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.13994499904312838,
"grade": "good",
"residentMemoryBytes": 18429919232,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "3",
"expected": [
"3",
"three"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3741.1324579989014,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 713
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 6705.934417001117,
"inferenceMs": 6698.781499999313,
"tokensPerSecond": 0.2985617608217562,
"inputTokens": 824,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 640,
"inputHeight": 320,
"visionGrid": [
[
1,
40,
80
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"requestMs": 6707.429541998863,
"totalMs": 10448.561999997764,
"artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
"precision": "fp32",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.14912164924619087,
"grade": "good",
"residentMemoryBytes": 18432081920,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "42",
"expected": [
"42"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3741.1324579989014,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 713
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 844.1981669966481,
"inferenceMs": 843.3275000024878,
"tokensPerSecond": 3.557336859038926,
"inputTokens": 22,
"outputTokens": 3,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": null,
"inputHeight": null,
"visionGrid": null,
"inputModality": "text",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"requestMs": 845.9493330010446,
"totalMs": 4587.081790999946,
"artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
"precision": "fp32",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 1.1845559953744493,
"grade": "good",
"residentMemoryBytes": 18433540096,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "READY",
"expected": [
"ready"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3741.1324579989014,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 713
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 710.5662499998289,
"inferenceMs": 709.6702079979877,
"tokensPerSecond": 2.8182104553072502,
"inputTokens": 15,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": null,
"inputHeight": null,
"visionGrid": null,
"inputModality": "text",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"requestMs": 711.9691659972887,
"totalMs": 4453.10162399619,
"artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
"precision": "fp32",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 1.4073283103443779,
"grade": "good",
"residentMemoryBytes": 18435031040,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "dog",
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3741.1324579989014,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 713
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 8325.148125000851,
"inferenceMs": 8308.921625000949,
"tokensPerSecond": 0.24070512279019982,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"requestMs": 8332.22370799922,
"totalMs": 12073.35616599812,
"artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
"precision": "fp32",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.12011798288572767,
"grade": "good",
"residentMemoryBytes": 18435031040,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "dog",
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7941.704166998534,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 713
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 7277.82308300084,
"inferenceMs": 7263.054041999567,
"tokensPerSecond": 0.27536625618296884,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"requestMs": 7285.103083002468,
"totalMs": 15226.807250001002,
"artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
"precision": "fp32",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.13740372479454027,
"grade": "good",
"residentMemoryBytes": 18250121216,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "LOCAL AI 2026",
"expected": [
"localai2026"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7941.704166998534,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 713
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 16904.474708000635,
"inferenceMs": 16896.785790999274,
"tokensPerSecond": 0.4734628289045073,
"inputTokens": 791,
"outputTokens": 8,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 768,
"inputHeight": 256,
"visionGrid": [
[
1,
32,
96
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"requestMs": 16906.74562499771,
"totalMs": 24848.449791996245,
"artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
"precision": "fp32",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.05915593458380076,
"grade": "moderate",
"residentMemoryBytes": 18222071808,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "3",
"expected": [
"3",
"three"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7941.704166998534,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 713
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 6450.158916999499,
"inferenceMs": 6440.879707999557,
"tokensPerSecond": 0.3105165894522149,
"inputTokens": 824,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 640,
"inputHeight": 320,
"visionGrid": [
[
1,
40,
80
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"requestMs": 6451.830875001178,
"totalMs": 14393.535041999712,
"artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
"precision": "fp32",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.15503493989341624,
"grade": "good",
"residentMemoryBytes": 18249777152,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "42",
"expected": [
"42"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7941.704166998534,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 713
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 1059.63679100023,
"inferenceMs": 1058.7157500012836,
"tokensPerSecond": 2.833621772412815,
"inputTokens": 22,
"outputTokens": 3,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": null,
"inputHeight": null,
"visionGrid": null,
"inputModality": "text",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"requestMs": 1061.3430419980432,
"totalMs": 9003.047208996577,
"artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
"precision": "fp32",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.9437195919330654,
"grade": "good",
"residentMemoryBytes": 18257313792,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "READY",
"expected": [
"ready"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7941.704166998534,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 713
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 960.1322080015962,
"inferenceMs": 959.1316250007367,
"tokensPerSecond": 2.0852195338658173,
"inputTokens": 15,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": null,
"inputHeight": null,
"visionGrid": null,
"inputModality": "text",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"requestMs": 962.295625002298,
"totalMs": 8903.999792000832,
"artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
"precision": "fp32",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 1.041523231557229,
"grade": "good",
"residentMemoryBytes": 18258411520,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "dog",
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7941.704166998534,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 713
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 7455.856999997195,
"inferenceMs": 7432.98833299923,
"tokensPerSecond": 0.2690707842390753,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"requestMs": 7465.716499998962,
"totalMs": 15407.420666997496,
"artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
"precision": "fp32",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.1341227440387304,
"grade": "good",
"residentMemoryBytes": 18259066880,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "dog",
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7119.960417003313,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 713
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 7106.457958998362,
"inferenceMs": 7093.278000000282,
"tokensPerSecond": 0.28195708669530795,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"requestMs": 7114.342583001417,
"totalMs": 14234.30300000473,
"artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
"precision": "fp32",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.1407170781519613,
"grade": "good",
"residentMemoryBytes": 18449825792,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "LOCAL AI 2026",
"expected": [
"localai2026"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7119.960417003313,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 713
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 7445.575707999524,
"inferenceMs": 7435.673832998873,
"tokensPerSecond": 1.0758944219011728,
"inputTokens": 791,
"outputTokens": 8,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 768,
"inputHeight": 256,
"visionGrid": [
[
1,
32,
96
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"requestMs": 7447.62858300237,
"totalMs": 14567.589000005682,
"artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
"precision": "fp32",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.1343079486688451,
"grade": "good",
"residentMemoryBytes": 18453168128,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "3",
"expected": [
"3",
"three"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7119.960417003313,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 713
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 6816.494291997515,
"inferenceMs": 6809.672999999748,
"tokensPerSecond": 0.29369985900939355,
"inputTokens": 824,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 640,
"inputHeight": 320,
"visionGrid": [
[
1,
40,
80
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"requestMs": 6818.0038329992385,
"totalMs": 13937.964250002551,
"artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
"precision": "fp32",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.1467029762166732,
"grade": "good",
"residentMemoryBytes": 18455298048,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "42",
"expected": [
"42"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7119.960417003313,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 713
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 708.3546249996289,
"inferenceMs": 707.6382500017644,
"tokensPerSecond": 4.239454269172872,
"inputTokens": 22,
"outputTokens": 3,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": null,
"inputHeight": null,
"visionGrid": null,
"inputModality": "text",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"requestMs": 709.6257079974748,
"totalMs": 7829.586125000787,
"artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
"precision": "fp32",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 1.4117222711724708,
"grade": "good",
"residentMemoryBytes": 18457362432,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "READY",
"expected": [
"ready"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7119.960417003313,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 713
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 584.1366250024294,
"inferenceMs": 583.3942919998663,
"tokensPerSecond": 3.4282131783360996,
"inputTokens": 15,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": null,
"inputHeight": null,
"visionGrid": null,
"inputModality": "text",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"requestMs": 585.4325830005109,
"totalMs": 7705.3930000038235,
"artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
"precision": "fp32",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 1.7119282667746456,
"grade": "good",
"residentMemoryBytes": 18457493504,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "dog",
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7119.960417003313,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 713
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 6628.192583000782,
"inferenceMs": 6615.071541000361,
"tokensPerSecond": 0.3023398896903768,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"requestMs": 6635.057333998702,
"totalMs": 13755.017751002015,
"artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
"precision": "fp32",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.15087069174252477,
"grade": "good",
"residentMemoryBytes": 18453463040,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "dog",
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3696.491624999908,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 713
},
"parameterDtype": "torch.bfloat16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 3573.611374999018,
"inferenceMs": 3548.659707998013,
"tokensPerSecond": 0.563593064585025,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 12675694592,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 3584.1817920008907,
"totalMs": 7280.6734170007985,
"artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
"precision": "bf16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.27982897272937934,
"grade": "good",
"residentMemoryBytes": 660766720,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "LOCAL AI 2026",
"expected": [
"localai2026"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3696.491624999908,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 713
},
"parameterDtype": "torch.bfloat16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 3993.279040998459,
"inferenceMs": 3980.546167000284,
"tokensPerSecond": 2.00977445414953,
"inputTokens": 791,
"outputTokens": 8,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 768,
"inputHeight": 256,
"visionGrid": [
[
1,
32,
96
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 14099972096,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 3998.594500000763,
"totalMs": 7695.086125000671,
"artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
"precision": "bf16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.25042076692691256,
"grade": "good",
"residentMemoryBytes": 696598528,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "3",
"expected": [
"3",
"three"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3696.491624999908,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 713
},
"parameterDtype": "torch.bfloat16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 4002.1517909990507,
"inferenceMs": 3979.4315829967672,
"tokensPerSecond": 0.5025843410766399,
"inputTokens": 824,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 640,
"inputHeight": 320,
"visionGrid": [
[
1,
40,
80
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15608709120,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 4007.612541998242,
"totalMs": 7704.10416699815,
"artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
"precision": "bf16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.24986558537060674,
"grade": "good",
"residentMemoryBytes": 729497600,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "42",
"expected": [
"42"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3696.491624999908,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 713
},
"parameterDtype": "torch.bfloat16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 342.77300000030664,
"inferenceMs": 338.4717919980176,
"tokensPerSecond": 8.863367851988006,
"inputTokens": 22,
"outputTokens": 3,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": null,
"inputHeight": null,
"visionGrid": null,
"inputModality": "text",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15625666560,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 345.7546249992447,
"totalMs": 4042.2462499991525,
"artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
"precision": "bf16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 2.9173826409871997,
"grade": "good",
"residentMemoryBytes": 742850560,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "READY",
"expected": [
"ready"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3696.491624999908,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 713
},
"parameterDtype": "torch.bfloat16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 636.409624999942,
"inferenceMs": 633.8471249982831,
"tokensPerSecond": 3.1553349713551473,
"inputTokens": 15,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": null,
"inputHeight": null,
"visionGrid": null,
"inputModality": "text",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15625781248,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 638.0790830007754,
"totalMs": 4334.570708000683,
"artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
"precision": "bf16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 1.571315015859622,
"grade": "good",
"residentMemoryBytes": 758710272,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "dog",
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 3696.491624999908,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 713
},
"parameterDtype": "torch.bfloat16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 3750.7840000034776,
"inferenceMs": 3734.5751249995374,
"tokensPerSecond": 0.5355361541964557,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15625781248,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 3759.758709002199,
"totalMs": 7456.250334002107,
"artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
"precision": "bf16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.26661092720857105,
"grade": "good",
"residentMemoryBytes": 765018112,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "dog",
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4015.286958001525,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 713
},
"parameterDtype": "torch.bfloat16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 4499.136832997465,
"inferenceMs": 4471.310459000961,
"tokensPerSecond": 0.44729616034017605,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 12675694592,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 4513.216499999544,
"totalMs": 8528.503458001069,
"artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
"precision": "bf16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.22226485593099174,
"grade": "good",
"residentMemoryBytes": 664322048,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "LOCAL AI 2026",
"expected": [
"localai2026"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4015.286958001525,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 713
},
"parameterDtype": "torch.bfloat16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 4978.793833000964,
"inferenceMs": 4954.102709001745,
"tokensPerSecond": 1.6148232020833506,
"inputTokens": 791,
"outputTokens": 8,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 768,
"inputHeight": 256,
"visionGrid": [
[
1,
32,
96
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 14099972096,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 4985.715875001915,
"totalMs": 9001.00283300344,
"artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
"precision": "bf16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.2008518596154143,
"grade": "good",
"residentMemoryBytes": 699482112,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "3",
"expected": [
"3",
"three"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4015.286958001525,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 713
},
"parameterDtype": "torch.bfloat16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 4676.366415998928,
"inferenceMs": 4634.414374999324,
"tokensPerSecond": 0.43155398679693846,
"inputTokens": 824,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 640,
"inputHeight": 320,
"visionGrid": [
[
1,
40,
80
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15608709120,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 4679.809165998449,
"totalMs": 8695.096123999974,
"artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
"precision": "bf16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.213841241477308,
"grade": "good",
"residentMemoryBytes": 732839936,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "42",
"expected": [
"42"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4015.286958001525,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 713
},
"parameterDtype": "torch.bfloat16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 443.8317089989141,
"inferenceMs": 440.730332997191,
"tokensPerSecond": 6.8068834282371045,
"inputTokens": 22,
"outputTokens": 3,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": null,
"inputHeight": null,
"visionGrid": null,
"inputModality": "text",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15625666560,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 447.4767499996233,
"totalMs": 4462.763708001148,
"artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
"precision": "bf16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 2.2531062556471073,
"grade": "good",
"residentMemoryBytes": 746553344,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "READY",
"expected": [
"ready"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4015.286958001525,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 713
},
"parameterDtype": "torch.bfloat16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 429.6830840030452,
"inferenceMs": 424.90379200171446,
"tokensPerSecond": 4.706947873018582,
"inputTokens": 15,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": null,
"inputHeight": null,
"visionGrid": null,
"inputModality": "text",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15625781248,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 433.0050409989781,
"totalMs": 4448.291999000503,
"artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
"precision": "bf16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 2.327296645433947,
"grade": "good",
"residentMemoryBytes": 759431168,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "dog",
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4015.286958001525,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 713
},
"parameterDtype": "torch.bfloat16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 4939.111083000171,
"inferenceMs": 4895.961666999938,
"tokensPerSecond": 0.4084999303569967,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15625781248,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 4960.963375000574,
"totalMs": 8976.250333002099,
"artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
"precision": "bf16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.20246558200358772,
"grade": "good",
"residentMemoryBytes": 761184256,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "dog",
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 6632.466375001968,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 713
},
"parameterDtype": "torch.bfloat16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 4214.732832999289,
"inferenceMs": 4167.440958000952,
"tokensPerSecond": 0.479910818210935,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 12675694592,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 4236.636833000375,
"totalMs": 10869.103208002343,
"artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
"precision": "bf16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.23726296294998606,
"grade": "good",
"residentMemoryBytes": 657571840,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "LOCAL AI 2026",
"expected": [
"localai2026"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 6632.466375001968,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 713
},
"parameterDtype": "torch.bfloat16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 4560.405457999877,
"inferenceMs": 4524.649584000144,
"tokensPerSecond": 1.7680927222053235,
"inputTokens": 791,
"outputTokens": 8,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 768,
"inputHeight": 256,
"visionGrid": [
[
1,
32,
96
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 14099972096,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 4572.066333999828,
"totalMs": 11204.532709001796,
"artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
"precision": "bf16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.21927874817485732,
"grade": "good",
"residentMemoryBytes": 693256192,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "3",
"expected": [
"3",
"three"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 6632.466375001968,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 713
},
"parameterDtype": "torch.bfloat16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 3857.208166999044,
"inferenceMs": 3827.064249999239,
"tokensPerSecond": 0.5225937871308007,
"inputTokens": 824,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 640,
"inputHeight": 320,
"visionGrid": [
[
1,
40,
80
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15608709120,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 3864.1452499978186,
"totalMs": 10496.611624999787,
"artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
"precision": "bf16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.25925486950786286,
"grade": "good",
"residentMemoryBytes": 727564288,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "42",
"expected": [
"42"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 6632.466375001968,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 713
},
"parameterDtype": "torch.bfloat16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 349.36691600159975,
"inferenceMs": 344.52020800017635,
"tokensPerSecond": 8.707762071241012,
"inputTokens": 22,
"outputTokens": 3,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": null,
"inputHeight": null,
"visionGrid": null,
"inputModality": "text",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15625666560,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 351.9629999973404,
"totalMs": 6984.4293749993085,
"artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
"precision": "bf16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 2.862320254718741,
"grade": "good",
"residentMemoryBytes": 741294080,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "READY",
"expected": [
"ready"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 6632.466375001968,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 713
},
"parameterDtype": "torch.bfloat16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 255.4954999977781,
"inferenceMs": 251.7043330008164,
"tokensPerSecond": 7.94583063452274,
"inputTokens": 15,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": null,
"inputHeight": null,
"visionGrid": null,
"inputModality": "text",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15625781248,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 257.34545799787156,
"totalMs": 6889.81183299984,
"artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
"precision": "bf16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 3.9139632596609193,
"grade": "good",
"residentMemoryBytes": 754286592,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "dog",
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 6632.466375001968,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 713
},
"parameterDtype": "torch.bfloat16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 4021.5398330001335,
"inferenceMs": 4002.245250001579,
"tokensPerSecond": 0.499719501197287,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15625781248,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 4031.9942079986504,
"totalMs": 10664.460583000618,
"artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
"precision": "bf16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.2486609710524697,
"grade": "good",
"residentMemoryBytes": 755941376,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "dog",
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 10236.078249999991,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 713
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 10601.683583001432,
"inferenceMs": 10581.3890419995,
"tokensPerSecond": 0.18901110166743026,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"requestMs": 10608.202708997851,
"totalMs": 20844.280958997842,
"artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
"precision": "fp32",
"cpuThreads": 1,
"fallback": false,
"requestsPerSecond": 0.0943246411922144,
"grade": "moderate",
"residentMemoryBytes": 18443010048,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "LOCAL AI 2026",
"expected": [
"localai2026"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 10236.078249999991,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 713
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 11403.46958399823,
"inferenceMs": 11389.23666700066,
"tokensPerSecond": 0.7024175749354051,
"inputTokens": 791,
"outputTokens": 8,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 768,
"inputHeight": 256,
"visionGrid": [
[
1,
32,
96
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"requestMs": 11405.566833000194,
"totalMs": 21641.645083000185,
"artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
"precision": "fp32",
"cpuThreads": 1,
"fallback": false,
"requestsPerSecond": 0.08769260904621844,
"grade": "moderate",
"residentMemoryBytes": 18447187968,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "3",
"expected": [
"3",
"three"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 10236.078249999991,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 713
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 10498.403166999196,
"inferenceMs": 10484.120332999737,
"tokensPerSecond": 0.19076469331478535,
"inputTokens": 824,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 640,
"inputHeight": 320,
"visionGrid": [
[
1,
40,
80
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"requestMs": 10499.87237499954,
"totalMs": 20735.950624999532,
"artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
"precision": "fp32",
"cpuThreads": 1,
"fallback": false,
"requestsPerSecond": 0.09525258118714774,
"grade": "moderate",
"residentMemoryBytes": 18449285120,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "42",
"expected": [
"42"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 10236.078249999991,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 713
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 667.4823749999632,
"inferenceMs": 666.752290999284,
"tokensPerSecond": 4.499422109977004,
"inputTokens": 22,
"outputTokens": 3,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": null,
"inputHeight": null,
"visionGrid": null,
"inputModality": "text",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"requestMs": 668.6668750007811,
"totalMs": 10904.745125000773,
"artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
"precision": "fp32",
"cpuThreads": 1,
"fallback": false,
"requestsPerSecond": 1.4981668991635249,
"grade": "good",
"residentMemoryBytes": 18452905984,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "READY",
"expected": [
"ready"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 10236.078249999991,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 713
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 498.6082499999611,
"inferenceMs": 497.7752500017232,
"tokensPerSecond": 4.017877546127648,
"inputTokens": 15,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": null,
"inputHeight": null,
"visionGrid": null,
"inputModality": "text",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"requestMs": 499.81516700063366,
"totalMs": 10735.893417000625,
"artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
"precision": "fp32",
"cpuThreads": 1,
"fallback": false,
"requestsPerSecond": 2.0055825389974555,
"grade": "good",
"residentMemoryBytes": 18452905984,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "dog",
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 10236.078249999991,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 713
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 11621.961249999003,
"inferenceMs": 11602.114707999135,
"tokensPerSecond": 0.17238236738179205,
"inputTokens": 826,
"outputTokens": 2,
"maxNewTokens": 32,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"requestMs": 11628.605249999964,
"totalMs": 21864.683499999956,
"artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
"precision": "fp32",
"cpuThreads": 1,
"fallback": false,
"requestsPerSecond": 0.08604399709215049,
"grade": "moderate",
"residentMemoryBytes": 18448793600,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
}
]Qwen3-VL 4B Thinking — Text, Image
20 configurations testedQuality criteria
{ "method": "Final-answer fixture semantic/normalized OCR plus completion/repetition/speed gates frozen in research/variant-onboarding/qwen3-vl-family/PLAN.md", "fixtures": [ "dog", "ocr", "circles", "arithmetic", "instruction" ] }
| Precision | Device | Result | Platform | Inference / load | |
|---|---|---|---|---|---|
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 4973 ms / 19685 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 5377 ms / 19685 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 5434 ms / 15106 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 5916 ms / 21061 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 5952 ms / 21061 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 6380 ms / 15106 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 6524 ms / 19685 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 6880 ms / 15106 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 8094 ms / 15106 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 8216 ms / 15106 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 8834 ms / 21061 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 8960 ms / 19685 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 9482 ms / 19685 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 10499 ms / 21061 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 14166 ms / 21061 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 22979 ms / 19685 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 27645 ms / 15106 ms |
FP16 | mpsAccelerated | Passed | Moderate | macos · aarch64 | 32455 ms / 21061 ms |
FP32 | cpu | Failed | — | macos · aarch64 | 134016 ms / 5941 ms |
BF16 | mps | Failed | — | macos · aarch64 | 46679 ms / 10534 ms |
Test setup & sample details
- Quality gate
- { "method": "Final-answer fixture semantic/normalized OCR plus completion/repetition/speed gates frozen in research/variant-onboarding/qwen3-vl-family/PLAN.md", "fixtures": [ "dog", "ocr", "circles", "arithmetic", "instruction" ] }
- Tested
- 2026-11-15T05:56:45.583669+00:00
- Environment
- { "os": "macos", "arch": "aarch64", "hardware": "Apple M3 Max", "osVersion": "27.2", "python": "App-owned CPython3.13.18 private model probe venv" }
- Runtime
- transformers 5.22.1
- Artifact revision
- 1de27d8c51f12e819435303b9e84c4e25ba8401e
- Fixture
{ "file": "model-samples/desktop/qwen3-vl-family/Qwen3-VL-4B-Thinking-dog-input.json", "url": "https://assets.localai.tools/samples/v1/files/7b3ce826dfe3c11238f8ed74eb4c0376337fae452991b11c6e9c5b322ca981f0/Qwen3-VL-4B-Thinking-dog-input.json", "size": 609, "sha1": "2e345d8ba592df6216208fae8065b18c3a4f6e41", "sha256": "7b3ce826dfe3c11238f8ed74eb4c0376337fae452991b11c6e9c5b322ca981f0" }- Rerun when
- model revision/checksum; pinned runtime/processor/attention/backend/driver; fixture checksum/question/expected answers; visual pixel and token budgets; quality gates
Per-case quality & failure details
[
{
"precision": "fp32",
"device": "cpu",
"status": "fail",
"quality": {
"passed": false,
"acceptedAnswer": true,
"finalAnswer": "42",
"expected": [
"42"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": false,
"maximumRequestSeconds": 120,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5941.389582992997,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 713
},
"parameterDtype": "torch.float32",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 134017.24229200045,
"inferenceMs": 134016.3947090041,
"tokensPerSecond": 3.5518042477829708,
"inputTokens": 24,
"outputTokens": 476,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": null,
"inputHeight": null,
"visionGrid": null,
"inputModality": "text",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"requestMs": 134018.6680839979,
"totalMs": 139960.0576669909,
"artifactRevision": "1de27d8c51f12e819435303b9e84c4e25ba8401e",
"precision": "fp32",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.007461726438312859,
"grade": "slow",
"residentMemoryBytes": 19143983104,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "fail",
"quality": {
"passed": false,
"acceptedAnswer": true,
"finalAnswer": "42",
"expected": [
"42"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": false,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 10534.1629589675,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 713
},
"parameterDtype": "torch.bfloat16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 46683.07541596005,
"inferenceMs": 46678.656041040085,
"tokensPerSecond": 11.589879527044445,
"inputTokens": 24,
"outputTokens": 541,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": null,
"inputHeight": null,
"visionGrid": null,
"inputModality": "text",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 19770343424,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 46684.89929201314,
"totalMs": 57219.06225098064,
"artifactRevision": "1de27d8c51f12e819435303b9e84c4e25ba8401e",
"precision": "bf16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.021421039447159455,
"grade": "slow",
"residentMemoryBytes": 827719680,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "Dog",
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 21061.0525420052,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 713
},
"parameterDtype": "torch.float16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 5942.261124961078,
"inferenceMs": 5915.678667020984,
"tokensPerSecond": 7.437861736015744,
"inputTokens": 828,
"outputTokens": 44,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 12676612096,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 5951.342332991771,
"totalMs": 27012.39487499697,
"artifactRevision": "1de27d8c51f12e819435303b9e84c4e25ba8401e",
"precision": "fp16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.16828610843091316,
"grade": "good",
"residentMemoryBytes": 2439331840,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "LOCAL AI 2026",
"expected": [
"localai2026"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 21061.0525420052,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 713
},
"parameterDtype": "torch.float16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 10512.467332999222,
"inferenceMs": 10499.336459033657,
"tokensPerSecond": 6.857576217404769,
"inputTokens": 793,
"outputTokens": 72,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 768,
"inputHeight": 256,
"visionGrid": [
[
1,
32,
96
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 13374078976,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 10516.864249948412,
"totalMs": 31577.916791953612,
"artifactRevision": "1de27d8c51f12e819435303b9e84c4e25ba8401e",
"precision": "fp16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.09512514696344826,
"grade": "moderate",
"residentMemoryBytes": 2480635904,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "3",
"expected": [
"3",
"three"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 21061.0525420052,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 713
},
"parameterDtype": "torch.float16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 8857.65929199988,
"inferenceMs": 8833.748042001389,
"tokensPerSecond": 8.71657190514059,
"inputTokens": 826,
"outputTokens": 77,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 640,
"inputHeight": 320,
"visionGrid": [
[
1,
40,
80
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 14900543488,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 8861.186750000343,
"totalMs": 29922.239292005543,
"artifactRevision": "1de27d8c51f12e819435303b9e84c4e25ba8401e",
"precision": "fp16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.1128966431236734,
"grade": "good",
"residentMemoryBytes": 2520481792,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "42",
"expected": [
"42"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 21061.0525420052,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 713
},
"parameterDtype": "torch.float16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 32457.96045899624,
"inferenceMs": 32454.874374961946,
"tokensPerSecond": 14.666518024399474,
"inputTokens": 24,
"outputTokens": 476,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": null,
"inputHeight": null,
"visionGrid": null,
"inputModality": "text",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 18914705408,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 32459.377208026126,
"totalMs": 53520.429750031326,
"artifactRevision": "1de27d8c51f12e819435303b9e84c4e25ba8401e",
"precision": "fp16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.03080908306802851,
"grade": "moderate",
"residentMemoryBytes": 2577825792,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "READY",
"expected": [
"ready"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 21061.0525420052,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 713
},
"parameterDtype": "torch.float16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 14168.662332987878,
"inferenceMs": 14166.351666033734,
"tokensPerSecond": 14.823859025292387,
"inputTokens": 17,
"outputTokens": 210,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": null,
"inputHeight": null,
"visionGrid": null,
"inputModality": "text",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 18914836480,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 14170.783000008669,
"totalMs": 35231.83554201387,
"artifactRevision": "1de27d8c51f12e819435303b9e84c4e25ba8401e",
"precision": "fp16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.07057829289020262,
"grade": "moderate",
"residentMemoryBytes": 2608087040,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "Dog",
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 21061.0525420052,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 713
},
"parameterDtype": "torch.float16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 5969.623000011779,
"inferenceMs": 5952.222875028383,
"tokensPerSecond": 7.392196314522276,
"inputTokens": 828,
"outputTokens": 44,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 19713851392,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 5978.216374991462,
"totalMs": 27039.26891699666,
"artifactRevision": "1de27d8c51f12e819435303b9e84c4e25ba8401e",
"precision": "fp16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.16751476600750614,
"grade": "good",
"residentMemoryBytes": 2613280768,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "Dog",
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 19684.861749992706,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 713
},
"parameterDtype": "torch.float16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 5416.185041016433,
"inferenceMs": 5377.04291602131,
"tokensPerSecond": 8.182936362456518,
"inputTokens": 828,
"outputTokens": 44,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 12676612096,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 5433.36491700029,
"totalMs": 25118.226666992996,
"artifactRevision": "1de27d8c51f12e819435303b9e84c4e25ba8401e",
"precision": "fp16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.18463180124516834,
"grade": "good",
"residentMemoryBytes": 2442002432,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "LOCAL AI 2026",
"expected": [
"localai2026"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 19684.861749992706,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 713
},
"parameterDtype": "torch.float16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 6543.261207989417,
"inferenceMs": 6523.7243750016205,
"tokensPerSecond": 11.03664040067329,
"inputTokens": 793,
"outputTokens": 72,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 768,
"inputHeight": 256,
"visionGrid": [
[
1,
32,
96
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 14028390400,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 6546.920291031711,
"totalMs": 26231.782041024417,
"artifactRevision": "1de27d8c51f12e819435303b9e84c4e25ba8401e",
"precision": "fp16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.1528289897366447,
"grade": "good",
"residentMemoryBytes": 2483994624,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "3",
"expected": [
"3",
"three"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 19684.861749992706,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 713
},
"parameterDtype": "torch.float16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 9496.606125030667,
"inferenceMs": 9481.694250018336,
"tokensPerSecond": 8.120911513240484,
"inputTokens": 826,
"outputTokens": 77,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 640,
"inputHeight": 320,
"visionGrid": [
[
1,
40,
80
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 14900543488,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 9500.49470900558,
"totalMs": 29185.356458998285,
"artifactRevision": "1de27d8c51f12e819435303b9e84c4e25ba8401e",
"precision": "fp16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.10530077659683615,
"grade": "good",
"residentMemoryBytes": 2524151808,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "42",
"expected": [
"42"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 19684.861749992706,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 713
},
"parameterDtype": "torch.float16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 22982.966082985513,
"inferenceMs": 22979.016625031363,
"tokensPerSecond": 20.71455048609376,
"inputTokens": 24,
"outputTokens": 476,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": null,
"inputHeight": null,
"visionGrid": null,
"inputModality": "text",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 18914705408,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 22984.96420797892,
"totalMs": 42669.825957971625,
"artifactRevision": "1de27d8c51f12e819435303b9e84c4e25ba8401e",
"precision": "fp16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.043510484956087045,
"grade": "moderate",
"residentMemoryBytes": 2582052864,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "READY",
"expected": [
"ready"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 19684.861749992706,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 713
},
"parameterDtype": "torch.float16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 8963.388292002492,
"inferenceMs": 8959.954124991782,
"tokensPerSecond": 23.437620000112737,
"inputTokens": 17,
"outputTokens": 210,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": null,
"inputHeight": null,
"visionGrid": null,
"inputModality": "text",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 18914836480,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 8964.92054197006,
"totalMs": 28649.782291962765,
"artifactRevision": "1de27d8c51f12e819435303b9e84c4e25ba8401e",
"precision": "fp16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.11156495372316311,
"grade": "good",
"residentMemoryBytes": 2612314112,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "Dog",
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 19684.861749992706,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 713
},
"parameterDtype": "torch.float16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 4997.112833021674,
"inferenceMs": 4973.179874999914,
"tokensPerSecond": 8.847457985822553,
"inputTokens": 828,
"outputTokens": 44,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 19713851392,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 5005.91574999271,
"totalMs": 24690.777499985415,
"artifactRevision": "1de27d8c51f12e819435303b9e84c4e25ba8401e",
"precision": "fp16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.2001155534035272,
"grade": "good",
"residentMemoryBytes": 2617393152,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "Dog",
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 15105.721917003393,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 713
},
"parameterDtype": "torch.float16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 6408.978249994107,
"inferenceMs": 6380.013666988816,
"tokensPerSecond": 6.896536950643672,
"inputTokens": 828,
"outputTokens": 44,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 12676612096,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 6421.450333029497,
"totalMs": 21527.17225003289,
"artifactRevision": "1de27d8c51f12e819435303b9e84c4e25ba8401e",
"precision": "fp16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.15603111151156107,
"grade": "good",
"residentMemoryBytes": 2446721024,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "LOCAL AI 2026",
"expected": [
"localai2026"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 15105.721917003393,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 713
},
"parameterDtype": "torch.float16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 8235.053249984048,
"inferenceMs": 8215.636208013166,
"tokensPerSecond": 8.763776556923785,
"inputTokens": 793,
"outputTokens": 72,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 768,
"inputHeight": 256,
"visionGrid": [
[
1,
32,
96
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 13374078976,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 8238.53762500221,
"totalMs": 23344.259542005602,
"artifactRevision": "1de27d8c51f12e819435303b9e84c4e25ba8401e",
"precision": "fp16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.12143212310156429,
"grade": "good",
"residentMemoryBytes": 2489729024,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "3",
"expected": [
"3",
"three"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 15105.721917003393,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 713
},
"parameterDtype": "torch.float16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 6896.21120900847,
"inferenceMs": 6879.89674997516,
"tokensPerSecond": 11.192028426920508,
"inputTokens": 826,
"outputTokens": 77,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 640,
"inputHeight": 320,
"visionGrid": [
[
1,
40,
80
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 14900543488,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 6898.716541996691,
"totalMs": 22004.438459000085,
"artifactRevision": "1de27d8c51f12e819435303b9e84c4e25ba8401e",
"precision": "fp16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.14500715968410413,
"grade": "good",
"residentMemoryBytes": 2529329152,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "42",
"expected": [
"42"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 15105.721917003393,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 713
},
"parameterDtype": "torch.float16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 27647.85262499936,
"inferenceMs": 27644.853041972965,
"tokensPerSecond": 17.218395021933844,
"inputTokens": 24,
"outputTokens": 476,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": null,
"inputHeight": null,
"visionGrid": null,
"inputModality": "text",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 18914705408,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 27650.592208025046,
"totalMs": 42756.31412502844,
"artifactRevision": "1de27d8c51f12e819435303b9e84c4e25ba8401e",
"precision": "fp16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.036169174277780755,
"grade": "moderate",
"residentMemoryBytes": 2587066368,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "READY",
"expected": [
"ready"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 15105.721917003393,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 713
},
"parameterDtype": "torch.float16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 8097.982249979395,
"inferenceMs": 8094.311750028282,
"tokensPerSecond": 25.944145281934098,
"inputTokens": 17,
"outputTokens": 210,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": null,
"inputHeight": null,
"visionGrid": null,
"inputModality": "text",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 18914836480,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 8099.707124987617,
"totalMs": 23205.42904199101,
"artifactRevision": "1de27d8c51f12e819435303b9e84c4e25ba8401e",
"precision": "fp16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.12348755148266032,
"grade": "good",
"residentMemoryBytes": 2617442304,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"acceptedAnswer": true,
"finalAnswer": "Dog",
"expected": [
"dog",
"labrador",
"puppy",
"canine"
],
"repetitive": false,
"eosWithinBudget": true,
"method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
"performancePassed": true,
"maximumRequestSeconds": 45,
"performanceGateVersion": 2,
"metric": "Complete response latency; token throughput retained as diagnostic"
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 15105.721917003393,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 713
},
"parameterDtype": "torch.float16",
"attentionImplementation": "sdpa",
"accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
"durationMs": 5454.548167006578,
"inferenceMs": 5433.968666999135,
"tokensPerSecond": 8.09721268126094,
"inputTokens": 828,
"outputTokens": 44,
"maxNewTokens": 1024,
"minPixels": 786432,
"maxPixels": 1048576,
"inputWidth": 700,
"inputHeight": 577,
"visionGrid": [
[
1,
52,
62
]
],
"inputModality": "image",
"videoFrameCount": 0,
"videoFps": null,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 19713851392,
"memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
"requestMs": 5464.136000024155,
"totalMs": 20569.857917027548,
"artifactRevision": "1de27d8c51f12e819435303b9e84c4e25ba8401e",
"precision": "fp16",
"cpuThreads": 4,
"fallback": false,
"requestsPerSecond": 0.18333324216454647,
"grade": "good",
"residentMemoryBytes": 2623209472,
"residentMemoryScope": "Post-inference process RSS, not peak"
}
}
]