A model is still running

A model is still running or downloading. Leaving will stop the current operation. Stay on this page to let it finish.

Skip to content
LocalAI.toolsBeta

Join the waitlist

Get an email when the new apps and model runs launch.

No account needed.

Qwen3-VL

Run local text questions, still-image questions using Qwen3-VL-8B Instruct. Answers can hallucinate or miss visual details.

Desktop inferencePrivate · Runs on your computer
About this model

Run local text questions, still-image questions using Qwen3-VL-8B Instruct. Answers can hallucinate or miss visual details.

License
Apache-2.0
Model released
Oct 15, 2025
Source repository created
Oct 11, 2025
Model weights · BF16
17.5 GB

Best for

  • Local visual questions and descriptions
  • Exploring printed-text reading
  • Image-content assistance with human verification

Keep in mind

  • Can hallucinate objects, text, counts or relationships; limited3-fixture validation is not broad visual/OCR accuracy.
  • Small visual processing budgets caused incorrect OCR even on a clear printed card; minimum786432/maximum1048576 pixels is the measured default. Other settings change quality, memory and speed.
  • Large images/output budgets require substantial memory; CPU recovery can be slow and needs room for FP32 weights.
  • No content-safety classifier; outputs may be biased or unsafe. Do not use answers as medical, legal, identity or safety-critical determinations.
  • Pinned source weights are BF16; FP32/FP16 casting does not create additional source precision.
  • The app downloads its own pinned CPython and isolated packages on request; installed Python, user packages and Python/pip configuration are not used. Automatic model-package provisioning is verified on Apple Silicon Mac; other platform stacks remain unverified.
  • Video and sampled video-frame inputs are not supported. Use a still image instead.

Processed on your deviceYour input is processed locally in the desktop app. Model and runtime files download from external hosts.

Target platforms · verification required

  • macos · aarch64, x86_64
  • windows · x86_64

Recommended configuration

BF16 · mps (verified platform only)

Sources & technical details
Original model
https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct
Release verification
Verified
Release evidence
Official news explicitly releases 8B Instruct/Thinking checkpoints on October 15.
Source updated
Oct 15, 2025
Native model files
https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct
Files updated
Oct 15, 2025
Artifact revision
0c351dd01ed87e9c1b53cbc748cba10e6187ff3b

Desktop app

Sample result

Qwen3-VL-8B Instruct — Text, Image

Run Qwen3-VL on your computer

Download the desktop app to load this model, provide input and view results locally.

Public desktop downloads are coming soon.

Device requirements

macOS Apple SiliconMPS / BF16Tested32.0 GiBRAM estimate
macOS Apple SiliconMPS / FP16Tested32.0 GiBRAM estimate
macOS Apple SiliconCPU / FP32Tested48.0 GiBRAM estimate
macOS Apple SiliconMPS / FP32Tested48.0 GiBRAM estimate
Windows Intel / AMDCPU / FP32Unverified48.0 GiBRAM estimate
macOS Intel / AMDCPU / FP32Unverified48.0 GiBRAM estimate
If your device falls short

MemoryAdvisory only: close other apps, reduce image/tile/token/batch settings or choose a smaller verified model/precision. CPU can need more RAM than GPU.

GPUIf the selected GPU runtime fails, existing CPU fallback may recover when enabled, but CPU has its own memory/runtime needs.

PlatformUse a compatible OS/architecture/runtime or another model. These estimates do not add hardware detection or block runs.

Benchmark evidence

Saved desktop measurements apply to the recorded hardware and runtime. Fallback cases report the device that actually ran; they do not prove GPU support on the requested device.

Qwen3-VL-8B Instruct — Text, Image

61 configurations tested
Passed61
Failed0
TestedNov 1, 2026
Runtimetransformers 5.22.1

Quality criteria

No explicit quality gate recorded. Review per-case evidence; full certification is pending.

PrecisionDeviceResultPlatformInference / load
FP16mpsAcceleratedPassedFastmacos · aarch64317 ms / 8203 ms
BF16mpsAcceleratedPassedFastmacos · aarch64372 ms / 12035 ms
FP32mpsAcceleratedPassedSlowmacos · aarch64542 ms / 19425 ms
FP16mpsAcceleratedPassedSlowmacos · aarch641672 ms / 8203 ms
BF16mpsAcceleratedPassedSlowmacos · aarch641773 ms / 12035 ms
FP16mpsAcceleratedPassedSlowmacos · aarch641871 ms / 8203 ms
BF16mpsAcceleratedPassedSlowmacos · aarch641939 ms / 12035 ms
FP32mpsAcceleratedPassedSlowmacos · aarch643074 ms / 19425 ms
FP32mpsAcceleratedPassedSlowmacos · aarch643322 ms / 19425 ms
BF16mpsAcceleratedPassedSlowmacos · aarch643539 ms / 4467 ms
BF16mpsAcceleratedPassedSlowmacos · aarch643543 ms / 4483 ms
BF16mpsAcceleratedPassedSlowmacos · aarch643556 ms / 4467 ms
BF16mpsAcceleratedPassedSlowmacos · aarch643596 ms / 4324 ms
BF16mpsAcceleratedPassedSlowmacos · aarch643599 ms / 4324 ms
BF16mpsAcceleratedPassedSlowmacos · aarch643701 ms / 4483 ms
FP16mpsAcceleratedPassedSlowmacos · aarch643733 ms / 3912 ms
FP16mpsAcceleratedPassedSlowmacos · aarch643757 ms / 3912 ms
FP16mpsAcceleratedPassedSlowmacos · aarch643819 ms / 3912 ms
BF16mpsAcceleratedPassedSlowmacos · aarch643835 ms / 4483 ms
BF16mpsAcceleratedPassedSlowmacos · aarch643835 ms / 4467 ms
BF16mpsAcceleratedPassedSlowmacos · aarch643991 ms / 4324 ms
BF16mpsAcceleratedPassedSlowmacos · aarch644022 ms / 4483 ms
BF16mpsAcceleratedPassedSlowmacos · aarch644275 ms / 4483 ms
FP16mpsAcceleratedPassedSlowmacos · aarch644296 ms / 3912 ms
FP16mpsAcceleratedPassedSlowmacos · aarch644354 ms / 8203 ms
FP32mpsAcceleratedPassedSlowmacos · aarch644455 ms / 14705 ms
FP32mpsAcceleratedPassedSlowmacos · aarch644465 ms / 14705 ms
FP16mpsAcceleratedPassedSlowmacos · aarch644492 ms / 3912 ms
BF16mpsAcceleratedPassedSlowmacos · aarch644517 ms / 12035 ms
FP32mpsAcceleratedPassedSlowmacos · aarch644528 ms / 14705 ms
FP32mpsAcceleratedPassedSlowmacos · aarch645514 ms / 14705 ms
FP32mpsAcceleratedPassedSlowmacos · aarch646360 ms / 14705 ms
FP32mpsAcceleratedPassedSlowmacos · aarch646896 ms / 19425 ms
BF16mpsAcceleratedPassedVery slowmacos · aarch643886 ms / 12035 ms
FP16mpsAcceleratedPassedVery slowmacos · aarch643896 ms / 8203 ms
BF16mpsAcceleratedPassedVery slowmacos · aarch644622 ms / 12035 ms
FP16mpsAcceleratedPassedVery slowmacos · aarch645040 ms / 8203 ms
FP32mpsAcceleratedPassedVery slowmacos · aarch646200 ms / 19425 ms
FP32mpsAcceleratedPassedVery slowmacos · aarch647158 ms / 19425 ms
FP32cpuPassedSlowmacos · aarch641040 ms / 10221 ms
FP32cpuPassedSlowmacos · aarch648718 ms / 3534 ms
FP32cpuPassedSlowmacos · aarch648783 ms / 3534 ms
FP32cpuPassedSlowmacos · aarch648991 ms / 3878 ms
FP32cpuPassedSlowmacos · aarch649002 ms / 4721 ms
FP32cpuPassedSlowmacos · aarch649024 ms / 4721 ms
FP32cpuPassedSlowmacos · aarch649106 ms / 3534 ms
FP32cpuPassedSlowmacos · aarch649197 ms / 3878 ms
FP32cpuPassedSlowmacos · aarch649585 ms / 3878 ms
FP32cpuPassedSlowmacos · aarch649769 ms / 3534 ms
FP32cpuPassedSlowmacos · aarch649800 ms / 4721 ms
FP32cpuPassedSlowmacos · aarch6410359 ms / 3534 ms
FP32cpuPassedSlowmacos · aarch6413715 ms / 4756 ms
FP32cpuPassedSlowmacos · aarch6413763 ms / 4756 ms
FP32cpuPassedSlowmacos · aarch6414163 ms / 4756 ms
FP32cpuPassedSlowmacos · aarch6414816 ms / 4756 ms
FP32cpuPassedSlowmacos · aarch6414948 ms / 4756 ms
FP32cpuPassedVery slowmacos · aarch645332 ms / 10221 ms
FP32cpuPassedVery slowmacos · aarch645603 ms / 10221 ms
FP32cpuPassedVery slowmacos · aarch649876 ms / 10221 ms
FP32cpuPassedVery slowmacos · aarch6410974 ms / 10221 ms
FP32cpuPassedVery slowmacos · aarch6411699 ms / 10221 ms
Test setup & sample details
Quality gate
No explicit quality gate recorded. Review the per-case quality evidence; full certification is pending.
Tested
2026-11-01T13:27:46Z
Environment
{ "platform": "macOS-27.2-arm64-arm-64bit-Mach-O", "hardware": "Apple M3 Max" }
Runtime
transformers 5.22.1
Artifact revision
0c351dd01ed87e9c1b53cbc748cba10e6187ff3b
Fixture
{
  "file": "model-samples/desktop/mobilenet-dog.jpg",
  "url": "https://assets.localai.tools/samples/v1/files/d4cb12ea8a768582562e1f1c0c930ea6fc4757f5b66551b986b41498efd75ca1/mobilenet-dog.jpg",
  "size": 83281,
  "sha1": "f1d042827c2f2154a6d8043b8e8f07c1be612941",
  "sha256": "d4cb12ea8a768582562e1f1c0c930ea6fc4757f5b66551b986b41498efd75ca1"
}
Rerun when
model revision/checksum; pinned runtime/processor/attention/backend/driver; fixture checksum/question/expected answers; visual pixel and token budgets; quality gates
Per-case quality & failure details
[
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "repetitive": false,
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 4483.0616250001185,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 750
      },
      "parameterDtype": "torch.bfloat16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 4304.363084000215,
      "inferenceMs": 4274.956541999927,
      "tokensPerSecond": 0.4678410132011195,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 20967997440,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "totalMs": 8801.348041999518,
      "requestMs": 4318.286416999399,
      "precision": "bf16",
      "cpuThreads": 4,
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "fallback": false,
      "grade": "slow"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "repetitive": false,
      "expected": [
        "localai2026"
      ],
      "method": "normalized exact printed text is contained in answer; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 4483.0616250001185,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 750
      },
      "parameterDtype": "torch.bfloat16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 4029.289042000528,
      "inferenceMs": 4021.5774579992285,
      "tokensPerSecond": 1.9892691570784946,
      "inputTokens": 791,
      "outputTokens": 8,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 768,
      "inputHeight": 256,
      "visionGrid": [
        [
          1,
          32,
          96
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 23469457408,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "totalMs": 8516.202999999223,
      "requestMs": 4033.1413749991043,
      "precision": "bf16",
      "cpuThreads": 4,
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "fallback": false,
      "grade": "slow"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "repetitive": false,
      "expected": [
        "3",
        "three"
      ],
      "method": "accepted count, with word boundaries; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 4483.0616250001185,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 750
      },
      "parameterDtype": "torch.bfloat16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 3844.211292000182,
      "inferenceMs": 3834.7400830007246,
      "tokensPerSecond": 0.5215477337997257,
      "inputTokens": 824,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 640,
      "inputHeight": 320,
      "visionGrid": [
        [
          1,
          40,
          80
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 25016041472,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "totalMs": 8330.259250000381,
      "requestMs": 3847.1976250002626,
      "precision": "bf16",
      "cpuThreads": 4,
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "fallback": false,
      "grade": "slow"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "repetitive": false,
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 4483.0616250001185,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 750
      },
      "parameterDtype": "torch.bfloat16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 3710.341709000204,
      "inferenceMs": 3701.039707999371,
      "tokensPerSecond": 0.5403886901502922,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 25016041472,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "totalMs": 8202.381042001434,
      "requestMs": 3719.3194170013157,
      "precision": "bf16",
      "cpuThreads": 4,
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "fallback": false,
      "grade": "slow"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "repetitive": false,
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 4483.0616250001185,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 750
      },
      "parameterDtype": "torch.bfloat16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 3552.740207998795,
      "inferenceMs": 3543.2400419995247,
      "tokensPerSecond": 0.5644551247708745,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 25016041472,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "totalMs": 8044.921875,
      "requestMs": 3561.8602499998815,
      "precision": "bf16",
      "cpuThreads": 4,
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "fallback": false,
      "grade": "slow"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "repetitive": false,
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 4324.244416000511,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 750
      },
      "parameterDtype": "torch.bfloat16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 4018.6762920002366,
      "inferenceMs": 3990.6664160007495,
      "tokensPerSecond": 0.5011694267355732,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 20967997440,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "totalMs": 8357.219833000272,
      "requestMs": 4032.9754169997614,
      "precision": "bf16",
      "cpuThreads": 4,
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "fallback": false,
      "grade": "slow"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "repetitive": false,
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 4324.244416000511,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 750
      },
      "parameterDtype": "torch.bfloat16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 3608.2952090000617,
      "inferenceMs": 3598.787249999077,
      "tokensPerSecond": 0.555742771401814,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 22041739264,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "totalMs": 7943.624916000772,
      "requestMs": 3619.3805000002612,
      "precision": "bf16",
      "cpuThreads": 4,
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "fallback": false,
      "grade": "slow"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "repetitive": false,
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 4324.244416000511,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 750
      },
      "parameterDtype": "torch.bfloat16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 3604.3845410003996,
      "inferenceMs": 3596.0347079999337,
      "tokensPerSecond": 0.5561681580966645,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 22041739264,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "totalMs": 7937.82445800025,
      "requestMs": 3613.5800419997395,
      "precision": "bf16",
      "cpuThreads": 4,
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "fallback": false,
      "grade": "slow"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "repetitive": false,
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 4467.237291999481,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 750
      },
      "parameterDtype": "torch.bfloat16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 3866.0650000001624,
      "inferenceMs": 3835.346500000014,
      "tokensPerSecond": 0.5214652704781674,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 20967997440,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "totalMs": 8346.760874999745,
      "requestMs": 3879.523583000264,
      "precision": "bf16",
      "cpuThreads": 4,
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "fallback": false,
      "grade": "slow"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "repetitive": false,
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 4467.237291999481,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 750
      },
      "parameterDtype": "torch.bfloat16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 3564.7898749994056,
      "inferenceMs": 3555.5632499999774,
      "tokensPerSecond": 0.5624987827174815,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 22041739264,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "totalMs": 8041.285374998552,
      "requestMs": 3574.04808299907,
      "precision": "bf16",
      "cpuThreads": 4,
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "fallback": false,
      "grade": "slow"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "repetitive": false,
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 4467.237291999481,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 750
      },
      "parameterDtype": "torch.bfloat16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 3547.0856669999193,
      "inferenceMs": 3539.3172090007283,
      "tokensPerSecond": 0.5650807435156876,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 22041739264,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "totalMs": 8023.073541999111,
      "requestMs": 3555.836249999629,
      "precision": "bf16",
      "cpuThreads": 4,
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "fallback": false,
      "grade": "slow"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "repetitive": false,
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3911.7337499992573,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 750
      },
      "parameterDtype": "torch.float16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 4521.122790998561,
      "inferenceMs": 4491.582208000182,
      "tokensPerSecond": 0.44527738943254785,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 20967997440,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "totalMs": 8446.037707999494,
      "requestMs": 4534.303958000237,
      "precision": "fp16",
      "cpuThreads": 4,
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "fallback": false,
      "grade": "slow"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "repetitive": false,
      "expected": [
        "localai2026"
      ],
      "method": "normalized exact printed text is contained in answer; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3911.7337499992573,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 750
      },
      "parameterDtype": "torch.float16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 4303.052041001138,
      "inferenceMs": 4296.136041999489,
      "tokensPerSecond": 1.8621384243401833,
      "inputTokens": 791,
      "outputTokens": 8,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 768,
      "inputHeight": 256,
      "visionGrid": [
        [
          1,
          32,
          96
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 23469457408,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "totalMs": 8218.96575000028,
      "requestMs": 4307.232000001022,
      "precision": "fp16",
      "cpuThreads": 4,
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "fallback": false,
      "grade": "slow"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "repetitive": false,
      "expected": [
        "3",
        "three"
      ],
      "method": "accepted count, with word boundaries; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3911.7337499992573,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 750
      },
      "parameterDtype": "torch.float16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 3832.534332999785,
      "inferenceMs": 3819.3320829996082,
      "tokensPerSecond": 0.5236517685650549,
      "inputTokens": 824,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 640,
      "inputHeight": 320,
      "visionGrid": [
        [
          1,
          40,
          80
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 25016041472,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "totalMs": 7747.81208299828,
      "requestMs": 3836.0783329990227,
      "precision": "fp16",
      "cpuThreads": 4,
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "fallback": false,
      "grade": "slow"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "repetitive": false,
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3911.7337499992573,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 750
      },
      "parameterDtype": "torch.float16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 3767.1306659995025,
      "inferenceMs": 3757.2959170011018,
      "tokensPerSecond": 0.5322977066965507,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 25016041472,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "totalMs": 7690.020624999306,
      "requestMs": 3778.286875000049,
      "precision": "fp16",
      "cpuThreads": 4,
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "fallback": false,
      "grade": "slow"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "repetitive": false,
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3911.7337499992573,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 750
      },
      "parameterDtype": "torch.float16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 3744.0704999989975,
      "inferenceMs": 3733.072582999739,
      "tokensPerSecond": 0.5357517046702812,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 25016041472,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "totalMs": 7670.307541999136,
      "requestMs": 3758.573791999879,
      "precision": "fp16",
      "cpuThreads": 4,
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "fallback": false,
      "grade": "slow"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "repetitive": false,
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3533.535499998834,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 750
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 9795.339166999838,
      "inferenceMs": 9769.395958001041,
      "tokensPerSecond": 0.2047209478045589,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 13345.9869169983,
      "requestMs": 9812.451416999465,
      "precision": "fp32",
      "cpuThreads": 4,
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "fallback": false,
      "grade": "slow"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "repetitive": false,
      "expected": [
        "localai2026"
      ],
      "method": "normalized exact printed text is contained in answer; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3533.535499998834,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 750
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 10364.259250000032,
      "inferenceMs": 10358.760166000138,
      "tokensPerSecond": 0.7722931964635944,
      "inputTokens": 791,
      "outputTokens": 8,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 768,
      "inputHeight": 256,
      "visionGrid": [
        [
          1,
          32,
          96
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 13901.250082999468,
      "requestMs": 10367.714583000634,
      "precision": "fp32",
      "cpuThreads": 4,
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "fallback": false,
      "grade": "slow"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "repetitive": false,
      "expected": [
        "3",
        "three"
      ],
      "method": "accepted count, with word boundaries; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3533.535499998834,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 750
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 8723.67254100027,
      "inferenceMs": 8718.475333998867,
      "tokensPerSecond": 0.2293979077053451,
      "inputTokens": 824,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 640,
      "inputHeight": 320,
      "visionGrid": [
        [
          1,
          40,
          80
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 12260.151790998862,
      "requestMs": 8726.616291000028,
      "precision": "fp32",
      "cpuThreads": 4,
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "fallback": false,
      "grade": "slow"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "repetitive": false,
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3533.535499998834,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 750
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 8788.448291999885,
      "inferenceMs": 8782.657707999533,
      "tokensPerSecond": 0.2277215014514723,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 12330.783790997884,
      "requestMs": 8797.24829099905,
      "precision": "fp32",
      "cpuThreads": 4,
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "fallback": false,
      "grade": "slow"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "repetitive": false,
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3533.535499998834,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 750
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 9111.44008300107,
      "inferenceMs": 9105.68366699954,
      "tokensPerSecond": 0.21964303539868416,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 12653.560249998918,
      "requestMs": 9120.024750000084,
      "precision": "fp32",
      "cpuThreads": 4,
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "fallback": false,
      "grade": "slow"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "repetitive": false,
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3877.8249580009287,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 750
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 9223.932999999306,
      "inferenceMs": 9197.179499999038,
      "tokensPerSecond": 0.2174579717619091,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 13116.675000001123,
      "requestMs": 9238.850042000195,
      "precision": "fp32",
      "cpuThreads": 4,
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "fallback": false,
      "grade": "slow"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "repetitive": false,
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3877.8249580009287,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 750
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 9592.003375000786,
      "inferenceMs": 9585.410166000656,
      "tokensPerSecond": 0.20865043491763952,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 13478.728083000533,
      "requestMs": 9600.903124999604,
      "precision": "fp32",
      "cpuThreads": 4,
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "fallback": false,
      "grade": "slow"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "repetitive": false,
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3877.8249580009287,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 750
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 8997.418625000137,
      "inferenceMs": 8990.72145900027,
      "tokensPerSecond": 0.22245155843393147,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 12884.291500002291,
      "requestMs": 9006.466542001363,
      "precision": "fp32",
      "cpuThreads": 4,
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "fallback": false,
      "grade": "slow"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "repetitive": false,
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 4720.506042000125,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 750
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 9825.947249999444,
      "inferenceMs": 9799.925082999835,
      "tokensPerSecond": 0.20408319278577425,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 14563.516000000163,
      "requestMs": 9843.009958000039,
      "precision": "fp32",
      "cpuThreads": 4,
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "fallback": false,
      "grade": "slow"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "repetitive": false,
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 4720.506042000125,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 750
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 9007.693583000218,
      "inferenceMs": 9001.534166000056,
      "tokensPerSecond": 0.22218434803638867,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 13737.47241700039,
      "requestMs": 9016.966375000266,
      "precision": "fp32",
      "cpuThreads": 4,
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "fallback": false,
      "grade": "slow"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "repetitive": false,
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 4720.506042000125,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 750
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 9029.408582999167,
      "inferenceMs": 9023.566374999064,
      "tokensPerSecond": 0.2216418561004055,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 13758.67745799951,
      "requestMs": 9038.171415999386,
      "precision": "fp32",
      "cpuThreads": 4,
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "fallback": false,
      "grade": "slow"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "repetitive": false,
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 4755.856499999936,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 750
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 14852.60958300023,
      "inferenceMs": 14816.483332999269,
      "tokensPerSecond": 0.13498479734024338,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 19625.404540998716,
      "requestMs": 14869.54804099878,
      "precision": "fp32",
      "cpuThreads": 1,
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "fallback": false,
      "grade": "slow"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "repetitive": false,
      "expected": [
        "localai2026"
      ],
      "method": "normalized exact printed text is contained in answer; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 4755.856499999936,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 750
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 14960.14683300018,
      "inferenceMs": 14948.099916000501,
      "tokensPerSecond": 0.535185076695719,
      "inputTokens": 791,
      "outputTokens": 8,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 768,
      "inputHeight": 256,
      "visionGrid": [
        [
          1,
          32,
          96
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 19718.798416000936,
      "requestMs": 14962.941916001,
      "precision": "fp32",
      "cpuThreads": 1,
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "fallback": false,
      "grade": "slow"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "repetitive": false,
      "expected": [
        "3",
        "three"
      ],
      "method": "accepted count, with word boundaries; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 4755.856499999936,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 750
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 13775.510957999359,
      "inferenceMs": 13762.813374998586,
      "tokensPerSecond": 0.14531912520394874,
      "inputTokens": 824,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 640,
      "inputHeight": 320,
      "visionGrid": [
        [
          1,
          40,
          80
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 18534.327625000515,
      "requestMs": 13778.471125000578,
      "precision": "fp32",
      "cpuThreads": 1,
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "fallback": false,
      "grade": "slow"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "repetitive": false,
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 4755.856499999936,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 750
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 13728.663750000123,
      "inferenceMs": 13714.888458000132,
      "tokensPerSecond": 0.1458269242308978,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 18493.118291000428,
      "requestMs": 13737.261791000492,
      "precision": "fp32",
      "cpuThreads": 1,
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "fallback": false,
      "grade": "slow"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "repetitive": false,
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 4755.856499999936,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 750
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 14176.67149999943,
      "inferenceMs": 14162.826333998964,
      "tokensPerSecond": 0.1412147514086821,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 18941.20537500021,
      "requestMs": 14185.348875000273,
      "precision": "fp32",
      "cpuThreads": 1,
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "fallback": false,
      "grade": "slow"
    }
  },
  {
    "precision": "fp32",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "repetitive": false,
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 14705.334291000327,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float32": 750
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 6390.0757920000615,
      "inferenceMs": 6360.303584000576,
      "tokensPerSecond": 0.31445039903928884,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 38604079104,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "totalMs": 21112.886165999953,
      "requestMs": 6407.551874999626,
      "precision": "fp32",
      "cpuThreads": 4,
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "fallback": false,
      "grade": "slow"
    }
  },
  {
    "precision": "fp32",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "repetitive": false,
      "expected": [
        "localai2026"
      ],
      "method": "normalized exact printed text is contained in answer; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 14705.334291000327,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float32": 750
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 5522.364541999195,
      "inferenceMs": 5513.772459000393,
      "tokensPerSecond": 1.4509122491881616,
      "inputTokens": 791,
      "outputTokens": 8,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 768,
      "inputHeight": 256,
      "visionGrid": [
        [
          1,
          32,
          96
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 41112616960,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "totalMs": 20232.68512499999,
      "requestMs": 5527.350833999662,
      "precision": "fp32",
      "cpuThreads": 4,
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "fallback": false,
      "grade": "slow"
    }
  },
  {
    "precision": "fp32",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "repetitive": false,
      "expected": [
        "3",
        "three"
      ],
      "method": "accepted count, with word boundaries; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 14705.334291000327,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float32": 750
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 4478.391374999774,
      "inferenceMs": 4464.863459001208,
      "tokensPerSecond": 0.4479420296645311,
      "inputTokens": 824,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 640,
      "inputHeight": 320,
      "visionGrid": [
        [
          1,
          40,
          80
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 43322982400,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "totalMs": 19187.62279099974,
      "requestMs": 4482.288499999413,
      "precision": "fp32",
      "cpuThreads": 4,
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "fallback": false,
      "grade": "slow"
    }
  },
  {
    "precision": "fp32",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "repetitive": false,
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 14705.334291000327,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float32": 750
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 4537.940041000184,
      "inferenceMs": 4527.87595799964,
      "tokensPerSecond": 0.4417082134210177,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 43989876736,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "totalMs": 19252.835166000295,
      "requestMs": 4547.500874999969,
      "precision": "fp32",
      "cpuThreads": 4,
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "fallback": false,
      "grade": "slow"
    }
  },
  {
    "precision": "fp32",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "repetitive": false,
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "method": "contains an accepted animal label; nonempty, nonrepetitive answer required. Limited3-fixture compatibility, not broad VQA/OCR accuracy."
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 14705.334291000327,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float32": 750
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 4464.924375000919,
      "inferenceMs": 4455.1144580000255,
      "tokensPerSecond": 0.44892224854259594,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 43989876736,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "totalMs": 19179.560041000514,
      "requestMs": 4474.225750000187,
      "precision": "fp32",
      "cpuThreads": 4,
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "fallback": false,
      "grade": "slow"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 4670.471958001144,
      "inferenceMs": 4621.939624950755,
      "tokensPerSecond": 0.43271876361243233,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "placement": {
        "mps:0/torch.bfloat16": 750
      },
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 20967997440,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "loadMs": 12035.147416987456,
      "totalMs": 16713.850374973845,
      "wallMs": 4677.5610410259105,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "device": "mps",
      "precision": "bf16",
      "requestedDevice": "mps",
      "requestedPrecision": "bf16",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "fallback": false,
      "parameterDtype": "torch.bfloat16",
      "grade": "very-slow"
    },
    "failure": null
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 4528.145957970992,
      "inferenceMs": 4516.854416986462,
      "tokensPerSecond": 1.7711440886636791,
      "inputTokens": 791,
      "outputTokens": 8,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 768,
      "inputHeight": 256,
      "visionGrid": [
        [
          1,
          32,
          96
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "placement": {
        "mps:0/torch.bfloat16": 750
      },
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 23469457408,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "loadMs": 12035.147416987456,
      "totalMs": 21244.47691696696,
      "wallMs": 4530.537542013917,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "device": "mps",
      "precision": "bf16",
      "requestedDevice": "mps",
      "requestedPrecision": "bf16",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "fallback": false,
      "parameterDtype": "torch.bfloat16",
      "grade": "slow"
    },
    "failure": null
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 3899.396665976383,
      "inferenceMs": 3886.4272499922663,
      "tokensPerSecond": 0.5146114596649094,
      "inputTokens": 824,
      "outputTokens": 2,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 640,
      "inputHeight": 320,
      "visionGrid": [
        [
          1,
          40,
          80
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "placement": {
        "mps:0/torch.bfloat16": 750
      },
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 25005555712,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "loadMs": 12035.147416987456,
      "totalMs": 25145.779124984983,
      "wallMs": 3901.1726250173524,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "device": "mps",
      "precision": "bf16",
      "requestedDevice": "mps",
      "requestedPrecision": "bf16",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "fallback": false,
      "parameterDtype": "torch.bfloat16",
      "grade": "very-slow"
    },
    "failure": null
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 1952.4582500453107,
      "inferenceMs": 1938.6689999955706,
      "tokensPerSecond": 2.063271244348127,
      "inputTokens": 426,
      "outputTokens": 4,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 320,
      "inputHeight": 240,
      "visionGrid": [
        [
          2,
          24,
          32
        ]
      ],
      "inputModality": "video",
      "videoFrameCount": 4,
      "videoFps": 1,
      "placement": {
        "mps:0/torch.bfloat16": 750
      },
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 25495011328,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "loadMs": 12035.147416987456,
      "totalMs": 27100.44933401514,
      "wallMs": 1954.5968750026077,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "device": "mps",
      "precision": "bf16",
      "requestedDevice": "mps",
      "requestedPrecision": "bf16",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "fallback": false,
      "parameterDtype": "torch.bfloat16",
      "grade": "slow"
    },
    "failure": null
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 1784.460750001017,
      "inferenceMs": 1773.1180000118911,
      "tokensPerSecond": 2.255913030025737,
      "inputTokens": 426,
      "outputTokens": 4,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 320,
      "inputHeight": 240,
      "visionGrid": [
        [
          2,
          24,
          32
        ]
      ],
      "inputModality": "video",
      "videoFrameCount": 4,
      "videoFps": 1,
      "placement": {
        "mps:0/torch.bfloat16": 750
      },
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 24823922688,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "loadMs": 12035.147416987456,
      "totalMs": 28886.86475000577,
      "wallMs": 1786.3384159863926,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "device": "mps",
      "precision": "bf16",
      "requestedDevice": "mps",
      "requestedPrecision": "bf16",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "fallback": false,
      "parameterDtype": "torch.bfloat16",
      "grade": "slow"
    },
    "failure": null
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 374.75775001803413,
      "inferenceMs": 372.2920840373263,
      "tokensPerSecond": 5.372126042302523,
      "inputTokens": 21,
      "outputTokens": 2,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": null,
      "inputHeight": null,
      "visionGrid": null,
      "inputModality": "text",
      "videoFrameCount": 0,
      "videoFps": null,
      "placement": {
        "mps:0/torch.bfloat16": 750
      },
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 24824102912,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "loadMs": 12035.147416987456,
      "totalMs": 29263.1937499973,
      "wallMs": 376.2596669839695,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "device": "mps",
      "precision": "bf16",
      "requestedDevice": "mps",
      "requestedPrecision": "bf16",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "fallback": false,
      "parameterDtype": "torch.bfloat16",
      "grade": "good"
    },
    "failure": null
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 5093.2034580037,
      "inferenceMs": 5039.896500005852,
      "tokensPerSecond": 0.3968335460852574,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "placement": {
        "mps:0/torch.float16": 750
      },
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 20967997440,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "loadMs": 8202.622334007174,
      "totalMs": 13305.446209036745,
      "wallMs": 5100.997000001371,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "device": "mps",
      "precision": "fp16",
      "requestedDevice": "mps",
      "requestedPrecision": "fp16",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "fallback": false,
      "parameterDtype": "torch.float16",
      "grade": "very-slow"
    },
    "failure": null
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 4366.2984580150805,
      "inferenceMs": 4353.796207986306,
      "tokensPerSecond": 1.8374769093062615,
      "inputTokens": 791,
      "outputTokens": 8,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 768,
      "inputHeight": 256,
      "visionGrid": [
        [
          1,
          32,
          96
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "placement": {
        "mps:0/torch.float16": 750
      },
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 23469457408,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "loadMs": 8202.622334007174,
      "totalMs": 17674.725249991752,
      "wallMs": 4369.10795798758,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "device": "mps",
      "precision": "fp16",
      "requestedDevice": "mps",
      "requestedPrecision": "fp16",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "fallback": false,
      "parameterDtype": "torch.float16",
      "grade": "slow"
    },
    "failure": null
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 3909.3517089495435,
      "inferenceMs": 3895.908042031806,
      "tokensPerSecond": 0.5133591394926647,
      "inputTokens": 824,
      "outputTokens": 2,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 640,
      "inputHeight": 320,
      "visionGrid": [
        [
          1,
          40,
          80
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "placement": {
        "mps:0/torch.float16": 750
      },
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 25016041472,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "loadMs": 8202.622334007174,
      "totalMs": 21586.266625032295,
      "wallMs": 3911.3699159934185,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "device": "mps",
      "precision": "fp16",
      "requestedDevice": "mps",
      "requestedPrecision": "fp16",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "fallback": false,
      "parameterDtype": "torch.float16",
      "grade": "very-slow"
    },
    "failure": null
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 1893.082750029862,
      "inferenceMs": 1871.4858750463463,
      "tokensPerSecond": 2.1373391342859813,
      "inputTokens": 426,
      "outputTokens": 4,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 320,
      "inputHeight": 240,
      "visionGrid": [
        [
          2,
          24,
          32
        ]
      ],
      "inputModality": "video",
      "videoFrameCount": 4,
      "videoFps": 1,
      "placement": {
        "mps:0/torch.float16": 750
      },
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 25505497088,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "loadMs": 8202.622334007174,
      "totalMs": 23481.45541700069,
      "wallMs": 1895.1128749758936,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "device": "mps",
      "precision": "fp16",
      "requestedDevice": "mps",
      "requestedPrecision": "fp16",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "fallback": false,
      "parameterDtype": "torch.float16",
      "grade": "slow"
    },
    "failure": null
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 1682.2373750037514,
      "inferenceMs": 1671.9542079954408,
      "tokensPerSecond": 2.3924100198866856,
      "inputTokens": 426,
      "outputTokens": 4,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 320,
      "inputHeight": 240,
      "visionGrid": [
        [
          2,
          24,
          32
        ]
      ],
      "inputModality": "video",
      "videoFrameCount": 4,
      "videoFps": 1,
      "placement": {
        "mps:0/torch.float16": 750
      },
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 24823922688,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "loadMs": 8202.622334007174,
      "totalMs": 25165.605584043078,
      "wallMs": 1684.0610840008594,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "device": "mps",
      "precision": "fp16",
      "requestedDevice": "mps",
      "requestedPrecision": "fp16",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "fallback": false,
      "parameterDtype": "torch.float16",
      "grade": "slow"
    },
    "failure": null
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 319.3915000301786,
      "inferenceMs": 317.1830829815008,
      "tokensPerSecond": 6.305506527019435,
      "inputTokens": 21,
      "outputTokens": 2,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": null,
      "inputHeight": null,
      "visionGrid": null,
      "inputModality": "text",
      "videoFrameCount": 0,
      "videoFps": null,
      "placement": {
        "mps:0/torch.float16": 750
      },
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 24824102912,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "loadMs": 8202.622334007174,
      "totalMs": 25486.61408398766,
      "wallMs": 320.93920902116224,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "device": "mps",
      "precision": "fp16",
      "requestedDevice": "mps",
      "requestedPrecision": "fp16",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "fallback": false,
      "parameterDtype": "torch.float16",
      "grade": "good"
    },
    "failure": null
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 11013.27591703739,
      "inferenceMs": 10973.914750036784,
      "tokensPerSecond": 0.18225036785467066,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "placement": {
        "cpu/torch.float32": 750
      },
      "decoding": "greedy",
      "outputLimitReached": false,
      "loadMs": 10221.168874995783,
      "totalMs": 21242.945792037062,
      "wallMs": 11019.916124991141,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "device": "cpu",
      "precision": "fp32",
      "requestedDevice": "cpu",
      "requestedPrecision": "fp32",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "fallback": false,
      "parameterDtype": "torch.float32",
      "grade": "very-slow"
    },
    "failure": null
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 11706.617666990496,
      "inferenceMs": 11698.730167001486,
      "tokensPerSecond": 0.6838349022328539,
      "inputTokens": 791,
      "outputTokens": 8,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 768,
      "inputHeight": 256,
      "visionGrid": [
        [
          1,
          32,
          96
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "placement": {
        "cpu/torch.float32": 750
      },
      "decoding": "greedy",
      "outputLimitReached": false,
      "loadMs": 10221.168874995783,
      "totalMs": 32951.75000000745,
      "wallMs": 11708.72087503085,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "device": "cpu",
      "precision": "fp32",
      "requestedDevice": "cpu",
      "requestedPrecision": "fp32",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "fallback": false,
      "parameterDtype": "torch.float32",
      "grade": "very-slow"
    },
    "failure": null
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 9883.481499971822,
      "inferenceMs": 9875.795457977802,
      "tokensPerSecond": 0.2025153324114639,
      "inputTokens": 824,
      "outputTokens": 2,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 640,
      "inputHeight": 320,
      "visionGrid": [
        [
          1,
          40,
          80
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "placement": {
        "cpu/torch.float32": 750
      },
      "decoding": "greedy",
      "outputLimitReached": false,
      "loadMs": 10221.168874995783,
      "totalMs": 42837.194583029486,
      "wallMs": 9885.31233300455,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "device": "cpu",
      "precision": "fp32",
      "requestedDevice": "cpu",
      "requestedPrecision": "fp32",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "fallback": false,
      "parameterDtype": "torch.float32",
      "grade": "very-slow"
    },
    "failure": null
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 5341.767417034134,
      "inferenceMs": 5331.669541017618,
      "tokensPerSecond": 0.7502340438069514,
      "inputTokens": 426,
      "outputTokens": 4,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 320,
      "inputHeight": 240,
      "visionGrid": [
        [
          2,
          24,
          32
        ]
      ],
      "inputModality": "video",
      "videoFrameCount": 4,
      "videoFps": 1,
      "placement": {
        "cpu/torch.float32": 750
      },
      "decoding": "greedy",
      "outputLimitReached": false,
      "loadMs": 10221.168874995783,
      "totalMs": 48180.604958033655,
      "wallMs": 5343.338917009532,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "device": "cpu",
      "precision": "fp32",
      "requestedDevice": "cpu",
      "requestedPrecision": "fp32",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "fallback": false,
      "parameterDtype": "torch.float32",
      "grade": "very-slow"
    },
    "failure": null
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 5610.365375003312,
      "inferenceMs": 5602.9090830124915,
      "tokensPerSecond": 0.7139148504350418,
      "inputTokens": 426,
      "outputTokens": 4,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 320,
      "inputHeight": 240,
      "visionGrid": [
        [
          2,
          24,
          32
        ]
      ],
      "inputModality": "video",
      "videoFrameCount": 4,
      "videoFps": 1,
      "placement": {
        "cpu/torch.float32": 750
      },
      "decoding": "greedy",
      "outputLimitReached": false,
      "loadMs": 10221.168874995783,
      "totalMs": 53792.62308299076,
      "wallMs": 5611.928957980126,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "device": "cpu",
      "precision": "fp32",
      "requestedDevice": "cpu",
      "requestedPrecision": "fp32",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "fallback": false,
      "parameterDtype": "torch.float32",
      "grade": "very-slow"
    },
    "failure": null
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 1041.0531670204364,
      "inferenceMs": 1040.2970829745755,
      "tokensPerSecond": 1.9225277401348624,
      "inputTokens": 21,
      "outputTokens": 2,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": null,
      "inputHeight": null,
      "visionGrid": null,
      "inputModality": "text",
      "videoFrameCount": 0,
      "videoFps": null,
      "placement": {
        "cpu/torch.float32": 750
      },
      "decoding": "greedy",
      "outputLimitReached": false,
      "loadMs": 10221.168874995783,
      "totalMs": 54834.98145802878,
      "wallMs": 1042.285958013963,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "device": "cpu",
      "precision": "fp32",
      "requestedDevice": "cpu",
      "requestedPrecision": "fp32",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "fallback": false,
      "parameterDtype": "torch.float32",
      "grade": "slow"
    },
    "failure": null
  },
  {
    "precision": "fp32",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 7214.809249970131,
      "inferenceMs": 7158.1280419486575,
      "tokensPerSecond": 0.2794026578288952,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "placement": {
        "mps:0/torch.float32": 750
      },
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 38604079104,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "loadMs": 19425.220499979332,
      "totalMs": 26651.330458989833,
      "wallMs": 7223.546083958354,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "device": "mps",
      "precision": "fp32",
      "requestedDevice": "mps",
      "requestedPrecision": "fp32",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "fallback": false,
      "parameterDtype": "torch.float32",
      "grade": "very-slow"
    },
    "failure": null
  },
  {
    "precision": "fp32",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 6917.717540985905,
      "inferenceMs": 6895.649957994465,
      "tokensPerSecond": 1.1601516969006247,
      "inputTokens": 791,
      "outputTokens": 8,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 768,
      "inputHeight": 256,
      "visionGrid": [
        [
          1,
          32,
          96
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "placement": {
        "mps:0/torch.float32": 750
      },
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 41104228352,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "loadMs": 19425.220499979332,
      "totalMs": 33572.96566700097,
      "wallMs": 6921.330375014804,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "device": "mps",
      "precision": "fp32",
      "requestedDevice": "mps",
      "requestedPrecision": "fp32",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "fallback": false,
      "parameterDtype": "torch.float32",
      "grade": "slow"
    },
    "failure": null
  },
  {
    "precision": "fp32",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 6212.775917025283,
      "inferenceMs": 6200.256666983478,
      "tokensPerSecond": 0.3225672915526304,
      "inputTokens": 824,
      "outputTokens": 2,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 640,
      "inputHeight": 320,
      "visionGrid": [
        [
          1,
          40,
          80
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "placement": {
        "mps:0/torch.float32": 750
      },
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 43312496640,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "loadMs": 19425.220499979332,
      "totalMs": 39787.781000020914,
      "wallMs": 6214.658958022483,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "device": "mps",
      "precision": "fp32",
      "requestedDevice": "mps",
      "requestedPrecision": "fp32",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "fallback": false,
      "parameterDtype": "torch.float32",
      "grade": "very-slow"
    },
    "failure": null
  },
  {
    "precision": "fp32",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 3334.3274999642745,
      "inferenceMs": 3322.058250021655,
      "tokensPerSecond": 1.2040728063615156,
      "inputTokens": 426,
      "outputTokens": 4,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 320,
      "inputHeight": 240,
      "visionGrid": [
        [
          2,
          24,
          32
        ]
      ],
      "inputModality": "video",
      "videoFrameCount": 4,
      "videoFps": 1,
      "placement": {
        "mps:0/torch.float32": 750
      },
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 42798858240,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "loadMs": 19425.220499979332,
      "totalMs": 43124.384749971796,
      "wallMs": 3336.5318340365775,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "device": "mps",
      "precision": "fp32",
      "requestedDevice": "mps",
      "requestedPrecision": "fp32",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "fallback": false,
      "parameterDtype": "torch.float32",
      "grade": "slow"
    },
    "failure": null
  },
  {
    "precision": "fp32",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 3085.90337500209,
      "inferenceMs": 3074.002458015457,
      "tokensPerSecond": 1.3012351338789616,
      "inputTokens": 426,
      "outputTokens": 4,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 320,
      "inputHeight": 240,
      "visionGrid": [
        [
          2,
          24,
          32
        ]
      ],
      "inputModality": "video",
      "videoFrameCount": 4,
      "videoFps": 1,
      "placement": {
        "mps:0/torch.float32": 750
      },
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 42798858240,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "loadMs": 19425.220499979332,
      "totalMs": 46212.73137500975,
      "wallMs": 3088.235333969351,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "device": "mps",
      "precision": "fp32",
      "requestedDevice": "mps",
      "requestedPrecision": "fp32",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "fallback": false,
      "parameterDtype": "torch.float32",
      "grade": "slow"
    },
    "failure": null
  },
  {
    "precision": "fp32",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 546.0952919675037,
      "inferenceMs": 541.8673329986632,
      "tokensPerSecond": 3.690940343150256,
      "inputTokens": 21,
      "outputTokens": 2,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": null,
      "inputHeight": null,
      "visionGrid": null,
      "inputModality": "text",
      "videoFrameCount": 0,
      "videoFps": null,
      "placement": {
        "mps:0/torch.float32": 750
      },
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 42824204288,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "loadMs": 19425.220499979332,
      "totalMs": 46760.43570897309,
      "wallMs": 547.5516250007786,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b",
      "device": "mps",
      "precision": "fp32",
      "requestedDevice": "mps",
      "requestedPrecision": "fp32",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "fallback": false,
      "parameterDtype": "torch.float32",
      "grade": "slow"
    },
    "failure": null
  }
]

Qwen3-VL 2B Instruct — Text, Image

45 configurations tested
Passed45
Failed0
TestedNov 1, 2026
Runtimetransformers 5.22.1

Quality criteria

No explicit quality gate recorded. Review per-case evidence; full certification is pending.

PrecisionDeviceResultPlatformInference / load
FP32mpsAcceleratedPassedFastmacos · aarch64162 ms / 5577 ms
FP32mpsAcceleratedPassedFastmacos · aarch641415 ms / 5577 ms
FP32mpsAcceleratedPassedFastmacos · aarch641505 ms / 5577 ms
FP32mpsAcceleratedPassedModeratemacos · aarch642034 ms / 0 ms
FP32mpsAcceleratedPassedModeratemacos · aarch642051 ms / 0 ms
FP32mpsAcceleratedPassedModeratemacos · aarch642053 ms / 0 ms
FP32mpsAcceleratedPassedModeratemacos · aarch642057 ms / 0 ms
FP32mpsAcceleratedPassedModeratemacos · aarch642057 ms / 0 ms
FP32mpsAcceleratedPassedModeratemacos · aarch642106 ms / 0 ms
FP32mpsAcceleratedPassedModeratemacos · aarch642171 ms / 0 ms
FP32mpsAcceleratedPassedModeratemacos · aarch642311 ms / 1164 ms
FP32mpsAcceleratedPassedModeratemacos · aarch642316 ms / 1171 ms
FP32mpsAcceleratedPassedModeratemacos · aarch642580 ms / 0 ms
FP32mpsAcceleratedPassedModeratemacos · aarch644522 ms / 1216 ms
FP32mpsAcceleratedPassedSlowmacos · aarch642433 ms / 5577 ms
FP32mpsAcceleratedPassedVery slowmacos · aarch642257 ms / 5577 ms
FP32mpsAcceleratedPassedVery slowmacos · aarch642565 ms / 5577 ms
FP32cpuPassedFastmacos · aarch64269 ms / 4074 ms
FP32cpuPassedFastmacos · aarch643109 ms / 4074 ms
FP32cpuPassedFastmacos · aarch643172 ms / 4074 ms
FP32cpuPassedModeratemacos · aarch643792 ms / 0 ms
FP32cpuPassedModeratemacos · aarch643819 ms / 0 ms
FP32cpuPassedModeratemacos · aarch643842 ms / 0 ms
FP32cpuPassedModeratemacos · aarch643874 ms / 0 ms
FP32cpuPassedModeratemacos · aarch643876 ms / 0 ms
FP32cpuPassedModeratemacos · aarch643895 ms / 0 ms
FP32cpuPassedModeratemacos · aarch643966 ms / 0 ms
FP32cpuPassedModeratemacos · aarch643971 ms / 492 ms
FP32cpuPassedModeratemacos · aarch643979 ms / 521 ms
FP32cpuPassedModeratemacos · aarch644130 ms / 0 ms
FP32cpuPassedModeratemacos · aarch644135 ms / 600 ms
FP32cpuPassedSlowmacos · aarch644440 ms / 4074 ms
FP32cpuPassedSlowmacos · aarch647702 ms / 0 ms
FP32cpuPassedSlowmacos · aarch647745 ms / 0 ms
FP32cpuPassedSlowmacos · aarch647793 ms / 0 ms
FP32cpuPassedSlowmacos · aarch647801 ms / 0 ms
FP32cpuPassedSlowmacos · aarch647834 ms / 0 ms
FP32cpuPassedSlowmacos · aarch647871 ms / 0 ms
FP32cpuPassedSlowmacos · aarch647883 ms / 0 ms
FP32cpuPassedSlowmacos · aarch647904 ms / 0 ms
FP32cpuPassedSlowmacos · aarch647993 ms / 554 ms
FP32cpuPassedSlowmacos · aarch648033 ms / 637 ms
FP32cpuPassedSlowmacos · aarch648062 ms / 587 ms
FP32cpuPassedVery slowmacos · aarch644109 ms / 4074 ms
FP32cpuPassedVery slowmacos · aarch644350 ms / 4074 ms
Test setup & sample details
Quality gate
No explicit quality gate recorded. Review the per-case quality evidence; full certification is pending.
Tested
2026-11-01T22:56:20.265785+00:00
Environment
{ "os": "macos", "arch": "aarch64", "hardware": "Apple M3 Max", "osVersion": "27.2", "python": "App-owned CPython3.13.18 private model probe venv" }
Runtime
transformers 5.22.1
Artifact revision
89644892e4d85e24eaac8bacfd4f463576704203
Fixture
{
  "file": "model-samples/desktop/mobilenet-dog.jpg",
  "size": 83281,
  "sha1": "f1d042827c2f2154a6d8043b8e8f07c1be612941",
  "sha256": "d4cb12ea8a768582562e1f1c0c930ea6fc4757f5b66551b986b41498efd75ca1",
  "url": "https://assets.localai.tools/samples/v1/files/d4cb12ea8a768582562e1f1c0c930ea6fc4757f5b66551b986b41498efd75ca1/mobilenet-dog.jpg"
}
Rerun when
model revision/checksum; pinned runtime/processor/attention/backend/driver; fixture checksum/question/expected answers; visual pixel and token budgets; quality gates
Per-case quality & failure details
[
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "repetitive": false,
      "eosWithinBudget": true,
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ]
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 636.9023750012275,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 8075.678832996346,
      "inferenceMs": 8033.141791995149,
      "tokensPerSecond": 0.2489685918394913,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 1,
      "totalMs": 8712.581207997573,
      "artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
      "fallback": false,
      "grade": "slow"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "repetitive": false,
      "eosWithinBudget": true,
      "expected": [
        "localai2026"
      ]
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 7812.195541999245,
      "inferenceMs": 7801.169833997847,
      "tokensPerSecond": 1.0254872243821231,
      "inputTokens": 791,
      "outputTokens": 8,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 768,
      "inputHeight": 256,
      "visionGrid": [
        [
          1,
          32,
          96
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 1,
      "totalMs": 7812.195541999245,
      "artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
      "fallback": false,
      "grade": "slow"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "repetitive": false,
      "eosWithinBudget": true,
      "expected": [
        "3",
        "three"
      ]
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 7713.636000000406,
      "inferenceMs": 7702.218833997904,
      "tokensPerSecond": 0.2596654344812847,
      "inputTokens": 824,
      "outputTokens": 2,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 640,
      "inputHeight": 320,
      "visionGrid": [
        [
          1,
          40,
          80
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 1,
      "totalMs": 7713.636000000406,
      "artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
      "fallback": false,
      "grade": "slow"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "repetitive": false,
      "eosWithinBudget": true,
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ]
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 7896.8052909986,
      "inferenceMs": 7882.819667000149,
      "tokensPerSecond": 0.2537163203634609,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 1,
      "totalMs": 7896.8052909986,
      "artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
      "fallback": false,
      "grade": "slow"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "repetitive": false,
      "eosWithinBudget": true,
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ]
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 7916.936333000194,
      "inferenceMs": 7903.680291005003,
      "tokensPerSecond": 0.2530466727349984,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 1,
      "totalMs": 7916.936333000194,
      "artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
      "fallback": false,
      "grade": "slow"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "repetitive": false,
      "eosWithinBudget": true,
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ]
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 554.4171250003274,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 8024.448666998069,
      "inferenceMs": 7992.687541001942,
      "tokensPerSecond": 0.250228723410009,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 1,
      "totalMs": 8578.865791998396,
      "artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
      "fallback": false,
      "grade": "slow"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "repetitive": false,
      "eosWithinBudget": true,
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ]
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 7885.141166996618,
      "inferenceMs": 7870.676416998322,
      "tokensPerSecond": 0.2541077658434279,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 1,
      "totalMs": 7885.141166996618,
      "artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
      "fallback": false,
      "grade": "slow"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "repetitive": false,
      "eosWithinBudget": true,
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ]
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 7847.544374999416,
      "inferenceMs": 7834.322167000209,
      "tokensPerSecond": 0.2552869230249957,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 1,
      "totalMs": 7847.544374999416,
      "artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
      "fallback": false,
      "grade": "slow"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "repetitive": false,
      "eosWithinBudget": true,
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ]
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 586.6140419966541,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 8093.579958003829,
      "inferenceMs": 8061.844624993682,
      "tokensPerSecond": 0.2480821813161113,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 1,
      "totalMs": 8680.194000000483,
      "artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
      "fallback": false,
      "grade": "slow"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "repetitive": false,
      "eosWithinBudget": true,
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ]
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 7806.9487499960815,
      "inferenceMs": 7793.158833002963,
      "tokensPerSecond": 0.25663534426249257,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 1,
      "totalMs": 7806.9487499960815,
      "artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
      "fallback": false,
      "grade": "slow"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "repetitive": false,
      "eosWithinBudget": true,
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ]
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 7758.417208002356,
      "inferenceMs": 7745.325916999718,
      "tokensPerSecond": 0.25822025069472265,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 1,
      "totalMs": 7758.417208002356,
      "artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
      "fallback": false,
      "grade": "slow"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "repetitive": false,
      "eosWithinBudget": true,
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ]
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 492.23937500210013,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 3995.2576249997946,
      "inferenceMs": 3971.08383399609,
      "tokensPerSecond": 0.5036408405378352,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 4487.497000001895,
      "artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
      "fallback": false,
      "grade": "moderate"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "repetitive": false,
      "eosWithinBudget": true,
      "expected": [
        "localai2026"
      ]
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 4135.194957998465,
      "inferenceMs": 4130.113082996104,
      "tokensPerSecond": 1.9369929682885505,
      "inputTokens": 791,
      "outputTokens": 8,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 768,
      "inputHeight": 256,
      "visionGrid": [
        [
          1,
          32,
          96
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 4135.194957998465,
      "artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
      "fallback": false,
      "grade": "moderate"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "repetitive": false,
      "eosWithinBudget": true,
      "expected": [
        "3",
        "three"
      ]
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 3846.928916995239,
      "inferenceMs": 3841.9710000016494,
      "tokensPerSecond": 0.5205661364958615,
      "inputTokens": 824,
      "outputTokens": 2,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 640,
      "inputHeight": 320,
      "visionGrid": [
        [
          1,
          40,
          80
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 3846.928916995239,
      "artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
      "fallback": false,
      "grade": "moderate"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "repetitive": false,
      "eosWithinBudget": true,
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ]
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 3798.2420839980477,
      "inferenceMs": 3792.3042500042357,
      "tokensPerSecond": 0.5273838458498593,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 3798.2420839980477,
      "artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
      "fallback": false,
      "grade": "moderate"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "repetitive": false,
      "eosWithinBudget": true,
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ]
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 3824.5153750031022,
      "inferenceMs": 3818.551166994439,
      "tokensPerSecond": 0.5237588584086433,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 3824.5153750031022,
      "artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
      "fallback": false,
      "grade": "moderate"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "repetitive": false,
      "eosWithinBudget": true,
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ]
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 520.9806249986286,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 4003.420166001888,
      "inferenceMs": 3978.965583999525,
      "tokensPerSecond": 0.502643201550305,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 4524.400791000517,
      "artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
      "fallback": false,
      "grade": "moderate"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "repetitive": false,
      "eosWithinBudget": true,
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ]
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 3901.0117080033524,
      "inferenceMs": 3894.970082998043,
      "tokensPerSecond": 0.5134827629948204,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 3901.0117080033524,
      "artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
      "fallback": false,
      "grade": "moderate"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "repetitive": false,
      "eosWithinBudget": true,
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ]
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 3881.6745840013027,
      "inferenceMs": 3875.833917001728,
      "tokensPerSecond": 0.5160179829240883,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 3881.6745840013027,
      "artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
      "fallback": false,
      "grade": "moderate"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "repetitive": false,
      "eosWithinBudget": true,
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ]
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 600.1735420068144,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 4163.385208994441,
      "inferenceMs": 4135.028584001702,
      "tokensPerSecond": 0.4836725936400871,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 4763.558751001256,
      "artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
      "fallback": false,
      "grade": "moderate"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "repetitive": false,
      "eosWithinBudget": true,
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ]
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 3972.2970000002533,
      "inferenceMs": 3966.473833002965,
      "tokensPerSecond": 0.5042261928867501,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 3972.2970000002533,
      "artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
      "fallback": false,
      "grade": "moderate"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "repetitive": false,
      "eosWithinBudget": true,
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ]
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 3878.9564589969814,
      "inferenceMs": 3873.626917004003,
      "tokensPerSecond": 0.5163119843112999,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 3878.9564589969814,
      "artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
      "fallback": false,
      "grade": "moderate"
    }
  },
  {
    "precision": "fp32",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "repetitive": false,
      "eosWithinBudget": true,
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ]
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1215.6197090007481,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 4551.082375000988,
      "inferenceMs": 4522.304082995106,
      "tokensPerSecond": 0.4422524366551236,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 11734597632,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 5766.702084001736,
      "artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
      "fallback": false,
      "grade": "moderate"
    }
  },
  {
    "precision": "fp32",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "repetitive": false,
      "eosWithinBudget": true,
      "expected": [
        "localai2026"
      ]
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 2587.805874994956,
      "inferenceMs": 2579.6897920008632,
      "tokensPerSecond": 3.1011480623780843,
      "inputTokens": 791,
      "outputTokens": 8,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 768,
      "inputHeight": 256,
      "visionGrid": [
        [
          1,
          32,
          96
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 13665746944,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 2587.805874994956,
      "artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
      "fallback": false,
      "grade": "moderate"
    }
  },
  {
    "precision": "fp32",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "repetitive": false,
      "eosWithinBudget": true,
      "expected": [
        "3",
        "three"
      ]
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 2179.3269169938867,
      "inferenceMs": 2170.8504580019508,
      "tokensPerSecond": 0.9212979146618873,
      "inputTokens": 824,
      "outputTokens": 2,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 640,
      "inputHeight": 320,
      "visionGrid": [
        [
          1,
          40,
          80
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 15760506880,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 2179.3269169938867,
      "artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
      "fallback": false,
      "grade": "moderate"
    }
  },
  {
    "precision": "fp32",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "repetitive": false,
      "eosWithinBudget": true,
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ]
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 2115.038416995958,
      "inferenceMs": 2106.0563749997527,
      "tokensPerSecond": 0.9496421955942347,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 16427401216,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 2115.038416995958,
      "artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
      "fallback": false,
      "grade": "moderate"
    }
  },
  {
    "precision": "fp32",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "repetitive": false,
      "eosWithinBudget": true,
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ]
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 2066.274583004997,
      "inferenceMs": 2056.9934170052875,
      "tokensPerSecond": 0.9722928539614568,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 16427401216,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 2066.274583004997,
      "artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
      "fallback": false,
      "grade": "moderate"
    }
  },
  {
    "precision": "fp32",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "repetitive": false,
      "eosWithinBudget": true,
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ]
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1164.4891250034561,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 2338.0629169987515,
      "inferenceMs": 2310.9830410030554,
      "tokensPerSecond": 0.8654325732879127,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 11734597632,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 3502.5520420022076,
      "artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
      "fallback": false,
      "grade": "moderate"
    }
  },
  {
    "precision": "fp32",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "repetitive": false,
      "eosWithinBudget": true,
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ]
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 2065.581499999098,
      "inferenceMs": 2056.772625001031,
      "tokensPerSecond": 0.9723972284000024,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 12401491968,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 2065.581499999098,
      "artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
      "fallback": false,
      "grade": "moderate"
    }
  },
  {
    "precision": "fp32",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "repetitive": false,
      "eosWithinBudget": true,
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ]
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 2060.4178329958813,
      "inferenceMs": 2050.706417001493,
      "tokensPerSecond": 0.9752736829703615,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 12401491968,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 2060.4178329958813,
      "artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
      "fallback": false,
      "grade": "moderate"
    }
  },
  {
    "precision": "fp32",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "repetitive": false,
      "eosWithinBudget": true,
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ]
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1170.8507910007029,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 2343.7331249951967,
      "inferenceMs": 2315.727458000765,
      "tokensPerSecond": 0.8636594920054445,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 11734597632,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 3514.5839159958996,
      "artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
      "fallback": false,
      "grade": "moderate"
    }
  },
  {
    "precision": "fp32",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "repetitive": false,
      "eosWithinBudget": true,
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ]
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 2063.3216659989557,
      "inferenceMs": 2053.3383330039214,
      "tokensPerSecond": 0.9740236023715145,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 12401491968,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 2063.3216659989557,
      "artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
      "fallback": false,
      "grade": "moderate"
    }
  },
  {
    "precision": "fp32",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "repetitive": false,
      "eosWithinBudget": true,
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ]
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 2042.1250840008724,
      "inferenceMs": 2033.7423340024543,
      "tokensPerSecond": 0.983408746802232,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 12401491968,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 2042.1250840008724,
      "artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
      "fallback": false,
      "grade": "moderate"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 4388.4951249929145,
      "inferenceMs": 4350.370042026043,
      "tokensPerSecond": 0.45973100694408175,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "placement": {
        "cpu/torch.float32": 625
      },
      "decoding": "greedy",
      "outputLimitReached": false,
      "loadMs": 4073.553374968469,
      "totalMs": 8468.864374968689,
      "wallMs": 4394.699833006598,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
      "device": "cpu",
      "precision": "fp32",
      "requestedDevice": "cpu",
      "requestedPrecision": "fp32",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "fallback": false,
      "parameterDtype": "torch.float32",
      "grade": "very-slow"
    },
    "failure": null
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 4447.265790950041,
      "inferenceMs": 4440.3853340190835,
      "tokensPerSecond": 1.8016454425046569,
      "inputTokens": 791,
      "outputTokens": 8,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 768,
      "inputHeight": 256,
      "visionGrid": [
        [
          1,
          32,
          96
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "placement": {
        "cpu/torch.float32": 625
      },
      "decoding": "greedy",
      "outputLimitReached": false,
      "loadMs": 4073.553374968469,
      "totalMs": 12917.885833012406,
      "wallMs": 4448.937541979831,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
      "device": "cpu",
      "precision": "fp32",
      "requestedDevice": "cpu",
      "requestedPrecision": "fp32",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "fallback": false,
      "parameterDtype": "torch.float32",
      "grade": "slow"
    },
    "failure": null
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 4116.308333002962,
      "inferenceMs": 4109.459625033196,
      "tokensPerSecond": 0.4866819928870439,
      "inputTokens": 824,
      "outputTokens": 2,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 640,
      "inputHeight": 320,
      "visionGrid": [
        [
          1,
          40,
          80
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "placement": {
        "cpu/torch.float32": 625
      },
      "decoding": "greedy",
      "outputLimitReached": false,
      "loadMs": 4073.553374968469,
      "totalMs": 17035.700124979485,
      "wallMs": 4117.6878749975,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
      "device": "cpu",
      "precision": "fp32",
      "requestedDevice": "cpu",
      "requestedPrecision": "fp32",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "fallback": false,
      "parameterDtype": "torch.float32",
      "grade": "very-slow"
    },
    "failure": null
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 3180.0782500067726,
      "inferenceMs": 3171.657375001814,
      "tokensPerSecond": 5.675266232056262,
      "inputTokens": 426,
      "outputTokens": 18,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 320,
      "inputHeight": 240,
      "visionGrid": [
        [
          2,
          24,
          32
        ]
      ],
      "inputModality": "video",
      "videoFrameCount": 4,
      "videoFps": 1,
      "placement": {
        "cpu/torch.float32": 625
      },
      "decoding": "greedy",
      "outputLimitReached": false,
      "loadMs": 4073.553374968469,
      "totalMs": 20217.279249976855,
      "wallMs": 3181.5036250045523,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
      "device": "cpu",
      "precision": "fp32",
      "requestedDevice": "cpu",
      "requestedPrecision": "fp32",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "fallback": false,
      "parameterDtype": "torch.float32",
      "grade": "good"
    },
    "failure": null
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 3116.1493330146186,
      "inferenceMs": 3108.579166990239,
      "tokensPerSecond": 5.790426761891929,
      "inputTokens": 426,
      "outputTokens": 18,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 320,
      "inputHeight": 240,
      "visionGrid": [
        [
          2,
          24,
          32
        ]
      ],
      "inputModality": "video",
      "videoFrameCount": 4,
      "videoFps": 1,
      "placement": {
        "cpu/torch.float32": 625
      },
      "decoding": "greedy",
      "outputLimitReached": false,
      "loadMs": 4073.553374968469,
      "totalMs": 23334.819875017274,
      "wallMs": 3117.4743750016205,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
      "device": "cpu",
      "precision": "fp32",
      "requestedDevice": "cpu",
      "requestedPrecision": "fp32",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "fallback": false,
      "parameterDtype": "torch.float32",
      "grade": "good"
    },
    "failure": null
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 269.33770801406354,
      "inferenceMs": 268.63662496907637,
      "tokensPerSecond": 7.445001217649405,
      "inputTokens": 21,
      "outputTokens": 2,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": null,
      "inputHeight": null,
      "visionGrid": null,
      "inputModality": "text",
      "videoFrameCount": 0,
      "videoFps": null,
      "placement": {
        "cpu/torch.float32": 625
      },
      "decoding": "greedy",
      "outputLimitReached": false,
      "loadMs": 4073.553374968469,
      "totalMs": 23605.354583007284,
      "wallMs": 270.4772499855608,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
      "device": "cpu",
      "precision": "fp32",
      "requestedDevice": "cpu",
      "requestedPrecision": "fp32",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "fallback": false,
      "parameterDtype": "torch.float32",
      "grade": "good"
    },
    "failure": null
  },
  {
    "precision": "fp32",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 2607.4020829983056,
      "inferenceMs": 2565.0439590099268,
      "tokensPerSecond": 0.77971373277048,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "placement": {
        "mps:0/torch.float32": 625
      },
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 11734597632,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "loadMs": 5577.300457982346,
      "totalMs": 8192.509624990635,
      "wallMs": 2614.388332993258,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
      "device": "mps",
      "precision": "fp32",
      "requestedDevice": "mps",
      "requestedPrecision": "fp32",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "fallback": false,
      "parameterDtype": "torch.float32",
      "grade": "very-slow"
    },
    "failure": null
  },
  {
    "precision": "fp32",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 2443.650291999802,
      "inferenceMs": 2432.5532920192927,
      "tokensPerSecond": 3.2887254829098116,
      "inputTokens": 791,
      "outputTokens": 8,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 768,
      "inputHeight": 256,
      "visionGrid": [
        [
          1,
          32,
          96
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "placement": {
        "mps:0/torch.float32": 625
      },
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 13665746944,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "loadMs": 5577.300457982346,
      "totalMs": 10638.538207975216,
      "wallMs": 2445.9031670121476,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
      "device": "mps",
      "precision": "fp32",
      "requestedDevice": "mps",
      "requestedPrecision": "fp32",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "fallback": false,
      "parameterDtype": "torch.float32",
      "grade": "slow"
    },
    "failure": null
  },
  {
    "precision": "fp32",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 2265.8370000426657,
      "inferenceMs": 2256.543666997459,
      "tokensPerSecond": 0.8863112330820462,
      "inputTokens": 824,
      "outputTokens": 2,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 640,
      "inputHeight": 320,
      "visionGrid": [
        [
          1,
          40,
          80
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "placement": {
        "mps:0/torch.float32": 625
      },
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 15760506880,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "loadMs": 5577.300457982346,
      "totalMs": 12906.167624983937,
      "wallMs": 2267.510832985863,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
      "device": "mps",
      "precision": "fp32",
      "requestedDevice": "mps",
      "requestedPrecision": "fp32",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "fallback": false,
      "parameterDtype": "torch.float32",
      "grade": "very-slow"
    },
    "failure": null
  },
  {
    "precision": "fp32",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 1517.6930840243585,
      "inferenceMs": 1504.8609169898555,
      "tokensPerSecond": 11.961238275763753,
      "inputTokens": 426,
      "outputTokens": 18,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 320,
      "inputHeight": 240,
      "visionGrid": [
        [
          2,
          24,
          32
        ]
      ],
      "inputModality": "video",
      "videoFrameCount": 4,
      "videoFps": 1,
      "placement": {
        "mps:0/torch.float32": 625
      },
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 15879962624,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "loadMs": 5577.300457982346,
      "totalMs": 14425.565124955028,
      "wallMs": 1519.3294169730507,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
      "device": "mps",
      "precision": "fp32",
      "requestedDevice": "mps",
      "requestedPrecision": "fp32",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "fallback": false,
      "parameterDtype": "torch.float32",
      "grade": "good"
    },
    "failure": null
  },
  {
    "precision": "fp32",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 1424.7224580030888,
      "inferenceMs": 1414.5067500066943,
      "tokensPerSecond": 12.725283919581729,
      "inputTokens": 426,
      "outputTokens": 18,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 320,
      "inputHeight": 240,
      "visionGrid": [
        [
          2,
          24,
          32
        ]
      ],
      "inputModality": "video",
      "videoFrameCount": 4,
      "videoFps": 1,
      "placement": {
        "mps:0/torch.float32": 625
      },
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 15879962624,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "loadMs": 5577.300457982346,
      "totalMs": 15852.031165966764,
      "wallMs": 1426.3948749867268,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
      "device": "mps",
      "precision": "fp32",
      "requestedDevice": "mps",
      "requestedPrecision": "fp32",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "fallback": false,
      "parameterDtype": "torch.float32",
      "grade": "good"
    },
    "failure": null
  },
  {
    "precision": "fp32",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 164.1484170104377,
      "inferenceMs": 162.03762497752905,
      "tokensPerSecond": 12.342812357792548,
      "inputTokens": 21,
      "outputTokens": 2,
      "maxNewTokens": 64,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": null,
      "inputHeight": null,
      "visionGrid": null,
      "inputModality": "text",
      "videoFrameCount": 0,
      "videoFps": null,
      "placement": {
        "mps:0/torch.float32": 625
      },
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 15896854528,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "loadMs": 5577.300457982346,
      "totalMs": 16017.471041006502,
      "wallMs": 165.37175001576543,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "89644892e4d85e24eaac8bacfd4f463576704203",
      "device": "mps",
      "precision": "fp32",
      "requestedDevice": "mps",
      "requestedPrecision": "fp32",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "fallback": false,
      "parameterDtype": "torch.float32",
      "grade": "good"
    },
    "failure": null
  }
]

Qwen3-VL 2B Thinking — Text, Image

42 configurations tested
Passed42
Failed0
TestedNov 6, 2026
Runtimetransformers 5.22.1

Quality criteria

{ "method": "Final-answer fixture semantic/normalized OCR plus completion/repetition/speed gates frozen in research/variant-onboarding/qwen3-vl-family/PLAN.md", "fixtures": [ "dog", "ocr", "circles", "arithmetic", "instruction" ] }

PrecisionDeviceResultPlatformInference / load
BF16mpsAcceleratedPassedFastmacos · aarch643100 ms / 2969 ms
BF16mpsAcceleratedPassedFastmacos · aarch643209 ms / 2690 ms
BF16mpsAcceleratedPassedFastmacos · aarch643212 ms / 2682 ms
BF16mpsAcceleratedPassedFastmacos · aarch643271 ms / 2682 ms
BF16mpsAcceleratedPassedFastmacos · aarch643277 ms / 2682 ms
BF16mpsAcceleratedPassedFastmacos · aarch643479 ms / 2690 ms
BF16mpsAcceleratedPassedFastmacos · aarch643493 ms / 2682 ms
BF16mpsAcceleratedPassedFastmacos · aarch643503 ms / 2969 ms
BF16mpsAcceleratedPassedFastmacos · aarch643611 ms / 2690 ms
BF16mpsAcceleratedPassedFastmacos · aarch643624 ms / 2969 ms
BF16mpsAcceleratedPassedFastmacos · aarch643636 ms / 2690 ms
BF16mpsAcceleratedPassedFastmacos · aarch643677 ms / 2682 ms
BF16mpsAcceleratedPassedFastmacos · aarch643688 ms / 2690 ms
BF16mpsAcceleratedPassedFastmacos · aarch644052 ms / 2969 ms
BF16mpsAcceleratedPassedFastmacos · aarch644537 ms / 2969 ms
BF16mpsAcceleratedPassedFastmacos · aarch645421 ms / 2969 ms
BF16mpsAcceleratedPassedFastmacos · aarch645654 ms / 2690 ms
BF16mpsAcceleratedPassedFastmacos · aarch645660 ms / 2682 ms
FP32cpuPassedFastmacos · aarch647855 ms / 8341 ms
FP32cpuPassedFastmacos · aarch648828 ms / 3022 ms
FP32cpuPassedFastmacos · aarch649656 ms / 3022 ms
FP32cpuPassedFastmacos · aarch649703 ms / 5031 ms
FP32cpuPassedModeratemacos · aarch6410773 ms / 5031 ms
FP32cpuPassedModeratemacos · aarch6410818 ms / 8341 ms
FP32cpuPassedModeratemacos · aarch6410988 ms / 8341 ms
FP32cpuPassedModeratemacos · aarch6411225 ms / 5031 ms
FP32cpuPassedModeratemacos · aarch6411906 ms / 3022 ms
FP32cpuPassedModeratemacos · aarch6412038 ms / 3022 ms
FP32cpuPassedModeratemacos · aarch6412123 ms / 8341 ms
FP32cpuPassedModeratemacos · aarch6412276 ms / 2296 ms
FP32cpuPassedModeratemacos · aarch6412396 ms / 2296 ms
FP32cpuPassedModeratemacos · aarch6412810 ms / 2296 ms
FP32cpuPassedModeratemacos · aarch6413093 ms / 8341 ms
FP32cpuPassedModeratemacos · aarch6413476 ms / 3022 ms
FP32cpuPassedModeratemacos · aarch6413521 ms / 5031 ms
FP32cpuPassedModeratemacos · aarch6414793 ms / 2296 ms
FP32cpuPassedModeratemacos · aarch6414916 ms / 2296 ms
FP32cpuPassedModeratemacos · aarch6416267 ms / 2296 ms
FP32cpuPassedModeratemacos · aarch6416311 ms / 5031 ms
FP32cpuPassedModeratemacos · aarch6418659 ms / 3022 ms
FP32cpuPassedModeratemacos · aarch6419973 ms / 8341 ms
FP32cpuPassedModeratemacos · aarch6421395 ms / 5031 ms
Test setup & sample details
Quality gate
{ "method": "Final-answer fixture semantic/normalized OCR plus completion/repetition/speed gates frozen in research/variant-onboarding/qwen3-vl-family/PLAN.md", "fixtures": [ "dog", "ocr", "circles", "arithmetic", "instruction" ] }
Tested
2026-11-06T16:23:55.435304+00:00
Environment
{ "os": "macos", "arch": "aarch64", "hardware": "Apple M3 Max", "osVersion": "27.2", "python": "App-owned CPython3.13.18 private model probe venv" }
Runtime
transformers 5.22.1
Artifact revision
33e0ad94c327808ebc7d3bbc97e78e600f1eeecd
Fixture
{
  "file": "model-samples/desktop/qwen3-vl-family/Qwen3-VL-2B-Thinking-dog-input.json",
  "url": "https://assets.localai.tools/samples/v1/files/7b3ce826dfe3c11238f8ed74eb4c0376337fae452991b11c6e9c5b322ca981f0/Qwen3-VL-2B-Thinking-dog-input.json",
  "size": 609,
  "sha1": "2e345d8ba592df6216208fae8065b18c3a4f6e41",
  "sha256": "7b3ce826dfe3c11238f8ed74eb4c0376337fae452991b11c6e9c5b322ca981f0"
}
Rerun when
model revision/checksum; pinned runtime/processor/attention/backend/driver; fixture checksum/question/expected answers; visual pixel and token budgets; quality gates
Per-case quality & failure details
[
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "dog",
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 120,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3022.400458001357,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 8840.984792001109,
      "inferenceMs": 8828.372374999162,
      "tokensPerSecond": 5.097202302820204,
      "inputTokens": 828,
      "outputTokens": 45,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "requestMs": 8847.723875001975,
      "totalMs": 11870.124333003332,
      "artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
      "precision": "fp32",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.11310957133471727,
      "grade": "good",
      "residentMemoryBytes": 9283813376,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "LOCAL AI 2026",
      "expected": [
        "localai2026"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 120,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3022.400458001357,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 12044.997207998676,
      "inferenceMs": 12037.956375002977,
      "tokensPerSecond": 6.313363965815269,
      "inputTokens": 793,
      "outputTokens": 76,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 768,
      "inputHeight": 256,
      "visionGrid": [
        [
          1,
          32,
          96
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "requestMs": 12046.859917001711,
      "totalMs": 15069.260375003068,
      "artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
      "precision": "fp32",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.08302202007451971,
      "grade": "moderate",
      "residentMemoryBytes": 9281355776,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "3",
      "expected": [
        "3",
        "three"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 120,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3022.400458001357,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 11912.995750000846,
      "inferenceMs": 11905.521249998856,
      "tokensPerSecond": 6.467587464934171,
      "inputTokens": 826,
      "outputTokens": 77,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 640,
      "inputHeight": 320,
      "visionGrid": [
        [
          1,
          40,
          80
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "requestMs": 11914.513666000857,
      "totalMs": 14936.914124002215,
      "artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
      "precision": "fp32",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.08394194214330422,
      "grade": "moderate",
      "residentMemoryBytes": 9314451456,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "17 + 25 = 42\n\nAnswer: 42",
      "expected": [
        "42"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 120,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3022.400458001357,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 18659.51454199967,
      "inferenceMs": 18658.54599999875,
      "tokensPerSecond": 10.826138328249883,
      "inputTokens": 24,
      "outputTokens": 202,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": null,
      "inputHeight": null,
      "visionGrid": null,
      "inputModality": "text",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "requestMs": 18660.8586250004,
      "totalMs": 21683.259083001758,
      "artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
      "precision": "fp32",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.053591962306905425,
      "grade": "moderate",
      "residentMemoryBytes": 9392455680,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "READY",
      "expected": [
        "ready"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 120,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3022.400458001357,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 13476.740750000317,
      "inferenceMs": 13475.579833000666,
      "tokensPerSecond": 11.279663055964658,
      "inputTokens": 17,
      "outputTokens": 152,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": null,
      "inputHeight": null,
      "visionGrid": null,
      "inputModality": "text",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "requestMs": 13477.901749996818,
      "totalMs": 16500.302207998175,
      "artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
      "precision": "fp32",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.0742019171066993,
      "grade": "moderate",
      "residentMemoryBytes": 9393029120,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "dog",
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 120,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3022.400458001357,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 9667.7203750005,
      "inferenceMs": 9655.712499999936,
      "tokensPerSecond": 4.660453591591537,
      "inputTokens": 828,
      "outputTokens": 45,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "requestMs": 9674.303040999803,
      "totalMs": 12696.70349900116,
      "artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
      "precision": "fp32",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.10343700078312912,
      "grade": "good",
      "residentMemoryBytes": 9398976512,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "dog",
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 120,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 5031.187874999887,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 9716.61687500091,
      "inferenceMs": 9703.423999999359,
      "tokensPerSecond": 4.637538254538086,
      "inputTokens": 828,
      "outputTokens": 45,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "requestMs": 9723.536583001987,
      "totalMs": 14754.724458001874,
      "artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
      "precision": "fp32",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.10291647935330438,
      "grade": "good",
      "residentMemoryBytes": 9282633728,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "LOCAL AI 2026",
      "expected": [
        "localai2026"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 120,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 5031.187874999887,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 10780.18399999928,
      "inferenceMs": 10773.396625001624,
      "tokensPerSecond": 7.054414001952568,
      "inputTokens": 793,
      "outputTokens": 76,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 768,
      "inputHeight": 256,
      "visionGrid": [
        [
          1,
          32,
          96
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "requestMs": 10781.960750002327,
      "totalMs": 15813.148625002214,
      "artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
      "precision": "fp32",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.0927627951433915,
      "grade": "moderate",
      "residentMemoryBytes": 9280864256,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "3",
      "expected": [
        "3",
        "three"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 120,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 5031.187874999887,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 11231.309457998577,
      "inferenceMs": 11224.980125000002,
      "tokensPerSecond": 6.859700341785682,
      "inputTokens": 826,
      "outputTokens": 77,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 640,
      "inputHeight": 320,
      "visionGrid": [
        [
          1,
          40,
          80
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "requestMs": 11232.843707999564,
      "totalMs": 16264.03158299945,
      "artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
      "precision": "fp32",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.08903681300383297,
      "grade": "moderate",
      "residentMemoryBytes": 9313845248,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "17 + 25 = 42\n\nAnswer: 42",
      "expected": [
        "42"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 120,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 5031.187874999887,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 21395.42891599922,
      "inferenceMs": 21394.57579199734,
      "tokensPerSecond": 9.441645488271764,
      "inputTokens": 24,
      "outputTokens": 202,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": null,
      "inputHeight": null,
      "visionGrid": null,
      "inputModality": "text",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "requestMs": 21396.755958998256,
      "totalMs": 26427.943833998143,
      "artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
      "precision": "fp32",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.04673895549961203,
      "grade": "moderate",
      "residentMemoryBytes": 9404366848,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "READY",
      "expected": [
        "ready"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 120,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 5031.187874999887,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 16311.731042002066,
      "inferenceMs": 16310.550500002137,
      "tokensPerSecond": 9.319121387103403,
      "inputTokens": 17,
      "outputTokens": 152,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": null,
      "inputHeight": null,
      "visionGrid": null,
      "inputModality": "text",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "requestMs": 16313.402583000425,
      "totalMs": 21344.59045800031,
      "artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
      "precision": "fp32",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.061305571887192066,
      "grade": "moderate",
      "residentMemoryBytes": 9372532736,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "dog",
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 120,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 5031.187874999887,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 13536.24729099829,
      "inferenceMs": 13520.908292000968,
      "tokensPerSecond": 3.3281787752840692,
      "inputTokens": 828,
      "outputTokens": 45,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "requestMs": 13544.194375001098,
      "totalMs": 18575.382250000985,
      "artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
      "precision": "fp32",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.07387571891250892,
      "grade": "moderate",
      "residentMemoryBytes": 9059581952,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "dog",
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 120,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 8340.832292000414,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 12136.681249998219,
      "inferenceMs": 12123.492708997219,
      "tokensPerSecond": 3.7118016301196857,
      "inputTokens": 828,
      "outputTokens": 45,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "requestMs": 12143.483416999516,
      "totalMs": 20484.31570899993,
      "artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
      "precision": "fp32",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.08239484743822754,
      "grade": "moderate",
      "residentMemoryBytes": 9071034368,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "LOCAL AI 2026",
      "expected": [
        "localai2026"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 120,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 8340.832292000414,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 10824.734625002748,
      "inferenceMs": 10817.748624998785,
      "tokensPerSecond": 7.025491406259086,
      "inputTokens": 793,
      "outputTokens": 76,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 768,
      "inputHeight": 256,
      "visionGrid": [
        [
          1,
          32,
          96
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "requestMs": 10826.52620799854,
      "totalMs": 19167.358499998954,
      "artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
      "precision": "fp32",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.092381017608526,
      "grade": "moderate",
      "residentMemoryBytes": 9069838336,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "3",
      "expected": [
        "3",
        "three"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 120,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 8340.832292000414,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 10994.872791998205,
      "inferenceMs": 10988.445499999216,
      "tokensPerSecond": 7.007360595273052,
      "inputTokens": 826,
      "outputTokens": 77,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 640,
      "inputHeight": 320,
      "visionGrid": [
        [
          1,
          40,
          80
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "requestMs": 10996.290833001694,
      "totalMs": 19337.12312500211,
      "artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
      "precision": "fp32",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.09095148428891102,
      "grade": "moderate",
      "residentMemoryBytes": 9102917632,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "17 + 25 = 42\n\nAnswer: 42",
      "expected": [
        "42"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 120,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 8340.832292000414,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 19973.961916999542,
      "inferenceMs": 19972.949457998766,
      "tokensPerSecond": 10.113679024962588,
      "inputTokens": 24,
      "outputTokens": 202,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": null,
      "inputHeight": null,
      "visionGrid": null,
      "inputModality": "text",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "requestMs": 19975.131375002093,
      "totalMs": 28315.963667002507,
      "artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
      "precision": "fp32",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.050065180065699176,
      "grade": "moderate",
      "residentMemoryBytes": 9198403584,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "READY",
      "expected": [
        "ready"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 120,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 8340.832292000414,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 13093.458666997321,
      "inferenceMs": 13092.54612499717,
      "tokensPerSecond": 11.609659309108055,
      "inputTokens": 17,
      "outputTokens": 152,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": null,
      "inputHeight": null,
      "visionGrid": null,
      "inputModality": "text",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "requestMs": 13095.176916998753,
      "totalMs": 21436.009208999167,
      "artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
      "precision": "fp32",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.0763740143405002,
      "grade": "moderate",
      "residentMemoryBytes": 9200435200,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "dog",
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 120,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 8340.832292000414,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 7871.886917000666,
      "inferenceMs": 7854.881041999761,
      "tokensPerSecond": 5.7289218970200375,
      "inputTokens": 828,
      "outputTokens": 45,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "requestMs": 7881.953541000257,
      "totalMs": 16222.785833000671,
      "artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
      "precision": "fp32",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.12703434520131782,
      "grade": "good",
      "residentMemoryBytes": 9206398976,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "dog",
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2969.3782499998633,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 625
      },
      "parameterDtype": "torch.bfloat16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 3642.0052079993184,
      "inferenceMs": 3624.2387919992325,
      "tokensPerSecond": 12.416400403676693,
      "inputTokens": 828,
      "outputTokens": 45,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 8272379904,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 3649.2836249999527,
      "totalMs": 6618.661874999816,
      "artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
      "precision": "bf16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.27457401702874973,
      "grade": "good",
      "residentMemoryBytes": 657293312,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "LOCAL AI 2026",
      "expected": [
        "localai2026"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2969.3782499998633,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 625
      },
      "parameterDtype": "torch.bfloat16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 4546.717124998395,
      "inferenceMs": 4536.961292000342,
      "tokensPerSecond": 14.988005324158026,
      "inputTokens": 793,
      "outputTokens": 68,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 768,
      "inputHeight": 256,
      "visionGrid": [
        [
          1,
          32,
          96
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 9593749504,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 4549.028875000658,
      "totalMs": 7518.407125000522,
      "artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
      "precision": "bf16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.2199389081194166,
      "grade": "good",
      "residentMemoryBytes": 697450496,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "3",
      "expected": [
        "3",
        "three"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2969.3782499998633,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 625
      },
      "parameterDtype": "torch.bfloat16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 4064.170166999247,
      "inferenceMs": 4051.7539579996082,
      "tokensPerSecond": 12.587141403121924,
      "inputTokens": 826,
      "outputTokens": 51,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 640,
      "inputHeight": 320,
      "visionGrid": [
        [
          1,
          40,
          80
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 11026055168,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 4066.2290830005077,
      "totalMs": 7035.607333000371,
      "artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
      "precision": "bf16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.24605268945673683,
      "grade": "good",
      "residentMemoryBytes": 734052352,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "17 + 25 = 42\n\n42",
      "expected": [
        "42"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2969.3782499998633,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 625
      },
      "parameterDtype": "torch.bfloat16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 5424.383916997613,
      "inferenceMs": 5421.142041999701,
      "tokensPerSecond": 44.640040442609994,
      "inputTokens": 24,
      "outputTokens": 242,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": null,
      "inputHeight": null,
      "visionGrid": null,
      "inputModality": "text",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 10459955200,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 5426.033374998951,
      "totalMs": 8395.411624998815,
      "artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
      "precision": "bf16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.18435273301110633,
      "grade": "good",
      "residentMemoryBytes": 763641856,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "READY",
      "expected": [
        "ready"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2969.3782499998633,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 625
      },
      "parameterDtype": "torch.bfloat16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 3103.133249998791,
      "inferenceMs": 3100.471124998876,
      "tokensPerSecond": 49.02480747988102,
      "inputTokens": 17,
      "outputTokens": 152,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": null,
      "inputHeight": null,
      "visionGrid": null,
      "inputModality": "text",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 10460053504,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 3104.879874998005,
      "totalMs": 6074.258124997868,
      "artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
      "precision": "bf16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.32225493378358455,
      "grade": "good",
      "residentMemoryBytes": 786235392,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "dog",
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2969.3782499998633,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 625
      },
      "parameterDtype": "torch.bfloat16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 3524.995334002597,
      "inferenceMs": 3503.0560000013793,
      "tokensPerSecond": 12.845926528146363,
      "inputTokens": 828,
      "outputTokens": 45,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 11210833920,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 3534.1916249999485,
      "totalMs": 6503.569874999812,
      "artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
      "precision": "bf16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.28368831877700956,
      "grade": "good",
      "residentMemoryBytes": 789757952,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "dog",
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2681.656999997358,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 625
      },
      "parameterDtype": "torch.bfloat16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 3286.652040998888,
      "inferenceMs": 3270.9619589986687,
      "tokensPerSecond": 13.757420772259833,
      "inputTokens": 828,
      "outputTokens": 45,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 8272379904,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 3293.6740000004647,
      "totalMs": 5975.330999997823,
      "artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
      "precision": "bf16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.30426098884994146,
      "grade": "good",
      "residentMemoryBytes": 657752064,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "LOCAL AI 2026",
      "expected": [
        "localai2026"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2681.656999997358,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 625
      },
      "parameterDtype": "torch.bfloat16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 3687.2162079998816,
      "inferenceMs": 3676.653916001669,
      "tokensPerSecond": 18.49507774012884,
      "inputTokens": 793,
      "outputTokens": 68,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 768,
      "inputHeight": 256,
      "visionGrid": [
        [
          1,
          32,
          96
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 9593749504,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 3689.3571250002424,
      "totalMs": 6371.0141249976,
      "artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
      "precision": "bf16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.27120731293987416,
      "grade": "good",
      "residentMemoryBytes": 700317696,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "3",
      "expected": [
        "3",
        "three"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2681.656999997358,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 625
      },
      "parameterDtype": "torch.bfloat16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 3502.547500000219,
      "inferenceMs": 3492.5216670017107,
      "tokensPerSecond": 14.602629521775567,
      "inputTokens": 826,
      "outputTokens": 51,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 640,
      "inputHeight": 320,
      "visionGrid": [
        [
          1,
          40,
          80
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 11026055168,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 3504.728667001473,
      "totalMs": 6186.385666998831,
      "artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
      "precision": "bf16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.28550647778508,
      "grade": "good",
      "residentMemoryBytes": 737050624,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "17 + 25 = 42\n\n42",
      "expected": [
        "42"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2681.656999997358,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 625
      },
      "parameterDtype": "torch.bfloat16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 5662.737250000646,
      "inferenceMs": 5660.2739580011985,
      "tokensPerSecond": 42.75411434068767,
      "inputTokens": 24,
      "outputTokens": 242,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": null,
      "inputHeight": null,
      "visionGrid": null,
      "inputModality": "text",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 10459955200,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 5664.1843329998665,
      "totalMs": 8345.841332997225,
      "artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
      "precision": "bf16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.17659304252548289,
      "grade": "good",
      "residentMemoryBytes": 766574592,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "READY",
      "expected": [
        "ready"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2681.656999997358,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 625
      },
      "parameterDtype": "torch.bfloat16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 3278.5739170030865,
      "inferenceMs": 3276.679750000767,
      "tokensPerSecond": 46.388421083862234,
      "inputTokens": 17,
      "outputTokens": 152,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": null,
      "inputHeight": null,
      "visionGrid": null,
      "inputModality": "text",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 10460053504,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 3279.8882909992244,
      "totalMs": 5961.5452909965825,
      "artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
      "precision": "bf16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.30501066174347247,
      "grade": "good",
      "residentMemoryBytes": 789151744,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "dog",
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2681.656999997358,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 625
      },
      "parameterDtype": "torch.bfloat16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 3228.9094160005334,
      "inferenceMs": 3211.6581250011222,
      "tokensPerSecond": 14.011453974412134,
      "inputTokens": 828,
      "outputTokens": 45,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 11210833920,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 3236.58420799984,
      "totalMs": 5918.241207997198,
      "artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
      "precision": "bf16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.3097020916860044,
      "grade": "good",
      "residentMemoryBytes": 793100288,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "dog",
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2689.8290419994737,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 625
      },
      "parameterDtype": "torch.bfloat16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 3229.67858300035,
      "inferenceMs": 3208.9673330010555,
      "tokensPerSecond": 14.023202896838338,
      "inputTokens": 828,
      "outputTokens": 45,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 8272379904,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 3237.862624999252,
      "totalMs": 5927.691666998726,
      "artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
      "precision": "bf16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.3096283343065695,
      "grade": "good",
      "residentMemoryBytes": 658423808,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "LOCAL AI 2026",
      "expected": [
        "localai2026"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2689.8290419994737,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 625
      },
      "parameterDtype": "torch.bfloat16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 3695.6460000001243,
      "inferenceMs": 3687.5166249992617,
      "tokensPerSecond": 18.440594827152438,
      "inputTokens": 793,
      "outputTokens": 68,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 768,
      "inputHeight": 256,
      "visionGrid": [
        [
          1,
          32,
          96
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 9593749504,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 3697.675374998653,
      "totalMs": 6387.5044169981265,
      "artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
      "precision": "bf16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.2705886873363862,
      "grade": "good",
      "residentMemoryBytes": 697745408,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "3",
      "expected": [
        "3",
        "three"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2689.8290419994737,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 625
      },
      "parameterDtype": "torch.bfloat16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 3621.6417079995153,
      "inferenceMs": 3611.137249998137,
      "tokensPerSecond": 14.122974694475074,
      "inputTokens": 826,
      "outputTokens": 51,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 640,
      "inputHeight": 320,
      "visionGrid": [
        [
          1,
          40,
          80
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 11026055168,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 3624.075374998938,
      "totalMs": 6313.904416998412,
      "artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
      "precision": "bf16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.2761178715694573,
      "grade": "good",
      "residentMemoryBytes": 734314496,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "17 + 25 = 42\n\n42",
      "expected": [
        "42"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2689.8290419994737,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 625
      },
      "parameterDtype": "torch.bfloat16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 5656.247999999323,
      "inferenceMs": 5654.064667000057,
      "tokensPerSecond": 42.801066887761785,
      "inputTokens": 24,
      "outputTokens": 242,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": null,
      "inputHeight": null,
      "visionGrid": null,
      "inputModality": "text",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 10459955200,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 5657.76150000238,
      "totalMs": 8347.590542001853,
      "artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
      "precision": "bf16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.1767956426238948,
      "grade": "good",
      "residentMemoryBytes": 762937344,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "READY",
      "expected": [
        "ready"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2689.8290419994737,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 625
      },
      "parameterDtype": "torch.bfloat16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 3482.4695410025015,
      "inferenceMs": 3478.8234170009673,
      "tokensPerSecond": 43.69293343754612,
      "inputTokens": 17,
      "outputTokens": 152,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": null,
      "inputHeight": null,
      "visionGrid": null,
      "inputModality": "text",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 10460053504,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 3484.2842500002007,
      "totalMs": 6174.113291999674,
      "artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
      "precision": "bf16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.28715254741671886,
      "grade": "good",
      "residentMemoryBytes": 785088512,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "dog",
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2689.8290419994737,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 625
      },
      "parameterDtype": "torch.bfloat16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 3656.1782080025296,
      "inferenceMs": 3636.0153749992605,
      "tokensPerSecond": 12.376185290473133,
      "inputTokens": 828,
      "outputTokens": 45,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 11210833920,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 3664.811208000174,
      "totalMs": 6354.6402499996475,
      "artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
      "precision": "bf16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.2735096439804906,
      "grade": "good",
      "residentMemoryBytes": 789331968,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "dog",
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 120,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2296.0705000004964,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 12416.36437499983,
      "inferenceMs": 12395.884958998067,
      "tokensPerSecond": 3.63023698177635,
      "inputTokens": 828,
      "outputTokens": 45,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "requestMs": 12422.926582999935,
      "totalMs": 14718.997083000431,
      "artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
      "precision": "fp32",
      "cpuThreads": 1,
      "fallback": false,
      "requestsPerSecond": 0.08053887352190513,
      "grade": "moderate",
      "residentMemoryBytes": 9279471616,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "LOCAL AI 2026",
      "expected": [
        "localai2026"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 120,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2296.0705000004964,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 14806.454499997926,
      "inferenceMs": 14793.110499998875,
      "tokensPerSecond": 5.137526688522051,
      "inputTokens": 793,
      "outputTokens": 76,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 768,
      "inputHeight": 256,
      "visionGrid": [
        [
          1,
          32,
          96
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "requestMs": 14808.299083000747,
      "totalMs": 17104.369583001244,
      "artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
      "precision": "fp32",
      "cpuThreads": 1,
      "fallback": false,
      "requestsPerSecond": 0.06753811319246887,
      "grade": "moderate",
      "residentMemoryBytes": 9277063168,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "3",
      "expected": [
        "3",
        "three"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 120,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2296.0705000004964,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 14929.877874998056,
      "inferenceMs": 14916.288415999588,
      "tokensPerSecond": 5.162142072649108,
      "inputTokens": 826,
      "outputTokens": 77,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 640,
      "inputHeight": 320,
      "visionGrid": [
        [
          1,
          40,
          80
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "requestMs": 14931.277916999534,
      "totalMs": 17227.34841700003,
      "artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
      "precision": "fp32",
      "cpuThreads": 1,
      "fallback": false,
      "requestsPerSecond": 0.06697978432058208,
      "grade": "moderate",
      "residentMemoryBytes": 9309601792,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "17 + 25 = 42\n\nAnswer: 42",
      "expected": [
        "42"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 120,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2296.0705000004964,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 16267.460209001001,
      "inferenceMs": 16266.589874998317,
      "tokensPerSecond": 12.418091410202281,
      "inputTokens": 24,
      "outputTokens": 202,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": null,
      "inputHeight": null,
      "visionGrid": null,
      "inputModality": "text",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "requestMs": 16268.699415999436,
      "totalMs": 18564.769915999932,
      "artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
      "precision": "fp32",
      "cpuThreads": 1,
      "fallback": false,
      "requestsPerSecond": 0.06147241100652496,
      "grade": "moderate",
      "residentMemoryBytes": 9429401600,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "READY",
      "expected": [
        "ready"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 120,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2296.0705000004964,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 12811.984916002984,
      "inferenceMs": 12810.033833000489,
      "tokensPerSecond": 11.865698559548386,
      "inputTokens": 17,
      "outputTokens": 152,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": null,
      "inputHeight": null,
      "visionGrid": null,
      "inputModality": "text",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "requestMs": 12813.1942910004,
      "totalMs": 15109.264791000896,
      "artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
      "precision": "fp32",
      "cpuThreads": 1,
      "fallback": false,
      "requestsPerSecond": 0.07805191830587752,
      "grade": "moderate",
      "residentMemoryBytes": 9432449024,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "dog",
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 120,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2296.0705000004964,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 625
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 12295.713666000665,
      "inferenceMs": 12275.892167002894,
      "tokensPerSecond": 3.6657213494395298,
      "inputTokens": 828,
      "outputTokens": 45,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "requestMs": 12302.271958000347,
      "totalMs": 14598.342458000843,
      "artifactRevision": "33e0ad94c327808ebc7d3bbc97e78e600f1eeecd",
      "precision": "fp32",
      "cpuThreads": 1,
      "fallback": false,
      "requestsPerSecond": 0.08132915479035081,
      "grade": "moderate",
      "residentMemoryBytes": 9438429184,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  }
]

Qwen3-VL 4B Instruct — Text, Image

42 configurations tested
Passed42
Failed0
TestedNov 6, 2026
Runtimetransformers 5.22.1

Quality criteria

{ "method": "Final-answer fixture semantic/normalized OCR plus completion/repetition/speed gates frozen in research/variant-onboarding/qwen3-vl-family/PLAN.md", "fixtures": [ "dog", "ocr", "circles", "arithmetic", "instruction" ] }

PrecisionDeviceResultPlatformInference / load
BF16mpsAcceleratedPassedFastmacos · aarch64252 ms / 6632 ms
BF16mpsAcceleratedPassedFastmacos · aarch64338 ms / 3696 ms
BF16mpsAcceleratedPassedFastmacos · aarch64345 ms / 6632 ms
BF16mpsAcceleratedPassedFastmacos · aarch64425 ms / 4015 ms
BF16mpsAcceleratedPassedFastmacos · aarch64441 ms / 4015 ms
BF16mpsAcceleratedPassedFastmacos · aarch64634 ms / 3696 ms
BF16mpsAcceleratedPassedFastmacos · aarch643549 ms / 3696 ms
BF16mpsAcceleratedPassedFastmacos · aarch643735 ms / 3696 ms
BF16mpsAcceleratedPassedFastmacos · aarch643827 ms / 6632 ms
BF16mpsAcceleratedPassedFastmacos · aarch643979 ms / 3696 ms
BF16mpsAcceleratedPassedFastmacos · aarch643981 ms / 3696 ms
BF16mpsAcceleratedPassedFastmacos · aarch644002 ms / 6632 ms
BF16mpsAcceleratedPassedFastmacos · aarch644167 ms / 6632 ms
BF16mpsAcceleratedPassedFastmacos · aarch644471 ms / 4015 ms
BF16mpsAcceleratedPassedFastmacos · aarch644525 ms / 6632 ms
BF16mpsAcceleratedPassedFastmacos · aarch644634 ms / 4015 ms
BF16mpsAcceleratedPassedFastmacos · aarch644896 ms / 4015 ms
BF16mpsAcceleratedPassedFastmacos · aarch644954 ms / 4015 ms
FP32cpuPassedFastmacos · aarch64498 ms / 10236 ms
FP32cpuPassedFastmacos · aarch64583 ms / 7120 ms
FP32cpuPassedFastmacos · aarch64667 ms / 10236 ms
FP32cpuPassedFastmacos · aarch64708 ms / 7120 ms
FP32cpuPassedFastmacos · aarch64710 ms / 3741 ms
FP32cpuPassedFastmacos · aarch64843 ms / 3741 ms
FP32cpuPassedFastmacos · aarch64959 ms / 7942 ms
FP32cpuPassedFastmacos · aarch641059 ms / 7942 ms
FP32cpuPassedFastmacos · aarch646161 ms / 3741 ms
FP32cpuPassedFastmacos · aarch646441 ms / 7942 ms
FP32cpuPassedFastmacos · aarch646615 ms / 7120 ms
FP32cpuPassedFastmacos · aarch646699 ms / 3741 ms
FP32cpuPassedFastmacos · aarch646810 ms / 7120 ms
FP32cpuPassedFastmacos · aarch647093 ms / 7120 ms
FP32cpuPassedFastmacos · aarch647137 ms / 3741 ms
FP32cpuPassedFastmacos · aarch647263 ms / 7942 ms
FP32cpuPassedFastmacos · aarch647433 ms / 7942 ms
FP32cpuPassedFastmacos · aarch647436 ms / 7120 ms
FP32cpuPassedFastmacos · aarch648309 ms / 3741 ms
FP32cpuPassedModeratemacos · aarch6410484 ms / 10236 ms
FP32cpuPassedModeratemacos · aarch6410581 ms / 10236 ms
FP32cpuPassedModeratemacos · aarch6411389 ms / 10236 ms
FP32cpuPassedModeratemacos · aarch6411602 ms / 10236 ms
FP32cpuPassedModeratemacos · aarch6416897 ms / 7942 ms
Test setup & sample details
Quality gate
{ "method": "Final-answer fixture semantic/normalized OCR plus completion/repetition/speed gates frozen in research/variant-onboarding/qwen3-vl-family/PLAN.md", "fixtures": [ "dog", "ocr", "circles", "arithmetic", "instruction" ] }
Tested
2026-11-06T16:16:04.465768+00:00
Environment
{ "os": "macos", "arch": "aarch64", "hardware": "Apple M3 Max", "osVersion": "27.2", "python": "App-owned CPython3.13.18 private model probe venv" }
Runtime
transformers 5.22.1
Artifact revision
ebb281ec70b05090aa6165b016eac8ec08e71b17
Fixture
{
  "file": "model-samples/desktop/qwen3-vl-family/Qwen3-VL-4B-Instruct-dog-input.json",
  "url": "https://assets.localai.tools/samples/v1/files/7246f398c23127a1d90d48dd4a0d56a81dd2b7b8909e5a2efa1975f1a2ed6a87/Qwen3-VL-4B-Instruct-dog-input.json",
  "size": 607,
  "sha1": "409fa36ae63ce99dbfa9da362719622bab8d19f3",
  "sha256": "7246f398c23127a1d90d48dd4a0d56a81dd2b7b8909e5a2efa1975f1a2ed6a87"
}
Rerun when
model revision/checksum; pinned runtime/processor/attention/backend/driver; fixture checksum/question/expected answers; visual pixel and token budgets; quality gates
Per-case quality & failure details
[
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "dog",
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3741.1324579989014,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 713
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 6174.910917001398,
      "inferenceMs": 6161.06495799977,
      "tokensPerSecond": 0.324619203601014,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "requestMs": 6181.928042002255,
      "totalMs": 9923.060500001156,
      "artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
      "precision": "fp32",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.1619456561303091,
      "grade": "good",
      "residentMemoryBytes": 18426576896,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "LOCAL AI 2026",
      "expected": [
        "localai2026"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3741.1324579989014,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 713
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 7145.66441700299,
      "inferenceMs": 7137.178999997559,
      "tokensPerSecond": 1.1208910411246147,
      "inputTokens": 791,
      "outputTokens": 8,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 768,
      "inputHeight": 256,
      "visionGrid": [
        [
          1,
          32,
          96
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "requestMs": 7147.677042001305,
      "totalMs": 10888.809500000207,
      "artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
      "precision": "fp32",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.13994499904312838,
      "grade": "good",
      "residentMemoryBytes": 18429919232,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "3",
      "expected": [
        "3",
        "three"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3741.1324579989014,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 713
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 6705.934417001117,
      "inferenceMs": 6698.781499999313,
      "tokensPerSecond": 0.2985617608217562,
      "inputTokens": 824,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 640,
      "inputHeight": 320,
      "visionGrid": [
        [
          1,
          40,
          80
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "requestMs": 6707.429541998863,
      "totalMs": 10448.561999997764,
      "artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
      "precision": "fp32",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.14912164924619087,
      "grade": "good",
      "residentMemoryBytes": 18432081920,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "42",
      "expected": [
        "42"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3741.1324579989014,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 713
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 844.1981669966481,
      "inferenceMs": 843.3275000024878,
      "tokensPerSecond": 3.557336859038926,
      "inputTokens": 22,
      "outputTokens": 3,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": null,
      "inputHeight": null,
      "visionGrid": null,
      "inputModality": "text",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "requestMs": 845.9493330010446,
      "totalMs": 4587.081790999946,
      "artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
      "precision": "fp32",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 1.1845559953744493,
      "grade": "good",
      "residentMemoryBytes": 18433540096,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "READY",
      "expected": [
        "ready"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3741.1324579989014,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 713
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 710.5662499998289,
      "inferenceMs": 709.6702079979877,
      "tokensPerSecond": 2.8182104553072502,
      "inputTokens": 15,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": null,
      "inputHeight": null,
      "visionGrid": null,
      "inputModality": "text",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "requestMs": 711.9691659972887,
      "totalMs": 4453.10162399619,
      "artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
      "precision": "fp32",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 1.4073283103443779,
      "grade": "good",
      "residentMemoryBytes": 18435031040,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "dog",
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3741.1324579989014,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 713
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 8325.148125000851,
      "inferenceMs": 8308.921625000949,
      "tokensPerSecond": 0.24070512279019982,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "requestMs": 8332.22370799922,
      "totalMs": 12073.35616599812,
      "artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
      "precision": "fp32",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.12011798288572767,
      "grade": "good",
      "residentMemoryBytes": 18435031040,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "dog",
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 7941.704166998534,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 713
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 7277.82308300084,
      "inferenceMs": 7263.054041999567,
      "tokensPerSecond": 0.27536625618296884,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "requestMs": 7285.103083002468,
      "totalMs": 15226.807250001002,
      "artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
      "precision": "fp32",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.13740372479454027,
      "grade": "good",
      "residentMemoryBytes": 18250121216,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "LOCAL AI 2026",
      "expected": [
        "localai2026"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 7941.704166998534,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 713
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 16904.474708000635,
      "inferenceMs": 16896.785790999274,
      "tokensPerSecond": 0.4734628289045073,
      "inputTokens": 791,
      "outputTokens": 8,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 768,
      "inputHeight": 256,
      "visionGrid": [
        [
          1,
          32,
          96
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "requestMs": 16906.74562499771,
      "totalMs": 24848.449791996245,
      "artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
      "precision": "fp32",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.05915593458380076,
      "grade": "moderate",
      "residentMemoryBytes": 18222071808,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "3",
      "expected": [
        "3",
        "three"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 7941.704166998534,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 713
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 6450.158916999499,
      "inferenceMs": 6440.879707999557,
      "tokensPerSecond": 0.3105165894522149,
      "inputTokens": 824,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 640,
      "inputHeight": 320,
      "visionGrid": [
        [
          1,
          40,
          80
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "requestMs": 6451.830875001178,
      "totalMs": 14393.535041999712,
      "artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
      "precision": "fp32",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.15503493989341624,
      "grade": "good",
      "residentMemoryBytes": 18249777152,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "42",
      "expected": [
        "42"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 7941.704166998534,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 713
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 1059.63679100023,
      "inferenceMs": 1058.7157500012836,
      "tokensPerSecond": 2.833621772412815,
      "inputTokens": 22,
      "outputTokens": 3,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": null,
      "inputHeight": null,
      "visionGrid": null,
      "inputModality": "text",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "requestMs": 1061.3430419980432,
      "totalMs": 9003.047208996577,
      "artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
      "precision": "fp32",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.9437195919330654,
      "grade": "good",
      "residentMemoryBytes": 18257313792,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "READY",
      "expected": [
        "ready"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 7941.704166998534,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 713
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 960.1322080015962,
      "inferenceMs": 959.1316250007367,
      "tokensPerSecond": 2.0852195338658173,
      "inputTokens": 15,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": null,
      "inputHeight": null,
      "visionGrid": null,
      "inputModality": "text",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "requestMs": 962.295625002298,
      "totalMs": 8903.999792000832,
      "artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
      "precision": "fp32",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 1.041523231557229,
      "grade": "good",
      "residentMemoryBytes": 18258411520,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "dog",
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 7941.704166998534,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 713
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 7455.856999997195,
      "inferenceMs": 7432.98833299923,
      "tokensPerSecond": 0.2690707842390753,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "requestMs": 7465.716499998962,
      "totalMs": 15407.420666997496,
      "artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
      "precision": "fp32",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.1341227440387304,
      "grade": "good",
      "residentMemoryBytes": 18259066880,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "dog",
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 7119.960417003313,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 713
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 7106.457958998362,
      "inferenceMs": 7093.278000000282,
      "tokensPerSecond": 0.28195708669530795,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "requestMs": 7114.342583001417,
      "totalMs": 14234.30300000473,
      "artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
      "precision": "fp32",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.1407170781519613,
      "grade": "good",
      "residentMemoryBytes": 18449825792,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "LOCAL AI 2026",
      "expected": [
        "localai2026"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 7119.960417003313,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 713
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 7445.575707999524,
      "inferenceMs": 7435.673832998873,
      "tokensPerSecond": 1.0758944219011728,
      "inputTokens": 791,
      "outputTokens": 8,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 768,
      "inputHeight": 256,
      "visionGrid": [
        [
          1,
          32,
          96
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "requestMs": 7447.62858300237,
      "totalMs": 14567.589000005682,
      "artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
      "precision": "fp32",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.1343079486688451,
      "grade": "good",
      "residentMemoryBytes": 18453168128,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "3",
      "expected": [
        "3",
        "three"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 7119.960417003313,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 713
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 6816.494291997515,
      "inferenceMs": 6809.672999999748,
      "tokensPerSecond": 0.29369985900939355,
      "inputTokens": 824,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 640,
      "inputHeight": 320,
      "visionGrid": [
        [
          1,
          40,
          80
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "requestMs": 6818.0038329992385,
      "totalMs": 13937.964250002551,
      "artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
      "precision": "fp32",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.1467029762166732,
      "grade": "good",
      "residentMemoryBytes": 18455298048,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "42",
      "expected": [
        "42"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 7119.960417003313,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 713
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 708.3546249996289,
      "inferenceMs": 707.6382500017644,
      "tokensPerSecond": 4.239454269172872,
      "inputTokens": 22,
      "outputTokens": 3,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": null,
      "inputHeight": null,
      "visionGrid": null,
      "inputModality": "text",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "requestMs": 709.6257079974748,
      "totalMs": 7829.586125000787,
      "artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
      "precision": "fp32",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 1.4117222711724708,
      "grade": "good",
      "residentMemoryBytes": 18457362432,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "READY",
      "expected": [
        "ready"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 7119.960417003313,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 713
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 584.1366250024294,
      "inferenceMs": 583.3942919998663,
      "tokensPerSecond": 3.4282131783360996,
      "inputTokens": 15,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": null,
      "inputHeight": null,
      "visionGrid": null,
      "inputModality": "text",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "requestMs": 585.4325830005109,
      "totalMs": 7705.3930000038235,
      "artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
      "precision": "fp32",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 1.7119282667746456,
      "grade": "good",
      "residentMemoryBytes": 18457493504,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "dog",
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 7119.960417003313,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 713
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 6628.192583000782,
      "inferenceMs": 6615.071541000361,
      "tokensPerSecond": 0.3023398896903768,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "requestMs": 6635.057333998702,
      "totalMs": 13755.017751002015,
      "artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
      "precision": "fp32",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.15087069174252477,
      "grade": "good",
      "residentMemoryBytes": 18453463040,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "dog",
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3696.491624999908,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 713
      },
      "parameterDtype": "torch.bfloat16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 3573.611374999018,
      "inferenceMs": 3548.659707998013,
      "tokensPerSecond": 0.563593064585025,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 12675694592,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 3584.1817920008907,
      "totalMs": 7280.6734170007985,
      "artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
      "precision": "bf16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.27982897272937934,
      "grade": "good",
      "residentMemoryBytes": 660766720,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "LOCAL AI 2026",
      "expected": [
        "localai2026"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3696.491624999908,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 713
      },
      "parameterDtype": "torch.bfloat16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 3993.279040998459,
      "inferenceMs": 3980.546167000284,
      "tokensPerSecond": 2.00977445414953,
      "inputTokens": 791,
      "outputTokens": 8,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 768,
      "inputHeight": 256,
      "visionGrid": [
        [
          1,
          32,
          96
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 14099972096,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 3998.594500000763,
      "totalMs": 7695.086125000671,
      "artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
      "precision": "bf16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.25042076692691256,
      "grade": "good",
      "residentMemoryBytes": 696598528,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "3",
      "expected": [
        "3",
        "three"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3696.491624999908,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 713
      },
      "parameterDtype": "torch.bfloat16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 4002.1517909990507,
      "inferenceMs": 3979.4315829967672,
      "tokensPerSecond": 0.5025843410766399,
      "inputTokens": 824,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 640,
      "inputHeight": 320,
      "visionGrid": [
        [
          1,
          40,
          80
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 15608709120,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 4007.612541998242,
      "totalMs": 7704.10416699815,
      "artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
      "precision": "bf16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.24986558537060674,
      "grade": "good",
      "residentMemoryBytes": 729497600,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "42",
      "expected": [
        "42"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3696.491624999908,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 713
      },
      "parameterDtype": "torch.bfloat16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 342.77300000030664,
      "inferenceMs": 338.4717919980176,
      "tokensPerSecond": 8.863367851988006,
      "inputTokens": 22,
      "outputTokens": 3,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": null,
      "inputHeight": null,
      "visionGrid": null,
      "inputModality": "text",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 15625666560,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 345.7546249992447,
      "totalMs": 4042.2462499991525,
      "artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
      "precision": "bf16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 2.9173826409871997,
      "grade": "good",
      "residentMemoryBytes": 742850560,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "READY",
      "expected": [
        "ready"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3696.491624999908,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 713
      },
      "parameterDtype": "torch.bfloat16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 636.409624999942,
      "inferenceMs": 633.8471249982831,
      "tokensPerSecond": 3.1553349713551473,
      "inputTokens": 15,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": null,
      "inputHeight": null,
      "visionGrid": null,
      "inputModality": "text",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 15625781248,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 638.0790830007754,
      "totalMs": 4334.570708000683,
      "artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
      "precision": "bf16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 1.571315015859622,
      "grade": "good",
      "residentMemoryBytes": 758710272,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "dog",
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3696.491624999908,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 713
      },
      "parameterDtype": "torch.bfloat16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 3750.7840000034776,
      "inferenceMs": 3734.5751249995374,
      "tokensPerSecond": 0.5355361541964557,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 15625781248,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 3759.758709002199,
      "totalMs": 7456.250334002107,
      "artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
      "precision": "bf16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.26661092720857105,
      "grade": "good",
      "residentMemoryBytes": 765018112,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "dog",
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 4015.286958001525,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 713
      },
      "parameterDtype": "torch.bfloat16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 4499.136832997465,
      "inferenceMs": 4471.310459000961,
      "tokensPerSecond": 0.44729616034017605,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 12675694592,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 4513.216499999544,
      "totalMs": 8528.503458001069,
      "artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
      "precision": "bf16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.22226485593099174,
      "grade": "good",
      "residentMemoryBytes": 664322048,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "LOCAL AI 2026",
      "expected": [
        "localai2026"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 4015.286958001525,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 713
      },
      "parameterDtype": "torch.bfloat16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 4978.793833000964,
      "inferenceMs": 4954.102709001745,
      "tokensPerSecond": 1.6148232020833506,
      "inputTokens": 791,
      "outputTokens": 8,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 768,
      "inputHeight": 256,
      "visionGrid": [
        [
          1,
          32,
          96
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 14099972096,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 4985.715875001915,
      "totalMs": 9001.00283300344,
      "artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
      "precision": "bf16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.2008518596154143,
      "grade": "good",
      "residentMemoryBytes": 699482112,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "3",
      "expected": [
        "3",
        "three"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 4015.286958001525,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 713
      },
      "parameterDtype": "torch.bfloat16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 4676.366415998928,
      "inferenceMs": 4634.414374999324,
      "tokensPerSecond": 0.43155398679693846,
      "inputTokens": 824,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 640,
      "inputHeight": 320,
      "visionGrid": [
        [
          1,
          40,
          80
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 15608709120,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 4679.809165998449,
      "totalMs": 8695.096123999974,
      "artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
      "precision": "bf16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.213841241477308,
      "grade": "good",
      "residentMemoryBytes": 732839936,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "42",
      "expected": [
        "42"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 4015.286958001525,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 713
      },
      "parameterDtype": "torch.bfloat16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 443.8317089989141,
      "inferenceMs": 440.730332997191,
      "tokensPerSecond": 6.8068834282371045,
      "inputTokens": 22,
      "outputTokens": 3,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": null,
      "inputHeight": null,
      "visionGrid": null,
      "inputModality": "text",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 15625666560,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 447.4767499996233,
      "totalMs": 4462.763708001148,
      "artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
      "precision": "bf16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 2.2531062556471073,
      "grade": "good",
      "residentMemoryBytes": 746553344,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "READY",
      "expected": [
        "ready"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 4015.286958001525,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 713
      },
      "parameterDtype": "torch.bfloat16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 429.6830840030452,
      "inferenceMs": 424.90379200171446,
      "tokensPerSecond": 4.706947873018582,
      "inputTokens": 15,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": null,
      "inputHeight": null,
      "visionGrid": null,
      "inputModality": "text",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 15625781248,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 433.0050409989781,
      "totalMs": 4448.291999000503,
      "artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
      "precision": "bf16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 2.327296645433947,
      "grade": "good",
      "residentMemoryBytes": 759431168,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "dog",
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 4015.286958001525,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 713
      },
      "parameterDtype": "torch.bfloat16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 4939.111083000171,
      "inferenceMs": 4895.961666999938,
      "tokensPerSecond": 0.4084999303569967,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 15625781248,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 4960.963375000574,
      "totalMs": 8976.250333002099,
      "artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
      "precision": "bf16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.20246558200358772,
      "grade": "good",
      "residentMemoryBytes": 761184256,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "dog",
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 6632.466375001968,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 713
      },
      "parameterDtype": "torch.bfloat16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 4214.732832999289,
      "inferenceMs": 4167.440958000952,
      "tokensPerSecond": 0.479910818210935,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 12675694592,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 4236.636833000375,
      "totalMs": 10869.103208002343,
      "artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
      "precision": "bf16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.23726296294998606,
      "grade": "good",
      "residentMemoryBytes": 657571840,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "LOCAL AI 2026",
      "expected": [
        "localai2026"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 6632.466375001968,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 713
      },
      "parameterDtype": "torch.bfloat16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 4560.405457999877,
      "inferenceMs": 4524.649584000144,
      "tokensPerSecond": 1.7680927222053235,
      "inputTokens": 791,
      "outputTokens": 8,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 768,
      "inputHeight": 256,
      "visionGrid": [
        [
          1,
          32,
          96
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 14099972096,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 4572.066333999828,
      "totalMs": 11204.532709001796,
      "artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
      "precision": "bf16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.21927874817485732,
      "grade": "good",
      "residentMemoryBytes": 693256192,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "3",
      "expected": [
        "3",
        "three"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 6632.466375001968,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 713
      },
      "parameterDtype": "torch.bfloat16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 3857.208166999044,
      "inferenceMs": 3827.064249999239,
      "tokensPerSecond": 0.5225937871308007,
      "inputTokens": 824,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 640,
      "inputHeight": 320,
      "visionGrid": [
        [
          1,
          40,
          80
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 15608709120,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 3864.1452499978186,
      "totalMs": 10496.611624999787,
      "artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
      "precision": "bf16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.25925486950786286,
      "grade": "good",
      "residentMemoryBytes": 727564288,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "42",
      "expected": [
        "42"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 6632.466375001968,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 713
      },
      "parameterDtype": "torch.bfloat16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 349.36691600159975,
      "inferenceMs": 344.52020800017635,
      "tokensPerSecond": 8.707762071241012,
      "inputTokens": 22,
      "outputTokens": 3,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": null,
      "inputHeight": null,
      "visionGrid": null,
      "inputModality": "text",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 15625666560,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 351.9629999973404,
      "totalMs": 6984.4293749993085,
      "artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
      "precision": "bf16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 2.862320254718741,
      "grade": "good",
      "residentMemoryBytes": 741294080,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "READY",
      "expected": [
        "ready"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 6632.466375001968,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 713
      },
      "parameterDtype": "torch.bfloat16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 255.4954999977781,
      "inferenceMs": 251.7043330008164,
      "tokensPerSecond": 7.94583063452274,
      "inputTokens": 15,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": null,
      "inputHeight": null,
      "visionGrid": null,
      "inputModality": "text",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 15625781248,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 257.34545799787156,
      "totalMs": 6889.81183299984,
      "artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
      "precision": "bf16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 3.9139632596609193,
      "grade": "good",
      "residentMemoryBytes": 754286592,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "dog",
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 6632.466375001968,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 713
      },
      "parameterDtype": "torch.bfloat16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 4021.5398330001335,
      "inferenceMs": 4002.245250001579,
      "tokensPerSecond": 0.499719501197287,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 15625781248,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 4031.9942079986504,
      "totalMs": 10664.460583000618,
      "artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
      "precision": "bf16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.2486609710524697,
      "grade": "good",
      "residentMemoryBytes": 755941376,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "dog",
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 10236.078249999991,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 713
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 10601.683583001432,
      "inferenceMs": 10581.3890419995,
      "tokensPerSecond": 0.18901110166743026,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "requestMs": 10608.202708997851,
      "totalMs": 20844.280958997842,
      "artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
      "precision": "fp32",
      "cpuThreads": 1,
      "fallback": false,
      "requestsPerSecond": 0.0943246411922144,
      "grade": "moderate",
      "residentMemoryBytes": 18443010048,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "LOCAL AI 2026",
      "expected": [
        "localai2026"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 10236.078249999991,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 713
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 11403.46958399823,
      "inferenceMs": 11389.23666700066,
      "tokensPerSecond": 0.7024175749354051,
      "inputTokens": 791,
      "outputTokens": 8,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 768,
      "inputHeight": 256,
      "visionGrid": [
        [
          1,
          32,
          96
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "requestMs": 11405.566833000194,
      "totalMs": 21641.645083000185,
      "artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
      "precision": "fp32",
      "cpuThreads": 1,
      "fallback": false,
      "requestsPerSecond": 0.08769260904621844,
      "grade": "moderate",
      "residentMemoryBytes": 18447187968,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "3",
      "expected": [
        "3",
        "three"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 10236.078249999991,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 713
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 10498.403166999196,
      "inferenceMs": 10484.120332999737,
      "tokensPerSecond": 0.19076469331478535,
      "inputTokens": 824,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 640,
      "inputHeight": 320,
      "visionGrid": [
        [
          1,
          40,
          80
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "requestMs": 10499.87237499954,
      "totalMs": 20735.950624999532,
      "artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
      "precision": "fp32",
      "cpuThreads": 1,
      "fallback": false,
      "requestsPerSecond": 0.09525258118714774,
      "grade": "moderate",
      "residentMemoryBytes": 18449285120,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "42",
      "expected": [
        "42"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 10236.078249999991,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 713
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 667.4823749999632,
      "inferenceMs": 666.752290999284,
      "tokensPerSecond": 4.499422109977004,
      "inputTokens": 22,
      "outputTokens": 3,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": null,
      "inputHeight": null,
      "visionGrid": null,
      "inputModality": "text",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "requestMs": 668.6668750007811,
      "totalMs": 10904.745125000773,
      "artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
      "precision": "fp32",
      "cpuThreads": 1,
      "fallback": false,
      "requestsPerSecond": 1.4981668991635249,
      "grade": "good",
      "residentMemoryBytes": 18452905984,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "READY",
      "expected": [
        "ready"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 10236.078249999991,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 713
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 498.6082499999611,
      "inferenceMs": 497.7752500017232,
      "tokensPerSecond": 4.017877546127648,
      "inputTokens": 15,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": null,
      "inputHeight": null,
      "visionGrid": null,
      "inputModality": "text",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "requestMs": 499.81516700063366,
      "totalMs": 10735.893417000625,
      "artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
      "precision": "fp32",
      "cpuThreads": 1,
      "fallback": false,
      "requestsPerSecond": 2.0055825389974555,
      "grade": "good",
      "residentMemoryBytes": 18452905984,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "dog",
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 10236.078249999991,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 713
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 11621.961249999003,
      "inferenceMs": 11602.114707999135,
      "tokensPerSecond": 0.17238236738179205,
      "inputTokens": 826,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "requestMs": 11628.605249999964,
      "totalMs": 21864.683499999956,
      "artifactRevision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
      "precision": "fp32",
      "cpuThreads": 1,
      "fallback": false,
      "requestsPerSecond": 0.08604399709215049,
      "grade": "moderate",
      "residentMemoryBytes": 18448793600,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  }
]

Qwen3-VL 4B Thinking — Text, Image

20 configurations tested
Passed18
Failed2
TestedNov 15, 2026
Runtimetransformers 5.22.1

Quality criteria

{ "method": "Final-answer fixture semantic/normalized OCR plus completion/repetition/speed gates frozen in research/variant-onboarding/qwen3-vl-family/PLAN.md", "fixtures": [ "dog", "ocr", "circles", "arithmetic", "instruction" ] }

PrecisionDeviceResultPlatformInference / load
FP16mpsAcceleratedPassedFastmacos · aarch644973 ms / 19685 ms
FP16mpsAcceleratedPassedFastmacos · aarch645377 ms / 19685 ms
FP16mpsAcceleratedPassedFastmacos · aarch645434 ms / 15106 ms
FP16mpsAcceleratedPassedFastmacos · aarch645916 ms / 21061 ms
FP16mpsAcceleratedPassedFastmacos · aarch645952 ms / 21061 ms
FP16mpsAcceleratedPassedFastmacos · aarch646380 ms / 15106 ms
FP16mpsAcceleratedPassedFastmacos · aarch646524 ms / 19685 ms
FP16mpsAcceleratedPassedFastmacos · aarch646880 ms / 15106 ms
FP16mpsAcceleratedPassedFastmacos · aarch648094 ms / 15106 ms
FP16mpsAcceleratedPassedFastmacos · aarch648216 ms / 15106 ms
FP16mpsAcceleratedPassedFastmacos · aarch648834 ms / 21061 ms
FP16mpsAcceleratedPassedFastmacos · aarch648960 ms / 19685 ms
FP16mpsAcceleratedPassedFastmacos · aarch649482 ms / 19685 ms
FP16mpsAcceleratedPassedModeratemacos · aarch6410499 ms / 21061 ms
FP16mpsAcceleratedPassedModeratemacos · aarch6414166 ms / 21061 ms
FP16mpsAcceleratedPassedModeratemacos · aarch6422979 ms / 19685 ms
FP16mpsAcceleratedPassedModeratemacos · aarch6427645 ms / 15106 ms
FP16mpsAcceleratedPassedModeratemacos · aarch6432455 ms / 21061 ms
FP32cpuFailed—macos · aarch64134016 ms / 5941 ms
BF16mpsFailed—macos · aarch6446679 ms / 10534 ms
Test setup & sample details
Quality gate
{ "method": "Final-answer fixture semantic/normalized OCR plus completion/repetition/speed gates frozen in research/variant-onboarding/qwen3-vl-family/PLAN.md", "fixtures": [ "dog", "ocr", "circles", "arithmetic", "instruction" ] }
Tested
2026-11-15T05:56:45.583669+00:00
Environment
{ "os": "macos", "arch": "aarch64", "hardware": "Apple M3 Max", "osVersion": "27.2", "python": "App-owned CPython3.13.18 private model probe venv" }
Runtime
transformers 5.22.1
Artifact revision
1de27d8c51f12e819435303b9e84c4e25ba8401e
Fixture
{
  "file": "model-samples/desktop/qwen3-vl-family/Qwen3-VL-4B-Thinking-dog-input.json",
  "url": "https://assets.localai.tools/samples/v1/files/7b3ce826dfe3c11238f8ed74eb4c0376337fae452991b11c6e9c5b322ca981f0/Qwen3-VL-4B-Thinking-dog-input.json",
  "size": 609,
  "sha1": "2e345d8ba592df6216208fae8065b18c3a4f6e41",
  "sha256": "7b3ce826dfe3c11238f8ed74eb4c0376337fae452991b11c6e9c5b322ca981f0"
}
Rerun when
model revision/checksum; pinned runtime/processor/attention/backend/driver; fixture checksum/question/expected answers; visual pixel and token budgets; quality gates
Per-case quality & failure details
[
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "passed": false,
      "acceptedAnswer": true,
      "finalAnswer": "42",
      "expected": [
        "42"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": false,
      "maximumRequestSeconds": 120,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 5941.389582992997,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 713
      },
      "parameterDtype": "torch.float32",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 134017.24229200045,
      "inferenceMs": 134016.3947090041,
      "tokensPerSecond": 3.5518042477829708,
      "inputTokens": 24,
      "outputTokens": 476,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": null,
      "inputHeight": null,
      "visionGrid": null,
      "inputModality": "text",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "requestMs": 134018.6680839979,
      "totalMs": 139960.0576669909,
      "artifactRevision": "1de27d8c51f12e819435303b9e84c4e25ba8401e",
      "precision": "fp32",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.007461726438312859,
      "grade": "slow",
      "residentMemoryBytes": 19143983104,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "fail",
    "quality": {
      "passed": false,
      "acceptedAnswer": true,
      "finalAnswer": "42",
      "expected": [
        "42"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": false,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 10534.1629589675,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 713
      },
      "parameterDtype": "torch.bfloat16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 46683.07541596005,
      "inferenceMs": 46678.656041040085,
      "tokensPerSecond": 11.589879527044445,
      "inputTokens": 24,
      "outputTokens": 541,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": null,
      "inputHeight": null,
      "visionGrid": null,
      "inputModality": "text",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 19770343424,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 46684.89929201314,
      "totalMs": 57219.06225098064,
      "artifactRevision": "1de27d8c51f12e819435303b9e84c4e25ba8401e",
      "precision": "bf16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.021421039447159455,
      "grade": "slow",
      "residentMemoryBytes": 827719680,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "Dog",
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 21061.0525420052,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 713
      },
      "parameterDtype": "torch.float16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 5942.261124961078,
      "inferenceMs": 5915.678667020984,
      "tokensPerSecond": 7.437861736015744,
      "inputTokens": 828,
      "outputTokens": 44,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 12676612096,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 5951.342332991771,
      "totalMs": 27012.39487499697,
      "artifactRevision": "1de27d8c51f12e819435303b9e84c4e25ba8401e",
      "precision": "fp16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.16828610843091316,
      "grade": "good",
      "residentMemoryBytes": 2439331840,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "LOCAL AI 2026",
      "expected": [
        "localai2026"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 21061.0525420052,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 713
      },
      "parameterDtype": "torch.float16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 10512.467332999222,
      "inferenceMs": 10499.336459033657,
      "tokensPerSecond": 6.857576217404769,
      "inputTokens": 793,
      "outputTokens": 72,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 768,
      "inputHeight": 256,
      "visionGrid": [
        [
          1,
          32,
          96
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 13374078976,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 10516.864249948412,
      "totalMs": 31577.916791953612,
      "artifactRevision": "1de27d8c51f12e819435303b9e84c4e25ba8401e",
      "precision": "fp16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.09512514696344826,
      "grade": "moderate",
      "residentMemoryBytes": 2480635904,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "3",
      "expected": [
        "3",
        "three"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 21061.0525420052,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 713
      },
      "parameterDtype": "torch.float16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 8857.65929199988,
      "inferenceMs": 8833.748042001389,
      "tokensPerSecond": 8.71657190514059,
      "inputTokens": 826,
      "outputTokens": 77,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 640,
      "inputHeight": 320,
      "visionGrid": [
        [
          1,
          40,
          80
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 14900543488,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 8861.186750000343,
      "totalMs": 29922.239292005543,
      "artifactRevision": "1de27d8c51f12e819435303b9e84c4e25ba8401e",
      "precision": "fp16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.1128966431236734,
      "grade": "good",
      "residentMemoryBytes": 2520481792,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "42",
      "expected": [
        "42"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 21061.0525420052,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 713
      },
      "parameterDtype": "torch.float16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 32457.96045899624,
      "inferenceMs": 32454.874374961946,
      "tokensPerSecond": 14.666518024399474,
      "inputTokens": 24,
      "outputTokens": 476,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": null,
      "inputHeight": null,
      "visionGrid": null,
      "inputModality": "text",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 18914705408,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 32459.377208026126,
      "totalMs": 53520.429750031326,
      "artifactRevision": "1de27d8c51f12e819435303b9e84c4e25ba8401e",
      "precision": "fp16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.03080908306802851,
      "grade": "moderate",
      "residentMemoryBytes": 2577825792,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "READY",
      "expected": [
        "ready"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 21061.0525420052,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 713
      },
      "parameterDtype": "torch.float16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 14168.662332987878,
      "inferenceMs": 14166.351666033734,
      "tokensPerSecond": 14.823859025292387,
      "inputTokens": 17,
      "outputTokens": 210,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": null,
      "inputHeight": null,
      "visionGrid": null,
      "inputModality": "text",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 18914836480,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 14170.783000008669,
      "totalMs": 35231.83554201387,
      "artifactRevision": "1de27d8c51f12e819435303b9e84c4e25ba8401e",
      "precision": "fp16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.07057829289020262,
      "grade": "moderate",
      "residentMemoryBytes": 2608087040,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "Dog",
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 21061.0525420052,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 713
      },
      "parameterDtype": "torch.float16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 5969.623000011779,
      "inferenceMs": 5952.222875028383,
      "tokensPerSecond": 7.392196314522276,
      "inputTokens": 828,
      "outputTokens": 44,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 19713851392,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 5978.216374991462,
      "totalMs": 27039.26891699666,
      "artifactRevision": "1de27d8c51f12e819435303b9e84c4e25ba8401e",
      "precision": "fp16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.16751476600750614,
      "grade": "good",
      "residentMemoryBytes": 2613280768,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "Dog",
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 19684.861749992706,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 713
      },
      "parameterDtype": "torch.float16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 5416.185041016433,
      "inferenceMs": 5377.04291602131,
      "tokensPerSecond": 8.182936362456518,
      "inputTokens": 828,
      "outputTokens": 44,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 12676612096,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 5433.36491700029,
      "totalMs": 25118.226666992996,
      "artifactRevision": "1de27d8c51f12e819435303b9e84c4e25ba8401e",
      "precision": "fp16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.18463180124516834,
      "grade": "good",
      "residentMemoryBytes": 2442002432,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "LOCAL AI 2026",
      "expected": [
        "localai2026"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 19684.861749992706,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 713
      },
      "parameterDtype": "torch.float16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 6543.261207989417,
      "inferenceMs": 6523.7243750016205,
      "tokensPerSecond": 11.03664040067329,
      "inputTokens": 793,
      "outputTokens": 72,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 768,
      "inputHeight": 256,
      "visionGrid": [
        [
          1,
          32,
          96
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 14028390400,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 6546.920291031711,
      "totalMs": 26231.782041024417,
      "artifactRevision": "1de27d8c51f12e819435303b9e84c4e25ba8401e",
      "precision": "fp16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.1528289897366447,
      "grade": "good",
      "residentMemoryBytes": 2483994624,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "3",
      "expected": [
        "3",
        "three"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 19684.861749992706,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 713
      },
      "parameterDtype": "torch.float16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 9496.606125030667,
      "inferenceMs": 9481.694250018336,
      "tokensPerSecond": 8.120911513240484,
      "inputTokens": 826,
      "outputTokens": 77,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 640,
      "inputHeight": 320,
      "visionGrid": [
        [
          1,
          40,
          80
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 14900543488,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 9500.49470900558,
      "totalMs": 29185.356458998285,
      "artifactRevision": "1de27d8c51f12e819435303b9e84c4e25ba8401e",
      "precision": "fp16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.10530077659683615,
      "grade": "good",
      "residentMemoryBytes": 2524151808,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "42",
      "expected": [
        "42"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 19684.861749992706,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 713
      },
      "parameterDtype": "torch.float16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 22982.966082985513,
      "inferenceMs": 22979.016625031363,
      "tokensPerSecond": 20.71455048609376,
      "inputTokens": 24,
      "outputTokens": 476,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": null,
      "inputHeight": null,
      "visionGrid": null,
      "inputModality": "text",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 18914705408,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 22984.96420797892,
      "totalMs": 42669.825957971625,
      "artifactRevision": "1de27d8c51f12e819435303b9e84c4e25ba8401e",
      "precision": "fp16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.043510484956087045,
      "grade": "moderate",
      "residentMemoryBytes": 2582052864,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "READY",
      "expected": [
        "ready"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 19684.861749992706,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 713
      },
      "parameterDtype": "torch.float16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 8963.388292002492,
      "inferenceMs": 8959.954124991782,
      "tokensPerSecond": 23.437620000112737,
      "inputTokens": 17,
      "outputTokens": 210,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": null,
      "inputHeight": null,
      "visionGrid": null,
      "inputModality": "text",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 18914836480,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 8964.92054197006,
      "totalMs": 28649.782291962765,
      "artifactRevision": "1de27d8c51f12e819435303b9e84c4e25ba8401e",
      "precision": "fp16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.11156495372316311,
      "grade": "good",
      "residentMemoryBytes": 2612314112,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "Dog",
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 19684.861749992706,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 713
      },
      "parameterDtype": "torch.float16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 4997.112833021674,
      "inferenceMs": 4973.179874999914,
      "tokensPerSecond": 8.847457985822553,
      "inputTokens": 828,
      "outputTokens": 44,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 19713851392,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 5005.91574999271,
      "totalMs": 24690.777499985415,
      "artifactRevision": "1de27d8c51f12e819435303b9e84c4e25ba8401e",
      "precision": "fp16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.2001155534035272,
      "grade": "good",
      "residentMemoryBytes": 2617393152,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "Dog",
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 15105.721917003393,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 713
      },
      "parameterDtype": "torch.float16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 6408.978249994107,
      "inferenceMs": 6380.013666988816,
      "tokensPerSecond": 6.896536950643672,
      "inputTokens": 828,
      "outputTokens": 44,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 12676612096,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 6421.450333029497,
      "totalMs": 21527.17225003289,
      "artifactRevision": "1de27d8c51f12e819435303b9e84c4e25ba8401e",
      "precision": "fp16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.15603111151156107,
      "grade": "good",
      "residentMemoryBytes": 2446721024,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "LOCAL AI 2026",
      "expected": [
        "localai2026"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 15105.721917003393,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 713
      },
      "parameterDtype": "torch.float16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 8235.053249984048,
      "inferenceMs": 8215.636208013166,
      "tokensPerSecond": 8.763776556923785,
      "inputTokens": 793,
      "outputTokens": 72,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 768,
      "inputHeight": 256,
      "visionGrid": [
        [
          1,
          32,
          96
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 13374078976,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 8238.53762500221,
      "totalMs": 23344.259542005602,
      "artifactRevision": "1de27d8c51f12e819435303b9e84c4e25ba8401e",
      "precision": "fp16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.12143212310156429,
      "grade": "good",
      "residentMemoryBytes": 2489729024,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "3",
      "expected": [
        "3",
        "three"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 15105.721917003393,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 713
      },
      "parameterDtype": "torch.float16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 6896.21120900847,
      "inferenceMs": 6879.89674997516,
      "tokensPerSecond": 11.192028426920508,
      "inputTokens": 826,
      "outputTokens": 77,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 640,
      "inputHeight": 320,
      "visionGrid": [
        [
          1,
          40,
          80
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 14900543488,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 6898.716541996691,
      "totalMs": 22004.438459000085,
      "artifactRevision": "1de27d8c51f12e819435303b9e84c4e25ba8401e",
      "precision": "fp16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.14500715968410413,
      "grade": "good",
      "residentMemoryBytes": 2529329152,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "42",
      "expected": [
        "42"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 15105.721917003393,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 713
      },
      "parameterDtype": "torch.float16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 27647.85262499936,
      "inferenceMs": 27644.853041972965,
      "tokensPerSecond": 17.218395021933844,
      "inputTokens": 24,
      "outputTokens": 476,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": null,
      "inputHeight": null,
      "visionGrid": null,
      "inputModality": "text",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 18914705408,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 27650.592208025046,
      "totalMs": 42756.31412502844,
      "artifactRevision": "1de27d8c51f12e819435303b9e84c4e25ba8401e",
      "precision": "fp16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.036169174277780755,
      "grade": "moderate",
      "residentMemoryBytes": 2587066368,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "READY",
      "expected": [
        "ready"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 15105.721917003393,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 713
      },
      "parameterDtype": "torch.float16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 8097.982249979395,
      "inferenceMs": 8094.311750028282,
      "tokensPerSecond": 25.944145281934098,
      "inputTokens": 17,
      "outputTokens": 210,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": null,
      "inputHeight": null,
      "visionGrid": null,
      "inputModality": "text",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 18914836480,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 8099.707124987617,
      "totalMs": 23205.42904199101,
      "artifactRevision": "1de27d8c51f12e819435303b9e84c4e25ba8401e",
      "precision": "fp16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.12348755148266032,
      "grade": "good",
      "residentMemoryBytes": 2617442304,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "acceptedAnswer": true,
      "finalAnswer": "Dog",
      "expected": [
        "dog",
        "labrador",
        "puppy",
        "canine"
      ],
      "repetitive": false,
      "eosWithinBudget": true,
      "method": "Final answer semantic label/exact normalized OCR; complete nonrepetitive response; HF EOS before budget",
      "performancePassed": true,
      "maximumRequestSeconds": 45,
      "performanceGateVersion": 2,
      "metric": "Complete response latency; token throughput retained as diagnostic"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 15105.721917003393,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 713
      },
      "parameterDtype": "torch.float16",
      "attentionImplementation": "sdpa",
      "accelerationDetails": "Model parameters and input tensors on selected device; image decoding/processing and text tokenization/decoding on CPU",
      "durationMs": 5454.548167006578,
      "inferenceMs": 5433.968666999135,
      "tokensPerSecond": 8.09721268126094,
      "inputTokens": 828,
      "outputTokens": 44,
      "maxNewTokens": 1024,
      "minPixels": 786432,
      "maxPixels": 1048576,
      "inputWidth": 700,
      "inputHeight": 577,
      "visionGrid": [
        [
          1,
          52,
          62
        ]
      ],
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 19713851392,
      "memoryMeasurementScope": "Post-inference MPS driver allocation, not peak",
      "requestMs": 5464.136000024155,
      "totalMs": 20569.857917027548,
      "artifactRevision": "1de27d8c51f12e819435303b9e84c4e25ba8401e",
      "precision": "fp16",
      "cpuThreads": 4,
      "fallback": false,
      "requestsPerSecond": 0.18333324216454647,
      "grade": "good",
      "residentMemoryBytes": 2623209472,
      "residentMemoryScope": "Post-inference process RSS, not peak"
    }
  }
]

Something not quite right?

Tell us what happened with Qwen3-VL. Loading, results, or the page itself—we’re here for all of it.

Attachments Optional

Up to 3 files · 50 MB each

Browser, OS, CPU/GPU, memory, screen, storage, and network details are included automatically. Model inputs, results, and browsing history are not collected.