A model is still running

A model is still running or downloading. Leaving will stop the current operation. Stay on this page to let it finish.

Skip to content
LocalAI.toolsBeta

Join the waitlist

Get an email when the new apps and model runs launch.

No account needed.

Granite 4.0

Local chat, drafting and text extraction using IBM Granite 4.0 Micro.

Desktop inferencePrivate · Runs on your computer
About this model

Local chat, drafting and text extraction using IBM Granite 4.0 Micro.

Created by
IBM Granite
License
Apache-2.0
Model released
Oct 2, 2025
Source repository created
Sep 16, 2025
Model weights · FP16
6.8 GB

Best for

  • Local text chat and drafting
  • Short text extraction and translation

Keep in mind

  • May hallucinate facts, produce unsafe content or fail instructions; review generated text.
  • Input plus output budget is limited to8192 tokens by this adapter; long-document quality is not validated.
  • Upstream supports12 languages; local quality coverage is short English/French/Spanish prompts only.
  • Tools are not executed; this adapter exposes text messages, not structured tool-call parsing.
  • Requires approximately6.81GB of weights plus runtime downloads; memory and speed depend on hardware.
  • If no system message is supplied, the adapter adds an explicit instruction-following prompt. The original upstream default added commentary on the date-only fixture; baseline failure is retained. User system messages take precedence.

Processed on your deviceYour input is processed locally in the desktop app. Model and runtime files download from external hosts.

Target platforms · verification required

  • macos · aarch64, x86_64
  • windows · x86_64

Recommended configuration

FP16 · mps (verified platform only)

Sources & technical details
Original model
https://huggingface.co/ibm-granite/granite-4.0-micro
Release verification
Verified
Release evidence
Official original checkpoint card explicitly states October 2nd, 2025; repository creation is earlier and distinct.
Source updated
Nov 3, 2025
Native model files
ibm-granite/granite-4.0-micro
Files updated
Nov 3, 2025
Artifact revision
56111ae135df9c53a78c99028e7bc24035a9e979

Desktop app

Sample result

Granite 4.0 Micro Desktop

Run Granite 4.0 on your computer

Download the desktop app to load this model, provide input and view results locally.

Public desktop downloads are coming soon.

Device requirements

macOS Apple SiliconCPU / FP32Tested24.0 GiBRAM estimate
macOS Apple SiliconMPS / FP16Tested16.0 GiBRAM estimate
Windows Intel / AMDCPU / FP32Unverified24.0 GiBRAM estimate
macOS Intel / AMDCPU / FP32Unverified24.0 GiBRAM estimate
If your device falls short

MemoryAdvisory only: close other apps, reduce image/tile/token/batch settings or choose a smaller verified model/precision. CPU can need more RAM than GPU.

GPUIf the selected GPU runtime fails, existing CPU fallback may recover when enabled, but CPU has its own memory/runtime needs.

PlatformUse a compatible OS/architecture/runtime or another model. These estimates do not add hardware detection or block runs.

Benchmark evidence

Saved desktop measurements apply to the recorded hardware and runtime. Fallback cases report the device that actually ran; they do not prove GPU support on the requested device.

Granite 4.0 Micro Desktop

77 configurations tested
Passed77
Failed0
TestedNov 4, 2026
Runtimetransformers 5.22.1

Quality criteria

No explicit quality gate recorded. Review per-case evidence; full certification is pending.

PrecisionDeviceResultPlatformInference / load
FP16mpsAcceleratedPassedFastmacos · aarch64317 ms / 1678 ms
FP16mpsAcceleratedPassedFastmacos · aarch64318 ms / 1635 ms
FP16mpsAcceleratedPassedFastmacos · aarch64323 ms / 1698 ms
FP16mpsAcceleratedPassedFastmacos · aarch64346 ms / 1698 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64353 ms / 1678 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64357 ms / 1635 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64143 ms / 1698 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64143 ms / 1635 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64143 ms / 1698 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64143 ms / 1678 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64144 ms / 1698 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64145 ms / 1678 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64145 ms / 1635 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64147 ms / 1635 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64148 ms / 1678 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64167 ms / 1698 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64167 ms / 1698 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64167 ms / 1635 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64168 ms / 1635 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64168 ms / 1678 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64168 ms / 1678 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64186 ms / 1698 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64189 ms / 1635 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64191 ms / 1678 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64197 ms / 1698 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64198 ms / 1698 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64200 ms / 1678 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64200 ms / 1635 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64201 ms / 1635 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64202 ms / 1698 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64205 ms / 1635 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64211 ms / 1678 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64237 ms / 1678 ms
FP32cpuPassedModeratemacos · aarch641309 ms / 1354 ms
FP32cpuPassedModeratemacos · aarch641319 ms / 1300 ms
FP32cpuPassedModeratemacos · aarch641351 ms / 1311 ms
FP32cpuPassedModeratemacos · aarch641369 ms / 4598 ms
FP32cpuPassedModeratemacos · aarch641376 ms / 1354 ms
FP32cpuPassedModeratemacos · aarch641396 ms / 4598 ms
FP32cpuPassedModeratemacos · aarch641417 ms / 1311 ms
FP32cpuPassedModeratemacos · aarch641418 ms / 1300 ms
FP32cpuPassedModeratemacos · aarch64534 ms / 1300 ms
FP32cpuPassedModeratemacos · aarch64535 ms / 1311 ms
FP32cpuPassedModeratemacos · aarch64539 ms / 1300 ms
FP32cpuPassedModeratemacos · aarch64540 ms / 1311 ms
FP32cpuPassedModeratemacos · aarch64542 ms / 1354 ms
FP32cpuPassedModeratemacos · aarch64544 ms / 1354 ms
FP32cpuPassedModeratemacos · aarch64548 ms / 1300 ms
FP32cpuPassedModeratemacos · aarch64549 ms / 1354 ms
FP32cpuPassedModeratemacos · aarch64553 ms / 1300 ms
FP32cpuPassedModeratemacos · aarch64556 ms / 4598 ms
FP32cpuPassedModeratemacos · aarch64563 ms / 4598 ms
FP32cpuPassedModeratemacos · aarch64574 ms / 1311 ms
FP32cpuPassedModeratemacos · aarch64576 ms / 1354 ms
FP32cpuPassedModeratemacos · aarch64595 ms / 4598 ms
FP32cpuPassedModeratemacos · aarch64598 ms / 4598 ms
FP32cpuPassedModeratemacos · aarch64599 ms / 1311 ms
FP32cpuPassedModeratemacos · aarch64599 ms / 1300 ms
FP32cpuPassedModeratemacos · aarch64601 ms / 1311 ms
FP32cpuPassedModeratemacos · aarch64605 ms / 1354 ms
FP32cpuPassedModeratemacos · aarch64608 ms / 1300 ms
FP32cpuPassedModeratemacos · aarch64610 ms / 1300 ms
FP32cpuPassedModeratemacos · aarch64611 ms / 1354 ms
FP32cpuPassedModeratemacos · aarch64612 ms / 1300 ms
FP32cpuPassedModeratemacos · aarch64612 ms / 1354 ms
FP32cpuPassedModeratemacos · aarch64614 ms / 1311 ms
FP32cpuPassedModeratemacos · aarch64614 ms / 1354 ms
FP32cpuPassedModeratemacos · aarch64614 ms / 1311 ms
FP32cpuPassedModeratemacos · aarch64619 ms / 1354 ms
FP32cpuPassedModeratemacos · aarch64624 ms / 1300 ms
FP32cpuPassedModeratemacos · aarch64625 ms / 1311 ms
FP32cpuPassedModeratemacos · aarch64627 ms / 1311 ms
FP32cpuPassedModeratemacos · aarch64633 ms / 4598 ms
FP32cpuPassedModeratemacos · aarch64641 ms / 4598 ms
FP32cpuPassedModeratemacos · aarch64646 ms / 4598 ms
FP32cpuPassedModeratemacos · aarch64673 ms / 4598 ms
FP32cpuPassedModeratemacos · aarch64688 ms / 4598 ms
Test setup & sample details
Quality gate
No explicit quality gate recorded. Review the per-case quality evidence; full certification is pending.
Tested
2026-11-04T10:38:04.717566+00:00
Environment
{ "os": "macos", "arch": "aarch64", "hardware": "Apple M3 Max", "osVersion": "27.2" }
Runtime
transformers 5.22.1
Artifact revision
56111ae135df9c53a78c99028e7bc24035a9e979
Fixture
{
  "file": "model-samples/desktop/granite4/granite-4.0-micro-3b-native-capital.json",
  "size": 244,
  "sha1": "9013b4f5a71cd1280d78b3dfa751ead1186d071f",
  "sha256": "8ff34789fa2a0f5bbef2080057f4b36376d3c73ea64baccefb6caec96eafbd68",
  "url": "https://assets.localai.tools/samples/v1/files/8ff34789fa2a0f5bbef2080057f4b36376d3c73ea64baccefb6caec96eafbd68/granite-4.0-micro-3b-native-capital.json"
}
Rerun when
Artifact/runtime revision; Input/generation processing; Fixture/quality gates; OS/backend behavior
Per-case quality & failure details
[
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "Paris"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 4597.526416997425,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 604.9998329835944,
      "inferenceMs": 594.7437499999069,
      "tokensPerSecond": 3.3627927994204447,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp32",
      "cpuThreads": 1,
      "rssBytes": 14061158400,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 5202.52624998102,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "56"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 4597.526416997425,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 598.7748749903403,
      "inferenceMs": 598.3119160227943,
      "tokensPerSecond": 3.3427380375352653,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp32",
      "cpuThreads": 1,
      "rssBytes": 14065467392,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 5196.301291987766,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 4597.526416997425,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 673.2560409873258,
      "inferenceMs": 672.8133749857079,
      "tokensPerSecond": 2.9725925113223033,
      "inputTokens": 68,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp32",
      "cpuThreads": 1,
      "rssBytes": 14072266752,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 5270.782457984751,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "hola"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 4597.526416997425,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 688.9472909970209,
      "inferenceMs": 688.3237499860115,
      "tokensPerSecond": 2.9056094607234533,
      "inputTokens": 69,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp32",
      "cpuThreads": 1,
      "rssBytes": 14080163840,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 5286.473707994446,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "2026-11-04"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 4597.526416997425,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 1369.7359159996267,
      "inferenceMs": 1369.301375001669,
      "tokensPerSecond": 5.112095940158877,
      "inputTokens": 72,
      "outputTokens": 7,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp32",
      "cpuThreads": 1,
      "rssBytes": 14084915200,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 5967.262332997052,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "Unknown"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 4597.526416997425,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 641.4512920018751,
      "inferenceMs": 641.05129201198,
      "tokensPerSecond": 3.1198751565149703,
      "inputTokens": 74,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp32",
      "cpuThreads": 1,
      "rssBytes": 14096252928,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 5238.9777089993,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "Paris"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 4597.526416997425,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 563.3985000022221,
      "inferenceMs": 563.0373330204748,
      "tokensPerSecond": 3.5521623215831593,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp32",
      "cpuThreads": 1,
      "rssBytes": 14096269312,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 5160.924916999647,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "56"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 4597.526416997425,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 556.2327920051757,
      "inferenceMs": 555.8991660072934,
      "tokensPerSecond": 3.5977747805683165,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp32",
      "cpuThreads": 1,
      "rssBytes": 14096269312,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 5153.759209002601,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 4597.526416997425,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 646.3567079917993,
      "inferenceMs": 645.9092920122202,
      "tokensPerSecond": 3.0964100141203126,
      "inputTokens": 68,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp32",
      "cpuThreads": 1,
      "rssBytes": 14096269312,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 5243.883124989225,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "hola"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 4597.526416997425,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 632.995584019227,
      "inferenceMs": 632.617041002959,
      "tokensPerSecond": 3.16147032149051,
      "inputTokens": 69,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp32",
      "cpuThreads": 1,
      "rssBytes": 14098956288,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 5230.522001016652,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "2026-11-04"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 4597.526416997425,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 1396.2768329947721,
      "inferenceMs": 1395.8155839936808,
      "tokensPerSecond": 5.014989143459578,
      "inputTokens": 72,
      "outputTokens": 7,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp32",
      "cpuThreads": 1,
      "rssBytes": 14098956288,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 5993.803249992197,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "Paris"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1310.5549169995356,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 583.8928329758346,
      "inferenceMs": 574.2461670015473,
      "tokensPerSecond": 3.4828269040838213,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp32",
      "cpuThreads": 4,
      "rssBytes": 14055866368,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1894.4477499753702,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "56"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1310.5549169995356,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 614.3457500147633,
      "inferenceMs": 613.9265839883592,
      "tokensPerSecond": 3.2577185157988247,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp32",
      "cpuThreads": 4,
      "rssBytes": 14069399552,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1924.900667014299,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1310.5549169995356,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 626.9493330037221,
      "inferenceMs": 626.5720420051366,
      "tokensPerSecond": 3.191971339161035,
      "inputTokens": 68,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp32",
      "cpuThreads": 4,
      "rssBytes": 14077362176,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1937.5042500032578,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "hola"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1310.5549169995356,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 625.2870000025723,
      "inferenceMs": 624.8886669927742,
      "tokensPerSecond": 3.200570126555242,
      "inputTokens": 69,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp32",
      "cpuThreads": 4,
      "rssBytes": 14083457024,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1935.841917002108,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "2026-11-04"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1310.5549169995356,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 1351.646916999016,
      "inferenceMs": 1351.2178749952,
      "tokensPerSecond": 5.18051169210951,
      "inputTokens": 72,
      "outputTokens": 7,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp32",
      "cpuThreads": 4,
      "rssBytes": 14090059776,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 2662.2018339985516,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "Unknown"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1310.5549169995356,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 614.6077919984236,
      "inferenceMs": 614.2186670040246,
      "tokensPerSecond": 3.256169353750519,
      "inputTokens": 74,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp32",
      "cpuThreads": 4,
      "rssBytes": 14096384000,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1925.1627089979593,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "Paris"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1310.5549169995356,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 539.8697500058915,
      "inferenceMs": 539.5069590013009,
      "tokensPerSecond": 3.7070884195864053,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp32",
      "cpuThreads": 4,
      "rssBytes": 14096384000,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1850.4246670054272,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "56"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1310.5549169995356,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 535.4864169785287,
      "inferenceMs": 535.1558749971446,
      "tokensPerSecond": 3.7372288961803353,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp32",
      "cpuThreads": 4,
      "rssBytes": 14096384000,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1846.0413339780644,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1310.5549169995356,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 601.2195000075735,
      "inferenceMs": 600.8753750065807,
      "tokensPerSecond": 3.3284772237139792,
      "inputTokens": 68,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp32",
      "cpuThreads": 4,
      "rssBytes": 14096416768,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1911.7744170071092,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "hola"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1310.5549169995356,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 599.3902499903925,
      "inferenceMs": 599.0102089999709,
      "tokensPerSecond": 3.3388412583801172,
      "inputTokens": 69,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp32",
      "cpuThreads": 4,
      "rssBytes": 14096695296,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1909.945166989928,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "2026-11-04"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1310.5549169995356,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 1416.9614580168854,
      "inferenceMs": 1416.6262499929871,
      "tokensPerSecond": 4.941317443492702,
      "inputTokens": 72,
      "outputTokens": 7,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp32",
      "cpuThreads": 4,
      "rssBytes": 14096695296,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 2727.516375016421,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "Paris"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1299.7478340112139,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 548.2074579922482,
      "inferenceMs": 538.9182500075549,
      "tokensPerSecond": 3.7111380065009167,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp32",
      "cpuThreads": 4,
      "rssBytes": 14053965824,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1847.955292003462,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "56"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1299.7478340112139,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 548.6775409954134,
      "inferenceMs": 548.1935419957153,
      "tokensPerSecond": 3.648346517762575,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp32",
      "cpuThreads": 4,
      "rssBytes": 14058192896,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1848.4253750066273,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1299.7478340112139,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 624.5060410001315,
      "inferenceMs": 624.1044170164969,
      "tokensPerSecond": 3.2045919648524683,
      "inputTokens": 68,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp32",
      "cpuThreads": 4,
      "rssBytes": 14068514816,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1924.2538750113454,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "hola"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1299.7478340112139,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 612.2970000142232,
      "inferenceMs": 611.8591249978635,
      "tokensPerSecond": 3.268726277485824,
      "inputTokens": 69,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp32",
      "cpuThreads": 4,
      "rssBytes": 14074576896,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1912.044834025437,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "2026-11-04"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1299.7478340112139,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 1319.8480420105625,
      "inferenceMs": 1319.473832991207,
      "tokensPerSecond": 5.305144994146048,
      "inputTokens": 72,
      "outputTokens": 7,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp32",
      "cpuThreads": 4,
      "rssBytes": 14084325376,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 2619.5958760217763,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "Unknown"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1299.7478340112139,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 608.1316669879016,
      "inferenceMs": 607.7278750017285,
      "tokensPerSecond": 3.290946626389832,
      "inputTokens": 74,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp32",
      "cpuThreads": 4,
      "rssBytes": 14092369920,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1907.8795009991154,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "Paris"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1299.7478340112139,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 553.8077499950305,
      "inferenceMs": 553.4598340163939,
      "tokensPerSecond": 3.6136316984129304,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp32",
      "cpuThreads": 4,
      "rssBytes": 14092337152,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1853.5555840062443,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "56"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1299.7478340112139,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 534.8343749938067,
      "inferenceMs": 534.4896249880549,
      "tokensPerSecond": 3.7418874127719457,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp32",
      "cpuThreads": 4,
      "rssBytes": 14092337152,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1834.5822090050206,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1299.7478340112139,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 599.478292017011,
      "inferenceMs": 599.0963749936782,
      "tokensPerSecond": 3.338361044199449,
      "inputTokens": 68,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp32",
      "cpuThreads": 4,
      "rssBytes": 14092337152,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1899.2261260282248,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "hola"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1299.7478340112139,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 610.735583002679,
      "inferenceMs": 610.3197500051465,
      "tokensPerSecond": 3.2769708009336664,
      "inputTokens": 69,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp32",
      "cpuThreads": 4,
      "rssBytes": 14092337152,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1910.483417013893,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "2026-11-04"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1299.7478340112139,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 1418.831582996063,
      "inferenceMs": 1418.4050830081105,
      "tokensPerSecond": 4.9351204982674,
      "inputTokens": 72,
      "outputTokens": 7,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp32",
      "cpuThreads": 4,
      "rssBytes": 14092353536,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 2718.579417007277,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "Paris"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1354.042249993654,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 551.7477910034359,
      "inferenceMs": 542.1407080139033,
      "tokensPerSecond": 3.6890791826477445,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp32",
      "cpuThreads": 4,
      "rssBytes": 14053277696,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1905.7900409970898,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "56"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1354.042249993654,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 576.3487500080373,
      "inferenceMs": 575.9893330105115,
      "tokensPerSecond": 3.47228652577061,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp32",
      "cpuThreads": 4,
      "rssBytes": 14055391232,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1930.3910000016913,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1354.042249993654,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 619.2984159861226,
      "inferenceMs": 618.8890000048559,
      "tokensPerSecond": 3.2315972653970384,
      "inputTokens": 68,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp32",
      "cpuThreads": 4,
      "rssBytes": 14076067840,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1973.3406659797765,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "hola"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1354.042249993654,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 605.3586670022923,
      "inferenceMs": 604.9653340014629,
      "tokensPerSecond": 3.305974553568657,
      "inputTokens": 69,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp32",
      "cpuThreads": 4,
      "rssBytes": 14081310720,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1959.4009169959463,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "2026-11-04"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1354.042249993654,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 1309.3345830275211,
      "inferenceMs": 1308.965125004761,
      "tokensPerSecond": 5.34773605979345,
      "inputTokens": 72,
      "outputTokens": 7,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp32",
      "cpuThreads": 4,
      "rssBytes": 14086062080,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 2663.376833021175,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "Unknown"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1354.042249993654,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 611.0214999935124,
      "inferenceMs": 610.6169160048012,
      "tokensPerSecond": 3.2753760133043452,
      "inputTokens": 74,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp32",
      "cpuThreads": 4,
      "rssBytes": 14093942784,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1965.0637499871664,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "Paris"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1354.042249993654,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 548.9256670116447,
      "inferenceMs": 548.5876670281868,
      "tokensPerSecond": 3.645725414926688,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp32",
      "cpuThreads": 4,
      "rssBytes": 14093991936,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1902.9679170052987,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "56"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1354.042249993654,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 544.4394580263179,
      "inferenceMs": 544.0798329946119,
      "tokensPerSecond": 3.675931138619884,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp32",
      "cpuThreads": 4,
      "rssBytes": 14093991936,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1898.4817080199718,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1354.042249993654,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 612.6240420271643,
      "inferenceMs": 612.267916003475,
      "tokensPerSecond": 3.2665438572297307,
      "inputTokens": 68,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp32",
      "cpuThreads": 4,
      "rssBytes": 14093991936,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1966.6662920208182,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "hola"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1354.042249993654,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 614.4819160108455,
      "inferenceMs": 614.1169159964193,
      "tokensPerSecond": 3.256708857718001,
      "inputTokens": 69,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp32",
      "cpuThreads": 4,
      "rssBytes": 14095450112,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1968.5241660044994,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "2026-11-04"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1354.042249993654,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 1376.1902500118595,
      "inferenceMs": 1375.8320409979206,
      "tokensPerSecond": 5.087830339321615,
      "inputTokens": 72,
      "outputTokens": 7,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp32",
      "cpuThreads": 4,
      "rssBytes": 14095450112,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 2730.2325000055134,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "Paris"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1697.501624992583,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 322
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 196.6235419968143,
      "inferenceMs": 185.52470800932497,
      "tokensPerSecond": 10.780235266018986,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 7570833408,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp16",
      "cpuThreads": 4,
      "rssBytes": 2113503232,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1894.1251669893973,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "56"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1697.501624992583,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 322
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 145.5222080112435,
      "inferenceMs": 144.35595902614295,
      "tokensPerSecond": 13.854641079540047,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 7570817024,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp16",
      "cpuThreads": 4,
      "rssBytes": 2114371584,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1843.0238330038264,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1697.501624992583,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 322
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 203.40866700280458,
      "inferenceMs": 201.74183300696313,
      "tokensPerSecond": 9.91366029637973,
      "inputTokens": 68,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 7580745728,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp16",
      "cpuThreads": 4,
      "rssBytes": 2125643776,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1900.9102919953875,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "hola"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1697.501624992583,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 322
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 199.2749590135645,
      "inferenceMs": 198.04854100220837,
      "tokensPerSecond": 10.098534378890974,
      "inputTokens": 69,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 7582334976,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp16",
      "cpuThreads": 4,
      "rssBytes": 2135425024,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1896.7765840061475,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "2026-11-04"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1697.501624992583,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 322
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 347.57487499155104,
      "inferenceMs": 345.96283300197683,
      "tokensPerSecond": 20.233387324470204,
      "inputTokens": 72,
      "outputTokens": 7,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 7584055296,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp16",
      "cpuThreads": 4,
      "rssBytes": 2145140736,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 2045.076499984134,
      "grade": "fast",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "Unknown"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1697.501624992583,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 322
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 198.3077499899082,
      "inferenceMs": 197.13629101170227,
      "tokensPerSecond": 10.145265439133565,
      "inputTokens": 74,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 7585873920,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp16",
      "cpuThreads": 4,
      "rssBytes": 2154627072,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1895.8093749824911,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "Paris"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1697.501624992583,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 322
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 144.55950001138262,
      "inferenceMs": 143.3369159931317,
      "tokensPerSecond": 13.95313960916973,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 7585873920,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp16",
      "cpuThreads": 4,
      "rssBytes": 2154971136,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1842.0611250039656,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "56"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1697.501624992583,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 322
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 144.5252090052236,
      "inferenceMs": 142.8537920000963,
      "tokensPerSecond": 14.00032839169332,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 7585873920,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp16",
      "cpuThreads": 4,
      "rssBytes": 2155298816,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1842.0268339978065,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1697.501624992583,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 322
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 167.80179098714143,
      "inferenceMs": 166.64208300062455,
      "tokensPerSecond": 12.001770285075615,
      "inputTokens": 68,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 7585873920,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp16",
      "cpuThreads": 4,
      "rssBytes": 2155511808,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1865.3034159797244,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "hola"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1697.501624992583,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 322
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 168.29345800215378,
      "inferenceMs": 166.95145799894817,
      "tokensPerSecond": 11.9795300021435,
      "inputTokens": 69,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 7585873920,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp16",
      "cpuThreads": 4,
      "rssBytes": 2155855872,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1865.7950829947367,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "2026-11-04"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1697.501624992583,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 322
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 324.51579201733693,
      "inferenceMs": 322.628750000149,
      "tokensPerSecond": 21.69676446998839,
      "inputTokens": 72,
      "outputTokens": 7,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 7585873920,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp16",
      "cpuThreads": 4,
      "rssBytes": 2156544000,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 2022.0174170099199,
      "grade": "fast",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "Paris"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1635.3245840000454,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 322
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 199.96629198431037,
      "inferenceMs": 188.96491700434126,
      "tokensPerSecond": 10.583975225168661,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 7570833408,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp16",
      "cpuThreads": 4,
      "rssBytes": 2110996480,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1835.2908759843558,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "56"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1635.3245840000454,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 322
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 144.64216699707322,
      "inferenceMs": 143.0024580040481,
      "tokensPerSecond": 13.98577358679656,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 7570833408,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp16",
      "cpuThreads": 4,
      "rssBytes": 2111913984,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1779.9667509971187,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1635.3245840000454,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 322
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 202.43075001053512,
      "inferenceMs": 200.85512500372715,
      "tokensPerSecond": 9.957425781208656,
      "inputTokens": 68,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 7580745728,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp16",
      "cpuThreads": 4,
      "rssBytes": 2122743808,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1837.7553340105806,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "hola"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1635.3245840000454,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 322
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 201.52783297817223,
      "inferenceMs": 199.88599998760037,
      "tokensPerSecond": 10.005703251473674,
      "inputTokens": 69,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 7582334976,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp16",
      "cpuThreads": 4,
      "rssBytes": 2132148224,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1836.8524169782177,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "2026-11-04"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1635.3245840000454,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 322
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 358.4991250245366,
      "inferenceMs": 356.81395800202154,
      "tokensPerSecond": 19.61806662271979,
      "inputTokens": 72,
      "outputTokens": 7,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 7584055296,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp16",
      "cpuThreads": 4,
      "rssBytes": 2141831168,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1993.823709024582,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "Unknown"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1635.3245840000454,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 322
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 206.46887499606237,
      "inferenceMs": 204.67158398241736,
      "tokensPerSecond": 9.771752194832347,
      "inputTokens": 74,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 7585873920,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp16",
      "cpuThreads": 4,
      "rssBytes": 2151071744,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1841.7934589961078,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "Paris"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1635.3245840000454,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 322
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 148.67225001216866,
      "inferenceMs": 146.939416998066,
      "tokensPerSecond": 13.611051689597513,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 7585873920,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp16",
      "cpuThreads": 4,
      "rssBytes": 2151366656,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1783.996834012214,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "56"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1635.3245840000454,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 322
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 146.70850001857616,
      "inferenceMs": 145.05137500236742,
      "tokensPerSecond": 13.788218139727096,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 7585873920,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp16",
      "cpuThreads": 4,
      "rssBytes": 2151596032,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1782.0330840186216,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1635.3245840000454,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 322
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 170.00762501265854,
      "inferenceMs": 167.72308299550787,
      "tokensPerSecond": 11.924417106341684,
      "inputTokens": 68,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 7585873920,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp16",
      "cpuThreads": 4,
      "rssBytes": 2151874560,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1805.332209012704,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "hola"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1635.3245840000454,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 322
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 169.15341702406295,
      "inferenceMs": 167.4354170099832,
      "tokensPerSecond": 11.944904105209423,
      "inputTokens": 69,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 7585873920,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp16",
      "cpuThreads": 4,
      "rssBytes": 2152202240,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1804.4780010241084,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "2026-11-04"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1635.3245840000454,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 322
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 319.94579199817963,
      "inferenceMs": 318.2375839969609,
      "tokensPerSecond": 21.996144867875973,
      "inputTokens": 72,
      "outputTokens": 7,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 7585873920,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp16",
      "cpuThreads": 4,
      "rssBytes": 2152792064,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1955.270375998225,
      "grade": "fast",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "Paris"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1677.6497910032049,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 322
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 202.12183400872163,
      "inferenceMs": 191.03404198540375,
      "tokensPerSecond": 10.469338235291138,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 7570833408,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp16",
      "cpuThreads": 4,
      "rssBytes": 2112946176,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1879.7716250119265,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "56"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1677.6497910032049,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 322
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 150.36466700257733,
      "inferenceMs": 148.35137501358986,
      "tokensPerSecond": 13.481506321170183,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 7570817024,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp16",
      "cpuThreads": 4,
      "rssBytes": 2114093056,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1828.0144580057822,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1677.6497910032049,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 322
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 239.3982089997735,
      "inferenceMs": 237.19783299020492,
      "tokensPerSecond": 8.43178023503524,
      "inputTokens": 68,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 7580745728,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp16",
      "cpuThreads": 4,
      "rssBytes": 2124644352,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1917.0480000029784,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "hola"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1677.6497910032049,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 322
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 212.91716600535437,
      "inferenceMs": 211.3567090127617,
      "tokensPerSecond": 9.462675726462225,
      "inputTokens": 69,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 7582334976,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp16",
      "cpuThreads": 4,
      "rssBytes": 2134589440,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1890.5669570085593,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "2026-11-04"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1677.6497910032049,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 322
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 354.79137499351054,
      "inferenceMs": 352.5634169927798,
      "tokensPerSecond": 19.854584062371263,
      "inputTokens": 72,
      "outputTokens": 7,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 7584055296,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp16",
      "cpuThreads": 4,
      "rssBytes": 2144796672,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 2032.4411659967154,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "Unknown"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1677.6497910032049,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 322
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 200.72204098687507,
      "inferenceMs": 199.5894580031745,
      "tokensPerSecond": 10.020569322695337,
      "inputTokens": 74,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 7585873920,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp16",
      "cpuThreads": 4,
      "rssBytes": 2154266624,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1878.37183199008,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "Paris"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1677.6497910032049,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 322
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 145.79770900309086,
      "inferenceMs": 144.51187499798834,
      "tokensPerSecond": 13.839693104998055,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 7585873920,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp16",
      "cpuThreads": 4,
      "rssBytes": 2154577920,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1823.4475000062957,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "56"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1677.6497910032049,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 322
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 144.72083299187943,
      "inferenceMs": 143.38570900144987,
      "tokensPerSecond": 13.9483914675191,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 7585873920,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp16",
      "cpuThreads": 4,
      "rssBytes": 2154856448,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1822.3706239950843,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1677.6497910032049,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 322
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 169.23595900880173,
      "inferenceMs": 168.0347920046188,
      "tokensPerSecond": 11.902296995404534,
      "inputTokens": 68,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 7585873920,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp16",
      "cpuThreads": 4,
      "rssBytes": 2155151360,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1846.8857500120066,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "hola"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1677.6497910032049,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 322
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 170.19470798550174,
      "inferenceMs": 168.23483299231157,
      "tokensPerSecond": 11.888144472978443,
      "inputTokens": 69,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 7585873920,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp16",
      "cpuThreads": 4,
      "rssBytes": 2155380736,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1847.8444989887066,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "passed": true,
      "expected": "2026-11-04"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1677.6497910032049,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 322
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 318.5505839937832,
      "inferenceMs": 316.9354590063449,
      "tokensPerSecond": 22.086515727670168,
      "inputTokens": 72,
      "outputTokens": 7,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 7585873920,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "artifactRevision": "56111ae135df9c53a78c99028e7bc24035a9e979",
      "precision": "fp16",
      "cpuThreads": 4,
      "rssBytes": 2156036096,
      "memoryScope": "Post-inference helper RSS, not peak/minimum",
      "totalMs": 1996.200374996988,
      "grade": "fast",
      "loadTimingScope": "Local immutable checkpoint load and one-token backend warmup; no network/provisioning timings"
    }
  }
]

Granite 4.0 Dense 1B

48 configurations tested
Passed42
Failed6
TestedNov 7, 2026
Runtimetransformers 5.22.1

Quality criteria

No explicit quality gate recorded. Review per-case evidence; full certification is pending.

PrecisionDeviceResultPlatformInference / load
BF16mpsAcceleratedPassedFastmacos · aarch64228 ms / 1547 ms
BF16mpsAcceleratedPassedFastmacos · aarch64229 ms / 1528 ms
BF16mpsAcceleratedPassedFastmacos · aarch64232 ms / 1602 ms
BF16mpsAcceleratedPassedModeratemacos · aarch64101 ms / 1528 ms
BF16mpsAcceleratedPassedModeratemacos · aarch64101 ms / 1547 ms
BF16mpsAcceleratedPassedModeratemacos · aarch64101 ms / 1602 ms
BF16mpsAcceleratedPassedModeratemacos · aarch64140 ms / 1547 ms
BF16mpsAcceleratedPassedModeratemacos · aarch64142 ms / 1528 ms
BF16mpsAcceleratedPassedModeratemacos · aarch64142 ms / 1547 ms
BF16mpsAcceleratedPassedModeratemacos · aarch64143 ms / 1547 ms
BF16mpsAcceleratedPassedModeratemacos · aarch64144 ms / 1602 ms
BF16mpsAcceleratedPassedModeratemacos · aarch64145 ms / 1528 ms
BF16mpsAcceleratedPassedModeratemacos · aarch64146 ms / 1547 ms
BF16mpsAcceleratedPassedModeratemacos · aarch64147 ms / 1602 ms
BF16mpsAcceleratedPassedModeratemacos · aarch64147 ms / 1528 ms
BF16mpsAcceleratedPassedModeratemacos · aarch64148 ms / 1528 ms
BF16mpsAcceleratedPassedModeratemacos · aarch64165 ms / 1602 ms
BF16mpsAcceleratedPassedModeratemacos · aarch64240 ms / 1602 ms
FP32cpuPassedModeratemacos · aarch64701 ms / 1069 ms
FP32cpuPassedModeratemacos · aarch64738 ms / 1013 ms
FP32cpuPassedModeratemacos · aarch64742 ms / 1008 ms
FP32cpuPassedModeratemacos · aarch64807 ms / 1038 ms
FP32cpuPassedModeratemacos · aarch64279 ms / 1069 ms
FP32cpuPassedModeratemacos · aarch64286 ms / 1013 ms
FP32cpuPassedModeratemacos · aarch64288 ms / 1069 ms
FP32cpuPassedModeratemacos · aarch64290 ms / 1008 ms
FP32cpuPassedModeratemacos · aarch64292 ms / 1013 ms
FP32cpuPassedModeratemacos · aarch64296 ms / 1008 ms
FP32cpuPassedModeratemacos · aarch64301 ms / 1038 ms
FP32cpuPassedModeratemacos · aarch64301 ms / 1038 ms
FP32cpuPassedModeratemacos · aarch64308 ms / 1069 ms
FP32cpuPassedModeratemacos · aarch64311 ms / 1069 ms
FP32cpuPassedModeratemacos · aarch64312 ms / 1008 ms
FP32cpuPassedModeratemacos · aarch64325 ms / 1013 ms
FP32cpuPassedModeratemacos · aarch64325 ms / 1013 ms
FP32cpuPassedModeratemacos · aarch64327 ms / 1008 ms
FP32cpuPassedModeratemacos · aarch64331 ms / 1038 ms
FP32cpuPassedModeratemacos · aarch64331 ms / 1069 ms
FP32cpuPassedModeratemacos · aarch64332 ms / 1013 ms
FP32cpuPassedModeratemacos · aarch64332 ms / 1008 ms
FP32cpuPassedModeratemacos · aarch64340 ms / 1038 ms
FP32cpuPassedModeratemacos · aarch64363 ms / 1038 ms
FP16mpsFailed—macos · aarch64789 ms / 1373 ms
FP16mpsFailed—macos · aarch64599 ms / 1373 ms
FP16mpsFailed—macos · aarch64666 ms / 1373 ms
FP16mpsFailed—macos · aarch64649 ms / 1373 ms
FP16mpsFailed—macos · aarch64653 ms / 1373 ms
FP16mpsFailed—macos · aarch64645 ms / 1373 ms
Test setup & sample details
Quality gate
No explicit quality gate recorded. Review the per-case quality evidence; full certification is pending.
Tested
2026-11-07T12:18:59.941540+00:00
Environment
{ "os": "macos", "arch": "aarch64", "hardware": "Apple M3 Max", "osVersion": "27.2" }
Runtime
transformers 5.22.1
Artifact revision
6a7381ba1f54d684ff508d991aeb7dc580157103
Fixture
{
  "file": "model-samples/desktop/granite4/granite-4.0-1b-native-capital.json",
  "size": 244,
  "sha1": "9013b4f5a71cd1280d78b3dfa751ead1186d071f",
  "sha256": "8ff34789fa2a0f5bbef2080057f4b36376d3c73ea64baccefb6caec96eafbd68",
  "url": "https://assets.localai.tools/samples/v1/files/8ff34789fa2a0f5bbef2080057f4b36376d3c73ea64baccefb6caec96eafbd68/granite-4.0-1b-native-capital.json"
}
Rerun when
Artifact/runtime revision; Input/generation processing; Fixture/quality gates; OS/backend behavior
Per-case quality & failure details
[
  {
    "precision": "fp16",
    "device": "mps",
    "status": "fail",
    "quality": {
      "exactNormalized": false,
      "streamParity": true,
      "withinTokenBudget": false,
      "plainParity": true,
      "passed": false,
      "expected": "Paris"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
      "torchVersion": "2.10.0",
      "loadMs": 1372.7315830037696,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 322
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 800.7019579963526,
      "inferenceMs": 789.4715830043424,
      "tokensPerSecond": 40.533441214215294,
      "inputTokens": 63,
      "outputTokens": 32,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": true,
      "driverAllocatedMemoryBytes": 4348821504,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 1745797120,
      "cpuThreads": 4,
      "totalMs": 2173.433541000122,
      "grade": "fast",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "fail",
    "quality": {
      "exactNormalized": false,
      "streamParity": true,
      "withinTokenBudget": false,
      "passed": false,
      "expected": "56"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
      "torchVersion": "2.10.0",
      "loadMs": 1372.7315830037696,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 322
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 600.4489580009249,
      "inferenceMs": 598.7156659975881,
      "tokensPerSecond": 53.4477412524043,
      "inputTokens": 63,
      "outputTokens": 32,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": true,
      "driverAllocatedMemoryBytes": 4348821504,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 1747779584,
      "cpuThreads": 4,
      "totalMs": 1973.1805410046945,
      "grade": "fast",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "fail",
    "quality": {
      "exactNormalized": false,
      "streamParity": true,
      "withinTokenBudget": false,
      "passed": false,
      "expected": "chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
      "torchVersion": "2.10.0",
      "loadMs": 1372.7315830037696,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 322
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 667.8376249983557,
      "inferenceMs": 666.1485830045422,
      "tokensPerSecond": 48.037331034571615,
      "inputTokens": 68,
      "outputTokens": 32,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": true,
      "driverAllocatedMemoryBytes": 4349476864,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 1758953472,
      "cpuThreads": 4,
      "totalMs": 2040.5692080021254,
      "grade": "fast",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "fail",
    "quality": {
      "exactNormalized": false,
      "streamParity": true,
      "withinTokenBudget": false,
      "passed": false,
      "expected": "hola"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
      "torchVersion": "2.10.0",
      "loadMs": 1372.7315830037696,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 322
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 650.5502920044819,
      "inferenceMs": 648.7943329993868,
      "tokensPerSecond": 49.32225571709832,
      "inputTokens": 69,
      "outputTokens": 32,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": true,
      "driverAllocatedMemoryBytes": 4350148608,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 1769439232,
      "cpuThreads": 4,
      "totalMs": 2023.2818750082515,
      "grade": "fast",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "fail",
    "quality": {
      "exactNormalized": false,
      "streamParity": true,
      "withinTokenBudget": false,
      "passed": false,
      "expected": "2026-11-04"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
      "torchVersion": "2.10.0",
      "loadMs": 1372.7315830037696,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 322
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 654.2875000013737,
      "inferenceMs": 653.0187089956598,
      "tokensPerSecond": 49.00319019835722,
      "inputTokens": 72,
      "outputTokens": 32,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": true,
      "driverAllocatedMemoryBytes": 4350869504,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 1779712000,
      "cpuThreads": 4,
      "totalMs": 2027.0190830051433,
      "grade": "fast",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "fail",
    "quality": {
      "exactNormalized": false,
      "streamParity": true,
      "withinTokenBudget": false,
      "passed": false,
      "expected": "Unknown"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
      "torchVersion": "2.10.0",
      "loadMs": 1372.7315830037696,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 322
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 646.8709999971907,
      "inferenceMs": 645.0455410013092,
      "tokensPerSecond": 49.60890040465384,
      "inputTokens": 74,
      "outputTokens": 32,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": true,
      "driverAllocatedMemoryBytes": 4351639552,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 1790148608,
      "cpuThreads": 4,
      "totalMs": 2019.6025830009603,
      "grade": "fast",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "plainParity": true,
      "passed": true,
      "expected": "Paris"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
      "torchVersion": "2.10.0",
      "loadMs": 1037.506749999011,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 311.1241250007879,
      "inferenceMs": 301.29562500223983,
      "tokensPerSecond": 6.637998809259616,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 6971244544,
      "cpuThreads": 4,
      "totalMs": 1348.6308749997988,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "56"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
      "torchVersion": "2.10.0",
      "loadMs": 1037.506749999011,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 301.17291700298665,
      "inferenceMs": 300.80766700120876,
      "tokensPerSecond": 6.6487667017874355,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 6972325888,
      "cpuThreads": 4,
      "totalMs": 1338.6796670019976,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
      "torchVersion": "2.10.0",
      "loadMs": 1037.506749999011,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 340.6294579981477,
      "inferenceMs": 340.2599159962847,
      "tokensPerSecond": 5.877859559636869,
      "inputTokens": 68,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 6984630272,
      "cpuThreads": 4,
      "totalMs": 1378.1362079971586,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "hola"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
      "torchVersion": "2.10.0",
      "loadMs": 1037.506749999011,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 331.48708299995633,
      "inferenceMs": 331.09712499572197,
      "tokensPerSecond": 6.040523607765672,
      "inputTokens": 69,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 6989004800,
      "cpuThreads": 4,
      "totalMs": 1368.9938329989673,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "2026-11-04"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
      "torchVersion": "2.10.0",
      "loadMs": 1037.506749999011,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 806.9329999998445,
      "inferenceMs": 806.5121250037919,
      "tokensPerSecond": 8.679348745026108,
      "inputTokens": 72,
      "outputTokens": 7,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 6991609856,
      "cpuThreads": 4,
      "totalMs": 1844.4397499988554,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "Unknown"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
      "torchVersion": "2.10.0",
      "loadMs": 1037.506749999011,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 363.110708000022,
      "inferenceMs": 362.6732080010697,
      "tokensPerSecond": 5.514606416678292,
      "inputTokens": 74,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 6994100224,
      "cpuThreads": 4,
      "totalMs": 1400.617457999033,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "Paris"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
      "torchVersion": "2.10.0",
      "loadMs": 1008.2986670022365,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 300.26425000687595,
      "inferenceMs": 290.22729199641617,
      "tokensPerSecond": 6.891150677947603,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 6966853632,
      "cpuThreads": 4,
      "totalMs": 1308.5629170091124,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "56"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
      "torchVersion": "2.10.0",
      "loadMs": 1008.2986670022365,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 296.5291669970611,
      "inferenceMs": 296.16458300006343,
      "tokensPerSecond": 6.753001927983981,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 6969163776,
      "cpuThreads": 4,
      "totalMs": 1304.8278339992976,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
      "torchVersion": "2.10.0",
      "loadMs": 1008.2986670022365,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 312.8949579986511,
      "inferenceMs": 312.4881670009927,
      "tokensPerSecond": 6.400242348996359,
      "inputTokens": 68,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 6973718528,
      "cpuThreads": 4,
      "totalMs": 1321.1936250008876,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "hola"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
      "torchVersion": "2.10.0",
      "loadMs": 1008.2986670022365,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 327.1032920019934,
      "inferenceMs": 326.75804199971026,
      "tokensPerSecond": 6.120736884577652,
      "inputTokens": 69,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 6976356352,
      "cpuThreads": 4,
      "totalMs": 1335.4019590042299,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "2026-11-04"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
      "torchVersion": "2.10.0",
      "loadMs": 1008.2986670022365,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 742.6865419984097,
      "inferenceMs": 742.2947910017683,
      "tokensPerSecond": 9.430215710598087,
      "inputTokens": 72,
      "outputTokens": 7,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 6978781184,
      "cpuThreads": 4,
      "totalMs": 1750.9852090006461,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "Unknown"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
      "torchVersion": "2.10.0",
      "loadMs": 1008.2986670022365,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 332.26579100301024,
      "inferenceMs": 331.8535409998731,
      "tokensPerSecond": 6.0267550377013,
      "inputTokens": 74,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 6987300864,
      "cpuThreads": 4,
      "totalMs": 1340.5644580052467,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "Paris"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
      "torchVersion": "2.10.0",
      "loadMs": 1013.4504999950877,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 301.7093750022468,
      "inferenceMs": 292.00954199768603,
      "tokensPerSecond": 6.84909125338051,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 6964969472,
      "cpuThreads": 4,
      "totalMs": 1315.1598749973346,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "56"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
      "torchVersion": "2.10.0",
      "loadMs": 1013.4504999950877,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 286.80916700250236,
      "inferenceMs": 286.4351660027751,
      "tokensPerSecond": 6.982382882347006,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 6969245696,
      "cpuThreads": 4,
      "totalMs": 1300.25966699759,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
      "torchVersion": "2.10.0",
      "loadMs": 1013.4504999950877,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 325.0256250030361,
      "inferenceMs": 324.6042920000036,
      "tokensPerSecond": 6.161347983655058,
      "inputTokens": 68,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 6971654144,
      "cpuThreads": 4,
      "totalMs": 1338.4761249981239,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "hola"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
      "torchVersion": "2.10.0",
      "loadMs": 1013.4504999950877,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 332.0834170008311,
      "inferenceMs": 331.6714999964461,
      "tokensPerSecond": 6.030062878545278,
      "inputTokens": 69,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 6975389696,
      "cpuThreads": 4,
      "totalMs": 1345.5339169959188,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "2026-11-04"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
      "torchVersion": "2.10.0",
      "loadMs": 1013.4504999950877,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 737.9900000014459,
      "inferenceMs": 737.5955410025199,
      "tokensPerSecond": 9.490295983196685,
      "inputTokens": 72,
      "outputTokens": 7,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 6984810496,
      "cpuThreads": 4,
      "totalMs": 1751.4404999965336,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "Unknown"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
      "torchVersion": "2.10.0",
      "loadMs": 1013.4504999950877,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 325.5090000020573,
      "inferenceMs": 325.0664169972879,
      "tokensPerSecond": 6.152588810848112,
      "inputTokens": 74,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 6993477632,
      "cpuThreads": 4,
      "totalMs": 1338.959499997145,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "plainParity": true,
      "passed": true,
      "expected": "Paris"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
      "torchVersion": "2.10.0",
      "loadMs": 1068.5277080046944,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 297.60641699976986,
      "inferenceMs": 288.0704999988666,
      "tokensPerSecond": 6.942744918371957,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 6971277312,
      "cpuThreads": 1,
      "totalMs": 1366.1341250044643,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "56"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
      "torchVersion": "2.10.0",
      "loadMs": 1068.5277080046944,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 279.09091600304237,
      "inferenceMs": 278.7209580055787,
      "tokensPerSecond": 7.175635496918639,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 6971277312,
      "cpuThreads": 1,
      "totalMs": 1347.6186240077368,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
      "torchVersion": "2.10.0",
      "loadMs": 1068.5277080046944,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 307.975791998615,
      "inferenceMs": 307.6092920018709,
      "tokensPerSecond": 6.501754179739914,
      "inputTokens": 68,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 6973521920,
      "cpuThreads": 1,
      "totalMs": 1376.5035000033095,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "hola"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
      "torchVersion": "2.10.0",
      "loadMs": 1068.5277080046944,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 311.829124999349,
      "inferenceMs": 311.4636669997708,
      "tokensPerSecond": 6.421294718788088,
      "inputTokens": 69,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 6976798720,
      "cpuThreads": 1,
      "totalMs": 1380.3568330040434,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "2026-11-04"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
      "torchVersion": "2.10.0",
      "loadMs": 1068.5277080046944,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 701.3185000032536,
      "inferenceMs": 700.8254159954959,
      "tokensPerSecond": 9.988222230862968,
      "inputTokens": 72,
      "outputTokens": 7,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 6982631424,
      "cpuThreads": 1,
      "totalMs": 1769.846208007948,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "Unknown"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
      "torchVersion": "2.10.0",
      "loadMs": 1068.5277080046944,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 322
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 331.74608400440775,
      "inferenceMs": 331.3417079989449,
      "tokensPerSecond": 6.036064738358772,
      "inputTokens": 74,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 6987595776,
      "cpuThreads": 1,
      "totalMs": 1400.2737920091022,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "plainParity": true,
      "passed": true,
      "expected": "Paris"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
      "torchVersion": "2.10.0",
      "loadMs": 1601.937542000087,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 322
      },
      "parameterDtype": "torch.bfloat16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 250.708334002411,
      "inferenceMs": 239.65508399851387,
      "tokensPerSecond": 8.34532681982412,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 4348821504,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 510246912,
      "cpuThreads": 4,
      "totalMs": 1852.645876002498,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "56"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
      "torchVersion": "2.10.0",
      "loadMs": 1601.937542000087,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 322
      },
      "parameterDtype": "torch.bfloat16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 102.42675000336021,
      "inferenceMs": 101.24408300180221,
      "tokensPerSecond": 19.754240847481416,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 4348821504,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 510623744,
      "cpuThreads": 4,
      "totalMs": 1704.3642920034472,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
      "torchVersion": "2.10.0",
      "loadMs": 1601.937542000087,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 322
      },
      "parameterDtype": "torch.bfloat16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 166.7393749958137,
      "inferenceMs": 164.99674999795388,
      "tokensPerSecond": 12.121450877213048,
      "inputTokens": 68,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 4349476864,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 520667136,
      "cpuThreads": 4,
      "totalMs": 1768.6769169959007,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "hola"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
      "torchVersion": "2.10.0",
      "loadMs": 1601.937542000087,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 322
      },
      "parameterDtype": "torch.bfloat16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 148.7805419965298,
      "inferenceMs": 147.075249995396,
      "tokensPerSecond": 13.598481050092435,
      "inputTokens": 69,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 4350148608,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 530038784,
      "cpuThreads": 4,
      "totalMs": 1750.7180839966168,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "2026-11-04"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
      "torchVersion": "2.10.0",
      "loadMs": 1601.937542000087,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 322
      },
      "parameterDtype": "torch.bfloat16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 233.8637499997276,
      "inferenceMs": 232.1823750025942,
      "tokensPerSecond": 30.148713914748214,
      "inputTokens": 72,
      "outputTokens": 7,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 4350869504,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 539426816,
      "cpuThreads": 4,
      "totalMs": 1835.8012919998146,
      "grade": "fast",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "Unknown"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
      "torchVersion": "2.10.0",
      "loadMs": 1601.937542000087,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 322
      },
      "parameterDtype": "torch.bfloat16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 146.2107500046841,
      "inferenceMs": 144.44766600354342,
      "tokensPerSecond": 13.845845040867177,
      "inputTokens": 74,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 4351639552,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 548716544,
      "cpuThreads": 4,
      "totalMs": 1748.148292004771,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "Paris"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
      "torchVersion": "2.10.0",
      "loadMs": 1546.7671250007697,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 322
      },
      "parameterDtype": "torch.bfloat16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 156.74033400136977,
      "inferenceMs": 145.57950000016717,
      "tokensPerSecond": 13.738198029239717,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 4348837888,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 505151488,
      "cpuThreads": 4,
      "totalMs": 1703.5074590021395,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "56"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
      "torchVersion": "2.10.0",
      "loadMs": 1546.7671250007697,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 322
      },
      "parameterDtype": "torch.bfloat16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 102.72541700396687,
      "inferenceMs": 101.03929099568632,
      "tokensPerSecond": 19.79427983204461,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 4348837888,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 505610240,
      "cpuThreads": 4,
      "totalMs": 1649.4925420047366,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
      "torchVersion": "2.10.0",
      "loadMs": 1546.7671250007697,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 322
      },
      "parameterDtype": "torch.bfloat16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 144.80779199948302,
      "inferenceMs": 143.03274999838322,
      "tokensPerSecond": 13.982811628963347,
      "inputTokens": 68,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 4349476864,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 515817472,
      "cpuThreads": 4,
      "totalMs": 1691.5749170002528,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "hola"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
      "torchVersion": "2.10.0",
      "loadMs": 1546.7671250007697,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 322
      },
      "parameterDtype": "torch.bfloat16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 141.60166699730325,
      "inferenceMs": 140.4084590030834,
      "tokensPerSecond": 14.244156044445154,
      "inputTokens": 69,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 4350148608,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 525156352,
      "cpuThreads": 4,
      "totalMs": 1688.368791998073,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "2026-11-04"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
      "torchVersion": "2.10.0",
      "loadMs": 1546.7671250007697,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 322
      },
      "parameterDtype": "torch.bfloat16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 229.66729199833935,
      "inferenceMs": 227.94625000096858,
      "tokensPerSecond": 30.708993896456978,
      "inputTokens": 72,
      "outputTokens": 7,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 4350869504,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 534528000,
      "cpuThreads": 4,
      "totalMs": 1776.434416999109,
      "grade": "fast",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "Unknown"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
      "torchVersion": "2.10.0",
      "loadMs": 1546.7671250007697,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 322
      },
      "parameterDtype": "torch.bfloat16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 144.36045799811836,
      "inferenceMs": 142.40566599619342,
      "tokensPerSecond": 14.044385003988964,
      "inputTokens": 74,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 4351639552,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 543686656,
      "cpuThreads": 4,
      "totalMs": 1691.127582998888,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "Paris"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
      "torchVersion": "2.10.0",
      "loadMs": 1527.7355420039385,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 322
      },
      "parameterDtype": "torch.bfloat16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 159.09820899832994,
      "inferenceMs": 147.9200000030687,
      "tokensPerSecond": 13.520822065701113,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 4348837888,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 509001728,
      "cpuThreads": 4,
      "totalMs": 1686.8337510022684,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "56"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
      "torchVersion": "2.10.0",
      "loadMs": 1527.7355420039385,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 322
      },
      "parameterDtype": "torch.bfloat16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 101.8580829986604,
      "inferenceMs": 100.7070410050801,
      "tokensPerSecond": 19.859584593485486,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 4348837888,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 509509632,
      "cpuThreads": 4,
      "totalMs": 1629.5936250025989,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
      "torchVersion": "2.10.0",
      "loadMs": 1527.7355420039385,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 322
      },
      "parameterDtype": "torch.bfloat16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 146.13899999676505,
      "inferenceMs": 144.56670799700078,
      "tokensPerSecond": 13.83444381981426,
      "inputTokens": 68,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 4349476864,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 519356416,
      "cpuThreads": 4,
      "totalMs": 1673.8745420007035,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "hola"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
      "torchVersion": "2.10.0",
      "loadMs": 1527.7355420039385,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 322
      },
      "parameterDtype": "torch.bfloat16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 144.12462500331458,
      "inferenceMs": 142.28900000307476,
      "tokensPerSecond": 14.055900315251224,
      "inputTokens": 69,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 4350148608,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 528515072,
      "cpuThreads": 4,
      "totalMs": 1671.860167007253,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "2026-11-04"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
      "torchVersion": "2.10.0",
      "loadMs": 1527.7355420039385,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 322
      },
      "parameterDtype": "torch.bfloat16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 230.75229200185277,
      "inferenceMs": 228.98787499434547,
      "tokensPerSecond": 30.5693041615101,
      "inputTokens": 72,
      "outputTokens": 7,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 4350869504,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 537837568,
      "cpuThreads": 4,
      "totalMs": 1758.4878340057912,
      "grade": "fast",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "Unknown"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "6a7381ba1f54d684ff508d991aeb7dc580157103",
      "torchVersion": "2.10.0",
      "loadMs": 1527.7355420039385,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 322
      },
      "parameterDtype": "torch.bfloat16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 151.71829099563183,
      "inferenceMs": 147.4896660001832,
      "tokensPerSecond": 13.560272080333517,
      "inputTokens": 74,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 4351639552,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 547045376,
      "cpuThreads": 4,
      "totalMs": 1679.4538329995703,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  }
]

Granite 4.0 Dense 350M

28 configurations tested
Passed28
Failed0
TestedNov 7, 2026
Runtimetransformers 5.22.1

Quality criteria

No explicit quality gate recorded. Review per-case evidence; full certification is pending.

PrecisionDeviceResultPlatformInference / load
FP16mpsAcceleratedPassedVery fastmacos · aarch6448 ms / 1027 ms
FP16mpsAcceleratedPassedVery fastmacos · aarch6452 ms / 1154 ms
FP16mpsAcceleratedPassedVery fastmacos · aarch6452 ms / 1107 ms
FP16mpsAcceleratedPassedVery fastmacos · aarch6479 ms / 1027 ms
FP16mpsAcceleratedPassedVery fastmacos · aarch6479 ms / 1027 ms
FP16mpsAcceleratedPassedVery fastmacos · aarch6482 ms / 1107 ms
FP16mpsAcceleratedPassedVery fastmacos · aarch6485 ms / 1107 ms
FP16mpsAcceleratedPassedVery fastmacos · aarch6492 ms / 1154 ms
FP16mpsAcceleratedPassedVery fastmacos · aarch6493 ms / 1154 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64104 ms / 1027 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64119 ms / 1107 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64120 ms / 1154 ms
FP32cpuPassedVery fastmacos · aarch6482 ms / 790 ms
FP32cpuPassedVery fastmacos · aarch6484 ms / 790 ms
FP32cpuPassedVery fastmacos · aarch6489 ms / 809 ms
FP32cpuPassedVery fastmacos · aarch6490 ms / 790 ms
FP32cpuPassedVery fastmacos · aarch6491 ms / 855 ms
FP32cpuPassedVery fastmacos · aarch6491 ms / 855 ms
FP32cpuPassedVery fastmacos · aarch6495 ms / 809 ms
FP32cpuPassedVery fastmacos · aarch6498 ms / 855 ms
FP32cpuPassedVery fastmacos · aarch6499 ms / 790 ms
FP32cpuPassedVery fastmacos · aarch64100 ms / 855 ms
FP32cpuPassedModeratemacos · aarch64100 ms / 819 ms
FP32cpuPassedModeratemacos · aarch64104 ms / 819 ms
FP32cpuPassedModeratemacos · aarch64104 ms / 809 ms
FP32cpuPassedModeratemacos · aarch64109 ms / 809 ms
FP32cpuPassedModeratemacos · aarch64114 ms / 819 ms
FP32cpuPassedModeratemacos · aarch64117 ms / 819 ms
Test setup & sample details
Quality gate
No explicit quality gate recorded. Review the per-case quality evidence; full certification is pending.
Tested
2026-11-07T12:16:29.639275+00:00
Environment
{ "os": "macos", "arch": "aarch64", "hardware": "Apple M3 Max", "osVersion": "27.2" }
Runtime
transformers 5.22.1
Artifact revision
bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2
Fixture
{
  "file": "model-samples/desktop/granite4/granite-4.0-350m-native-capital.json",
  "size": 244,
  "sha1": "9013b4f5a71cd1280d78b3dfa751ead1186d071f",
  "sha256": "8ff34789fa2a0f5bbef2080057f4b36376d3c73ea64baccefb6caec96eafbd68",
  "url": "https://assets.localai.tools/samples/v1/files/8ff34789fa2a0f5bbef2080057f4b36376d3c73ea64baccefb6caec96eafbd68/granite-4.0-350m-native-capital.json"
}
Rerun when
Artifact/runtime revision; Input/generation processing; Fixture/quality gates; OS/backend behavior
Per-case quality & failure details
[
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "plainParity": true,
      "passed": true,
      "expected": "Paris"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
      "torchVersion": "2.10.0",
      "loadMs": 1153.7135419930564,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 226
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 131.51083400589414,
      "inferenceMs": 119.536458994844,
      "tokensPerSecond": 16.731297018646558,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 1136001024,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 1037860864,
      "cpuThreads": 4,
      "totalMs": 1285.2243759989506,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "56"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
      "torchVersion": "2.10.0",
      "loadMs": 1153.7135419930564,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 226
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 53.727541999251116,
      "inferenceMs": 51.9242089940235,
      "tokensPerSecond": 38.51767872342939,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 1136001024,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 1038106624,
      "cpuThreads": 4,
      "totalMs": 1207.4410839923075,
      "grade": "fast",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "summer"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
      "torchVersion": "2.10.0",
      "loadMs": 1153.7135419930564,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 226
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 93.75641599763185,
      "inferenceMs": 91.96091699413955,
      "tokensPerSecond": 21.748369474474195,
      "inputTokens": 67,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 1136640000,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 1047920640,
      "cpuThreads": 4,
      "totalMs": 1247.4699579906883,
      "grade": "fast",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "7"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
      "torchVersion": "2.10.0",
      "loadMs": 1153.7135419930564,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 226
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 94.54145900235744,
      "inferenceMs": 92.6774170002318,
      "tokensPerSecond": 21.580230273303773,
      "inputTokens": 75,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 1137426432,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 1057226752,
      "cpuThreads": 4,
      "totalMs": 1248.2550009954139,
      "grade": "fast",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "Paris"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
      "torchVersion": "2.10.0",
      "loadMs": 1106.7504169986933,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 226
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 130.84404199616984,
      "inferenceMs": 118.98054199991748,
      "tokensPerSecond": 16.809471249520676,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 1136001024,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 1201274880,
      "cpuThreads": 4,
      "totalMs": 1237.594458994863,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "56"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
      "torchVersion": "2.10.0",
      "loadMs": 1106.7504169986933,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 226
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 53.620083002897445,
      "inferenceMs": 52.36283299745992,
      "tokensPerSecond": 38.19503043498465,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 1136001024,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 1201700864,
      "cpuThreads": 4,
      "totalMs": 1160.3705000015907,
      "grade": "fast",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "summer"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
      "torchVersion": "2.10.0",
      "loadMs": 1106.7504169986933,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 226
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 86.38733300176682,
      "inferenceMs": 84.71229099814082,
      "tokensPerSecond": 23.60932488585268,
      "inputTokens": 67,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 1136640000,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 1211514880,
      "cpuThreads": 4,
      "totalMs": 1193.13775000046,
      "grade": "fast",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "7"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
      "torchVersion": "2.10.0",
      "loadMs": 1106.7504169986933,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 226
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 83.82970799721079,
      "inferenceMs": 82.03133400093066,
      "tokensPerSecond": 24.380927414606102,
      "inputTokens": 75,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 1137426432,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 1220919296,
      "cpuThreads": 4,
      "totalMs": 1190.580124995904,
      "grade": "fast",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "Paris"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
      "torchVersion": "2.10.0",
      "loadMs": 1026.9828749951557,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 226
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 115.35945899959188,
      "inferenceMs": 103.9500830011093,
      "tokensPerSecond": 19.24000387742506,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 1136001024,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 1044250624,
      "cpuThreads": 4,
      "totalMs": 1142.3423339947476,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "56"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
      "torchVersion": "2.10.0",
      "loadMs": 1026.9828749951557,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 226
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 49.815040998510085,
      "inferenceMs": 48.15483299898915,
      "tokensPerSecond": 41.532695171884065,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 1136001024,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 1044676608,
      "cpuThreads": 4,
      "totalMs": 1076.7979159936658,
      "grade": "fast",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "summer"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
      "torchVersion": "2.10.0",
      "loadMs": 1026.9828749951557,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 226
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 80.7666250038892,
      "inferenceMs": 78.96066600369522,
      "tokensPerSecond": 25.329067005417652,
      "inputTokens": 67,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 1136640000,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 1054539776,
      "cpuThreads": 4,
      "totalMs": 1107.749499999045,
      "grade": "fast",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "7"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
      "torchVersion": "2.10.0",
      "loadMs": 1026.9828749951557,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 226
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 81.00491600634996,
      "inferenceMs": 79.22775000042748,
      "tokensPerSecond": 25.243680402248064,
      "inputTokens": 75,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 1137426432,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 1063911424,
      "cpuThreads": 4,
      "totalMs": 1107.9877910015057,
      "grade": "fast",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "plainParity": true,
      "passed": true,
      "expected": "Paris"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
      "torchVersion": "2.10.0",
      "loadMs": 818.9316249990952,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 226
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 123.5065000000759,
      "inferenceMs": 113.90570900402963,
      "tokensPerSecond": 17.55838243304597,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 1834057728,
      "cpuThreads": 4,
      "totalMs": 942.4381249991711,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "56"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
      "torchVersion": "2.10.0",
      "loadMs": 818.9316249990952,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 226
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 104.3223750020843,
      "inferenceMs": 103.98008400079561,
      "tokensPerSecond": 19.234452628300403,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 1836793856,
      "cpuThreads": 4,
      "totalMs": 923.2540000011795,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "summer"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
      "torchVersion": "2.10.0",
      "loadMs": 818.9316249990952,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 226
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 117.16191600135062,
      "inferenceMs": 116.7806669982383,
      "tokensPerSecond": 17.126122425984867,
      "inputTokens": 67,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 1838317568,
      "cpuThreads": 4,
      "totalMs": 936.0935410004458,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "7"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
      "torchVersion": "2.10.0",
      "loadMs": 818.9316249990952,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 226
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 100.62195800128393,
      "inferenceMs": 100.16212499613175,
      "tokensPerSecond": 19.967627484712807,
      "inputTokens": 75,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 1839104000,
      "cpuThreads": 4,
      "totalMs": 919.5535830003791,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "Paris"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
      "torchVersion": "2.10.0",
      "loadMs": 855.4552500063437,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 226
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 102.5817499976256,
      "inferenceMs": 91.46650000184309,
      "tokensPerSecond": 21.86592905555257,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 1833025536,
      "cpuThreads": 4,
      "totalMs": 958.0370000039693,
      "grade": "fast",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "56"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
      "torchVersion": "2.10.0",
      "loadMs": 855.4552500063437,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 226
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 91.43366699572653,
      "inferenceMs": 91.08829199976753,
      "tokensPerSecond": 21.956718652767186,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 1834205184,
      "cpuThreads": 4,
      "totalMs": 946.8889170020702,
      "grade": "fast",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "summer"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
      "torchVersion": "2.10.0",
      "loadMs": 855.4552500063437,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 226
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 98.27200000290759,
      "inferenceMs": 97.87729099480202,
      "tokensPerSecond": 20.433749030775832,
      "inputTokens": 67,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 1836040192,
      "cpuThreads": 4,
      "totalMs": 953.7272500092513,
      "grade": "fast",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "7"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
      "torchVersion": "2.10.0",
      "loadMs": 855.4552500063437,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 226
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 100.29762499470962,
      "inferenceMs": 99.89679099817295,
      "tokensPerSecond": 20.020663126572092,
      "inputTokens": 75,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 1838088192,
      "cpuThreads": 4,
      "totalMs": 955.7528750010533,
      "grade": "fast",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "Paris"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
      "torchVersion": "2.10.0",
      "loadMs": 809.3695409988868,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 226
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 113.83824999938952,
      "inferenceMs": 104.44516599818598,
      "tokensPerSecond": 19.148803880830027,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 1836826624,
      "cpuThreads": 4,
      "totalMs": 923.2077909982763,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "56"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
      "torchVersion": "2.10.0",
      "loadMs": 809.3695409988868,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 226
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 95.27779200288933,
      "inferenceMs": 94.86583399848314,
      "tokensPerSecond": 21.082405706062513,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 1840807936,
      "cpuThreads": 4,
      "totalMs": 904.6473330017761,
      "grade": "fast",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "summer"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
      "torchVersion": "2.10.0",
      "loadMs": 809.3695409988868,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 226
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 109.13429200445535,
      "inferenceMs": 108.78875000344124,
      "tokensPerSecond": 18.384253885964636,
      "inputTokens": 67,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 1842069504,
      "cpuThreads": 4,
      "totalMs": 918.5038330033422,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "7"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
      "torchVersion": "2.10.0",
      "loadMs": 809.3695409988868,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 226
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 89.68825000192737,
      "inferenceMs": 89.30945800238987,
      "tokensPerSecond": 22.394044760035168,
      "inputTokens": 75,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 1844150272,
      "cpuThreads": 4,
      "totalMs": 899.0577910008142,
      "grade": "fast",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "plainParity": true,
      "passed": true,
      "expected": "Paris"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
      "torchVersion": "2.10.0",
      "loadMs": 790.2471249981318,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 226
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 91.50879200024065,
      "inferenceMs": 81.92720799706876,
      "tokensPerSecond": 24.411914538471237,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 1827979264,
      "cpuThreads": 1,
      "totalMs": 881.7559169983724,
      "grade": "fast",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "56"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
      "torchVersion": "2.10.0",
      "loadMs": 790.2471249981318,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 226
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 83.9100840021274,
      "inferenceMs": 83.55279199895449,
      "tokensPerSecond": 23.936961915348398,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 1828405248,
      "cpuThreads": 1,
      "totalMs": 874.1572090002592,
      "grade": "fast",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "summer"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
      "torchVersion": "2.10.0",
      "loadMs": 790.2471249981318,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 226
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 90.5117500005872,
      "inferenceMs": 90.14249999745516,
      "tokensPerSecond": 22.187092659472086,
      "inputTokens": 67,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 1832271872,
      "cpuThreads": 1,
      "totalMs": 880.758874998719,
      "grade": "fast",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "7"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "bd8a1497065c0d6ba1ef19af6b0d2b14bacf71c2",
      "torchVersion": "2.10.0",
      "loadMs": 790.2471249981318,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 226
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 99.82495800068136,
      "inferenceMs": 99.42108300310792,
      "tokensPerSecond": 20.11645759217368,
      "inputTokens": 75,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 1836056576,
      "cpuThreads": 1,
      "totalMs": 890.0720829988131,
      "grade": "fast",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  }
]

Granite 4.0 Hybrid 1B

30 configurations tested
Passed18
Failed12
TestedNov 7, 2026
Runtimetransformers 5.22.1

Quality criteria

No explicit quality gate recorded. Review per-case evidence; full certification is pending.

PrecisionDeviceResultPlatformInference / load
FP16mpsAcceleratedPassedModeratemacos · aarch64373 ms / 1419 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64384 ms / 1616 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64384 ms / 1446 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64175 ms / 1446 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64263 ms / 1446 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64265 ms / 1616 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64268 ms / 1419 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64180 ms / 1419 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64180 ms / 1616 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64233 ms / 1446 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64234 ms / 1446 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64235 ms / 1419 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64241 ms / 1419 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64241 ms / 1446 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64242 ms / 1419 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64243 ms / 1616 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64274 ms / 1616 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64361 ms / 1616 ms
FP32cpuFailed—macos · aarch645659 ms / 4055 ms
FP32cpuFailed—macos · aarch645433 ms / 4055 ms
FP32cpuFailed—macos · aarch645294 ms / 4055 ms
FP32cpuFailed—macos · aarch647330 ms / 4055 ms
FP32cpuFailed—macos · aarch6414982 ms / 4055 ms
FP32cpuFailed—macos · aarch644431 ms / 4055 ms
FP32cpuFailed—macos · aarch641606 ms / 3678 ms
FP32cpuFailed—macos · aarch641606 ms / 3678 ms
FP32cpuFailed—macos · aarch641643 ms / 3678 ms
FP32cpuFailed—macos · aarch641999 ms / 3678 ms
FP32cpuFailed—macos · aarch643420 ms / 3678 ms
FP32cpuFailed—macos · aarch641651 ms / 3678 ms
Test setup & sample details
Quality gate
No explicit quality gate recorded. Review the per-case quality evidence; full certification is pending.
Tested
2026-11-07T12:20:59.105634+00:00
Environment
{ "os": "macos", "arch": "aarch64", "hardware": "Apple M3 Max", "osVersion": "27.2" }
Runtime
transformers 5.22.1
Artifact revision
d18cca4c121edb87d022116d281ce212c9136f57
Fixture
{
  "file": "model-samples/desktop/granite4/granite-4.0-h-1b-native-capital.json",
  "size": 244,
  "sha1": "9013b4f5a71cd1280d78b3dfa751ead1186d071f",
  "sha256": "8ff34789fa2a0f5bbef2080057f4b36376d3c73ea64baccefb6caec96eafbd68",
  "url": "https://assets.localai.tools/samples/v1/files/8ff34789fa2a0f5bbef2080057f4b36376d3c73ea64baccefb6caec96eafbd68/granite-4.0-h-1b-native-capital.json"
}
Rerun when
Artifact/runtime revision; Input/generation processing; Fixture/quality gates; OS/backend behavior
Per-case quality & failure details
[
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "plainParity": true,
      "passed": true,
      "expected": "Paris"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
      "torchVersion": "2.10.0",
      "loadMs": 1615.5484580012853,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 466
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 372.524541002349,
      "inferenceMs": 361.1726669987547,
      "tokensPerSecond": 5.537517599599794,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 3258384384,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 1599078400,
      "cpuThreads": 4,
      "totalMs": 1988.0729990036343,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "56"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
      "torchVersion": "2.10.0",
      "loadMs": 1615.5484580012853,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 466
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 181.17112499749055,
      "inferenceMs": 179.51862500194693,
      "tokensPerSecond": 11.140905295917399,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 3258384384,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 1600225280,
      "cpuThreads": 4,
      "totalMs": 1796.7195829987759,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
      "torchVersion": "2.10.0",
      "loadMs": 1615.5484580012853,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 466
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 275.88537499832455,
      "inferenceMs": 274.11974999995437,
      "tokensPerSecond": 7.296081365900607,
      "inputTokens": 68,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 3267346432,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 1615806464,
      "cpuThreads": 4,
      "totalMs": 1891.4338329996099,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "hola"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
      "torchVersion": "2.10.0",
      "loadMs": 1615.5484580012853,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 466
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 266.7036250059027,
      "inferenceMs": 264.7699160006596,
      "tokensPerSecond": 11.330592407607693,
      "inputTokens": 69,
      "outputTokens": 3,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 3267936256,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 1630928896,
      "cpuThreads": 4,
      "totalMs": 1882.252083007188,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "2026-11-04"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
      "torchVersion": "2.10.0",
      "loadMs": 1615.5484580012853,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 466
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 385.33874999848194,
      "inferenceMs": 383.50454199826345,
      "tokensPerSecond": 18.252717330350933,
      "inputTokens": 72,
      "outputTokens": 7,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 3268558848,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 1647083520,
      "cpuThreads": 4,
      "totalMs": 2000.8872079997673,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "Unknown"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
      "torchVersion": "2.10.0",
      "loadMs": 1615.5484580012853,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 466
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 244.55133300216403,
      "inferenceMs": 242.83783300052164,
      "tokensPerSecond": 8.235948967621136,
      "inputTokens": 74,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 3277602816,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 1661960192,
      "cpuThreads": 4,
      "totalMs": 1860.0997910034494,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "Paris"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
      "torchVersion": "2.10.0",
      "loadMs": 1446.4713749985094,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 466
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 252.28354099817807,
      "inferenceMs": 241.0489169997163,
      "tokensPerSecond": 8.297071087867007,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 3258400768,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 1592852480,
      "cpuThreads": 4,
      "totalMs": 1698.7549159966875,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "56"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
      "torchVersion": "2.10.0",
      "loadMs": 1446.4713749985094,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 466
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 176.63241700211074,
      "inferenceMs": 174.99975000100676,
      "tokensPerSecond": 11.428587755059617,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 3258400768,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 1594228736,
      "cpuThreads": 4,
      "totalMs": 1623.1037920006202,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
      "torchVersion": "2.10.0",
      "loadMs": 1446.4713749985094,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 466
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 235.52391699922737,
      "inferenceMs": 233.8634589978028,
      "tokensPerSecond": 8.551998711430974,
      "inputTokens": 68,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 3267346432,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 1609695232,
      "cpuThreads": 4,
      "totalMs": 1681.9952919977368,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "hola"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
      "torchVersion": "2.10.0",
      "loadMs": 1446.4713749985094,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 466
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 265.2217080030823,
      "inferenceMs": 263.47412500035716,
      "tokensPerSecond": 11.386317347086486,
      "inputTokens": 69,
      "outputTokens": 3,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 3267936256,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 1626259456,
      "cpuThreads": 4,
      "totalMs": 1711.6930830015917,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "2026-11-04"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
      "torchVersion": "2.10.0",
      "loadMs": 1446.4713749985094,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 466
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 385.5130830052076,
      "inferenceMs": 383.80429199605715,
      "tokensPerSecond": 18.238462013009254,
      "inputTokens": 72,
      "outputTokens": 7,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 3268558848,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 1642430464,
      "cpuThreads": 4,
      "totalMs": 1831.984458003717,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "Unknown"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
      "torchVersion": "2.10.0",
      "loadMs": 1446.4713749985094,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 466
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 234.76033299812116,
      "inferenceMs": 232.5196249948931,
      "tokensPerSecond": 8.601424503604488,
      "inputTokens": 74,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 3277602816,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 1657372672,
      "cpuThreads": 4,
      "totalMs": 1681.2317079966306,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "Paris"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
      "torchVersion": "2.10.0",
      "loadMs": 1418.5662919990136,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 466
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 245.96404100157088,
      "inferenceMs": 235.08370800118428,
      "tokensPerSecond": 8.5076078517101,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 3258400768,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 1596850176,
      "cpuThreads": 4,
      "totalMs": 1664.5303330005845,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "56"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
      "torchVersion": "2.10.0",
      "loadMs": 1418.5662919990136,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 466
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 181.15433299681172,
      "inferenceMs": 179.51779199938755,
      "tokensPerSecond": 11.140956992200657,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 3258400768,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 1598242816,
      "cpuThreads": 4,
      "totalMs": 1599.7206249958253,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
      "torchVersion": "2.10.0",
      "loadMs": 1418.5662919990136,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 466
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 242.39862499962328,
      "inferenceMs": 240.71654200088233,
      "tokensPerSecond": 8.308527462947142,
      "inputTokens": 68,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 3267346432,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 1614168064,
      "cpuThreads": 4,
      "totalMs": 1660.964916998637,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "hola"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
      "torchVersion": "2.10.0",
      "loadMs": 1418.5662919990136,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 466
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 269.2904169962276,
      "inferenceMs": 267.83616700413404,
      "tokensPerSecond": 11.200877139022435,
      "inputTokens": 69,
      "outputTokens": 3,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 3267936256,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 1630846976,
      "cpuThreads": 4,
      "totalMs": 1687.8567089952412,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "2026-11-04"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
      "torchVersion": "2.10.0",
      "loadMs": 1418.5662919990136,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 466
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 374.7637910055346,
      "inferenceMs": 373.08862499776296,
      "tokensPerSecond": 18.762298100195288,
      "inputTokens": 72,
      "outputTokens": 7,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 3268558848,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 1647329280,
      "cpuThreads": 4,
      "totalMs": 1793.3300830045482,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "Unknown"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
      "torchVersion": "2.10.0",
      "loadMs": 1418.5662919990136,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 466
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 243.1170830022893,
      "inferenceMs": 241.72537500271574,
      "tokensPerSecond": 8.273852093424326,
      "inputTokens": 74,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 3277602816,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 1662746624,
      "cpuThreads": 4,
      "totalMs": 1661.683375001303,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "plainParity": true,
      "passed": false,
      "expected": "Paris"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
      "torchVersion": "2.10.0",
      "loadMs": 4054.5797909944667,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 466
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 5669.2278330010595,
      "inferenceMs": 5659.269166993909,
      "tokensPerSecond": 0.35340252265512234,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 6458064896,
      "cpuThreads": 4,
      "totalMs": 9723.807623995526,
      "grade": "slow",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": false,
      "expected": "56"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
      "torchVersion": "2.10.0",
      "loadMs": 4054.5797909944667,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 466
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 5433.794791999389,
      "inferenceMs": 5433.386249998875,
      "tokensPerSecond": 0.3680945745391125,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 6462586880,
      "cpuThreads": 4,
      "totalMs": 9488.374582993856,
      "grade": "slow",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": false,
      "expected": "chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
      "torchVersion": "2.10.0",
      "loadMs": 4054.5797909944667,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 466
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 5294.674124997982,
      "inferenceMs": 5294.235000001208,
      "tokensPerSecond": 0.3777694038892387,
      "inputTokens": 68,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 6462980096,
      "cpuThreads": 4,
      "totalMs": 9349.253915992449,
      "grade": "slow",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": false,
      "expected": "hola"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
      "torchVersion": "2.10.0",
      "loadMs": 4054.5797909944667,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 466
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 7330.640082996979,
      "inferenceMs": 7330.113250005525,
      "tokensPerSecond": 0.4092706207503327,
      "inputTokens": 69,
      "outputTokens": 3,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 6468009984,
      "cpuThreads": 4,
      "totalMs": 11385.219873991446,
      "grade": "slow",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": false,
      "expected": "2026-11-04"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
      "torchVersion": "2.10.0",
      "loadMs": 4054.5797909944667,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 466
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 14983.009625000705,
      "inferenceMs": 14982.482708001044,
      "tokensPerSecond": 0.46721228626960565,
      "inputTokens": 72,
      "outputTokens": 7,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 6468059136,
      "cpuThreads": 4,
      "totalMs": 19037.58941599517,
      "grade": "slow",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": false,
      "expected": "Unknown"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
      "torchVersion": "2.10.0",
      "loadMs": 4054.5797909944667,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 466
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 4431.094125000527,
      "inferenceMs": 4430.652041999565,
      "tokensPerSecond": 0.4514008279236017,
      "inputTokens": 74,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 6468108288,
      "cpuThreads": 4,
      "totalMs": 8485.673915994994,
      "grade": "slow",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "plainParity": true,
      "passed": false,
      "expected": "Paris"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
      "torchVersion": "2.10.0",
      "loadMs": 3678.4359160010354,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 466
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 1616.2001249977038,
      "inferenceMs": 1606.1470419954276,
      "tokensPerSecond": 1.2452160030847872,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 6462767104,
      "cpuThreads": 1,
      "totalMs": 5294.636040998739,
      "grade": "slow",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": false,
      "expected": "56"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
      "torchVersion": "2.10.0",
      "loadMs": 3678.4359160010354,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 466
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 1606.2157499982277,
      "inferenceMs": 1605.7623329979833,
      "tokensPerSecond": 1.245514332289741,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 6462783488,
      "cpuThreads": 1,
      "totalMs": 5284.651665999263,
      "grade": "slow",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": false,
      "expected": "chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
      "torchVersion": "2.10.0",
      "loadMs": 3678.4359160010354,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 466
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 1643.326707999222,
      "inferenceMs": 1642.9632090002997,
      "tokensPerSecond": 1.2173127122042786,
      "inputTokens": 68,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 6463127552,
      "cpuThreads": 1,
      "totalMs": 5321.762624000257,
      "grade": "slow",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": false,
      "expected": "hola"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
      "torchVersion": "2.10.0",
      "loadMs": 3678.4359160010354,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 466
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 1999.6779580033035,
      "inferenceMs": 1999.2964999983087,
      "tokensPerSecond": 1.5005278106586681,
      "inputTokens": 69,
      "outputTokens": 3,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 6476218368,
      "cpuThreads": 1,
      "totalMs": 5678.113874004339,
      "grade": "slow",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": false,
      "expected": "2026-11-04",
      "fixturePassed": true
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
      "torchVersion": "2.10.0",
      "loadMs": 3678.4359160010354,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 466
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 3420.493332996557,
      "inferenceMs": 3420.0012920045992,
      "tokensPerSecond": 2.046782852499164,
      "inputTokens": 72,
      "outputTokens": 7,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 6476218368,
      "cpuThreads": 1,
      "totalMs": 7098.929248997592,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    },
    "failure": "This fixture passed but the backend did not pass the complete repeated profile."
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": false,
      "expected": "Unknown"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d18cca4c121edb87d022116d281ce212c9136f57",
      "torchVersion": "2.10.0",
      "loadMs": 3678.4359160010354,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 466
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 1651.6368750017136,
      "inferenceMs": 1651.1553329983144,
      "tokensPerSecond": 1.2112730765119613,
      "inputTokens": 74,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 6476251136,
      "cpuThreads": 1,
      "totalMs": 5330.072791002749,
      "grade": "slow",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  }
]

Granite 4.0 Hybrid 350M

20 configurations tested
Passed12
Failed8
TestedNov 7, 2026
Runtimetransformers 5.22.1

Quality criteria

No explicit quality gate recorded. Review per-case evidence; full certification is pending.

PrecisionDeviceResultPlatformInference / load
FP16mpsAcceleratedPassedFastmacos · aarch64284 ms / 1212 ms
FP16mpsAcceleratedPassedFastmacos · aarch64301 ms / 1159 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64103 ms / 3600 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64105 ms / 1159 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64109 ms / 1212 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64399 ms / 3600 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64185 ms / 1212 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64210 ms / 1159 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64230 ms / 1212 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64284 ms / 1159 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64382 ms / 3600 ms
FP16mpsAcceleratedPassedModeratemacos · aarch641104 ms / 3600 ms
FP32cpuFailed—macos · aarch643470 ms / 2002 ms
FP32cpuFailed—macos · aarch648195 ms / 2002 ms
FP32cpuFailed—macos · aarch642988 ms / 2002 ms
FP32cpuFailed—macos · aarch644133 ms / 2002 ms
FP32cpuFailed—macos · aarch641157 ms / 1625 ms
FP32cpuFailed—macos · aarch641761 ms / 1625 ms
FP32cpuFailed—macos · aarch641007 ms / 1625 ms
FP32cpuFailed—macos · aarch641139 ms / 1625 ms
Test setup & sample details
Quality gate
No explicit quality gate recorded. Review the per-case quality evidence; full certification is pending.
Tested
2026-11-07T12:16:16.883639+00:00
Environment
{ "os": "macos", "arch": "aarch64", "hardware": "Apple M3 Max", "osVersion": "27.2" }
Runtime
transformers 5.22.1
Artifact revision
3b17b717b8f2f5d305b0a92c1491e239aeda19c8
Fixture
{
  "file": "model-samples/desktop/granite4/granite-4.0-h-350m-native-document-place.json",
  "size": 304,
  "sha1": "7dcdd80b3065bd555172453dbf8def706517421c",
  "sha256": "09d6cface3de3505ba7aceeabaaff63d9207d7e0596302661e2e987262d30589",
  "url": "https://assets.localai.tools/samples/v1/files/09d6cface3de3505ba7aceeabaaff63d9207d7e0596302661e2e987262d30589/granite-4.0-h-350m-native-document-place.json"
}
Rerun when
Artifact/runtime revision; Input/generation processing; Fixture/quality gates; OS/backend behavior
Per-case quality & failure details
[
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "plainParity": true,
      "passed": true,
      "expected": "Cedar"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "3b17b717b8f2f5d305b0a92c1491e239aeda19c8",
      "torchVersion": "2.10.0",
      "loadMs": 1158.8636669985135,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 370
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 295.71645799296675,
      "inferenceMs": 284.3197499969392,
      "tokensPerSecond": 10.551500555386307,
      "inputTokens": 76,
      "outputTokens": 3,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 1127825408,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 828424192,
      "cpuThreads": 4,
      "totalMs": 1454.5801249914803,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "2027-03-14"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "3b17b717b8f2f5d305b0a92c1491e239aeda19c8",
      "torchVersion": "2.10.0",
      "loadMs": 1158.8636669985135,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 370
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 303.1012919964269,
      "inferenceMs": 301.3171670027077,
      "tokensPerSecond": 23.231334841061663,
      "inputTokens": 79,
      "outputTokens": 7,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 1128513536,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 845545472,
      "cpuThreads": 4,
      "totalMs": 1461.9649589949404,
      "grade": "fast",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "Unknown"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "3b17b717b8f2f5d305b0a92c1491e239aeda19c8",
      "torchVersion": "2.10.0",
      "loadMs": 1158.8636669985135,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 370
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 107.04383299889741,
      "inferenceMs": 105.24533400166547,
      "tokensPerSecond": 19.003217757552566,
      "inputTokens": 76,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 1128513536,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 846020608,
      "cpuThreads": 4,
      "totalMs": 1265.907499997411,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "Dana"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "3b17b717b8f2f5d305b0a92c1491e239aeda19c8",
      "torchVersion": "2.10.0",
      "loadMs": 1158.8636669985135,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 370
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 211.8947089984431,
      "inferenceMs": 209.56008299981477,
      "tokensPerSecond": 14.315703434812304,
      "inputTokens": 71,
      "outputTokens": 3,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 1129103360,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 861863936,
      "cpuThreads": 4,
      "totalMs": 1370.7583759969566,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "Cedar"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "3b17b717b8f2f5d305b0a92c1491e239aeda19c8",
      "torchVersion": "2.10.0",
      "loadMs": 3599.5609579986194,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 370
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 1115.9589170056279,
      "inferenceMs": 1104.3106249999255,
      "tokensPerSecond": 2.7166269454305056,
      "inputTokens": 76,
      "outputTokens": 3,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 1127809024,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 821100544,
      "cpuThreads": 4,
      "totalMs": 4715.519875004247,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "2027-03-14"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "3b17b717b8f2f5d305b0a92c1491e239aeda19c8",
      "torchVersion": "2.10.0",
      "loadMs": 3599.5609579986194,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 370
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 401.51600000535836,
      "inferenceMs": 399.0823330022977,
      "tokensPerSecond": 17.54024024902074,
      "inputTokens": 79,
      "outputTokens": 7,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 1128513536,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 837550080,
      "cpuThreads": 4,
      "totalMs": 4001.076958003978,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "Unknown"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "3b17b717b8f2f5d305b0a92c1491e239aeda19c8",
      "torchVersion": "2.10.0",
      "loadMs": 3599.5609579986194,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 370
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 105.00237499945797,
      "inferenceMs": 103.29120800452074,
      "tokensPerSecond": 19.3627322076867,
      "inputTokens": 76,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 1128513536,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 837992448,
      "cpuThreads": 4,
      "totalMs": 3704.5633329980774,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "Dana"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "3b17b717b8f2f5d305b0a92c1491e239aeda19c8",
      "torchVersion": "2.10.0",
      "loadMs": 3599.5609579986194,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 370
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 383.9327499954379,
      "inferenceMs": 382.22833400504896,
      "tokensPerSecond": 7.848711707385806,
      "inputTokens": 71,
      "outputTokens": 3,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 1129103360,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 853852160,
      "cpuThreads": 4,
      "totalMs": 3983.4937079940573,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "Cedar"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "3b17b717b8f2f5d305b0a92c1491e239aeda19c8",
      "torchVersion": "2.10.0",
      "loadMs": 1211.986207999871,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 370
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 242.2459999943385,
      "inferenceMs": 230.48787500010803,
      "tokensPerSecond": 13.015869056012573,
      "inputTokens": 76,
      "outputTokens": 3,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 1127825408,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 824459264,
      "cpuThreads": 4,
      "totalMs": 1454.2322079942096,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "2027-03-14"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "3b17b717b8f2f5d305b0a92c1491e239aeda19c8",
      "torchVersion": "2.10.0",
      "loadMs": 1211.986207999871,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 370
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 285.60583300713915,
      "inferenceMs": 283.85195799637586,
      "tokensPerSecond": 24.66074234404039,
      "inputTokens": 79,
      "outputTokens": 7,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 1128529920,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 841728000,
      "cpuThreads": 4,
      "totalMs": 1497.5920410070103,
      "grade": "fast",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "Unknown"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "3b17b717b8f2f5d305b0a92c1491e239aeda19c8",
      "torchVersion": "2.10.0",
      "loadMs": 1211.986207999871,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 370
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 110.52754100091988,
      "inferenceMs": 108.57416599901626,
      "tokensPerSecond": 18.42058819054729,
      "inputTokens": 76,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 1128513536,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 842317824,
      "cpuThreads": 4,
      "totalMs": 1322.513749000791,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "Dana"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "3b17b717b8f2f5d305b0a92c1491e239aeda19c8",
      "torchVersion": "2.10.0",
      "loadMs": 1211.986207999871,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 370
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 186.4249159989413,
      "inferenceMs": 184.65708299481776,
      "tokensPerSecond": 16.246330502709135,
      "inputTokens": 71,
      "outputTokens": 3,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 1129103360,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 858308608,
      "cpuThreads": 4,
      "totalMs": 1398.4111239988124,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "plainParity": true,
      "passed": false,
      "expected": "Cedar"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "3b17b717b8f2f5d305b0a92c1491e239aeda19c8",
      "torchVersion": "2.10.0",
      "loadMs": 2002.2069170008763,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 370
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 3479.801708002924,
      "inferenceMs": 3470.2159160005976,
      "tokensPerSecond": 0.8644995218215359,
      "inputTokens": 76,
      "outputTokens": 3,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 1897349120,
      "cpuThreads": 4,
      "totalMs": 5482.0086250038,
      "grade": "slow",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": false,
      "expected": "2027-03-14"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "3b17b717b8f2f5d305b0a92c1491e239aeda19c8",
      "torchVersion": "2.10.0",
      "loadMs": 2002.2069170008763,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 370
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 8195.009707997087,
      "inferenceMs": 8194.530834000034,
      "tokensPerSecond": 0.8542282824730135,
      "inputTokens": 79,
      "outputTokens": 7,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 1902346240,
      "cpuThreads": 4,
      "totalMs": 10197.216624997964,
      "grade": "slow",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": false,
      "expected": "Unknown"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "3b17b717b8f2f5d305b0a92c1491e239aeda19c8",
      "torchVersion": "2.10.0",
      "loadMs": 2002.2069170008763,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 370
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 2988.5040419976576,
      "inferenceMs": 2988.031792003312,
      "tokensPerSecond": 0.6693369211641182,
      "inputTokens": 76,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 1902362624,
      "cpuThreads": 4,
      "totalMs": 4990.710958998534,
      "grade": "slow",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": false,
      "expected": "Dana"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "3b17b717b8f2f5d305b0a92c1491e239aeda19c8",
      "torchVersion": "2.10.0",
      "loadMs": 2002.2069170008763,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 370
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 4134.1128749991185,
      "inferenceMs": 4132.940999996208,
      "tokensPerSecond": 0.7258753512336016,
      "inputTokens": 71,
      "outputTokens": 3,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 1902395392,
      "cpuThreads": 4,
      "totalMs": 6136.319791999995,
      "grade": "slow",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "plainParity": true,
      "passed": false,
      "expected": "Cedar",
      "fixturePassed": true
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "3b17b717b8f2f5d305b0a92c1491e239aeda19c8",
      "torchVersion": "2.10.0",
      "loadMs": 1625.3247919958085,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 370
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 1167.05683300097,
      "inferenceMs": 1157.2270000033313,
      "tokensPerSecond": 2.5924040832017954,
      "inputTokens": 76,
      "outputTokens": 3,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 1897660416,
      "cpuThreads": 1,
      "totalMs": 2792.3816249967786,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    },
    "failure": "This fixture passed but the backend did not pass the complete repeated profile."
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": false,
      "expected": "2027-03-14",
      "fixturePassed": true
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "3b17b717b8f2f5d305b0a92c1491e239aeda19c8",
      "torchVersion": "2.10.0",
      "loadMs": 1625.3247919958085,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 370
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 1761.850166003569,
      "inferenceMs": 1761.3827090026462,
      "tokensPerSecond": 3.974150514946087,
      "inputTokens": 79,
      "outputTokens": 7,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 1902592000,
      "cpuThreads": 1,
      "totalMs": 3387.1749579993775,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    },
    "failure": "This fixture passed but the backend did not pass the complete repeated profile."
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": false,
      "expected": "Unknown"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "3b17b717b8f2f5d305b0a92c1491e239aeda19c8",
      "torchVersion": "2.10.0",
      "loadMs": 1625.3247919958085,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 370
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 1007.1682919951854,
      "inferenceMs": 1006.7262499942444,
      "tokensPerSecond": 1.9866373803319763,
      "inputTokens": 76,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 1902592000,
      "cpuThreads": 1,
      "totalMs": 2632.493083990994,
      "grade": "slow",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": false,
      "expected": "Dana",
      "fixturePassed": true
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "3b17b717b8f2f5d305b0a92c1491e239aeda19c8",
      "torchVersion": "2.10.0",
      "loadMs": 1625.3247919958085,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 370
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 1139.8297499981709,
      "inferenceMs": 1139.3076250024023,
      "tokensPerSecond": 2.633178198902754,
      "inputTokens": 71,
      "outputTokens": 3,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 1902592000,
      "cpuThreads": 1,
      "totalMs": 2765.1545419939794,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    },
    "failure": "This fixture passed but the backend did not pass the complete repeated profile."
  }
]

Granite 4.0 Hybrid Micro

30 configurations tested
Passed18
Failed12
TestedNov 7, 2026
Runtimetransformers 5.22.1

Quality criteria

No explicit quality gate recorded. Review per-case evidence; full certification is pending.

PrecisionDeviceResultPlatformInference / load
FP16mpsAcceleratedPassedModeratemacos · aarch64489 ms / 4238 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64492 ms / 1925 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64501 ms / 1962 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64344 ms / 1925 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64346 ms / 1962 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64234 ms / 1925 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64237 ms / 1962 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64357 ms / 4238 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64238 ms / 4238 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64293 ms / 1925 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64305 ms / 1925 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64307 ms / 1962 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64307 ms / 1925 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64311 ms / 4238 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64311 ms / 1962 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64312 ms / 1962 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64362 ms / 4238 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64369 ms / 4238 ms
FP32cpuFailed—macos · aarch646014 ms / 4618 ms
FP32cpuFailed—macos · aarch645414 ms / 4618 ms
FP32cpuFailed—macos · aarch645755 ms / 4618 ms
FP32cpuFailed—macos · aarch648517 ms / 4618 ms
FP32cpuFailed—macos · aarch6417597 ms / 4618 ms
FP32cpuFailed—macos · aarch645506 ms / 4618 ms
FP32cpuFailed—macos · aarch642417 ms / 6964 ms
FP32cpuFailed—macos · aarch642600 ms / 6964 ms
FP32cpuFailed—macos · aarch642958 ms / 6964 ms
FP32cpuFailed—macos · aarch643573 ms / 6964 ms
FP32cpuFailed—macos · aarch645862 ms / 6964 ms
FP32cpuFailed—macos · aarch642611 ms / 6964 ms
Test setup & sample details
Quality gate
No explicit quality gate recorded. Review the per-case quality evidence; full certification is pending.
Tested
2026-11-07T12:22:53.450203+00:00
Environment
{ "os": "macos", "arch": "aarch64", "hardware": "Apple M3 Max", "osVersion": "27.2" }
Runtime
transformers 5.22.1
Artifact revision
d5f01a3ea75f088947be3aae039f4ad52837dfde
Fixture
{
  "file": "model-samples/desktop/granite4/granite-4.0-h-micro-native-capital.json",
  "size": 244,
  "sha1": "9013b4f5a71cd1280d78b3dfa751ead1186d071f",
  "sha256": "8ff34789fa2a0f5bbef2080057f4b36376d3c73ea64baccefb6caec96eafbd68",
  "url": "https://assets.localai.tools/samples/v1/files/8ff34789fa2a0f5bbef2080057f4b36376d3c73ea64baccefb6caec96eafbd68/granite-4.0-h-micro-native-capital.json"
}
Rerun when
Artifact/runtime revision; Input/generation processing; Fixture/quality gates; OS/backend behavior
Per-case quality & failure details
[
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "plainParity": true,
      "passed": true,
      "expected": "Paris"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
      "torchVersion": "2.10.0",
      "loadMs": 4238.254416995915,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 466
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 380.27199999487493,
      "inferenceMs": 369.1712500003632,
      "tokensPerSecond": 5.417539962816802,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 7554039808,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 2933342208,
      "cpuThreads": 4,
      "totalMs": 4618.52641699079,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "56"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
      "torchVersion": "2.10.0",
      "loadMs": 4238.254416995915,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 466
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 239.99333300162107,
      "inferenceMs": 238.32175000279676,
      "tokensPerSecond": 8.392016255237005,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 7554039808,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 2934128640,
      "cpuThreads": 4,
      "totalMs": 4478.247749997536,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
      "torchVersion": "2.10.0",
      "loadMs": 4238.254416995915,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 466
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 363.3048750052694,
      "inferenceMs": 361.5772499979357,
      "tokensPerSecond": 5.5313214534692605,
      "inputTokens": 68,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 7563755520,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 2951036928,
      "cpuThreads": 4,
      "totalMs": 4601.559292001184,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "hola"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
      "torchVersion": "2.10.0",
      "loadMs": 4238.254416995915,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 466
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 358.55141700449167,
      "inferenceMs": 357.23649999999907,
      "tokensPerSecond": 8.3977980973389,
      "inputTokens": 69,
      "outputTokens": 3,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 7565115392,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 2966192128,
      "cpuThreads": 4,
      "totalMs": 4596.805834000406,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "2026-11-04"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
      "torchVersion": "2.10.0",
      "loadMs": 4238.254416995915,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 466
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 491.55462499766145,
      "inferenceMs": 489.41774999548215,
      "tokensPerSecond": 14.302709699565693,
      "inputTokens": 72,
      "outputTokens": 7,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 7566589952,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 2981249024,
      "cpuThreads": 4,
      "totalMs": 4729.809041993576,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "Unknown"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
      "torchVersion": "2.10.0",
      "loadMs": 4238.254416995915,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 466
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 312.47787499887636,
      "inferenceMs": 310.7872909968137,
      "tokensPerSecond": 6.435269581279322,
      "inputTokens": 74,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 7568146432,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 2996502528,
      "cpuThreads": 4,
      "totalMs": 4550.732291994791,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "Paris"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
      "torchVersion": "2.10.0",
      "loadMs": 1925.2609999966808,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 466
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 303.6869579955237,
      "inferenceMs": 292.9782499995781,
      "tokensPerSecond": 6.8264453078099825,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 7554039808,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 2935439360,
      "cpuThreads": 4,
      "totalMs": 2228.9479579922045,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "56"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
      "torchVersion": "2.10.0",
      "loadMs": 1925.2609999966808,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 466
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 235.4229580014362,
      "inferenceMs": 233.82262499944773,
      "tokensPerSecond": 8.553492203779356,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 7554039808,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 2936766464,
      "cpuThreads": 4,
      "totalMs": 2160.683957998117,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
      "torchVersion": "2.10.0",
      "loadMs": 1925.2609999966808,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 466
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 306.73191699315794,
      "inferenceMs": 305.0386670001899,
      "tokensPerSecond": 6.556545829643148,
      "inputTokens": 68,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 7563755520,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 2954297344,
      "cpuThreads": 4,
      "totalMs": 2231.9929169898387,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "hola"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
      "torchVersion": "2.10.0",
      "loadMs": 1925.2609999966808,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 466
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 345.6560839986196,
      "inferenceMs": 343.80783300002804,
      "tokensPerSecond": 8.725804685199698,
      "inputTokens": 69,
      "outputTokens": 3,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 7565115392,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 2970222592,
      "cpuThreads": 4,
      "totalMs": 2270.9170839953003,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "2026-11-04"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
      "torchVersion": "2.10.0",
      "loadMs": 1925.2609999966808,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 466
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 493.6798339986126,
      "inferenceMs": 491.99379199853865,
      "tokensPerSecond": 14.227821801501088,
      "inputTokens": 72,
      "outputTokens": 7,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 7566589952,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 2985934848,
      "cpuThreads": 4,
      "totalMs": 2418.9408339952934,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "Unknown"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
      "torchVersion": "2.10.0",
      "loadMs": 1925.2609999966808,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 466
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 308.8843329969677,
      "inferenceMs": 307.2142500022892,
      "tokensPerSecond": 6.510114683759289,
      "inputTokens": 74,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 7568146432,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 3000975360,
      "cpuThreads": 4,
      "totalMs": 2234.1453329936485,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "Paris"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
      "torchVersion": "2.10.0",
      "loadMs": 1961.5091249943362,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 466
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 318.30379200255265,
      "inferenceMs": 306.7352090001805,
      "tokensPerSecond": 6.5202817978382885,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 7554039808,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 2934865920,
      "cpuThreads": 4,
      "totalMs": 2279.812916996889,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "56"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
      "torchVersion": "2.10.0",
      "loadMs": 1961.5091249943362,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 466
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 238.54745899734553,
      "inferenceMs": 237.10237499471987,
      "tokensPerSecond": 8.435174890359233,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 7554039808,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 2936815616,
      "cpuThreads": 4,
      "totalMs": 2200.0565839916817,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
      "torchVersion": "2.10.0",
      "loadMs": 1961.5091249943362,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 466
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 312.9267920012353,
      "inferenceMs": 311.21004200394964,
      "tokensPerSecond": 6.4265278431298745,
      "inputTokens": 68,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 7563755520,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 2954395648,
      "cpuThreads": 4,
      "totalMs": 2274.4359169955715,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "hola"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
      "torchVersion": "2.10.0",
      "loadMs": 1961.5091249943362,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 466
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 347.84408300038194,
      "inferenceMs": 346.2073339978815,
      "tokensPerSecond": 8.66532769643279,
      "inputTokens": 69,
      "outputTokens": 3,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 7565115392,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 2970320896,
      "cpuThreads": 4,
      "totalMs": 2309.353207994718,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "2026-11-04"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
      "torchVersion": "2.10.0",
      "loadMs": 1961.5091249943362,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 466
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 502.90058300015517,
      "inferenceMs": 500.97775000176625,
      "tokensPerSecond": 13.97267643118945,
      "inputTokens": 72,
      "outputTokens": 7,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 7566589952,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 2985754624,
      "cpuThreads": 4,
      "totalMs": 2464.4097079944913,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "Unknown"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
      "torchVersion": "2.10.0",
      "loadMs": 1961.5091249943362,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 466
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 314.43504100025166,
      "inferenceMs": 312.2585410019383,
      "tokensPerSecond": 6.404948904144099,
      "inputTokens": 74,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 7568146432,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "rssBytes": 3000975360,
      "cpuThreads": 4,
      "totalMs": 2275.944165994588,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "plainParity": true,
      "passed": false,
      "expected": "Paris"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
      "torchVersion": "2.10.0",
      "loadMs": 4617.936540998926,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 466
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 6023.984749997908,
      "inferenceMs": 6013.853958000254,
      "tokensPerSecond": 0.33256544205557104,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 13423312896,
      "cpuThreads": 4,
      "totalMs": 10641.921290996834,
      "grade": "slow",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": false,
      "expected": "56"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
      "torchVersion": "2.10.0",
      "loadMs": 4617.936540998926,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 466
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 5414.4182500022,
      "inferenceMs": 5413.9613330044085,
      "tokensPerSecond": 0.36941527229011917,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 13427589120,
      "cpuThreads": 4,
      "totalMs": 10032.354791001126,
      "grade": "slow",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": false,
      "expected": "chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
      "torchVersion": "2.10.0",
      "loadMs": 4617.936540998926,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 466
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 5755.799167003715,
      "inferenceMs": 5755.414541999926,
      "tokensPerSecond": 0.34749886136004166,
      "inputTokens": 68,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 13427818496,
      "cpuThreads": 4,
      "totalMs": 10373.735708002641,
      "grade": "slow",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": false,
      "expected": "hola"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
      "torchVersion": "2.10.0",
      "loadMs": 4617.936540998926,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 466
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 8517.614333002712,
      "inferenceMs": 8517.201750000822,
      "tokensPerSecond": 0.352228359507829,
      "inputTokens": 69,
      "outputTokens": 3,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 13432111104,
      "cpuThreads": 4,
      "totalMs": 13135.550874001638,
      "grade": "slow",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": false,
      "expected": "2026-11-04"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
      "torchVersion": "2.10.0",
      "loadMs": 4617.936540998926,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 466
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 17596.97354200034,
      "inferenceMs": 17596.55304199987,
      "tokensPerSecond": 0.39780518282712723,
      "inputTokens": 72,
      "outputTokens": 7,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 13432127488,
      "cpuThreads": 4,
      "totalMs": 22214.910082999268,
      "grade": "slow",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": false,
      "expected": "Unknown"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
      "torchVersion": "2.10.0",
      "loadMs": 4617.936540998926,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 466
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 5506.182166995131,
      "inferenceMs": 5505.780542000139,
      "tokensPerSecond": 0.36325458029851665,
      "inputTokens": 74,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 13432143872,
      "cpuThreads": 4,
      "totalMs": 10124.118707994057,
      "grade": "slow",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "plainParity": true,
      "passed": false,
      "expected": "Paris"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
      "torchVersion": "2.10.0",
      "loadMs": 6963.629167003091,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 466
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 2427.8841669947724,
      "inferenceMs": 2417.44362499594,
      "tokensPerSecond": 0.8273202234461038,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 13416546304,
      "cpuThreads": 1,
      "totalMs": 9391.513333997864,
      "grade": "slow",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": false,
      "expected": "56"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
      "torchVersion": "2.10.0",
      "loadMs": 6963.629167003091,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 466
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 2600.4014999998617,
      "inferenceMs": 2599.905208000564,
      "tokensPerSecond": 0.7692588152235303,
      "inputTokens": 63,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 13420773376,
      "cpuThreads": 1,
      "totalMs": 9564.030667002953,
      "grade": "slow",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": false,
      "expected": "chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
      "torchVersion": "2.10.0",
      "loadMs": 6963.629167003091,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 466
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 2958.3785840004566,
      "inferenceMs": 2957.889666999108,
      "tokensPerSecond": 0.6761577425668741,
      "inputTokens": 68,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 13420806144,
      "cpuThreads": 1,
      "totalMs": 9922.007751003548,
      "grade": "slow",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": false,
      "expected": "hola"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
      "torchVersion": "2.10.0",
      "loadMs": 6963.629167003091,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 466
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 3573.3351669987314,
      "inferenceMs": 3572.786709002685,
      "tokensPerSecond": 0.839680687470265,
      "inputTokens": 69,
      "outputTokens": 3,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 13433454592,
      "cpuThreads": 1,
      "totalMs": 10536.964334001823,
      "grade": "slow",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": false,
      "expected": "2026-11-04"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
      "torchVersion": "2.10.0",
      "loadMs": 6963.629167003091,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 466
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 5862.599625004805,
      "inferenceMs": 5862.039291001565,
      "tokensPerSecond": 1.1941236918601423,
      "inputTokens": 72,
      "outputTokens": 7,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 13458685952,
      "cpuThreads": 1,
      "totalMs": 12826.228792007896,
      "grade": "slow",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "exactNormalized": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": false,
      "expected": "Unknown"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "artifactRevision": "d5f01a3ea75f088947be3aae039f4ad52837dfde",
      "torchVersion": "2.10.0",
      "loadMs": 6963.629167003091,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 466
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 2611.811167000269,
      "inferenceMs": 2611.232666997239,
      "tokensPerSecond": 0.7659217906077592,
      "inputTokens": 74,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "outputLimitReached": false,
      "rssBytes": 13458685952,
      "cpuThreads": 1,
      "totalMs": 9575.44033400336,
      "grade": "slow",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  }
]

Granite 4.0 Hybrid Small

30 configurations tested
Passed18
Failed12
TestedNov 7, 2026
Runtimellama.cpp 0.3.36

Quality criteria

No explicit quality gate recorded. Review per-case evidence; full certification is pending.

PrecisionDeviceResultPlatformInference / load
Q4mpsAcceleratedPassedModeratemacos · aarch64552 ms / 1374 ms
Q4mpsAcceleratedPassedModeratemacos · aarch64558 ms / 1278 ms
Q4mpsAcceleratedPassedModeratemacos · aarch64564 ms / 1559 ms
Q4mpsAcceleratedPassedModeratemacos · aarch64402 ms / 1559 ms
Q4mpsAcceleratedPassedModeratemacos · aarch64403 ms / 1278 ms
Q4mpsAcceleratedPassedModeratemacos · aarch64405 ms / 1374 ms
Q4mpsAcceleratedPassedModeratemacos · aarch64330 ms / 1278 ms
Q4mpsAcceleratedPassedModeratemacos · aarch64333 ms / 1559 ms
Q4mpsAcceleratedPassedModeratemacos · aarch64334 ms / 1374 ms
Q4mpsAcceleratedPassedModeratemacos · aarch64357 ms / 1278 ms
Q4mpsAcceleratedPassedModeratemacos · aarch64358 ms / 1374 ms
Q4mpsAcceleratedPassedModeratemacos · aarch64365 ms / 1374 ms
Q4mpsAcceleratedPassedModeratemacos · aarch64366 ms / 1559 ms
Q4mpsAcceleratedPassedModeratemacos · aarch64366 ms / 1278 ms
Q4mpsAcceleratedPassedModeratemacos · aarch64378 ms / 1374 ms
Q4mpsAcceleratedPassedModeratemacos · aarch64379 ms / 1559 ms
Q4mpsAcceleratedPassedModeratemacos · aarch64385 ms / 1278 ms
Q4mpsAcceleratedPassedModeratemacos · aarch64412 ms / 1559 ms
Q4cpuFailed—macos · aarch643833 ms / 34820 ms
Q4cpuFailed—macos · aarch642350 ms / 34820 ms
Q4cpuFailed—macos · aarch642191 ms / 34820 ms
Q4cpuFailed—macos · aarch642424 ms / 34820 ms
Q4cpuFailed—macos · aarch642686 ms / 34820 ms
Q4cpuFailed—macos · aarch642411 ms / 34820 ms
Q4cpuFailed—macos · aarch647620 ms / 19608 ms
Q4cpuFailed—macos · aarch647652 ms / 19608 ms
Q4cpuFailed—macos · aarch648072 ms / 19608 ms
Q4cpuFailed—macos · aarch648438 ms / 19608 ms
Q4cpuFailed—macos · aarch649486 ms / 19608 ms
Q4cpuFailed—macos · aarch648769 ms / 19608 ms
Test setup & sample details
Quality gate
No explicit quality gate recorded. Review the per-case quality evidence; full certification is pending.
Tested
2026-11-07T12:55:47.397494+00:00
Environment
{ "os": "macos", "arch": "aarch64", "hardware": "Apple M3 Max", "osVersion": "27.2" }
Runtime
llama.cpp 0.3.36
Artifact revision
6522095069a86fc3186b8632850e4373dd850cac
Fixture
{
  "file": "model-samples/desktop/granite4/granite-4.0-h-small-native-capital.json",
  "size": 244,
  "sha1": "9013b4f5a71cd1280d78b3dfa751ead1186d071f",
  "sha256": "8ff34789fa2a0f5bbef2080057f4b36376d3c73ea64baccefb6caec96eafbd68",
  "url": "https://assets.localai.tools/samples/v1/files/8ff34789fa2a0f5bbef2080057f4b36376d3c73ea64baccefb6caec96eafbd68/granite-4.0-h-small-native-capital.json"
}
Rerun when
Artifact/runtime revision; Input/generation processing; Fixture/quality gates; OS/backend behavior
Per-case quality & failure details
[
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "plainParity": true,
      "passed": true,
      "expected": "Paris"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "loadMs": 1558.7679159943946,
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer   0 assigned to device MTL0, is_swa = 0\nload_tensors: layer   1 assigned to device MTL0, is_swa = 0\nload_tensors: layer   2 assigned to device MTL0, is_swa = 0\nload_tensors: layer   3 assigned to device MTL0, is_swa = 0\nload_tensors: layer   4 assigned to device MTL0, is_swa = 0\nload_tensors: layer   5 assigned to device MTL0, is_swa = 0\nload_tensors: layer   6 assigned to device MTL0, is_swa = 0\nload_tensors: layer   7 assigned to device MTL0, is_swa = 0\nload_tensors: layer   8 assigned to device MTL0, is_swa = 0\nload_tensors: layer   9 assigned to device MTL0, is_swa = 0\nload_tensors: layer  10 assigned to device MTL0, is_swa = 0\nload_tensors: layer  11 assigned to device MTL0, is_swa = 0\nload_tensors: layer  12 assigned to device MTL0, is_swa = 0\nload_tensors: layer  13 assigned to device MTL0, is_swa = 0\nload_tensors: layer  14 assigned to device MTL0, is_swa = 0\nload_tensors: layer  15 assigned to device MTL0, is_swa = 0\nload_tensors: layer  16 assigned to device MTL0, is_swa = 0\nload_tensors: layer  17 assigned to device MTL0, is_swa = 0\nload_tensors: layer  18 assigned to device MTL0, is_swa = 0\nload_tensors: layer  19 assigned to device MTL0, is_swa = 0\nload_tensors: layer  20 assigned to device MTL0, is_swa = 0\nload_tensors: layer  21 assigned to device MTL0, is_swa = 0\nload_tensors: layer  22 assigned to device MTL0, is_swa = 0\nload_tensors: layer  23 assigned to device MTL0, is_swa = 0\nload_tensors: layer  24 assigned to device MTL0, is_swa = 0\nload_tensors: layer  25 assigned to device MTL0, is_swa = 0\nload_tensors: layer  26 assigned to device MTL0, is_swa = 0\nload_tensors: layer  27 assigned to device MTL0, is_swa = 0\nload_tensors: layer  28 assigned to device MTL0, is_swa = 0\nload_tensors: layer  29 assigned to device MTL0, is_swa = 0\nload_tensors: layer  30 assigned to device MTL0, is_swa = 0\nload_tensors: layer  31 assigned to device MTL0, is_swa = 0\nload_tensors: layer  32 assigned to device MTL0, is_swa = 0\nload_tensors: layer  33 assigned to device MTL0, is_swa = 0\nload_tensors: layer  34 assigned to device MTL0, is_swa = 0\nload_tensors: layer  35 assigned to device MTL0, is_swa = 0\nload_tensors: layer  36 assigned to device MTL0, is_swa = 0\nload_tensors: layer  37 assigned to device MTL0, is_swa = 0\nload_tensors: layer  38 assigned to device MTL0, is_swa = 0\nload_tensors: layer  39 assigned to device MTL0, is_swa = 0\nload_tensors: layer  40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =   321.56 MiB\nload_tensors:  MTL0_Mapped model buffer size = 18570.93 MiB\nllama_kv_cache: layer   5: dev = MTL0\nllama_kv_cache: layer  15: dev = MTL0\nllama_kv_cache: layer  25: dev = MTL0\nllama_kv_cache: layer  35: dev = MTL0\nllama_kv_cache:       MTL0 KV buffer size =   128.00 MiB\nllama_memory_recurrent, layer   0: dev = MTL0\nllama_memory_recurrent, layer   1: dev = MTL0\nllama_memory_recurrent, layer   2: dev = MTL0\nllama_memory_recurrent, layer   3: dev = MTL0\nllama_memory_recurrent, layer   4: dev = MTL0\nllama_memory_recurrent, layer   6: dev = MTL0\nllama_memory_recurrent, layer   7: dev = MTL0\nllama_memory_recurrent, layer   8: dev = MTL0\nllama_memory_recurrent, layer   9: dev = MTL0\nllama_memory_recurrent, layer  10: dev = MTL0\nllama_memory_recurrent, layer  11: dev = MTL0\nllama_memory_recurrent, layer  12: dev = MTL0\nllama_memory_recurrent, layer  13: dev = MTL0\nllama_memory_recurrent, layer  14: dev = MTL0\nllama_memory_recurrent, layer  16: dev = MTL0\nllama_memory_recurrent, layer  17: dev = MTL0\nllama_memory_recurrent, layer  18: dev = MTL0\nllama_memory_recurrent, layer  19: dev = MTL0\nllama_memory_recurrent, layer  20: dev = MTL0\nllama_memory_recurrent, layer  21: dev = MTL0\nllama_memory_recurrent, layer  22: dev = MTL0\nllama_memory_recurrent, layer  23: dev = MTL0\nllama_memory_recurrent, layer  24: dev = MTL0\nllama_memory_recurrent, layer  26: dev = MTL0\nllama_memory_recurrent, layer  27: dev = MTL0\nllama_memory_recurrent, layer  28: dev = MTL0\nllama_memory_recurrent, layer  29: dev = MTL0\nllama_memory_recurrent, layer  30: dev = MTL0\nllama_memory_recurrent, layer  31: dev = MTL0\nllama_memory_recurrent, layer  32: dev = MTL0\nllama_memory_recurrent, layer  33: dev = MTL0\nllama_memory_recurrent, layer  34: dev = MTL0\nllama_memory_recurrent, layer  36: dev = MTL0\nllama_memory_recurrent, layer  37: dev = MTL0\nllama_memory_recurrent, layer  38: dev = MTL0\nllama_memory_recurrent, layer  39: dev = MTL0\nllama_memory_recurrent:       MTL0 RS buffer size =   147.48 MiB\nsched_reserve:       MTL0 compute buffer size =   282.28 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 41,
      "totalLayers": 41,
      "inferenceMs": 411.71433399722446,
      "durationMs": 421.66816600365564,
      "tokensPerSecond": 4.857737112484121,
      "inputTokens": 63,
      "outputTokens": 2,
      "outputLimitReached": false,
      "rssBytes": 19944783872,
      "cpuThreads": 4,
      "totalMs": 1980.4360819980502,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "56"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "loadMs": 1558.7679159943946,
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer   0 assigned to device MTL0, is_swa = 0\nload_tensors: layer   1 assigned to device MTL0, is_swa = 0\nload_tensors: layer   2 assigned to device MTL0, is_swa = 0\nload_tensors: layer   3 assigned to device MTL0, is_swa = 0\nload_tensors: layer   4 assigned to device MTL0, is_swa = 0\nload_tensors: layer   5 assigned to device MTL0, is_swa = 0\nload_tensors: layer   6 assigned to device MTL0, is_swa = 0\nload_tensors: layer   7 assigned to device MTL0, is_swa = 0\nload_tensors: layer   8 assigned to device MTL0, is_swa = 0\nload_tensors: layer   9 assigned to device MTL0, is_swa = 0\nload_tensors: layer  10 assigned to device MTL0, is_swa = 0\nload_tensors: layer  11 assigned to device MTL0, is_swa = 0\nload_tensors: layer  12 assigned to device MTL0, is_swa = 0\nload_tensors: layer  13 assigned to device MTL0, is_swa = 0\nload_tensors: layer  14 assigned to device MTL0, is_swa = 0\nload_tensors: layer  15 assigned to device MTL0, is_swa = 0\nload_tensors: layer  16 assigned to device MTL0, is_swa = 0\nload_tensors: layer  17 assigned to device MTL0, is_swa = 0\nload_tensors: layer  18 assigned to device MTL0, is_swa = 0\nload_tensors: layer  19 assigned to device MTL0, is_swa = 0\nload_tensors: layer  20 assigned to device MTL0, is_swa = 0\nload_tensors: layer  21 assigned to device MTL0, is_swa = 0\nload_tensors: layer  22 assigned to device MTL0, is_swa = 0\nload_tensors: layer  23 assigned to device MTL0, is_swa = 0\nload_tensors: layer  24 assigned to device MTL0, is_swa = 0\nload_tensors: layer  25 assigned to device MTL0, is_swa = 0\nload_tensors: layer  26 assigned to device MTL0, is_swa = 0\nload_tensors: layer  27 assigned to device MTL0, is_swa = 0\nload_tensors: layer  28 assigned to device MTL0, is_swa = 0\nload_tensors: layer  29 assigned to device MTL0, is_swa = 0\nload_tensors: layer  30 assigned to device MTL0, is_swa = 0\nload_tensors: layer  31 assigned to device MTL0, is_swa = 0\nload_tensors: layer  32 assigned to device MTL0, is_swa = 0\nload_tensors: layer  33 assigned to device MTL0, is_swa = 0\nload_tensors: layer  34 assigned to device MTL0, is_swa = 0\nload_tensors: layer  35 assigned to device MTL0, is_swa = 0\nload_tensors: layer  36 assigned to device MTL0, is_swa = 0\nload_tensors: layer  37 assigned to device MTL0, is_swa = 0\nload_tensors: layer  38 assigned to device MTL0, is_swa = 0\nload_tensors: layer  39 assigned to device MTL0, is_swa = 0\nload_tensors: layer  40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =   321.56 MiB\nload_tensors:  MTL0_Mapped model buffer size = 18570.93 MiB\nllama_kv_cache: layer   5: dev = MTL0\nllama_kv_cache: layer  15: dev = MTL0\nllama_kv_cache: layer  25: dev = MTL0\nllama_kv_cache: layer  35: dev = MTL0\nllama_kv_cache:       MTL0 KV buffer size =   128.00 MiB\nllama_memory_recurrent, layer   0: dev = MTL0\nllama_memory_recurrent, layer   1: dev = MTL0\nllama_memory_recurrent, layer   2: dev = MTL0\nllama_memory_recurrent, layer   3: dev = MTL0\nllama_memory_recurrent, layer   4: dev = MTL0\nllama_memory_recurrent, layer   6: dev = MTL0\nllama_memory_recurrent, layer   7: dev = MTL0\nllama_memory_recurrent, layer   8: dev = MTL0\nllama_memory_recurrent, layer   9: dev = MTL0\nllama_memory_recurrent, layer  10: dev = MTL0\nllama_memory_recurrent, layer  11: dev = MTL0\nllama_memory_recurrent, layer  12: dev = MTL0\nllama_memory_recurrent, layer  13: dev = MTL0\nllama_memory_recurrent, layer  14: dev = MTL0\nllama_memory_recurrent, layer  16: dev = MTL0\nllama_memory_recurrent, layer  17: dev = MTL0\nllama_memory_recurrent, layer  18: dev = MTL0\nllama_memory_recurrent, layer  19: dev = MTL0\nllama_memory_recurrent, layer  20: dev = MTL0\nllama_memory_recurrent, layer  21: dev = MTL0\nllama_memory_recurrent, layer  22: dev = MTL0\nllama_memory_recurrent, layer  23: dev = MTL0\nllama_memory_recurrent, layer  24: dev = MTL0\nllama_memory_recurrent, layer  26: dev = MTL0\nllama_memory_recurrent, layer  27: dev = MTL0\nllama_memory_recurrent, layer  28: dev = MTL0\nllama_memory_recurrent, layer  29: dev = MTL0\nllama_memory_recurrent, layer  30: dev = MTL0\nllama_memory_recurrent, layer  31: dev = MTL0\nllama_memory_recurrent, layer  32: dev = MTL0\nllama_memory_recurrent, layer  33: dev = MTL0\nllama_memory_recurrent, layer  34: dev = MTL0\nllama_memory_recurrent, layer  36: dev = MTL0\nllama_memory_recurrent, layer  37: dev = MTL0\nllama_memory_recurrent, layer  38: dev = MTL0\nllama_memory_recurrent, layer  39: dev = MTL0\nllama_memory_recurrent:       MTL0 RS buffer size =   147.48 MiB\nsched_reserve:       MTL0 compute buffer size =   282.28 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 41,
      "totalLayers": 41,
      "inferenceMs": 333.4497090036166,
      "durationMs": 343.2479159964714,
      "tokensPerSecond": 5.997905969017679,
      "inputTokens": 63,
      "outputTokens": 2,
      "outputLimitReached": false,
      "rssBytes": 19946078208,
      "cpuThreads": 4,
      "totalMs": 1902.015831990866,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "chat"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "loadMs": 1558.7679159943946,
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer   0 assigned to device MTL0, is_swa = 0\nload_tensors: layer   1 assigned to device MTL0, is_swa = 0\nload_tensors: layer   2 assigned to device MTL0, is_swa = 0\nload_tensors: layer   3 assigned to device MTL0, is_swa = 0\nload_tensors: layer   4 assigned to device MTL0, is_swa = 0\nload_tensors: layer   5 assigned to device MTL0, is_swa = 0\nload_tensors: layer   6 assigned to device MTL0, is_swa = 0\nload_tensors: layer   7 assigned to device MTL0, is_swa = 0\nload_tensors: layer   8 assigned to device MTL0, is_swa = 0\nload_tensors: layer   9 assigned to device MTL0, is_swa = 0\nload_tensors: layer  10 assigned to device MTL0, is_swa = 0\nload_tensors: layer  11 assigned to device MTL0, is_swa = 0\nload_tensors: layer  12 assigned to device MTL0, is_swa = 0\nload_tensors: layer  13 assigned to device MTL0, is_swa = 0\nload_tensors: layer  14 assigned to device MTL0, is_swa = 0\nload_tensors: layer  15 assigned to device MTL0, is_swa = 0\nload_tensors: layer  16 assigned to device MTL0, is_swa = 0\nload_tensors: layer  17 assigned to device MTL0, is_swa = 0\nload_tensors: layer  18 assigned to device MTL0, is_swa = 0\nload_tensors: layer  19 assigned to device MTL0, is_swa = 0\nload_tensors: layer  20 assigned to device MTL0, is_swa = 0\nload_tensors: layer  21 assigned to device MTL0, is_swa = 0\nload_tensors: layer  22 assigned to device MTL0, is_swa = 0\nload_tensors: layer  23 assigned to device MTL0, is_swa = 0\nload_tensors: layer  24 assigned to device MTL0, is_swa = 0\nload_tensors: layer  25 assigned to device MTL0, is_swa = 0\nload_tensors: layer  26 assigned to device MTL0, is_swa = 0\nload_tensors: layer  27 assigned to device MTL0, is_swa = 0\nload_tensors: layer  28 assigned to device MTL0, is_swa = 0\nload_tensors: layer  29 assigned to device MTL0, is_swa = 0\nload_tensors: layer  30 assigned to device MTL0, is_swa = 0\nload_tensors: layer  31 assigned to device MTL0, is_swa = 0\nload_tensors: layer  32 assigned to device MTL0, is_swa = 0\nload_tensors: layer  33 assigned to device MTL0, is_swa = 0\nload_tensors: layer  34 assigned to device MTL0, is_swa = 0\nload_tensors: layer  35 assigned to device MTL0, is_swa = 0\nload_tensors: layer  36 assigned to device MTL0, is_swa = 0\nload_tensors: layer  37 assigned to device MTL0, is_swa = 0\nload_tensors: layer  38 assigned to device MTL0, is_swa = 0\nload_tensors: layer  39 assigned to device MTL0, is_swa = 0\nload_tensors: layer  40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =   321.56 MiB\nload_tensors:  MTL0_Mapped model buffer size = 18570.93 MiB\nllama_kv_cache: layer   5: dev = MTL0\nllama_kv_cache: layer  15: dev = MTL0\nllama_kv_cache: layer  25: dev = MTL0\nllama_kv_cache: layer  35: dev = MTL0\nllama_kv_cache:       MTL0 KV buffer size =   128.00 MiB\nllama_memory_recurrent, layer   0: dev = MTL0\nllama_memory_recurrent, layer   1: dev = MTL0\nllama_memory_recurrent, layer   2: dev = MTL0\nllama_memory_recurrent, layer   3: dev = MTL0\nllama_memory_recurrent, layer   4: dev = MTL0\nllama_memory_recurrent, layer   6: dev = MTL0\nllama_memory_recurrent, layer   7: dev = MTL0\nllama_memory_recurrent, layer   8: dev = MTL0\nllama_memory_recurrent, layer   9: dev = MTL0\nllama_memory_recurrent, layer  10: dev = MTL0\nllama_memory_recurrent, layer  11: dev = MTL0\nllama_memory_recurrent, layer  12: dev = MTL0\nllama_memory_recurrent, layer  13: dev = MTL0\nllama_memory_recurrent, layer  14: dev = MTL0\nllama_memory_recurrent, layer  16: dev = MTL0\nllama_memory_recurrent, layer  17: dev = MTL0\nllama_memory_recurrent, layer  18: dev = MTL0\nllama_memory_recurrent, layer  19: dev = MTL0\nllama_memory_recurrent, layer  20: dev = MTL0\nllama_memory_recurrent, layer  21: dev = MTL0\nllama_memory_recurrent, layer  22: dev = MTL0\nllama_memory_recurrent, layer  23: dev = MTL0\nllama_memory_recurrent, layer  24: dev = MTL0\nllama_memory_recurrent, layer  26: dev = MTL0\nllama_memory_recurrent, layer  27: dev = MTL0\nllama_memory_recurrent, layer  28: dev = MTL0\nllama_memory_recurrent, layer  29: dev = MTL0\nllama_memory_recurrent, layer  30: dev = MTL0\nllama_memory_recurrent, layer  31: dev = MTL0\nllama_memory_recurrent, layer  32: dev = MTL0\nllama_memory_recurrent, layer  33: dev = MTL0\nllama_memory_recurrent, layer  34: dev = MTL0\nllama_memory_recurrent, layer  36: dev = MTL0\nllama_memory_recurrent, layer  37: dev = MTL0\nllama_memory_recurrent, layer  38: dev = MTL0\nllama_memory_recurrent, layer  39: dev = MTL0\nllama_memory_recurrent:       MTL0 RS buffer size =   147.48 MiB\nsched_reserve:       MTL0 compute buffer size =   282.28 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 41,
      "totalLayers": 41,
      "inferenceMs": 366.17212500277674,
      "durationMs": 375.0527919983142,
      "tokensPerSecond": 5.461912208595435,
      "inputTokens": 68,
      "outputTokens": 2,
      "outputLimitReached": false,
      "rssBytes": 19948191744,
      "cpuThreads": 4,
      "totalMs": 1933.8207079927088,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "hola"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "loadMs": 1558.7679159943946,
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer   0 assigned to device MTL0, is_swa = 0\nload_tensors: layer   1 assigned to device MTL0, is_swa = 0\nload_tensors: layer   2 assigned to device MTL0, is_swa = 0\nload_tensors: layer   3 assigned to device MTL0, is_swa = 0\nload_tensors: layer   4 assigned to device MTL0, is_swa = 0\nload_tensors: layer   5 assigned to device MTL0, is_swa = 0\nload_tensors: layer   6 assigned to device MTL0, is_swa = 0\nload_tensors: layer   7 assigned to device MTL0, is_swa = 0\nload_tensors: layer   8 assigned to device MTL0, is_swa = 0\nload_tensors: layer   9 assigned to device MTL0, is_swa = 0\nload_tensors: layer  10 assigned to device MTL0, is_swa = 0\nload_tensors: layer  11 assigned to device MTL0, is_swa = 0\nload_tensors: layer  12 assigned to device MTL0, is_swa = 0\nload_tensors: layer  13 assigned to device MTL0, is_swa = 0\nload_tensors: layer  14 assigned to device MTL0, is_swa = 0\nload_tensors: layer  15 assigned to device MTL0, is_swa = 0\nload_tensors: layer  16 assigned to device MTL0, is_swa = 0\nload_tensors: layer  17 assigned to device MTL0, is_swa = 0\nload_tensors: layer  18 assigned to device MTL0, is_swa = 0\nload_tensors: layer  19 assigned to device MTL0, is_swa = 0\nload_tensors: layer  20 assigned to device MTL0, is_swa = 0\nload_tensors: layer  21 assigned to device MTL0, is_swa = 0\nload_tensors: layer  22 assigned to device MTL0, is_swa = 0\nload_tensors: layer  23 assigned to device MTL0, is_swa = 0\nload_tensors: layer  24 assigned to device MTL0, is_swa = 0\nload_tensors: layer  25 assigned to device MTL0, is_swa = 0\nload_tensors: layer  26 assigned to device MTL0, is_swa = 0\nload_tensors: layer  27 assigned to device MTL0, is_swa = 0\nload_tensors: layer  28 assigned to device MTL0, is_swa = 0\nload_tensors: layer  29 assigned to device MTL0, is_swa = 0\nload_tensors: layer  30 assigned to device MTL0, is_swa = 0\nload_tensors: layer  31 assigned to device MTL0, is_swa = 0\nload_tensors: layer  32 assigned to device MTL0, is_swa = 0\nload_tensors: layer  33 assigned to device MTL0, is_swa = 0\nload_tensors: layer  34 assigned to device MTL0, is_swa = 0\nload_tensors: layer  35 assigned to device MTL0, is_swa = 0\nload_tensors: layer  36 assigned to device MTL0, is_swa = 0\nload_tensors: layer  37 assigned to device MTL0, is_swa = 0\nload_tensors: layer  38 assigned to device MTL0, is_swa = 0\nload_tensors: layer  39 assigned to device MTL0, is_swa = 0\nload_tensors: layer  40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =   321.56 MiB\nload_tensors:  MTL0_Mapped model buffer size = 18570.93 MiB\nllama_kv_cache: layer   5: dev = MTL0\nllama_kv_cache: layer  15: dev = MTL0\nllama_kv_cache: layer  25: dev = MTL0\nllama_kv_cache: layer  35: dev = MTL0\nllama_kv_cache:       MTL0 KV buffer size =   128.00 MiB\nllama_memory_recurrent, layer   0: dev = MTL0\nllama_memory_recurrent, layer   1: dev = MTL0\nllama_memory_recurrent, layer   2: dev = MTL0\nllama_memory_recurrent, layer   3: dev = MTL0\nllama_memory_recurrent, layer   4: dev = MTL0\nllama_memory_recurrent, layer   6: dev = MTL0\nllama_memory_recurrent, layer   7: dev = MTL0\nllama_memory_recurrent, layer   8: dev = MTL0\nllama_memory_recurrent, layer   9: dev = MTL0\nllama_memory_recurrent, layer  10: dev = MTL0\nllama_memory_recurrent, layer  11: dev = MTL0\nllama_memory_recurrent, layer  12: dev = MTL0\nllama_memory_recurrent, layer  13: dev = MTL0\nllama_memory_recurrent, layer  14: dev = MTL0\nllama_memory_recurrent, layer  16: dev = MTL0\nllama_memory_recurrent, layer  17: dev = MTL0\nllama_memory_recurrent, layer  18: dev = MTL0\nllama_memory_recurrent, layer  19: dev = MTL0\nllama_memory_recurrent, layer  20: dev = MTL0\nllama_memory_recurrent, layer  21: dev = MTL0\nllama_memory_recurrent, layer  22: dev = MTL0\nllama_memory_recurrent, layer  23: dev = MTL0\nllama_memory_recurrent, layer  24: dev = MTL0\nllama_memory_recurrent, layer  26: dev = MTL0\nllama_memory_recurrent, layer  27: dev = MTL0\nllama_memory_recurrent, layer  28: dev = MTL0\nllama_memory_recurrent, layer  29: dev = MTL0\nllama_memory_recurrent, layer  30: dev = MTL0\nllama_memory_recurrent, layer  31: dev = MTL0\nllama_memory_recurrent, layer  32: dev = MTL0\nllama_memory_recurrent, layer  33: dev = MTL0\nllama_memory_recurrent, layer  34: dev = MTL0\nllama_memory_recurrent, layer  36: dev = MTL0\nllama_memory_recurrent, layer  37: dev = MTL0\nllama_memory_recurrent, layer  38: dev = MTL0\nllama_memory_recurrent, layer  39: dev = MTL0\nllama_memory_recurrent:       MTL0 RS buffer size =   147.48 MiB\nsched_reserve:       MTL0 compute buffer size =   282.28 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 41,
      "totalLayers": 41,
      "inferenceMs": 401.8288749939529,
      "durationMs": 410.65070900367573,
      "tokensPerSecond": 7.465864667005691,
      "inputTokens": 69,
      "outputTokens": 3,
      "outputLimitReached": false,
      "rssBytes": 19948191744,
      "cpuThreads": 4,
      "totalMs": 1969.4186249980703,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "2026-11-04"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "loadMs": 1558.7679159943946,
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer   0 assigned to device MTL0, is_swa = 0\nload_tensors: layer   1 assigned to device MTL0, is_swa = 0\nload_tensors: layer   2 assigned to device MTL0, is_swa = 0\nload_tensors: layer   3 assigned to device MTL0, is_swa = 0\nload_tensors: layer   4 assigned to device MTL0, is_swa = 0\nload_tensors: layer   5 assigned to device MTL0, is_swa = 0\nload_tensors: layer   6 assigned to device MTL0, is_swa = 0\nload_tensors: layer   7 assigned to device MTL0, is_swa = 0\nload_tensors: layer   8 assigned to device MTL0, is_swa = 0\nload_tensors: layer   9 assigned to device MTL0, is_swa = 0\nload_tensors: layer  10 assigned to device MTL0, is_swa = 0\nload_tensors: layer  11 assigned to device MTL0, is_swa = 0\nload_tensors: layer  12 assigned to device MTL0, is_swa = 0\nload_tensors: layer  13 assigned to device MTL0, is_swa = 0\nload_tensors: layer  14 assigned to device MTL0, is_swa = 0\nload_tensors: layer  15 assigned to device MTL0, is_swa = 0\nload_tensors: layer  16 assigned to device MTL0, is_swa = 0\nload_tensors: layer  17 assigned to device MTL0, is_swa = 0\nload_tensors: layer  18 assigned to device MTL0, is_swa = 0\nload_tensors: layer  19 assigned to device MTL0, is_swa = 0\nload_tensors: layer  20 assigned to device MTL0, is_swa = 0\nload_tensors: layer  21 assigned to device MTL0, is_swa = 0\nload_tensors: layer  22 assigned to device MTL0, is_swa = 0\nload_tensors: layer  23 assigned to device MTL0, is_swa = 0\nload_tensors: layer  24 assigned to device MTL0, is_swa = 0\nload_tensors: layer  25 assigned to device MTL0, is_swa = 0\nload_tensors: layer  26 assigned to device MTL0, is_swa = 0\nload_tensors: layer  27 assigned to device MTL0, is_swa = 0\nload_tensors: layer  28 assigned to device MTL0, is_swa = 0\nload_tensors: layer  29 assigned to device MTL0, is_swa = 0\nload_tensors: layer  30 assigned to device MTL0, is_swa = 0\nload_tensors: layer  31 assigned to device MTL0, is_swa = 0\nload_tensors: layer  32 assigned to device MTL0, is_swa = 0\nload_tensors: layer  33 assigned to device MTL0, is_swa = 0\nload_tensors: layer  34 assigned to device MTL0, is_swa = 0\nload_tensors: layer  35 assigned to device MTL0, is_swa = 0\nload_tensors: layer  36 assigned to device MTL0, is_swa = 0\nload_tensors: layer  37 assigned to device MTL0, is_swa = 0\nload_tensors: layer  38 assigned to device MTL0, is_swa = 0\nload_tensors: layer  39 assigned to device MTL0, is_swa = 0\nload_tensors: layer  40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =   321.56 MiB\nload_tensors:  MTL0_Mapped model buffer size = 18570.93 MiB\nllama_kv_cache: layer   5: dev = MTL0\nllama_kv_cache: layer  15: dev = MTL0\nllama_kv_cache: layer  25: dev = MTL0\nllama_kv_cache: layer  35: dev = MTL0\nllama_kv_cache:       MTL0 KV buffer size =   128.00 MiB\nllama_memory_recurrent, layer   0: dev = MTL0\nllama_memory_recurrent, layer   1: dev = MTL0\nllama_memory_recurrent, layer   2: dev = MTL0\nllama_memory_recurrent, layer   3: dev = MTL0\nllama_memory_recurrent, layer   4: dev = MTL0\nllama_memory_recurrent, layer   6: dev = MTL0\nllama_memory_recurrent, layer   7: dev = MTL0\nllama_memory_recurrent, layer   8: dev = MTL0\nllama_memory_recurrent, layer   9: dev = MTL0\nllama_memory_recurrent, layer  10: dev = MTL0\nllama_memory_recurrent, layer  11: dev = MTL0\nllama_memory_recurrent, layer  12: dev = MTL0\nllama_memory_recurrent, layer  13: dev = MTL0\nllama_memory_recurrent, layer  14: dev = MTL0\nllama_memory_recurrent, layer  16: dev = MTL0\nllama_memory_recurrent, layer  17: dev = MTL0\nllama_memory_recurrent, layer  18: dev = MTL0\nllama_memory_recurrent, layer  19: dev = MTL0\nllama_memory_recurrent, layer  20: dev = MTL0\nllama_memory_recurrent, layer  21: dev = MTL0\nllama_memory_recurrent, layer  22: dev = MTL0\nllama_memory_recurrent, layer  23: dev = MTL0\nllama_memory_recurrent, layer  24: dev = MTL0\nllama_memory_recurrent, layer  26: dev = MTL0\nllama_memory_recurrent, layer  27: dev = MTL0\nllama_memory_recurrent, layer  28: dev = MTL0\nllama_memory_recurrent, layer  29: dev = MTL0\nllama_memory_recurrent, layer  30: dev = MTL0\nllama_memory_recurrent, layer  31: dev = MTL0\nllama_memory_recurrent, layer  32: dev = MTL0\nllama_memory_recurrent, layer  33: dev = MTL0\nllama_memory_recurrent, layer  34: dev = MTL0\nllama_memory_recurrent, layer  36: dev = MTL0\nllama_memory_recurrent, layer  37: dev = MTL0\nllama_memory_recurrent, layer  38: dev = MTL0\nllama_memory_recurrent, layer  39: dev = MTL0\nllama_memory_recurrent:       MTL0 RS buffer size =   147.48 MiB\nsched_reserve:       MTL0 compute buffer size =   282.28 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 41,
      "totalLayers": 41,
      "inferenceMs": 563.7482079982874,
      "durationMs": 571.8166250007926,
      "tokensPerSecond": 12.416890911023996,
      "inputTokens": 72,
      "outputTokens": 7,
      "outputLimitReached": false,
      "rssBytes": 19949109248,
      "cpuThreads": 4,
      "totalMs": 2130.584540995187,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "Unknown"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "loadMs": 1558.7679159943946,
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer   0 assigned to device MTL0, is_swa = 0\nload_tensors: layer   1 assigned to device MTL0, is_swa = 0\nload_tensors: layer   2 assigned to device MTL0, is_swa = 0\nload_tensors: layer   3 assigned to device MTL0, is_swa = 0\nload_tensors: layer   4 assigned to device MTL0, is_swa = 0\nload_tensors: layer   5 assigned to device MTL0, is_swa = 0\nload_tensors: layer   6 assigned to device MTL0, is_swa = 0\nload_tensors: layer   7 assigned to device MTL0, is_swa = 0\nload_tensors: layer   8 assigned to device MTL0, is_swa = 0\nload_tensors: layer   9 assigned to device MTL0, is_swa = 0\nload_tensors: layer  10 assigned to device MTL0, is_swa = 0\nload_tensors: layer  11 assigned to device MTL0, is_swa = 0\nload_tensors: layer  12 assigned to device MTL0, is_swa = 0\nload_tensors: layer  13 assigned to device MTL0, is_swa = 0\nload_tensors: layer  14 assigned to device MTL0, is_swa = 0\nload_tensors: layer  15 assigned to device MTL0, is_swa = 0\nload_tensors: layer  16 assigned to device MTL0, is_swa = 0\nload_tensors: layer  17 assigned to device MTL0, is_swa = 0\nload_tensors: layer  18 assigned to device MTL0, is_swa = 0\nload_tensors: layer  19 assigned to device MTL0, is_swa = 0\nload_tensors: layer  20 assigned to device MTL0, is_swa = 0\nload_tensors: layer  21 assigned to device MTL0, is_swa = 0\nload_tensors: layer  22 assigned to device MTL0, is_swa = 0\nload_tensors: layer  23 assigned to device MTL0, is_swa = 0\nload_tensors: layer  24 assigned to device MTL0, is_swa = 0\nload_tensors: layer  25 assigned to device MTL0, is_swa = 0\nload_tensors: layer  26 assigned to device MTL0, is_swa = 0\nload_tensors: layer  27 assigned to device MTL0, is_swa = 0\nload_tensors: layer  28 assigned to device MTL0, is_swa = 0\nload_tensors: layer  29 assigned to device MTL0, is_swa = 0\nload_tensors: layer  30 assigned to device MTL0, is_swa = 0\nload_tensors: layer  31 assigned to device MTL0, is_swa = 0\nload_tensors: layer  32 assigned to device MTL0, is_swa = 0\nload_tensors: layer  33 assigned to device MTL0, is_swa = 0\nload_tensors: layer  34 assigned to device MTL0, is_swa = 0\nload_tensors: layer  35 assigned to device MTL0, is_swa = 0\nload_tensors: layer  36 assigned to device MTL0, is_swa = 0\nload_tensors: layer  37 assigned to device MTL0, is_swa = 0\nload_tensors: layer  38 assigned to device MTL0, is_swa = 0\nload_tensors: layer  39 assigned to device MTL0, is_swa = 0\nload_tensors: layer  40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =   321.56 MiB\nload_tensors:  MTL0_Mapped model buffer size = 18570.93 MiB\nllama_kv_cache: layer   5: dev = MTL0\nllama_kv_cache: layer  15: dev = MTL0\nllama_kv_cache: layer  25: dev = MTL0\nllama_kv_cache: layer  35: dev = MTL0\nllama_kv_cache:       MTL0 KV buffer size =   128.00 MiB\nllama_memory_recurrent, layer   0: dev = MTL0\nllama_memory_recurrent, layer   1: dev = MTL0\nllama_memory_recurrent, layer   2: dev = MTL0\nllama_memory_recurrent, layer   3: dev = MTL0\nllama_memory_recurrent, layer   4: dev = MTL0\nllama_memory_recurrent, layer   6: dev = MTL0\nllama_memory_recurrent, layer   7: dev = MTL0\nllama_memory_recurrent, layer   8: dev = MTL0\nllama_memory_recurrent, layer   9: dev = MTL0\nllama_memory_recurrent, layer  10: dev = MTL0\nllama_memory_recurrent, layer  11: dev = MTL0\nllama_memory_recurrent, layer  12: dev = MTL0\nllama_memory_recurrent, layer  13: dev = MTL0\nllama_memory_recurrent, layer  14: dev = MTL0\nllama_memory_recurrent, layer  16: dev = MTL0\nllama_memory_recurrent, layer  17: dev = MTL0\nllama_memory_recurrent, layer  18: dev = MTL0\nllama_memory_recurrent, layer  19: dev = MTL0\nllama_memory_recurrent, layer  20: dev = MTL0\nllama_memory_recurrent, layer  21: dev = MTL0\nllama_memory_recurrent, layer  22: dev = MTL0\nllama_memory_recurrent, layer  23: dev = MTL0\nllama_memory_recurrent, layer  24: dev = MTL0\nllama_memory_recurrent, layer  26: dev = MTL0\nllama_memory_recurrent, layer  27: dev = MTL0\nllama_memory_recurrent, layer  28: dev = MTL0\nllama_memory_recurrent, layer  29: dev = MTL0\nllama_memory_recurrent, layer  30: dev = MTL0\nllama_memory_recurrent, layer  31: dev = MTL0\nllama_memory_recurrent, layer  32: dev = MTL0\nllama_memory_recurrent, layer  33: dev = MTL0\nllama_memory_recurrent, layer  34: dev = MTL0\nllama_memory_recurrent, layer  36: dev = MTL0\nllama_memory_recurrent, layer  37: dev = MTL0\nllama_memory_recurrent, layer  38: dev = MTL0\nllama_memory_recurrent, layer  39: dev = MTL0\nllama_memory_recurrent:       MTL0 RS buffer size =   147.48 MiB\nsched_reserve:       MTL0 compute buffer size =   282.28 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 41,
      "totalLayers": 41,
      "inferenceMs": 379.13874999503605,
      "durationMs": 396.48783300071955,
      "tokensPerSecond": 5.275113662283756,
      "inputTokens": 74,
      "outputTokens": 2,
      "outputLimitReached": false,
      "rssBytes": 19949142016,
      "cpuThreads": 4,
      "totalMs": 1955.2557489951141,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "Paris"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "loadMs": 1277.7895419931156,
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer   0 assigned to device MTL0, is_swa = 0\nload_tensors: layer   1 assigned to device MTL0, is_swa = 0\nload_tensors: layer   2 assigned to device MTL0, is_swa = 0\nload_tensors: layer   3 assigned to device MTL0, is_swa = 0\nload_tensors: layer   4 assigned to device MTL0, is_swa = 0\nload_tensors: layer   5 assigned to device MTL0, is_swa = 0\nload_tensors: layer   6 assigned to device MTL0, is_swa = 0\nload_tensors: layer   7 assigned to device MTL0, is_swa = 0\nload_tensors: layer   8 assigned to device MTL0, is_swa = 0\nload_tensors: layer   9 assigned to device MTL0, is_swa = 0\nload_tensors: layer  10 assigned to device MTL0, is_swa = 0\nload_tensors: layer  11 assigned to device MTL0, is_swa = 0\nload_tensors: layer  12 assigned to device MTL0, is_swa = 0\nload_tensors: layer  13 assigned to device MTL0, is_swa = 0\nload_tensors: layer  14 assigned to device MTL0, is_swa = 0\nload_tensors: layer  15 assigned to device MTL0, is_swa = 0\nload_tensors: layer  16 assigned to device MTL0, is_swa = 0\nload_tensors: layer  17 assigned to device MTL0, is_swa = 0\nload_tensors: layer  18 assigned to device MTL0, is_swa = 0\nload_tensors: layer  19 assigned to device MTL0, is_swa = 0\nload_tensors: layer  20 assigned to device MTL0, is_swa = 0\nload_tensors: layer  21 assigned to device MTL0, is_swa = 0\nload_tensors: layer  22 assigned to device MTL0, is_swa = 0\nload_tensors: layer  23 assigned to device MTL0, is_swa = 0\nload_tensors: layer  24 assigned to device MTL0, is_swa = 0\nload_tensors: layer  25 assigned to device MTL0, is_swa = 0\nload_tensors: layer  26 assigned to device MTL0, is_swa = 0\nload_tensors: layer  27 assigned to device MTL0, is_swa = 0\nload_tensors: layer  28 assigned to device MTL0, is_swa = 0\nload_tensors: layer  29 assigned to device MTL0, is_swa = 0\nload_tensors: layer  30 assigned to device MTL0, is_swa = 0\nload_tensors: layer  31 assigned to device MTL0, is_swa = 0\nload_tensors: layer  32 assigned to device MTL0, is_swa = 0\nload_tensors: layer  33 assigned to device MTL0, is_swa = 0\nload_tensors: layer  34 assigned to device MTL0, is_swa = 0\nload_tensors: layer  35 assigned to device MTL0, is_swa = 0\nload_tensors: layer  36 assigned to device MTL0, is_swa = 0\nload_tensors: layer  37 assigned to device MTL0, is_swa = 0\nload_tensors: layer  38 assigned to device MTL0, is_swa = 0\nload_tensors: layer  39 assigned to device MTL0, is_swa = 0\nload_tensors: layer  40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =   321.56 MiB\nload_tensors:  MTL0_Mapped model buffer size = 18570.93 MiB\nllama_kv_cache: layer   5: dev = MTL0\nllama_kv_cache: layer  15: dev = MTL0\nllama_kv_cache: layer  25: dev = MTL0\nllama_kv_cache: layer  35: dev = MTL0\nllama_kv_cache:       MTL0 KV buffer size =   128.00 MiB\nllama_memory_recurrent, layer   0: dev = MTL0\nllama_memory_recurrent, layer   1: dev = MTL0\nllama_memory_recurrent, layer   2: dev = MTL0\nllama_memory_recurrent, layer   3: dev = MTL0\nllama_memory_recurrent, layer   4: dev = MTL0\nllama_memory_recurrent, layer   6: dev = MTL0\nllama_memory_recurrent, layer   7: dev = MTL0\nllama_memory_recurrent, layer   8: dev = MTL0\nllama_memory_recurrent, layer   9: dev = MTL0\nllama_memory_recurrent, layer  10: dev = MTL0\nllama_memory_recurrent, layer  11: dev = MTL0\nllama_memory_recurrent, layer  12: dev = MTL0\nllama_memory_recurrent, layer  13: dev = MTL0\nllama_memory_recurrent, layer  14: dev = MTL0\nllama_memory_recurrent, layer  16: dev = MTL0\nllama_memory_recurrent, layer  17: dev = MTL0\nllama_memory_recurrent, layer  18: dev = MTL0\nllama_memory_recurrent, layer  19: dev = MTL0\nllama_memory_recurrent, layer  20: dev = MTL0\nllama_memory_recurrent, layer  21: dev = MTL0\nllama_memory_recurrent, layer  22: dev = MTL0\nllama_memory_recurrent, layer  23: dev = MTL0\nllama_memory_recurrent, layer  24: dev = MTL0\nllama_memory_recurrent, layer  26: dev = MTL0\nllama_memory_recurrent, layer  27: dev = MTL0\nllama_memory_recurrent, layer  28: dev = MTL0\nllama_memory_recurrent, layer  29: dev = MTL0\nllama_memory_recurrent, layer  30: dev = MTL0\nllama_memory_recurrent, layer  31: dev = MTL0\nllama_memory_recurrent, layer  32: dev = MTL0\nllama_memory_recurrent, layer  33: dev = MTL0\nllama_memory_recurrent, layer  34: dev = MTL0\nllama_memory_recurrent, layer  36: dev = MTL0\nllama_memory_recurrent, layer  37: dev = MTL0\nllama_memory_recurrent, layer  38: dev = MTL0\nllama_memory_recurrent, layer  39: dev = MTL0\nllama_memory_recurrent:       MTL0 RS buffer size =   147.48 MiB\nsched_reserve:       MTL0 compute buffer size =   282.28 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 41,
      "totalLayers": 41,
      "inferenceMs": 357.1684999988065,
      "durationMs": 365.3875829986646,
      "tokensPerSecond": 5.599597948885982,
      "inputTokens": 63,
      "outputTokens": 2,
      "outputLimitReached": false,
      "rssBytes": 19940950016,
      "cpuThreads": 4,
      "totalMs": 1643.1771249917801,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "56"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "loadMs": 1277.7895419931156,
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer   0 assigned to device MTL0, is_swa = 0\nload_tensors: layer   1 assigned to device MTL0, is_swa = 0\nload_tensors: layer   2 assigned to device MTL0, is_swa = 0\nload_tensors: layer   3 assigned to device MTL0, is_swa = 0\nload_tensors: layer   4 assigned to device MTL0, is_swa = 0\nload_tensors: layer   5 assigned to device MTL0, is_swa = 0\nload_tensors: layer   6 assigned to device MTL0, is_swa = 0\nload_tensors: layer   7 assigned to device MTL0, is_swa = 0\nload_tensors: layer   8 assigned to device MTL0, is_swa = 0\nload_tensors: layer   9 assigned to device MTL0, is_swa = 0\nload_tensors: layer  10 assigned to device MTL0, is_swa = 0\nload_tensors: layer  11 assigned to device MTL0, is_swa = 0\nload_tensors: layer  12 assigned to device MTL0, is_swa = 0\nload_tensors: layer  13 assigned to device MTL0, is_swa = 0\nload_tensors: layer  14 assigned to device MTL0, is_swa = 0\nload_tensors: layer  15 assigned to device MTL0, is_swa = 0\nload_tensors: layer  16 assigned to device MTL0, is_swa = 0\nload_tensors: layer  17 assigned to device MTL0, is_swa = 0\nload_tensors: layer  18 assigned to device MTL0, is_swa = 0\nload_tensors: layer  19 assigned to device MTL0, is_swa = 0\nload_tensors: layer  20 assigned to device MTL0, is_swa = 0\nload_tensors: layer  21 assigned to device MTL0, is_swa = 0\nload_tensors: layer  22 assigned to device MTL0, is_swa = 0\nload_tensors: layer  23 assigned to device MTL0, is_swa = 0\nload_tensors: layer  24 assigned to device MTL0, is_swa = 0\nload_tensors: layer  25 assigned to device MTL0, is_swa = 0\nload_tensors: layer  26 assigned to device MTL0, is_swa = 0\nload_tensors: layer  27 assigned to device MTL0, is_swa = 0\nload_tensors: layer  28 assigned to device MTL0, is_swa = 0\nload_tensors: layer  29 assigned to device MTL0, is_swa = 0\nload_tensors: layer  30 assigned to device MTL0, is_swa = 0\nload_tensors: layer  31 assigned to device MTL0, is_swa = 0\nload_tensors: layer  32 assigned to device MTL0, is_swa = 0\nload_tensors: layer  33 assigned to device MTL0, is_swa = 0\nload_tensors: layer  34 assigned to device MTL0, is_swa = 0\nload_tensors: layer  35 assigned to device MTL0, is_swa = 0\nload_tensors: layer  36 assigned to device MTL0, is_swa = 0\nload_tensors: layer  37 assigned to device MTL0, is_swa = 0\nload_tensors: layer  38 assigned to device MTL0, is_swa = 0\nload_tensors: layer  39 assigned to device MTL0, is_swa = 0\nload_tensors: layer  40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =   321.56 MiB\nload_tensors:  MTL0_Mapped model buffer size = 18570.93 MiB\nllama_kv_cache: layer   5: dev = MTL0\nllama_kv_cache: layer  15: dev = MTL0\nllama_kv_cache: layer  25: dev = MTL0\nllama_kv_cache: layer  35: dev = MTL0\nllama_kv_cache:       MTL0 KV buffer size =   128.00 MiB\nllama_memory_recurrent, layer   0: dev = MTL0\nllama_memory_recurrent, layer   1: dev = MTL0\nllama_memory_recurrent, layer   2: dev = MTL0\nllama_memory_recurrent, layer   3: dev = MTL0\nllama_memory_recurrent, layer   4: dev = MTL0\nllama_memory_recurrent, layer   6: dev = MTL0\nllama_memory_recurrent, layer   7: dev = MTL0\nllama_memory_recurrent, layer   8: dev = MTL0\nllama_memory_recurrent, layer   9: dev = MTL0\nllama_memory_recurrent, layer  10: dev = MTL0\nllama_memory_recurrent, layer  11: dev = MTL0\nllama_memory_recurrent, layer  12: dev = MTL0\nllama_memory_recurrent, layer  13: dev = MTL0\nllama_memory_recurrent, layer  14: dev = MTL0\nllama_memory_recurrent, layer  16: dev = MTL0\nllama_memory_recurrent, layer  17: dev = MTL0\nllama_memory_recurrent, layer  18: dev = MTL0\nllama_memory_recurrent, layer  19: dev = MTL0\nllama_memory_recurrent, layer  20: dev = MTL0\nllama_memory_recurrent, layer  21: dev = MTL0\nllama_memory_recurrent, layer  22: dev = MTL0\nllama_memory_recurrent, layer  23: dev = MTL0\nllama_memory_recurrent, layer  24: dev = MTL0\nllama_memory_recurrent, layer  26: dev = MTL0\nllama_memory_recurrent, layer  27: dev = MTL0\nllama_memory_recurrent, layer  28: dev = MTL0\nllama_memory_recurrent, layer  29: dev = MTL0\nllama_memory_recurrent, layer  30: dev = MTL0\nllama_memory_recurrent, layer  31: dev = MTL0\nllama_memory_recurrent, layer  32: dev = MTL0\nllama_memory_recurrent, layer  33: dev = MTL0\nllama_memory_recurrent, layer  34: dev = MTL0\nllama_memory_recurrent, layer  36: dev = MTL0\nllama_memory_recurrent, layer  37: dev = MTL0\nllama_memory_recurrent, layer  38: dev = MTL0\nllama_memory_recurrent, layer  39: dev = MTL0\nllama_memory_recurrent:       MTL0 RS buffer size =   147.48 MiB\nsched_reserve:       MTL0 compute buffer size =   282.28 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 41,
      "totalLayers": 41,
      "inferenceMs": 329.7040420002304,
      "durationMs": 337.9441669967491,
      "tokensPerSecond": 6.066046348314415,
      "inputTokens": 63,
      "outputTokens": 2,
      "outputLimitReached": false,
      "rssBytes": 19942162432,
      "cpuThreads": 4,
      "totalMs": 1615.7337089898647,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "chat"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "loadMs": 1277.7895419931156,
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer   0 assigned to device MTL0, is_swa = 0\nload_tensors: layer   1 assigned to device MTL0, is_swa = 0\nload_tensors: layer   2 assigned to device MTL0, is_swa = 0\nload_tensors: layer   3 assigned to device MTL0, is_swa = 0\nload_tensors: layer   4 assigned to device MTL0, is_swa = 0\nload_tensors: layer   5 assigned to device MTL0, is_swa = 0\nload_tensors: layer   6 assigned to device MTL0, is_swa = 0\nload_tensors: layer   7 assigned to device MTL0, is_swa = 0\nload_tensors: layer   8 assigned to device MTL0, is_swa = 0\nload_tensors: layer   9 assigned to device MTL0, is_swa = 0\nload_tensors: layer  10 assigned to device MTL0, is_swa = 0\nload_tensors: layer  11 assigned to device MTL0, is_swa = 0\nload_tensors: layer  12 assigned to device MTL0, is_swa = 0\nload_tensors: layer  13 assigned to device MTL0, is_swa = 0\nload_tensors: layer  14 assigned to device MTL0, is_swa = 0\nload_tensors: layer  15 assigned to device MTL0, is_swa = 0\nload_tensors: layer  16 assigned to device MTL0, is_swa = 0\nload_tensors: layer  17 assigned to device MTL0, is_swa = 0\nload_tensors: layer  18 assigned to device MTL0, is_swa = 0\nload_tensors: layer  19 assigned to device MTL0, is_swa = 0\nload_tensors: layer  20 assigned to device MTL0, is_swa = 0\nload_tensors: layer  21 assigned to device MTL0, is_swa = 0\nload_tensors: layer  22 assigned to device MTL0, is_swa = 0\nload_tensors: layer  23 assigned to device MTL0, is_swa = 0\nload_tensors: layer  24 assigned to device MTL0, is_swa = 0\nload_tensors: layer  25 assigned to device MTL0, is_swa = 0\nload_tensors: layer  26 assigned to device MTL0, is_swa = 0\nload_tensors: layer  27 assigned to device MTL0, is_swa = 0\nload_tensors: layer  28 assigned to device MTL0, is_swa = 0\nload_tensors: layer  29 assigned to device MTL0, is_swa = 0\nload_tensors: layer  30 assigned to device MTL0, is_swa = 0\nload_tensors: layer  31 assigned to device MTL0, is_swa = 0\nload_tensors: layer  32 assigned to device MTL0, is_swa = 0\nload_tensors: layer  33 assigned to device MTL0, is_swa = 0\nload_tensors: layer  34 assigned to device MTL0, is_swa = 0\nload_tensors: layer  35 assigned to device MTL0, is_swa = 0\nload_tensors: layer  36 assigned to device MTL0, is_swa = 0\nload_tensors: layer  37 assigned to device MTL0, is_swa = 0\nload_tensors: layer  38 assigned to device MTL0, is_swa = 0\nload_tensors: layer  39 assigned to device MTL0, is_swa = 0\nload_tensors: layer  40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =   321.56 MiB\nload_tensors:  MTL0_Mapped model buffer size = 18570.93 MiB\nllama_kv_cache: layer   5: dev = MTL0\nllama_kv_cache: layer  15: dev = MTL0\nllama_kv_cache: layer  25: dev = MTL0\nllama_kv_cache: layer  35: dev = MTL0\nllama_kv_cache:       MTL0 KV buffer size =   128.00 MiB\nllama_memory_recurrent, layer   0: dev = MTL0\nllama_memory_recurrent, layer   1: dev = MTL0\nllama_memory_recurrent, layer   2: dev = MTL0\nllama_memory_recurrent, layer   3: dev = MTL0\nllama_memory_recurrent, layer   4: dev = MTL0\nllama_memory_recurrent, layer   6: dev = MTL0\nllama_memory_recurrent, layer   7: dev = MTL0\nllama_memory_recurrent, layer   8: dev = MTL0\nllama_memory_recurrent, layer   9: dev = MTL0\nllama_memory_recurrent, layer  10: dev = MTL0\nllama_memory_recurrent, layer  11: dev = MTL0\nllama_memory_recurrent, layer  12: dev = MTL0\nllama_memory_recurrent, layer  13: dev = MTL0\nllama_memory_recurrent, layer  14: dev = MTL0\nllama_memory_recurrent, layer  16: dev = MTL0\nllama_memory_recurrent, layer  17: dev = MTL0\nllama_memory_recurrent, layer  18: dev = MTL0\nllama_memory_recurrent, layer  19: dev = MTL0\nllama_memory_recurrent, layer  20: dev = MTL0\nllama_memory_recurrent, layer  21: dev = MTL0\nllama_memory_recurrent, layer  22: dev = MTL0\nllama_memory_recurrent, layer  23: dev = MTL0\nllama_memory_recurrent, layer  24: dev = MTL0\nllama_memory_recurrent, layer  26: dev = MTL0\nllama_memory_recurrent, layer  27: dev = MTL0\nllama_memory_recurrent, layer  28: dev = MTL0\nllama_memory_recurrent, layer  29: dev = MTL0\nllama_memory_recurrent, layer  30: dev = MTL0\nllama_memory_recurrent, layer  31: dev = MTL0\nllama_memory_recurrent, layer  32: dev = MTL0\nllama_memory_recurrent, layer  33: dev = MTL0\nllama_memory_recurrent, layer  34: dev = MTL0\nllama_memory_recurrent, layer  36: dev = MTL0\nllama_memory_recurrent, layer  37: dev = MTL0\nllama_memory_recurrent, layer  38: dev = MTL0\nllama_memory_recurrent, layer  39: dev = MTL0\nllama_memory_recurrent:       MTL0 RS buffer size =   147.48 MiB\nsched_reserve:       MTL0 compute buffer size =   282.28 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 41,
      "totalLayers": 41,
      "inferenceMs": 366.49279199627927,
      "durationMs": 376.4192090020515,
      "tokensPerSecond": 5.4571332470306935,
      "inputTokens": 68,
      "outputTokens": 2,
      "outputLimitReached": false,
      "rssBytes": 19944161280,
      "cpuThreads": 4,
      "totalMs": 1654.208750995167,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "hola"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "loadMs": 1277.7895419931156,
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer   0 assigned to device MTL0, is_swa = 0\nload_tensors: layer   1 assigned to device MTL0, is_swa = 0\nload_tensors: layer   2 assigned to device MTL0, is_swa = 0\nload_tensors: layer   3 assigned to device MTL0, is_swa = 0\nload_tensors: layer   4 assigned to device MTL0, is_swa = 0\nload_tensors: layer   5 assigned to device MTL0, is_swa = 0\nload_tensors: layer   6 assigned to device MTL0, is_swa = 0\nload_tensors: layer   7 assigned to device MTL0, is_swa = 0\nload_tensors: layer   8 assigned to device MTL0, is_swa = 0\nload_tensors: layer   9 assigned to device MTL0, is_swa = 0\nload_tensors: layer  10 assigned to device MTL0, is_swa = 0\nload_tensors: layer  11 assigned to device MTL0, is_swa = 0\nload_tensors: layer  12 assigned to device MTL0, is_swa = 0\nload_tensors: layer  13 assigned to device MTL0, is_swa = 0\nload_tensors: layer  14 assigned to device MTL0, is_swa = 0\nload_tensors: layer  15 assigned to device MTL0, is_swa = 0\nload_tensors: layer  16 assigned to device MTL0, is_swa = 0\nload_tensors: layer  17 assigned to device MTL0, is_swa = 0\nload_tensors: layer  18 assigned to device MTL0, is_swa = 0\nload_tensors: layer  19 assigned to device MTL0, is_swa = 0\nload_tensors: layer  20 assigned to device MTL0, is_swa = 0\nload_tensors: layer  21 assigned to device MTL0, is_swa = 0\nload_tensors: layer  22 assigned to device MTL0, is_swa = 0\nload_tensors: layer  23 assigned to device MTL0, is_swa = 0\nload_tensors: layer  24 assigned to device MTL0, is_swa = 0\nload_tensors: layer  25 assigned to device MTL0, is_swa = 0\nload_tensors: layer  26 assigned to device MTL0, is_swa = 0\nload_tensors: layer  27 assigned to device MTL0, is_swa = 0\nload_tensors: layer  28 assigned to device MTL0, is_swa = 0\nload_tensors: layer  29 assigned to device MTL0, is_swa = 0\nload_tensors: layer  30 assigned to device MTL0, is_swa = 0\nload_tensors: layer  31 assigned to device MTL0, is_swa = 0\nload_tensors: layer  32 assigned to device MTL0, is_swa = 0\nload_tensors: layer  33 assigned to device MTL0, is_swa = 0\nload_tensors: layer  34 assigned to device MTL0, is_swa = 0\nload_tensors: layer  35 assigned to device MTL0, is_swa = 0\nload_tensors: layer  36 assigned to device MTL0, is_swa = 0\nload_tensors: layer  37 assigned to device MTL0, is_swa = 0\nload_tensors: layer  38 assigned to device MTL0, is_swa = 0\nload_tensors: layer  39 assigned to device MTL0, is_swa = 0\nload_tensors: layer  40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =   321.56 MiB\nload_tensors:  MTL0_Mapped model buffer size = 18570.93 MiB\nllama_kv_cache: layer   5: dev = MTL0\nllama_kv_cache: layer  15: dev = MTL0\nllama_kv_cache: layer  25: dev = MTL0\nllama_kv_cache: layer  35: dev = MTL0\nllama_kv_cache:       MTL0 KV buffer size =   128.00 MiB\nllama_memory_recurrent, layer   0: dev = MTL0\nllama_memory_recurrent, layer   1: dev = MTL0\nllama_memory_recurrent, layer   2: dev = MTL0\nllama_memory_recurrent, layer   3: dev = MTL0\nllama_memory_recurrent, layer   4: dev = MTL0\nllama_memory_recurrent, layer   6: dev = MTL0\nllama_memory_recurrent, layer   7: dev = MTL0\nllama_memory_recurrent, layer   8: dev = MTL0\nllama_memory_recurrent, layer   9: dev = MTL0\nllama_memory_recurrent, layer  10: dev = MTL0\nllama_memory_recurrent, layer  11: dev = MTL0\nllama_memory_recurrent, layer  12: dev = MTL0\nllama_memory_recurrent, layer  13: dev = MTL0\nllama_memory_recurrent, layer  14: dev = MTL0\nllama_memory_recurrent, layer  16: dev = MTL0\nllama_memory_recurrent, layer  17: dev = MTL0\nllama_memory_recurrent, layer  18: dev = MTL0\nllama_memory_recurrent, layer  19: dev = MTL0\nllama_memory_recurrent, layer  20: dev = MTL0\nllama_memory_recurrent, layer  21: dev = MTL0\nllama_memory_recurrent, layer  22: dev = MTL0\nllama_memory_recurrent, layer  23: dev = MTL0\nllama_memory_recurrent, layer  24: dev = MTL0\nllama_memory_recurrent, layer  26: dev = MTL0\nllama_memory_recurrent, layer  27: dev = MTL0\nllama_memory_recurrent, layer  28: dev = MTL0\nllama_memory_recurrent, layer  29: dev = MTL0\nllama_memory_recurrent, layer  30: dev = MTL0\nllama_memory_recurrent, layer  31: dev = MTL0\nllama_memory_recurrent, layer  32: dev = MTL0\nllama_memory_recurrent, layer  33: dev = MTL0\nllama_memory_recurrent, layer  34: dev = MTL0\nllama_memory_recurrent, layer  36: dev = MTL0\nllama_memory_recurrent, layer  37: dev = MTL0\nllama_memory_recurrent, layer  38: dev = MTL0\nllama_memory_recurrent, layer  39: dev = MTL0\nllama_memory_recurrent:       MTL0 RS buffer size =   147.48 MiB\nsched_reserve:       MTL0 compute buffer size =   282.28 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 41,
      "totalLayers": 41,
      "inferenceMs": 403.24220799811883,
      "durationMs": 411.4744590042392,
      "tokensPerSecond": 7.439697384094265,
      "inputTokens": 69,
      "outputTokens": 3,
      "outputLimitReached": false,
      "rssBytes": 19945586688,
      "cpuThreads": 4,
      "totalMs": 1689.2640009973547,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "2026-11-04"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "loadMs": 1277.7895419931156,
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer   0 assigned to device MTL0, is_swa = 0\nload_tensors: layer   1 assigned to device MTL0, is_swa = 0\nload_tensors: layer   2 assigned to device MTL0, is_swa = 0\nload_tensors: layer   3 assigned to device MTL0, is_swa = 0\nload_tensors: layer   4 assigned to device MTL0, is_swa = 0\nload_tensors: layer   5 assigned to device MTL0, is_swa = 0\nload_tensors: layer   6 assigned to device MTL0, is_swa = 0\nload_tensors: layer   7 assigned to device MTL0, is_swa = 0\nload_tensors: layer   8 assigned to device MTL0, is_swa = 0\nload_tensors: layer   9 assigned to device MTL0, is_swa = 0\nload_tensors: layer  10 assigned to device MTL0, is_swa = 0\nload_tensors: layer  11 assigned to device MTL0, is_swa = 0\nload_tensors: layer  12 assigned to device MTL0, is_swa = 0\nload_tensors: layer  13 assigned to device MTL0, is_swa = 0\nload_tensors: layer  14 assigned to device MTL0, is_swa = 0\nload_tensors: layer  15 assigned to device MTL0, is_swa = 0\nload_tensors: layer  16 assigned to device MTL0, is_swa = 0\nload_tensors: layer  17 assigned to device MTL0, is_swa = 0\nload_tensors: layer  18 assigned to device MTL0, is_swa = 0\nload_tensors: layer  19 assigned to device MTL0, is_swa = 0\nload_tensors: layer  20 assigned to device MTL0, is_swa = 0\nload_tensors: layer  21 assigned to device MTL0, is_swa = 0\nload_tensors: layer  22 assigned to device MTL0, is_swa = 0\nload_tensors: layer  23 assigned to device MTL0, is_swa = 0\nload_tensors: layer  24 assigned to device MTL0, is_swa = 0\nload_tensors: layer  25 assigned to device MTL0, is_swa = 0\nload_tensors: layer  26 assigned to device MTL0, is_swa = 0\nload_tensors: layer  27 assigned to device MTL0, is_swa = 0\nload_tensors: layer  28 assigned to device MTL0, is_swa = 0\nload_tensors: layer  29 assigned to device MTL0, is_swa = 0\nload_tensors: layer  30 assigned to device MTL0, is_swa = 0\nload_tensors: layer  31 assigned to device MTL0, is_swa = 0\nload_tensors: layer  32 assigned to device MTL0, is_swa = 0\nload_tensors: layer  33 assigned to device MTL0, is_swa = 0\nload_tensors: layer  34 assigned to device MTL0, is_swa = 0\nload_tensors: layer  35 assigned to device MTL0, is_swa = 0\nload_tensors: layer  36 assigned to device MTL0, is_swa = 0\nload_tensors: layer  37 assigned to device MTL0, is_swa = 0\nload_tensors: layer  38 assigned to device MTL0, is_swa = 0\nload_tensors: layer  39 assigned to device MTL0, is_swa = 0\nload_tensors: layer  40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =   321.56 MiB\nload_tensors:  MTL0_Mapped model buffer size = 18570.93 MiB\nllama_kv_cache: layer   5: dev = MTL0\nllama_kv_cache: layer  15: dev = MTL0\nllama_kv_cache: layer  25: dev = MTL0\nllama_kv_cache: layer  35: dev = MTL0\nllama_kv_cache:       MTL0 KV buffer size =   128.00 MiB\nllama_memory_recurrent, layer   0: dev = MTL0\nllama_memory_recurrent, layer   1: dev = MTL0\nllama_memory_recurrent, layer   2: dev = MTL0\nllama_memory_recurrent, layer   3: dev = MTL0\nllama_memory_recurrent, layer   4: dev = MTL0\nllama_memory_recurrent, layer   6: dev = MTL0\nllama_memory_recurrent, layer   7: dev = MTL0\nllama_memory_recurrent, layer   8: dev = MTL0\nllama_memory_recurrent, layer   9: dev = MTL0\nllama_memory_recurrent, layer  10: dev = MTL0\nllama_memory_recurrent, layer  11: dev = MTL0\nllama_memory_recurrent, layer  12: dev = MTL0\nllama_memory_recurrent, layer  13: dev = MTL0\nllama_memory_recurrent, layer  14: dev = MTL0\nllama_memory_recurrent, layer  16: dev = MTL0\nllama_memory_recurrent, layer  17: dev = MTL0\nllama_memory_recurrent, layer  18: dev = MTL0\nllama_memory_recurrent, layer  19: dev = MTL0\nllama_memory_recurrent, layer  20: dev = MTL0\nllama_memory_recurrent, layer  21: dev = MTL0\nllama_memory_recurrent, layer  22: dev = MTL0\nllama_memory_recurrent, layer  23: dev = MTL0\nllama_memory_recurrent, layer  24: dev = MTL0\nllama_memory_recurrent, layer  26: dev = MTL0\nllama_memory_recurrent, layer  27: dev = MTL0\nllama_memory_recurrent, layer  28: dev = MTL0\nllama_memory_recurrent, layer  29: dev = MTL0\nllama_memory_recurrent, layer  30: dev = MTL0\nllama_memory_recurrent, layer  31: dev = MTL0\nllama_memory_recurrent, layer  32: dev = MTL0\nllama_memory_recurrent, layer  33: dev = MTL0\nllama_memory_recurrent, layer  34: dev = MTL0\nllama_memory_recurrent, layer  36: dev = MTL0\nllama_memory_recurrent, layer  37: dev = MTL0\nllama_memory_recurrent, layer  38: dev = MTL0\nllama_memory_recurrent, layer  39: dev = MTL0\nllama_memory_recurrent:       MTL0 RS buffer size =   147.48 MiB\nsched_reserve:       MTL0 compute buffer size =   282.28 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 41,
      "totalLayers": 41,
      "inferenceMs": 557.7706249969197,
      "durationMs": 566.5681669997866,
      "tokensPerSecond": 12.549961733891344,
      "inputTokens": 72,
      "outputTokens": 7,
      "outputLimitReached": false,
      "rssBytes": 19945783296,
      "cpuThreads": 4,
      "totalMs": 1844.3577089929022,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "Unknown"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "loadMs": 1277.7895419931156,
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer   0 assigned to device MTL0, is_swa = 0\nload_tensors: layer   1 assigned to device MTL0, is_swa = 0\nload_tensors: layer   2 assigned to device MTL0, is_swa = 0\nload_tensors: layer   3 assigned to device MTL0, is_swa = 0\nload_tensors: layer   4 assigned to device MTL0, is_swa = 0\nload_tensors: layer   5 assigned to device MTL0, is_swa = 0\nload_tensors: layer   6 assigned to device MTL0, is_swa = 0\nload_tensors: layer   7 assigned to device MTL0, is_swa = 0\nload_tensors: layer   8 assigned to device MTL0, is_swa = 0\nload_tensors: layer   9 assigned to device MTL0, is_swa = 0\nload_tensors: layer  10 assigned to device MTL0, is_swa = 0\nload_tensors: layer  11 assigned to device MTL0, is_swa = 0\nload_tensors: layer  12 assigned to device MTL0, is_swa = 0\nload_tensors: layer  13 assigned to device MTL0, is_swa = 0\nload_tensors: layer  14 assigned to device MTL0, is_swa = 0\nload_tensors: layer  15 assigned to device MTL0, is_swa = 0\nload_tensors: layer  16 assigned to device MTL0, is_swa = 0\nload_tensors: layer  17 assigned to device MTL0, is_swa = 0\nload_tensors: layer  18 assigned to device MTL0, is_swa = 0\nload_tensors: layer  19 assigned to device MTL0, is_swa = 0\nload_tensors: layer  20 assigned to device MTL0, is_swa = 0\nload_tensors: layer  21 assigned to device MTL0, is_swa = 0\nload_tensors: layer  22 assigned to device MTL0, is_swa = 0\nload_tensors: layer  23 assigned to device MTL0, is_swa = 0\nload_tensors: layer  24 assigned to device MTL0, is_swa = 0\nload_tensors: layer  25 assigned to device MTL0, is_swa = 0\nload_tensors: layer  26 assigned to device MTL0, is_swa = 0\nload_tensors: layer  27 assigned to device MTL0, is_swa = 0\nload_tensors: layer  28 assigned to device MTL0, is_swa = 0\nload_tensors: layer  29 assigned to device MTL0, is_swa = 0\nload_tensors: layer  30 assigned to device MTL0, is_swa = 0\nload_tensors: layer  31 assigned to device MTL0, is_swa = 0\nload_tensors: layer  32 assigned to device MTL0, is_swa = 0\nload_tensors: layer  33 assigned to device MTL0, is_swa = 0\nload_tensors: layer  34 assigned to device MTL0, is_swa = 0\nload_tensors: layer  35 assigned to device MTL0, is_swa = 0\nload_tensors: layer  36 assigned to device MTL0, is_swa = 0\nload_tensors: layer  37 assigned to device MTL0, is_swa = 0\nload_tensors: layer  38 assigned to device MTL0, is_swa = 0\nload_tensors: layer  39 assigned to device MTL0, is_swa = 0\nload_tensors: layer  40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =   321.56 MiB\nload_tensors:  MTL0_Mapped model buffer size = 18570.93 MiB\nllama_kv_cache: layer   5: dev = MTL0\nllama_kv_cache: layer  15: dev = MTL0\nllama_kv_cache: layer  25: dev = MTL0\nllama_kv_cache: layer  35: dev = MTL0\nllama_kv_cache:       MTL0 KV buffer size =   128.00 MiB\nllama_memory_recurrent, layer   0: dev = MTL0\nllama_memory_recurrent, layer   1: dev = MTL0\nllama_memory_recurrent, layer   2: dev = MTL0\nllama_memory_recurrent, layer   3: dev = MTL0\nllama_memory_recurrent, layer   4: dev = MTL0\nllama_memory_recurrent, layer   6: dev = MTL0\nllama_memory_recurrent, layer   7: dev = MTL0\nllama_memory_recurrent, layer   8: dev = MTL0\nllama_memory_recurrent, layer   9: dev = MTL0\nllama_memory_recurrent, layer  10: dev = MTL0\nllama_memory_recurrent, layer  11: dev = MTL0\nllama_memory_recurrent, layer  12: dev = MTL0\nllama_memory_recurrent, layer  13: dev = MTL0\nllama_memory_recurrent, layer  14: dev = MTL0\nllama_memory_recurrent, layer  16: dev = MTL0\nllama_memory_recurrent, layer  17: dev = MTL0\nllama_memory_recurrent, layer  18: dev = MTL0\nllama_memory_recurrent, layer  19: dev = MTL0\nllama_memory_recurrent, layer  20: dev = MTL0\nllama_memory_recurrent, layer  21: dev = MTL0\nllama_memory_recurrent, layer  22: dev = MTL0\nllama_memory_recurrent, layer  23: dev = MTL0\nllama_memory_recurrent, layer  24: dev = MTL0\nllama_memory_recurrent, layer  26: dev = MTL0\nllama_memory_recurrent, layer  27: dev = MTL0\nllama_memory_recurrent, layer  28: dev = MTL0\nllama_memory_recurrent, layer  29: dev = MTL0\nllama_memory_recurrent, layer  30: dev = MTL0\nllama_memory_recurrent, layer  31: dev = MTL0\nllama_memory_recurrent, layer  32: dev = MTL0\nllama_memory_recurrent, layer  33: dev = MTL0\nllama_memory_recurrent, layer  34: dev = MTL0\nllama_memory_recurrent, layer  36: dev = MTL0\nllama_memory_recurrent, layer  37: dev = MTL0\nllama_memory_recurrent, layer  38: dev = MTL0\nllama_memory_recurrent, layer  39: dev = MTL0\nllama_memory_recurrent:       MTL0 RS buffer size =   147.48 MiB\nsched_reserve:       MTL0 compute buffer size =   282.28 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 41,
      "totalLayers": 41,
      "inferenceMs": 385.34866699774284,
      "durationMs": 393.4925000066869,
      "tokensPerSecond": 5.190104887560737,
      "inputTokens": 74,
      "outputTokens": 2,
      "outputLimitReached": false,
      "rssBytes": 19945848832,
      "cpuThreads": 4,
      "totalMs": 1671.2820419998025,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "Paris"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "loadMs": 1374.4882920000236,
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer   0 assigned to device MTL0, is_swa = 0\nload_tensors: layer   1 assigned to device MTL0, is_swa = 0\nload_tensors: layer   2 assigned to device MTL0, is_swa = 0\nload_tensors: layer   3 assigned to device MTL0, is_swa = 0\nload_tensors: layer   4 assigned to device MTL0, is_swa = 0\nload_tensors: layer   5 assigned to device MTL0, is_swa = 0\nload_tensors: layer   6 assigned to device MTL0, is_swa = 0\nload_tensors: layer   7 assigned to device MTL0, is_swa = 0\nload_tensors: layer   8 assigned to device MTL0, is_swa = 0\nload_tensors: layer   9 assigned to device MTL0, is_swa = 0\nload_tensors: layer  10 assigned to device MTL0, is_swa = 0\nload_tensors: layer  11 assigned to device MTL0, is_swa = 0\nload_tensors: layer  12 assigned to device MTL0, is_swa = 0\nload_tensors: layer  13 assigned to device MTL0, is_swa = 0\nload_tensors: layer  14 assigned to device MTL0, is_swa = 0\nload_tensors: layer  15 assigned to device MTL0, is_swa = 0\nload_tensors: layer  16 assigned to device MTL0, is_swa = 0\nload_tensors: layer  17 assigned to device MTL0, is_swa = 0\nload_tensors: layer  18 assigned to device MTL0, is_swa = 0\nload_tensors: layer  19 assigned to device MTL0, is_swa = 0\nload_tensors: layer  20 assigned to device MTL0, is_swa = 0\nload_tensors: layer  21 assigned to device MTL0, is_swa = 0\nload_tensors: layer  22 assigned to device MTL0, is_swa = 0\nload_tensors: layer  23 assigned to device MTL0, is_swa = 0\nload_tensors: layer  24 assigned to device MTL0, is_swa = 0\nload_tensors: layer  25 assigned to device MTL0, is_swa = 0\nload_tensors: layer  26 assigned to device MTL0, is_swa = 0\nload_tensors: layer  27 assigned to device MTL0, is_swa = 0\nload_tensors: layer  28 assigned to device MTL0, is_swa = 0\nload_tensors: layer  29 assigned to device MTL0, is_swa = 0\nload_tensors: layer  30 assigned to device MTL0, is_swa = 0\nload_tensors: layer  31 assigned to device MTL0, is_swa = 0\nload_tensors: layer  32 assigned to device MTL0, is_swa = 0\nload_tensors: layer  33 assigned to device MTL0, is_swa = 0\nload_tensors: layer  34 assigned to device MTL0, is_swa = 0\nload_tensors: layer  35 assigned to device MTL0, is_swa = 0\nload_tensors: layer  36 assigned to device MTL0, is_swa = 0\nload_tensors: layer  37 assigned to device MTL0, is_swa = 0\nload_tensors: layer  38 assigned to device MTL0, is_swa = 0\nload_tensors: layer  39 assigned to device MTL0, is_swa = 0\nload_tensors: layer  40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =   321.56 MiB\nload_tensors:  MTL0_Mapped model buffer size = 18570.93 MiB\nllama_kv_cache: layer   5: dev = MTL0\nllama_kv_cache: layer  15: dev = MTL0\nllama_kv_cache: layer  25: dev = MTL0\nllama_kv_cache: layer  35: dev = MTL0\nllama_kv_cache:       MTL0 KV buffer size =   128.00 MiB\nllama_memory_recurrent, layer   0: dev = MTL0\nllama_memory_recurrent, layer   1: dev = MTL0\nllama_memory_recurrent, layer   2: dev = MTL0\nllama_memory_recurrent, layer   3: dev = MTL0\nllama_memory_recurrent, layer   4: dev = MTL0\nllama_memory_recurrent, layer   6: dev = MTL0\nllama_memory_recurrent, layer   7: dev = MTL0\nllama_memory_recurrent, layer   8: dev = MTL0\nllama_memory_recurrent, layer   9: dev = MTL0\nllama_memory_recurrent, layer  10: dev = MTL0\nllama_memory_recurrent, layer  11: dev = MTL0\nllama_memory_recurrent, layer  12: dev = MTL0\nllama_memory_recurrent, layer  13: dev = MTL0\nllama_memory_recurrent, layer  14: dev = MTL0\nllama_memory_recurrent, layer  16: dev = MTL0\nllama_memory_recurrent, layer  17: dev = MTL0\nllama_memory_recurrent, layer  18: dev = MTL0\nllama_memory_recurrent, layer  19: dev = MTL0\nllama_memory_recurrent, layer  20: dev = MTL0\nllama_memory_recurrent, layer  21: dev = MTL0\nllama_memory_recurrent, layer  22: dev = MTL0\nllama_memory_recurrent, layer  23: dev = MTL0\nllama_memory_recurrent, layer  24: dev = MTL0\nllama_memory_recurrent, layer  26: dev = MTL0\nllama_memory_recurrent, layer  27: dev = MTL0\nllama_memory_recurrent, layer  28: dev = MTL0\nllama_memory_recurrent, layer  29: dev = MTL0\nllama_memory_recurrent, layer  30: dev = MTL0\nllama_memory_recurrent, layer  31: dev = MTL0\nllama_memory_recurrent, layer  32: dev = MTL0\nllama_memory_recurrent, layer  33: dev = MTL0\nllama_memory_recurrent, layer  34: dev = MTL0\nllama_memory_recurrent, layer  36: dev = MTL0\nllama_memory_recurrent, layer  37: dev = MTL0\nllama_memory_recurrent, layer  38: dev = MTL0\nllama_memory_recurrent, layer  39: dev = MTL0\nllama_memory_recurrent:       MTL0 RS buffer size =   147.48 MiB\nsched_reserve:       MTL0 compute buffer size =   282.28 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 41,
      "totalLayers": 41,
      "inferenceMs": 357.9981660004705,
      "durationMs": 367.01845800416777,
      "tokensPerSecond": 5.586620798491385,
      "inputTokens": 63,
      "outputTokens": 2,
      "outputLimitReached": false,
      "rssBytes": 19943505920,
      "cpuThreads": 4,
      "totalMs": 1741.5067500041914,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "56"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "loadMs": 1374.4882920000236,
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer   0 assigned to device MTL0, is_swa = 0\nload_tensors: layer   1 assigned to device MTL0, is_swa = 0\nload_tensors: layer   2 assigned to device MTL0, is_swa = 0\nload_tensors: layer   3 assigned to device MTL0, is_swa = 0\nload_tensors: layer   4 assigned to device MTL0, is_swa = 0\nload_tensors: layer   5 assigned to device MTL0, is_swa = 0\nload_tensors: layer   6 assigned to device MTL0, is_swa = 0\nload_tensors: layer   7 assigned to device MTL0, is_swa = 0\nload_tensors: layer   8 assigned to device MTL0, is_swa = 0\nload_tensors: layer   9 assigned to device MTL0, is_swa = 0\nload_tensors: layer  10 assigned to device MTL0, is_swa = 0\nload_tensors: layer  11 assigned to device MTL0, is_swa = 0\nload_tensors: layer  12 assigned to device MTL0, is_swa = 0\nload_tensors: layer  13 assigned to device MTL0, is_swa = 0\nload_tensors: layer  14 assigned to device MTL0, is_swa = 0\nload_tensors: layer  15 assigned to device MTL0, is_swa = 0\nload_tensors: layer  16 assigned to device MTL0, is_swa = 0\nload_tensors: layer  17 assigned to device MTL0, is_swa = 0\nload_tensors: layer  18 assigned to device MTL0, is_swa = 0\nload_tensors: layer  19 assigned to device MTL0, is_swa = 0\nload_tensors: layer  20 assigned to device MTL0, is_swa = 0\nload_tensors: layer  21 assigned to device MTL0, is_swa = 0\nload_tensors: layer  22 assigned to device MTL0, is_swa = 0\nload_tensors: layer  23 assigned to device MTL0, is_swa = 0\nload_tensors: layer  24 assigned to device MTL0, is_swa = 0\nload_tensors: layer  25 assigned to device MTL0, is_swa = 0\nload_tensors: layer  26 assigned to device MTL0, is_swa = 0\nload_tensors: layer  27 assigned to device MTL0, is_swa = 0\nload_tensors: layer  28 assigned to device MTL0, is_swa = 0\nload_tensors: layer  29 assigned to device MTL0, is_swa = 0\nload_tensors: layer  30 assigned to device MTL0, is_swa = 0\nload_tensors: layer  31 assigned to device MTL0, is_swa = 0\nload_tensors: layer  32 assigned to device MTL0, is_swa = 0\nload_tensors: layer  33 assigned to device MTL0, is_swa = 0\nload_tensors: layer  34 assigned to device MTL0, is_swa = 0\nload_tensors: layer  35 assigned to device MTL0, is_swa = 0\nload_tensors: layer  36 assigned to device MTL0, is_swa = 0\nload_tensors: layer  37 assigned to device MTL0, is_swa = 0\nload_tensors: layer  38 assigned to device MTL0, is_swa = 0\nload_tensors: layer  39 assigned to device MTL0, is_swa = 0\nload_tensors: layer  40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =   321.56 MiB\nload_tensors:  MTL0_Mapped model buffer size = 18570.93 MiB\nllama_kv_cache: layer   5: dev = MTL0\nllama_kv_cache: layer  15: dev = MTL0\nllama_kv_cache: layer  25: dev = MTL0\nllama_kv_cache: layer  35: dev = MTL0\nllama_kv_cache:       MTL0 KV buffer size =   128.00 MiB\nllama_memory_recurrent, layer   0: dev = MTL0\nllama_memory_recurrent, layer   1: dev = MTL0\nllama_memory_recurrent, layer   2: dev = MTL0\nllama_memory_recurrent, layer   3: dev = MTL0\nllama_memory_recurrent, layer   4: dev = MTL0\nllama_memory_recurrent, layer   6: dev = MTL0\nllama_memory_recurrent, layer   7: dev = MTL0\nllama_memory_recurrent, layer   8: dev = MTL0\nllama_memory_recurrent, layer   9: dev = MTL0\nllama_memory_recurrent, layer  10: dev = MTL0\nllama_memory_recurrent, layer  11: dev = MTL0\nllama_memory_recurrent, layer  12: dev = MTL0\nllama_memory_recurrent, layer  13: dev = MTL0\nllama_memory_recurrent, layer  14: dev = MTL0\nllama_memory_recurrent, layer  16: dev = MTL0\nllama_memory_recurrent, layer  17: dev = MTL0\nllama_memory_recurrent, layer  18: dev = MTL0\nllama_memory_recurrent, layer  19: dev = MTL0\nllama_memory_recurrent, layer  20: dev = MTL0\nllama_memory_recurrent, layer  21: dev = MTL0\nllama_memory_recurrent, layer  22: dev = MTL0\nllama_memory_recurrent, layer  23: dev = MTL0\nllama_memory_recurrent, layer  24: dev = MTL0\nllama_memory_recurrent, layer  26: dev = MTL0\nllama_memory_recurrent, layer  27: dev = MTL0\nllama_memory_recurrent, layer  28: dev = MTL0\nllama_memory_recurrent, layer  29: dev = MTL0\nllama_memory_recurrent, layer  30: dev = MTL0\nllama_memory_recurrent, layer  31: dev = MTL0\nllama_memory_recurrent, layer  32: dev = MTL0\nllama_memory_recurrent, layer  33: dev = MTL0\nllama_memory_recurrent, layer  34: dev = MTL0\nllama_memory_recurrent, layer  36: dev = MTL0\nllama_memory_recurrent, layer  37: dev = MTL0\nllama_memory_recurrent, layer  38: dev = MTL0\nllama_memory_recurrent, layer  39: dev = MTL0\nllama_memory_recurrent:       MTL0 RS buffer size =   147.48 MiB\nsched_reserve:       MTL0 compute buffer size =   282.28 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 41,
      "totalLayers": 41,
      "inferenceMs": 333.6505830011447,
      "durationMs": 342.3435000004247,
      "tokensPerSecond": 5.994294935768592,
      "inputTokens": 63,
      "outputTokens": 2,
      "outputLimitReached": false,
      "rssBytes": 19944849408,
      "cpuThreads": 4,
      "totalMs": 1716.8317920004483,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "chat"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "loadMs": 1374.4882920000236,
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer   0 assigned to device MTL0, is_swa = 0\nload_tensors: layer   1 assigned to device MTL0, is_swa = 0\nload_tensors: layer   2 assigned to device MTL0, is_swa = 0\nload_tensors: layer   3 assigned to device MTL0, is_swa = 0\nload_tensors: layer   4 assigned to device MTL0, is_swa = 0\nload_tensors: layer   5 assigned to device MTL0, is_swa = 0\nload_tensors: layer   6 assigned to device MTL0, is_swa = 0\nload_tensors: layer   7 assigned to device MTL0, is_swa = 0\nload_tensors: layer   8 assigned to device MTL0, is_swa = 0\nload_tensors: layer   9 assigned to device MTL0, is_swa = 0\nload_tensors: layer  10 assigned to device MTL0, is_swa = 0\nload_tensors: layer  11 assigned to device MTL0, is_swa = 0\nload_tensors: layer  12 assigned to device MTL0, is_swa = 0\nload_tensors: layer  13 assigned to device MTL0, is_swa = 0\nload_tensors: layer  14 assigned to device MTL0, is_swa = 0\nload_tensors: layer  15 assigned to device MTL0, is_swa = 0\nload_tensors: layer  16 assigned to device MTL0, is_swa = 0\nload_tensors: layer  17 assigned to device MTL0, is_swa = 0\nload_tensors: layer  18 assigned to device MTL0, is_swa = 0\nload_tensors: layer  19 assigned to device MTL0, is_swa = 0\nload_tensors: layer  20 assigned to device MTL0, is_swa = 0\nload_tensors: layer  21 assigned to device MTL0, is_swa = 0\nload_tensors: layer  22 assigned to device MTL0, is_swa = 0\nload_tensors: layer  23 assigned to device MTL0, is_swa = 0\nload_tensors: layer  24 assigned to device MTL0, is_swa = 0\nload_tensors: layer  25 assigned to device MTL0, is_swa = 0\nload_tensors: layer  26 assigned to device MTL0, is_swa = 0\nload_tensors: layer  27 assigned to device MTL0, is_swa = 0\nload_tensors: layer  28 assigned to device MTL0, is_swa = 0\nload_tensors: layer  29 assigned to device MTL0, is_swa = 0\nload_tensors: layer  30 assigned to device MTL0, is_swa = 0\nload_tensors: layer  31 assigned to device MTL0, is_swa = 0\nload_tensors: layer  32 assigned to device MTL0, is_swa = 0\nload_tensors: layer  33 assigned to device MTL0, is_swa = 0\nload_tensors: layer  34 assigned to device MTL0, is_swa = 0\nload_tensors: layer  35 assigned to device MTL0, is_swa = 0\nload_tensors: layer  36 assigned to device MTL0, is_swa = 0\nload_tensors: layer  37 assigned to device MTL0, is_swa = 0\nload_tensors: layer  38 assigned to device MTL0, is_swa = 0\nload_tensors: layer  39 assigned to device MTL0, is_swa = 0\nload_tensors: layer  40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =   321.56 MiB\nload_tensors:  MTL0_Mapped model buffer size = 18570.93 MiB\nllama_kv_cache: layer   5: dev = MTL0\nllama_kv_cache: layer  15: dev = MTL0\nllama_kv_cache: layer  25: dev = MTL0\nllama_kv_cache: layer  35: dev = MTL0\nllama_kv_cache:       MTL0 KV buffer size =   128.00 MiB\nllama_memory_recurrent, layer   0: dev = MTL0\nllama_memory_recurrent, layer   1: dev = MTL0\nllama_memory_recurrent, layer   2: dev = MTL0\nllama_memory_recurrent, layer   3: dev = MTL0\nllama_memory_recurrent, layer   4: dev = MTL0\nllama_memory_recurrent, layer   6: dev = MTL0\nllama_memory_recurrent, layer   7: dev = MTL0\nllama_memory_recurrent, layer   8: dev = MTL0\nllama_memory_recurrent, layer   9: dev = MTL0\nllama_memory_recurrent, layer  10: dev = MTL0\nllama_memory_recurrent, layer  11: dev = MTL0\nllama_memory_recurrent, layer  12: dev = MTL0\nllama_memory_recurrent, layer  13: dev = MTL0\nllama_memory_recurrent, layer  14: dev = MTL0\nllama_memory_recurrent, layer  16: dev = MTL0\nllama_memory_recurrent, layer  17: dev = MTL0\nllama_memory_recurrent, layer  18: dev = MTL0\nllama_memory_recurrent, layer  19: dev = MTL0\nllama_memory_recurrent, layer  20: dev = MTL0\nllama_memory_recurrent, layer  21: dev = MTL0\nllama_memory_recurrent, layer  22: dev = MTL0\nllama_memory_recurrent, layer  23: dev = MTL0\nllama_memory_recurrent, layer  24: dev = MTL0\nllama_memory_recurrent, layer  26: dev = MTL0\nllama_memory_recurrent, layer  27: dev = MTL0\nllama_memory_recurrent, layer  28: dev = MTL0\nllama_memory_recurrent, layer  29: dev = MTL0\nllama_memory_recurrent, layer  30: dev = MTL0\nllama_memory_recurrent, layer  31: dev = MTL0\nllama_memory_recurrent, layer  32: dev = MTL0\nllama_memory_recurrent, layer  33: dev = MTL0\nllama_memory_recurrent, layer  34: dev = MTL0\nllama_memory_recurrent, layer  36: dev = MTL0\nllama_memory_recurrent, layer  37: dev = MTL0\nllama_memory_recurrent, layer  38: dev = MTL0\nllama_memory_recurrent, layer  39: dev = MTL0\nllama_memory_recurrent:       MTL0 RS buffer size =   147.48 MiB\nsched_reserve:       MTL0 compute buffer size =   282.28 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 41,
      "totalLayers": 41,
      "inferenceMs": 365.4482080019079,
      "durationMs": 374.9816249983269,
      "tokensPerSecond": 5.472731720139009,
      "inputTokens": 68,
      "outputTokens": 2,
      "outputLimitReached": false,
      "rssBytes": 19946733568,
      "cpuThreads": 4,
      "totalMs": 1749.4699169983505,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "hola"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "loadMs": 1374.4882920000236,
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer   0 assigned to device MTL0, is_swa = 0\nload_tensors: layer   1 assigned to device MTL0, is_swa = 0\nload_tensors: layer   2 assigned to device MTL0, is_swa = 0\nload_tensors: layer   3 assigned to device MTL0, is_swa = 0\nload_tensors: layer   4 assigned to device MTL0, is_swa = 0\nload_tensors: layer   5 assigned to device MTL0, is_swa = 0\nload_tensors: layer   6 assigned to device MTL0, is_swa = 0\nload_tensors: layer   7 assigned to device MTL0, is_swa = 0\nload_tensors: layer   8 assigned to device MTL0, is_swa = 0\nload_tensors: layer   9 assigned to device MTL0, is_swa = 0\nload_tensors: layer  10 assigned to device MTL0, is_swa = 0\nload_tensors: layer  11 assigned to device MTL0, is_swa = 0\nload_tensors: layer  12 assigned to device MTL0, is_swa = 0\nload_tensors: layer  13 assigned to device MTL0, is_swa = 0\nload_tensors: layer  14 assigned to device MTL0, is_swa = 0\nload_tensors: layer  15 assigned to device MTL0, is_swa = 0\nload_tensors: layer  16 assigned to device MTL0, is_swa = 0\nload_tensors: layer  17 assigned to device MTL0, is_swa = 0\nload_tensors: layer  18 assigned to device MTL0, is_swa = 0\nload_tensors: layer  19 assigned to device MTL0, is_swa = 0\nload_tensors: layer  20 assigned to device MTL0, is_swa = 0\nload_tensors: layer  21 assigned to device MTL0, is_swa = 0\nload_tensors: layer  22 assigned to device MTL0, is_swa = 0\nload_tensors: layer  23 assigned to device MTL0, is_swa = 0\nload_tensors: layer  24 assigned to device MTL0, is_swa = 0\nload_tensors: layer  25 assigned to device MTL0, is_swa = 0\nload_tensors: layer  26 assigned to device MTL0, is_swa = 0\nload_tensors: layer  27 assigned to device MTL0, is_swa = 0\nload_tensors: layer  28 assigned to device MTL0, is_swa = 0\nload_tensors: layer  29 assigned to device MTL0, is_swa = 0\nload_tensors: layer  30 assigned to device MTL0, is_swa = 0\nload_tensors: layer  31 assigned to device MTL0, is_swa = 0\nload_tensors: layer  32 assigned to device MTL0, is_swa = 0\nload_tensors: layer  33 assigned to device MTL0, is_swa = 0\nload_tensors: layer  34 assigned to device MTL0, is_swa = 0\nload_tensors: layer  35 assigned to device MTL0, is_swa = 0\nload_tensors: layer  36 assigned to device MTL0, is_swa = 0\nload_tensors: layer  37 assigned to device MTL0, is_swa = 0\nload_tensors: layer  38 assigned to device MTL0, is_swa = 0\nload_tensors: layer  39 assigned to device MTL0, is_swa = 0\nload_tensors: layer  40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =   321.56 MiB\nload_tensors:  MTL0_Mapped model buffer size = 18570.93 MiB\nllama_kv_cache: layer   5: dev = MTL0\nllama_kv_cache: layer  15: dev = MTL0\nllama_kv_cache: layer  25: dev = MTL0\nllama_kv_cache: layer  35: dev = MTL0\nllama_kv_cache:       MTL0 KV buffer size =   128.00 MiB\nllama_memory_recurrent, layer   0: dev = MTL0\nllama_memory_recurrent, layer   1: dev = MTL0\nllama_memory_recurrent, layer   2: dev = MTL0\nllama_memory_recurrent, layer   3: dev = MTL0\nllama_memory_recurrent, layer   4: dev = MTL0\nllama_memory_recurrent, layer   6: dev = MTL0\nllama_memory_recurrent, layer   7: dev = MTL0\nllama_memory_recurrent, layer   8: dev = MTL0\nllama_memory_recurrent, layer   9: dev = MTL0\nllama_memory_recurrent, layer  10: dev = MTL0\nllama_memory_recurrent, layer  11: dev = MTL0\nllama_memory_recurrent, layer  12: dev = MTL0\nllama_memory_recurrent, layer  13: dev = MTL0\nllama_memory_recurrent, layer  14: dev = MTL0\nllama_memory_recurrent, layer  16: dev = MTL0\nllama_memory_recurrent, layer  17: dev = MTL0\nllama_memory_recurrent, layer  18: dev = MTL0\nllama_memory_recurrent, layer  19: dev = MTL0\nllama_memory_recurrent, layer  20: dev = MTL0\nllama_memory_recurrent, layer  21: dev = MTL0\nllama_memory_recurrent, layer  22: dev = MTL0\nllama_memory_recurrent, layer  23: dev = MTL0\nllama_memory_recurrent, layer  24: dev = MTL0\nllama_memory_recurrent, layer  26: dev = MTL0\nllama_memory_recurrent, layer  27: dev = MTL0\nllama_memory_recurrent, layer  28: dev = MTL0\nllama_memory_recurrent, layer  29: dev = MTL0\nllama_memory_recurrent, layer  30: dev = MTL0\nllama_memory_recurrent, layer  31: dev = MTL0\nllama_memory_recurrent, layer  32: dev = MTL0\nllama_memory_recurrent, layer  33: dev = MTL0\nllama_memory_recurrent, layer  34: dev = MTL0\nllama_memory_recurrent, layer  36: dev = MTL0\nllama_memory_recurrent, layer  37: dev = MTL0\nllama_memory_recurrent, layer  38: dev = MTL0\nllama_memory_recurrent, layer  39: dev = MTL0\nllama_memory_recurrent:       MTL0 RS buffer size =   147.48 MiB\nsched_reserve:       MTL0 compute buffer size =   282.28 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 41,
      "totalLayers": 41,
      "inferenceMs": 405.2658340006019,
      "durationMs": 413.8391250016866,
      "tokensPerSecond": 7.402548520770553,
      "inputTokens": 69,
      "outputTokens": 3,
      "outputLimitReached": false,
      "rssBytes": 19948175360,
      "cpuThreads": 4,
      "totalMs": 1788.3274170017103,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "2026-11-04"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "loadMs": 1374.4882920000236,
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer   0 assigned to device MTL0, is_swa = 0\nload_tensors: layer   1 assigned to device MTL0, is_swa = 0\nload_tensors: layer   2 assigned to device MTL0, is_swa = 0\nload_tensors: layer   3 assigned to device MTL0, is_swa = 0\nload_tensors: layer   4 assigned to device MTL0, is_swa = 0\nload_tensors: layer   5 assigned to device MTL0, is_swa = 0\nload_tensors: layer   6 assigned to device MTL0, is_swa = 0\nload_tensors: layer   7 assigned to device MTL0, is_swa = 0\nload_tensors: layer   8 assigned to device MTL0, is_swa = 0\nload_tensors: layer   9 assigned to device MTL0, is_swa = 0\nload_tensors: layer  10 assigned to device MTL0, is_swa = 0\nload_tensors: layer  11 assigned to device MTL0, is_swa = 0\nload_tensors: layer  12 assigned to device MTL0, is_swa = 0\nload_tensors: layer  13 assigned to device MTL0, is_swa = 0\nload_tensors: layer  14 assigned to device MTL0, is_swa = 0\nload_tensors: layer  15 assigned to device MTL0, is_swa = 0\nload_tensors: layer  16 assigned to device MTL0, is_swa = 0\nload_tensors: layer  17 assigned to device MTL0, is_swa = 0\nload_tensors: layer  18 assigned to device MTL0, is_swa = 0\nload_tensors: layer  19 assigned to device MTL0, is_swa = 0\nload_tensors: layer  20 assigned to device MTL0, is_swa = 0\nload_tensors: layer  21 assigned to device MTL0, is_swa = 0\nload_tensors: layer  22 assigned to device MTL0, is_swa = 0\nload_tensors: layer  23 assigned to device MTL0, is_swa = 0\nload_tensors: layer  24 assigned to device MTL0, is_swa = 0\nload_tensors: layer  25 assigned to device MTL0, is_swa = 0\nload_tensors: layer  26 assigned to device MTL0, is_swa = 0\nload_tensors: layer  27 assigned to device MTL0, is_swa = 0\nload_tensors: layer  28 assigned to device MTL0, is_swa = 0\nload_tensors: layer  29 assigned to device MTL0, is_swa = 0\nload_tensors: layer  30 assigned to device MTL0, is_swa = 0\nload_tensors: layer  31 assigned to device MTL0, is_swa = 0\nload_tensors: layer  32 assigned to device MTL0, is_swa = 0\nload_tensors: layer  33 assigned to device MTL0, is_swa = 0\nload_tensors: layer  34 assigned to device MTL0, is_swa = 0\nload_tensors: layer  35 assigned to device MTL0, is_swa = 0\nload_tensors: layer  36 assigned to device MTL0, is_swa = 0\nload_tensors: layer  37 assigned to device MTL0, is_swa = 0\nload_tensors: layer  38 assigned to device MTL0, is_swa = 0\nload_tensors: layer  39 assigned to device MTL0, is_swa = 0\nload_tensors: layer  40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =   321.56 MiB\nload_tensors:  MTL0_Mapped model buffer size = 18570.93 MiB\nllama_kv_cache: layer   5: dev = MTL0\nllama_kv_cache: layer  15: dev = MTL0\nllama_kv_cache: layer  25: dev = MTL0\nllama_kv_cache: layer  35: dev = MTL0\nllama_kv_cache:       MTL0 KV buffer size =   128.00 MiB\nllama_memory_recurrent, layer   0: dev = MTL0\nllama_memory_recurrent, layer   1: dev = MTL0\nllama_memory_recurrent, layer   2: dev = MTL0\nllama_memory_recurrent, layer   3: dev = MTL0\nllama_memory_recurrent, layer   4: dev = MTL0\nllama_memory_recurrent, layer   6: dev = MTL0\nllama_memory_recurrent, layer   7: dev = MTL0\nllama_memory_recurrent, layer   8: dev = MTL0\nllama_memory_recurrent, layer   9: dev = MTL0\nllama_memory_recurrent, layer  10: dev = MTL0\nllama_memory_recurrent, layer  11: dev = MTL0\nllama_memory_recurrent, layer  12: dev = MTL0\nllama_memory_recurrent, layer  13: dev = MTL0\nllama_memory_recurrent, layer  14: dev = MTL0\nllama_memory_recurrent, layer  16: dev = MTL0\nllama_memory_recurrent, layer  17: dev = MTL0\nllama_memory_recurrent, layer  18: dev = MTL0\nllama_memory_recurrent, layer  19: dev = MTL0\nllama_memory_recurrent, layer  20: dev = MTL0\nllama_memory_recurrent, layer  21: dev = MTL0\nllama_memory_recurrent, layer  22: dev = MTL0\nllama_memory_recurrent, layer  23: dev = MTL0\nllama_memory_recurrent, layer  24: dev = MTL0\nllama_memory_recurrent, layer  26: dev = MTL0\nllama_memory_recurrent, layer  27: dev = MTL0\nllama_memory_recurrent, layer  28: dev = MTL0\nllama_memory_recurrent, layer  29: dev = MTL0\nllama_memory_recurrent, layer  30: dev = MTL0\nllama_memory_recurrent, layer  31: dev = MTL0\nllama_memory_recurrent, layer  32: dev = MTL0\nllama_memory_recurrent, layer  33: dev = MTL0\nllama_memory_recurrent, layer  34: dev = MTL0\nllama_memory_recurrent, layer  36: dev = MTL0\nllama_memory_recurrent, layer  37: dev = MTL0\nllama_memory_recurrent, layer  38: dev = MTL0\nllama_memory_recurrent, layer  39: dev = MTL0\nllama_memory_recurrent:       MTL0 RS buffer size =   147.48 MiB\nsched_reserve:       MTL0 compute buffer size =   282.28 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 41,
      "totalLayers": 41,
      "inferenceMs": 551.9109160013613,
      "durationMs": 561.6470420063706,
      "tokensPerSecond": 12.683206287557345,
      "inputTokens": 72,
      "outputTokens": 7,
      "outputLimitReached": false,
      "rssBytes": 19948519424,
      "cpuThreads": 4,
      "totalMs": 1936.1353340063943,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "Unknown"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "loadMs": 1374.4882920000236,
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer   0 assigned to device MTL0, is_swa = 0\nload_tensors: layer   1 assigned to device MTL0, is_swa = 0\nload_tensors: layer   2 assigned to device MTL0, is_swa = 0\nload_tensors: layer   3 assigned to device MTL0, is_swa = 0\nload_tensors: layer   4 assigned to device MTL0, is_swa = 0\nload_tensors: layer   5 assigned to device MTL0, is_swa = 0\nload_tensors: layer   6 assigned to device MTL0, is_swa = 0\nload_tensors: layer   7 assigned to device MTL0, is_swa = 0\nload_tensors: layer   8 assigned to device MTL0, is_swa = 0\nload_tensors: layer   9 assigned to device MTL0, is_swa = 0\nload_tensors: layer  10 assigned to device MTL0, is_swa = 0\nload_tensors: layer  11 assigned to device MTL0, is_swa = 0\nload_tensors: layer  12 assigned to device MTL0, is_swa = 0\nload_tensors: layer  13 assigned to device MTL0, is_swa = 0\nload_tensors: layer  14 assigned to device MTL0, is_swa = 0\nload_tensors: layer  15 assigned to device MTL0, is_swa = 0\nload_tensors: layer  16 assigned to device MTL0, is_swa = 0\nload_tensors: layer  17 assigned to device MTL0, is_swa = 0\nload_tensors: layer  18 assigned to device MTL0, is_swa = 0\nload_tensors: layer  19 assigned to device MTL0, is_swa = 0\nload_tensors: layer  20 assigned to device MTL0, is_swa = 0\nload_tensors: layer  21 assigned to device MTL0, is_swa = 0\nload_tensors: layer  22 assigned to device MTL0, is_swa = 0\nload_tensors: layer  23 assigned to device MTL0, is_swa = 0\nload_tensors: layer  24 assigned to device MTL0, is_swa = 0\nload_tensors: layer  25 assigned to device MTL0, is_swa = 0\nload_tensors: layer  26 assigned to device MTL0, is_swa = 0\nload_tensors: layer  27 assigned to device MTL0, is_swa = 0\nload_tensors: layer  28 assigned to device MTL0, is_swa = 0\nload_tensors: layer  29 assigned to device MTL0, is_swa = 0\nload_tensors: layer  30 assigned to device MTL0, is_swa = 0\nload_tensors: layer  31 assigned to device MTL0, is_swa = 0\nload_tensors: layer  32 assigned to device MTL0, is_swa = 0\nload_tensors: layer  33 assigned to device MTL0, is_swa = 0\nload_tensors: layer  34 assigned to device MTL0, is_swa = 0\nload_tensors: layer  35 assigned to device MTL0, is_swa = 0\nload_tensors: layer  36 assigned to device MTL0, is_swa = 0\nload_tensors: layer  37 assigned to device MTL0, is_swa = 0\nload_tensors: layer  38 assigned to device MTL0, is_swa = 0\nload_tensors: layer  39 assigned to device MTL0, is_swa = 0\nload_tensors: layer  40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =   321.56 MiB\nload_tensors:  MTL0_Mapped model buffer size = 18570.93 MiB\nllama_kv_cache: layer   5: dev = MTL0\nllama_kv_cache: layer  15: dev = MTL0\nllama_kv_cache: layer  25: dev = MTL0\nllama_kv_cache: layer  35: dev = MTL0\nllama_kv_cache:       MTL0 KV buffer size =   128.00 MiB\nllama_memory_recurrent, layer   0: dev = MTL0\nllama_memory_recurrent, layer   1: dev = MTL0\nllama_memory_recurrent, layer   2: dev = MTL0\nllama_memory_recurrent, layer   3: dev = MTL0\nllama_memory_recurrent, layer   4: dev = MTL0\nllama_memory_recurrent, layer   6: dev = MTL0\nllama_memory_recurrent, layer   7: dev = MTL0\nllama_memory_recurrent, layer   8: dev = MTL0\nllama_memory_recurrent, layer   9: dev = MTL0\nllama_memory_recurrent, layer  10: dev = MTL0\nllama_memory_recurrent, layer  11: dev = MTL0\nllama_memory_recurrent, layer  12: dev = MTL0\nllama_memory_recurrent, layer  13: dev = MTL0\nllama_memory_recurrent, layer  14: dev = MTL0\nllama_memory_recurrent, layer  16: dev = MTL0\nllama_memory_recurrent, layer  17: dev = MTL0\nllama_memory_recurrent, layer  18: dev = MTL0\nllama_memory_recurrent, layer  19: dev = MTL0\nllama_memory_recurrent, layer  20: dev = MTL0\nllama_memory_recurrent, layer  21: dev = MTL0\nllama_memory_recurrent, layer  22: dev = MTL0\nllama_memory_recurrent, layer  23: dev = MTL0\nllama_memory_recurrent, layer  24: dev = MTL0\nllama_memory_recurrent, layer  26: dev = MTL0\nllama_memory_recurrent, layer  27: dev = MTL0\nllama_memory_recurrent, layer  28: dev = MTL0\nllama_memory_recurrent, layer  29: dev = MTL0\nllama_memory_recurrent, layer  30: dev = MTL0\nllama_memory_recurrent, layer  31: dev = MTL0\nllama_memory_recurrent, layer  32: dev = MTL0\nllama_memory_recurrent, layer  33: dev = MTL0\nllama_memory_recurrent, layer  34: dev = MTL0\nllama_memory_recurrent, layer  36: dev = MTL0\nllama_memory_recurrent, layer  37: dev = MTL0\nllama_memory_recurrent, layer  38: dev = MTL0\nllama_memory_recurrent, layer  39: dev = MTL0\nllama_memory_recurrent:       MTL0 RS buffer size =   147.48 MiB\nsched_reserve:       MTL0 compute buffer size =   282.28 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 41,
      "totalLayers": 41,
      "inferenceMs": 377.84820800152374,
      "durationMs": 386.29999999830034,
      "tokensPerSecond": 5.293130833088229,
      "inputTokens": 74,
      "outputTokens": 2,
      "outputLimitReached": false,
      "rssBytes": 19948535808,
      "cpuThreads": 4,
      "totalMs": 1760.788291998324,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "q4",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "plainParity": true,
      "passed": false,
      "expected": "Paris"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "loadMs": 34819.97774999763,
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size = 18552.93 MiB\nload_tensors:   CPU_REPACK model buffer size = 18517.69 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 0,
      "totalLayers": 41,
      "inferenceMs": 3832.869332996779,
      "durationMs": 3841.931292001391,
      "tokensPerSecond": 0.521802291244892,
      "inputTokens": 63,
      "outputTokens": 2,
      "outputLimitReached": false,
      "rssBytes": 37500960768,
      "cpuThreads": 4,
      "totalMs": 38661.909041999024,
      "grade": "slow",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "q4",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": false,
      "expected": "56"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "loadMs": 34819.97774999763,
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size = 18552.93 MiB\nload_tensors:   CPU_REPACK model buffer size = 18517.69 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 0,
      "totalLayers": 41,
      "inferenceMs": 2350.014083000133,
      "durationMs": 2358.5027090011863,
      "tokensPerSecond": 0.8510587295914035,
      "inputTokens": 63,
      "outputTokens": 2,
      "outputLimitReached": false,
      "rssBytes": 37859672064,
      "cpuThreads": 4,
      "totalMs": 37178.48045899882,
      "grade": "slow",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "q4",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": false,
      "expected": "chat"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "loadMs": 34819.97774999763,
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size = 18552.93 MiB\nload_tensors:   CPU_REPACK model buffer size = 18517.69 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 0,
      "totalLayers": 41,
      "inferenceMs": 2191.428083999199,
      "durationMs": 2199.4122079995577,
      "tokensPerSecond": 0.9126468783543851,
      "inputTokens": 68,
      "outputTokens": 2,
      "outputLimitReached": false,
      "rssBytes": 38124797952,
      "cpuThreads": 4,
      "totalMs": 37019.38995799719,
      "grade": "slow",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "q4",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": false,
      "expected": "hola"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "loadMs": 34819.97774999763,
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size = 18552.93 MiB\nload_tensors:   CPU_REPACK model buffer size = 18517.69 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 0,
      "totalLayers": 41,
      "inferenceMs": 2424.1190839966293,
      "durationMs": 2432.439500000328,
      "tokensPerSecond": 1.2375629645445965,
      "inputTokens": 69,
      "outputTokens": 3,
      "outputLimitReached": false,
      "rssBytes": 38216499200,
      "cpuThreads": 4,
      "totalMs": 37252.41724999796,
      "grade": "slow",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "q4",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": false,
      "expected": "2026-11-04",
      "fixturePassed": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "loadMs": 34819.97774999763,
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size = 18552.93 MiB\nload_tensors:   CPU_REPACK model buffer size = 18517.69 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 0,
      "totalLayers": 41,
      "inferenceMs": 2686.0973330039997,
      "durationMs": 2694.8029580016737,
      "tokensPerSecond": 2.6060112989917394,
      "inputTokens": 72,
      "outputTokens": 7,
      "outputLimitReached": false,
      "rssBytes": 38413828096,
      "cpuThreads": 4,
      "totalMs": 37514.78070799931,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    },
    "failure": "This fixture passed but the backend did not pass the complete repeated profile."
  },
  {
    "precision": "q4",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": false,
      "expected": "Unknown"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "loadMs": 34819.97774999763,
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size = 18552.93 MiB\nload_tensors:   CPU_REPACK model buffer size = 18517.69 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 0,
      "totalLayers": 41,
      "inferenceMs": 2411.2892499979353,
      "durationMs": 2419.3728329992155,
      "tokensPerSecond": 0.8294318070723836,
      "inputTokens": 74,
      "outputTokens": 2,
      "outputLimitReached": false,
      "rssBytes": 38546784256,
      "cpuThreads": 4,
      "totalMs": 37239.35058299685,
      "grade": "slow",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "q4",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "plainParity": true,
      "passed": false,
      "expected": "Paris"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "loadMs": 19607.6884590002,
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size = 18552.93 MiB\nload_tensors:   CPU_REPACK model buffer size = 18517.69 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 0,
      "totalLayers": 41,
      "inferenceMs": 7619.532499993511,
      "durationMs": 7627.649542002473,
      "tokensPerSecond": 0.2624832953992523,
      "inputTokens": 63,
      "outputTokens": 2,
      "outputLimitReached": false,
      "rssBytes": 38995722240,
      "cpuThreads": 1,
      "totalMs": 27235.338001002674,
      "grade": "slow",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "q4",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": false,
      "expected": "56"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "loadMs": 19607.6884590002,
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size = 18552.93 MiB\nload_tensors:   CPU_REPACK model buffer size = 18517.69 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 0,
      "totalLayers": 41,
      "inferenceMs": 7651.7220420064405,
      "durationMs": 7660.638916997414,
      "tokensPerSecond": 0.26137907114508285,
      "inputTokens": 63,
      "outputTokens": 2,
      "outputLimitReached": false,
      "rssBytes": 38997016576,
      "cpuThreads": 1,
      "totalMs": 27268.327375997615,
      "grade": "slow",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "q4",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": false,
      "expected": "chat"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "loadMs": 19607.6884590002,
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size = 18552.93 MiB\nload_tensors:   CPU_REPACK model buffer size = 18517.69 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 0,
      "totalLayers": 41,
      "inferenceMs": 8071.8802500050515,
      "durationMs": 8080.053291996592,
      "tokensPerSecond": 0.24777374515668124,
      "inputTokens": 68,
      "outputTokens": 2,
      "outputLimitReached": false,
      "rssBytes": 38732054528,
      "cpuThreads": 1,
      "totalMs": 27687.741750996793,
      "grade": "slow",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "q4",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": false,
      "expected": "hola"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "loadMs": 19607.6884590002,
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size = 18552.93 MiB\nload_tensors:   CPU_REPACK model buffer size = 18517.69 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 0,
      "totalLayers": 41,
      "inferenceMs": 8437.868083994545,
      "durationMs": 8446.016834001057,
      "tokensPerSecond": 0.355540045202956,
      "inputTokens": 69,
      "outputTokens": 3,
      "outputLimitReached": false,
      "rssBytes": 38664617984,
      "cpuThreads": 1,
      "totalMs": 28053.705293001258,
      "grade": "slow",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "q4",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": false,
      "expected": "2026-11-04"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "loadMs": 19607.6884590002,
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size = 18552.93 MiB\nload_tensors:   CPU_REPACK model buffer size = 18517.69 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 0,
      "totalLayers": 41,
      "inferenceMs": 9485.951374997967,
      "durationMs": 9494.039042001532,
      "tokensPerSecond": 0.7379333630625426,
      "inputTokens": 72,
      "outputTokens": 7,
      "outputLimitReached": false,
      "rssBytes": 38755663872,
      "cpuThreads": 1,
      "totalMs": 29101.727501001733,
      "grade": "slow",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "q4",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": false,
      "expected": "Unknown"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "6522095069a86fc3186b8632850e4373dd850cac",
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "loadMs": 19607.6884590002,
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size = 18552.93 MiB\nload_tensors:   CPU_REPACK model buffer size = 18517.69 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 0,
      "totalLayers": 41,
      "inferenceMs": 8769.063250001636,
      "durationMs": 8776.766625000164,
      "tokensPerSecond": 0.22807453236235087,
      "inputTokens": 74,
      "outputTokens": 2,
      "outputLimitReached": false,
      "rssBytes": 38781222912,
      "cpuThreads": 1,
      "totalMs": 28384.455084000365,
      "grade": "slow",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  }
]

Granite 4.0 Hybrid Tiny

36 configurations tested
Passed18
Failed18
TestedNov 7, 2026
Runtimellama.cpp 0.3.36

Quality criteria

No explicit quality gate recorded. Review per-case evidence; full certification is pending.

PrecisionDeviceResultPlatformInference / load
Q4mpsAcceleratedPassedVery fastmacos · aarch6494 ms / 1236 ms
Q4mpsAcceleratedPassedVery fastmacos · aarch6496 ms / 925 ms
Q4mpsAcceleratedPassedVery fastmacos · aarch6497 ms / 1116 ms
Q4mpsAcceleratedPassedVery fastmacos · aarch6497 ms / 1236 ms
Q4mpsAcceleratedPassedVery fastmacos · aarch6497 ms / 1236 ms
Q4mpsAcceleratedPassedVery fastmacos · aarch6499 ms / 925 ms
Q4mpsAcceleratedPassedFastmacos · aarch64165 ms / 1116 ms
Q4mpsAcceleratedPassedFastmacos · aarch64169 ms / 1236 ms
Q4mpsAcceleratedPassedFastmacos · aarch64170 ms / 925 ms
Q4mpsAcceleratedPassedModeratemacos · aarch64100 ms / 1236 ms
Q4mpsAcceleratedPassedModeratemacos · aarch64100 ms / 925 ms
Q4mpsAcceleratedPassedModeratemacos · aarch64102 ms / 925 ms
Q4mpsAcceleratedPassedModeratemacos · aarch64104 ms / 1116 ms
Q4mpsAcceleratedPassedModeratemacos · aarch64108 ms / 1236 ms
Q4mpsAcceleratedPassedModeratemacos · aarch64109 ms / 1116 ms
Q4mpsAcceleratedPassedModeratemacos · aarch64126 ms / 925 ms
Q4mpsAcceleratedPassedModeratemacos · aarch64217 ms / 1116 ms
Q4mpsAcceleratedPassedModeratemacos · aarch64280 ms / 1116 ms
Q4cpuFailed—macos · aarch64452 ms / 5334 ms
Q4cpuFailed—macos · aarch64429 ms / 5334 ms
Q4cpuFailed—macos · aarch64419 ms / 5334 ms
Q4cpuFailed—macos · aarch64409 ms / 5334 ms
Q4cpuFailed—macos · aarch64551 ms / 5334 ms
Q4cpuFailed—macos · aarch64530 ms / 5334 ms
Q4cpuFailed—macos · aarch641444 ms / 4297 ms
Q4cpuFailed—macos · aarch641737 ms / 4297 ms
Q4cpuFailed—macos · aarch641108 ms / 4297 ms
Q4cpuFailed—macos · aarch64915 ms / 4297 ms
Q4cpuFailed—macos · aarch641209 ms / 4297 ms
Q4cpuFailed—macos · aarch64658 ms / 4297 ms
Q4cpuFailed—macos · aarch641374 ms / 4240 ms
Q4cpuFailed—macos · aarch641358 ms / 4240 ms
Q4cpuFailed—macos · aarch641435 ms / 4240 ms
Q4cpuFailed—macos · aarch641456 ms / 4240 ms
Q4cpuFailed—macos · aarch641682 ms / 4240 ms
Q4cpuFailed—macos · aarch641633 ms / 4240 ms
Test setup & sample details
Quality gate
No explicit quality gate recorded. Review the per-case quality evidence; full certification is pending.
Tested
2026-11-07T12:40:44.259304+00:00
Environment
{ "os": "macos", "arch": "aarch64", "hardware": "Apple M3 Max", "osVersion": "27.2" }
Runtime
llama.cpp 0.3.36
Artifact revision
08d5a8a9741dd5c1a95d2d39e25253226aa1464e
Fixture
{
  "file": "model-samples/desktop/granite4/granite-4.0-h-tiny-native-capital.json",
  "size": 244,
  "sha1": "9013b4f5a71cd1280d78b3dfa751ead1186d071f",
  "sha256": "8ff34789fa2a0f5bbef2080057f4b36376d3c73ea64baccefb6caec96eafbd68",
  "url": "https://assets.localai.tools/samples/v1/files/8ff34789fa2a0f5bbef2080057f4b36376d3c73ea64baccefb6caec96eafbd68/granite-4.0-h-tiny-native-capital.json"
}
Rerun when
Artifact/runtime revision; Input/generation processing; Fixture/quality gates; OS/backend behavior
Per-case quality & failure details
[
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "plainParity": true,
      "passed": true,
      "expected": "Paris"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "loadMs": 1116.3811659935163,
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer   0 assigned to device MTL0, is_swa = 0\nload_tensors: layer   1 assigned to device MTL0, is_swa = 0\nload_tensors: layer   2 assigned to device MTL0, is_swa = 0\nload_tensors: layer   3 assigned to device MTL0, is_swa = 0\nload_tensors: layer   4 assigned to device MTL0, is_swa = 0\nload_tensors: layer   5 assigned to device MTL0, is_swa = 0\nload_tensors: layer   6 assigned to device MTL0, is_swa = 0\nload_tensors: layer   7 assigned to device MTL0, is_swa = 0\nload_tensors: layer   8 assigned to device MTL0, is_swa = 0\nload_tensors: layer   9 assigned to device MTL0, is_swa = 0\nload_tensors: layer  10 assigned to device MTL0, is_swa = 0\nload_tensors: layer  11 assigned to device MTL0, is_swa = 0\nload_tensors: layer  12 assigned to device MTL0, is_swa = 0\nload_tensors: layer  13 assigned to device MTL0, is_swa = 0\nload_tensors: layer  14 assigned to device MTL0, is_swa = 0\nload_tensors: layer  15 assigned to device MTL0, is_swa = 0\nload_tensors: layer  16 assigned to device MTL0, is_swa = 0\nload_tensors: layer  17 assigned to device MTL0, is_swa = 0\nload_tensors: layer  18 assigned to device MTL0, is_swa = 0\nload_tensors: layer  19 assigned to device MTL0, is_swa = 0\nload_tensors: layer  20 assigned to device MTL0, is_swa = 0\nload_tensors: layer  21 assigned to device MTL0, is_swa = 0\nload_tensors: layer  22 assigned to device MTL0, is_swa = 0\nload_tensors: layer  23 assigned to device MTL0, is_swa = 0\nload_tensors: layer  24 assigned to device MTL0, is_swa = 0\nload_tensors: layer  25 assigned to device MTL0, is_swa = 0\nload_tensors: layer  26 assigned to device MTL0, is_swa = 0\nload_tensors: layer  27 assigned to device MTL0, is_swa = 0\nload_tensors: layer  28 assigned to device MTL0, is_swa = 0\nload_tensors: layer  29 assigned to device MTL0, is_swa = 0\nload_tensors: layer  30 assigned to device MTL0, is_swa = 0\nload_tensors: layer  31 assigned to device MTL0, is_swa = 0\nload_tensors: layer  32 assigned to device MTL0, is_swa = 0\nload_tensors: layer  33 assigned to device MTL0, is_swa = 0\nload_tensors: layer  34 assigned to device MTL0, is_swa = 0\nload_tensors: layer  35 assigned to device MTL0, is_swa = 0\nload_tensors: layer  36 assigned to device MTL0, is_swa = 0\nload_tensors: layer  37 assigned to device MTL0, is_swa = 0\nload_tensors: layer  38 assigned to device MTL0, is_swa = 0\nload_tensors: layer  39 assigned to device MTL0, is_swa = 0\nload_tensors: layer  40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =   120.59 MiB\nload_tensors:  MTL0_Mapped model buffer size =  4031.55 MiB\nllama_kv_cache: layer   5: dev = MTL0\nllama_kv_cache: layer  15: dev = MTL0\nllama_kv_cache: layer  25: dev = MTL0\nllama_kv_cache: layer  35: dev = MTL0\nllama_kv_cache:       MTL0 KV buffer size =    64.00 MiB\nllama_memory_recurrent, layer   0: dev = MTL0\nllama_memory_recurrent, layer   1: dev = MTL0\nllama_memory_recurrent, layer   2: dev = MTL0\nllama_memory_recurrent, layer   3: dev = MTL0\nllama_memory_recurrent, layer   4: dev = MTL0\nllama_memory_recurrent, layer   6: dev = MTL0\nllama_memory_recurrent, layer   7: dev = MTL0\nllama_memory_recurrent, layer   8: dev = MTL0\nllama_memory_recurrent, layer   9: dev = MTL0\nllama_memory_recurrent, layer  10: dev = MTL0\nllama_memory_recurrent, layer  11: dev = MTL0\nllama_memory_recurrent, layer  12: dev = MTL0\nllama_memory_recurrent, layer  13: dev = MTL0\nllama_memory_recurrent, layer  14: dev = MTL0\nllama_memory_recurrent, layer  16: dev = MTL0\nllama_memory_recurrent, layer  17: dev = MTL0\nllama_memory_recurrent, layer  18: dev = MTL0\nllama_memory_recurrent, layer  19: dev = MTL0\nllama_memory_recurrent, layer  20: dev = MTL0\nllama_memory_recurrent, layer  21: dev = MTL0\nllama_memory_recurrent, layer  22: dev = MTL0\nllama_memory_recurrent, layer  23: dev = MTL0\nllama_memory_recurrent, layer  24: dev = MTL0\nllama_memory_recurrent, layer  26: dev = MTL0\nllama_memory_recurrent, layer  27: dev = MTL0\nllama_memory_recurrent, layer  28: dev = MTL0\nllama_memory_recurrent, layer  29: dev = MTL0\nllama_memory_recurrent, layer  30: dev = MTL0\nllama_memory_recurrent, layer  31: dev = MTL0\nllama_memory_recurrent, layer  32: dev = MTL0\nllama_memory_recurrent, layer  33: dev = MTL0\nllama_memory_recurrent, layer  34: dev = MTL0\nllama_memory_recurrent, layer  36: dev = MTL0\nllama_memory_recurrent, layer  37: dev = MTL0\nllama_memory_recurrent, layer  38: dev = MTL0\nllama_memory_recurrent, layer  39: dev = MTL0\nllama_memory_recurrent:       MTL0 RS buffer size =    55.37 MiB\nsched_reserve:       MTL0 compute buffer size =   123.63 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 41,
      "totalLayers": 41,
      "inferenceMs": 279.5718749985099,
      "durationMs": 289.8012080040644,
      "tokensPerSecond": 7.1537954238446195,
      "inputTokens": 63,
      "outputTokens": 2,
      "outputLimitReached": false,
      "rssBytes": 4529979392,
      "cpuThreads": 4,
      "totalMs": 1406.1823739975807,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "56"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "loadMs": 1116.3811659935163,
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer   0 assigned to device MTL0, is_swa = 0\nload_tensors: layer   1 assigned to device MTL0, is_swa = 0\nload_tensors: layer   2 assigned to device MTL0, is_swa = 0\nload_tensors: layer   3 assigned to device MTL0, is_swa = 0\nload_tensors: layer   4 assigned to device MTL0, is_swa = 0\nload_tensors: layer   5 assigned to device MTL0, is_swa = 0\nload_tensors: layer   6 assigned to device MTL0, is_swa = 0\nload_tensors: layer   7 assigned to device MTL0, is_swa = 0\nload_tensors: layer   8 assigned to device MTL0, is_swa = 0\nload_tensors: layer   9 assigned to device MTL0, is_swa = 0\nload_tensors: layer  10 assigned to device MTL0, is_swa = 0\nload_tensors: layer  11 assigned to device MTL0, is_swa = 0\nload_tensors: layer  12 assigned to device MTL0, is_swa = 0\nload_tensors: layer  13 assigned to device MTL0, is_swa = 0\nload_tensors: layer  14 assigned to device MTL0, is_swa = 0\nload_tensors: layer  15 assigned to device MTL0, is_swa = 0\nload_tensors: layer  16 assigned to device MTL0, is_swa = 0\nload_tensors: layer  17 assigned to device MTL0, is_swa = 0\nload_tensors: layer  18 assigned to device MTL0, is_swa = 0\nload_tensors: layer  19 assigned to device MTL0, is_swa = 0\nload_tensors: layer  20 assigned to device MTL0, is_swa = 0\nload_tensors: layer  21 assigned to device MTL0, is_swa = 0\nload_tensors: layer  22 assigned to device MTL0, is_swa = 0\nload_tensors: layer  23 assigned to device MTL0, is_swa = 0\nload_tensors: layer  24 assigned to device MTL0, is_swa = 0\nload_tensors: layer  25 assigned to device MTL0, is_swa = 0\nload_tensors: layer  26 assigned to device MTL0, is_swa = 0\nload_tensors: layer  27 assigned to device MTL0, is_swa = 0\nload_tensors: layer  28 assigned to device MTL0, is_swa = 0\nload_tensors: layer  29 assigned to device MTL0, is_swa = 0\nload_tensors: layer  30 assigned to device MTL0, is_swa = 0\nload_tensors: layer  31 assigned to device MTL0, is_swa = 0\nload_tensors: layer  32 assigned to device MTL0, is_swa = 0\nload_tensors: layer  33 assigned to device MTL0, is_swa = 0\nload_tensors: layer  34 assigned to device MTL0, is_swa = 0\nload_tensors: layer  35 assigned to device MTL0, is_swa = 0\nload_tensors: layer  36 assigned to device MTL0, is_swa = 0\nload_tensors: layer  37 assigned to device MTL0, is_swa = 0\nload_tensors: layer  38 assigned to device MTL0, is_swa = 0\nload_tensors: layer  39 assigned to device MTL0, is_swa = 0\nload_tensors: layer  40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =   120.59 MiB\nload_tensors:  MTL0_Mapped model buffer size =  4031.55 MiB\nllama_kv_cache: layer   5: dev = MTL0\nllama_kv_cache: layer  15: dev = MTL0\nllama_kv_cache: layer  25: dev = MTL0\nllama_kv_cache: layer  35: dev = MTL0\nllama_kv_cache:       MTL0 KV buffer size =    64.00 MiB\nllama_memory_recurrent, layer   0: dev = MTL0\nllama_memory_recurrent, layer   1: dev = MTL0\nllama_memory_recurrent, layer   2: dev = MTL0\nllama_memory_recurrent, layer   3: dev = MTL0\nllama_memory_recurrent, layer   4: dev = MTL0\nllama_memory_recurrent, layer   6: dev = MTL0\nllama_memory_recurrent, layer   7: dev = MTL0\nllama_memory_recurrent, layer   8: dev = MTL0\nllama_memory_recurrent, layer   9: dev = MTL0\nllama_memory_recurrent, layer  10: dev = MTL0\nllama_memory_recurrent, layer  11: dev = MTL0\nllama_memory_recurrent, layer  12: dev = MTL0\nllama_memory_recurrent, layer  13: dev = MTL0\nllama_memory_recurrent, layer  14: dev = MTL0\nllama_memory_recurrent, layer  16: dev = MTL0\nllama_memory_recurrent, layer  17: dev = MTL0\nllama_memory_recurrent, layer  18: dev = MTL0\nllama_memory_recurrent, layer  19: dev = MTL0\nllama_memory_recurrent, layer  20: dev = MTL0\nllama_memory_recurrent, layer  21: dev = MTL0\nllama_memory_recurrent, layer  22: dev = MTL0\nllama_memory_recurrent, layer  23: dev = MTL0\nllama_memory_recurrent, layer  24: dev = MTL0\nllama_memory_recurrent, layer  26: dev = MTL0\nllama_memory_recurrent, layer  27: dev = MTL0\nllama_memory_recurrent, layer  28: dev = MTL0\nllama_memory_recurrent, layer  29: dev = MTL0\nllama_memory_recurrent, layer  30: dev = MTL0\nllama_memory_recurrent, layer  31: dev = MTL0\nllama_memory_recurrent, layer  32: dev = MTL0\nllama_memory_recurrent, layer  33: dev = MTL0\nllama_memory_recurrent, layer  34: dev = MTL0\nllama_memory_recurrent, layer  36: dev = MTL0\nllama_memory_recurrent, layer  37: dev = MTL0\nllama_memory_recurrent, layer  38: dev = MTL0\nllama_memory_recurrent, layer  39: dev = MTL0\nllama_memory_recurrent:       MTL0 RS buffer size =    55.37 MiB\nsched_reserve:       MTL0 compute buffer size =   123.63 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 41,
      "totalLayers": 41,
      "inferenceMs": 109.4386660042801,
      "durationMs": 134.49054199736565,
      "tokensPerSecond": 18.2750765613479,
      "inputTokens": 63,
      "outputTokens": 2,
      "outputLimitReached": false,
      "rssBytes": 4531257344,
      "cpuThreads": 4,
      "totalMs": 1250.871707990882,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "chat"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "loadMs": 1116.3811659935163,
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer   0 assigned to device MTL0, is_swa = 0\nload_tensors: layer   1 assigned to device MTL0, is_swa = 0\nload_tensors: layer   2 assigned to device MTL0, is_swa = 0\nload_tensors: layer   3 assigned to device MTL0, is_swa = 0\nload_tensors: layer   4 assigned to device MTL0, is_swa = 0\nload_tensors: layer   5 assigned to device MTL0, is_swa = 0\nload_tensors: layer   6 assigned to device MTL0, is_swa = 0\nload_tensors: layer   7 assigned to device MTL0, is_swa = 0\nload_tensors: layer   8 assigned to device MTL0, is_swa = 0\nload_tensors: layer   9 assigned to device MTL0, is_swa = 0\nload_tensors: layer  10 assigned to device MTL0, is_swa = 0\nload_tensors: layer  11 assigned to device MTL0, is_swa = 0\nload_tensors: layer  12 assigned to device MTL0, is_swa = 0\nload_tensors: layer  13 assigned to device MTL0, is_swa = 0\nload_tensors: layer  14 assigned to device MTL0, is_swa = 0\nload_tensors: layer  15 assigned to device MTL0, is_swa = 0\nload_tensors: layer  16 assigned to device MTL0, is_swa = 0\nload_tensors: layer  17 assigned to device MTL0, is_swa = 0\nload_tensors: layer  18 assigned to device MTL0, is_swa = 0\nload_tensors: layer  19 assigned to device MTL0, is_swa = 0\nload_tensors: layer  20 assigned to device MTL0, is_swa = 0\nload_tensors: layer  21 assigned to device MTL0, is_swa = 0\nload_tensors: layer  22 assigned to device MTL0, is_swa = 0\nload_tensors: layer  23 assigned to device MTL0, is_swa = 0\nload_tensors: layer  24 assigned to device MTL0, is_swa = 0\nload_tensors: layer  25 assigned to device MTL0, is_swa = 0\nload_tensors: layer  26 assigned to device MTL0, is_swa = 0\nload_tensors: layer  27 assigned to device MTL0, is_swa = 0\nload_tensors: layer  28 assigned to device MTL0, is_swa = 0\nload_tensors: layer  29 assigned to device MTL0, is_swa = 0\nload_tensors: layer  30 assigned to device MTL0, is_swa = 0\nload_tensors: layer  31 assigned to device MTL0, is_swa = 0\nload_tensors: layer  32 assigned to device MTL0, is_swa = 0\nload_tensors: layer  33 assigned to device MTL0, is_swa = 0\nload_tensors: layer  34 assigned to device MTL0, is_swa = 0\nload_tensors: layer  35 assigned to device MTL0, is_swa = 0\nload_tensors: layer  36 assigned to device MTL0, is_swa = 0\nload_tensors: layer  37 assigned to device MTL0, is_swa = 0\nload_tensors: layer  38 assigned to device MTL0, is_swa = 0\nload_tensors: layer  39 assigned to device MTL0, is_swa = 0\nload_tensors: layer  40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =   120.59 MiB\nload_tensors:  MTL0_Mapped model buffer size =  4031.55 MiB\nllama_kv_cache: layer   5: dev = MTL0\nllama_kv_cache: layer  15: dev = MTL0\nllama_kv_cache: layer  25: dev = MTL0\nllama_kv_cache: layer  35: dev = MTL0\nllama_kv_cache:       MTL0 KV buffer size =    64.00 MiB\nllama_memory_recurrent, layer   0: dev = MTL0\nllama_memory_recurrent, layer   1: dev = MTL0\nllama_memory_recurrent, layer   2: dev = MTL0\nllama_memory_recurrent, layer   3: dev = MTL0\nllama_memory_recurrent, layer   4: dev = MTL0\nllama_memory_recurrent, layer   6: dev = MTL0\nllama_memory_recurrent, layer   7: dev = MTL0\nllama_memory_recurrent, layer   8: dev = MTL0\nllama_memory_recurrent, layer   9: dev = MTL0\nllama_memory_recurrent, layer  10: dev = MTL0\nllama_memory_recurrent, layer  11: dev = MTL0\nllama_memory_recurrent, layer  12: dev = MTL0\nllama_memory_recurrent, layer  13: dev = MTL0\nllama_memory_recurrent, layer  14: dev = MTL0\nllama_memory_recurrent, layer  16: dev = MTL0\nllama_memory_recurrent, layer  17: dev = MTL0\nllama_memory_recurrent, layer  18: dev = MTL0\nllama_memory_recurrent, layer  19: dev = MTL0\nllama_memory_recurrent, layer  20: dev = MTL0\nllama_memory_recurrent, layer  21: dev = MTL0\nllama_memory_recurrent, layer  22: dev = MTL0\nllama_memory_recurrent, layer  23: dev = MTL0\nllama_memory_recurrent, layer  24: dev = MTL0\nllama_memory_recurrent, layer  26: dev = MTL0\nllama_memory_recurrent, layer  27: dev = MTL0\nllama_memory_recurrent, layer  28: dev = MTL0\nllama_memory_recurrent, layer  29: dev = MTL0\nllama_memory_recurrent, layer  30: dev = MTL0\nllama_memory_recurrent, layer  31: dev = MTL0\nllama_memory_recurrent, layer  32: dev = MTL0\nllama_memory_recurrent, layer  33: dev = MTL0\nllama_memory_recurrent, layer  34: dev = MTL0\nllama_memory_recurrent, layer  36: dev = MTL0\nllama_memory_recurrent, layer  37: dev = MTL0\nllama_memory_recurrent, layer  38: dev = MTL0\nllama_memory_recurrent, layer  39: dev = MTL0\nllama_memory_recurrent:       MTL0 RS buffer size =    55.37 MiB\nsched_reserve:       MTL0 compute buffer size =   123.63 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 41,
      "totalLayers": 41,
      "inferenceMs": 216.89704199525295,
      "durationMs": 233.35537499951897,
      "tokensPerSecond": 9.220964848583654,
      "inputTokens": 68,
      "outputTokens": 2,
      "outputLimitReached": false,
      "rssBytes": 4533026816,
      "cpuThreads": 4,
      "totalMs": 1349.7365409930353,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "hola"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "loadMs": 1116.3811659935163,
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer   0 assigned to device MTL0, is_swa = 0\nload_tensors: layer   1 assigned to device MTL0, is_swa = 0\nload_tensors: layer   2 assigned to device MTL0, is_swa = 0\nload_tensors: layer   3 assigned to device MTL0, is_swa = 0\nload_tensors: layer   4 assigned to device MTL0, is_swa = 0\nload_tensors: layer   5 assigned to device MTL0, is_swa = 0\nload_tensors: layer   6 assigned to device MTL0, is_swa = 0\nload_tensors: layer   7 assigned to device MTL0, is_swa = 0\nload_tensors: layer   8 assigned to device MTL0, is_swa = 0\nload_tensors: layer   9 assigned to device MTL0, is_swa = 0\nload_tensors: layer  10 assigned to device MTL0, is_swa = 0\nload_tensors: layer  11 assigned to device MTL0, is_swa = 0\nload_tensors: layer  12 assigned to device MTL0, is_swa = 0\nload_tensors: layer  13 assigned to device MTL0, is_swa = 0\nload_tensors: layer  14 assigned to device MTL0, is_swa = 0\nload_tensors: layer  15 assigned to device MTL0, is_swa = 0\nload_tensors: layer  16 assigned to device MTL0, is_swa = 0\nload_tensors: layer  17 assigned to device MTL0, is_swa = 0\nload_tensors: layer  18 assigned to device MTL0, is_swa = 0\nload_tensors: layer  19 assigned to device MTL0, is_swa = 0\nload_tensors: layer  20 assigned to device MTL0, is_swa = 0\nload_tensors: layer  21 assigned to device MTL0, is_swa = 0\nload_tensors: layer  22 assigned to device MTL0, is_swa = 0\nload_tensors: layer  23 assigned to device MTL0, is_swa = 0\nload_tensors: layer  24 assigned to device MTL0, is_swa = 0\nload_tensors: layer  25 assigned to device MTL0, is_swa = 0\nload_tensors: layer  26 assigned to device MTL0, is_swa = 0\nload_tensors: layer  27 assigned to device MTL0, is_swa = 0\nload_tensors: layer  28 assigned to device MTL0, is_swa = 0\nload_tensors: layer  29 assigned to device MTL0, is_swa = 0\nload_tensors: layer  30 assigned to device MTL0, is_swa = 0\nload_tensors: layer  31 assigned to device MTL0, is_swa = 0\nload_tensors: layer  32 assigned to device MTL0, is_swa = 0\nload_tensors: layer  33 assigned to device MTL0, is_swa = 0\nload_tensors: layer  34 assigned to device MTL0, is_swa = 0\nload_tensors: layer  35 assigned to device MTL0, is_swa = 0\nload_tensors: layer  36 assigned to device MTL0, is_swa = 0\nload_tensors: layer  37 assigned to device MTL0, is_swa = 0\nload_tensors: layer  38 assigned to device MTL0, is_swa = 0\nload_tensors: layer  39 assigned to device MTL0, is_swa = 0\nload_tensors: layer  40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =   120.59 MiB\nload_tensors:  MTL0_Mapped model buffer size =  4031.55 MiB\nllama_kv_cache: layer   5: dev = MTL0\nllama_kv_cache: layer  15: dev = MTL0\nllama_kv_cache: layer  25: dev = MTL0\nllama_kv_cache: layer  35: dev = MTL0\nllama_kv_cache:       MTL0 KV buffer size =    64.00 MiB\nllama_memory_recurrent, layer   0: dev = MTL0\nllama_memory_recurrent, layer   1: dev = MTL0\nllama_memory_recurrent, layer   2: dev = MTL0\nllama_memory_recurrent, layer   3: dev = MTL0\nllama_memory_recurrent, layer   4: dev = MTL0\nllama_memory_recurrent, layer   6: dev = MTL0\nllama_memory_recurrent, layer   7: dev = MTL0\nllama_memory_recurrent, layer   8: dev = MTL0\nllama_memory_recurrent, layer   9: dev = MTL0\nllama_memory_recurrent, layer  10: dev = MTL0\nllama_memory_recurrent, layer  11: dev = MTL0\nllama_memory_recurrent, layer  12: dev = MTL0\nllama_memory_recurrent, layer  13: dev = MTL0\nllama_memory_recurrent, layer  14: dev = MTL0\nllama_memory_recurrent, layer  16: dev = MTL0\nllama_memory_recurrent, layer  17: dev = MTL0\nllama_memory_recurrent, layer  18: dev = MTL0\nllama_memory_recurrent, layer  19: dev = MTL0\nllama_memory_recurrent, layer  20: dev = MTL0\nllama_memory_recurrent, layer  21: dev = MTL0\nllama_memory_recurrent, layer  22: dev = MTL0\nllama_memory_recurrent, layer  23: dev = MTL0\nllama_memory_recurrent, layer  24: dev = MTL0\nllama_memory_recurrent, layer  26: dev = MTL0\nllama_memory_recurrent, layer  27: dev = MTL0\nllama_memory_recurrent, layer  28: dev = MTL0\nllama_memory_recurrent, layer  29: dev = MTL0\nllama_memory_recurrent, layer  30: dev = MTL0\nllama_memory_recurrent, layer  31: dev = MTL0\nllama_memory_recurrent, layer  32: dev = MTL0\nllama_memory_recurrent, layer  33: dev = MTL0\nllama_memory_recurrent, layer  34: dev = MTL0\nllama_memory_recurrent, layer  36: dev = MTL0\nllama_memory_recurrent, layer  37: dev = MTL0\nllama_memory_recurrent, layer  38: dev = MTL0\nllama_memory_recurrent, layer  39: dev = MTL0\nllama_memory_recurrent:       MTL0 RS buffer size =    55.37 MiB\nsched_reserve:       MTL0 compute buffer size =   123.63 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 41,
      "totalLayers": 41,
      "inferenceMs": 96.98383299837587,
      "durationMs": 111.57320799975423,
      "tokensPerSecond": 20.62199377120404,
      "inputTokens": 69,
      "outputTokens": 2,
      "outputLimitReached": false,
      "rssBytes": 4533059584,
      "cpuThreads": 4,
      "totalMs": 1227.9543739932706,
      "grade": "fast",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "2026-11-04"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "loadMs": 1116.3811659935163,
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer   0 assigned to device MTL0, is_swa = 0\nload_tensors: layer   1 assigned to device MTL0, is_swa = 0\nload_tensors: layer   2 assigned to device MTL0, is_swa = 0\nload_tensors: layer   3 assigned to device MTL0, is_swa = 0\nload_tensors: layer   4 assigned to device MTL0, is_swa = 0\nload_tensors: layer   5 assigned to device MTL0, is_swa = 0\nload_tensors: layer   6 assigned to device MTL0, is_swa = 0\nload_tensors: layer   7 assigned to device MTL0, is_swa = 0\nload_tensors: layer   8 assigned to device MTL0, is_swa = 0\nload_tensors: layer   9 assigned to device MTL0, is_swa = 0\nload_tensors: layer  10 assigned to device MTL0, is_swa = 0\nload_tensors: layer  11 assigned to device MTL0, is_swa = 0\nload_tensors: layer  12 assigned to device MTL0, is_swa = 0\nload_tensors: layer  13 assigned to device MTL0, is_swa = 0\nload_tensors: layer  14 assigned to device MTL0, is_swa = 0\nload_tensors: layer  15 assigned to device MTL0, is_swa = 0\nload_tensors: layer  16 assigned to device MTL0, is_swa = 0\nload_tensors: layer  17 assigned to device MTL0, is_swa = 0\nload_tensors: layer  18 assigned to device MTL0, is_swa = 0\nload_tensors: layer  19 assigned to device MTL0, is_swa = 0\nload_tensors: layer  20 assigned to device MTL0, is_swa = 0\nload_tensors: layer  21 assigned to device MTL0, is_swa = 0\nload_tensors: layer  22 assigned to device MTL0, is_swa = 0\nload_tensors: layer  23 assigned to device MTL0, is_swa = 0\nload_tensors: layer  24 assigned to device MTL0, is_swa = 0\nload_tensors: layer  25 assigned to device MTL0, is_swa = 0\nload_tensors: layer  26 assigned to device MTL0, is_swa = 0\nload_tensors: layer  27 assigned to device MTL0, is_swa = 0\nload_tensors: layer  28 assigned to device MTL0, is_swa = 0\nload_tensors: layer  29 assigned to device MTL0, is_swa = 0\nload_tensors: layer  30 assigned to device MTL0, is_swa = 0\nload_tensors: layer  31 assigned to device MTL0, is_swa = 0\nload_tensors: layer  32 assigned to device MTL0, is_swa = 0\nload_tensors: layer  33 assigned to device MTL0, is_swa = 0\nload_tensors: layer  34 assigned to device MTL0, is_swa = 0\nload_tensors: layer  35 assigned to device MTL0, is_swa = 0\nload_tensors: layer  36 assigned to device MTL0, is_swa = 0\nload_tensors: layer  37 assigned to device MTL0, is_swa = 0\nload_tensors: layer  38 assigned to device MTL0, is_swa = 0\nload_tensors: layer  39 assigned to device MTL0, is_swa = 0\nload_tensors: layer  40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =   120.59 MiB\nload_tensors:  MTL0_Mapped model buffer size =  4031.55 MiB\nllama_kv_cache: layer   5: dev = MTL0\nllama_kv_cache: layer  15: dev = MTL0\nllama_kv_cache: layer  25: dev = MTL0\nllama_kv_cache: layer  35: dev = MTL0\nllama_kv_cache:       MTL0 KV buffer size =    64.00 MiB\nllama_memory_recurrent, layer   0: dev = MTL0\nllama_memory_recurrent, layer   1: dev = MTL0\nllama_memory_recurrent, layer   2: dev = MTL0\nllama_memory_recurrent, layer   3: dev = MTL0\nllama_memory_recurrent, layer   4: dev = MTL0\nllama_memory_recurrent, layer   6: dev = MTL0\nllama_memory_recurrent, layer   7: dev = MTL0\nllama_memory_recurrent, layer   8: dev = MTL0\nllama_memory_recurrent, layer   9: dev = MTL0\nllama_memory_recurrent, layer  10: dev = MTL0\nllama_memory_recurrent, layer  11: dev = MTL0\nllama_memory_recurrent, layer  12: dev = MTL0\nllama_memory_recurrent, layer  13: dev = MTL0\nllama_memory_recurrent, layer  14: dev = MTL0\nllama_memory_recurrent, layer  16: dev = MTL0\nllama_memory_recurrent, layer  17: dev = MTL0\nllama_memory_recurrent, layer  18: dev = MTL0\nllama_memory_recurrent, layer  19: dev = MTL0\nllama_memory_recurrent, layer  20: dev = MTL0\nllama_memory_recurrent, layer  21: dev = MTL0\nllama_memory_recurrent, layer  22: dev = MTL0\nllama_memory_recurrent, layer  23: dev = MTL0\nllama_memory_recurrent, layer  24: dev = MTL0\nllama_memory_recurrent, layer  26: dev = MTL0\nllama_memory_recurrent, layer  27: dev = MTL0\nllama_memory_recurrent, layer  28: dev = MTL0\nllama_memory_recurrent, layer  29: dev = MTL0\nllama_memory_recurrent, layer  30: dev = MTL0\nllama_memory_recurrent, layer  31: dev = MTL0\nllama_memory_recurrent, layer  32: dev = MTL0\nllama_memory_recurrent, layer  33: dev = MTL0\nllama_memory_recurrent, layer  34: dev = MTL0\nllama_memory_recurrent, layer  36: dev = MTL0\nllama_memory_recurrent, layer  37: dev = MTL0\nllama_memory_recurrent, layer  38: dev = MTL0\nllama_memory_recurrent, layer  39: dev = MTL0\nllama_memory_recurrent:       MTL0 RS buffer size =    55.37 MiB\nsched_reserve:       MTL0 compute buffer size =   123.63 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 41,
      "totalLayers": 41,
      "inferenceMs": 165.19404200516874,
      "durationMs": 179.3914159934502,
      "tokensPerSecond": 42.374409603591985,
      "inputTokens": 72,
      "outputTokens": 7,
      "outputLimitReached": false,
      "rssBytes": 4533174272,
      "cpuThreads": 4,
      "totalMs": 1295.7725819869665,
      "grade": "fast",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "Unknown"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "loadMs": 1116.3811659935163,
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer   0 assigned to device MTL0, is_swa = 0\nload_tensors: layer   1 assigned to device MTL0, is_swa = 0\nload_tensors: layer   2 assigned to device MTL0, is_swa = 0\nload_tensors: layer   3 assigned to device MTL0, is_swa = 0\nload_tensors: layer   4 assigned to device MTL0, is_swa = 0\nload_tensors: layer   5 assigned to device MTL0, is_swa = 0\nload_tensors: layer   6 assigned to device MTL0, is_swa = 0\nload_tensors: layer   7 assigned to device MTL0, is_swa = 0\nload_tensors: layer   8 assigned to device MTL0, is_swa = 0\nload_tensors: layer   9 assigned to device MTL0, is_swa = 0\nload_tensors: layer  10 assigned to device MTL0, is_swa = 0\nload_tensors: layer  11 assigned to device MTL0, is_swa = 0\nload_tensors: layer  12 assigned to device MTL0, is_swa = 0\nload_tensors: layer  13 assigned to device MTL0, is_swa = 0\nload_tensors: layer  14 assigned to device MTL0, is_swa = 0\nload_tensors: layer  15 assigned to device MTL0, is_swa = 0\nload_tensors: layer  16 assigned to device MTL0, is_swa = 0\nload_tensors: layer  17 assigned to device MTL0, is_swa = 0\nload_tensors: layer  18 assigned to device MTL0, is_swa = 0\nload_tensors: layer  19 assigned to device MTL0, is_swa = 0\nload_tensors: layer  20 assigned to device MTL0, is_swa = 0\nload_tensors: layer  21 assigned to device MTL0, is_swa = 0\nload_tensors: layer  22 assigned to device MTL0, is_swa = 0\nload_tensors: layer  23 assigned to device MTL0, is_swa = 0\nload_tensors: layer  24 assigned to device MTL0, is_swa = 0\nload_tensors: layer  25 assigned to device MTL0, is_swa = 0\nload_tensors: layer  26 assigned to device MTL0, is_swa = 0\nload_tensors: layer  27 assigned to device MTL0, is_swa = 0\nload_tensors: layer  28 assigned to device MTL0, is_swa = 0\nload_tensors: layer  29 assigned to device MTL0, is_swa = 0\nload_tensors: layer  30 assigned to device MTL0, is_swa = 0\nload_tensors: layer  31 assigned to device MTL0, is_swa = 0\nload_tensors: layer  32 assigned to device MTL0, is_swa = 0\nload_tensors: layer  33 assigned to device MTL0, is_swa = 0\nload_tensors: layer  34 assigned to device MTL0, is_swa = 0\nload_tensors: layer  35 assigned to device MTL0, is_swa = 0\nload_tensors: layer  36 assigned to device MTL0, is_swa = 0\nload_tensors: layer  37 assigned to device MTL0, is_swa = 0\nload_tensors: layer  38 assigned to device MTL0, is_swa = 0\nload_tensors: layer  39 assigned to device MTL0, is_swa = 0\nload_tensors: layer  40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =   120.59 MiB\nload_tensors:  MTL0_Mapped model buffer size =  4031.55 MiB\nllama_kv_cache: layer   5: dev = MTL0\nllama_kv_cache: layer  15: dev = MTL0\nllama_kv_cache: layer  25: dev = MTL0\nllama_kv_cache: layer  35: dev = MTL0\nllama_kv_cache:       MTL0 KV buffer size =    64.00 MiB\nllama_memory_recurrent, layer   0: dev = MTL0\nllama_memory_recurrent, layer   1: dev = MTL0\nllama_memory_recurrent, layer   2: dev = MTL0\nllama_memory_recurrent, layer   3: dev = MTL0\nllama_memory_recurrent, layer   4: dev = MTL0\nllama_memory_recurrent, layer   6: dev = MTL0\nllama_memory_recurrent, layer   7: dev = MTL0\nllama_memory_recurrent, layer   8: dev = MTL0\nllama_memory_recurrent, layer   9: dev = MTL0\nllama_memory_recurrent, layer  10: dev = MTL0\nllama_memory_recurrent, layer  11: dev = MTL0\nllama_memory_recurrent, layer  12: dev = MTL0\nllama_memory_recurrent, layer  13: dev = MTL0\nllama_memory_recurrent, layer  14: dev = MTL0\nllama_memory_recurrent, layer  16: dev = MTL0\nllama_memory_recurrent, layer  17: dev = MTL0\nllama_memory_recurrent, layer  18: dev = MTL0\nllama_memory_recurrent, layer  19: dev = MTL0\nllama_memory_recurrent, layer  20: dev = MTL0\nllama_memory_recurrent, layer  21: dev = MTL0\nllama_memory_recurrent, layer  22: dev = MTL0\nllama_memory_recurrent, layer  23: dev = MTL0\nllama_memory_recurrent, layer  24: dev = MTL0\nllama_memory_recurrent, layer  26: dev = MTL0\nllama_memory_recurrent, layer  27: dev = MTL0\nllama_memory_recurrent, layer  28: dev = MTL0\nllama_memory_recurrent, layer  29: dev = MTL0\nllama_memory_recurrent, layer  30: dev = MTL0\nllama_memory_recurrent, layer  31: dev = MTL0\nllama_memory_recurrent, layer  32: dev = MTL0\nllama_memory_recurrent, layer  33: dev = MTL0\nllama_memory_recurrent, layer  34: dev = MTL0\nllama_memory_recurrent, layer  36: dev = MTL0\nllama_memory_recurrent, layer  37: dev = MTL0\nllama_memory_recurrent, layer  38: dev = MTL0\nllama_memory_recurrent, layer  39: dev = MTL0\nllama_memory_recurrent:       MTL0 RS buffer size =    55.37 MiB\nsched_reserve:       MTL0 compute buffer size =   123.63 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 41,
      "totalLayers": 41,
      "inferenceMs": 104.0853340018657,
      "durationMs": 128.2852499934961,
      "tokensPerSecond": 19.215002950983955,
      "inputTokens": 74,
      "outputTokens": 2,
      "outputLimitReached": false,
      "rssBytes": 4534435840,
      "cpuThreads": 4,
      "totalMs": 1244.6664159870124,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "Paris"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "loadMs": 925.3593329995056,
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer   0 assigned to device MTL0, is_swa = 0\nload_tensors: layer   1 assigned to device MTL0, is_swa = 0\nload_tensors: layer   2 assigned to device MTL0, is_swa = 0\nload_tensors: layer   3 assigned to device MTL0, is_swa = 0\nload_tensors: layer   4 assigned to device MTL0, is_swa = 0\nload_tensors: layer   5 assigned to device MTL0, is_swa = 0\nload_tensors: layer   6 assigned to device MTL0, is_swa = 0\nload_tensors: layer   7 assigned to device MTL0, is_swa = 0\nload_tensors: layer   8 assigned to device MTL0, is_swa = 0\nload_tensors: layer   9 assigned to device MTL0, is_swa = 0\nload_tensors: layer  10 assigned to device MTL0, is_swa = 0\nload_tensors: layer  11 assigned to device MTL0, is_swa = 0\nload_tensors: layer  12 assigned to device MTL0, is_swa = 0\nload_tensors: layer  13 assigned to device MTL0, is_swa = 0\nload_tensors: layer  14 assigned to device MTL0, is_swa = 0\nload_tensors: layer  15 assigned to device MTL0, is_swa = 0\nload_tensors: layer  16 assigned to device MTL0, is_swa = 0\nload_tensors: layer  17 assigned to device MTL0, is_swa = 0\nload_tensors: layer  18 assigned to device MTL0, is_swa = 0\nload_tensors: layer  19 assigned to device MTL0, is_swa = 0\nload_tensors: layer  20 assigned to device MTL0, is_swa = 0\nload_tensors: layer  21 assigned to device MTL0, is_swa = 0\nload_tensors: layer  22 assigned to device MTL0, is_swa = 0\nload_tensors: layer  23 assigned to device MTL0, is_swa = 0\nload_tensors: layer  24 assigned to device MTL0, is_swa = 0\nload_tensors: layer  25 assigned to device MTL0, is_swa = 0\nload_tensors: layer  26 assigned to device MTL0, is_swa = 0\nload_tensors: layer  27 assigned to device MTL0, is_swa = 0\nload_tensors: layer  28 assigned to device MTL0, is_swa = 0\nload_tensors: layer  29 assigned to device MTL0, is_swa = 0\nload_tensors: layer  30 assigned to device MTL0, is_swa = 0\nload_tensors: layer  31 assigned to device MTL0, is_swa = 0\nload_tensors: layer  32 assigned to device MTL0, is_swa = 0\nload_tensors: layer  33 assigned to device MTL0, is_swa = 0\nload_tensors: layer  34 assigned to device MTL0, is_swa = 0\nload_tensors: layer  35 assigned to device MTL0, is_swa = 0\nload_tensors: layer  36 assigned to device MTL0, is_swa = 0\nload_tensors: layer  37 assigned to device MTL0, is_swa = 0\nload_tensors: layer  38 assigned to device MTL0, is_swa = 0\nload_tensors: layer  39 assigned to device MTL0, is_swa = 0\nload_tensors: layer  40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =   120.59 MiB\nload_tensors:  MTL0_Mapped model buffer size =  4031.55 MiB\nllama_kv_cache: layer   5: dev = MTL0\nllama_kv_cache: layer  15: dev = MTL0\nllama_kv_cache: layer  25: dev = MTL0\nllama_kv_cache: layer  35: dev = MTL0\nllama_kv_cache:       MTL0 KV buffer size =    64.00 MiB\nllama_memory_recurrent, layer   0: dev = MTL0\nllama_memory_recurrent, layer   1: dev = MTL0\nllama_memory_recurrent, layer   2: dev = MTL0\nllama_memory_recurrent, layer   3: dev = MTL0\nllama_memory_recurrent, layer   4: dev = MTL0\nllama_memory_recurrent, layer   6: dev = MTL0\nllama_memory_recurrent, layer   7: dev = MTL0\nllama_memory_recurrent, layer   8: dev = MTL0\nllama_memory_recurrent, layer   9: dev = MTL0\nllama_memory_recurrent, layer  10: dev = MTL0\nllama_memory_recurrent, layer  11: dev = MTL0\nllama_memory_recurrent, layer  12: dev = MTL0\nllama_memory_recurrent, layer  13: dev = MTL0\nllama_memory_recurrent, layer  14: dev = MTL0\nllama_memory_recurrent, layer  16: dev = MTL0\nllama_memory_recurrent, layer  17: dev = MTL0\nllama_memory_recurrent, layer  18: dev = MTL0\nllama_memory_recurrent, layer  19: dev = MTL0\nllama_memory_recurrent, layer  20: dev = MTL0\nllama_memory_recurrent, layer  21: dev = MTL0\nllama_memory_recurrent, layer  22: dev = MTL0\nllama_memory_recurrent, layer  23: dev = MTL0\nllama_memory_recurrent, layer  24: dev = MTL0\nllama_memory_recurrent, layer  26: dev = MTL0\nllama_memory_recurrent, layer  27: dev = MTL0\nllama_memory_recurrent, layer  28: dev = MTL0\nllama_memory_recurrent, layer  29: dev = MTL0\nllama_memory_recurrent, layer  30: dev = MTL0\nllama_memory_recurrent, layer  31: dev = MTL0\nllama_memory_recurrent, layer  32: dev = MTL0\nllama_memory_recurrent, layer  33: dev = MTL0\nllama_memory_recurrent, layer  34: dev = MTL0\nllama_memory_recurrent, layer  36: dev = MTL0\nllama_memory_recurrent, layer  37: dev = MTL0\nllama_memory_recurrent, layer  38: dev = MTL0\nllama_memory_recurrent, layer  39: dev = MTL0\nllama_memory_recurrent:       MTL0 RS buffer size =    55.37 MiB\nsched_reserve:       MTL0 compute buffer size =   123.63 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 41,
      "totalLayers": 41,
      "inferenceMs": 126.38304200663697,
      "durationMs": 171.15341699536657,
      "tokensPerSecond": 15.82490790097433,
      "inputTokens": 63,
      "outputTokens": 2,
      "outputLimitReached": false,
      "rssBytes": 4530241536,
      "cpuThreads": 4,
      "totalMs": 1096.5127499948721,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "56"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "loadMs": 925.3593329995056,
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer   0 assigned to device MTL0, is_swa = 0\nload_tensors: layer   1 assigned to device MTL0, is_swa = 0\nload_tensors: layer   2 assigned to device MTL0, is_swa = 0\nload_tensors: layer   3 assigned to device MTL0, is_swa = 0\nload_tensors: layer   4 assigned to device MTL0, is_swa = 0\nload_tensors: layer   5 assigned to device MTL0, is_swa = 0\nload_tensors: layer   6 assigned to device MTL0, is_swa = 0\nload_tensors: layer   7 assigned to device MTL0, is_swa = 0\nload_tensors: layer   8 assigned to device MTL0, is_swa = 0\nload_tensors: layer   9 assigned to device MTL0, is_swa = 0\nload_tensors: layer  10 assigned to device MTL0, is_swa = 0\nload_tensors: layer  11 assigned to device MTL0, is_swa = 0\nload_tensors: layer  12 assigned to device MTL0, is_swa = 0\nload_tensors: layer  13 assigned to device MTL0, is_swa = 0\nload_tensors: layer  14 assigned to device MTL0, is_swa = 0\nload_tensors: layer  15 assigned to device MTL0, is_swa = 0\nload_tensors: layer  16 assigned to device MTL0, is_swa = 0\nload_tensors: layer  17 assigned to device MTL0, is_swa = 0\nload_tensors: layer  18 assigned to device MTL0, is_swa = 0\nload_tensors: layer  19 assigned to device MTL0, is_swa = 0\nload_tensors: layer  20 assigned to device MTL0, is_swa = 0\nload_tensors: layer  21 assigned to device MTL0, is_swa = 0\nload_tensors: layer  22 assigned to device MTL0, is_swa = 0\nload_tensors: layer  23 assigned to device MTL0, is_swa = 0\nload_tensors: layer  24 assigned to device MTL0, is_swa = 0\nload_tensors: layer  25 assigned to device MTL0, is_swa = 0\nload_tensors: layer  26 assigned to device MTL0, is_swa = 0\nload_tensors: layer  27 assigned to device MTL0, is_swa = 0\nload_tensors: layer  28 assigned to device MTL0, is_swa = 0\nload_tensors: layer  29 assigned to device MTL0, is_swa = 0\nload_tensors: layer  30 assigned to device MTL0, is_swa = 0\nload_tensors: layer  31 assigned to device MTL0, is_swa = 0\nload_tensors: layer  32 assigned to device MTL0, is_swa = 0\nload_tensors: layer  33 assigned to device MTL0, is_swa = 0\nload_tensors: layer  34 assigned to device MTL0, is_swa = 0\nload_tensors: layer  35 assigned to device MTL0, is_swa = 0\nload_tensors: layer  36 assigned to device MTL0, is_swa = 0\nload_tensors: layer  37 assigned to device MTL0, is_swa = 0\nload_tensors: layer  38 assigned to device MTL0, is_swa = 0\nload_tensors: layer  39 assigned to device MTL0, is_swa = 0\nload_tensors: layer  40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =   120.59 MiB\nload_tensors:  MTL0_Mapped model buffer size =  4031.55 MiB\nllama_kv_cache: layer   5: dev = MTL0\nllama_kv_cache: layer  15: dev = MTL0\nllama_kv_cache: layer  25: dev = MTL0\nllama_kv_cache: layer  35: dev = MTL0\nllama_kv_cache:       MTL0 KV buffer size =    64.00 MiB\nllama_memory_recurrent, layer   0: dev = MTL0\nllama_memory_recurrent, layer   1: dev = MTL0\nllama_memory_recurrent, layer   2: dev = MTL0\nllama_memory_recurrent, layer   3: dev = MTL0\nllama_memory_recurrent, layer   4: dev = MTL0\nllama_memory_recurrent, layer   6: dev = MTL0\nllama_memory_recurrent, layer   7: dev = MTL0\nllama_memory_recurrent, layer   8: dev = MTL0\nllama_memory_recurrent, layer   9: dev = MTL0\nllama_memory_recurrent, layer  10: dev = MTL0\nllama_memory_recurrent, layer  11: dev = MTL0\nllama_memory_recurrent, layer  12: dev = MTL0\nllama_memory_recurrent, layer  13: dev = MTL0\nllama_memory_recurrent, layer  14: dev = MTL0\nllama_memory_recurrent, layer  16: dev = MTL0\nllama_memory_recurrent, layer  17: dev = MTL0\nllama_memory_recurrent, layer  18: dev = MTL0\nllama_memory_recurrent, layer  19: dev = MTL0\nllama_memory_recurrent, layer  20: dev = MTL0\nllama_memory_recurrent, layer  21: dev = MTL0\nllama_memory_recurrent, layer  22: dev = MTL0\nllama_memory_recurrent, layer  23: dev = MTL0\nllama_memory_recurrent, layer  24: dev = MTL0\nllama_memory_recurrent, layer  26: dev = MTL0\nllama_memory_recurrent, layer  27: dev = MTL0\nllama_memory_recurrent, layer  28: dev = MTL0\nllama_memory_recurrent, layer  29: dev = MTL0\nllama_memory_recurrent, layer  30: dev = MTL0\nllama_memory_recurrent, layer  31: dev = MTL0\nllama_memory_recurrent, layer  32: dev = MTL0\nllama_memory_recurrent, layer  33: dev = MTL0\nllama_memory_recurrent, layer  34: dev = MTL0\nllama_memory_recurrent, layer  36: dev = MTL0\nllama_memory_recurrent, layer  37: dev = MTL0\nllama_memory_recurrent, layer  38: dev = MTL0\nllama_memory_recurrent, layer  39: dev = MTL0\nllama_memory_recurrent:       MTL0 RS buffer size =    55.37 MiB\nsched_reserve:       MTL0 compute buffer size =   123.63 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 41,
      "totalLayers": 41,
      "inferenceMs": 96.4625420019729,
      "durationMs": 107.24950000440003,
      "tokensPerSecond": 20.733436611685963,
      "inputTokens": 63,
      "outputTokens": 2,
      "outputLimitReached": false,
      "rssBytes": 4530307072,
      "cpuThreads": 4,
      "totalMs": 1032.6088330039056,
      "grade": "fast",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "chat"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "loadMs": 925.3593329995056,
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer   0 assigned to device MTL0, is_swa = 0\nload_tensors: layer   1 assigned to device MTL0, is_swa = 0\nload_tensors: layer   2 assigned to device MTL0, is_swa = 0\nload_tensors: layer   3 assigned to device MTL0, is_swa = 0\nload_tensors: layer   4 assigned to device MTL0, is_swa = 0\nload_tensors: layer   5 assigned to device MTL0, is_swa = 0\nload_tensors: layer   6 assigned to device MTL0, is_swa = 0\nload_tensors: layer   7 assigned to device MTL0, is_swa = 0\nload_tensors: layer   8 assigned to device MTL0, is_swa = 0\nload_tensors: layer   9 assigned to device MTL0, is_swa = 0\nload_tensors: layer  10 assigned to device MTL0, is_swa = 0\nload_tensors: layer  11 assigned to device MTL0, is_swa = 0\nload_tensors: layer  12 assigned to device MTL0, is_swa = 0\nload_tensors: layer  13 assigned to device MTL0, is_swa = 0\nload_tensors: layer  14 assigned to device MTL0, is_swa = 0\nload_tensors: layer  15 assigned to device MTL0, is_swa = 0\nload_tensors: layer  16 assigned to device MTL0, is_swa = 0\nload_tensors: layer  17 assigned to device MTL0, is_swa = 0\nload_tensors: layer  18 assigned to device MTL0, is_swa = 0\nload_tensors: layer  19 assigned to device MTL0, is_swa = 0\nload_tensors: layer  20 assigned to device MTL0, is_swa = 0\nload_tensors: layer  21 assigned to device MTL0, is_swa = 0\nload_tensors: layer  22 assigned to device MTL0, is_swa = 0\nload_tensors: layer  23 assigned to device MTL0, is_swa = 0\nload_tensors: layer  24 assigned to device MTL0, is_swa = 0\nload_tensors: layer  25 assigned to device MTL0, is_swa = 0\nload_tensors: layer  26 assigned to device MTL0, is_swa = 0\nload_tensors: layer  27 assigned to device MTL0, is_swa = 0\nload_tensors: layer  28 assigned to device MTL0, is_swa = 0\nload_tensors: layer  29 assigned to device MTL0, is_swa = 0\nload_tensors: layer  30 assigned to device MTL0, is_swa = 0\nload_tensors: layer  31 assigned to device MTL0, is_swa = 0\nload_tensors: layer  32 assigned to device MTL0, is_swa = 0\nload_tensors: layer  33 assigned to device MTL0, is_swa = 0\nload_tensors: layer  34 assigned to device MTL0, is_swa = 0\nload_tensors: layer  35 assigned to device MTL0, is_swa = 0\nload_tensors: layer  36 assigned to device MTL0, is_swa = 0\nload_tensors: layer  37 assigned to device MTL0, is_swa = 0\nload_tensors: layer  38 assigned to device MTL0, is_swa = 0\nload_tensors: layer  39 assigned to device MTL0, is_swa = 0\nload_tensors: layer  40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =   120.59 MiB\nload_tensors:  MTL0_Mapped model buffer size =  4031.55 MiB\nllama_kv_cache: layer   5: dev = MTL0\nllama_kv_cache: layer  15: dev = MTL0\nllama_kv_cache: layer  25: dev = MTL0\nllama_kv_cache: layer  35: dev = MTL0\nllama_kv_cache:       MTL0 KV buffer size =    64.00 MiB\nllama_memory_recurrent, layer   0: dev = MTL0\nllama_memory_recurrent, layer   1: dev = MTL0\nllama_memory_recurrent, layer   2: dev = MTL0\nllama_memory_recurrent, layer   3: dev = MTL0\nllama_memory_recurrent, layer   4: dev = MTL0\nllama_memory_recurrent, layer   6: dev = MTL0\nllama_memory_recurrent, layer   7: dev = MTL0\nllama_memory_recurrent, layer   8: dev = MTL0\nllama_memory_recurrent, layer   9: dev = MTL0\nllama_memory_recurrent, layer  10: dev = MTL0\nllama_memory_recurrent, layer  11: dev = MTL0\nllama_memory_recurrent, layer  12: dev = MTL0\nllama_memory_recurrent, layer  13: dev = MTL0\nllama_memory_recurrent, layer  14: dev = MTL0\nllama_memory_recurrent, layer  16: dev = MTL0\nllama_memory_recurrent, layer  17: dev = MTL0\nllama_memory_recurrent, layer  18: dev = MTL0\nllama_memory_recurrent, layer  19: dev = MTL0\nllama_memory_recurrent, layer  20: dev = MTL0\nllama_memory_recurrent, layer  21: dev = MTL0\nllama_memory_recurrent, layer  22: dev = MTL0\nllama_memory_recurrent, layer  23: dev = MTL0\nllama_memory_recurrent, layer  24: dev = MTL0\nllama_memory_recurrent, layer  26: dev = MTL0\nllama_memory_recurrent, layer  27: dev = MTL0\nllama_memory_recurrent, layer  28: dev = MTL0\nllama_memory_recurrent, layer  29: dev = MTL0\nllama_memory_recurrent, layer  30: dev = MTL0\nllama_memory_recurrent, layer  31: dev = MTL0\nllama_memory_recurrent, layer  32: dev = MTL0\nllama_memory_recurrent, layer  33: dev = MTL0\nllama_memory_recurrent, layer  34: dev = MTL0\nllama_memory_recurrent, layer  36: dev = MTL0\nllama_memory_recurrent, layer  37: dev = MTL0\nllama_memory_recurrent, layer  38: dev = MTL0\nllama_memory_recurrent, layer  39: dev = MTL0\nllama_memory_recurrent:       MTL0 RS buffer size =    55.37 MiB\nsched_reserve:       MTL0 compute buffer size =   123.63 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 41,
      "totalLayers": 41,
      "inferenceMs": 98.73174999665935,
      "durationMs": 112.96225000114646,
      "tokensPerSecond": 20.256908239423197,
      "inputTokens": 68,
      "outputTokens": 2,
      "outputLimitReached": false,
      "rssBytes": 4532060160,
      "cpuThreads": 4,
      "totalMs": 1038.321583000652,
      "grade": "fast",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "hola"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "loadMs": 925.3593329995056,
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer   0 assigned to device MTL0, is_swa = 0\nload_tensors: layer   1 assigned to device MTL0, is_swa = 0\nload_tensors: layer   2 assigned to device MTL0, is_swa = 0\nload_tensors: layer   3 assigned to device MTL0, is_swa = 0\nload_tensors: layer   4 assigned to device MTL0, is_swa = 0\nload_tensors: layer   5 assigned to device MTL0, is_swa = 0\nload_tensors: layer   6 assigned to device MTL0, is_swa = 0\nload_tensors: layer   7 assigned to device MTL0, is_swa = 0\nload_tensors: layer   8 assigned to device MTL0, is_swa = 0\nload_tensors: layer   9 assigned to device MTL0, is_swa = 0\nload_tensors: layer  10 assigned to device MTL0, is_swa = 0\nload_tensors: layer  11 assigned to device MTL0, is_swa = 0\nload_tensors: layer  12 assigned to device MTL0, is_swa = 0\nload_tensors: layer  13 assigned to device MTL0, is_swa = 0\nload_tensors: layer  14 assigned to device MTL0, is_swa = 0\nload_tensors: layer  15 assigned to device MTL0, is_swa = 0\nload_tensors: layer  16 assigned to device MTL0, is_swa = 0\nload_tensors: layer  17 assigned to device MTL0, is_swa = 0\nload_tensors: layer  18 assigned to device MTL0, is_swa = 0\nload_tensors: layer  19 assigned to device MTL0, is_swa = 0\nload_tensors: layer  20 assigned to device MTL0, is_swa = 0\nload_tensors: layer  21 assigned to device MTL0, is_swa = 0\nload_tensors: layer  22 assigned to device MTL0, is_swa = 0\nload_tensors: layer  23 assigned to device MTL0, is_swa = 0\nload_tensors: layer  24 assigned to device MTL0, is_swa = 0\nload_tensors: layer  25 assigned to device MTL0, is_swa = 0\nload_tensors: layer  26 assigned to device MTL0, is_swa = 0\nload_tensors: layer  27 assigned to device MTL0, is_swa = 0\nload_tensors: layer  28 assigned to device MTL0, is_swa = 0\nload_tensors: layer  29 assigned to device MTL0, is_swa = 0\nload_tensors: layer  30 assigned to device MTL0, is_swa = 0\nload_tensors: layer  31 assigned to device MTL0, is_swa = 0\nload_tensors: layer  32 assigned to device MTL0, is_swa = 0\nload_tensors: layer  33 assigned to device MTL0, is_swa = 0\nload_tensors: layer  34 assigned to device MTL0, is_swa = 0\nload_tensors: layer  35 assigned to device MTL0, is_swa = 0\nload_tensors: layer  36 assigned to device MTL0, is_swa = 0\nload_tensors: layer  37 assigned to device MTL0, is_swa = 0\nload_tensors: layer  38 assigned to device MTL0, is_swa = 0\nload_tensors: layer  39 assigned to device MTL0, is_swa = 0\nload_tensors: layer  40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =   120.59 MiB\nload_tensors:  MTL0_Mapped model buffer size =  4031.55 MiB\nllama_kv_cache: layer   5: dev = MTL0\nllama_kv_cache: layer  15: dev = MTL0\nllama_kv_cache: layer  25: dev = MTL0\nllama_kv_cache: layer  35: dev = MTL0\nllama_kv_cache:       MTL0 KV buffer size =    64.00 MiB\nllama_memory_recurrent, layer   0: dev = MTL0\nllama_memory_recurrent, layer   1: dev = MTL0\nllama_memory_recurrent, layer   2: dev = MTL0\nllama_memory_recurrent, layer   3: dev = MTL0\nllama_memory_recurrent, layer   4: dev = MTL0\nllama_memory_recurrent, layer   6: dev = MTL0\nllama_memory_recurrent, layer   7: dev = MTL0\nllama_memory_recurrent, layer   8: dev = MTL0\nllama_memory_recurrent, layer   9: dev = MTL0\nllama_memory_recurrent, layer  10: dev = MTL0\nllama_memory_recurrent, layer  11: dev = MTL0\nllama_memory_recurrent, layer  12: dev = MTL0\nllama_memory_recurrent, layer  13: dev = MTL0\nllama_memory_recurrent, layer  14: dev = MTL0\nllama_memory_recurrent, layer  16: dev = MTL0\nllama_memory_recurrent, layer  17: dev = MTL0\nllama_memory_recurrent, layer  18: dev = MTL0\nllama_memory_recurrent, layer  19: dev = MTL0\nllama_memory_recurrent, layer  20: dev = MTL0\nllama_memory_recurrent, layer  21: dev = MTL0\nllama_memory_recurrent, layer  22: dev = MTL0\nllama_memory_recurrent, layer  23: dev = MTL0\nllama_memory_recurrent, layer  24: dev = MTL0\nllama_memory_recurrent, layer  26: dev = MTL0\nllama_memory_recurrent, layer  27: dev = MTL0\nllama_memory_recurrent, layer  28: dev = MTL0\nllama_memory_recurrent, layer  29: dev = MTL0\nllama_memory_recurrent, layer  30: dev = MTL0\nllama_memory_recurrent, layer  31: dev = MTL0\nllama_memory_recurrent, layer  32: dev = MTL0\nllama_memory_recurrent, layer  33: dev = MTL0\nllama_memory_recurrent, layer  34: dev = MTL0\nllama_memory_recurrent, layer  36: dev = MTL0\nllama_memory_recurrent, layer  37: dev = MTL0\nllama_memory_recurrent, layer  38: dev = MTL0\nllama_memory_recurrent, layer  39: dev = MTL0\nllama_memory_recurrent:       MTL0 RS buffer size =    55.37 MiB\nsched_reserve:       MTL0 compute buffer size =   123.63 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 41,
      "totalLayers": 41,
      "inferenceMs": 101.52958300022874,
      "durationMs": 119.607291999273,
      "tokensPerSecond": 19.698692153551878,
      "inputTokens": 69,
      "outputTokens": 2,
      "outputLimitReached": false,
      "rssBytes": 4533436416,
      "cpuThreads": 4,
      "totalMs": 1044.9666249987786,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "2026-11-04"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "loadMs": 925.3593329995056,
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer   0 assigned to device MTL0, is_swa = 0\nload_tensors: layer   1 assigned to device MTL0, is_swa = 0\nload_tensors: layer   2 assigned to device MTL0, is_swa = 0\nload_tensors: layer   3 assigned to device MTL0, is_swa = 0\nload_tensors: layer   4 assigned to device MTL0, is_swa = 0\nload_tensors: layer   5 assigned to device MTL0, is_swa = 0\nload_tensors: layer   6 assigned to device MTL0, is_swa = 0\nload_tensors: layer   7 assigned to device MTL0, is_swa = 0\nload_tensors: layer   8 assigned to device MTL0, is_swa = 0\nload_tensors: layer   9 assigned to device MTL0, is_swa = 0\nload_tensors: layer  10 assigned to device MTL0, is_swa = 0\nload_tensors: layer  11 assigned to device MTL0, is_swa = 0\nload_tensors: layer  12 assigned to device MTL0, is_swa = 0\nload_tensors: layer  13 assigned to device MTL0, is_swa = 0\nload_tensors: layer  14 assigned to device MTL0, is_swa = 0\nload_tensors: layer  15 assigned to device MTL0, is_swa = 0\nload_tensors: layer  16 assigned to device MTL0, is_swa = 0\nload_tensors: layer  17 assigned to device MTL0, is_swa = 0\nload_tensors: layer  18 assigned to device MTL0, is_swa = 0\nload_tensors: layer  19 assigned to device MTL0, is_swa = 0\nload_tensors: layer  20 assigned to device MTL0, is_swa = 0\nload_tensors: layer  21 assigned to device MTL0, is_swa = 0\nload_tensors: layer  22 assigned to device MTL0, is_swa = 0\nload_tensors: layer  23 assigned to device MTL0, is_swa = 0\nload_tensors: layer  24 assigned to device MTL0, is_swa = 0\nload_tensors: layer  25 assigned to device MTL0, is_swa = 0\nload_tensors: layer  26 assigned to device MTL0, is_swa = 0\nload_tensors: layer  27 assigned to device MTL0, is_swa = 0\nload_tensors: layer  28 assigned to device MTL0, is_swa = 0\nload_tensors: layer  29 assigned to device MTL0, is_swa = 0\nload_tensors: layer  30 assigned to device MTL0, is_swa = 0\nload_tensors: layer  31 assigned to device MTL0, is_swa = 0\nload_tensors: layer  32 assigned to device MTL0, is_swa = 0\nload_tensors: layer  33 assigned to device MTL0, is_swa = 0\nload_tensors: layer  34 assigned to device MTL0, is_swa = 0\nload_tensors: layer  35 assigned to device MTL0, is_swa = 0\nload_tensors: layer  36 assigned to device MTL0, is_swa = 0\nload_tensors: layer  37 assigned to device MTL0, is_swa = 0\nload_tensors: layer  38 assigned to device MTL0, is_swa = 0\nload_tensors: layer  39 assigned to device MTL0, is_swa = 0\nload_tensors: layer  40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =   120.59 MiB\nload_tensors:  MTL0_Mapped model buffer size =  4031.55 MiB\nllama_kv_cache: layer   5: dev = MTL0\nllama_kv_cache: layer  15: dev = MTL0\nllama_kv_cache: layer  25: dev = MTL0\nllama_kv_cache: layer  35: dev = MTL0\nllama_kv_cache:       MTL0 KV buffer size =    64.00 MiB\nllama_memory_recurrent, layer   0: dev = MTL0\nllama_memory_recurrent, layer   1: dev = MTL0\nllama_memory_recurrent, layer   2: dev = MTL0\nllama_memory_recurrent, layer   3: dev = MTL0\nllama_memory_recurrent, layer   4: dev = MTL0\nllama_memory_recurrent, layer   6: dev = MTL0\nllama_memory_recurrent, layer   7: dev = MTL0\nllama_memory_recurrent, layer   8: dev = MTL0\nllama_memory_recurrent, layer   9: dev = MTL0\nllama_memory_recurrent, layer  10: dev = MTL0\nllama_memory_recurrent, layer  11: dev = MTL0\nllama_memory_recurrent, layer  12: dev = MTL0\nllama_memory_recurrent, layer  13: dev = MTL0\nllama_memory_recurrent, layer  14: dev = MTL0\nllama_memory_recurrent, layer  16: dev = MTL0\nllama_memory_recurrent, layer  17: dev = MTL0\nllama_memory_recurrent, layer  18: dev = MTL0\nllama_memory_recurrent, layer  19: dev = MTL0\nllama_memory_recurrent, layer  20: dev = MTL0\nllama_memory_recurrent, layer  21: dev = MTL0\nllama_memory_recurrent, layer  22: dev = MTL0\nllama_memory_recurrent, layer  23: dev = MTL0\nllama_memory_recurrent, layer  24: dev = MTL0\nllama_memory_recurrent, layer  26: dev = MTL0\nllama_memory_recurrent, layer  27: dev = MTL0\nllama_memory_recurrent, layer  28: dev = MTL0\nllama_memory_recurrent, layer  29: dev = MTL0\nllama_memory_recurrent, layer  30: dev = MTL0\nllama_memory_recurrent, layer  31: dev = MTL0\nllama_memory_recurrent, layer  32: dev = MTL0\nllama_memory_recurrent, layer  33: dev = MTL0\nllama_memory_recurrent, layer  34: dev = MTL0\nllama_memory_recurrent, layer  36: dev = MTL0\nllama_memory_recurrent, layer  37: dev = MTL0\nllama_memory_recurrent, layer  38: dev = MTL0\nllama_memory_recurrent, layer  39: dev = MTL0\nllama_memory_recurrent:       MTL0 RS buffer size =    55.37 MiB\nsched_reserve:       MTL0 compute buffer size =   123.63 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 41,
      "totalLayers": 41,
      "inferenceMs": 170.4097090041614,
      "durationMs": 191.52187500003492,
      "tokensPerSecond": 41.07747170572927,
      "inputTokens": 72,
      "outputTokens": 7,
      "outputLimitReached": false,
      "rssBytes": 4533567488,
      "cpuThreads": 4,
      "totalMs": 1116.8812079995405,
      "grade": "fast",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "Unknown"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "loadMs": 925.3593329995056,
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer   0 assigned to device MTL0, is_swa = 0\nload_tensors: layer   1 assigned to device MTL0, is_swa = 0\nload_tensors: layer   2 assigned to device MTL0, is_swa = 0\nload_tensors: layer   3 assigned to device MTL0, is_swa = 0\nload_tensors: layer   4 assigned to device MTL0, is_swa = 0\nload_tensors: layer   5 assigned to device MTL0, is_swa = 0\nload_tensors: layer   6 assigned to device MTL0, is_swa = 0\nload_tensors: layer   7 assigned to device MTL0, is_swa = 0\nload_tensors: layer   8 assigned to device MTL0, is_swa = 0\nload_tensors: layer   9 assigned to device MTL0, is_swa = 0\nload_tensors: layer  10 assigned to device MTL0, is_swa = 0\nload_tensors: layer  11 assigned to device MTL0, is_swa = 0\nload_tensors: layer  12 assigned to device MTL0, is_swa = 0\nload_tensors: layer  13 assigned to device MTL0, is_swa = 0\nload_tensors: layer  14 assigned to device MTL0, is_swa = 0\nload_tensors: layer  15 assigned to device MTL0, is_swa = 0\nload_tensors: layer  16 assigned to device MTL0, is_swa = 0\nload_tensors: layer  17 assigned to device MTL0, is_swa = 0\nload_tensors: layer  18 assigned to device MTL0, is_swa = 0\nload_tensors: layer  19 assigned to device MTL0, is_swa = 0\nload_tensors: layer  20 assigned to device MTL0, is_swa = 0\nload_tensors: layer  21 assigned to device MTL0, is_swa = 0\nload_tensors: layer  22 assigned to device MTL0, is_swa = 0\nload_tensors: layer  23 assigned to device MTL0, is_swa = 0\nload_tensors: layer  24 assigned to device MTL0, is_swa = 0\nload_tensors: layer  25 assigned to device MTL0, is_swa = 0\nload_tensors: layer  26 assigned to device MTL0, is_swa = 0\nload_tensors: layer  27 assigned to device MTL0, is_swa = 0\nload_tensors: layer  28 assigned to device MTL0, is_swa = 0\nload_tensors: layer  29 assigned to device MTL0, is_swa = 0\nload_tensors: layer  30 assigned to device MTL0, is_swa = 0\nload_tensors: layer  31 assigned to device MTL0, is_swa = 0\nload_tensors: layer  32 assigned to device MTL0, is_swa = 0\nload_tensors: layer  33 assigned to device MTL0, is_swa = 0\nload_tensors: layer  34 assigned to device MTL0, is_swa = 0\nload_tensors: layer  35 assigned to device MTL0, is_swa = 0\nload_tensors: layer  36 assigned to device MTL0, is_swa = 0\nload_tensors: layer  37 assigned to device MTL0, is_swa = 0\nload_tensors: layer  38 assigned to device MTL0, is_swa = 0\nload_tensors: layer  39 assigned to device MTL0, is_swa = 0\nload_tensors: layer  40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =   120.59 MiB\nload_tensors:  MTL0_Mapped model buffer size =  4031.55 MiB\nllama_kv_cache: layer   5: dev = MTL0\nllama_kv_cache: layer  15: dev = MTL0\nllama_kv_cache: layer  25: dev = MTL0\nllama_kv_cache: layer  35: dev = MTL0\nllama_kv_cache:       MTL0 KV buffer size =    64.00 MiB\nllama_memory_recurrent, layer   0: dev = MTL0\nllama_memory_recurrent, layer   1: dev = MTL0\nllama_memory_recurrent, layer   2: dev = MTL0\nllama_memory_recurrent, layer   3: dev = MTL0\nllama_memory_recurrent, layer   4: dev = MTL0\nllama_memory_recurrent, layer   6: dev = MTL0\nllama_memory_recurrent, layer   7: dev = MTL0\nllama_memory_recurrent, layer   8: dev = MTL0\nllama_memory_recurrent, layer   9: dev = MTL0\nllama_memory_recurrent, layer  10: dev = MTL0\nllama_memory_recurrent, layer  11: dev = MTL0\nllama_memory_recurrent, layer  12: dev = MTL0\nllama_memory_recurrent, layer  13: dev = MTL0\nllama_memory_recurrent, layer  14: dev = MTL0\nllama_memory_recurrent, layer  16: dev = MTL0\nllama_memory_recurrent, layer  17: dev = MTL0\nllama_memory_recurrent, layer  18: dev = MTL0\nllama_memory_recurrent, layer  19: dev = MTL0\nllama_memory_recurrent, layer  20: dev = MTL0\nllama_memory_recurrent, layer  21: dev = MTL0\nllama_memory_recurrent, layer  22: dev = MTL0\nllama_memory_recurrent, layer  23: dev = MTL0\nllama_memory_recurrent, layer  24: dev = MTL0\nllama_memory_recurrent, layer  26: dev = MTL0\nllama_memory_recurrent, layer  27: dev = MTL0\nllama_memory_recurrent, layer  28: dev = MTL0\nllama_memory_recurrent, layer  29: dev = MTL0\nllama_memory_recurrent, layer  30: dev = MTL0\nllama_memory_recurrent, layer  31: dev = MTL0\nllama_memory_recurrent, layer  32: dev = MTL0\nllama_memory_recurrent, layer  33: dev = MTL0\nllama_memory_recurrent, layer  34: dev = MTL0\nllama_memory_recurrent, layer  36: dev = MTL0\nllama_memory_recurrent, layer  37: dev = MTL0\nllama_memory_recurrent, layer  38: dev = MTL0\nllama_memory_recurrent, layer  39: dev = MTL0\nllama_memory_recurrent:       MTL0 RS buffer size =    55.37 MiB\nsched_reserve:       MTL0 compute buffer size =   123.63 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 41,
      "totalLayers": 41,
      "inferenceMs": 100.23616599937668,
      "durationMs": 125.43708400335163,
      "tokensPerSecond": 19.952878086063638,
      "inputTokens": 74,
      "outputTokens": 2,
      "outputLimitReached": false,
      "rssBytes": 4533633024,
      "cpuThreads": 4,
      "totalMs": 1050.7964170028572,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "Paris"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "loadMs": 1236.3420840047183,
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer   0 assigned to device MTL0, is_swa = 0\nload_tensors: layer   1 assigned to device MTL0, is_swa = 0\nload_tensors: layer   2 assigned to device MTL0, is_swa = 0\nload_tensors: layer   3 assigned to device MTL0, is_swa = 0\nload_tensors: layer   4 assigned to device MTL0, is_swa = 0\nload_tensors: layer   5 assigned to device MTL0, is_swa = 0\nload_tensors: layer   6 assigned to device MTL0, is_swa = 0\nload_tensors: layer   7 assigned to device MTL0, is_swa = 0\nload_tensors: layer   8 assigned to device MTL0, is_swa = 0\nload_tensors: layer   9 assigned to device MTL0, is_swa = 0\nload_tensors: layer  10 assigned to device MTL0, is_swa = 0\nload_tensors: layer  11 assigned to device MTL0, is_swa = 0\nload_tensors: layer  12 assigned to device MTL0, is_swa = 0\nload_tensors: layer  13 assigned to device MTL0, is_swa = 0\nload_tensors: layer  14 assigned to device MTL0, is_swa = 0\nload_tensors: layer  15 assigned to device MTL0, is_swa = 0\nload_tensors: layer  16 assigned to device MTL0, is_swa = 0\nload_tensors: layer  17 assigned to device MTL0, is_swa = 0\nload_tensors: layer  18 assigned to device MTL0, is_swa = 0\nload_tensors: layer  19 assigned to device MTL0, is_swa = 0\nload_tensors: layer  20 assigned to device MTL0, is_swa = 0\nload_tensors: layer  21 assigned to device MTL0, is_swa = 0\nload_tensors: layer  22 assigned to device MTL0, is_swa = 0\nload_tensors: layer  23 assigned to device MTL0, is_swa = 0\nload_tensors: layer  24 assigned to device MTL0, is_swa = 0\nload_tensors: layer  25 assigned to device MTL0, is_swa = 0\nload_tensors: layer  26 assigned to device MTL0, is_swa = 0\nload_tensors: layer  27 assigned to device MTL0, is_swa = 0\nload_tensors: layer  28 assigned to device MTL0, is_swa = 0\nload_tensors: layer  29 assigned to device MTL0, is_swa = 0\nload_tensors: layer  30 assigned to device MTL0, is_swa = 0\nload_tensors: layer  31 assigned to device MTL0, is_swa = 0\nload_tensors: layer  32 assigned to device MTL0, is_swa = 0\nload_tensors: layer  33 assigned to device MTL0, is_swa = 0\nload_tensors: layer  34 assigned to device MTL0, is_swa = 0\nload_tensors: layer  35 assigned to device MTL0, is_swa = 0\nload_tensors: layer  36 assigned to device MTL0, is_swa = 0\nload_tensors: layer  37 assigned to device MTL0, is_swa = 0\nload_tensors: layer  38 assigned to device MTL0, is_swa = 0\nload_tensors: layer  39 assigned to device MTL0, is_swa = 0\nload_tensors: layer  40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =   120.59 MiB\nload_tensors:  MTL0_Mapped model buffer size =  4031.55 MiB\nllama_kv_cache: layer   5: dev = MTL0\nllama_kv_cache: layer  15: dev = MTL0\nllama_kv_cache: layer  25: dev = MTL0\nllama_kv_cache: layer  35: dev = MTL0\nllama_kv_cache:       MTL0 KV buffer size =    64.00 MiB\nllama_memory_recurrent, layer   0: dev = MTL0\nllama_memory_recurrent, layer   1: dev = MTL0\nllama_memory_recurrent, layer   2: dev = MTL0\nllama_memory_recurrent, layer   3: dev = MTL0\nllama_memory_recurrent, layer   4: dev = MTL0\nllama_memory_recurrent, layer   6: dev = MTL0\nllama_memory_recurrent, layer   7: dev = MTL0\nllama_memory_recurrent, layer   8: dev = MTL0\nllama_memory_recurrent, layer   9: dev = MTL0\nllama_memory_recurrent, layer  10: dev = MTL0\nllama_memory_recurrent, layer  11: dev = MTL0\nllama_memory_recurrent, layer  12: dev = MTL0\nllama_memory_recurrent, layer  13: dev = MTL0\nllama_memory_recurrent, layer  14: dev = MTL0\nllama_memory_recurrent, layer  16: dev = MTL0\nllama_memory_recurrent, layer  17: dev = MTL0\nllama_memory_recurrent, layer  18: dev = MTL0\nllama_memory_recurrent, layer  19: dev = MTL0\nllama_memory_recurrent, layer  20: dev = MTL0\nllama_memory_recurrent, layer  21: dev = MTL0\nllama_memory_recurrent, layer  22: dev = MTL0\nllama_memory_recurrent, layer  23: dev = MTL0\nllama_memory_recurrent, layer  24: dev = MTL0\nllama_memory_recurrent, layer  26: dev = MTL0\nllama_memory_recurrent, layer  27: dev = MTL0\nllama_memory_recurrent, layer  28: dev = MTL0\nllama_memory_recurrent, layer  29: dev = MTL0\nllama_memory_recurrent, layer  30: dev = MTL0\nllama_memory_recurrent, layer  31: dev = MTL0\nllama_memory_recurrent, layer  32: dev = MTL0\nllama_memory_recurrent, layer  33: dev = MTL0\nllama_memory_recurrent, layer  34: dev = MTL0\nllama_memory_recurrent, layer  36: dev = MTL0\nllama_memory_recurrent, layer  37: dev = MTL0\nllama_memory_recurrent, layer  38: dev = MTL0\nllama_memory_recurrent, layer  39: dev = MTL0\nllama_memory_recurrent:       MTL0 RS buffer size =    55.37 MiB\nsched_reserve:       MTL0 compute buffer size =   123.63 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 41,
      "totalLayers": 41,
      "inferenceMs": 97.12879200378666,
      "durationMs": 107.12225000315811,
      "tokensPerSecond": 20.591216659237645,
      "inputTokens": 63,
      "outputTokens": 2,
      "outputLimitReached": false,
      "rssBytes": 4531077120,
      "cpuThreads": 4,
      "totalMs": 1343.4643340078765,
      "grade": "fast",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "56"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "loadMs": 1236.3420840047183,
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer   0 assigned to device MTL0, is_swa = 0\nload_tensors: layer   1 assigned to device MTL0, is_swa = 0\nload_tensors: layer   2 assigned to device MTL0, is_swa = 0\nload_tensors: layer   3 assigned to device MTL0, is_swa = 0\nload_tensors: layer   4 assigned to device MTL0, is_swa = 0\nload_tensors: layer   5 assigned to device MTL0, is_swa = 0\nload_tensors: layer   6 assigned to device MTL0, is_swa = 0\nload_tensors: layer   7 assigned to device MTL0, is_swa = 0\nload_tensors: layer   8 assigned to device MTL0, is_swa = 0\nload_tensors: layer   9 assigned to device MTL0, is_swa = 0\nload_tensors: layer  10 assigned to device MTL0, is_swa = 0\nload_tensors: layer  11 assigned to device MTL0, is_swa = 0\nload_tensors: layer  12 assigned to device MTL0, is_swa = 0\nload_tensors: layer  13 assigned to device MTL0, is_swa = 0\nload_tensors: layer  14 assigned to device MTL0, is_swa = 0\nload_tensors: layer  15 assigned to device MTL0, is_swa = 0\nload_tensors: layer  16 assigned to device MTL0, is_swa = 0\nload_tensors: layer  17 assigned to device MTL0, is_swa = 0\nload_tensors: layer  18 assigned to device MTL0, is_swa = 0\nload_tensors: layer  19 assigned to device MTL0, is_swa = 0\nload_tensors: layer  20 assigned to device MTL0, is_swa = 0\nload_tensors: layer  21 assigned to device MTL0, is_swa = 0\nload_tensors: layer  22 assigned to device MTL0, is_swa = 0\nload_tensors: layer  23 assigned to device MTL0, is_swa = 0\nload_tensors: layer  24 assigned to device MTL0, is_swa = 0\nload_tensors: layer  25 assigned to device MTL0, is_swa = 0\nload_tensors: layer  26 assigned to device MTL0, is_swa = 0\nload_tensors: layer  27 assigned to device MTL0, is_swa = 0\nload_tensors: layer  28 assigned to device MTL0, is_swa = 0\nload_tensors: layer  29 assigned to device MTL0, is_swa = 0\nload_tensors: layer  30 assigned to device MTL0, is_swa = 0\nload_tensors: layer  31 assigned to device MTL0, is_swa = 0\nload_tensors: layer  32 assigned to device MTL0, is_swa = 0\nload_tensors: layer  33 assigned to device MTL0, is_swa = 0\nload_tensors: layer  34 assigned to device MTL0, is_swa = 0\nload_tensors: layer  35 assigned to device MTL0, is_swa = 0\nload_tensors: layer  36 assigned to device MTL0, is_swa = 0\nload_tensors: layer  37 assigned to device MTL0, is_swa = 0\nload_tensors: layer  38 assigned to device MTL0, is_swa = 0\nload_tensors: layer  39 assigned to device MTL0, is_swa = 0\nload_tensors: layer  40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =   120.59 MiB\nload_tensors:  MTL0_Mapped model buffer size =  4031.55 MiB\nllama_kv_cache: layer   5: dev = MTL0\nllama_kv_cache: layer  15: dev = MTL0\nllama_kv_cache: layer  25: dev = MTL0\nllama_kv_cache: layer  35: dev = MTL0\nllama_kv_cache:       MTL0 KV buffer size =    64.00 MiB\nllama_memory_recurrent, layer   0: dev = MTL0\nllama_memory_recurrent, layer   1: dev = MTL0\nllama_memory_recurrent, layer   2: dev = MTL0\nllama_memory_recurrent, layer   3: dev = MTL0\nllama_memory_recurrent, layer   4: dev = MTL0\nllama_memory_recurrent, layer   6: dev = MTL0\nllama_memory_recurrent, layer   7: dev = MTL0\nllama_memory_recurrent, layer   8: dev = MTL0\nllama_memory_recurrent, layer   9: dev = MTL0\nllama_memory_recurrent, layer  10: dev = MTL0\nllama_memory_recurrent, layer  11: dev = MTL0\nllama_memory_recurrent, layer  12: dev = MTL0\nllama_memory_recurrent, layer  13: dev = MTL0\nllama_memory_recurrent, layer  14: dev = MTL0\nllama_memory_recurrent, layer  16: dev = MTL0\nllama_memory_recurrent, layer  17: dev = MTL0\nllama_memory_recurrent, layer  18: dev = MTL0\nllama_memory_recurrent, layer  19: dev = MTL0\nllama_memory_recurrent, layer  20: dev = MTL0\nllama_memory_recurrent, layer  21: dev = MTL0\nllama_memory_recurrent, layer  22: dev = MTL0\nllama_memory_recurrent, layer  23: dev = MTL0\nllama_memory_recurrent, layer  24: dev = MTL0\nllama_memory_recurrent, layer  26: dev = MTL0\nllama_memory_recurrent, layer  27: dev = MTL0\nllama_memory_recurrent, layer  28: dev = MTL0\nllama_memory_recurrent, layer  29: dev = MTL0\nllama_memory_recurrent, layer  30: dev = MTL0\nllama_memory_recurrent, layer  31: dev = MTL0\nllama_memory_recurrent, layer  32: dev = MTL0\nllama_memory_recurrent, layer  33: dev = MTL0\nllama_memory_recurrent, layer  34: dev = MTL0\nllama_memory_recurrent, layer  36: dev = MTL0\nllama_memory_recurrent, layer  37: dev = MTL0\nllama_memory_recurrent, layer  38: dev = MTL0\nllama_memory_recurrent, layer  39: dev = MTL0\nllama_memory_recurrent:       MTL0 RS buffer size =    55.37 MiB\nsched_reserve:       MTL0 compute buffer size =   123.63 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 41,
      "totalLayers": 41,
      "inferenceMs": 93.80783399683423,
      "durationMs": 230.77516599732917,
      "tokensPerSecond": 21.3201809996753,
      "inputTokens": 63,
      "outputTokens": 2,
      "outputLimitReached": false,
      "rssBytes": 4532355072,
      "cpuThreads": 4,
      "totalMs": 1467.1172500020475,
      "grade": "fast",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "chat"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "loadMs": 1236.3420840047183,
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer   0 assigned to device MTL0, is_swa = 0\nload_tensors: layer   1 assigned to device MTL0, is_swa = 0\nload_tensors: layer   2 assigned to device MTL0, is_swa = 0\nload_tensors: layer   3 assigned to device MTL0, is_swa = 0\nload_tensors: layer   4 assigned to device MTL0, is_swa = 0\nload_tensors: layer   5 assigned to device MTL0, is_swa = 0\nload_tensors: layer   6 assigned to device MTL0, is_swa = 0\nload_tensors: layer   7 assigned to device MTL0, is_swa = 0\nload_tensors: layer   8 assigned to device MTL0, is_swa = 0\nload_tensors: layer   9 assigned to device MTL0, is_swa = 0\nload_tensors: layer  10 assigned to device MTL0, is_swa = 0\nload_tensors: layer  11 assigned to device MTL0, is_swa = 0\nload_tensors: layer  12 assigned to device MTL0, is_swa = 0\nload_tensors: layer  13 assigned to device MTL0, is_swa = 0\nload_tensors: layer  14 assigned to device MTL0, is_swa = 0\nload_tensors: layer  15 assigned to device MTL0, is_swa = 0\nload_tensors: layer  16 assigned to device MTL0, is_swa = 0\nload_tensors: layer  17 assigned to device MTL0, is_swa = 0\nload_tensors: layer  18 assigned to device MTL0, is_swa = 0\nload_tensors: layer  19 assigned to device MTL0, is_swa = 0\nload_tensors: layer  20 assigned to device MTL0, is_swa = 0\nload_tensors: layer  21 assigned to device MTL0, is_swa = 0\nload_tensors: layer  22 assigned to device MTL0, is_swa = 0\nload_tensors: layer  23 assigned to device MTL0, is_swa = 0\nload_tensors: layer  24 assigned to device MTL0, is_swa = 0\nload_tensors: layer  25 assigned to device MTL0, is_swa = 0\nload_tensors: layer  26 assigned to device MTL0, is_swa = 0\nload_tensors: layer  27 assigned to device MTL0, is_swa = 0\nload_tensors: layer  28 assigned to device MTL0, is_swa = 0\nload_tensors: layer  29 assigned to device MTL0, is_swa = 0\nload_tensors: layer  30 assigned to device MTL0, is_swa = 0\nload_tensors: layer  31 assigned to device MTL0, is_swa = 0\nload_tensors: layer  32 assigned to device MTL0, is_swa = 0\nload_tensors: layer  33 assigned to device MTL0, is_swa = 0\nload_tensors: layer  34 assigned to device MTL0, is_swa = 0\nload_tensors: layer  35 assigned to device MTL0, is_swa = 0\nload_tensors: layer  36 assigned to device MTL0, is_swa = 0\nload_tensors: layer  37 assigned to device MTL0, is_swa = 0\nload_tensors: layer  38 assigned to device MTL0, is_swa = 0\nload_tensors: layer  39 assigned to device MTL0, is_swa = 0\nload_tensors: layer  40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =   120.59 MiB\nload_tensors:  MTL0_Mapped model buffer size =  4031.55 MiB\nllama_kv_cache: layer   5: dev = MTL0\nllama_kv_cache: layer  15: dev = MTL0\nllama_kv_cache: layer  25: dev = MTL0\nllama_kv_cache: layer  35: dev = MTL0\nllama_kv_cache:       MTL0 KV buffer size =    64.00 MiB\nllama_memory_recurrent, layer   0: dev = MTL0\nllama_memory_recurrent, layer   1: dev = MTL0\nllama_memory_recurrent, layer   2: dev = MTL0\nllama_memory_recurrent, layer   3: dev = MTL0\nllama_memory_recurrent, layer   4: dev = MTL0\nllama_memory_recurrent, layer   6: dev = MTL0\nllama_memory_recurrent, layer   7: dev = MTL0\nllama_memory_recurrent, layer   8: dev = MTL0\nllama_memory_recurrent, layer   9: dev = MTL0\nllama_memory_recurrent, layer  10: dev = MTL0\nllama_memory_recurrent, layer  11: dev = MTL0\nllama_memory_recurrent, layer  12: dev = MTL0\nllama_memory_recurrent, layer  13: dev = MTL0\nllama_memory_recurrent, layer  14: dev = MTL0\nllama_memory_recurrent, layer  16: dev = MTL0\nllama_memory_recurrent, layer  17: dev = MTL0\nllama_memory_recurrent, layer  18: dev = MTL0\nllama_memory_recurrent, layer  19: dev = MTL0\nllama_memory_recurrent, layer  20: dev = MTL0\nllama_memory_recurrent, layer  21: dev = MTL0\nllama_memory_recurrent, layer  22: dev = MTL0\nllama_memory_recurrent, layer  23: dev = MTL0\nllama_memory_recurrent, layer  24: dev = MTL0\nllama_memory_recurrent, layer  26: dev = MTL0\nllama_memory_recurrent, layer  27: dev = MTL0\nllama_memory_recurrent, layer  28: dev = MTL0\nllama_memory_recurrent, layer  29: dev = MTL0\nllama_memory_recurrent, layer  30: dev = MTL0\nllama_memory_recurrent, layer  31: dev = MTL0\nllama_memory_recurrent, layer  32: dev = MTL0\nllama_memory_recurrent, layer  33: dev = MTL0\nllama_memory_recurrent, layer  34: dev = MTL0\nllama_memory_recurrent, layer  36: dev = MTL0\nllama_memory_recurrent, layer  37: dev = MTL0\nllama_memory_recurrent, layer  38: dev = MTL0\nllama_memory_recurrent, layer  39: dev = MTL0\nllama_memory_recurrent:       MTL0 RS buffer size =    55.37 MiB\nsched_reserve:       MTL0 compute buffer size =   123.63 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 41,
      "totalLayers": 41,
      "inferenceMs": 97.01829199912027,
      "durationMs": 106.974624999566,
      "tokensPerSecond": 20.61466924214802,
      "inputTokens": 68,
      "outputTokens": 2,
      "outputLimitReached": false,
      "rssBytes": 4532961280,
      "cpuThreads": 4,
      "totalMs": 1343.3167090042843,
      "grade": "fast",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "hola"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "loadMs": 1236.3420840047183,
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer   0 assigned to device MTL0, is_swa = 0\nload_tensors: layer   1 assigned to device MTL0, is_swa = 0\nload_tensors: layer   2 assigned to device MTL0, is_swa = 0\nload_tensors: layer   3 assigned to device MTL0, is_swa = 0\nload_tensors: layer   4 assigned to device MTL0, is_swa = 0\nload_tensors: layer   5 assigned to device MTL0, is_swa = 0\nload_tensors: layer   6 assigned to device MTL0, is_swa = 0\nload_tensors: layer   7 assigned to device MTL0, is_swa = 0\nload_tensors: layer   8 assigned to device MTL0, is_swa = 0\nload_tensors: layer   9 assigned to device MTL0, is_swa = 0\nload_tensors: layer  10 assigned to device MTL0, is_swa = 0\nload_tensors: layer  11 assigned to device MTL0, is_swa = 0\nload_tensors: layer  12 assigned to device MTL0, is_swa = 0\nload_tensors: layer  13 assigned to device MTL0, is_swa = 0\nload_tensors: layer  14 assigned to device MTL0, is_swa = 0\nload_tensors: layer  15 assigned to device MTL0, is_swa = 0\nload_tensors: layer  16 assigned to device MTL0, is_swa = 0\nload_tensors: layer  17 assigned to device MTL0, is_swa = 0\nload_tensors: layer  18 assigned to device MTL0, is_swa = 0\nload_tensors: layer  19 assigned to device MTL0, is_swa = 0\nload_tensors: layer  20 assigned to device MTL0, is_swa = 0\nload_tensors: layer  21 assigned to device MTL0, is_swa = 0\nload_tensors: layer  22 assigned to device MTL0, is_swa = 0\nload_tensors: layer  23 assigned to device MTL0, is_swa = 0\nload_tensors: layer  24 assigned to device MTL0, is_swa = 0\nload_tensors: layer  25 assigned to device MTL0, is_swa = 0\nload_tensors: layer  26 assigned to device MTL0, is_swa = 0\nload_tensors: layer  27 assigned to device MTL0, is_swa = 0\nload_tensors: layer  28 assigned to device MTL0, is_swa = 0\nload_tensors: layer  29 assigned to device MTL0, is_swa = 0\nload_tensors: layer  30 assigned to device MTL0, is_swa = 0\nload_tensors: layer  31 assigned to device MTL0, is_swa = 0\nload_tensors: layer  32 assigned to device MTL0, is_swa = 0\nload_tensors: layer  33 assigned to device MTL0, is_swa = 0\nload_tensors: layer  34 assigned to device MTL0, is_swa = 0\nload_tensors: layer  35 assigned to device MTL0, is_swa = 0\nload_tensors: layer  36 assigned to device MTL0, is_swa = 0\nload_tensors: layer  37 assigned to device MTL0, is_swa = 0\nload_tensors: layer  38 assigned to device MTL0, is_swa = 0\nload_tensors: layer  39 assigned to device MTL0, is_swa = 0\nload_tensors: layer  40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =   120.59 MiB\nload_tensors:  MTL0_Mapped model buffer size =  4031.55 MiB\nllama_kv_cache: layer   5: dev = MTL0\nllama_kv_cache: layer  15: dev = MTL0\nllama_kv_cache: layer  25: dev = MTL0\nllama_kv_cache: layer  35: dev = MTL0\nllama_kv_cache:       MTL0 KV buffer size =    64.00 MiB\nllama_memory_recurrent, layer   0: dev = MTL0\nllama_memory_recurrent, layer   1: dev = MTL0\nllama_memory_recurrent, layer   2: dev = MTL0\nllama_memory_recurrent, layer   3: dev = MTL0\nllama_memory_recurrent, layer   4: dev = MTL0\nllama_memory_recurrent, layer   6: dev = MTL0\nllama_memory_recurrent, layer   7: dev = MTL0\nllama_memory_recurrent, layer   8: dev = MTL0\nllama_memory_recurrent, layer   9: dev = MTL0\nllama_memory_recurrent, layer  10: dev = MTL0\nllama_memory_recurrent, layer  11: dev = MTL0\nllama_memory_recurrent, layer  12: dev = MTL0\nllama_memory_recurrent, layer  13: dev = MTL0\nllama_memory_recurrent, layer  14: dev = MTL0\nllama_memory_recurrent, layer  16: dev = MTL0\nllama_memory_recurrent, layer  17: dev = MTL0\nllama_memory_recurrent, layer  18: dev = MTL0\nllama_memory_recurrent, layer  19: dev = MTL0\nllama_memory_recurrent, layer  20: dev = MTL0\nllama_memory_recurrent, layer  21: dev = MTL0\nllama_memory_recurrent, layer  22: dev = MTL0\nllama_memory_recurrent, layer  23: dev = MTL0\nllama_memory_recurrent, layer  24: dev = MTL0\nllama_memory_recurrent, layer  26: dev = MTL0\nllama_memory_recurrent, layer  27: dev = MTL0\nllama_memory_recurrent, layer  28: dev = MTL0\nllama_memory_recurrent, layer  29: dev = MTL0\nllama_memory_recurrent, layer  30: dev = MTL0\nllama_memory_recurrent, layer  31: dev = MTL0\nllama_memory_recurrent, layer  32: dev = MTL0\nllama_memory_recurrent, layer  33: dev = MTL0\nllama_memory_recurrent, layer  34: dev = MTL0\nllama_memory_recurrent, layer  36: dev = MTL0\nllama_memory_recurrent, layer  37: dev = MTL0\nllama_memory_recurrent, layer  38: dev = MTL0\nllama_memory_recurrent, layer  39: dev = MTL0\nllama_memory_recurrent:       MTL0 RS buffer size =    55.37 MiB\nsched_reserve:       MTL0 compute buffer size =   123.63 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 41,
      "totalLayers": 41,
      "inferenceMs": 100.1509170018835,
      "durationMs": 109.32479199982481,
      "tokensPerSecond": 19.96986208286427,
      "inputTokens": 69,
      "outputTokens": 2,
      "outputLimitReached": false,
      "rssBytes": 4534452224,
      "cpuThreads": 4,
      "totalMs": 1345.6668760045432,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "2026-11-04"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "loadMs": 1236.3420840047183,
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer   0 assigned to device MTL0, is_swa = 0\nload_tensors: layer   1 assigned to device MTL0, is_swa = 0\nload_tensors: layer   2 assigned to device MTL0, is_swa = 0\nload_tensors: layer   3 assigned to device MTL0, is_swa = 0\nload_tensors: layer   4 assigned to device MTL0, is_swa = 0\nload_tensors: layer   5 assigned to device MTL0, is_swa = 0\nload_tensors: layer   6 assigned to device MTL0, is_swa = 0\nload_tensors: layer   7 assigned to device MTL0, is_swa = 0\nload_tensors: layer   8 assigned to device MTL0, is_swa = 0\nload_tensors: layer   9 assigned to device MTL0, is_swa = 0\nload_tensors: layer  10 assigned to device MTL0, is_swa = 0\nload_tensors: layer  11 assigned to device MTL0, is_swa = 0\nload_tensors: layer  12 assigned to device MTL0, is_swa = 0\nload_tensors: layer  13 assigned to device MTL0, is_swa = 0\nload_tensors: layer  14 assigned to device MTL0, is_swa = 0\nload_tensors: layer  15 assigned to device MTL0, is_swa = 0\nload_tensors: layer  16 assigned to device MTL0, is_swa = 0\nload_tensors: layer  17 assigned to device MTL0, is_swa = 0\nload_tensors: layer  18 assigned to device MTL0, is_swa = 0\nload_tensors: layer  19 assigned to device MTL0, is_swa = 0\nload_tensors: layer  20 assigned to device MTL0, is_swa = 0\nload_tensors: layer  21 assigned to device MTL0, is_swa = 0\nload_tensors: layer  22 assigned to device MTL0, is_swa = 0\nload_tensors: layer  23 assigned to device MTL0, is_swa = 0\nload_tensors: layer  24 assigned to device MTL0, is_swa = 0\nload_tensors: layer  25 assigned to device MTL0, is_swa = 0\nload_tensors: layer  26 assigned to device MTL0, is_swa = 0\nload_tensors: layer  27 assigned to device MTL0, is_swa = 0\nload_tensors: layer  28 assigned to device MTL0, is_swa = 0\nload_tensors: layer  29 assigned to device MTL0, is_swa = 0\nload_tensors: layer  30 assigned to device MTL0, is_swa = 0\nload_tensors: layer  31 assigned to device MTL0, is_swa = 0\nload_tensors: layer  32 assigned to device MTL0, is_swa = 0\nload_tensors: layer  33 assigned to device MTL0, is_swa = 0\nload_tensors: layer  34 assigned to device MTL0, is_swa = 0\nload_tensors: layer  35 assigned to device MTL0, is_swa = 0\nload_tensors: layer  36 assigned to device MTL0, is_swa = 0\nload_tensors: layer  37 assigned to device MTL0, is_swa = 0\nload_tensors: layer  38 assigned to device MTL0, is_swa = 0\nload_tensors: layer  39 assigned to device MTL0, is_swa = 0\nload_tensors: layer  40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =   120.59 MiB\nload_tensors:  MTL0_Mapped model buffer size =  4031.55 MiB\nllama_kv_cache: layer   5: dev = MTL0\nllama_kv_cache: layer  15: dev = MTL0\nllama_kv_cache: layer  25: dev = MTL0\nllama_kv_cache: layer  35: dev = MTL0\nllama_kv_cache:       MTL0 KV buffer size =    64.00 MiB\nllama_memory_recurrent, layer   0: dev = MTL0\nllama_memory_recurrent, layer   1: dev = MTL0\nllama_memory_recurrent, layer   2: dev = MTL0\nllama_memory_recurrent, layer   3: dev = MTL0\nllama_memory_recurrent, layer   4: dev = MTL0\nllama_memory_recurrent, layer   6: dev = MTL0\nllama_memory_recurrent, layer   7: dev = MTL0\nllama_memory_recurrent, layer   8: dev = MTL0\nllama_memory_recurrent, layer   9: dev = MTL0\nllama_memory_recurrent, layer  10: dev = MTL0\nllama_memory_recurrent, layer  11: dev = MTL0\nllama_memory_recurrent, layer  12: dev = MTL0\nllama_memory_recurrent, layer  13: dev = MTL0\nllama_memory_recurrent, layer  14: dev = MTL0\nllama_memory_recurrent, layer  16: dev = MTL0\nllama_memory_recurrent, layer  17: dev = MTL0\nllama_memory_recurrent, layer  18: dev = MTL0\nllama_memory_recurrent, layer  19: dev = MTL0\nllama_memory_recurrent, layer  20: dev = MTL0\nllama_memory_recurrent, layer  21: dev = MTL0\nllama_memory_recurrent, layer  22: dev = MTL0\nllama_memory_recurrent, layer  23: dev = MTL0\nllama_memory_recurrent, layer  24: dev = MTL0\nllama_memory_recurrent, layer  26: dev = MTL0\nllama_memory_recurrent, layer  27: dev = MTL0\nllama_memory_recurrent, layer  28: dev = MTL0\nllama_memory_recurrent, layer  29: dev = MTL0\nllama_memory_recurrent, layer  30: dev = MTL0\nllama_memory_recurrent, layer  31: dev = MTL0\nllama_memory_recurrent, layer  32: dev = MTL0\nllama_memory_recurrent, layer  33: dev = MTL0\nllama_memory_recurrent, layer  34: dev = MTL0\nllama_memory_recurrent, layer  36: dev = MTL0\nllama_memory_recurrent, layer  37: dev = MTL0\nllama_memory_recurrent, layer  38: dev = MTL0\nllama_memory_recurrent, layer  39: dev = MTL0\nllama_memory_recurrent:       MTL0 RS buffer size =    55.37 MiB\nsched_reserve:       MTL0 compute buffer size =   123.63 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 41,
      "totalLayers": 41,
      "inferenceMs": 168.81433300295612,
      "durationMs": 184.7548750010901,
      "tokensPerSecond": 41.46567341457566,
      "inputTokens": 72,
      "outputTokens": 7,
      "outputLimitReached": false,
      "rssBytes": 4534534144,
      "cpuThreads": 4,
      "totalMs": 1421.0969590058085,
      "grade": "fast",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": true,
      "expected": "Unknown"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "loadMs": 1236.3420840047183,
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: layer   0 assigned to device MTL0, is_swa = 0\nload_tensors: layer   1 assigned to device MTL0, is_swa = 0\nload_tensors: layer   2 assigned to device MTL0, is_swa = 0\nload_tensors: layer   3 assigned to device MTL0, is_swa = 0\nload_tensors: layer   4 assigned to device MTL0, is_swa = 0\nload_tensors: layer   5 assigned to device MTL0, is_swa = 0\nload_tensors: layer   6 assigned to device MTL0, is_swa = 0\nload_tensors: layer   7 assigned to device MTL0, is_swa = 0\nload_tensors: layer   8 assigned to device MTL0, is_swa = 0\nload_tensors: layer   9 assigned to device MTL0, is_swa = 0\nload_tensors: layer  10 assigned to device MTL0, is_swa = 0\nload_tensors: layer  11 assigned to device MTL0, is_swa = 0\nload_tensors: layer  12 assigned to device MTL0, is_swa = 0\nload_tensors: layer  13 assigned to device MTL0, is_swa = 0\nload_tensors: layer  14 assigned to device MTL0, is_swa = 0\nload_tensors: layer  15 assigned to device MTL0, is_swa = 0\nload_tensors: layer  16 assigned to device MTL0, is_swa = 0\nload_tensors: layer  17 assigned to device MTL0, is_swa = 0\nload_tensors: layer  18 assigned to device MTL0, is_swa = 0\nload_tensors: layer  19 assigned to device MTL0, is_swa = 0\nload_tensors: layer  20 assigned to device MTL0, is_swa = 0\nload_tensors: layer  21 assigned to device MTL0, is_swa = 0\nload_tensors: layer  22 assigned to device MTL0, is_swa = 0\nload_tensors: layer  23 assigned to device MTL0, is_swa = 0\nload_tensors: layer  24 assigned to device MTL0, is_swa = 0\nload_tensors: layer  25 assigned to device MTL0, is_swa = 0\nload_tensors: layer  26 assigned to device MTL0, is_swa = 0\nload_tensors: layer  27 assigned to device MTL0, is_swa = 0\nload_tensors: layer  28 assigned to device MTL0, is_swa = 0\nload_tensors: layer  29 assigned to device MTL0, is_swa = 0\nload_tensors: layer  30 assigned to device MTL0, is_swa = 0\nload_tensors: layer  31 assigned to device MTL0, is_swa = 0\nload_tensors: layer  32 assigned to device MTL0, is_swa = 0\nload_tensors: layer  33 assigned to device MTL0, is_swa = 0\nload_tensors: layer  34 assigned to device MTL0, is_swa = 0\nload_tensors: layer  35 assigned to device MTL0, is_swa = 0\nload_tensors: layer  36 assigned to device MTL0, is_swa = 0\nload_tensors: layer  37 assigned to device MTL0, is_swa = 0\nload_tensors: layer  38 assigned to device MTL0, is_swa = 0\nload_tensors: layer  39 assigned to device MTL0, is_swa = 0\nload_tensors: layer  40 assigned to device MTL0, is_swa = 0\nload_tensors: offloaded 41/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =   120.59 MiB\nload_tensors:  MTL0_Mapped model buffer size =  4031.55 MiB\nllama_kv_cache: layer   5: dev = MTL0\nllama_kv_cache: layer  15: dev = MTL0\nllama_kv_cache: layer  25: dev = MTL0\nllama_kv_cache: layer  35: dev = MTL0\nllama_kv_cache:       MTL0 KV buffer size =    64.00 MiB\nllama_memory_recurrent, layer   0: dev = MTL0\nllama_memory_recurrent, layer   1: dev = MTL0\nllama_memory_recurrent, layer   2: dev = MTL0\nllama_memory_recurrent, layer   3: dev = MTL0\nllama_memory_recurrent, layer   4: dev = MTL0\nllama_memory_recurrent, layer   6: dev = MTL0\nllama_memory_recurrent, layer   7: dev = MTL0\nllama_memory_recurrent, layer   8: dev = MTL0\nllama_memory_recurrent, layer   9: dev = MTL0\nllama_memory_recurrent, layer  10: dev = MTL0\nllama_memory_recurrent, layer  11: dev = MTL0\nllama_memory_recurrent, layer  12: dev = MTL0\nllama_memory_recurrent, layer  13: dev = MTL0\nllama_memory_recurrent, layer  14: dev = MTL0\nllama_memory_recurrent, layer  16: dev = MTL0\nllama_memory_recurrent, layer  17: dev = MTL0\nllama_memory_recurrent, layer  18: dev = MTL0\nllama_memory_recurrent, layer  19: dev = MTL0\nllama_memory_recurrent, layer  20: dev = MTL0\nllama_memory_recurrent, layer  21: dev = MTL0\nllama_memory_recurrent, layer  22: dev = MTL0\nllama_memory_recurrent, layer  23: dev = MTL0\nllama_memory_recurrent, layer  24: dev = MTL0\nllama_memory_recurrent, layer  26: dev = MTL0\nllama_memory_recurrent, layer  27: dev = MTL0\nllama_memory_recurrent, layer  28: dev = MTL0\nllama_memory_recurrent, layer  29: dev = MTL0\nllama_memory_recurrent, layer  30: dev = MTL0\nllama_memory_recurrent, layer  31: dev = MTL0\nllama_memory_recurrent, layer  32: dev = MTL0\nllama_memory_recurrent, layer  33: dev = MTL0\nllama_memory_recurrent, layer  34: dev = MTL0\nllama_memory_recurrent, layer  36: dev = MTL0\nllama_memory_recurrent, layer  37: dev = MTL0\nllama_memory_recurrent, layer  38: dev = MTL0\nllama_memory_recurrent, layer  39: dev = MTL0\nllama_memory_recurrent:       MTL0 RS buffer size =    55.37 MiB\nsched_reserve:       MTL0 compute buffer size =   123.63 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 41,
      "totalLayers": 41,
      "inferenceMs": 107.77562500152271,
      "durationMs": 118.25495799712371,
      "tokensPerSecond": 18.55707169382449,
      "inputTokens": 74,
      "outputTokens": 2,
      "outputLimitReached": false,
      "rssBytes": 4535271424,
      "cpuThreads": 4,
      "totalMs": 1354.597042001842,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "q4",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "plainParity": true,
      "passed": false,
      "expected": "Paris",
      "fixturePassed": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "loadMs": 5334.060290995694,
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =  4029.02 MiB\nload_tensors:   CPU_REPACK model buffer size =  4013.35 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 0,
      "totalLayers": 41,
      "inferenceMs": 451.78691600449383,
      "durationMs": 460.1550000006682,
      "tokensPerSecond": 4.426865695198899,
      "inputTokens": 63,
      "outputTokens": 2,
      "outputLimitReached": false,
      "rssBytes": 8748482560,
      "cpuThreads": 4,
      "totalMs": 5794.2152909963625,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    },
    "failure": "This fixture passed but the backend did not pass the complete repeated profile."
  },
  {
    "precision": "q4",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": false,
      "expected": "56",
      "fixturePassed": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "loadMs": 5334.060290995694,
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =  4029.02 MiB\nload_tensors:   CPU_REPACK model buffer size =  4013.35 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 0,
      "totalLayers": 41,
      "inferenceMs": 428.8514580039191,
      "durationMs": 437.12595800025156,
      "tokensPerSecond": 4.6636194483492295,
      "inputTokens": 63,
      "outputTokens": 2,
      "outputLimitReached": false,
      "rssBytes": 8750235648,
      "cpuThreads": 4,
      "totalMs": 5771.186248995946,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    },
    "failure": "This fixture passed but the backend did not pass the complete repeated profile."
  },
  {
    "precision": "q4",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": false,
      "expected": "chat",
      "fixturePassed": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "loadMs": 5334.060290995694,
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =  4029.02 MiB\nload_tensors:   CPU_REPACK model buffer size =  4013.35 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 0,
      "totalLayers": 41,
      "inferenceMs": 418.81191699940246,
      "durationMs": 427.0358750000014,
      "tokensPerSecond": 4.775413303253387,
      "inputTokens": 68,
      "outputTokens": 2,
      "outputLimitReached": false,
      "rssBytes": 8752988160,
      "cpuThreads": 4,
      "totalMs": 5761.096165995696,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    },
    "failure": "This fixture passed but the backend did not pass the complete repeated profile."
  },
  {
    "precision": "q4",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": false,
      "expected": "hola",
      "fixturePassed": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "loadMs": 5334.060290995694,
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =  4029.02 MiB\nload_tensors:   CPU_REPACK model buffer size =  4013.35 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 0,
      "totalLayers": 41,
      "inferenceMs": 408.98524999647634,
      "durationMs": 417.00212500290945,
      "tokensPerSecond": 4.890151906498416,
      "inputTokens": 69,
      "outputTokens": 2,
      "outputLimitReached": false,
      "rssBytes": 8753217536,
      "cpuThreads": 4,
      "totalMs": 5751.062415998604,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    },
    "failure": "This fixture passed but the backend did not pass the complete repeated profile."
  },
  {
    "precision": "q4",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": false,
      "expected": "2026-11-04",
      "fixturePassed": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "loadMs": 5334.060290995694,
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =  4029.02 MiB\nload_tensors:   CPU_REPACK model buffer size =  4013.35 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 0,
      "totalLayers": 41,
      "inferenceMs": 551.460708004015,
      "durationMs": 559.5853339982568,
      "tokensPerSecond": 12.69356075310634,
      "inputTokens": 72,
      "outputTokens": 7,
      "outputLimitReached": false,
      "rssBytes": 8754282496,
      "cpuThreads": 4,
      "totalMs": 5893.645624993951,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    },
    "failure": "This fixture passed but the backend did not pass the complete repeated profile."
  },
  {
    "precision": "q4",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": false,
      "expected": "Unknown",
      "fixturePassed": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "loadMs": 5334.060290995694,
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =  4029.02 MiB\nload_tensors:   CPU_REPACK model buffer size =  4013.35 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 0,
      "totalLayers": 41,
      "inferenceMs": 530.0160419938038,
      "durationMs": 538.2825420019799,
      "tokensPerSecond": 3.7734706905784203,
      "inputTokens": 74,
      "outputTokens": 2,
      "outputLimitReached": false,
      "rssBytes": 8754462720,
      "cpuThreads": 4,
      "totalMs": 5872.342832997674,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    },
    "failure": "This fixture passed but the backend did not pass the complete repeated profile."
  },
  {
    "precision": "q4",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": false,
      "expected": "Paris"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "loadMs": 4297.2847080018255,
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =  4029.02 MiB\nload_tensors:   CPU_REPACK model buffer size =  4013.35 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 0,
      "totalLayers": 41,
      "inferenceMs": 1443.9783339985297,
      "durationMs": 1453.518833994167,
      "tokensPerSecond": 1.3850623329380485,
      "inputTokens": 63,
      "outputTokens": 2,
      "outputLimitReached": false,
      "rssBytes": 7349977088,
      "cpuThreads": 4,
      "totalMs": 5750.803541995992,
      "grade": "slow",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "q4",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": false,
      "expected": "56"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "loadMs": 4297.2847080018255,
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =  4029.02 MiB\nload_tensors:   CPU_REPACK model buffer size =  4013.35 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 0,
      "totalLayers": 41,
      "inferenceMs": 1736.786374996882,
      "durationMs": 1747.559250004997,
      "tokensPerSecond": 1.151552101509082,
      "inputTokens": 63,
      "outputTokens": 2,
      "outputLimitReached": false,
      "rssBytes": 6324764672,
      "cpuThreads": 4,
      "totalMs": 6044.8439580068225,
      "grade": "slow",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "q4",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": false,
      "expected": "chat"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "loadMs": 4297.2847080018255,
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =  4029.02 MiB\nload_tensors:   CPU_REPACK model buffer size =  4013.35 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 0,
      "totalLayers": 41,
      "inferenceMs": 1108.1044170059613,
      "durationMs": 1117.1746249965508,
      "tokensPerSecond": 1.8048840608395846,
      "inputTokens": 68,
      "outputTokens": 2,
      "outputLimitReached": false,
      "rssBytes": 7378649088,
      "cpuThreads": 4,
      "totalMs": 5414.459332998376,
      "grade": "slow",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "q4",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": false,
      "expected": "hola",
      "fixturePassed": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "loadMs": 4297.2847080018255,
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =  4029.02 MiB\nload_tensors:   CPU_REPACK model buffer size =  4013.35 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 0,
      "totalLayers": 41,
      "inferenceMs": 914.9198749946663,
      "durationMs": 923.8642920026905,
      "tokensPerSecond": 2.1859837726354554,
      "inputTokens": 69,
      "outputTokens": 2,
      "outputLimitReached": false,
      "rssBytes": 7150141440,
      "cpuThreads": 4,
      "totalMs": 5221.149000004516,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    },
    "failure": "This fixture passed but the backend did not pass the complete repeated profile."
  },
  {
    "precision": "q4",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": false,
      "expected": "2026-11-04",
      "fixturePassed": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "loadMs": 4297.2847080018255,
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =  4029.02 MiB\nload_tensors:   CPU_REPACK model buffer size =  4013.35 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 0,
      "totalLayers": 41,
      "inferenceMs": 1209.3559999993886,
      "durationMs": 1220.7180000041262,
      "tokensPerSecond": 5.78820463122814,
      "inputTokens": 72,
      "outputTokens": 7,
      "outputLimitReached": false,
      "rssBytes": 7941373952,
      "cpuThreads": 4,
      "totalMs": 5518.002708005952,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    },
    "failure": "This fixture passed but the backend did not pass the complete repeated profile."
  },
  {
    "precision": "q4",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": false,
      "expected": "Unknown",
      "fixturePassed": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "loadMs": 4297.2847080018255,
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =  4029.02 MiB\nload_tensors:   CPU_REPACK model buffer size =  4013.35 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 0,
      "totalLayers": 41,
      "inferenceMs": 658.3934589943965,
      "durationMs": 672.8037499997299,
      "tokensPerSecond": 3.0376972502957713,
      "inputTokens": 74,
      "outputTokens": 2,
      "outputLimitReached": false,
      "rssBytes": 8082325504,
      "cpuThreads": 4,
      "totalMs": 4970.088458001555,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    },
    "failure": "This fixture passed but the backend did not pass the complete repeated profile."
  },
  {
    "precision": "q4",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "plainParity": true,
      "passed": false,
      "expected": "Paris"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "loadMs": 4240.411375001713,
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =  4029.02 MiB\nload_tensors:   CPU_REPACK model buffer size =  4013.35 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 0,
      "totalLayers": 41,
      "inferenceMs": 1374.3917920000968,
      "durationMs": 1382.5569580003503,
      "tokensPerSecond": 1.455189132852344,
      "inputTokens": 63,
      "outputTokens": 2,
      "outputLimitReached": false,
      "rssBytes": 8748171264,
      "cpuThreads": 1,
      "totalMs": 5622.968333002063,
      "grade": "slow",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "q4",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": false,
      "expected": "56"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "loadMs": 4240.411375001713,
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =  4029.02 MiB\nload_tensors:   CPU_REPACK model buffer size =  4013.35 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 0,
      "totalLayers": 41,
      "inferenceMs": 1357.6007499941625,
      "durationMs": 1365.4761249999865,
      "tokensPerSecond": 1.4731871649368193,
      "inputTokens": 63,
      "outputTokens": 2,
      "outputLimitReached": false,
      "rssBytes": 8749498368,
      "cpuThreads": 1,
      "totalMs": 5605.8875000017,
      "grade": "slow",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "q4",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": false,
      "expected": "chat"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "loadMs": 4240.411375001713,
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =  4029.02 MiB\nload_tensors:   CPU_REPACK model buffer size =  4013.35 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 0,
      "totalLayers": 41,
      "inferenceMs": 1434.74079099542,
      "durationMs": 1443.079624994425,
      "tokensPerSecond": 1.3939800224209171,
      "inputTokens": 68,
      "outputTokens": 2,
      "outputLimitReached": false,
      "rssBytes": 8752103424,
      "cpuThreads": 1,
      "totalMs": 5683.490999996138,
      "grade": "slow",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "q4",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": false,
      "expected": "hola"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "loadMs": 4240.411375001713,
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =  4029.02 MiB\nload_tensors:   CPU_REPACK model buffer size =  4013.35 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 0,
      "totalLayers": 41,
      "inferenceMs": 1456.053082998551,
      "durationMs": 1464.2745000019204,
      "tokensPerSecond": 1.3735762956397588,
      "inputTokens": 69,
      "outputTokens": 2,
      "outputLimitReached": false,
      "rssBytes": 8752758784,
      "cpuThreads": 1,
      "totalMs": 5704.685875003634,
      "grade": "slow",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  },
  {
    "precision": "q4",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": false,
      "expected": "2026-11-04",
      "fixturePassed": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "loadMs": 4240.411375001713,
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =  4029.02 MiB\nload_tensors:   CPU_REPACK model buffer size =  4013.35 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 0,
      "totalLayers": 41,
      "inferenceMs": 1682.3193750024075,
      "durationMs": 1690.0682080013212,
      "tokensPerSecond": 4.160922179232456,
      "inputTokens": 72,
      "outputTokens": 7,
      "outputLimitReached": false,
      "rssBytes": 8754397184,
      "cpuThreads": 1,
      "totalMs": 5930.479583003034,
      "grade": "acceptable",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    },
    "failure": "This fixture passed but the backend did not pass the complete repeated profile."
  },
  {
    "precision": "q4",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "expectedText": true,
      "streamParity": true,
      "withinTokenBudget": true,
      "passed": false,
      "expected": "Unknown"
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "artifactRevision": "08d5a8a9741dd5c1a95d2d39e25253226aa1464e",
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "loadMs": 4240.411375001713,
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nllama_prepare_model_devices: using device MTL0 (Apple M3 Max) (unknown id) - 79625 MiB free\nload_tensors: offloaded 0/41 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =  4029.02 MiB\nload_tensors:   CPU_REPACK model buffer size =  4013.35 MiB",
      "parameterDtype": "Q4_K_M mixed tensor types",
      "offloadedLayers": 0,
      "totalLayers": 41,
      "inferenceMs": 1633.4471669979393,
      "durationMs": 1641.847124999913,
      "tokensPerSecond": 1.2244044621753738,
      "inputTokens": 74,
      "outputTokens": 2,
      "outputLimitReached": false,
      "rssBytes": 8756117504,
      "cpuThreads": 1,
      "totalMs": 5882.258500001626,
      "grade": "slow",
      "loadTimingScope": "Local snapshot load with one-token warmup. Excludes downloads."
    }
  }
]

Something not quite right?

Tell us what happened with Granite 4.0. Loading, results, or the page itself—we’re here for all of it.

Attachments Optional

Up to 3 files · 50 MB each

Browser, OS, CPU/GPU, memory, screen, storage, and network details are included automatically. Model inputs, results, and browsing history are not collected.