A model is still running

A model is still running or downloading. Leaving will stop the current operation. Stay on this page to let it finish.

Skip to content
LocalAI.toolsBeta

Join the waitlist

Get an email when the new apps and model runs launch.

No account needed.

SmolLM3 Desktop

General chat, drafting and short multilingual replies locally. Strict-format output is not guaranteed.

Desktop inferencePrivate · Runs on your computer
About this model

General chat, drafting and short multilingual replies locally. Strict-format output is not guaranteed.

Created by
Hugging Face
License
Apache-2.0
Model released
Jul 8, 2025
Source repository created
Jul 8, 2025
Model weights · FP16
6.2 GB

Best for

  • Local chat and drafting
  • Short multilingual text tasks

Keep in mind

  • May hallucinate facts, produce unsafe content or fail instructions; review output.
  • Desktop adapter limits input plus output budget to 8192 tokens; upstream long-context claims are not validated here.
  • Six primary upstream languages; local tests cover only short English, French and Spanish prompts.
  • Requires a multi-gigabyte download and suitable memory. Performance depends on runtime and hardware.
  • May add commentary despite exact-format instructions; not verified for strict extraction or structured output.
  • Strict-format extraction failed: the model added commentary around a requested date. General-chat scope was explicitly approved; do not rely on exact formatting.
  • Extended thinking is disabled in this general-chat integration. Browser quantizations repeated/truncated reasoning at 1024 tokens. Native direct probes can complete with larger budgets, but this feature is outside the shipped scope.

Processed on your deviceYour input is processed locally in the desktop app. Model and runtime files download from external hosts.

Target platforms · verification required

  • macos · aarch64, x86_64
  • windows · x86_64

Recommended configuration

FP16 · mps (verified platform only)

Sources & technical details
Original model
HuggingFaceTB/SmolLM3-3B
Release verification
Verified
Release evidence
Official launch article explicitly releases this instruct/reasoning checkpoint on July 8, 2025.
Source updated
Sep 10, 2025
Native model files
HuggingFaceTB/SmolLM3-3B
Files updated
Sep 10, 2025
Artifact revision
a07cc9a04f16550a088caea529712d1d335b0ac1

Desktop app

Sample result

SmolLM3 Desktop

Run SmolLM3 Desktop on your computer

Download the desktop app to load this model, provide input and view results locally.

Public desktop downloads are coming soon.

Device requirements

macOS Apple SiliconCPU / FP32Tested22.4 GiBRAM estimate
macOS Apple SiliconMPS / FP16Tested14.9 GiBRAM estimate
Windows Intel / AMDCPU / FP32Unverified22.4 GiBRAM estimate
macOS Intel / AMDCPU / FP32Unverified22.4 GiBRAM estimate
If your device falls short

MemoryAdvisory only: close other apps, reduce image/tile/token/batch settings or choose a smaller verified model/precision. CPU can need more RAM than GPU.

GPUIf the selected GPU runtime fails, existing CPU fallback may recover when enabled, but CPU has its own memory/runtime needs.

PlatformUse a compatible OS/architecture/runtime or another model. These estimates do not add hardware detection or block runs.

Benchmark evidence

Saved desktop measurements apply to the recorded hardware and runtime. Fallback cases report the device that actually ran; they do not prove GPU support on the requested device.

SmolLM3 Desktop

42 configurations tested
Passed35
Failed7
TestedNov 4, 2026
Runtimetransformers 5.22.1

Quality criteria

No explicit quality gate recorded. Review per-case evidence; full certification is pending.

PrecisionDeviceResultPlatformInference / load
FP16mpsAcceleratedPassedModeratemacos · aarch64151 ms / 1724 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64152 ms / 1583 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64153 ms / 1724 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64153 ms / 1664 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64155 ms / 1583 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64155 ms / 1664 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64178 ms / 1664 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64181 ms / 1664 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64186 ms / 1724 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64187 ms / 1583 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64187 ms / 1724 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64192 ms / 1583 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64195 ms / 1583 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64196 ms / 1724 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64196 ms / 1664 ms
FP32cpuPassedModeratemacos · aarch64511 ms / 1284 ms
FP32cpuPassedModeratemacos · aarch64515 ms / 1284 ms
FP32cpuPassedModeratemacos · aarch64521 ms / 1284 ms
FP32cpuPassedModeratemacos · aarch64522 ms / 1364 ms
FP32cpuPassedModeratemacos · aarch64522 ms / 1449 ms
FP32cpuPassedModeratemacos · aarch64531 ms / 1364 ms
FP32cpuPassedModeratemacos · aarch64531 ms / 1432 ms
FP32cpuPassedModeratemacos · aarch64534 ms / 1449 ms
FP32cpuPassedModeratemacos · aarch64536 ms / 1364 ms
FP32cpuPassedModeratemacos · aarch64537 ms / 1432 ms
FP32cpuPassedModeratemacos · aarch64537 ms / 1364 ms
FP32cpuPassedModeratemacos · aarch64542 ms / 1449 ms
FP32cpuPassedModeratemacos · aarch64542 ms / 1284 ms
FP32cpuPassedModeratemacos · aarch64544 ms / 1432 ms
FP32cpuPassedModeratemacos · aarch64546 ms / 1449 ms
FP32cpuPassedModeratemacos · aarch64551 ms / 1449 ms
FP32cpuPassedModeratemacos · aarch64571 ms / 1432 ms
FP32cpuPassedModeratemacos · aarch64582 ms / 1432 ms
FP32cpuPassedModeratemacos · aarch64587 ms / 1364 ms
FP32cpuPassedModeratemacos · aarch64611 ms / 1284 ms
FP32cpuFailed—macos · aarch642288 ms / 1449 ms
FP32cpuFailed—macos · aarch642748 ms / 1284 ms
FP32cpuFailed—macos · aarch642344 ms / 1432 ms
FP32cpuFailed—macos · aarch642461 ms / 1364 ms
FP16mpsFailed—macos · aarch64549 ms / 1724 ms
FP16mpsFailed—macos · aarch64558 ms / 1583 ms
FP16mpsFailed—macos · aarch64541 ms / 1664 ms
Test setup & sample details
Quality gate
No explicit quality gate recorded. Review the per-case quality evidence; full certification is pending.
Tested
2026-11-04T09:43:32.286323+00:00
Environment
{ "os": "macos", "arch": "aarch64", "hardware": "Apple M3 Max", "osVersion": "27.2" }
Runtime
transformers 5.22.1
Artifact revision
a07cc9a04f16550a088caea529712d1d335b0ac1
Fixture
{
  "file": "model-samples/desktop/smollm3/smollm3-3b-native-capital.json",
  "size": 244,
  "sha1": "9013b4f5a71cd1280d78b3dfa751ead1186d071f",
  "sha256": "8ff34789fa2a0f5bbef2080057f4b36376d3c73ea64baccefb6caec96eafbd68",
  "url": "https://assets.localai.tools/samples/v1/files/8ff34789fa2a0f5bbef2080057f4b36376d3c73ea64baccefb6caec96eafbd68/smollm3-3b-native-capital.json"
}
Rerun when
Artifact/runtime revisions; Generation/preprocessing changes; Fixture/quality scope; OS/backend behavior
Per-case quality & failure details
[
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "exactNormalized": true,
      "expected": "Paris",
      "scope": "general-chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1448.5713749891147,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 326
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 553.6669589928351,
      "inferenceMs": 546.4499579975381,
      "tokensPerSecond": 3.659987471366976,
      "inputTokens": 82,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 1,
      "peakHelperRssBytes": 12849774592,
      "memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
      "artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
      "totalMs": 2002.2383339819498,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "exactNormalized": true,
      "expected": "56",
      "scope": "general-chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1448.5713749891147,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 326
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 522.884082980454,
      "inferenceMs": 522.4799169809557,
      "tokensPerSecond": 3.827898326803821,
      "inputTokens": 82,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 1,
      "peakHelperRssBytes": 12853297152,
      "memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
      "artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
      "totalMs": 1971.4554579695687,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "exactNormalized": true,
      "expected": "chat",
      "scope": "general-chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1448.5713749891147,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 326
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 541.8647920014337,
      "inferenceMs": 541.5140829863958,
      "tokensPerSecond": 3.693348082417729,
      "inputTokens": 87,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 1,
      "peakHelperRssBytes": 12870057984,
      "memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
      "artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
      "totalMs": 1990.4361669905484,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "exactNormalized": true,
      "expected": "hola",
      "scope": "general-chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1448.5713749891147,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 326
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 551.3195410021581,
      "inferenceMs": 550.8882080030162,
      "tokensPerSecond": 3.6305006550240946,
      "inputTokens": 88,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 1,
      "peakHelperRssBytes": 12883214336,
      "memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
      "artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
      "totalMs": 1999.8909159912728,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "passed": false,
      "exactNormalized": false,
      "expected": "2026-11-04",
      "scope": "diagnostic-strict-format"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1448.5713749891147,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 326
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 2288.609541981714,
      "inferenceMs": 2288.246625015745,
      "tokensPerSecond": 6.992253293453413,
      "inputTokens": 91,
      "outputTokens": 16,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 1,
      "peakHelperRssBytes": 12901138432,
      "memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
      "artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
      "totalMs": 3737.1809169708285,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "exactNormalized": true,
      "expected": "Paris",
      "scope": "general-chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1448.5713749891147,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 326
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 534.2767080001067,
      "inferenceMs": 533.941583998967,
      "tokensPerSecond": 3.7457281094702477,
      "inputTokens": 82,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 1,
      "peakHelperRssBytes": 12901138432,
      "memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
      "artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
      "totalMs": 1982.8480829892214,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "exactNormalized": true,
      "expected": "Paris",
      "scope": "general-chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1284.4667080207728,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 326
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 549.6564589848276,
      "inferenceMs": 541.5752919798251,
      "tokensPerSecond": 3.692930659167709,
      "inputTokens": 82,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "peakHelperRssBytes": 12846383104,
      "memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
      "artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
      "totalMs": 1834.1231670056004,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "exactNormalized": true,
      "expected": "56",
      "scope": "general-chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1284.4667080207728,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 326
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 515.1305420149583,
      "inferenceMs": 514.6995830000378,
      "tokensPerSecond": 3.8857618425539955,
      "inputTokens": 82,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "peakHelperRssBytes": 12847448064,
      "memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
      "artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
      "totalMs": 1799.5972500357311,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "exactNormalized": true,
      "expected": "chat",
      "scope": "general-chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1284.4667080207728,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 326
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 511.64462498854846,
      "inferenceMs": 511.2174169917125,
      "tokensPerSecond": 3.9122297745039907,
      "inputTokens": 87,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "peakHelperRssBytes": 12876529664,
      "memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
      "artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
      "totalMs": 1796.1113330093212,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "exactNormalized": true,
      "expected": "hola",
      "scope": "general-chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1284.4667080207728,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 326
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 521.4637909957673,
      "inferenceMs": 521.062083018478,
      "tokensPerSecond": 3.8383142147172427,
      "inputTokens": 88,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "peakHelperRssBytes": 12888981504,
      "memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
      "artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
      "totalMs": 1805.9304990165401,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "passed": false,
      "exactNormalized": false,
      "expected": "2026-11-04",
      "scope": "diagnostic-strict-format"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1284.4667080207728,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 326
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 2748.0667080089916,
      "inferenceMs": 2747.6530830026604,
      "tokensPerSecond": 5.8231514374860796,
      "inputTokens": 91,
      "outputTokens": 16,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "peakHelperRssBytes": 12903923712,
      "memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
      "artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
      "totalMs": 4032.5334160297643,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "exactNormalized": true,
      "expected": "Paris",
      "scope": "general-chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1284.4667080207728,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 326
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 611.310209002113,
      "inferenceMs": 610.8436659851577,
      "tokensPerSecond": 3.2741601679284598,
      "inputTokens": 82,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "peakHelperRssBytes": 12903940096,
      "memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
      "artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
      "totalMs": 1895.7769170228858,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "exactNormalized": true,
      "expected": "Paris",
      "scope": "general-chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1431.9112499943003,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 326
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 544.5574159966782,
      "inferenceMs": 536.6443329839967,
      "tokensPerSecond": 3.726863170023715,
      "inputTokens": 82,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "peakHelperRssBytes": 12847661056,
      "memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
      "artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
      "totalMs": 1976.4686659909785,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "exactNormalized": true,
      "expected": "56",
      "scope": "general-chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1431.9112499943003,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 326
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 582.4415419774596,
      "inferenceMs": 582.0458329981193,
      "tokensPerSecond": 3.436155516650632,
      "inputTokens": 82,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "peakHelperRssBytes": 12848119808,
      "memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
      "artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
      "totalMs": 2014.3527919717599,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "exactNormalized": true,
      "expected": "chat",
      "scope": "general-chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1431.9112499943003,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 326
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 571.7759580002166,
      "inferenceMs": 571.3317499903496,
      "tokensPerSecond": 3.5005931318078893,
      "inputTokens": 87,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "peakHelperRssBytes": 12859703296,
      "memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
      "artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
      "totalMs": 2003.687207994517,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "exactNormalized": true,
      "expected": "hola",
      "scope": "general-chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1431.9112499943003,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 326
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 544.6160410065204,
      "inferenceMs": 544.201166019775,
      "tokensPerSecond": 3.675111566973976,
      "inputTokens": 88,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "peakHelperRssBytes": 12876873728,
      "memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
      "artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
      "totalMs": 1976.5272910008207,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "passed": false,
      "exactNormalized": false,
      "expected": "2026-11-04",
      "scope": "diagnostic-strict-format"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1431.9112499943003,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 326
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 2344.899040996097,
      "inferenceMs": 2344.4360840076115,
      "tokensPerSecond": 6.824668886962948,
      "inputTokens": 91,
      "outputTokens": 16,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "peakHelperRssBytes": 12903743488,
      "memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
      "artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
      "totalMs": 3776.8102909903973,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "exactNormalized": true,
      "expected": "Paris",
      "scope": "general-chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1431.9112499943003,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 326
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 531.6927919921,
      "inferenceMs": 531.2627909879666,
      "tokensPerSecond": 3.7646152411327094,
      "inputTokens": 82,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "peakHelperRssBytes": 12903759872,
      "memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
      "artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
      "totalMs": 1963.6040419864003,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "exactNormalized": true,
      "expected": "Paris",
      "scope": "general-chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1364.149791013915,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 326
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 594.6649170073215,
      "inferenceMs": 586.9804579997435,
      "tokensPerSecond": 3.407268457991618,
      "inputTokens": 82,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "peakHelperRssBytes": 12847251456,
      "memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
      "artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
      "totalMs": 1958.8147080212366,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "exactNormalized": true,
      "expected": "56",
      "scope": "general-chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1364.149791013915,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 326
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 536.728334001964,
      "inferenceMs": 536.3179999985732,
      "tokensPerSecond": 3.7291308514823682,
      "inputTokens": 82,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "peakHelperRssBytes": 12848742400,
      "memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
      "artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
      "totalMs": 1900.878125015879,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "exactNormalized": true,
      "expected": "chat",
      "scope": "general-chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1364.149791013915,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 326
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 537.1224579866976,
      "inferenceMs": 536.7711249855347,
      "tokensPerSecond": 3.725982838689204,
      "inputTokens": 87,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "peakHelperRssBytes": 12871221248,
      "memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
      "artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
      "totalMs": 1901.2722490006126,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "exactNormalized": true,
      "expected": "hola",
      "scope": "general-chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1364.149791013915,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 326
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 522.4633339967113,
      "inferenceMs": 521.9061250099912,
      "tokensPerSecond": 3.8321067796660033,
      "inputTokens": 88,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "peakHelperRssBytes": 12888211456,
      "memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
      "artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
      "totalMs": 1886.6131250106264,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "fail",
    "quality": {
      "passed": false,
      "exactNormalized": false,
      "expected": "2026-11-04",
      "scope": "diagnostic-strict-format"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1364.149791013915,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 326
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 2461.52837498812,
      "inferenceMs": 2461.1574160226155,
      "tokensPerSecond": 6.501006354098634,
      "inputTokens": 91,
      "outputTokens": 16,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "peakHelperRssBytes": 12903841792,
      "memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
      "artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
      "totalMs": 3825.678166002035,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "exactNormalized": true,
      "expected": "Paris",
      "scope": "general-chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1364.149791013915,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 326
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 531.0165830014739,
      "inferenceMs": 530.6597090093419,
      "tokensPerSecond": 3.768893635685447,
      "inputTokens": 82,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "peakHelperRssBytes": 12903890944,
      "memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
      "artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
      "totalMs": 1895.166374015389,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "exactNormalized": true,
      "expected": "Paris",
      "scope": "general-chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1723.9172920235433,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 326
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 205.20191700779833,
      "inferenceMs": 195.98945797770284,
      "tokensPerSecond": 10.20463049715426,
      "inputTokens": 82,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 6530220032,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "precision": "fp16",
      "cpuThreads": 4,
      "peakHelperRssBytes": 2719154176,
      "memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
      "artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
      "totalMs": 1929.1192090313416,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "exactNormalized": true,
      "expected": "56",
      "scope": "general-chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1723.9172920235433,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 326
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 154.83412498724647,
      "inferenceMs": 153.10120797948912,
      "tokensPerSecond": 13.06325421199772,
      "inputTokens": 82,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 6530203648,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "precision": "fp16",
      "cpuThreads": 4,
      "peakHelperRssBytes": 2719891456,
      "memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
      "artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
      "totalMs": 1878.7514170107897,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "exactNormalized": true,
      "expected": "chat",
      "scope": "general-chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1723.9172920235433,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 326
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 188.111374998698,
      "inferenceMs": 185.74812498991378,
      "tokensPerSecond": 10.767268849193504,
      "inputTokens": 87,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 6539624448,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "precision": "fp16",
      "cpuThreads": 4,
      "peakHelperRssBytes": 2729558016,
      "memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
      "artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
      "totalMs": 1912.0286670222413,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "exactNormalized": true,
      "expected": "hola",
      "scope": "general-chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1723.9172920235433,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 326
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 189.1775419935584,
      "inferenceMs": 187.25770901073702,
      "tokensPerSecond": 10.680468166388406,
      "inputTokens": 88,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 6540673024,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "precision": "fp16",
      "cpuThreads": 4,
      "peakHelperRssBytes": 2739109888,
      "memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
      "artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
      "totalMs": 1913.0948340171017,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "fail",
    "quality": {
      "passed": false,
      "exactNormalized": false,
      "expected": "2026-11-04",
      "scope": "diagnostic-strict-format"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1723.9172920235433,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 326
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 551.2816250266042,
      "inferenceMs": 549.4946249818895,
      "tokensPerSecond": 29.117664254727398,
      "inputTokens": 91,
      "outputTokens": 16,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 6541803520,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "precision": "fp16",
      "cpuThreads": 4,
      "peakHelperRssBytes": 2749186048,
      "memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
      "artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
      "totalMs": 2275.1989170501474,
      "grade": "fast",
      "loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "exactNormalized": true,
      "expected": "Paris",
      "scope": "general-chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1723.9172920235433,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 326
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 153.5433329991065,
      "inferenceMs": 151.2529170140624,
      "tokensPerSecond": 13.222885478724715,
      "inputTokens": 82,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 6541803520,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "precision": "fp16",
      "cpuThreads": 4,
      "peakHelperRssBytes": 2749628416,
      "memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
      "artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
      "totalMs": 1877.4606250226498,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "exactNormalized": true,
      "expected": "Paris",
      "scope": "general-chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1583.3670000138227,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 326
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 203.67754099424928,
      "inferenceMs": 194.61916599539109,
      "tokensPerSecond": 10.276480169724719,
      "inputTokens": 82,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 6530220032,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "precision": "fp16",
      "cpuThreads": 4,
      "peakHelperRssBytes": 2719744000,
      "memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
      "artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
      "totalMs": 1787.044541008072,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "exactNormalized": true,
      "expected": "56",
      "scope": "general-chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1583.3670000138227,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 326
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 156.57429199200124,
      "inferenceMs": 154.56175000872463,
      "tokensPerSecond": 12.939812080848625,
      "inputTokens": 82,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 6530220032,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "precision": "fp16",
      "cpuThreads": 4,
      "peakHelperRssBytes": 2720595968,
      "memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
      "artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
      "totalMs": 1739.941292005824,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "exactNormalized": true,
      "expected": "chat",
      "scope": "general-chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1583.3670000138227,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 326
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 188.49683398730122,
      "inferenceMs": 186.79758301004767,
      "tokensPerSecond": 10.706776649740815,
      "inputTokens": 87,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 6539624448,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "precision": "fp16",
      "cpuThreads": 4,
      "peakHelperRssBytes": 2729885696,
      "memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
      "artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
      "totalMs": 1771.863834001124,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "exactNormalized": true,
      "expected": "hola",
      "scope": "general-chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1583.3670000138227,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 326
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 193.88779098517261,
      "inferenceMs": 192.10408398066647,
      "tokensPerSecond": 10.411022808871058,
      "inputTokens": 88,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 6540673024,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "precision": "fp16",
      "cpuThreads": 4,
      "peakHelperRssBytes": 2739077120,
      "memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
      "artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
      "totalMs": 1777.2547909989953,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "fail",
    "quality": {
      "passed": false,
      "exactNormalized": false,
      "expected": "2026-11-04",
      "scope": "diagnostic-strict-format"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1583.3670000138227,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 326
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 559.2794999829493,
      "inferenceMs": 557.5622920005117,
      "tokensPerSecond": 28.69634519686155,
      "inputTokens": 91,
      "outputTokens": 16,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 6541803520,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "precision": "fp16",
      "cpuThreads": 4,
      "peakHelperRssBytes": 2748809216,
      "memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
      "artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
      "totalMs": 2142.646499996772,
      "grade": "fast",
      "loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "exactNormalized": true,
      "expected": "Paris",
      "scope": "general-chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1583.3670000138227,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 326
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 154.20708298915997,
      "inferenceMs": 152.4789580143988,
      "tokensPerSecond": 13.116563924913082,
      "inputTokens": 82,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 6541803520,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "precision": "fp16",
      "cpuThreads": 4,
      "peakHelperRssBytes": 2749251584,
      "memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
      "artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
      "totalMs": 1737.5740830029827,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "exactNormalized": true,
      "expected": "Paris",
      "scope": "general-chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1663.9372499776073,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 326
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 205.40504198288545,
      "inferenceMs": 196.4990830165334,
      "tokensPerSecond": 10.178164545590883,
      "inputTokens": 82,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 6530220032,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "precision": "fp16",
      "cpuThreads": 4,
      "peakHelperRssBytes": 2718875648,
      "memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
      "artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
      "totalMs": 1869.3422919604927,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "exactNormalized": true,
      "expected": "56",
      "scope": "general-chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1663.9372499776073,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 326
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 157.03620901331306,
      "inferenceMs": 155.24758299579844,
      "tokensPerSecond": 12.882648228115263,
      "inputTokens": 82,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 6530220032,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "precision": "fp16",
      "cpuThreads": 4,
      "peakHelperRssBytes": 2719449088,
      "memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
      "artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
      "totalMs": 1820.9734589909203,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "exactNormalized": true,
      "expected": "chat",
      "scope": "general-chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1663.9372499776073,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 326
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 182.60575001477264,
      "inferenceMs": 181.0010000190232,
      "tokensPerSecond": 11.049662707884487,
      "inputTokens": 87,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 6539624448,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "precision": "fp16",
      "cpuThreads": 4,
      "peakHelperRssBytes": 2729099264,
      "memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
      "artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
      "totalMs": 1846.54299999238,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "exactNormalized": true,
      "expected": "hola",
      "scope": "general-chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1663.9372499776073,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 326
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 180.093915987527,
      "inferenceMs": 178.21354197803885,
      "tokensPerSecond": 11.222491724262227,
      "inputTokens": 88,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 6540673024,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "precision": "fp16",
      "cpuThreads": 4,
      "peakHelperRssBytes": 2738487296,
      "memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
      "artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
      "totalMs": 1844.0311659651343,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "fail",
    "quality": {
      "passed": false,
      "exactNormalized": false,
      "expected": "2026-11-04",
      "scope": "diagnostic-strict-format"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1663.9372499776073,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 326
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 542.4867499968968,
      "inferenceMs": 540.7002919819206,
      "tokensPerSecond": 29.591254595688273,
      "inputTokens": 91,
      "outputTokens": 16,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 6541803520,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "precision": "fp16",
      "cpuThreads": 4,
      "peakHelperRssBytes": 2748497920,
      "memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
      "artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
      "totalMs": 2206.423999974504,
      "grade": "fast",
      "loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "exactNormalized": true,
      "expected": "Paris",
      "scope": "general-chat"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 1663.9372499776073,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 326
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 155.09383301832713,
      "inferenceMs": 153.16183300456032,
      "tokensPerSecond": 13.058083471359677,
      "inputTokens": 82,
      "outputTokens": 2,
      "maxNewTokens": 32,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 6541803520,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "precision": "fp16",
      "cpuThreads": 4,
      "peakHelperRssBytes": 2748825600,
      "memoryScope": "Instantaneous helper RSS after request, not peak/minimum",
      "artifactRevision": "a07cc9a04f16550a088caea529712d1d335b0ac1",
      "totalMs": 1819.0310829959344,
      "grade": "acceptable",
      "loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
    }
  }
]

Something not quite right?

Tell us what happened with SmolLM3 Desktop. Loading, results, or the page itself—we’re here for all of it.

Attachments Optional

Up to 3 files · 50 MB each

Browser, OS, CPU/GPU, memory, screen, storage, and network details are included automatically. Model inputs, results, and browsing history are not collected.