Liquid LFM3 Desktop
Local text chat with Liquid LFM3 1.7B.
Desktop app
Run Liquid LFM3 Desktop on your computer
Download the desktop app to load this model, provide input and view results locally.
Public desktop downloads are coming soon.Device requirements
macOS Apple SiliconCPU / Q4TestedNot establishedRAM estimate
macOS Apple SiliconMETAL / Q4TestedNot establishedRAM estimate
If your device falls short
MemoryShow advisory requirements.
GPUUse verified CPU recovery when permitted.
PlatformNo verified configuration.
Benchmark evidence
Saved desktop measurements apply to the recorded hardware and runtime. Fallback cases report the device that actually ran; they do not prove GPU support on the requested device.
LFM3 Omni 2B Desktop
10 configurations testedPassed10
Failed0
TestedNov 15, 2026
Runtimellama.cpp b12359
Quality criteria
No explicit quality gate recorded. Review per-case evidence; full certification is pending.
| Precision | Device | Result | Platform | Inference / load | |
|---|---|---|---|---|---|
Q4 | metalAccelerated | Passed | Very fast | macos · aarch64 | 88 ms / 754 ms |
Q4 | metalAccelerated | Passed | Fast | macos · aarch64 | 228 ms / 754 ms |
Q4 | metalAccelerated | Passed | Fast | macos · aarch64 | 264 ms / 754 ms |
Q4 | metalAccelerated | Passed | Fast | macos · aarch64 | 394 ms / 754 ms |
Q4 | metalAccelerated | Passed | Fast | macos · aarch64 | 625 ms / 754 ms |
Q4 | cpu | Passed | Fast | macos · aarch64 | 107 ms / 7635 ms |
Q4 | cpu | Passed | Fast | macos · aarch64 | 477 ms / 7635 ms |
Q4 | cpu | Passed | Fast | macos · aarch64 | 640 ms / 7635 ms |
Q4 | cpu | Passed | Fast | macos · aarch64 | 846 ms / 7635 ms |
Q4 | cpu | Passed | Fast | macos · aarch64 | 933 ms / 7635 ms |
Test setup & sample details
- Quality gate
- No explicit quality gate recorded. Review the per-case quality evidence; full certification is pending.
- Tested
- 2026-11-15T22:36:20.409Z
- Environment
- { "scope": "DesktopAI TypeScript client using native JSON-lines transport with verified cached sources.", "hardware": "Apple M3 Max", "os": "macOS", "arch": "arm64", "runtime": { "name": "llama.cpp", "version": "b12359" }, "speedGradePolicy": "Short requests completing in 2 seconds or less are interactive." }
- Runtime
- llama.cpp b12359
- Artifact revision
- d508b88f08a5b4286f2d2805a6a8d69a0ff44c72
- Fixture
{ "file": "model-samples/voxtral-realtime/jfk.wav", "url": "https://assets.localai.tools/samples/v1/files/4d968ac99a1d0d4bc42ae8dd1552f4235e7b952a8121b39797f3dc2ca16123e9/jfk.wav", "size": 352078, "sha1": "069db705beb788608e93b99f1a4300451ebfb972", "sha256": "4d968ac99a1d0d4bc42ae8dd1552f4235e7b952a8121b39797f3dc2ca16123e9" }- Rerun when
- artifactRevision; runtimeVersion; fixtureChecksum; qualityGate; nativeBackend
Per-case quality & failure details
[
{
"precision": "q4",
"device": "cpu",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true
},
"runtime": {
"acceleration": "cpu",
"cpuThreads": 4,
"device": "cpu",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 107.171584,
"loadMs": 7634.52925,
"offloadedLayers": 0,
"precision": "q4",
"provider": "llama.cpp CPU",
"requestedDevice": "cpu",
"requestedPrecision": "q4",
"selectionReason": "Exact Q4_0 package with verified layer placement",
"version": "b12359",
"artifactRevision": "d508b88f08a5b4286f2d2805a6a8d69a0ff44c72",
"totalMs": 7741.700833999999,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 18.661663151306975
},
"grade": "good",
"platform": "macos",
"arch": "aarch64"
}
},
{
"precision": "q4",
"device": "metal",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true
},
"runtime": {
"acceleration": "partial",
"cpuThreads": 4,
"device": "metal",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 87.70400000000001,
"loadMs": 753.985542,
"offloadedLayers": 25,
"precision": "q4",
"provider": "llama.cpp Metal",
"requestedDevice": "metal",
"requestedPrecision": "q4",
"selectionReason": "Exact Q4_0 package with verified layer placement",
"version": "b12359",
"artifactRevision": "d508b88f08a5b4286f2d2805a6a8d69a0ff44c72",
"totalMs": 841.6895420000001,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 22.80397701359117
},
"grade": "good",
"platform": "macos",
"arch": "aarch64"
}
},
{
"precision": "q4",
"device": "cpu",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true
},
"runtime": {
"acceleration": "cpu",
"cpuThreads": 4,
"device": "cpu",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 477.129041,
"loadMs": 7634.52925,
"offloadedLayers": 0,
"precision": "q4",
"provider": "llama.cpp CPU",
"requestedDevice": "cpu",
"requestedPrecision": "q4",
"selectionReason": "Exact Q4_0 package with verified layer placement",
"version": "b12359",
"visualizationMs": 0.018292,
"artifactRevision": "d508b88f08a5b4286f2d2805a6a8d69a0ff44c72",
"totalMs": 8111.658291,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 6.287607213579775
},
"grade": "good",
"platform": "macos",
"arch": "aarch64"
}
},
{
"precision": "q4",
"device": "metal",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true
},
"runtime": {
"acceleration": "partial",
"cpuThreads": 4,
"device": "metal",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 228.486834,
"loadMs": 753.985542,
"offloadedLayers": 25,
"precision": "q4",
"provider": "llama.cpp Metal",
"requestedDevice": "metal",
"requestedPrecision": "q4",
"selectionReason": "Exact Q4_0 package with verified layer placement",
"version": "b12359",
"visualizationMs": 0.036875000000000005,
"artifactRevision": "d508b88f08a5b4286f2d2805a6a8d69a0ff44c72",
"totalMs": 982.4723759999999,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 13.12985937736789
},
"grade": "good",
"platform": "macos",
"arch": "aarch64"
}
},
{
"precision": "q4",
"device": "cpu",
"status": "pass",
"quality": {
"metric": "normalizedWordMatch",
"score": 1,
"passed": true
},
"runtime": {
"acceleration": "cpu",
"cpuThreads": 4,
"device": "cpu",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 845.560792,
"loadMs": 7634.52925,
"offloadedLayers": 0,
"precision": "q4",
"provider": "llama.cpp CPU",
"requestedDevice": "cpu",
"requestedPrecision": "q4",
"selectionReason": "Exact Q4_0 package with verified layer placement",
"version": "b12359",
"artifactRevision": "d508b88f08a5b4286f2d2805a6a8d69a0ff44c72",
"totalMs": 8480.090042,
"performanceMetric": {
"name": "audioSecondsPerSecond",
"value": 13.009117858908482
},
"grade": "good",
"platform": "macos",
"arch": "aarch64"
}
},
{
"precision": "q4",
"device": "metal",
"status": "pass",
"quality": {
"metric": "normalizedWordMatch",
"score": 1,
"passed": true
},
"runtime": {
"acceleration": "partial",
"cpuThreads": 4,
"device": "metal",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 393.675,
"loadMs": 753.985542,
"offloadedLayers": 25,
"precision": "q4",
"provider": "llama.cpp Metal",
"requestedDevice": "metal",
"requestedPrecision": "q4",
"selectionReason": "Exact Q4_0 package with verified layer placement",
"version": "b12359",
"artifactRevision": "d508b88f08a5b4286f2d2805a6a8d69a0ff44c72",
"totalMs": 1147.660542,
"performanceMetric": {
"name": "audioSecondsPerSecond",
"value": 27.941830189877436
},
"grade": "good",
"platform": "macos",
"arch": "aarch64"
}
},
{
"precision": "q4",
"device": "cpu",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true
},
"runtime": {
"acceleration": "cpu",
"cpuThreads": 4,
"device": "cpu",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 639.875333,
"loadMs": 7634.52925,
"offloadedLayers": 0,
"precision": "q4",
"provider": "llama.cpp CPU",
"requestedDevice": "cpu",
"requestedPrecision": "q4",
"selectionReason": "Exact Q4_0 package with verified layer placement",
"version": "b12359",
"artifactRevision": "d508b88f08a5b4286f2d2805a6a8d69a0ff44c72",
"totalMs": 8274.404583,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 14.065239798828127
},
"grade": "good",
"platform": "macos",
"arch": "aarch64"
}
},
{
"precision": "q4",
"device": "metal",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true
},
"runtime": {
"acceleration": "partial",
"cpuThreads": 4,
"device": "metal",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 264.48062500000003,
"loadMs": 753.985542,
"offloadedLayers": 25,
"precision": "q4",
"provider": "llama.cpp Metal",
"requestedDevice": "metal",
"requestedPrecision": "q4",
"selectionReason": "Exact Q4_0 package with verified layer placement",
"version": "b12359",
"artifactRevision": "d508b88f08a5b4286f2d2805a6a8d69a0ff44c72",
"totalMs": 1018.466167,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 34.028957697751956
},
"grade": "good",
"platform": "macos",
"arch": "aarch64"
}
},
{
"precision": "q4",
"device": "cpu",
"status": "pass",
"quality": {
"metric": "independentWhisperWordMatch",
"score": 1,
"passed": true
},
"runtime": {
"acceleration": "cpu",
"cpuThreads": 4,
"device": "cpu",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 932.528125,
"loadMs": 7634.52925,
"offloadedLayers": 0,
"precision": "q4",
"provider": "llama.cpp CPU",
"requestedDevice": "cpu",
"requestedPrecision": "q4",
"selectionReason": "Exact Q4_0 package with verified layer placement",
"version": "b12359",
"artifactRevision": "d508b88f08a5b4286f2d2805a6a8d69a0ff44c72",
"totalMs": 8567.057375,
"performanceMetric": {
"name": "audioSecondsPerSecond",
"value": 3.2599552962544696
},
"grade": "good",
"platform": "macos",
"arch": "aarch64"
}
},
{
"precision": "q4",
"device": "metal",
"status": "pass",
"quality": {
"metric": "independentWhisperWordMatch",
"score": 1,
"passed": true
},
"runtime": {
"acceleration": "partial",
"cpuThreads": 4,
"device": "metal",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 624.8902499999999,
"loadMs": 753.985542,
"offloadedLayers": 25,
"precision": "q4",
"provider": "llama.cpp Metal",
"requestedDevice": "metal",
"requestedPrecision": "q4",
"selectionReason": "Exact Q4_0 package with verified layer placement",
"version": "b12359",
"artifactRevision": "d508b88f08a5b4286f2d2805a6a8d69a0ff44c72",
"totalMs": 1378.8757919999998,
"performanceMetric": {
"name": "audioSecondsPerSecond",
"value": 4.608809306914295
},
"grade": "good",
"platform": "macos",
"arch": "aarch64"
}
}
]LFM3-VL 2B Desktop
78 configurations testedPassed75
Failed3
TestedNov 16, 2026
Runtimellama.cpp b12359
Quality criteria
No explicit quality gate recorded. Review per-case evidence; full certification is pending.
| Precision | Device | Result | Platform | Inference / load | |
|---|---|---|---|---|---|
Q5_K_M | metalAccelerated | Passed | Very fast | macos · aarch64 | 64 ms / 2474 ms |
FP16 | metalAccelerated | Passed | Fast | macos · aarch64 | 117 ms / 2055 ms |
Q5_K_M | metalAccelerated | Passed | Fast | macos · aarch64 | 133 ms / 2474 ms |
FP16 | metalAccelerated | Passed | Fast | macos · aarch64 | 136 ms / 2055 ms |
FP16 | metalAccelerated | Passed | Fast | macos · aarch64 | 138 ms / 2055 ms |
BF16 | metalAccelerated | Passed | Fast | macos · aarch64 | 139 ms / 3233 ms |
Q5_K_M | metalAccelerated | Passed | Fast | macos · aarch64 | 160 ms / 2474 ms |
BF16 | metalAccelerated | Passed | Fast | macos · aarch64 | 169 ms / 3233 ms |
BF16 | metalAccelerated | Passed | Fast | macos · aarch64 | 186 ms / 3233 ms |
FP16 | metalAccelerated | Passed | Fast | macos · aarch64 | 221 ms / 2055 ms |
BF16 | metalAccelerated | Passed | Fast | macos · aarch64 | 231 ms / 3233 ms |
Q5_K_M | metalAccelerated | Passed | Fast | macos · aarch64 | 246 ms / 2474 ms |
FP16 | metalAccelerated | Passed | Fast | macos · aarch64 | 323 ms / 2055 ms |
Q5_K_M | metalAccelerated | Passed | Fast | macos · aarch64 | 329 ms / 2474 ms |
BF16 | metalAccelerated | Passed | Fast | macos · aarch64 | 349 ms / 3233 ms |
NVFP4 | metalAccelerated | Passed | Fast | macos · aarch64 | 402 ms / 2785 ms |
Q4_K_M | metalAccelerated | Passed | Fast | macos · aarch64 | 543 ms / 6791 ms |
Q4_K_M | metalAccelerated | Passed | Fast | macos · aarch64 | 590 ms / 6791 ms |
Q4_K_M | metalAccelerated | Passed | Fast | macos · aarch64 | 618 ms / 6791 ms |
Q6_K | metalAccelerated | Passed | Fast | macos · aarch64 | 695 ms / 2187 ms |
Q4 | metalAccelerated | Passed | Fast | macos · aarch64 | 731 ms / 2902 ms |
Q4_K_M | metalAccelerated | Passed | Fast | macos · aarch64 | 744 ms / 6791 ms |
Q8_0 | metalAccelerated | Passed | Fast | macos · aarch64 | 759 ms / 2178 ms |
NVFP4 | metalAccelerated | Passed | Fast | macos · aarch64 | 800 ms / 2785 ms |
NVFP4 | metalAccelerated | Passed | Fast | macos · aarch64 | 805 ms / 2785 ms |
Q8_0 | metalAccelerated | Passed | Fast | macos · aarch64 | 819 ms / 2178 ms |
Q6_K | metalAccelerated | Passed | Fast | macos · aarch64 | 829 ms / 2187 ms |
Q4 | metalAccelerated | Passed | Fast | macos · aarch64 | 879 ms / 2902 ms |
Q4 | metalAccelerated | Passed | Fast | macos · aarch64 | 880 ms / 2902 ms |
Q4 | metalAccelerated | Passed | Fast | macos · aarch64 | 888 ms / 2902 ms |
Q4 | metalAccelerated | Passed | Fast | macos · aarch64 | 925 ms / 2902 ms |
NVFP4 | metalAccelerated | Passed | Fast | macos · aarch64 | 954 ms / 2785 ms |
Q6_K | metalAccelerated | Passed | Fast | macos · aarch64 | 975 ms / 2187 ms |
Q6_K | metalAccelerated | Passed | Fast | macos · aarch64 | 996 ms / 2187 ms |
Q8_0 | metalAccelerated | Passed | Fast | macos · aarch64 | 999 ms / 2178 ms |
Q8_0 | metalAccelerated | Passed | Fast | macos · aarch64 | 1040 ms / 2178 ms |
Q4_K_M | metalAccelerated | Passed | Fast | macos · aarch64 | 1141 ms / 6791 ms |
NVFP4 | metalAccelerated | Passed | Fast | macos · aarch64 | 1149 ms / 2785 ms |
Q6_K | metalAccelerated | Passed | Fast | macos · aarch64 | 1352 ms / 2187 ms |
Q8_0 | metalAccelerated | Passed | Fast | macos · aarch64 | 1449 ms / 2178 ms |
Q8_0 | cpu | Passed | Fast | macos · aarch64 | 120 ms / 3520 ms |
Q5_K_M | cpu | Passed | Fast | macos · aarch64 | 152 ms / 2893 ms |
Q6_K | cpu | Passed | Fast | macos · aarch64 | 157 ms / 3147 ms |
NVFP4 | cpu | Passed | Fast | macos · aarch64 | 247 ms / 3251 ms |
FP16 | cpu | Passed | Fast | macos · aarch64 | 306 ms / 2542 ms |
Q8_0 | cpu | Passed | Fast | macos · aarch64 | 484 ms / 3520 ms |
Q8_0 | cpu | Passed | Fast | macos · aarch64 | 498 ms / 3520 ms |
Q6_K | cpu | Passed | Fast | macos · aarch64 | 656 ms / 3147 ms |
Q6_K | cpu | Passed | Fast | macos · aarch64 | 659 ms / 3147 ms |
Q5_K_M | cpu | Passed | Fast | macos · aarch64 | 659 ms / 2893 ms |
Q5_K_M | cpu | Passed | Fast | macos · aarch64 | 663 ms / 2893 ms |
NVFP4 | cpu | Passed | Fast | macos · aarch64 | 761 ms / 3251 ms |
NVFP4 | cpu | Passed | Fast | macos · aarch64 | 788 ms / 3251 ms |
Q4_K_M | cpu | Passed | Fast | macos · aarch64 | 815 ms / 2799 ms |
FP16 | cpu | Passed | Fast | macos · aarch64 | 911 ms / 2542 ms |
FP16 | cpu | Passed | Fast | macos · aarch64 | 954 ms / 2542 ms |
Q4 | cpu | Passed | Fast | macos · aarch64 | 1143 ms / 7105 ms |
Q8_0 | cpu | Passed | Fast | macos · aarch64 | 1295 ms / 3520 ms |
Q8_0 | cpu | Passed | Fast | macos · aarch64 | 1335 ms / 3520 ms |
Q4_K_M | cpu | Passed | Fast | macos · aarch64 | 1381 ms / 2799 ms |
NVFP4 | cpu | Passed | Fast | macos · aarch64 | 1401 ms / 3251 ms |
NVFP4 | cpu | Passed | Fast | macos · aarch64 | 1421 ms / 3251 ms |
Q4_K_M | cpu | Passed | Fast | macos · aarch64 | 1631 ms / 2799 ms |
Q6_K | cpu | Passed | Fast | macos · aarch64 | 1645 ms / 3147 ms |
Q6_K | cpu | Passed | Fast | macos · aarch64 | 1656 ms / 3147 ms |
Q5_K_M | cpu | Passed | Fast | macos · aarch64 | 1687 ms / 2893 ms |
Q5_K_M | cpu | Passed | Fast | macos · aarch64 | 1693 ms / 2893 ms |
FP16 | cpu | Passed | Fast | macos · aarch64 | 1727 ms / 2542 ms |
FP16 | cpu | Passed | Fast | macos · aarch64 | 1809 ms / 2542 ms |
Q4 | cpu | Passed | Moderate | macos · aarch64 | 2016 ms / 7105 ms |
Q4_K_M | cpu | Passed | Moderate | macos · aarch64 | 2046 ms / 2799 ms |
Q4 | cpu | Passed | Moderate | macos · aarch64 | 2118 ms / 7105 ms |
Q4 | cpu | Passed | Moderate | macos · aarch64 | 2239 ms / 7105 ms |
Q4_K_M | cpu | Passed | Moderate | macos · aarch64 | 2541 ms / 2799 ms |
Q4 | cpu | Passed | Moderate | macos · aarch64 | 3452 ms / 7105 ms |
BF16 | cpu | Failed | — | macos · aarch64 | 39035 ms / 3972 ms |
BF16 | cpu | Failed | — | macos · aarch64 | 39429 ms / 3972 ms |
BF16 | cpu | Failed | — | macos · aarch64 | 43616 ms / 3972 ms |
Test setup & sample details
- Quality gate
- No explicit quality gate recorded. Review the per-case quality evidence; full certification is pending.
- Tested
- 2026-11-16T01:57:03.731Z
- Environment
- { "scope": "DesktopAI TypeScript client using native JSON-lines transport with verified cached sources.", "hardware": "Apple M3 Max", "os": "macOS", "arch": "arm64", "runtime": { "name": "llama.cpp", "version": "b12359" }, "speedGradePolicy": "Short requests completing in 2 seconds or less are interactive." }
- Runtime
- llama.cpp b12359
- Artifact revision
- 1add221a5ea6c047fcf1eaaf2d247939d7cf8849
- Fixture
{ "file": "model-samples/lfm3-vl-2b-native/vl-samples/desktopai-arithmetic-input.json", "url": "https://assets.localai.tools/samples/v1/files/712f8b4b7e032c2e4b12e94dbb2e8d7c9681d69c344dbcf7874175c28b7678b5/desktopai-arithmetic-input.json", "size": 219, "sha1": "30f7cac931b376f1ddcdf626b6cad3d91a3c6e93", "sha256": "712f8b4b7e032c2e4b12e94dbb2e8d7c9681d69c344dbcf7874175c28b7678b5" }- Rerun when
- artifactRevision; runtimeVersion; fixtureChecksum; qualityGate; nativeBackend
Per-case quality & failure details
[
{
"precision": "q4",
"device": "cpu",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": "15"
},
"runtime": {
"acceleration": "cpu",
"cpuThreads": 4,
"device": "cpu",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 1143.015709,
"loadMs": 7105.099291,
"offloadedLayers": 0,
"precision": "q4",
"provider": "llama.cpp CPU",
"requestedDevice": "cpu",
"requestedPrecision": "q4",
"selectionReason": "Exact QAD Q4_0 package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 8248.115,
"charactersPerSecond": 1.749757229277065,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 1.749757229277065
},
"grade": "good"
}
},
{
"precision": "q4",
"device": "cpu",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": "red"
},
"runtime": {
"acceleration": "cpu",
"cpuThreads": 4,
"device": "cpu",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 2015.819,
"loadMs": 7105.099291,
"offloadedLayers": 0,
"precision": "q4",
"provider": "llama.cpp CPU",
"requestedDevice": "cpu",
"requestedPrecision": "q4",
"selectionReason": "Exact QAD Q4_0 package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 9120.918291,
"charactersPerSecond": 1.4882288538802344,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 1.4882288538802344
},
"grade": "moderate"
}
},
{
"precision": "q4",
"device": "cpu",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": "blue"
},
"runtime": {
"acceleration": "cpu",
"cpuThreads": 4,
"device": "cpu",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 2117.605291,
"loadMs": 7105.099291,
"offloadedLayers": 0,
"precision": "q4",
"provider": "llama.cpp CPU",
"requestedDevice": "cpu",
"requestedPrecision": "q4",
"selectionReason": "Exact QAD Q4_0 package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 9222.704582,
"charactersPerSecond": 1.888926145491011,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 1.888926145491011
},
"grade": "moderate"
}
},
{
"precision": "q4",
"device": "cpu",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": [
"localai2026"
]
},
"runtime": {
"acceleration": "cpu",
"cpuThreads": 4,
"device": "cpu",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 2239.10725,
"loadMs": 7105.099291,
"offloadedLayers": 0,
"precision": "q4",
"provider": "llama.cpp CPU",
"requestedDevice": "cpu",
"requestedPrecision": "q4",
"selectionReason": "Exact QAD Q4_0 package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 9344.206541,
"charactersPerSecond": 5.805885358997431,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 5.805885358997431
},
"grade": "moderate"
}
},
{
"precision": "q4",
"device": "cpu",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": [
"3"
]
},
"runtime": {
"acceleration": "cpu",
"cpuThreads": 4,
"device": "cpu",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 3451.848541,
"loadMs": 7105.099291,
"offloadedLayers": 0,
"precision": "q4",
"provider": "llama.cpp CPU",
"requestedDevice": "cpu",
"requestedPrecision": "q4",
"selectionReason": "Exact QAD Q4_0 package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 10556.947832,
"charactersPerSecond": 0.28969984868174437,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 0.28969984868174437
},
"grade": "moderate"
}
},
{
"precision": "q4",
"device": "metal",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": "15"
},
"runtime": {
"acceleration": "partial",
"cpuThreads": 4,
"device": "metal",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 879.5343340000001,
"loadMs": 2901.979416,
"offloadedLayers": 25,
"precision": "q4",
"provider": "llama.cpp Metal",
"requestedDevice": "metal",
"requestedPrecision": "q4",
"selectionReason": "Exact QAD Q4_0 package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 3781.51375,
"charactersPerSecond": 2.2739305592588726,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 2.2739305592588726
},
"grade": "good"
}
},
{
"precision": "q4",
"device": "metal",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": "red"
},
"runtime": {
"acceleration": "partial",
"cpuThreads": 4,
"device": "metal",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 924.6751250000001,
"loadMs": 2901.979416,
"offloadedLayers": 25,
"precision": "q4",
"provider": "llama.cpp Metal",
"requestedDevice": "metal",
"requestedPrecision": "q4",
"selectionReason": "Exact QAD Q4_0 package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 3826.6545410000003,
"charactersPerSecond": 3.2443827230671958,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 3.2443827230671958
},
"grade": "good"
}
},
{
"precision": "q4",
"device": "metal",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": "blue"
},
"runtime": {
"acceleration": "partial",
"cpuThreads": 4,
"device": "metal",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 879.015125,
"loadMs": 2901.979416,
"offloadedLayers": 25,
"precision": "q4",
"provider": "llama.cpp Metal",
"requestedDevice": "metal",
"requestedPrecision": "q4",
"selectionReason": "Exact QAD Q4_0 package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 3780.994541,
"charactersPerSecond": 4.550547409522674,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 4.550547409522674
},
"grade": "good"
}
},
{
"precision": "q4",
"device": "metal",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": [
"localai2026"
]
},
"runtime": {
"acceleration": "partial",
"cpuThreads": 4,
"device": "metal",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 731.3244589999999,
"loadMs": 2901.979416,
"offloadedLayers": 25,
"precision": "q4",
"provider": "llama.cpp Metal",
"requestedDevice": "metal",
"requestedPrecision": "q4",
"selectionReason": "Exact QAD Q4_0 package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 3633.303875,
"charactersPerSecond": 17.775967752775518,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 17.775967752775518
},
"grade": "good"
}
},
{
"precision": "q4",
"device": "metal",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": [
"3"
]
},
"runtime": {
"acceleration": "partial",
"cpuThreads": 4,
"device": "metal",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 887.623542,
"loadMs": 2901.979416,
"offloadedLayers": 25,
"precision": "q4",
"provider": "llama.cpp Metal",
"requestedDevice": "metal",
"requestedPrecision": "q4",
"selectionReason": "Exact QAD Q4_0 package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 3789.6029580000004,
"charactersPerSecond": 1.1266037376011822,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 1.1266037376011822
},
"grade": "good"
}
},
{
"precision": "q4_k_m",
"device": "cpu",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": "15"
},
"runtime": {
"acceleration": "cpu",
"cpuThreads": 4,
"device": "cpu",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 815.113167,
"loadMs": 2799.4535,
"offloadedLayers": 0,
"precision": "q4_k_m",
"provider": "llama.cpp CPU",
"requestedDevice": "cpu",
"requestedPrecision": "q4_k_m",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 3614.566667,
"charactersPerSecond": 2.453647028376368,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 2.453647028376368
},
"grade": "good"
}
},
{
"precision": "q4_k_m",
"device": "cpu",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": "red"
},
"runtime": {
"acceleration": "cpu",
"cpuThreads": 4,
"device": "cpu",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 1631.486708,
"loadMs": 2799.4535,
"offloadedLayers": 0,
"precision": "q4_k_m",
"provider": "llama.cpp CPU",
"requestedDevice": "cpu",
"requestedPrecision": "q4_k_m",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 4430.940208,
"charactersPerSecond": 1.8388136325533584,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 1.8388136325533584
},
"grade": "good"
}
},
{
"precision": "q4_k_m",
"device": "cpu",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": "blue"
},
"runtime": {
"acceleration": "cpu",
"cpuThreads": 4,
"device": "cpu",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 1380.7507079999998,
"loadMs": 2799.4535,
"offloadedLayers": 0,
"precision": "q4_k_m",
"provider": "llama.cpp CPU",
"requestedDevice": "cpu",
"requestedPrecision": "q4_k_m",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 4180.204208,
"charactersPerSecond": 2.896974795539993,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 2.896974795539993
},
"grade": "good"
}
},
{
"precision": "q4_k_m",
"device": "cpu",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": [
"localai2026"
]
},
"runtime": {
"acceleration": "cpu",
"cpuThreads": 4,
"device": "cpu",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 2046.1294169999999,
"loadMs": 2799.4535,
"offloadedLayers": 0,
"precision": "q4_k_m",
"provider": "llama.cpp CPU",
"requestedDevice": "cpu",
"requestedPrecision": "q4_k_m",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 4845.582917,
"charactersPerSecond": 6.353459312979518,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 6.353459312979518
},
"grade": "moderate"
}
},
{
"precision": "q4_k_m",
"device": "cpu",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": [
"3",
"three"
]
},
"runtime": {
"acceleration": "cpu",
"cpuThreads": 4,
"device": "cpu",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 2541.326666,
"loadMs": 2799.4535,
"offloadedLayers": 0,
"precision": "q4_k_m",
"provider": "llama.cpp CPU",
"requestedDevice": "cpu",
"requestedPrecision": "q4_k_m",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 5340.7801660000005,
"charactersPerSecond": 0.3934952611086323,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 0.3934952611086323
},
"grade": "moderate"
}
},
{
"precision": "q4_k_m",
"device": "metal",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": "15"
},
"runtime": {
"acceleration": "partial",
"cpuThreads": 4,
"device": "metal",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 542.529083,
"loadMs": 6790.742958,
"offloadedLayers": 25,
"precision": "q4_k_m",
"provider": "llama.cpp Metal",
"requestedDevice": "metal",
"requestedPrecision": "q4_k_m",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 7333.272041,
"charactersPerSecond": 3.6864383176302455,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 3.6864383176302455
},
"grade": "good"
}
},
{
"precision": "q4_k_m",
"device": "metal",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": "red"
},
"runtime": {
"acceleration": "partial",
"cpuThreads": 4,
"device": "metal",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 744.162208,
"loadMs": 6790.742958,
"offloadedLayers": 25,
"precision": "q4_k_m",
"provider": "llama.cpp Metal",
"requestedDevice": "metal",
"requestedPrecision": "q4_k_m",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 7534.905166,
"charactersPerSecond": 4.031379137167901,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 4.031379137167901
},
"grade": "good"
}
},
{
"precision": "q4_k_m",
"device": "metal",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": "blue"
},
"runtime": {
"acceleration": "partial",
"cpuThreads": 4,
"device": "metal",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 617.913791,
"loadMs": 6790.742958,
"offloadedLayers": 25,
"precision": "q4_k_m",
"provider": "llama.cpp Metal",
"requestedDevice": "metal",
"requestedPrecision": "q4_k_m",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 7408.656749,
"charactersPerSecond": 6.473394927027936,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 6.473394927027936
},
"grade": "good"
}
},
{
"precision": "q4_k_m",
"device": "metal",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": [
"localai2026"
]
},
"runtime": {
"acceleration": "partial",
"cpuThreads": 4,
"device": "metal",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 590.235708,
"loadMs": 6790.742958,
"offloadedLayers": 25,
"precision": "q4_k_m",
"provider": "llama.cpp Metal",
"requestedDevice": "metal",
"requestedPrecision": "q4_k_m",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 7380.978666,
"charactersPerSecond": 22.025099165975906,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 22.025099165975906
},
"grade": "good"
}
},
{
"precision": "q4_k_m",
"device": "metal",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": [
"3",
"three"
]
},
"runtime": {
"acceleration": "partial",
"cpuThreads": 4,
"device": "metal",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 1140.5493330000002,
"loadMs": 6790.742958,
"offloadedLayers": 25,
"precision": "q4_k_m",
"provider": "llama.cpp Metal",
"requestedDevice": "metal",
"requestedPrecision": "q4_k_m",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 7931.292291,
"charactersPerSecond": 0.8767704921361783,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 0.8767704921361783
},
"grade": "good"
}
},
{
"precision": "q5_k_m",
"device": "cpu",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": "15"
},
"runtime": {
"acceleration": "cpu",
"cpuThreads": 4,
"device": "cpu",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 152.28225,
"loadMs": 2893.1373750000002,
"offloadedLayers": 0,
"precision": "q5_k_m",
"provider": "llama.cpp CPU",
"requestedDevice": "cpu",
"requestedPrecision": "q5_k_m",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 3045.4196250000005,
"charactersPerSecond": 13.133507023963725,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 13.133507023963725
},
"grade": "good"
}
},
{
"precision": "q5_k_m",
"device": "cpu",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": "red"
},
"runtime": {
"acceleration": "cpu",
"cpuThreads": 4,
"device": "cpu",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 662.718042,
"loadMs": 2893.1373750000002,
"offloadedLayers": 0,
"precision": "q5_k_m",
"provider": "llama.cpp CPU",
"requestedDevice": "cpu",
"requestedPrecision": "q5_k_m",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 3555.855417,
"charactersPerSecond": 4.526812022419635,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 4.526812022419635
},
"grade": "good"
}
},
{
"precision": "q5_k_m",
"device": "cpu",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": "blue"
},
"runtime": {
"acceleration": "cpu",
"cpuThreads": 4,
"device": "cpu",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 659.1525419999999,
"loadMs": 2893.1373750000002,
"offloadedLayers": 0,
"precision": "q5_k_m",
"provider": "llama.cpp CPU",
"requestedDevice": "cpu",
"requestedPrecision": "q5_k_m",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 3552.289917,
"charactersPerSecond": 6.068398049203003,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 6.068398049203003
},
"grade": "good"
}
},
{
"precision": "q5_k_m",
"device": "cpu",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": [
"localai2026"
]
},
"runtime": {
"acceleration": "cpu",
"cpuThreads": 4,
"device": "cpu",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 1693.2307500000002,
"loadMs": 2893.1373750000002,
"offloadedLayers": 0,
"precision": "q5_k_m",
"provider": "llama.cpp CPU",
"requestedDevice": "cpu",
"requestedPrecision": "q5_k_m",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 4586.368125000001,
"charactersPerSecond": 7.677630470625459,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 7.677630470625459
},
"grade": "good"
}
},
{
"precision": "q5_k_m",
"device": "cpu",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": [
"3",
"three"
]
},
"runtime": {
"acceleration": "cpu",
"cpuThreads": 4,
"device": "cpu",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 1686.535958,
"loadMs": 2893.1373750000002,
"offloadedLayers": 0,
"precision": "q5_k_m",
"provider": "llama.cpp CPU",
"requestedDevice": "cpu",
"requestedPrecision": "q5_k_m",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 4579.673333000001,
"charactersPerSecond": 0.5929313248594252,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 0.5929313248594252
},
"grade": "good"
}
},
{
"precision": "q5_k_m",
"device": "metal",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": "15"
},
"runtime": {
"acceleration": "partial",
"cpuThreads": 4,
"device": "metal",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 63.886083,
"loadMs": 2473.8635000000004,
"offloadedLayers": 25,
"precision": "q5_k_m",
"provider": "llama.cpp Metal",
"requestedDevice": "metal",
"requestedPrecision": "q5_k_m",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 2537.7495830000003,
"charactersPerSecond": 31.30572271898404,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 31.30572271898404
},
"grade": "good"
}
},
{
"precision": "q5_k_m",
"device": "metal",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": "red"
},
"runtime": {
"acceleration": "partial",
"cpuThreads": 4,
"device": "metal",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 160.091625,
"loadMs": 2473.8635000000004,
"offloadedLayers": 25,
"precision": "q5_k_m",
"provider": "llama.cpp Metal",
"requestedDevice": "metal",
"requestedPrecision": "q5_k_m",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 2633.9551250000004,
"charactersPerSecond": 18.739268840578013,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 18.739268840578013
},
"grade": "good"
}
},
{
"precision": "q5_k_m",
"device": "metal",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": "blue"
},
"runtime": {
"acceleration": "partial",
"cpuThreads": 4,
"device": "metal",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 133.380958,
"loadMs": 2473.8635000000004,
"offloadedLayers": 25,
"precision": "q5_k_m",
"provider": "llama.cpp Metal",
"requestedDevice": "metal",
"requestedPrecision": "q5_k_m",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 2607.2444580000006,
"charactersPerSecond": 29.989288276067114,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 29.989288276067114
},
"grade": "good"
}
},
{
"precision": "q5_k_m",
"device": "metal",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": [
"localai2026"
]
},
"runtime": {
"acceleration": "partial",
"cpuThreads": 4,
"device": "metal",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 328.665416,
"loadMs": 2473.8635000000004,
"offloadedLayers": 25,
"precision": "q5_k_m",
"provider": "llama.cpp Metal",
"requestedDevice": "metal",
"requestedPrecision": "q5_k_m",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 2802.528916,
"charactersPerSecond": 39.55390304892925,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 39.55390304892925
},
"grade": "good"
}
},
{
"precision": "q5_k_m",
"device": "metal",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": [
"3",
"three"
]
},
"runtime": {
"acceleration": "partial",
"cpuThreads": 4,
"device": "metal",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 245.967584,
"loadMs": 2473.8635000000004,
"offloadedLayers": 25,
"precision": "q5_k_m",
"provider": "llama.cpp Metal",
"requestedDevice": "metal",
"requestedPrecision": "q5_k_m",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 2719.8310840000004,
"charactersPerSecond": 4.0655763809917325,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 4.0655763809917325
},
"grade": "good"
}
},
{
"precision": "q6_k",
"device": "metal",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": "15"
},
"runtime": {
"acceleration": "partial",
"cpuThreads": 4,
"device": "metal",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 995.9282499999999,
"loadMs": 2187.437667,
"offloadedLayers": 25,
"precision": "q6_k",
"provider": "llama.cpp Metal",
"requestedDevice": "metal",
"requestedPrecision": "q6_k",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 3183.365917,
"charactersPerSecond": 2.008176793860401,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 2.008176793860401
},
"grade": "good"
}
},
{
"precision": "q6_k",
"device": "metal",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": "red"
},
"runtime": {
"acceleration": "partial",
"cpuThreads": 4,
"device": "metal",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 829.275584,
"loadMs": 2187.437667,
"offloadedLayers": 25,
"precision": "q6_k",
"provider": "llama.cpp Metal",
"requestedDevice": "metal",
"requestedPrecision": "q6_k",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 3016.713251,
"charactersPerSecond": 3.617615251047835,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 3.617615251047835
},
"grade": "good"
}
},
{
"precision": "q6_k",
"device": "metal",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": "blue"
},
"runtime": {
"acceleration": "partial",
"cpuThreads": 4,
"device": "metal",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 975.455916,
"loadMs": 2187.437667,
"offloadedLayers": 25,
"precision": "q6_k",
"provider": "llama.cpp Metal",
"requestedDevice": "metal",
"requestedPrecision": "q6_k",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 3162.893583,
"charactersPerSecond": 4.100646614972193,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 4.100646614972193
},
"grade": "good"
}
},
{
"precision": "q6_k",
"device": "metal",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": [
"localai2026"
]
},
"runtime": {
"acceleration": "partial",
"cpuThreads": 4,
"device": "metal",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 694.940333,
"loadMs": 2187.437667,
"offloadedLayers": 25,
"precision": "q6_k",
"provider": "llama.cpp Metal",
"requestedDevice": "metal",
"requestedPrecision": "q6_k",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 2882.378,
"charactersPerSecond": 18.706641970081193,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 18.706641970081193
},
"grade": "good"
}
},
{
"precision": "q6_k",
"device": "metal",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": [
"3",
"three"
]
},
"runtime": {
"acceleration": "partial",
"cpuThreads": 4,
"device": "metal",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 1351.511792,
"loadMs": 2187.437667,
"offloadedLayers": 25,
"precision": "q6_k",
"provider": "llama.cpp Metal",
"requestedDevice": "metal",
"requestedPrecision": "q6_k",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 3538.9494590000004,
"charactersPerSecond": 0.7399121531305144,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 0.7399121531305144
},
"grade": "good"
}
},
{
"precision": "q8_0",
"device": "metal",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": "15"
},
"runtime": {
"acceleration": "partial",
"cpuThreads": 4,
"device": "metal",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 1039.566417,
"loadMs": 2177.748666,
"offloadedLayers": 25,
"precision": "q8_0",
"provider": "llama.cpp Metal",
"requestedDevice": "metal",
"requestedPrecision": "q8_0",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 3217.315083,
"charactersPerSecond": 1.9238790011817013,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 1.9238790011817013
},
"grade": "good"
}
},
{
"precision": "q8_0",
"device": "metal",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": "red"
},
"runtime": {
"acceleration": "partial",
"cpuThreads": 4,
"device": "metal",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 819.099209,
"loadMs": 2177.748666,
"offloadedLayers": 25,
"precision": "q8_0",
"provider": "llama.cpp Metal",
"requestedDevice": "metal",
"requestedPrecision": "q8_0",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 2996.847875,
"charactersPerSecond": 3.662560001324577,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 3.662560001324577
},
"grade": "good"
}
},
{
"precision": "q8_0",
"device": "metal",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": "blue"
},
"runtime": {
"acceleration": "partial",
"cpuThreads": 4,
"device": "metal",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 998.7096250000001,
"loadMs": 2177.748666,
"offloadedLayers": 25,
"precision": "q8_0",
"provider": "llama.cpp Metal",
"requestedDevice": "metal",
"requestedPrecision": "q8_0",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 3176.458291,
"charactersPerSecond": 4.005168168875913,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 4.005168168875913
},
"grade": "good"
}
},
{
"precision": "q8_0",
"device": "metal",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": [
"localai2026"
]
},
"runtime": {
"acceleration": "partial",
"cpuThreads": 4,
"device": "metal",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 758.8914589999999,
"loadMs": 2177.748666,
"offloadedLayers": 25,
"precision": "q8_0",
"provider": "llama.cpp Metal",
"requestedDevice": "metal",
"requestedPrecision": "q8_0",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 2936.640125,
"charactersPerSecond": 17.130249452445085,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 17.130249452445085
},
"grade": "good"
}
},
{
"precision": "q8_0",
"device": "metal",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": [
"3",
"three"
]
},
"runtime": {
"acceleration": "partial",
"cpuThreads": 4,
"device": "metal",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 1449.183166,
"loadMs": 2177.748666,
"offloadedLayers": 25,
"precision": "q8_0",
"provider": "llama.cpp Metal",
"requestedDevice": "metal",
"requestedPrecision": "q8_0",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 3626.931832,
"charactersPerSecond": 0.6900438974599571,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 0.6900438974599571
},
"grade": "good"
}
},
{
"precision": "q6_k",
"device": "cpu",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": "15"
},
"runtime": {
"acceleration": "cpu",
"cpuThreads": 4,
"device": "cpu",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 156.909417,
"loadMs": 3146.91175,
"offloadedLayers": 0,
"precision": "q6_k",
"provider": "llama.cpp CPU",
"requestedDevice": "cpu",
"requestedPrecision": "q6_k",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 3303.821167,
"charactersPerSecond": 12.746207577840915,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 12.746207577840915
},
"grade": "good"
}
},
{
"precision": "q6_k",
"device": "cpu",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": "red"
},
"runtime": {
"acceleration": "cpu",
"cpuThreads": 4,
"device": "cpu",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 656.397708,
"loadMs": 3146.91175,
"offloadedLayers": 0,
"precision": "q6_k",
"provider": "llama.cpp CPU",
"requestedDevice": "cpu",
"requestedPrecision": "q6_k",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 3803.309458,
"charactersPerSecond": 4.57039987104891,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 4.57039987104891
},
"grade": "good"
}
},
{
"precision": "q6_k",
"device": "cpu",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": "blue"
},
"runtime": {
"acceleration": "cpu",
"cpuThreads": 4,
"device": "cpu",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 658.636416,
"loadMs": 3146.91175,
"offloadedLayers": 0,
"precision": "q6_k",
"provider": "llama.cpp CPU",
"requestedDevice": "cpu",
"requestedPrecision": "q6_k",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 3805.5481660000005,
"charactersPerSecond": 6.073153416406298,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 6.073153416406298
},
"grade": "good"
}
},
{
"precision": "q6_k",
"device": "cpu",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": [
"localai2026"
]
},
"runtime": {
"acceleration": "cpu",
"cpuThreads": 4,
"device": "cpu",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 1656.087375,
"loadMs": 3146.91175,
"offloadedLayers": 0,
"precision": "q6_k",
"provider": "llama.cpp CPU",
"requestedDevice": "cpu",
"requestedPrecision": "q6_k",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 4802.999125,
"charactersPerSecond": 7.849827367955148,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 7.849827367955148
},
"grade": "good"
}
},
{
"precision": "q6_k",
"device": "cpu",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": [
"3",
"three"
]
},
"runtime": {
"acceleration": "cpu",
"cpuThreads": 4,
"device": "cpu",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 1644.980041,
"loadMs": 3146.91175,
"offloadedLayers": 0,
"precision": "q6_k",
"provider": "llama.cpp CPU",
"requestedDevice": "cpu",
"requestedPrecision": "q6_k",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 4791.891791,
"charactersPerSecond": 0.6079101114151452,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 0.6079101114151452
},
"grade": "good"
}
},
{
"precision": "q8_0",
"device": "cpu",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": "15"
},
"runtime": {
"acceleration": "cpu",
"cpuThreads": 4,
"device": "cpu",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 120.412959,
"loadMs": 3520.119667,
"offloadedLayers": 0,
"precision": "q8_0",
"provider": "llama.cpp CPU",
"requestedDevice": "cpu",
"requestedPrecision": "q8_0",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 3640.532626,
"charactersPerSecond": 16.609507951714733,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 16.609507951714733
},
"grade": "good"
}
},
{
"precision": "q8_0",
"device": "cpu",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": "red"
},
"runtime": {
"acceleration": "cpu",
"cpuThreads": 4,
"device": "cpu",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 498.074833,
"loadMs": 3520.119667,
"offloadedLayers": 0,
"precision": "q8_0",
"provider": "llama.cpp CPU",
"requestedDevice": "cpu",
"requestedPrecision": "q8_0",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 4018.1945,
"charactersPerSecond": 6.023191298244133,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 6.023191298244133
},
"grade": "good"
}
},
{
"precision": "q8_0",
"device": "cpu",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": "blue"
},
"runtime": {
"acceleration": "cpu",
"cpuThreads": 4,
"device": "cpu",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 483.856792,
"loadMs": 3520.119667,
"offloadedLayers": 0,
"precision": "q8_0",
"provider": "llama.cpp CPU",
"requestedDevice": "cpu",
"requestedPrecision": "q8_0",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 4003.976459,
"charactersPerSecond": 8.266908858437601,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 8.266908858437601
},
"grade": "good"
}
},
{
"precision": "q8_0",
"device": "cpu",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": [
"localai2026"
]
},
"runtime": {
"acceleration": "cpu",
"cpuThreads": 4,
"device": "cpu",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 1334.946417,
"loadMs": 3520.119667,
"offloadedLayers": 0,
"precision": "q8_0",
"provider": "llama.cpp CPU",
"requestedDevice": "cpu",
"requestedPrecision": "q8_0",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 4855.066084,
"charactersPerSecond": 9.738218579000836,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 9.738218579000836
},
"grade": "good"
}
},
{
"precision": "q8_0",
"device": "cpu",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": [
"3",
"three"
]
},
"runtime": {
"acceleration": "cpu",
"cpuThreads": 4,
"device": "cpu",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 1294.901709,
"loadMs": 3520.119667,
"offloadedLayers": 0,
"precision": "q8_0",
"provider": "llama.cpp CPU",
"requestedDevice": "cpu",
"requestedPrecision": "q8_0",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 4815.021376,
"charactersPerSecond": 0.7722593869864142,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 0.7722593869864142
},
"grade": "good"
}
},
{
"precision": "nvfp4",
"device": "cpu",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": "15"
},
"runtime": {
"acceleration": "cpu",
"cpuThreads": 4,
"device": "cpu",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 247.41470800000002,
"loadMs": 3250.873542,
"offloadedLayers": 0,
"precision": "nvfp4",
"provider": "llama.cpp CPU",
"requestedDevice": "cpu",
"requestedPrecision": "nvfp4",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 3498.2882499999996,
"charactersPerSecond": 8.08359380154554,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 8.08359380154554
},
"grade": "good"
}
},
{
"precision": "nvfp4",
"device": "cpu",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": "red"
},
"runtime": {
"acceleration": "cpu",
"cpuThreads": 4,
"device": "cpu",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 760.594625,
"loadMs": 3250.873542,
"offloadedLayers": 0,
"precision": "nvfp4",
"provider": "llama.cpp CPU",
"requestedDevice": "cpu",
"requestedPrecision": "nvfp4",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 4011.468167,
"charactersPerSecond": 3.9442824093057456,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 3.9442824093057456
},
"grade": "good"
}
},
{
"precision": "nvfp4",
"device": "cpu",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": "blue"
},
"runtime": {
"acceleration": "cpu",
"cpuThreads": 4,
"device": "cpu",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 787.9119999999999,
"loadMs": 3250.873542,
"offloadedLayers": 0,
"precision": "nvfp4",
"provider": "llama.cpp CPU",
"requestedDevice": "cpu",
"requestedPrecision": "nvfp4",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 4038.7855419999996,
"charactersPerSecond": 5.076709074109799,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 5.076709074109799
},
"grade": "good"
}
},
{
"precision": "nvfp4",
"device": "cpu",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": [
"localai2026"
]
},
"runtime": {
"acceleration": "cpu",
"cpuThreads": 4,
"device": "cpu",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 1420.912959,
"loadMs": 3250.873542,
"offloadedLayers": 0,
"precision": "nvfp4",
"provider": "llama.cpp CPU",
"requestedDevice": "cpu",
"requestedPrecision": "nvfp4",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 4671.786501,
"charactersPerSecond": 9.14904739073465,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 9.14904739073465
},
"grade": "good"
}
},
{
"precision": "nvfp4",
"device": "cpu",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": [
"3",
"three"
]
},
"runtime": {
"acceleration": "cpu",
"cpuThreads": 4,
"device": "cpu",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 1400.7547499999998,
"loadMs": 3250.873542,
"offloadedLayers": 0,
"precision": "nvfp4",
"provider": "llama.cpp CPU",
"requestedDevice": "cpu",
"requestedPrecision": "nvfp4",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 4651.628291999999,
"charactersPerSecond": 0.7139008452407534,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 0.7139008452407534
},
"grade": "good"
}
},
{
"precision": "nvfp4",
"device": "metal",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": "15"
},
"runtime": {
"acceleration": "partial",
"cpuThreads": 4,
"device": "metal",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 401.596,
"loadMs": 2784.6329170000004,
"offloadedLayers": 25,
"precision": "nvfp4",
"provider": "llama.cpp Metal",
"requestedDevice": "metal",
"requestedPrecision": "nvfp4",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 3186.2289170000004,
"charactersPerSecond": 4.980129284156217,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 4.980129284156217
},
"grade": "good"
}
},
{
"precision": "nvfp4",
"device": "metal",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": "red"
},
"runtime": {
"acceleration": "partial",
"cpuThreads": 4,
"device": "metal",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 800.3299579999999,
"loadMs": 2784.6329170000004,
"offloadedLayers": 25,
"precision": "nvfp4",
"provider": "llama.cpp Metal",
"requestedDevice": "metal",
"requestedPrecision": "nvfp4",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 3584.962875,
"charactersPerSecond": 3.748453959535525,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 3.748453959535525
},
"grade": "good"
}
},
{
"precision": "nvfp4",
"device": "metal",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": "blue"
},
"runtime": {
"acceleration": "partial",
"cpuThreads": 4,
"device": "metal",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 805.1493330000001,
"loadMs": 2784.6329170000004,
"offloadedLayers": 25,
"precision": "nvfp4",
"provider": "llama.cpp Metal",
"requestedDevice": "metal",
"requestedPrecision": "nvfp4",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 3589.78225,
"charactersPerSecond": 4.968022497262629,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 4.968022497262629
},
"grade": "good"
}
},
{
"precision": "nvfp4",
"device": "metal",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": [
"localai2026"
]
},
"runtime": {
"acceleration": "partial",
"cpuThreads": 4,
"device": "metal",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 1149.2737909999998,
"loadMs": 2784.6329170000004,
"offloadedLayers": 25,
"precision": "nvfp4",
"provider": "llama.cpp Metal",
"requestedDevice": "metal",
"requestedPrecision": "nvfp4",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 3933.9067080000004,
"charactersPerSecond": 11.31149087519738,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 11.31149087519738
},
"grade": "good"
}
},
{
"precision": "nvfp4",
"device": "metal",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": [
"3",
"three"
]
},
"runtime": {
"acceleration": "partial",
"cpuThreads": 4,
"device": "metal",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 954.409958,
"loadMs": 2784.6329170000004,
"offloadedLayers": 25,
"precision": "nvfp4",
"provider": "llama.cpp Metal",
"requestedDevice": "metal",
"requestedPrecision": "nvfp4",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 3739.042875,
"charactersPerSecond": 1.0477677769577505,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 1.0477677769577505
},
"grade": "good"
}
},
{
"precision": "bf16",
"device": "cpu",
"status": "fail",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": "15"
},
"runtime": {
"acceleration": "cpu",
"cpuThreads": 4,
"device": "cpu",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 39035.376291,
"loadMs": 3971.8835,
"offloadedLayers": 0,
"precision": "bf16",
"provider": "llama.cpp CPU",
"requestedDevice": "cpu",
"requestedPrecision": "bf16",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 43007.259791000004,
"charactersPerSecond": 0.05123557629086107,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 0.05123557629086107
},
"grade": "moderate"
}
},
{
"precision": "bf16",
"device": "cpu",
"status": "fail",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": "red"
},
"runtime": {
"acceleration": "cpu",
"cpuThreads": 4,
"device": "cpu",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 39428.568833,
"loadMs": 3971.8835,
"offloadedLayers": 0,
"precision": "bf16",
"provider": "llama.cpp CPU",
"requestedDevice": "cpu",
"requestedPrecision": "bf16",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 43400.452332999994,
"charactersPerSecond": 0.07608696153052176,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 0.07608696153052176
},
"grade": "moderate"
}
},
{
"precision": "bf16",
"device": "cpu",
"status": "fail",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": "blue"
},
"runtime": {
"acceleration": "cpu",
"cpuThreads": 4,
"device": "cpu",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 43616.044167,
"loadMs": 3971.8835,
"offloadedLayers": 0,
"precision": "bf16",
"provider": "llama.cpp CPU",
"requestedDevice": "cpu",
"requestedPrecision": "bf16",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 47587.927666999996,
"charactersPerSecond": 0.09170937154879372,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 0.09170937154879372
},
"grade": "moderate"
}
},
{
"precision": "bf16",
"device": "metal",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": "15"
},
"runtime": {
"acceleration": "partial",
"cpuThreads": 4,
"device": "metal",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 186.08333399999998,
"loadMs": 3232.747792,
"offloadedLayers": 25,
"precision": "bf16",
"provider": "llama.cpp Metal",
"requestedDevice": "metal",
"requestedPrecision": "bf16",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 3418.831126,
"charactersPerSecond": 10.747872778332745,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 10.747872778332745
},
"grade": "good"
}
},
{
"precision": "bf16",
"device": "metal",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": "red"
},
"runtime": {
"acceleration": "partial",
"cpuThreads": 4,
"device": "metal",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 169.448625,
"loadMs": 3232.747792,
"offloadedLayers": 25,
"precision": "bf16",
"provider": "llama.cpp Metal",
"requestedDevice": "metal",
"requestedPrecision": "bf16",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 3402.196417,
"charactersPerSecond": 17.704481225504193,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 17.704481225504193
},
"grade": "good"
}
},
{
"precision": "bf16",
"device": "metal",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": "blue"
},
"runtime": {
"acceleration": "partial",
"cpuThreads": 4,
"device": "metal",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 139.03975,
"loadMs": 3232.747792,
"offloadedLayers": 25,
"precision": "bf16",
"provider": "llama.cpp Metal",
"requestedDevice": "metal",
"requestedPrecision": "bf16",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 3371.787542,
"charactersPerSecond": 28.768751382248603,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 28.768751382248603
},
"grade": "good"
}
},
{
"precision": "bf16",
"device": "metal",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": [
"localai2026"
]
},
"runtime": {
"acceleration": "partial",
"cpuThreads": 4,
"device": "metal",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 349.22225000000003,
"loadMs": 3232.747792,
"offloadedLayers": 25,
"precision": "bf16",
"provider": "llama.cpp Metal",
"requestedDevice": "metal",
"requestedPrecision": "bf16",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 3581.970042,
"charactersPerSecond": 37.225577694433845,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 37.225577694433845
},
"grade": "good"
}
},
{
"precision": "bf16",
"device": "metal",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": [
"3",
"three"
]
},
"runtime": {
"acceleration": "partial",
"cpuThreads": 4,
"device": "metal",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 230.592916,
"loadMs": 3232.747792,
"offloadedLayers": 25,
"precision": "bf16",
"provider": "llama.cpp Metal",
"requestedDevice": "metal",
"requestedPrecision": "bf16",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 3463.340708,
"charactersPerSecond": 4.336646664375414,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 4.336646664375414
},
"grade": "good"
}
},
{
"precision": "fp16",
"device": "cpu",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": "15"
},
"runtime": {
"acceleration": "cpu",
"cpuThreads": 4,
"device": "cpu",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 306.45054100000004,
"loadMs": 2542.124958,
"offloadedLayers": 0,
"precision": "fp16",
"provider": "llama.cpp CPU",
"requestedDevice": "cpu",
"requestedPrecision": "fp16",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 2848.575499,
"charactersPerSecond": 6.5263386172322,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 6.5263386172322
},
"grade": "good"
}
},
{
"precision": "fp16",
"device": "cpu",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": "red"
},
"runtime": {
"acceleration": "cpu",
"cpuThreads": 4,
"device": "cpu",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 910.6492910000001,
"loadMs": 2542.124958,
"offloadedLayers": 0,
"precision": "fp16",
"provider": "llama.cpp CPU",
"requestedDevice": "cpu",
"requestedPrecision": "fp16",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 3452.774249,
"charactersPerSecond": 3.294352754292102,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 3.294352754292102
},
"grade": "good"
}
},
{
"precision": "fp16",
"device": "cpu",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": "blue"
},
"runtime": {
"acceleration": "cpu",
"cpuThreads": 4,
"device": "cpu",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 953.506333,
"loadMs": 2542.124958,
"offloadedLayers": 0,
"precision": "fp16",
"provider": "llama.cpp CPU",
"requestedDevice": "cpu",
"requestedPrecision": "fp16",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 3495.6312909999997,
"charactersPerSecond": 4.1950429289912226,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 4.1950429289912226
},
"grade": "good"
}
},
{
"precision": "fp16",
"device": "cpu",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": [
"localai2026"
]
},
"runtime": {
"acceleration": "cpu",
"cpuThreads": 4,
"device": "cpu",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 1808.945417,
"loadMs": 2542.124958,
"offloadedLayers": 0,
"precision": "fp16",
"provider": "llama.cpp CPU",
"requestedDevice": "cpu",
"requestedPrecision": "fp16",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 4351.070374999999,
"charactersPerSecond": 7.186507607045172,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 7.186507607045172
},
"grade": "good"
}
},
{
"precision": "fp16",
"device": "cpu",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": [
"3",
"three"
]
},
"runtime": {
"acceleration": "cpu",
"cpuThreads": 4,
"device": "cpu",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 1727.031792,
"loadMs": 2542.124958,
"offloadedLayers": 0,
"precision": "fp16",
"provider": "llama.cpp CPU",
"requestedDevice": "cpu",
"requestedPrecision": "fp16",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 4269.15675,
"charactersPerSecond": 0.5790281363853433,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 0.5790281363853433
},
"grade": "good"
}
},
{
"precision": "fp16",
"device": "metal",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": "15"
},
"runtime": {
"acceleration": "partial",
"cpuThreads": 4,
"device": "metal",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 116.532125,
"loadMs": 2055.0385420000002,
"offloadedLayers": 25,
"precision": "fp16",
"provider": "llama.cpp Metal",
"requestedDevice": "metal",
"requestedPrecision": "fp16",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 2171.5706670000004,
"charactersPerSecond": 17.16264935527435,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 17.16264935527435
},
"grade": "good"
}
},
{
"precision": "fp16",
"device": "metal",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": "red"
},
"runtime": {
"acceleration": "partial",
"cpuThreads": 4,
"device": "metal",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 135.75216700000001,
"loadMs": 2055.0385420000002,
"offloadedLayers": 25,
"precision": "fp16",
"provider": "llama.cpp Metal",
"requestedDevice": "metal",
"requestedPrecision": "fp16",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 2190.7907090000003,
"charactersPerSecond": 22.09909474225925,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 22.09909474225925
},
"grade": "good"
}
},
{
"precision": "fp16",
"device": "metal",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": "blue"
},
"runtime": {
"acceleration": "partial",
"cpuThreads": 4,
"device": "metal",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 137.64950000000002,
"loadMs": 2055.0385420000002,
"offloadedLayers": 25,
"precision": "fp16",
"provider": "llama.cpp Metal",
"requestedDevice": "metal",
"requestedPrecision": "fp16",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 2192.688042,
"charactersPerSecond": 29.059313691658883,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 29.059313691658883
},
"grade": "good"
}
},
{
"precision": "fp16",
"device": "metal",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": [
"localai2026"
]
},
"runtime": {
"acceleration": "partial",
"cpuThreads": 4,
"device": "metal",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 323.403167,
"loadMs": 2055.0385420000002,
"offloadedLayers": 25,
"precision": "fp16",
"provider": "llama.cpp Metal",
"requestedDevice": "metal",
"requestedPrecision": "fp16",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 2378.441709,
"charactersPerSecond": 40.19750369358628,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 40.19750369358628
},
"grade": "good"
}
},
{
"precision": "fp16",
"device": "metal",
"status": "pass",
"quality": {
"metric": "frozenAnswerMatch",
"score": 1,
"passed": true,
"expected": [
"3",
"three"
]
},
"runtime": {
"acceleration": "partial",
"cpuThreads": 4,
"device": "metal",
"engine": "llama.cpp",
"fallback": false,
"fallbackReason": null,
"inferenceMs": 220.765,
"loadMs": 2055.0385420000002,
"offloadedLayers": 25,
"precision": "fp16",
"provider": "llama.cpp Metal",
"requestedDevice": "metal",
"requestedPrecision": "fp16",
"selectionReason": "Pinned package with verified layer placement",
"version": "b12359",
"artifactRevision": "1add221a5ea6c047fcf1eaaf2d247939d7cf8849",
"totalMs": 2275.803542,
"charactersPerSecond": 4.529703530903903,
"performanceMetric": {
"name": "charactersPerSecond",
"value": 4.529703530903903
},
"grade": "good"
}
}
]Release validation still required
- Full production lint fails on seven pre-existing errors in vendored generated DuckDB worker; changed catalog sources pass.
- Strict native Clippy retains pre-existing findings in unrelated Gradium/GPA/NeuDecide/NeuTTS/Voxtral adapters; LFM3 findings are repaired without behavior changes.