EttinX NLI Desktop
Compare two English statements and score contradiction, entailment and neutral relationships locally with the original XS · 32M checkpoint.
Desktop app
Run EttinX NLI Desktop on your computer
Download the desktop app to load this model, provide input and view results locally.
Public desktop downloads are coming soon.Device requirements
If your device falls short
MemoryAdvisory only: close other apps, reduce image/tile/token/batch settings or choose a smaller verified model/precision. CPU can need more RAM than GPU.
GPUIf the selected GPU runtime fails, existing CPU fallback may recover when enabled, but CPU has its own memory/runtime needs.
PlatformUse a compatible OS/architecture/runtime or another model. These estimates do not add hardware detection or block runs.
Benchmark evidence
Saved desktop measurements apply to the recorded hardware and runtime. Fallback cases report the device that actually ran; they do not prove GPU support on the requested device.
EttinX NLI XS Desktop
72 configurations testedQuality criteria
{ "file": "research/ettinx-nli-family-expansion/quality-gates.json", "sha256": "b98612c89959b9c069902c28bfd95bd8d943b61cefa34ea3c6da14ace9f0c18c", "description": "All six expected labels, per-case minimum margins, complete finite normalized three-label probabilities, <=1000ms/pair." }
| Precision | Device | Result | Platform | Inference / load | |
|---|---|---|---|---|---|
FP32 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 7 ms / 1779 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 7 ms / 1778 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 9 ms / 1836 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 10 ms / 1771 ms |
FP32 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 11 ms / 1764 ms |
FP32 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 11 ms / 1729 ms |
FP32 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 21 ms / 1729 ms |
FP32 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 22 ms / 1764 ms |
FP32 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 22 ms / 1764 ms |
FP32 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 22 ms / 1764 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 22 ms / 1836 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 23 ms / 1771 ms |
FP32 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 23 ms / 1764 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 24 ms / 1778 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 24 ms / 1778 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 24 ms / 1771 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 25 ms / 1778 ms |
FP32 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 26 ms / 1729 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 26 ms / 1836 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 26 ms / 1778 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 26 ms / 1771 ms |
FP32 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 26 ms / 1779 ms |
FP32 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 26 ms / 1729 ms |
FP32 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 27 ms / 1779 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 28 ms / 1771 ms |
FP32 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 28 ms / 1779 ms |
FP32 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 28 ms / 1779 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 31 ms / 1836 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 72 ms / 1778 ms |
FP32 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 72 ms / 1764 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 78 ms / 1771 ms |
FP32 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 82 ms / 1779 ms |
FP32 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 100 ms / 1729 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 106 ms / 1836 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 402 ms / 1729 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 418 ms / 1836 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 3 ms / 1583 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 3 ms / 1605 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 3 ms / 1601 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 3 ms / 1583 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 4 ms / 1583 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 4 ms / 1605 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 4 ms / 1601 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 4 ms / 1605 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 4 ms / 1682 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 4 ms / 1682 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 4 ms / 1601 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 4 ms / 1682 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 4 ms / 1611 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 4 ms / 1605 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 4 ms / 1677 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 4 ms / 1611 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 4 ms / 1611 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 4 ms / 1677 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 4 ms / 1601 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 4 ms / 1677 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 4 ms / 1583 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 4 ms / 1611 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 4 ms / 1682 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 4 ms / 1677 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 5 ms / 1583 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 5 ms / 1605 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 5 ms / 1601 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 5 ms / 1682 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 6 ms / 1611 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 6 ms / 1677 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 66 ms / 1682 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 66 ms / 1605 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 67 ms / 1601 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 68 ms / 1583 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 68 ms / 1611 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 70 ms / 1677 ms |
Test setup & sample details
- Quality gate
- { "file": "research/ettinx-nli-family-expansion/quality-gates.json", "sha256": "b98612c89959b9c069902c28bfd95bd8d943b61cefa34ea3c6da14ace9f0c18c", "description": "All six expected labels, per-case minimum margins, complete finite normalized three-label probabilities, <=1000ms/pair." }
- Tested
- 2026-11-12T22:03:07.787398+00:00
- Environment
- { "os": "macos", "osVersion": "27.2", "architecture": "aarch64", "hardware": "Apple M3 Max;Mac15,11;96GiB RAM", "cpu": "Apple M3 Max", "gpu": "Apple M3 Max", "driver": "OS-managed Metal/MPS macOS27.2 build26B118", "runtime": "App-owned CPython3.13.18;Torch2.10.0;Transformers5.22.1", "timingScope": "Fresh helper/model process per cold repetition; runtime/tokenizer/model startup in load. OS files/driver cache may be warm. Five warm pairs follow six fixtures. Snapshot loads, not Internet transfer.", "concurrentUploadActivity": "none; uploads excluded" }
- Runtime
- transformers 5.22.1
- Artifact revision
- 802360b6ffe6d4627f1d28fd3225522e247cd3e8
- Fixture
{ "file": "model-samples/native/ettinx-nli/quality-gates.json", "url": "https://assets.localai.tools/samples/v1/files/b98612c89959b9c069902c28bfd95bd8d943b61cefa34ea3c6da14ace9f0c18c/ettinx-nli-quality-gates.json", "size": 2262, "sha1": "9104fbed107fd4340666e85c8fcaf3376ba1a2a4", "sha256": "b98612c89959b9c069902c28bfd95bd8d943b61cefa34ea3c6da14ace9f0c18c" }- Rerun when
- artifact-revision; runtime-version; fixture-checksum; quality-gate; backend-behavior
Per-case quality & failure details
[
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.934897530823946,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 67.37174998852424,
"inferenceMs": 66.93391699809581,
"pairsPerSecond": 14.940108764715635,
"inputTokens": 14,
"placement": {
"cpu/torch.float32": 66
},
"loadMs": 1601.1102499905974,
"totalMs": 1668.0441669886932,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 1,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.9997462833562167,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 5.500166997080669,
"inferenceMs": 5.259833007585257,
"pairsPerSecond": 190.12010429948825,
"inputTokens": 26,
"placement": {
"cpu/torch.float32": 66
},
"loadMs": 1601.1102499905974,
"totalMs": 1606.3700829981826,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 1,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9828259916976094,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 4.119207995245233,
"inferenceMs": 3.9364160038530827,
"pairsPerSecond": 254.0381908368348,
"inputTokens": 18,
"placement": {
"cpu/torch.float32": 66
},
"loadMs": 1601.1102499905974,
"totalMs": 1605.0466659944504,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 1,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9937077884096652,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 3.6121250013820827,
"inferenceMs": 3.4612089802976698,
"pairsPerSecond": 288.9163889532028,
"inputTokens": 16,
"placement": {
"cpu/torch.float32": 66
},
"loadMs": 1601.1102499905974,
"totalMs": 1604.571458970895,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 1,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.9895947277545929,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 3.6834999918937683,
"inferenceMs": 3.545208979630843,
"pairsPerSecond": 282.07081888417434,
"inputTokens": 14,
"placement": {
"cpu/torch.float32": 66
},
"loadMs": 1601.1102499905974,
"totalMs": 1604.6554589702282,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 1,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9861082676798105,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 3.8606250018347055,
"inferenceMs": 3.7147499970160425,
"pairsPerSecond": 269.1971198070591,
"inputTokens": 15,
"placement": {
"cpu/torch.float32": 66
},
"loadMs": 1601.1102499905974,
"totalMs": 1604.8249999876134,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 1,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.934897530823946,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 66.45591699634679,
"inferenceMs": 65.99516599089839,
"pairsPerSecond": 15.152624968591084,
"inputTokens": 14,
"placement": {
"cpu/torch.float32": 66
},
"loadMs": 1605.0223339989316,
"totalMs": 1671.01749998983,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 1,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.9997462833562167,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 5.384291987866163,
"inferenceMs": 5.161624983884394,
"pairsPerSecond": 193.7374379429339,
"inputTokens": 26,
"placement": {
"cpu/torch.float32": 66
},
"loadMs": 1605.0223339989316,
"totalMs": 1610.183958982816,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 1,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9828259916976094,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 4.005166993010789,
"inferenceMs": 3.834792005363852,
"pairsPerSecond": 260.77033607070905,
"inputTokens": 18,
"placement": {
"cpu/torch.float32": 66
},
"loadMs": 1605.0223339989316,
"totalMs": 1608.8571260042954,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 1,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9937077884096652,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 3.6725419922731817,
"inferenceMs": 3.5236670228186995,
"pairsPerSecond": 283.79526031380414,
"inputTokens": 16,
"placement": {
"cpu/torch.float32": 66
},
"loadMs": 1605.0223339989316,
"totalMs": 1608.5460010217503,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 1,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.9895947277545929,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 3.4848749928642064,
"inferenceMs": 3.3582499891053885,
"pairsPerSecond": 297.7741392821063,
"inputTokens": 14,
"placement": {
"cpu/torch.float32": 66
},
"loadMs": 1605.0223339989316,
"totalMs": 1608.380583988037,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 1,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9861082676798105,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 3.7288340099621564,
"inferenceMs": 3.6121250013820827,
"pairsPerSecond": 276.84534716195503,
"inputTokens": 15,
"placement": {
"cpu/torch.float32": 66
},
"loadMs": 1605.0223339989316,
"totalMs": 1608.6344590003137,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 1,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.934897530823946,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 68.40912500047125,
"inferenceMs": 67.79166599153541,
"pairsPerSecond": 14.751075746167114,
"inputTokens": 14,
"placement": {
"cpu/torch.float32": 66
},
"loadMs": 1583.1185409915633,
"totalMs": 1650.9102069830988,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 1,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.9997462833562167,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 5.390290985815227,
"inferenceMs": 5.148083000676706,
"pairsPerSecond": 194.2470624247029,
"inputTokens": 26,
"placement": {
"cpu/torch.float32": 66
},
"loadMs": 1583.1185409915633,
"totalMs": 1588.26662399224,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 1,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9828259916976094,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 4.207457997836173,
"inferenceMs": 4.032083001220599,
"pairsPerSecond": 248.01076756040942,
"inputTokens": 18,
"placement": {
"cpu/torch.float32": 66
},
"loadMs": 1583.1185409915633,
"totalMs": 1587.150623992784,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 1,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9937077884096652,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 3.4230419842060655,
"inferenceMs": 3.250667010433972,
"pairsPerSecond": 307.6291717331261,
"inputTokens": 16,
"placement": {
"cpu/torch.float32": 66
},
"loadMs": 1583.1185409915633,
"totalMs": 1586.3692080019973,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 1,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.9895947277545929,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 3.6484999873209745,
"inferenceMs": 3.511957998853177,
"pairsPerSecond": 284.74144631756644,
"inputTokens": 14,
"placement": {
"cpu/torch.float32": 66
},
"loadMs": 1583.1185409915633,
"totalMs": 1586.6304989904165,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 1,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9861082676798105,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 3.5977500083390623,
"inferenceMs": 3.462416003458202,
"pairsPerSecond": 288.8156706187867,
"inputTokens": 15,
"placement": {
"cpu/torch.float32": 66
},
"loadMs": 1583.1185409915633,
"totalMs": 1586.5809569950216,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 1,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.934897530823946,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 70.29912498546764,
"inferenceMs": 69.76466698688455,
"pairsPerSecond": 14.333903438369392,
"inputTokens": 14,
"placement": {
"cpu/torch.float32": 66
},
"loadMs": 1677.209833986126,
"totalMs": 1746.9745009730104,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.9997462833562167,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 6.056458019884303,
"inferenceMs": 5.774833989562467,
"pairsPerSecond": 173.16515103419718,
"inputTokens": 26,
"placement": {
"cpu/torch.float32": 66
},
"loadMs": 1677.209833986126,
"totalMs": 1682.9846679756884,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9828259916976094,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 4.596874990966171,
"inferenceMs": 4.4043330126442015,
"pairsPerSecond": 227.04913482453415,
"inputTokens": 18,
"placement": {
"cpu/torch.float32": 66
},
"loadMs": 1677.209833986126,
"totalMs": 1681.61416699877,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9937077884096652,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 4.046582995215431,
"inferenceMs": 3.8731660170014948,
"pairsPerSecond": 258.1867122685782,
"inputTokens": 16,
"placement": {
"cpu/torch.float32": 66
},
"loadMs": 1677.209833986126,
"totalMs": 1681.0830000031274,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.9895947277545929,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 4.026959009934217,
"inferenceMs": 3.848916996503249,
"pairsPerSecond": 259.81334513280035,
"inputTokens": 14,
"placement": {
"cpu/torch.float32": 66
},
"loadMs": 1677.209833986126,
"totalMs": 1681.0587509826291,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9861082676798105,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 4.178834002232179,
"inferenceMs": 4.003749985713512,
"pairsPerSecond": 249.76584541199546,
"inputTokens": 15,
"placement": {
"cpu/torch.float32": 66
},
"loadMs": 1677.209833986126,
"totalMs": 1681.2135839718394,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.934897530823946,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 66.21812502271496,
"inferenceMs": 65.72812498779967,
"pairsPerSecond": 15.214187232415622,
"inputTokens": 14,
"placement": {
"cpu/torch.float32": 66
},
"loadMs": 1681.628250022186,
"totalMs": 1747.3563750099856,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.9997462833562167,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 5.722582980524749,
"inferenceMs": 5.475792015204206,
"pairsPerSecond": 182.62198367348097,
"inputTokens": 26,
"placement": {
"cpu/torch.float32": 66
},
"loadMs": 1681.628250022186,
"totalMs": 1687.1040420373902,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9828259916976094,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 4.437750001670793,
"inferenceMs": 4.246666008839384,
"pairsPerSecond": 235.47884338408346,
"inputTokens": 18,
"placement": {
"cpu/torch.float32": 66
},
"loadMs": 1681.628250022186,
"totalMs": 1685.8749160310254,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9937077884096652,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 3.889958985382691,
"inferenceMs": 3.728791984030977,
"pairsPerSecond": 268.18336991782496,
"inputTokens": 16,
"placement": {
"cpu/torch.float32": 66
},
"loadMs": 1681.628250022186,
"totalMs": 1685.357042006217,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.9895947277545929,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 3.8690420042257756,
"inferenceMs": 3.7107919924892485,
"pairsPerSecond": 269.4842508079217,
"inputTokens": 14,
"placement": {
"cpu/torch.float32": 66
},
"loadMs": 1681.628250022186,
"totalMs": 1685.3390420146752,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9861082676798105,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 3.767624992178753,
"inferenceMs": 3.638916998170316,
"pairsPerSecond": 274.8070375067115,
"inputTokens": 15,
"placement": {
"cpu/torch.float32": 66
},
"loadMs": 1681.628250022186,
"totalMs": 1685.2671670203563,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.934897530823946,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 68.71270798728801,
"inferenceMs": 68.19349998841062,
"pairsPerSecond": 14.664154210737804,
"inputTokens": 14,
"placement": {
"cpu/torch.float32": 66
},
"loadMs": 1611.1022499972023,
"totalMs": 1679.295749985613,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.9997462833562167,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 5.953874991973862,
"inferenceMs": 5.723000009311363,
"pairsPerSecond": 174.73353108037614,
"inputTokens": 26,
"placement": {
"cpu/torch.float32": 66
},
"loadMs": 1611.1022499972023,
"totalMs": 1616.8252500065137,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9828259916976094,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 4.296625003917143,
"inferenceMs": 4.123833001358435,
"pairsPerSecond": 242.49284577493543,
"inputTokens": 18,
"placement": {
"cpu/torch.float32": 66
},
"loadMs": 1611.1022499972023,
"totalMs": 1615.2260829985607,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9937077884096652,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 3.903042001184076,
"inferenceMs": 3.7576250033453107,
"pairsPerSecond": 266.1255444888001,
"inputTokens": 16,
"placement": {
"cpu/torch.float32": 66
},
"loadMs": 1611.1022499972023,
"totalMs": 1614.8598750005476,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.9895947277545929,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 4.021666012704372,
"inferenceMs": 3.860541997710243,
"pairsPerSecond": 259.03098595821984,
"inputTokens": 14,
"placement": {
"cpu/torch.float32": 66
},
"loadMs": 1611.1022499972023,
"totalMs": 1614.9627919949125,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9861082676798105,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 4.011374985566363,
"inferenceMs": 3.858499985653907,
"pairsPerSecond": 259.1680714573148,
"inputTokens": 15,
"placement": {
"cpu/torch.float32": 66
},
"loadMs": 1611.1022499972023,
"totalMs": 1614.9607499828562,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9348973371088505,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 403.3629169862252,
"inferenceMs": 402.47854200424626,
"pairsPerSecond": 2.4846045084049466,
"inputTokens": 14,
"placement": {
"mps:0/torch.float32": 66
},
"driverAllocatedMemoryBytes": 1110212608,
"loadMs": 1729.397125018295,
"totalMs": 2131.875667022541,
"grade": "good",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.9997462833562167,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 27.251167019130662,
"inferenceMs": 26.461375004146248,
"pairsPerSecond": 37.79093111538268,
"inputTokens": 26,
"placement": {
"mps:0/torch.float32": 66
},
"driverAllocatedMemoryBytes": 1110310912,
"loadMs": 1729.397125018295,
"totalMs": 1755.8585000224411,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9828259600326419,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 21.842249989276752,
"inferenceMs": 21.240040980046615,
"pairsPerSecond": 47.080888447410395,
"inputTokens": 18,
"placement": {
"mps:0/torch.float32": 66
},
"driverAllocatedMemoryBytes": 1110376448,
"loadMs": 1729.397125018295,
"totalMs": 1750.6371659983415,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9937077942304313,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 26.210625015664846,
"inferenceMs": 25.67866598838009,
"pairsPerSecond": 38.94283295138899,
"inputTokens": 16,
"placement": {
"mps:0/torch.float32": 66
},
"driverAllocatedMemoryBytes": 1110441984,
"loadMs": 1729.397125018295,
"totalMs": 1755.075791006675,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.9895947277545929,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 11.26433297758922,
"inferenceMs": 10.644458001479506,
"pairsPerSecond": 93.94560059901659,
"inputTokens": 14,
"placement": {
"mps:0/torch.float32": 66
},
"driverAllocatedMemoryBytes": 1110441984,
"loadMs": 1729.397125018295,
"totalMs": 1740.0415830197744,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9861082504503429,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 100.18437501275912,
"inferenceMs": 99.57466699415818,
"pairsPerSecond": 10.042714981498936,
"inputTokens": 15,
"placement": {
"mps:0/torch.float32": 66
},
"driverAllocatedMemoryBytes": 1110507520,
"loadMs": 1729.397125018295,
"totalMs": 1828.971792012453,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9348973371088505,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 73.4710420074407,
"inferenceMs": 72.34912499552593,
"pairsPerSecond": 13.821867231453595,
"inputTokens": 14,
"placement": {
"mps:0/torch.float32": 66
},
"driverAllocatedMemoryBytes": 1110212608,
"loadMs": 1764.048167009605,
"totalMs": 1836.397292005131,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.9997462833562167,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 22.692291997373104,
"inferenceMs": 22.011541994288564,
"pairsPerSecond": 45.43071086339497,
"inputTokens": 26,
"placement": {
"mps:0/torch.float32": 66
},
"driverAllocatedMemoryBytes": 1110310912,
"loadMs": 1764.048167009605,
"totalMs": 1786.0597090038937,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9828259600326419,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 22.883750003529713,
"inferenceMs": 22.322416974930093,
"pairsPerSecond": 44.798016322474496,
"inputTokens": 18,
"placement": {
"mps:0/torch.float32": 66
},
"driverAllocatedMemoryBytes": 1110376448,
"loadMs": 1764.048167009605,
"totalMs": 1786.3705839845352,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9937077942304313,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 24.19745799852535,
"inferenceMs": 23.286207986529917,
"pairsPerSecond": 42.943874785386164,
"inputTokens": 16,
"placement": {
"mps:0/torch.float32": 66
},
"driverAllocatedMemoryBytes": 1110441984,
"loadMs": 1764.048167009605,
"totalMs": 1787.334374996135,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.9895947277545929,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 11.087499995483086,
"inferenceMs": 10.568375000730157,
"pairsPerSecond": 94.62192625932664,
"inputTokens": 14,
"placement": {
"mps:0/torch.float32": 66
},
"driverAllocatedMemoryBytes": 1110441984,
"loadMs": 1764.048167009605,
"totalMs": 1774.6165420103353,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9861082504503429,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 22.737207997124642,
"inferenceMs": 22.126541007310152,
"pairsPerSecond": 45.19459230747457,
"inputTokens": 15,
"placement": {
"mps:0/torch.float32": 66
},
"driverAllocatedMemoryBytes": 1110507520,
"loadMs": 1764.048167009605,
"totalMs": 1786.1747080169152,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9348973371088505,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 82.78416702523828,
"inferenceMs": 81.90141702652909,
"pairsPerSecond": 12.209800957120011,
"inputTokens": 14,
"placement": {
"mps:0/torch.float32": 66
},
"driverAllocatedMemoryBytes": 1110212608,
"loadMs": 1779.3654589913785,
"totalMs": 1861.2668760179076,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.9997462833562167,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 28.70383401750587,
"inferenceMs": 27.881999994860962,
"pairsPerSecond": 35.86543290238555,
"inputTokens": 26,
"placement": {
"mps:0/torch.float32": 66
},
"driverAllocatedMemoryBytes": 1110310912,
"loadMs": 1779.3654589913785,
"totalMs": 1807.2474589862395,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9828259600326419,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 28.429958008928224,
"inferenceMs": 27.80250000068918,
"pairsPerSecond": 35.967988489352095,
"inputTokens": 18,
"placement": {
"mps:0/torch.float32": 66
},
"driverAllocatedMemoryBytes": 1110376448,
"loadMs": 1779.3654589913785,
"totalMs": 1807.1679589920677,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9937077942304313,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 30.076707975240424,
"inferenceMs": 26.437124994117767,
"pairsPerSecond": 37.82559564334242,
"inputTokens": 16,
"placement": {
"mps:0/torch.float32": 66
},
"driverAllocatedMemoryBytes": 1110441984,
"loadMs": 1779.3654589913785,
"totalMs": 1805.8025839854963,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.9895947277545929,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 8.605667011579499,
"inferenceMs": 7.2109170141629875,
"pairsPerSecond": 138.6786171628236,
"inputTokens": 14,
"placement": {
"mps:0/torch.float32": 66
},
"driverAllocatedMemoryBytes": 1110441984,
"loadMs": 1779.3654589913785,
"totalMs": 1786.5763760055415,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9861082504503429,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 27.578957990044728,
"inferenceMs": 26.97937499033287,
"pairsPerSecond": 37.0653508599927,
"inputTokens": 15,
"placement": {
"mps:0/torch.float32": 66
},
"driverAllocatedMemoryBytes": 1110507520,
"loadMs": 1779.3654589913785,
"totalMs": 1806.3448339817114,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9351372569799423,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 419.44487500586547,
"inferenceMs": 418.47995901480317,
"pairsPerSecond": 2.3896006928365865,
"inputTokens": 14,
"placement": {
"mps:0/torch.float16": 66
},
"driverAllocatedMemoryBytes": 1126989824,
"loadMs": 1835.9750410018023,
"totalMs": 2254.4550000166055,
"grade": "good",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp16",
"precision": "fp16",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.9997481048776535,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 32.0542500121519,
"inferenceMs": 31.36416699271649,
"pairsPerSecond": 31.883518546251327,
"inputTokens": 26,
"placement": {
"mps:0/torch.float16": 66
},
"driverAllocatedMemoryBytes": 1127088128,
"loadMs": 1835.9750410018023,
"totalMs": 1867.3392079945188,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp16",
"precision": "fp16",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9828035058453679,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 22.955624997848645,
"inferenceMs": 22.33862501452677,
"pairsPerSecond": 44.76551261994423,
"inputTokens": 18,
"placement": {
"mps:0/torch.float16": 66
},
"driverAllocatedMemoryBytes": 1127153664,
"loadMs": 1835.9750410018023,
"totalMs": 1858.313666016329,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp16",
"precision": "fp16",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9937266162596643,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 26.538250007433817,
"inferenceMs": 25.913374993251637,
"pairsPerSecond": 38.59011032952752,
"inputTokens": 16,
"placement": {
"mps:0/torch.float16": 66
},
"driverAllocatedMemoryBytes": 1127219200,
"loadMs": 1835.9750410018023,
"totalMs": 1861.888415995054,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp16",
"precision": "fp16",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.9896271242760122,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 9.83908399939537,
"inferenceMs": 9.17779200244695,
"pairsPerSecond": 108.95866889698347,
"inputTokens": 14,
"placement": {
"mps:0/torch.float16": 66
},
"driverAllocatedMemoryBytes": 1127219200,
"loadMs": 1835.9750410018023,
"totalMs": 1845.1528330042493,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp16",
"precision": "fp16",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9860793617554009,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 107.3271659843158,
"inferenceMs": 106.46091599483043,
"pairsPerSecond": 9.393118504151875,
"inputTokens": 15,
"placement": {
"mps:0/torch.float16": 66
},
"driverAllocatedMemoryBytes": 1127284736,
"loadMs": 1835.9750410018023,
"totalMs": 1942.4359569966327,
"grade": "good",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp16",
"precision": "fp16",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9351372569799423,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 79.29158399929293,
"inferenceMs": 78.38454202283174,
"pairsPerSecond": 12.757617435702073,
"inputTokens": 14,
"placement": {
"mps:0/torch.float16": 66
},
"driverAllocatedMemoryBytes": 1126989824,
"loadMs": 1770.9310000063851,
"totalMs": 1849.3155420292169,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp16",
"precision": "fp16",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.9997481048776535,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 27.11816699593328,
"inferenceMs": 26.393790991278365,
"pairsPerSecond": 37.887698676194056,
"inputTokens": 26,
"placement": {
"mps:0/torch.float16": 66
},
"driverAllocatedMemoryBytes": 1127088128,
"loadMs": 1770.9310000063851,
"totalMs": 1797.3247909976635,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp16",
"precision": "fp16",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9828035058453679,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 23.44195800833404,
"inferenceMs": 22.785167006077245,
"pairsPerSecond": 43.88820146603624,
"inputTokens": 18,
"placement": {
"mps:0/torch.float16": 66
},
"driverAllocatedMemoryBytes": 1127153664,
"loadMs": 1770.9310000063851,
"totalMs": 1793.7161670124624,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp16",
"precision": "fp16",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9937266162596643,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 28.265749977435917,
"inferenceMs": 27.661959000397474,
"pairsPerSecond": 36.15072959892794,
"inputTokens": 16,
"placement": {
"mps:0/torch.float16": 66
},
"driverAllocatedMemoryBytes": 1127219200,
"loadMs": 1770.9310000063851,
"totalMs": 1798.5929590067826,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp16",
"precision": "fp16",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.9896271242760122,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 10.620999993989244,
"inferenceMs": 9.977375011658296,
"pairsPerSecond": 100.22676293429151,
"inputTokens": 14,
"placement": {
"mps:0/torch.float16": 66
},
"driverAllocatedMemoryBytes": 1127219200,
"loadMs": 1770.9310000063851,
"totalMs": 1780.9083750180434,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp16",
"precision": "fp16",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9860793617554009,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 25.31062500202097,
"inferenceMs": 24.12587497383356,
"pairsPerSecond": 41.4492739054886,
"inputTokens": 15,
"placement": {
"mps:0/torch.float16": 66
},
"driverAllocatedMemoryBytes": 1127284736,
"loadMs": 1770.9310000063851,
"totalMs": 1795.0568749802187,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp16",
"precision": "fp16",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9351372569799423,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 73.04449999355711,
"inferenceMs": 71.97845799964853,
"pairsPerSecond": 13.89304561101994,
"inputTokens": 14,
"placement": {
"mps:0/torch.float16": 66
},
"driverAllocatedMemoryBytes": 1126989824,
"loadMs": 1777.745250001317,
"totalMs": 1849.7237080009654,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp16",
"precision": "fp16",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.9997481048776535,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 27.022750000469387,
"inferenceMs": 26.262792001944035,
"pairsPerSecond": 38.07668278094643,
"inputTokens": 26,
"placement": {
"mps:0/torch.float16": 66
},
"driverAllocatedMemoryBytes": 1127088128,
"loadMs": 1777.745250001317,
"totalMs": 1804.008042003261,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp16",
"precision": "fp16",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9828035058453679,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 24.61175000644289,
"inferenceMs": 23.88275001430884,
"pairsPerSecond": 41.87122502228057,
"inputTokens": 18,
"placement": {
"mps:0/torch.float16": 66
},
"driverAllocatedMemoryBytes": 1127153664,
"loadMs": 1777.745250001317,
"totalMs": 1801.6280000156257,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp16",
"precision": "fp16",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9937266162596643,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 24.653667001985013,
"inferenceMs": 24.02399998391047,
"pairsPerSecond": 41.62504165291905,
"inputTokens": 16,
"placement": {
"mps:0/torch.float16": 66
},
"driverAllocatedMemoryBytes": 1127219200,
"loadMs": 1777.745250001317,
"totalMs": 1801.7692499852274,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp16",
"precision": "fp16",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.9896271242760122,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 7.966167002450675,
"inferenceMs": 7.31566600734368,
"pairsPerSecond": 136.6929544071819,
"inputTokens": 14,
"placement": {
"mps:0/torch.float16": 66
},
"driverAllocatedMemoryBytes": 1127219200,
"loadMs": 1777.745250001317,
"totalMs": 1785.0609160086606,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp16",
"precision": "fp16",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9860793617554009,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 25.47954098554328,
"inferenceMs": 24.715749983442947,
"pairsPerSecond": 40.46003057442719,
"inputTokens": 15,
"placement": {
"mps:0/torch.float16": 66
},
"driverAllocatedMemoryBytes": 1127284736,
"loadMs": 1777.745250001317,
"totalMs": 1802.4609999847598,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "802360b6ffe6d4627f1d28fd3225522e247cd3e8",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp16",
"precision": "fp16",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
}
]EttinX NLI XXS Desktop
72 configurations testedQuality criteria
{ "file": "research/ettinx-nli-family-expansion/quality-gates.json", "sha256": "b98612c89959b9c069902c28bfd95bd8d943b61cefa34ea3c6da14ace9f0c18c", "description": "All six expected labels, per-case minimum margins, complete finite normalized three-label probabilities, <=1000ms/pair." }
| Precision | Device | Result | Platform | Inference / load | |
|---|---|---|---|---|---|
FP32 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 4 ms / 1717 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 5 ms / 1699 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 6 ms / 1798 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 8 ms / 1703 ms |
FP32 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 9 ms / 1739 ms |
FP32 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 10 ms / 1709 ms |
FP32 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 21 ms / 1717 ms |
FP32 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 21 ms / 1709 ms |
FP32 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 22 ms / 1709 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 23 ms / 1798 ms |
FP32 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 23 ms / 1739 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 23 ms / 1699 ms |
FP32 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 23 ms / 1709 ms |
FP32 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 23 ms / 1717 ms |
FP32 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 23 ms / 1717 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 24 ms / 1699 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 24 ms / 1699 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 24 ms / 1798 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 24 ms / 1798 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 24 ms / 1798 ms |
FP32 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 25 ms / 1709 ms |
FP32 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 25 ms / 1717 ms |
FP32 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 27 ms / 1739 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 27 ms / 1703 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 27 ms / 1699 ms |
FP32 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 43 ms / 1739 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 47 ms / 1703 ms |
FP32 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 66 ms / 1717 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 68 ms / 1703 ms |
FP32 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 69 ms / 1709 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 70 ms / 1699 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 73 ms / 1798 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 222 ms / 1739 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 243 ms / 1703 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 554 ms / 1739 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 566 ms / 1703 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 2 ms / 10387 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 2 ms / 1586 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 2 ms / 1612 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 2 ms / 1623 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 2 ms / 1586 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 2 ms / 1612 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 2 ms / 1612 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 2 ms / 10387 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 2 ms / 1586 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 2 ms / 1614 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 2 ms / 1614 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 2 ms / 1623 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 2 ms / 10387 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 2 ms / 1614 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 2 ms / 1623 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 2 ms / 1612 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 2 ms / 1586 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 2 ms / 1681 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 2 ms / 10387 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 2 ms / 1681 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 3 ms / 1614 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 3 ms / 1623 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 3 ms / 1681 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 3 ms / 1681 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 3 ms / 1612 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 3 ms / 1586 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 4 ms / 10387 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 4 ms / 1614 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 4 ms / 1623 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 4 ms / 1681 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 22 ms / 1612 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 23 ms / 1623 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 23 ms / 1586 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 23 ms / 1614 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 23 ms / 10387 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 24 ms / 1681 ms |
Test setup & sample details
- Quality gate
- { "file": "research/ettinx-nli-family-expansion/quality-gates.json", "sha256": "b98612c89959b9c069902c28bfd95bd8d943b61cefa34ea3c6da14ace9f0c18c", "description": "All six expected labels, per-case minimum margins, complete finite normalized three-label probabilities, <=1000ms/pair." }
- Tested
- 2026-11-12T22:03:07.787398+00:00
- Environment
- { "os": "macos", "osVersion": "27.2", "architecture": "aarch64", "hardware": "Apple M3 Max;Mac15,11;96GiB RAM", "cpu": "Apple M3 Max", "gpu": "Apple M3 Max", "driver": "OS-managed Metal/MPS macOS27.2 build26B118", "runtime": "App-owned CPython3.13.18;Torch2.10.0;Transformers5.22.1", "timingScope": "Fresh helper/model process per cold repetition; runtime/tokenizer/model startup in load. OS files/driver cache may be warm. Five warm pairs follow six fixtures. Snapshot loads, not Internet transfer.", "concurrentUploadActivity": "none; uploads excluded" }
- Runtime
- transformers 5.22.1
- Artifact revision
- 04ded685adf9835dff917af61327bb4aa1dbec01
- Fixture
{ "file": "model-samples/native/ettinx-nli/quality-gates.json", "url": "https://assets.localai.tools/samples/v1/files/b98612c89959b9c069902c28bfd95bd8d943b61cefa34ea3c6da14ace9f0c18c/ettinx-nli-quality-gates.json", "size": 2262, "sha1": "9104fbed107fd4340666e85c8fcaf3376ba1a2a4", "sha256": "b98612c89959b9c069902c28bfd95bd8d943b61cefa34ea3c6da14ace9f0c18c" }- Rerun when
- artifact-revision; runtime-version; fixture-checksum; quality-gate; backend-behavior
Per-case quality & failure details
[
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9350983984768391,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 24.097957997582853,
"inferenceMs": 23.155000002589077,
"pairsPerSecond": 43.18721657906219,
"inputTokens": 14,
"placement": {
"cpu/torch.float32": 48
},
"loadMs": 10387.186416017357,
"totalMs": 10410.341416019946,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 1,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.999408537405543,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 3.7819579883944243,
"inferenceMs": 3.5465830005705357,
"pairsPerSecond": 281.96153870898576,
"inputTokens": 26,
"placement": {
"cpu/torch.float32": 48
},
"loadMs": 10387.186416017357,
"totalMs": 10390.732999017928,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 1,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9768406730145216,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 2.58004100760445,
"inferenceMs": 2.419374999590218,
"pairsPerSecond": 413.32988898760016,
"inputTokens": 18,
"placement": {
"cpu/torch.float32": 48
},
"loadMs": 10387.186416017357,
"totalMs": 10389.605791016947,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 1,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9936191779561341,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 1.8935840053018183,
"inferenceMs": 1.764917018590495,
"pairsPerSecond": 566.5988765854975,
"inputTokens": 16,
"placement": {
"cpu/torch.float32": 48
},
"loadMs": 10387.186416017357,
"totalMs": 10388.951333035948,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 1,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.8207429498434067,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 2.1315830235835165,
"inferenceMs": 1.9974579918198287,
"pairsPerSecond": 500.6363107986705,
"inputTokens": 14,
"placement": {
"cpu/torch.float32": 48
},
"loadMs": 10387.186416017357,
"totalMs": 10389.183874009177,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 1,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9908071178942919,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 2.1797079825773835,
"inferenceMs": 2.064542000880465,
"pairsPerSecond": 484.36893004527406,
"inputTokens": 15,
"placement": {
"cpu/torch.float32": 48
},
"loadMs": 10387.186416017357,
"totalMs": 10389.250958018238,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 1,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9350983984768391,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 23.26845799689181,
"inferenceMs": 22.794874996179715,
"pairsPerSecond": 43.869510149434646,
"inputTokens": 14,
"placement": {
"cpu/torch.float32": 48
},
"loadMs": 1585.6987080187537,
"totalMs": 1608.4935830149334,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 1,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.999408537405543,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 3.6877499951515347,
"inferenceMs": 3.4679579839576036,
"pairsPerSecond": 288.35412788329364,
"inputTokens": 26,
"placement": {
"cpu/torch.float32": 48
},
"loadMs": 1585.6987080187537,
"totalMs": 1589.1666660027113,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 1,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9768406730145216,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 2.352707990212366,
"inferenceMs": 2.2236660006456077,
"pairsPerSecond": 449.70782469564455,
"inputTokens": 18,
"placement": {
"cpu/torch.float32": 48
},
"loadMs": 1585.6987080187537,
"totalMs": 1587.9223740193993,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 1,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9936191779561341,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 1.8783329869620502,
"inferenceMs": 1.7728749953676015,
"pairsPerSecond": 564.0555609464458,
"inputTokens": 16,
"placement": {
"cpu/torch.float32": 48
},
"loadMs": 1585.6987080187537,
"totalMs": 1587.4715830141213,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 1,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.8207429498434067,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 2.105916995787993,
"inferenceMs": 2.004207985009998,
"pairsPerSecond": 498.950212492548,
"inputTokens": 14,
"placement": {
"cpu/torch.float32": 48
},
"loadMs": 1585.6987080187537,
"totalMs": 1587.7029160037637,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 1,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9908071178942919,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 2.0364579977467656,
"inferenceMs": 1.9127920095343143,
"pairsPerSecond": 522.7959940314988,
"inputTokens": 15,
"placement": {
"cpu/torch.float32": 48
},
"loadMs": 1585.6987080187537,
"totalMs": 1587.611500028288,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 1,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9350983984768391,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 22.57716699386947,
"inferenceMs": 22.135249979328364,
"pairsPerSecond": 45.17681078523525,
"inputTokens": 14,
"placement": {
"cpu/torch.float32": 48
},
"loadMs": 1612.2754579992034,
"totalMs": 1634.4107079785317,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 1,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.999408537405543,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 3.649292018963024,
"inferenceMs": 3.4455419809091836,
"pairsPerSecond": 290.2301018361493,
"inputTokens": 26,
"placement": {
"cpu/torch.float32": 48
},
"loadMs": 1612.2754579992034,
"totalMs": 1615.7209999801125,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 1,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9768406730145216,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 2.285916998516768,
"inferenceMs": 2.1607920061796904,
"pairsPerSecond": 462.79327077297626,
"inputTokens": 18,
"placement": {
"cpu/torch.float32": 48
},
"loadMs": 1612.2754579992034,
"totalMs": 1614.436250005383,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 1,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9936191779561341,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 1.95537501713261,
"inferenceMs": 1.8093750113621354,
"pairsPerSecond": 552.6770258903814,
"inputTokens": 16,
"placement": {
"cpu/torch.float32": 48
},
"loadMs": 1612.2754579992034,
"totalMs": 1614.0848330105655,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 1,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.8207429498434067,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 2.0876670023426414,
"inferenceMs": 1.9934169831685722,
"pairsPerSecond": 501.6511891107108,
"inputTokens": 14,
"placement": {
"cpu/torch.float32": 48
},
"loadMs": 1612.2754579992034,
"totalMs": 1614.268874982372,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 1,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9908071178942919,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 2.029082999797538,
"inferenceMs": 1.923417003126815,
"pairsPerSecond": 519.9080586135735,
"inputTokens": 15,
"placement": {
"cpu/torch.float32": 48
},
"loadMs": 1612.2754579992034,
"totalMs": 1614.1988750023302,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 1,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9350983984768391,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 23.31858398974873,
"inferenceMs": 22.842333011794835,
"pairsPerSecond": 43.77836534839245,
"inputTokens": 14,
"placement": {
"cpu/torch.float32": 48
},
"loadMs": 1613.9189589885063,
"totalMs": 1636.7612920003012,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.999408537405543,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 3.922290983609855,
"inferenceMs": 3.6986659979447722,
"pairsPerSecond": 270.3677489548035,
"inputTokens": 26,
"placement": {
"cpu/torch.float32": 48
},
"loadMs": 1613.9189589885063,
"totalMs": 1617.6176249864511,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9768406730145216,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 2.6880829827859998,
"inferenceMs": 2.526542026316747,
"pairsPerSecond": 395.79788880766165,
"inputTokens": 18,
"placement": {
"cpu/torch.float32": 48
},
"loadMs": 1613.9189589885063,
"totalMs": 1616.445501014823,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9936191779561341,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 2.1497500129044056,
"inferenceMs": 2.021666005020961,
"pairsPerSecond": 494.6415468808517,
"inputTokens": 16,
"placement": {
"cpu/torch.float32": 48
},
"loadMs": 1613.9189589885063,
"totalMs": 1615.9406249935273,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.8207429498434067,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 2.1741249947808683,
"inferenceMs": 2.065250009763986,
"pairsPerSecond": 484.20287871795176,
"inputTokens": 14,
"placement": {
"cpu/torch.float32": 48
},
"loadMs": 1613.9189589885063,
"totalMs": 1615.9842089982703,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9908071178942919,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 2.1328330039978027,
"inferenceMs": 2.006208000238985,
"pairsPerSecond": 498.45280244165974,
"inputTokens": 15,
"placement": {
"cpu/torch.float32": 48
},
"loadMs": 1613.9189589885063,
"totalMs": 1615.9251669887453,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9350983984768391,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 23.26866702060215,
"inferenceMs": 22.739042004104704,
"pairsPerSecond": 43.97722647328267,
"inputTokens": 14,
"placement": {
"cpu/torch.float32": 48
},
"loadMs": 1622.9219579836354,
"totalMs": 1645.66099998774,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.999408537405543,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 4.198999988147989,
"inferenceMs": 3.9350420120172203,
"pairsPerSecond": 254.12689291400224,
"inputTokens": 26,
"placement": {
"cpu/torch.float32": 48
},
"loadMs": 1622.9219579836354,
"totalMs": 1626.8569999956526,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9768406730145216,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 2.7622080233413726,
"inferenceMs": 2.596332982648164,
"pairsPerSecond": 385.15860896241315,
"inputTokens": 18,
"placement": {
"cpu/torch.float32": 48
},
"loadMs": 1622.9219579836354,
"totalMs": 1625.5182909662835,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9936191779561341,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 1.9673750211950392,
"inferenceMs": 1.8412499921396375,
"pairsPerSecond": 543.1093030653285,
"inputTokens": 16,
"placement": {
"cpu/torch.float32": 48
},
"loadMs": 1622.9219579836354,
"totalMs": 1624.763207975775,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.8207429498434067,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 2.1747499995399266,
"inferenceMs": 2.0687499782070518,
"pairsPerSecond": 483.38369089273993,
"inputTokens": 14,
"placement": {
"cpu/torch.float32": 48
},
"loadMs": 1622.9219579836354,
"totalMs": 1624.9907079618424,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9908071178942919,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 2.1874999802093953,
"inferenceMs": 2.058790996670723,
"pairsPerSecond": 485.72196090671804,
"inputTokens": 15,
"placement": {
"cpu/torch.float32": 48
},
"loadMs": 1622.9219579836354,
"totalMs": 1624.980748980306,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9350983984768391,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 24.119584006257355,
"inferenceMs": 23.622250009793788,
"pairsPerSecond": 42.33296995778979,
"inputTokens": 14,
"placement": {
"cpu/torch.float32": 48
},
"loadMs": 1681.1614580219612,
"totalMs": 1704.783708031755,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.999408537405543,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 4.225834010867402,
"inferenceMs": 3.972915990743786,
"pairsPerSecond": 251.70429033229718,
"inputTokens": 26,
"placement": {
"cpu/torch.float32": 48
},
"loadMs": 1681.1614580219612,
"totalMs": 1685.134374012705,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9768406730145216,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 2.9890410078223795,
"inferenceMs": 2.8310839843470603,
"pairsPerSecond": 353.2215948127842,
"inputTokens": 18,
"placement": {
"cpu/torch.float32": 48
},
"loadMs": 1681.1614580219612,
"totalMs": 1683.9925420063082,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9936191779561341,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 2.4772089964244515,
"inferenceMs": 2.308916999027133,
"pairsPerSecond": 433.1034854961664,
"inputTokens": 16,
"placement": {
"cpu/torch.float32": 48
},
"loadMs": 1681.1614580219612,
"totalMs": 1683.4703750209883,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.8207429498434067,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 2.81649999669753,
"inferenceMs": 2.622041996801272,
"pairsPerSecond": 381.38214461093213,
"inputTokens": 14,
"placement": {
"cpu/torch.float32": 48
},
"loadMs": 1681.1614580219612,
"totalMs": 1683.7835000187624,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9908071178942919,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 2.5808749778661877,
"inferenceMs": 2.4289170105475932,
"pairsPerSecond": 411.70612073508124,
"inputTokens": 15,
"placement": {
"cpu/torch.float32": 48
},
"loadMs": 1681.1614580219612,
"totalMs": 1683.5903750325087,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9350983835756779,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 555.3770830156282,
"inferenceMs": 554.4127080065664,
"pairsPerSecond": 1.8037104589387516,
"inputTokens": 14,
"placement": {
"mps:0/torch.float32": 48
},
"driverAllocatedMemoryBytes": 1110212608,
"loadMs": 1738.9782500104047,
"totalMs": 2293.390958016971,
"grade": "good",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.9994085376674775,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 30.477457999950275,
"inferenceMs": 26.561375008895993,
"pairsPerSecond": 37.64865334212095,
"inputTokens": 26,
"placement": {
"mps:0/torch.float32": 48
},
"driverAllocatedMemoryBytes": 1110294528,
"loadMs": 1738.9782500104047,
"totalMs": 1765.5396250193007,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9768408071249723,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 23.18108401959762,
"inferenceMs": 22.60216700960882,
"pairsPerSecond": 44.24354530142494,
"inputTokens": 18,
"placement": {
"mps:0/torch.float32": 48
},
"driverAllocatedMemoryBytes": 1110360064,
"loadMs": 1738.9782500104047,
"totalMs": 1761.5804170200136,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9936191795859486,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 43.72479100129567,
"inferenceMs": 43.070374988019466,
"pairsPerSecond": 23.217815035930425,
"inputTokens": 16,
"placement": {
"mps:0/torch.float32": 48
},
"driverAllocatedMemoryBytes": 1110425600,
"loadMs": 1738.9782500104047,
"totalMs": 1782.0486249984242,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.8207431510090828,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 9.917582996422425,
"inferenceMs": 9.303375001763925,
"pairsPerSecond": 107.48787400383189,
"inputTokens": 14,
"placement": {
"mps:0/torch.float32": 48
},
"driverAllocatedMemoryBytes": 1110425600,
"loadMs": 1738.9782500104047,
"totalMs": 1748.2816250121687,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9908071213867515,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 222.5927910185419,
"inferenceMs": 221.98145900620148,
"pairsPerSecond": 4.504880743089733,
"inputTokens": 15,
"placement": {
"mps:0/torch.float32": 48
},
"driverAllocatedMemoryBytes": 1110491136,
"loadMs": 1738.9782500104047,
"totalMs": 1960.9597090166062,
"grade": "good",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9350983835756779,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 66.47625000914559,
"inferenceMs": 65.61154199880548,
"pairsPerSecond": 15.24122082084591,
"inputTokens": 14,
"placement": {
"mps:0/torch.float32": 48
},
"driverAllocatedMemoryBytes": 1110212608,
"loadMs": 1716.7617499944754,
"totalMs": 1782.3732919932809,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.9994085376674775,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 26.018042000941932,
"inferenceMs": 25.337041006423533,
"pairsPerSecond": 39.46790786447701,
"inputTokens": 26,
"placement": {
"mps:0/torch.float32": 48
},
"driverAllocatedMemoryBytes": 1110294528,
"loadMs": 1716.7617499944754,
"totalMs": 1742.098791000899,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9768408071249723,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 21.45870801177807,
"inferenceMs": 20.856417017057538,
"pairsPerSecond": 47.946874057137634,
"inputTokens": 18,
"placement": {
"mps:0/torch.float32": 48
},
"driverAllocatedMemoryBytes": 1110360064,
"loadMs": 1716.7617499944754,
"totalMs": 1737.618167011533,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9936191795859486,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 23.38495900039561,
"inferenceMs": 22.831125010270625,
"pairsPerSecond": 43.79985653576633,
"inputTokens": 16,
"placement": {
"mps:0/torch.float32": 48
},
"driverAllocatedMemoryBytes": 1110425600,
"loadMs": 1716.7617499944754,
"totalMs": 1739.592875004746,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.8207431510090828,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 4.656082979636267,
"inferenceMs": 4.01108298683539,
"pairsPerSecond": 249.3092272790313,
"inputTokens": 14,
"placement": {
"mps:0/torch.float32": 48
},
"driverAllocatedMemoryBytes": 1110425600,
"loadMs": 1716.7617499944754,
"totalMs": 1720.7728329813108,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9908071213867515,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 23.487666010623798,
"inferenceMs": 23.005499999271706,
"pairsPerSecond": 43.46786638115483,
"inputTokens": 15,
"placement": {
"mps:0/torch.float32": 48
},
"driverAllocatedMemoryBytes": 1110491136,
"loadMs": 1716.7617499944754,
"totalMs": 1739.767249993747,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9350983835756779,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 70.25395799428225,
"inferenceMs": 69.47154199588113,
"pairsPerSecond": 14.394383243419135,
"inputTokens": 14,
"placement": {
"mps:0/torch.float32": 48
},
"driverAllocatedMemoryBytes": 1110212608,
"loadMs": 1708.8568750186823,
"totalMs": 1778.3284170145635,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.9994085376674775,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 25.35354101564735,
"inferenceMs": 24.622208991786465,
"pairsPerSecond": 40.61374023482549,
"inputTokens": 26,
"placement": {
"mps:0/torch.float32": 48
},
"driverAllocatedMemoryBytes": 1110294528,
"loadMs": 1708.8568750186823,
"totalMs": 1733.4790840104688,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9768408071249723,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 21.49729200755246,
"inferenceMs": 20.942292001564056,
"pairsPerSecond": 47.75026534465835,
"inputTokens": 18,
"placement": {
"mps:0/torch.float32": 48
},
"driverAllocatedMemoryBytes": 1110360064,
"loadMs": 1708.8568750186823,
"totalMs": 1729.7991670202464,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9936191795859486,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 23.198415990918875,
"inferenceMs": 22.629959014011547,
"pairsPerSecond": 44.18920950677997,
"inputTokens": 16,
"placement": {
"mps:0/torch.float32": 48
},
"driverAllocatedMemoryBytes": 1110425600,
"loadMs": 1708.8568750186823,
"totalMs": 1731.4868340326939,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.8207431510090828,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 10.633124999003485,
"inferenceMs": 10.110584000358358,
"pairsPerSecond": 98.90625506543995,
"inputTokens": 14,
"placement": {
"mps:0/torch.float32": 48
},
"driverAllocatedMemoryBytes": 1110425600,
"loadMs": 1708.8568750186823,
"totalMs": 1718.9674590190407,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9908071213867515,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 22.70829101325944,
"inferenceMs": 22.120000008726493,
"pairsPerSecond": 45.20795658252681,
"inputTokens": 15,
"placement": {
"mps:0/torch.float32": 48
},
"driverAllocatedMemoryBytes": 1110491136,
"loadMs": 1708.8568750186823,
"totalMs": 1730.9768750274088,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9347764737904072,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 567.3714580188971,
"inferenceMs": 566.3940829981584,
"pairsPerSecond": 1.7655551673608345,
"inputTokens": 14,
"placement": {
"mps:0/torch.float16": 48
},
"driverAllocatedMemoryBytes": 1101824000,
"loadMs": 1702.5347909948323,
"totalMs": 2268.9288739929907,
"grade": "good",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp16",
"precision": "fp16",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.9994080024480354,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 47.96075000194833,
"inferenceMs": 47.36220798804425,
"pairsPerSecond": 21.113880506846986,
"inputTokens": 26,
"placement": {
"mps:0/torch.float16": 48
},
"driverAllocatedMemoryBytes": 1101905920,
"loadMs": 1702.5347909948323,
"totalMs": 1749.8969989828765,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp16",
"precision": "fp16",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9769495939835906,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 27.198625000892207,
"inferenceMs": 26.56854101223871,
"pairsPerSecond": 37.63849883737889,
"inputTokens": 18,
"placement": {
"mps:0/torch.float16": 48
},
"driverAllocatedMemoryBytes": 1101971456,
"loadMs": 1702.5347909948323,
"totalMs": 1729.103332007071,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp16",
"precision": "fp16",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.993615276645869,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 69.7937500081025,
"inferenceMs": 68.35162499919534,
"pairsPerSecond": 14.630230078828008,
"inputTokens": 16,
"placement": {
"mps:0/torch.float16": 48
},
"driverAllocatedMemoryBytes": 1102036992,
"loadMs": 1702.5347909948323,
"totalMs": 1770.8864159940276,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp16",
"precision": "fp16",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.8206676840782166,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 8.29233301919885,
"inferenceMs": 7.6804170093964785,
"pairsPerSecond": 130.20126365229473,
"inputTokens": 14,
"placement": {
"mps:0/torch.float16": 48
},
"driverAllocatedMemoryBytes": 1102036992,
"loadMs": 1702.5347909948323,
"totalMs": 1710.2152080042288,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp16",
"precision": "fp16",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9907999183051288,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 243.53587499354035,
"inferenceMs": 242.9982080066111,
"pairsPerSecond": 4.1152566852377515,
"inputTokens": 15,
"placement": {
"mps:0/torch.float16": 48
},
"driverAllocatedMemoryBytes": 1102102528,
"loadMs": 1702.5347909948323,
"totalMs": 1945.5329990014434,
"grade": "good",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp16",
"precision": "fp16",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9347764737904072,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 74.32691700523719,
"inferenceMs": 73.4237500000745,
"pairsPerSecond": 13.619571323978757,
"inputTokens": 14,
"placement": {
"mps:0/torch.float16": 48
},
"driverAllocatedMemoryBytes": 1101824000,
"loadMs": 1797.8989169932902,
"totalMs": 1871.3226669933647,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp16",
"precision": "fp16",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.9994080024480354,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 25.472417008131742,
"inferenceMs": 23.74145801877603,
"pairsPerSecond": 42.1204122850899,
"inputTokens": 26,
"placement": {
"mps:0/torch.float16": 48
},
"driverAllocatedMemoryBytes": 1101905920,
"loadMs": 1797.8989169932902,
"totalMs": 1821.6403750120662,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp16",
"precision": "fp16",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9769495939835906,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 23.10370802297257,
"inferenceMs": 22.51895802328363,
"pairsPerSecond": 44.40702802350105,
"inputTokens": 18,
"placement": {
"mps:0/torch.float16": 48
},
"driverAllocatedMemoryBytes": 1101971456,
"loadMs": 1797.8989169932902,
"totalMs": 1820.4178750165738,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp16",
"precision": "fp16",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.993615276645869,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 25.614708982175216,
"inferenceMs": 23.936290992423892,
"pairsPerSecond": 41.77756697211407,
"inputTokens": 16,
"placement": {
"mps:0/torch.float16": 48
},
"driverAllocatedMemoryBytes": 1102036992,
"loadMs": 1797.8989169932902,
"totalMs": 1821.835207985714,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp16",
"precision": "fp16",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.8206676840782166,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 6.822833005571738,
"inferenceMs": 6.102415994973853,
"pairsPerSecond": 163.86952328776542,
"inputTokens": 14,
"placement": {
"mps:0/torch.float16": 48
},
"driverAllocatedMemoryBytes": 1102036992,
"loadMs": 1797.8989169932902,
"totalMs": 1804.001332988264,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp16",
"precision": "fp16",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9907999183051288,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 24.74141598213464,
"inferenceMs": 24.16783300577663,
"pairsPerSecond": 41.377313380185086,
"inputTokens": 15,
"placement": {
"mps:0/torch.float16": 48
},
"driverAllocatedMemoryBytes": 1102102528,
"loadMs": 1797.8989169932902,
"totalMs": 1822.0667499990668,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp16",
"precision": "fp16",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9347764737904072,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 70.83483299356885,
"inferenceMs": 69.93904200498946,
"pairsPerSecond": 14.298165535762699,
"inputTokens": 14,
"placement": {
"mps:0/torch.float16": 48
},
"driverAllocatedMemoryBytes": 1101824000,
"loadMs": 1699.4528750074096,
"totalMs": 1769.391917012399,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp16",
"precision": "fp16",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.9994080024480354,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 27.361290995031595,
"inferenceMs": 26.621167024131864,
"pairsPerSecond": 37.56409323052999,
"inputTokens": 26,
"placement": {
"mps:0/torch.float16": 48
},
"driverAllocatedMemoryBytes": 1101905920,
"loadMs": 1699.4528750074096,
"totalMs": 1726.0740420315415,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp16",
"precision": "fp16",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9769495939835906,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 23.28304198454134,
"inferenceMs": 22.61487499345094,
"pairsPerSecond": 44.218683512050845,
"inputTokens": 18,
"placement": {
"mps:0/torch.float16": 48
},
"driverAllocatedMemoryBytes": 1101971456,
"loadMs": 1699.4528750074096,
"totalMs": 1722.0677500008605,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp16",
"precision": "fp16",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.993615276645869,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 24.307332991156727,
"inferenceMs": 23.73883401742205,
"pairsPerSecond": 42.1250681169132,
"inputTokens": 16,
"placement": {
"mps:0/torch.float16": 48
},
"driverAllocatedMemoryBytes": 1102036992,
"loadMs": 1699.4528750074096,
"totalMs": 1723.1917090248317,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp16",
"precision": "fp16",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.8206676840782166,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 5.859209020854905,
"inferenceMs": 5.298291012877598,
"pairsPerSecond": 188.74010460533043,
"inputTokens": 14,
"placement": {
"mps:0/torch.float16": 48
},
"driverAllocatedMemoryBytes": 1102036992,
"loadMs": 1699.4528750074096,
"totalMs": 1704.7511660202872,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp16",
"precision": "fp16",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9907999183051288,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 24.171249999199063,
"inferenceMs": 23.518874979345128,
"pairsPerSecond": 42.519040595191115,
"inputTokens": 15,
"placement": {
"mps:0/torch.float16": 48
},
"driverAllocatedMemoryBytes": 1102102528,
"loadMs": 1699.4528750074096,
"totalMs": 1722.9717499867547,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "04ded685adf9835dff917af61327bb4aa1dbec01",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp16",
"precision": "fp16",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
}
]EttinX NLI S Desktop
72 configurations testedQuality criteria
{ "file": "research/ettinx-nli-family-expansion/quality-gates.json", "sha256": "b98612c89959b9c069902c28bfd95bd8d943b61cefa34ea3c6da14ace9f0c18c", "description": "All six expected labels, per-case minimum margins, complete finite normalized three-label probabilities, <=1000ms/pair." }
| Precision | Device | Result | Platform | Inference / load | |
|---|---|---|---|---|---|
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 9 ms / 1900 ms |
FP32 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 11 ms / 1849 ms |
FP32 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 11 ms / 1932 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 11 ms / 1799 ms |
FP32 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 12 ms / 1842 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 13 ms / 1859 ms |
FP32 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 24 ms / 1849 ms |
FP32 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 24 ms / 1842 ms |
FP32 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 25 ms / 1849 ms |
FP32 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 25 ms / 1842 ms |
FP32 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 26 ms / 1842 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 26 ms / 1900 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 26 ms / 1900 ms |
FP32 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 26 ms / 1849 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 27 ms / 1799 ms |
FP32 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 27 ms / 1842 ms |
FP32 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 27 ms / 1849 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 28 ms / 1900 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 29 ms / 1859 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 29 ms / 1900 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 29 ms / 1799 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 29 ms / 1799 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 30 ms / 1799 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 30 ms / 1859 ms |
FP32 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 31 ms / 1932 ms |
FP32 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 31 ms / 1932 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 31 ms / 1859 ms |
FP32 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 33 ms / 1932 ms |
FP32 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 79 ms / 1842 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 85 ms / 1799 ms |
FP32 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 86 ms / 1849 ms |
FP16 | mpsAccelerated | Passed | Very fast | macos · aarch64 | 88 ms / 1900 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 181 ms / 1932 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 186 ms / 1859 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 196 ms / 1932 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 258 ms / 1859 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 8 ms / 1619 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 8 ms / 1685 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 9 ms / 1685 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 9 ms / 1622 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 9 ms / 1619 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 9 ms / 1619 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 9 ms / 1708 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 9 ms / 1663 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 9 ms / 1664 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 9 ms / 1685 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 9 ms / 1708 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 9 ms / 1708 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 9 ms / 1663 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 9 ms / 1622 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 9 ms / 1708 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 10 ms / 1663 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 10 ms / 1619 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 10 ms / 1622 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 10 ms / 1685 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 10 ms / 1622 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 10 ms / 1664 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 10 ms / 1663 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 10 ms / 1664 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 10 ms / 1664 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 11 ms / 1619 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 11 ms / 1685 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 11 ms / 1622 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 11 ms / 1663 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 11 ms / 1664 ms |
FP32 | cpu | Passed | Very fast | macos · aarch64 | 12 ms / 1708 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 216 ms / 1619 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 217 ms / 1685 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 218 ms / 1622 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 221 ms / 1663 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 223 ms / 1664 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 226 ms / 1708 ms |
Test setup & sample details
- Quality gate
- { "file": "research/ettinx-nli-family-expansion/quality-gates.json", "sha256": "b98612c89959b9c069902c28bfd95bd8d943b61cefa34ea3c6da14ace9f0c18c", "description": "All six expected labels, per-case minimum margins, complete finite normalized three-label probabilities, <=1000ms/pair." }
- Tested
- 2026-11-12T22:03:07.787398+00:00
- Environment
- { "os": "macos", "osVersion": "27.2", "architecture": "aarch64", "hardware": "Apple M3 Max;Mac15,11;96GiB RAM", "cpu": "Apple M3 Max", "gpu": "Apple M3 Max", "driver": "OS-managed Metal/MPS macOS27.2 build26B118", "runtime": "App-owned CPython3.13.18;Torch2.10.0;Transformers5.22.1", "timingScope": "Fresh helper/model process per cold repetition; runtime/tokenizer/model startup in load. OS files/driver cache may be warm. Five warm pairs follow six fixtures. Snapshot loads, not Internet transfer.", "concurrentUploadActivity": "none; uploads excluded" }
- Runtime
- transformers 5.22.1
- Artifact revision
- db0dfbf65c68dcd4747f384257a44f18776be709
- Fixture
{ "file": "model-samples/native/ettinx-nli/quality-gates.json", "url": "https://assets.localai.tools/samples/v1/files/b98612c89959b9c069902c28bfd95bd8d943b61cefa34ea3c6da14ace9f0c18c/ettinx-nli-quality-gates.json", "size": 2262, "sha1": "9104fbed107fd4340666e85c8fcaf3376ba1a2a4", "sha256": "b98612c89959b9c069902c28bfd95bd8d943b61cefa34ea3c6da14ace9f0c18c" }- Rerun when
- artifact-revision; runtime-version; fixture-checksum; quality-gate; backend-behavior
Per-case quality & failure details
[
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9052759148180485,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 216.26566699706018,
"inferenceMs": 215.80570799415,
"pairsPerSecond": 4.6337977308139955,
"inputTokens": 14,
"placement": {
"cpu/torch.float32": 120
},
"loadMs": 1618.503875011811,
"totalMs": 1834.309583005961,
"grade": "good",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 1,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.9997192201844882,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 11.050541012082249,
"inferenceMs": 10.772582987556234,
"pairsPerSecond": 92.82824752012893,
"inputTokens": 26,
"placement": {
"cpu/torch.float32": 120
},
"loadMs": 1618.503875011811,
"totalMs": 1629.2764579993673,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 1,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9867861345410347,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 9.75666698650457,
"inferenceMs": 9.566708002239466,
"pairsPerSecond": 104.52916507600216,
"inputTokens": 18,
"placement": {
"cpu/torch.float32": 120
},
"loadMs": 1618.503875011811,
"totalMs": 1628.0705830140505,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 1,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9980845952522941,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 8.473416994092986,
"inferenceMs": 8.296917018014938,
"pairsPerSecond": 120.5266965824437,
"inputTokens": 16,
"placement": {
"cpu/torch.float32": 120
},
"loadMs": 1618.503875011811,
"totalMs": 1626.800792029826,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 1,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.9949608948081732,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 9.125957993092015,
"inferenceMs": 8.972791983978823,
"pairsPerSecond": 111.44803109060464,
"inputTokens": 14,
"placement": {
"cpu/torch.float32": 120
},
"loadMs": 1618.503875011811,
"totalMs": 1627.4766669957899,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 1,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9816788006573915,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 9.125749988015741,
"inferenceMs": 8.963917003711686,
"pairsPerSecond": 111.55837337471223,
"inputTokens": 15,
"placement": {
"cpu/torch.float32": 120
},
"loadMs": 1618.503875011811,
"totalMs": 1627.4677920155227,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 1,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9052759148180485,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 226.32870799861848,
"inferenceMs": 225.72666601627134,
"pairsPerSecond": 4.430136756318882,
"inputTokens": 14,
"placement": {
"cpu/torch.float32": 120
},
"loadMs": 1707.8512080188375,
"totalMs": 1933.5778740351088,
"grade": "good",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 1,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.9997192201844882,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 11.902750004082918,
"inferenceMs": 11.625749990344048,
"pairsPerSecond": 86.01595603127248,
"inputTokens": 26,
"placement": {
"cpu/torch.float32": 120
},
"loadMs": 1707.8512080188375,
"totalMs": 1719.4769580091815,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 1,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9867861345410347,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 9.559083002386615,
"inferenceMs": 9.345124999526888,
"pairsPerSecond": 107.0076644293818,
"inputTokens": 18,
"placement": {
"cpu/torch.float32": 120
},
"loadMs": 1707.8512080188375,
"totalMs": 1717.1963330183644,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 1,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9980845952522941,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 9.142291994066909,
"inferenceMs": 8.97495899698697,
"pairsPerSecond": 111.42112184977282,
"inputTokens": 16,
"placement": {
"cpu/torch.float32": 120
},
"loadMs": 1707.8512080188375,
"totalMs": 1716.8261670158245,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 1,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.9949608948081732,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 9.384417004184797,
"inferenceMs": 9.215833008056507,
"pairsPerSecond": 108.50891060263324,
"inputTokens": 14,
"placement": {
"cpu/torch.float32": 120
},
"loadMs": 1707.8512080188375,
"totalMs": 1717.067041026894,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 1,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9816788006573915,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 9.681583003839478,
"inferenceMs": 9.492166980635375,
"pairsPerSecond": 105.35002197496775,
"inputTokens": 15,
"placement": {
"cpu/torch.float32": 120
},
"loadMs": 1707.8512080188375,
"totalMs": 1717.3433749994729,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 1,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9052759148180485,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 217.0542500098236,
"inferenceMs": 216.5741250209976,
"pairsPerSecond": 4.617356759045184,
"inputTokens": 14,
"placement": {
"cpu/torch.float32": 120
},
"loadMs": 1685.442875023,
"totalMs": 1902.0170000439975,
"grade": "good",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 1,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.9997192201844882,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 11.32845800020732,
"inferenceMs": 11.07570799649693,
"pairsPerSecond": 90.28768186343338,
"inputTokens": 26,
"placement": {
"cpu/torch.float32": 120
},
"loadMs": 1685.442875023,
"totalMs": 1696.5185830194969,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 1,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9867861345410347,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 9.7996249969583,
"inferenceMs": 9.581041987985373,
"pairsPerSecond": 104.37278129602188,
"inputTokens": 18,
"placement": {
"cpu/torch.float32": 120
},
"loadMs": 1685.442875023,
"totalMs": 1695.0239170109853,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 1,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9980845952522941,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 8.537499990779907,
"inferenceMs": 8.368624985450879,
"pairsPerSecond": 119.49394335850057,
"inputTokens": 16,
"placement": {
"cpu/torch.float32": 120
},
"loadMs": 1685.442875023,
"totalMs": 1693.8115000084508,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 1,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.9949608948081732,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 9.302124992245808,
"inferenceMs": 9.153999999398366,
"pairsPerSecond": 109.24186148849941,
"inputTokens": 14,
"placement": {
"cpu/torch.float32": 120
},
"loadMs": 1685.442875023,
"totalMs": 1694.5968750223983,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 1,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9816788006573915,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 8.963000000221655,
"inferenceMs": 8.80262500140816,
"pairsPerSecond": 113.6024765158154,
"inputTokens": 15,
"placement": {
"cpu/torch.float32": 120
},
"loadMs": 1685.442875023,
"totalMs": 1694.245500024408,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 1,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9052759148180485,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 218.09249999932945,
"inferenceMs": 217.5912079983391,
"pairsPerSecond": 4.595773924871235,
"inputTokens": 14,
"placement": {
"cpu/torch.float32": 120
},
"loadMs": 1622.0134579925798,
"totalMs": 1839.6046659909189,
"grade": "good",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.9997192201844882,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 11.42470899503678,
"inferenceMs": 11.174124985700473,
"pairsPerSecond": 89.4924659675545,
"inputTokens": 26,
"placement": {
"cpu/torch.float32": 120
},
"loadMs": 1622.0134579925798,
"totalMs": 1633.1875829782803,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9867861345410347,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 9.775666985660791,
"inferenceMs": 9.569167013978586,
"pairsPerSecond": 104.50230396639598,
"inputTokens": 18,
"placement": {
"cpu/torch.float32": 120
},
"loadMs": 1622.0134579925798,
"totalMs": 1631.5826250065584,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9980845952522941,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 9.796792000997812,
"inferenceMs": 9.620416996767744,
"pairsPerSecond": 103.94559823508469,
"inputTokens": 16,
"placement": {
"cpu/torch.float32": 120
},
"loadMs": 1622.0134579925798,
"totalMs": 1631.6338749893475,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.9949608948081732,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 8.981082995887846,
"inferenceMs": 8.81854200270027,
"pairsPerSecond": 113.3974300619985,
"inputTokens": 14,
"placement": {
"cpu/torch.float32": 120
},
"loadMs": 1622.0134579925798,
"totalMs": 1630.83199999528,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9816788006573915,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 9.612291003577411,
"inferenceMs": 9.453374979784712,
"pairsPerSecond": 105.7823266440208,
"inputTokens": 15,
"placement": {
"cpu/torch.float32": 120
},
"loadMs": 1622.0134579925798,
"totalMs": 1631.4668329723645,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9052759148180485,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 223.87854199041612,
"inferenceMs": 223.3853750221897,
"pairsPerSecond": 4.47656879910185,
"inputTokens": 14,
"placement": {
"cpu/torch.float32": 120
},
"loadMs": 1664.0752090024762,
"totalMs": 1887.4605840246659,
"grade": "good",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.9997192201844882,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 11.762124981032684,
"inferenceMs": 11.46958299796097,
"pairsPerSecond": 87.18712791718558,
"inputTokens": 26,
"placement": {
"cpu/torch.float32": 120
},
"loadMs": 1664.0752090024762,
"totalMs": 1675.5447920004372,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9867861345410347,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 9.99645801493898,
"inferenceMs": 9.775207989150658,
"pairsPerSecond": 102.29961358468111,
"inputTokens": 18,
"placement": {
"cpu/torch.float32": 120
},
"loadMs": 1664.0752090024762,
"totalMs": 1673.8504169916268,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9980845952522941,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 9.828500013099983,
"inferenceMs": 9.645832993555814,
"pairsPerSecond": 103.67170991536757,
"inputTokens": 16,
"placement": {
"cpu/torch.float32": 120
},
"loadMs": 1664.0752090024762,
"totalMs": 1673.721041996032,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.9949608948081732,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 9.30150001659058,
"inferenceMs": 9.132917009992525,
"pairsPerSecond": 109.49404214511947,
"inputTokens": 14,
"placement": {
"cpu/torch.float32": 120
},
"loadMs": 1664.0752090024762,
"totalMs": 1673.2081260124687,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9816788006573915,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 9.846875007497147,
"inferenceMs": 9.67812500311993,
"pairsPerSecond": 103.3257991271688,
"inputTokens": 15,
"placement": {
"cpu/torch.float32": 120
},
"loadMs": 1664.0752090024762,
"totalMs": 1673.7533340055961,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9052759148180485,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 221.1161660088692,
"inferenceMs": 220.66424999502487,
"pairsPerSecond": 4.531771684912921,
"inputTokens": 14,
"placement": {
"cpu/torch.float32": 120
},
"loadMs": 1662.9761250223964,
"totalMs": 1883.6403750174213,
"grade": "good",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.9997192201844882,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 11.51258300524205,
"inferenceMs": 11.225582973565906,
"pairsPerSecond": 89.08223317709273,
"inputTokens": 26,
"placement": {
"cpu/torch.float32": 120
},
"loadMs": 1662.9761250223964,
"totalMs": 1674.2017079959624,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9867861345410347,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 9.875458985334262,
"inferenceMs": 9.65029199142009,
"pairsPerSecond": 103.62380753754218,
"inputTokens": 18,
"placement": {
"cpu/torch.float32": 120
},
"loadMs": 1662.9761250223964,
"totalMs": 1672.6264170138165,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9980845952522941,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 9.589207998942584,
"inferenceMs": 9.405166987562552,
"pairsPerSecond": 106.32453430358078,
"inputTokens": 16,
"placement": {
"cpu/torch.float32": 120
},
"loadMs": 1662.9761250223964,
"totalMs": 1672.381292009959,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.9949608948081732,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 9.238375001586974,
"inferenceMs": 9.055417001945898,
"pairsPerSecond": 110.43113749318361,
"inputTokens": 14,
"placement": {
"cpu/torch.float32": 120
},
"loadMs": 1662.9761250223964,
"totalMs": 1672.0315420243423,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9816788006573915,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 9.717584005557,
"inferenceMs": 9.559458994772285,
"pairsPerSecond": 104.60843030414829,
"inputTokens": 15,
"placement": {
"cpu/torch.float32": 120
},
"loadMs": 1662.9761250223964,
"totalMs": 1672.5355840171687,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "cpu",
"device": "cpu",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "PyTorch CPU",
"acceleration": "cpu",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9052758924663067,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 196.74625000334345,
"inferenceMs": 195.81829197704792,
"pairsPerSecond": 5.10677521442793,
"inputTokens": 14,
"placement": {
"mps:0/torch.float32": 120
},
"driverAllocatedMemoryBytes": 1118601216,
"loadMs": 1932.2858749947045,
"totalMs": 2128.1041669717524,
"grade": "good",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.9997192201844882,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 31.34529199451208,
"inferenceMs": 30.58983400114812,
"pairsPerSecond": 32.69059910434517,
"inputTokens": 26,
"placement": {
"mps:0/torch.float32": 120
},
"driverAllocatedMemoryBytes": 1118699520,
"loadMs": 1932.2858749947045,
"totalMs": 1962.8757089958526,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9867861312814057,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 31.456790980882943,
"inferenceMs": 30.69375001359731,
"pairsPerSecond": 32.57992260825089,
"inputTokens": 18,
"placement": {
"mps:0/torch.float32": 120
},
"driverAllocatedMemoryBytes": 1118781440,
"loadMs": 1932.2858749947045,
"totalMs": 1962.9796250083018,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9980845952522941,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 33.592040999792516,
"inferenceMs": 32.87445902242325,
"pairsPerSecond": 30.41875150912484,
"inputTokens": 16,
"placement": {
"mps:0/torch.float32": 120
},
"driverAllocatedMemoryBytes": 1118863360,
"loadMs": 1932.2858749947045,
"totalMs": 1965.1603340171278,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.9949608956230804,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 11.940833006519824,
"inferenceMs": 11.138917005155236,
"pairsPerSecond": 89.77533449052426,
"inputTokens": 14,
"placement": {
"mps:0/torch.float32": 120
},
"driverAllocatedMemoryBytes": 1118863360,
"loadMs": 1932.2858749947045,
"totalMs": 1943.4247919998597,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9816788006573915,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 181.9457079982385,
"inferenceMs": 180.74508299469016,
"pairsPerSecond": 5.532653964530684,
"inputTokens": 15,
"placement": {
"mps:0/torch.float32": 120
},
"driverAllocatedMemoryBytes": 1118928896,
"loadMs": 1932.2858749947045,
"totalMs": 2113.0309579893947,
"grade": "good",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9052758924663067,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 80.29054102371447,
"inferenceMs": 78.7571250111796,
"pairsPerSecond": 12.697263896543324,
"inputTokens": 14,
"placement": {
"mps:0/torch.float32": 120
},
"driverAllocatedMemoryBytes": 1118601216,
"loadMs": 1841.5201659954619,
"totalMs": 1920.2772910066415,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.9997192201844882,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 25.851458980469033,
"inferenceMs": 25.164708000374958,
"pairsPerSecond": 39.738192073800334,
"inputTokens": 26,
"placement": {
"mps:0/torch.float32": 120
},
"driverAllocatedMemoryBytes": 1118699520,
"loadMs": 1841.5201659954619,
"totalMs": 1866.6848739958368,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9867861312814057,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 24.797292018774897,
"inferenceMs": 24.249957990832627,
"pairsPerSecond": 41.23718483875464,
"inputTokens": 18,
"placement": {
"mps:0/torch.float32": 120
},
"driverAllocatedMemoryBytes": 1118781440,
"loadMs": 1841.5201659954619,
"totalMs": 1865.7701239862945,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9980845952522941,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 26.20116699836217,
"inferenceMs": 25.588916993001476,
"pairsPerSecond": 39.079418651187865,
"inputTokens": 16,
"placement": {
"mps:0/torch.float32": 120
},
"driverAllocatedMemoryBytes": 1118863360,
"loadMs": 1841.5201659954619,
"totalMs": 1867.1090829884633,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.9949608956230804,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 12.279999995371327,
"inferenceMs": 11.710041988408193,
"pairsPerSecond": 85.39679029246037,
"inputTokens": 14,
"placement": {
"mps:0/torch.float32": 120
},
"driverAllocatedMemoryBytes": 1118863360,
"loadMs": 1841.5201659954619,
"totalMs": 1853.23020798387,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9816788006573915,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 27.458124997792765,
"inferenceMs": 26.85716698761098,
"pairsPerSecond": 37.23400909936974,
"inputTokens": 15,
"placement": {
"mps:0/torch.float32": 120
},
"driverAllocatedMemoryBytes": 1118928896,
"loadMs": 1841.5201659954619,
"totalMs": 1868.3773329830728,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9052758924663067,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 87.4681249842979,
"inferenceMs": 85.83658401039429,
"pairsPerSecond": 11.650044226817158,
"inputTokens": 14,
"placement": {
"mps:0/torch.float32": 120
},
"driverAllocatedMemoryBytes": 1118601216,
"loadMs": 1848.542375024408,
"totalMs": 1934.3789590348024,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.9997192201844882,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 29.201375000411645,
"inferenceMs": 24.94608299457468,
"pairsPerSecond": 40.086453661582134,
"inputTokens": 26,
"placement": {
"mps:0/torch.float32": 120
},
"driverAllocatedMemoryBytes": 1118699520,
"loadMs": 1848.542375024408,
"totalMs": 1873.4884580189828,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9867861312814057,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 24.36587499687448,
"inferenceMs": 23.7409999826923,
"pairsPerSecond": 42.12122491592694,
"inputTokens": 18,
"placement": {
"mps:0/torch.float32": 120
},
"driverAllocatedMemoryBytes": 1118781440,
"loadMs": 1848.542375024408,
"totalMs": 1872.2833750071004,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9980845952522941,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 27.868042001500726,
"inferenceMs": 27.29774999897927,
"pairsPerSecond": 36.63305583930516,
"inputTokens": 16,
"placement": {
"mps:0/torch.float32": 120
},
"driverAllocatedMemoryBytes": 1118863360,
"loadMs": 1848.542375024408,
"totalMs": 1875.8401250233874,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.9949608956230804,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 12.351915996987373,
"inferenceMs": 10.702416999265552,
"pairsPerSecond": 93.43683768522797,
"inputTokens": 14,
"placement": {
"mps:0/torch.float32": 120
},
"driverAllocatedMemoryBytes": 1118863360,
"loadMs": 1848.542375024408,
"totalMs": 1859.2447920236737,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9816788006573915,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 26.917624985799193,
"inferenceMs": 26.296375028323382,
"pairsPerSecond": 38.02805515676274,
"inputTokens": 15,
"placement": {
"mps:0/torch.float32": 120
},
"driverAllocatedMemoryBytes": 1118928896,
"loadMs": 1848.542375024408,
"totalMs": 1874.8387500527315,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp32",
"precision": "fp32",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9052134975790977,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 259.67624998884276,
"inferenceMs": 258.0718329991214,
"pairsPerSecond": 3.874890135737535,
"inputTokens": 14,
"placement": {
"mps:0/torch.float16": 120
},
"driverAllocatedMemoryBytes": 1126989824,
"loadMs": 1858.6206249892712,
"totalMs": 2116.6924579883926,
"grade": "good",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp16",
"precision": "fp16",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.9997197498159949,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 29.855917004169896,
"inferenceMs": 28.8093329872936,
"pairsPerSecond": 34.710973712617765,
"inputTokens": 26,
"placement": {
"mps:0/torch.float16": 120
},
"driverAllocatedMemoryBytes": 1127088128,
"loadMs": 1858.6206249892712,
"totalMs": 1887.4299579765648,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp16",
"precision": "fp16",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9867661232128739,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 31.502500001806766,
"inferenceMs": 30.892667011357844,
"pairsPerSecond": 32.370141420044604,
"inputTokens": 18,
"placement": {
"mps:0/torch.float16": 120
},
"driverAllocatedMemoryBytes": 1127170048,
"loadMs": 1858.6206249892712,
"totalMs": 1889.513292000629,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp16",
"precision": "fp16",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9980876952176914,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 30.657541996333748,
"inferenceMs": 29.975084005855024,
"pairsPerSecond": 33.36104078322751,
"inputTokens": 16,
"placement": {
"mps:0/torch.float16": 120
},
"driverAllocatedMemoryBytes": 1127251968,
"loadMs": 1858.6206249892712,
"totalMs": 1888.5957089951262,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp16",
"precision": "fp16",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.9949509820435196,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 13.746499986154959,
"inferenceMs": 13.158666988601908,
"pairsPerSecond": 75.99553973561335,
"inputTokens": 14,
"placement": {
"mps:0/torch.float16": 120
},
"driverAllocatedMemoryBytes": 1127251968,
"loadMs": 1858.6206249892712,
"totalMs": 1871.779291977873,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp16",
"precision": "fp16",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9817087044939399,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 186.7764589842409,
"inferenceMs": 186.18420799612068,
"pairsPerSecond": 5.371024807973166,
"inputTokens": 15,
"placement": {
"mps:0/torch.float16": 120
},
"driverAllocatedMemoryBytes": 1127317504,
"loadMs": 1858.6206249892712,
"totalMs": 2044.8048329853918,
"grade": "good",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp16",
"precision": "fp16",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9052134975790977,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 86.25554200261831,
"inferenceMs": 85.31774999573827,
"pairsPerSecond": 11.7208904366319,
"inputTokens": 14,
"placement": {
"mps:0/torch.float16": 120
},
"driverAllocatedMemoryBytes": 1126989824,
"loadMs": 1798.9354169985745,
"totalMs": 1884.2531669943128,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp16",
"precision": "fp16",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.9997197498159949,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 30.238250008551404,
"inferenceMs": 29.32112498092465,
"pairsPerSecond": 34.10510342459803,
"inputTokens": 26,
"placement": {
"mps:0/torch.float16": 120
},
"driverAllocatedMemoryBytes": 1127088128,
"loadMs": 1798.9354169985745,
"totalMs": 1828.2565419794992,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp16",
"precision": "fp16",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9867661232128739,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 30.04337500897236,
"inferenceMs": 29.15558297536336,
"pairsPerSecond": 34.298748230999394,
"inputTokens": 18,
"placement": {
"mps:0/torch.float16": 120
},
"driverAllocatedMemoryBytes": 1127170048,
"loadMs": 1798.9354169985745,
"totalMs": 1828.0909999739379,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp16",
"precision": "fp16",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9980876952176914,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 31.11800001352094,
"inferenceMs": 29.847625002730638,
"pairsPerSecond": 33.5035032069893,
"inputTokens": 16,
"placement": {
"mps:0/torch.float16": 120
},
"driverAllocatedMemoryBytes": 1127251968,
"loadMs": 1798.9354169985745,
"totalMs": 1828.7830420013051,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp16",
"precision": "fp16",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.9949509820435196,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 11.797791987191886,
"inferenceMs": 11.278333986410871,
"pairsPerSecond": 88.66557784198339,
"inputTokens": 14,
"placement": {
"mps:0/torch.float16": 120
},
"driverAllocatedMemoryBytes": 1127251968,
"loadMs": 1798.9354169985745,
"totalMs": 1810.2137509849854,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp16",
"precision": "fp16",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9817087044939399,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 27.16250001685694,
"inferenceMs": 26.553292002063245,
"pairsPerSecond": 37.66011385414276,
"inputTokens": 15,
"placement": {
"mps:0/torch.float16": 120
},
"driverAllocatedMemoryBytes": 1127317504,
"loadMs": 1798.9354169985745,
"totalMs": 1825.4887090006378,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp16",
"precision": "fp16",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9052134975790977,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 89.42645898787305,
"inferenceMs": 88.48787500755861,
"pairsPerSecond": 11.300983325846397,
"inputTokens": 14,
"placement": {
"mps:0/torch.float16": 120
},
"driverAllocatedMemoryBytes": 1126989824,
"loadMs": 1899.735042010434,
"totalMs": 1988.2229170179926,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp16",
"precision": "fp16",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.9997197498159949,
"minimumMargin": 0.3,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 28.764208022039384,
"inferenceMs": 27.856375003466383,
"pairsPerSecond": 35.898425400848545,
"inputTokens": 26,
"placement": {
"mps:0/torch.float16": 120
},
"driverAllocatedMemoryBytes": 1127088128,
"loadMs": 1899.735042010434,
"totalMs": 1927.5914170139004,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp16",
"precision": "fp16",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "entailment",
"actualLabel": "entailment",
"finite": true,
"allLabels": true,
"margin": 0.9867661232128739,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 26.816708006663248,
"inferenceMs": 26.164834009250626,
"pairsPerSecond": 38.21923730326163,
"inputTokens": 18,
"placement": {
"mps:0/torch.float16": 120
},
"driverAllocatedMemoryBytes": 1127170048,
"loadMs": 1899.735042010434,
"totalMs": 1925.8998760196846,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp16",
"precision": "fp16",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9980876952176914,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 29.8120410006959,
"inferenceMs": 28.873917006421834,
"pairsPerSecond": 34.633333599233886,
"inputTokens": 16,
"placement": {
"mps:0/torch.float16": 120
},
"driverAllocatedMemoryBytes": 1127251968,
"loadMs": 1899.735042010434,
"totalMs": 1928.6089590168558,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp16",
"precision": "fp16",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "contradiction",
"actualLabel": "contradiction",
"finite": true,
"allLabels": true,
"margin": 0.9949509820435196,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 10.069416981423274,
"inferenceMs": 9.434875013539568,
"pairsPerSecond": 105.9897453400225,
"inputTokens": 14,
"placement": {
"mps:0/torch.float16": 120
},
"driverAllocatedMemoryBytes": 1127251968,
"loadMs": 1899.735042010434,
"totalMs": 1909.1699170239735,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp16",
"precision": "fp16",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"expectedLabel": "neutral",
"actualLabel": "neutral",
"finite": true,
"allLabels": true,
"margin": 0.9817087044939399,
"minimumMargin": 0.15,
"passed": true,
"speedPassed": true
},
"runtime": {
"durationMs": 26.70245899935253,
"inferenceMs": 26.100458024302498,
"pairsPerSecond": 38.3135038882799,
"inputTokens": 15,
"placement": {
"mps:0/torch.float16": 120
},
"driverAllocatedMemoryBytes": 1127317504,
"loadMs": 1899.735042010434,
"totalMs": 1925.8355000347365,
"grade": "excellent",
"engine": "transformers",
"version": "5.22.1",
"artifactRevision": "db0dfbf65c68dcd4747f384257a44f18776be709",
"requestedDevice": "mps",
"device": "mps",
"requestedPrecision": "fp16",
"precision": "fp16",
"cpuThreads": 4,
"provider": "Metal/MPS",
"acceleration": "partial",
"fallback": false
}
}
]Release validation still required
- Sample publication, complete dependency backup and forced model/runtime-source recovery pending under user upload exclusion.