BEN2 Base
7 configurations testedQuality criteria
{ "version": 1, "scope": "Compatibility and portrait matte fidelity on the existing annotated portrait and its mirrored form; not broad segmentation certification", "fixture": "model-samples/modnet-portrait-pexels-5965592.jpg", "reference": "test-app/public/test-images/modnet-portrait-pexels-5965592-mask.png", "minimumIoU": 0.7, "maximumAlphaMAE": 0.16, "minimumBoundaryF1": 0.6, "boundaryTolerancePixels": 12, "minimumForegroundRecall": 0.85, "maximumWarmInferenceMs": 10000, "initializationTimeoutMs": 180000, "preprocessing": "Check original BEN2.py ImageNet-normalized Lanczos1024 first; retain published onnx_run.py unnormalized bilinear alternative as a separately identified diagnostic because the two official sources conflict.", "postprocessing": "Output is already sigmoid probability. Finite min-max normalize with constant-mask guard; resize to original width,height. Do not repeat the official onnx_run.py width/height interpolation inversion.", "nativePlan": "ONNX runtime1.23.2 CPU compatibility first; original reviewed Torch2.10 CPU/MPS if needed, exact retained app-managed runtime. No Transformers5.22.1 built-in BEN2 architecture. GPU claims require measured placement.", "repetitions": "Compatibility one cold process then recommend only after repeated timing and suite; failures terminate affected case." }
| Precision | Device | Result | |
|---|---|---|---|
FP16 | CPU · WASM | Passed | Very slow |
FP16 | CPU · WASM | Passed | Very slow |
FP16 | CPU · WASM | Passed | Very slow |
FP16 | CPU · WASM | Passed | Very slow |
FP16 | CPU · WASM | Passed | Very slow |
FP16 | CPU · WASM | Passed | Very slow |
FP16 | CPU · WASM | Passed | Very slow |
Test setup & sample details
- Quality gate
- { "version": 1, "scope": "Compatibility and portrait matte fidelity on the existing annotated portrait and its mirrored form; not broad segmentation certification", "fixture": "model-samples/modnet-portrait-pexels-5965592.jpg", "reference": "test-app/public/test-images/modnet-portrait-pexels-5965592-mask.png", "minimumIoU": 0.7, "maximumAlphaMAE": 0.16, "minimumBoundaryF1": 0.6, "boundaryTolerancePixels": 12, "minimumForegroundRecall": 0.85, "maximumWarmInferenceMs": 10000, "initializationTimeoutMs": 180000, "preprocessing": "Check original BEN2.py ImageNet-normalized Lanczos1024 first; retain published onnx_run.py unnormalized bilinear alternative as a separately identified diagnostic because the two official sources conflict.", "postprocessing": "Output is already sigmoid probability. Finite min-max normalize with constant-mask guard; resize to original width,height. Do not repeat the official onnx_run.py width/height interpolation inversion.", "nativePlan": "ONNX runtime1.23.2 CPU compatibility first; original reviewed Torch2.10 CPU/MPS if needed, exact retained app-managed runtime. No Transformers5.22.1 built-in BEN2 architecture. GPU claims require measured placement.", "repetitions": "Compatibility one cold process then recommend only after repeated timing and suite; failures terminate affected case." }
- Tested
- 2026-11-16T11:44:41.571Z
- Environment
- Apple M3 Max /macOS27.2 /Chrome157; eight-thread cross-origin isolated WASM
- Runtime
- onnxruntime-web 1.31.0-dev.20260914-8d85527a0
- Artifact revision
- e48a20765fb421d19dcdb0bf3cc61e802ca5ec8f
- Fixture
{ "file": "model-samples/modnet-portrait-pexels-5965592.jpg", "size": 78124, "sha1": "38212739a4a4f0be25d84863cf991d7fb6c6d818", "sha256": "2b5d6423b8948eb53319661633039912afa6455a9055f0c984ff88eeae1b5bfe", "url": "https://assets.localai.tools/samples/v1/files/2b5d6423b8948eb53319661633039912afa6455a9055f0c984ff88eeae1b5bfe/modnet-portrait-pexels-5965592.jpg", "width": 1024, "height": 683 }- Rerun when
- model artifact revision change; Transformers.js or ONNX Runtime version change; benchmark fixture, reference matte, or quality-gate change; target browser/backend behavior change
Per-case quality & failure details
[
{
"precision": "fp16",
"device": "wasm",
"status": "pass",
"quality": {
"id": "portrait",
"iou": 0.9889790244147549,
"boundaryF1": 0.9735125582198727,
"foregroundRecall": 0.9920196680181895,
"alphaMAE": 0.0045040353256968965,
"passed": true,
"alphaAndDimensionsValid": true,
"inferenceMs": 7050.370000001043,
"speedPassed": true
},
"runtime": {
"durationMs": 7050.255000002682,
"inferenceMs": 7050.370000001043,
"loadMs": 5093.434999998659,
"totalMs": 12143.804999999702,
"imagesPerSecond": 0.141836527728311,
"grade": "acceptable",
"engine": "onnxruntime-web",
"version": "1.31.0-dev.20260914-8d85527a0",
"actualDevice": "wasm",
"actualPrecision": "fp16"
}
},
{
"precision": "fp16",
"device": "wasm",
"status": "pass",
"quality": {
"id": "portrait-repeat-1",
"iou": 0.9889790244147549,
"boundaryF1": 0.9735125582198727,
"foregroundRecall": 0.9920196680181895,
"alphaMAE": 0.0045040353256968965,
"passed": true,
"alphaAndDimensionsValid": true,
"inferenceMs": 6923.02499999851,
"speedPassed": true
},
"runtime": {
"durationMs": 6923,
"inferenceMs": 6923.02499999851,
"loadMs": 5093.434999998659,
"totalMs": 12016.459999997169,
"imagesPerSecond": 0.14444552778593392,
"grade": "acceptable",
"engine": "onnxruntime-web",
"version": "1.31.0-dev.20260914-8d85527a0",
"actualDevice": "wasm",
"actualPrecision": "fp16"
}
},
{
"precision": "fp16",
"device": "wasm",
"status": "pass",
"quality": {
"id": "portrait-repeat-2",
"iou": 0.9889790244147549,
"boundaryF1": 0.9735125582198727,
"foregroundRecall": 0.9920196680181895,
"alphaMAE": 0.0045040353256968965,
"passed": true,
"alphaAndDimensionsValid": true,
"inferenceMs": 7187.72500000149,
"speedPassed": true
},
"runtime": {
"durationMs": 7187.704999998212,
"inferenceMs": 7187.72500000149,
"loadMs": 5093.434999998659,
"totalMs": 12281.160000000149,
"imagesPerSecond": 0.13912607953139453,
"grade": "acceptable",
"engine": "onnxruntime-web",
"version": "1.31.0-dev.20260914-8d85527a0",
"actualDevice": "wasm",
"actualPrecision": "fp16"
}
},
{
"precision": "fp16",
"device": "wasm",
"status": "pass",
"quality": {
"id": "portrait-repeat-3",
"iou": 0.9889790244147549,
"boundaryF1": 0.9735125582198727,
"foregroundRecall": 0.9920196680181895,
"alphaMAE": 0.0045040353256968965,
"passed": true,
"alphaAndDimensionsValid": true,
"inferenceMs": 7317.780000001192,
"speedPassed": true
},
"runtime": {
"durationMs": 7317.755000002682,
"inferenceMs": 7317.780000001192,
"loadMs": 5093.434999998659,
"totalMs": 12411.214999999851,
"imagesPerSecond": 0.13665346594183442,
"grade": "acceptable",
"engine": "onnxruntime-web",
"version": "1.31.0-dev.20260914-8d85527a0",
"actualDevice": "wasm",
"actualPrecision": "fp16"
}
},
{
"precision": "fp16",
"device": "wasm",
"status": "pass",
"quality": {
"id": "portrait-repeat-4",
"iou": 0.9889790244147549,
"boundaryF1": 0.9735125582198727,
"foregroundRecall": 0.9920196680181895,
"alphaMAE": 0.0045040353256968965,
"passed": true,
"alphaAndDimensionsValid": true,
"inferenceMs": 7416.030000001192,
"speedPassed": true
},
"runtime": {
"durationMs": 7416.004999998957,
"inferenceMs": 7416.030000001192,
"loadMs": 5093.434999998659,
"totalMs": 12509.464999999851,
"imagesPerSecond": 0.13484303596396444,
"grade": "acceptable",
"engine": "onnxruntime-web",
"version": "1.31.0-dev.20260914-8d85527a0",
"actualDevice": "wasm",
"actualPrecision": "fp16"
}
},
{
"precision": "fp16",
"device": "wasm",
"status": "pass",
"quality": {
"id": "portrait-repeat-5",
"iou": 0.9889790244147549,
"boundaryF1": 0.9735125582198727,
"foregroundRecall": 0.9920196680181895,
"alphaMAE": 0.0045040353256968965,
"passed": true,
"alphaAndDimensionsValid": true,
"inferenceMs": 7414.865000002086,
"speedPassed": true
},
"runtime": {
"durationMs": 7414.839999999851,
"inferenceMs": 7414.865000002086,
"loadMs": 5093.434999998659,
"totalMs": 12508.300000000745,
"imagesPerSecond": 0.1348642220727847,
"grade": "acceptable",
"engine": "onnxruntime-web",
"version": "1.31.0-dev.20260914-8d85527a0",
"actualDevice": "wasm",
"actualPrecision": "fp16"
}
},
{
"precision": "fp16",
"device": "wasm",
"status": "pass",
"quality": {
"platform": "browser",
"device": "wasm",
"id": "dog",
"reused": false,
"probeAccuracy": 1,
"foregroundRatio": 0.3673483535528596,
"probes": [
{
"point": [
200,
150
],
"foreground": true,
"passed": true
},
{
"point": [
350,
200
],
"foreground": true,
"passed": true
},
{
"point": [
500,
400
],
"foreground": true,
"passed": true
},
{
"point": [
0,
0
],
"foreground": false,
"passed": true
},
{
"point": [
690,
570
],
"foreground": false,
"passed": true
},
{
"point": [
680,
10
],
"foreground": false,
"passed": true
}
],
"passed": true,
"runtime": {
"durationMs": 7651.829999998212,
"inferenceMs": 7651.954999998212,
"loadMs": 4979.054999999702
}
},
"runtime": {
"durationMs": 7651.829999998212,
"inferenceMs": 7651.954999998212,
"loadMs": 4979.054999999702,
"totalMs": 12631.009999997914,
"imagesPerSecond": 0.1306855568283182,
"grade": "acceptable",
"engine": "onnxruntime-web",
"version": "1.31.0-dev.20260914-8d85527a0",
"actualDevice": "wasm",
"actualPrecision": "fp16"
}
}
]