OpenAI Privacy Filter — PII Span Detection
10 configurations testedQuality criteria
Finite spans must detect the fictional name Harry Potter as private_person and harry.potter@hogwarts.edu as private_email at score >= 0.9, with no more than one unexpected span and valid offsets when present.
| Precision | Device | Result | |
|---|---|---|---|
Q4 | GPU · WebGPUAccelerated | Passed | Fast |
Q4F16 | GPU · WebGPUAccelerated | Passed | Moderate |
Q4 | CPU · WASM | Failed | — |
Q4F16 | CPU · WASM | Failed | — |
Q8 | CPU · WASM | Failed | — |
Q8 | GPU · WebGPU | Failed | — |
FP16 | CPU · WASM | Failed | — |
FP16 | GPU · WebGPU | Failed | — |
FP32 | CPU · WASM | Failed | — |
FP32 | GPU · WebGPU | Failed | — |
Test setup & sample details
- Quality gate
- Finite spans must detect the fictional name Harry Potter as private_person and harry.potter@hogwarts.edu as private_email at score >= 0.9, with no more than one unexpected span and valid offsets when present.
- Tested
- 2026-09-15T12:41:45.169Z
- Environment
- Isolated headless Chrome on macOS; Transformers.js 4.2.0
- Runtime
- Transformers.js 4.2.0
- Artifact revision
- 7ffa9a043d54d1be65afb281eddf0ffbe629385b
- Fixture
{ "file": "privacy-filter.json", "url": "https://f005.backblazeb2.com/file/hubters-webai-public/assets/benchmarks/privacy-filter.json", "size": 899, "sha1": "39b9188d3224c759479738aa156538e01a283414", "source": "Official OpenAI Privacy Filter model-card Transformers.js example.", "sourceUrl": "https://huggingface.co/openai/privacy-filter/blob/7ffa9a043d54d1be65afb281eddf0ffbe629385b/README.md#transformersjs", "license": "Apache-2.0", "licenseUrl": "https://github.com/openai/privacy-filter/blob/f7f00ca7fb869683eb732c010299d901457f19c3/LICENSE", "preprocessing": "Pinned tokenizer followed by constrained model-specific sequence decoding and simple span aggregation." }- Rerun when
- artifact revision change; runtime version change; fixture or quality-gate change; browser/backend behavior change
Per-case quality & failure details
[
{
"precision": "q4",
"device": "wasm",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": null,
"inferenceMs": null,
"totalMs": 31244.40000000596,
"grade": "failed"
},
"failure": "Error: Can't create a session. ERROR_CODE: 9, ERROR_MESSAGE: Could not find an implementation for GatherBlockQuantized(1) node with name '/model/embed_tokens/Gather_Quant'"
},
{
"precision": "q4",
"device": "webgpu",
"status": "pass",
"quality": {
"finite": true,
"validOffsets": true,
"expectedMatches": [
true,
true
],
"unexpectedEntities": 0,
"grade": "privacy-span-pass"
},
"runtime": {
"loadMs": 31144.5,
"inferenceMs": 166.2000000178814,
"totalMs": 31406.5,
"grade": "good",
"charactersPerSecond": 397.1119133146756
}
},
{
"precision": "q4f16",
"device": "wasm",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": null,
"inferenceMs": null,
"totalMs": 29940.5,
"grade": "failed"
},
"failure": "Error: Can't create a session. ERROR_CODE: 9, ERROR_MESSAGE: Failed to find kernel for com.microsoft.GatherBlockQuantized(1) (node:'/model/embed_tokens/Gather_Quant' ep:'CPUExecutionProvider'). Kernel not found"
},
{
"precision": "q4f16",
"device": "webgpu",
"status": "pass",
"quality": {
"finite": true,
"validOffsets": true,
"expectedMatches": [
true,
true
],
"unexpectedEntities": 0,
"grade": "privacy-span-pass"
},
"runtime": {
"loadMs": 30789.5,
"inferenceMs": 638.5,
"totalMs": 31589.899999976158,
"grade": "slow",
"charactersPerSecond": 103.3672670321065
}
},
{
"precision": "q8",
"device": "wasm",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": null,
"inferenceMs": null,
"totalMs": 52944.90000000596,
"grade": "failed"
},
"failure": "Error: Can't create a session. ERROR_CODE: 9, ERROR_MESSAGE: Could not find an implementation for GatherBlockQuantized(1) node with name '/model/embed_tokens/Gather_Quant'"
},
{
"precision": "q8",
"device": "webgpu",
"status": "fail",
"quality": {
"finite": true,
"validOffsets": true,
"expectedMatches": [
false,
false
],
"unexpectedEntities": 9,
"grade": "failed"
},
"runtime": {
"loadMs": 54158.70000001788,
"inferenceMs": 635.5,
"totalMs": 54955.90000000596,
"grade": "slow",
"charactersPerSecond": 103.8552321007081
}
},
{
"precision": "fp16",
"device": "wasm",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": null,
"inferenceMs": null,
"totalMs": 88576.5,
"grade": "failed"
},
"failure": "Error: Can't create a session. ERROR_CODE: 6, ERROR_MESSAGE: std::bad_alloc"
},
{
"precision": "fp16",
"device": "webgpu",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": null,
"inferenceMs": null,
"totalMs": 85955.40000000596,
"grade": "failed"
},
"failure": "Error: Can't create a session. ERROR_CODE: 6, ERROR_MESSAGE: std::bad_alloc"
},
{
"precision": "fp32",
"device": "wasm",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": null,
"inferenceMs": null,
"totalMs": 144554.5,
"grade": "failed"
},
"failure": "Error: Can't create a session. ERROR_CODE: 6, ERROR_MESSAGE: std::bad_alloc"
},
{
"precision": "fp32",
"device": "webgpu",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": null,
"inferenceMs": null,
"totalMs": 143015,
"grade": "failed"
},
"failure": "Error: Can't create a session. ERROR_CODE: 6, ERROR_MESSAGE: std::bad_alloc"
}
]