patronus-studio/orca-sonar-document-classifier

Model Card for Orca Sonar Document Classifier

Multilingual Document Topic Classifier for Real-World AI Security & DLP

Read more

• Blog post (EN): Orca-Sonar: Our Multilingual Document Classifier for AI Security
• Blogbeitrag (DE): Orca-Sonar: unser mehrsprachiger Dokumenten-Klassifikator für AI-Security
• Blog post (EN): Our AI-Security Model Zoo Is Now Open Source
• Blogbeitrag (DE): Unser AI-Security-Model-Zoo ist jetzt Open Source
• Product page: Patronus AI models

Orca-Sonar is a Multilingual ModernBERT-based (mmBERT) classifier that assigns a document/text to one of 9 topic classes. It is part of the Patronus Protect security stack and is designed for topic-/risk-routing of incoming texts (e.g. before they reach an LLM, a DLP gate, or a storage tier).

It classifies German and English text and is robust to user-to-AI wrappers (e.g. "Summarize this contract: …"), i.e. the topic of the content determines the class, not the surface format of the request.

Intended Uses

The model maps an input text to one of:

id	label	description
0	legal	contracts, NDAs, ToS/AGB, privacy policies, statutes/judgments, compliance, legal correspondence
1	hr	CVs, job ads, employment contracts, terminations, HR policies, performance reviews, recruiting
2	finance	invoices, balance sheets, quarterly/annual reports, cash-flow, SEC filings, forecasts
3	internal_and_tech	ADRs, RFCs, postmortems, specs, READMEs, wikis, architecture & strategy memos, runbooks
4	source_code	raw program code & configs (Python/Go/Rust/JS/TS/SQL/Bash/Dockerfile/k8s/Terraform …)
5	marketing	press releases, newsletters, landing-page/sales copy, outbound pitches, case studies
6	other	conversational / non-business: smalltalk, recipes, travel, hobby, learning, creative
7	education	curricula, course materials, transcripts, grades, assignments, and academic records
8	medical	clinical notes, diagnoses, treatments, medications, and healthcare records

Disambiguation: on a tie, the more sensitive class wins: legal > hr > finance > internal_and_tech > source_code > marketing > other.

Limitations

• Highly accurate on German and English; other languages were not actively tested.
• The model can produce false positives; for high-stakes routing combine it with a confidence/abstention gate.
• Robustness against adversarial / out-of-distribution / pure-PII / pathological-length inputs is partial; pair the model with a deterministic pre-gate (length + PII) for production DLP use.

Model Variants

• orca-sonar: full model (model.safetensors, fp32).
• ONNX under onnx/ (inputs input_ids, attention_mask; output logits):

Variant	Path	Size	argmax agreement with fp32 PyTorch (test / OOD)
fp32	onnx/onnx_fp32/model.onnx	563 MB	100 % / 100 %
fp16	onnx/onnx_fp16/model_fp16.onnx	282 MB	100 % / 99.9 %
mixed (fp16 embeddings, int8 matmuls)	onnx/onnx_mixed/model_mixed.onnx	240 MB	99.5 % / 99.5 %
int8	onnx/int8/model.onnx	142 MB	99.6 % / 99.6 %
int8 + int4 embeddings	onnx/int8_int4_embeddings/model.onnx	96 MB	99.0 % / 98.9 %

Agreement was measured on the held-out test set (3,367 samples) and the OOD benchmark (845 samples), max length 256. Build details: onnx/quantization_manifest.json.

Training Data

Trained on our own in-house dataset (German + English, 9 topic classes), purpose-built for this model. The dataset will be published soon.

This version adds 1,890 hard examples for the seven original classes (270 per class). They come from an evolutionary generation loop: LLM-generated candidates are checked by two independent LLM judges, and only the examples the previous model got wrong or barely right are kept. education and medical received no new examples.

Benchmark

Both models were evaluated with the same script and max length 256. "Previous" is the model that was in this repository before this update.

Held-out test set (3,367 samples, 100 % real data):

Metric	Previous	This version
Accuracy	0.915	0.936
F1 (macro)	0.913	0.936
F1 legal	0.901	0.932
F1 hr	0.852	0.913
F1 finance	0.915	0.948
F1 internal_and_tech	0.890	0.917
F1 source_code	0.932	0.955
F1 marketing	0.942	0.930
F1 other	0.914	0.933
F1 education	0.938	0.953
F1 medical	0.937	0.942

Earlier versions of this card listed accuracy 0.942 for the previous model. On the current test split we measure 0.915 for that model, so the two numbers are not comparable.

Out-of-distribution benchmark (845 real samples from 23 public sources not used in training, judge-verified and deduplicated against the training data; seven classes, no education/medical):

Metric	Previous	This version
Accuracy	0.807	0.898
F1 (macro)	0.777	0.890

On the OOD set the largest gains are marketing (F1 0.54 → 0.84) and hr (0.68 → 0.84). Website-style promotional copy is still the weakest case: on a 35-sample diagnostic set of such pages, 31 % are classified as marketing.

Usage

from transformers import pipeline

clf = pipeline("text-classification", model="patronus-studio/orca-sonar-document-classifier")
clf("Fasse mir diesen Dienstleistungsvertrag zusammen: Laufzeit 24 Monate, Gerichtsstand München …")
# -> [{'label': 'legal', 'score': 0.99}]

ONNX

An FP16 ONNX version is available under onnx/onnx_fp16/:

import torch
from optimum.onnxruntime import ORTModelForSequenceClassification
from transformers import AutoTokenizer

model_id = "patronus-studio/orca-sonar-document-classifier"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = ORTModelForSequenceClassification.from_pretrained(model_id, subfolder="onnx/onnx_fp16", file_name="model_fp16.onnx")

inputs = tokenizer("def add(a, b):\n    return a + b", return_tensors="pt")
logits = model(**inputs).logits
print(model.config.id2label[int(torch.argmax(logits, dim=-1))])

Citation

@misc{orcasonar2026,
  title={Orca-Sonar: Multilingual Document Topic Classification for Real-World AI Security},
  author={Patronus Protect},
  year={2026},
  howpublished={\url{https://huggingface.co/patronus-studio/orca-sonar-document-classifier}}
}

License

This model is released under the Apache License 2.0. A copy of the license is included as LICENSE in this repository.

The model is derived from jhu-clsp/mmBERT-small, which is distributed under the MIT License. The upstream copyright and permission notice are retained; the MIT terms continue to apply to the portions originating from that work.

Patronus Ark

This model is built to run inside Patronus Ark, Patronus' open-source on-device AI-security scanning library (L1 native rules → L2 NTDB cascade → L3 transformer).

Note: the L2 heads under l2/ were trained and calibrated against the previous L3 model and have not been retrained for this update. Ark is open source: GitHub repository · product page.

More information

• Blog post (EN): Orca-Sonar: Our Multilingual Document Classifier for AI Security
• Blogbeitrag (DE): Orca-Sonar: unser mehrsprachiger Dokumenten-Klassifikator für AI-Security
• Blog post (EN): Our AI-Security Model Zoo Is Now Open Source
• Blogbeitrag (DE): Unser AI-Security-Model-Zoo ist jetzt Open Source
• Product page: Patronus AI models
• Patronus Ark, the open-source scanning library this model runs in: GitHub · product page
• Patronus Protect, the on-device AI firewall: patronus.studio

---

🛡️ Patronus Protect

Brought to you by Patronus Protect, a local AI firewall that secures every AI interaction (prompts, tools, documents) before it reaches your models. Try it for free at patronus.studio.