LocalLLaMA/typed-decisions
Benchmark • Updated • 3.2k • 38.4k • 269
Gjxv PRO
Gjusev
AI & ML interests
None yet
Recent Activity
liked a dataset about 19 hours ago
LocalLLaMA/typed-decisions updated a Space 2 days ago
Gjusev/MK-SH-V-demo updated a model 2 days ago
Gjusev/MK-SH-VOrganizations
None yet
liked a
dataset about 19 hours ago
updated a
Space 2 days ago
updated a
model 2 days ago
liked a
model 3 days ago
replied to their post 6 days ago
Thanks, I hadn't run that comparison before, so I did it now on the per-case logs behind that update.
Raw errors do repeat. 128/335 and 152/335 episodes have an error per seed, 92 shared (about 58 expected if independent, φ 0.42), and 238 calls wrong in both seeds vs about 76 expected.
Confident mistakes don't. Past each seed's abstention threshold, s42 made 2 accepted-and-wrong calls and s43 made 39 over 17 episodes, with one episode in common. Weak evidence at n=2, but it points at s43's own calibration, not at screens both models fall for.
The shared errors do have a direction. 172 of the 238 are mid-episode screens accepted as "task complete", and they sit on the templated tasks (the social-media one is 24% of the set but 38% of shared errors). The two I inspected were busy content screens, a video feed and a lecture slide, with no trace of the goal's artifact yet. Fits the reading that the goal text is under-used.
Per-case JSONLs available if you want to dig in.
reacted to their post with 🔥 10 days ago
Post
2,664
MK-SH-V: A 260M Visual Decision Model
I've released MK-SH-V, an open-source visual decision model built on H Company's NeoMME-260M.
Instead of generating text, it scores candidate answers from screenshots and documents, returning probabilities, a decision or an abstention.
Key results:
82.6% GUI accuracy, compared with 73.1% for a 4B zero-shot VLM.
+29.4 percentage points in real-image accuracy after incorporating real training data.
86% coverage* with a 95% selective-risk upper bound below 5% on held-out synthetic data.
The limitations are documented too. Receipt accuracy remains weak, and calibrated abstention does not yet generalize reliably to real images.
The entire experimental process, code and benchmarks are available under Apache 2.0.
Model: Gjusev/MK-SH-V
GitHub: https://github.com/Gjusev/mk-sh-v
Feedback and contributions are welcome!
I've released MK-SH-V, an open-source visual decision model built on H Company's NeoMME-260M.
Instead of generating text, it scores candidate answers from screenshots and documents, returning probabilities, a decision or an abstention.
Key results:
82.6% GUI accuracy, compared with 73.1% for a 4B zero-shot VLM.
+29.4 percentage points in real-image accuracy after incorporating real training data.
86% coverage* with a 95% selective-risk upper bound below 5% on held-out synthetic data.
The limitations are documented too. Receipt accuracy remains weak, and calibrated abstention does not yet generalize reliably to real images.
The entire experimental process, code and benchmarks are available under Apache 2.0.
Model: Gjusev/MK-SH-V
GitHub: https://github.com/Gjusev/mk-sh-v
Feedback and contributions are welcome!
- 4 replies
posted an update 10 days ago
Post
2,664
MK-SH-V: A 260M Visual Decision Model
I've released MK-SH-V, an open-source visual decision model built on H Company's NeoMME-260M.
Instead of generating text, it scores candidate answers from screenshots and documents, returning probabilities, a decision or an abstention.
Key results:
82.6% GUI accuracy, compared with 73.1% for a 4B zero-shot VLM.
+29.4 percentage points in real-image accuracy after incorporating real training data.
86% coverage* with a 95% selective-risk upper bound below 5% on held-out synthetic data.
The limitations are documented too. Receipt accuracy remains weak, and calibrated abstention does not yet generalize reliably to real images.
The entire experimental process, code and benchmarks are available under Apache 2.0.
Model: Gjusev/MK-SH-V
GitHub: https://github.com/Gjusev/mk-sh-v
Feedback and contributions are welcome!
I've released MK-SH-V, an open-source visual decision model built on H Company's NeoMME-260M.
Instead of generating text, it scores candidate answers from screenshots and documents, returning probabilities, a decision or an abstention.
Key results:
82.6% GUI accuracy, compared with 73.1% for a 4B zero-shot VLM.
+29.4 percentage points in real-image accuracy after incorporating real training data.
86% coverage* with a 95% selective-risk upper bound below 5% on held-out synthetic data.
The limitations are documented too. Receipt accuracy remains weak, and calibrated abstention does not yet generalize reliably to real images.
The entire experimental process, code and benchmarks are available under Apache 2.0.
Model: Gjusev/MK-SH-V
GitHub: https://github.com/Gjusev/mk-sh-v
Feedback and contributions are welcome!
- 4 replies
liked a
Space 10 days ago
upvoted a paper 10 days ago
published a
Space 10 days ago
liked 2
models 10 days ago
published a
model 10 days ago
liked a
Space 12 days ago
updated a
dataset 13 days ago
published a
dataset 13 days ago
posted an update 18 days ago
Post
111
A transcript can read fine and still get your phone number wrong. So I stopped scoring words and started scoring slots.
VoxEntity-EU is an open German/French benchmark: does a speech model capture a dictated name, phone number, email, date, address or reference code exactly? One wrong character fails the slot.
Whisper large-v3-turbo 34.2% · Voxtral Mini 3B 37.1% · correct-text ceiling 90.8%. 240 items per model, 95% CIs in the report. Emails: 0 to 20%.
My first extractor version produced a wrong headline: "ASR is the bottleneck." It was my parser. I retracted it in public commits and shipped what survived.
Read with care: synthetic speech (Qwen3-TTS), simulated phone line, provisional. A second TTS scores 10 points higher, so both results are published.
Dataset
Gjusev/voxentity-eu
Demo
Gjusev/voxentity-eu-results
Full record, retraction included https://github.com/Gjusev/voxentity-eu
VoxEntity-EU is an open German/French benchmark: does a speech model capture a dictated name, phone number, email, date, address or reference code exactly? One wrong character fails the slot.
Whisper large-v3-turbo 34.2% · Voxtral Mini 3B 37.1% · correct-text ceiling 90.8%. 240 items per model, 95% CIs in the report. Emails: 0 to 20%.
My first extractor version produced a wrong headline: "ASR is the bottleneck." It was my parser. I retracted it in public commits and shipped what survived.
Read with care: synthetic speech (Qwen3-TTS), simulated phone line, provisional. A second TTS scores 10 points higher, so both results are published.
Dataset
Gjusev/voxentity-eu
Demo
Gjusev/voxentity-eu-results
Full record, retraction included https://github.com/Gjusev/voxentity-eu
- 1 reply
liked a
Space 19 days ago
liked a
dataset 19 days ago
updated a
Space 19 days ago