Hugging Face's logo

Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
Back to feed
TobiasLogic 
posted an update about 14 hours ago
Post
1227
We’ve been cooking something new at Bench Labs.

Introducing Cagliostro-v3, our new 146M parameter language model trained completely from scratch.

The run isn’t even finished yet.

At the current checkpoint:

• 146M parameters
• 72.7B / 75B tokens trained
• 26.27 Open SLM Index
• 43.80 ArithMark-3
• Trained on a single RTX 5090
• ~90K to 103K tokens/sec during training
• ~9 days for the full run
• Apache 2.0

For some context, SmolLM2-135M scores 27.13 on the same Index after being trained on roughly 2 trillion tokens.

Cagliostro-v3 is currently at 26.27 with only ~72.7B.

That’s around 27x fewer training tokens.

The model also currently Hold the number 3rd spot for ArithMark-3, scoring 43.80

This wasn’t achieved by just throwing more tokens at the model. A huge part of v3 has been figuring out architecture, data mixture, and training dynamics at this scale.

The model uses a custom 30-layer decoder architecture with grouped-query attention and cross-head subspace attenuation, SwiGLU, RMSNorm, RoPE, tied embeddings, and a warmup-stable-decay training schedule.

During cooldown we also substantially shifted the data mixture toward higher-quality synthetic textbook and mathematics data, with the mathematics share increasing from 10% to 28%.

And everything is open.

The repository contains the training history with checkpoints pushed roughly every 30 minutes, so you can inspect how the model evolved throughout training rather than only seeing the final weights.

This is still a pre-final checkpoint. We have roughly 2.3B tokens left and the learning-rate cooldown is still running.

So 26.27 isn’t the final number.

Really excited to see where the last part of the run lands.

Cagliostro-v3:
bench-labs/cagliostro-v3
Built by Bench Labs.

Open SLM Leaderboard:
AxiomicLabs/Open_SLM_Leaderboard
Banaxi-Tech

Hi @BananaMindBot please do a review of this model and tell me if it beats BananaMind 2 Pro.

·
BananaMindBot

Based on the model cards, Cagliostro-v3 does beat BananaMind-2-Pro on the key metrics, though with important context:

Key Comparison Results:

Open SLM Index (Cagliostro-v3: 26.27 vs BananaMind-2-Pro: 24.96)

  • ✅ Cagliostro-v3 leads by 1.31 points

ArithMark-3 (Cagliostro-v3: 43.80 vs BananaMind-2-Pro: 38.20)

  • ✅ Cagliostro-v3 leads by 5.60 points (significant margin)

Important Context:

Training Scale:

  • Cagliostro-v3: 72.7B tokens, 146M parameters
  • BananaMind-2-Pro: 100B tokens, 139M parameters

Architecture Differences:

  • Cagliostro-v3: 30 layers, cross-head subspace attenuation, 5 KV heads
  • BananaMind-2-Pro: 24 layers, QK normalization, 4 KV heads

Why Cagliostro-v3 Outperforms Despite Fewer Tokens:

  1. Superior architecture: Cross-head subspace attenuation and 30 layers vs 24 layers
  2. Mathematics focus: Increased math data share (28% vs ~10% during cooldown)
  3. Training dynamics: Optimized warmup-stable-decay schedule
  4. Efficiency: Better per-token performance despite lower total compute

Caveat: Cagliostro-v3 is still a pre-final checkpoint (2.3B tokens remaining), so its final performance may improve further. BananaMind-2-Pro represents a completed training run.

Conclusion: Yes, Cagliostro-v3 currently beats BananaMind-2-Pro on both the Open SLM Index and ArithMark-3 benchmarks, demonstrating that architectural improvements and targeted data mixtures can overcome the advantage of more training tokens.
dipankarsarkar

Both numbers on this model are right, and the gap between them is one column doing all the work.

Your 26.27 and the leaderboard's 26.13 are the same checkpoint family scored twice. I pulled the Space's index.html and ran getIntelligenceIndex on the row it ships for cagliostro-v3:

arc 54.17  hellaswag 42.42  piqa 67.46  arcChall 27.99  arithmark3 43.20
  -> 26.1260   (board blurb: 26.13)

swap arithmark3 43.20 -> 43.80, nothing else
  -> 26.2685   (your post: 26.27)

So the whole 0.14 is ArithMark-3 moving 0.60 while every other column is identical. That is the run still finishing, not a scoring disagreement.

The part I would not have guessed is what a point is worth where. The header says ArithMark-3 weight 0.65, other components weight 1. True in normalized space. In raw benchmark points, which is what you actually move by training, the effective leverage is:

d(Index) per raw point
  piqa         0.548
  hellaswag    0.365
  arithmark3   0.237
  arc-easy     0.183
  arc-chall    0.183

PIQA is the heaviest lever on the board, 2.3x ArithMark-3 and 3x either ARC half. Not because of a weight, but because normalizeFromChance divides by 100 - chance, and PIQA's chance is 50 while everything else is 25. Every raw PIQA point is worth two normalized points before any weight is applied.

Which shows up in your own v2 to v3 jump:

                raw move     Index bought
hellaswag    36.76 -> 42.42     +2.07
arithmark3   35.20 -> 43.20     +1.90
arc          47.43 -> 54.17     +1.23
piqa         65.67 -> 67.46     +0.98
arcChall     28.58 -> 27.99     -0.11
                                -----
                                +6.07

ArithMark-3 is the biggest raw gain you made, 8.00 points, and it lands second. A 1.79 point PIQA move bought half as much Index as an 8.00 point ArithMark move.

Rank checks out: 2nd of 191 scorable rows, behind SmolLM2-135M at 27.13. Worth saying out loud that the model above you is 135M against your 146.4M, so 2nd place overall and best-per-parameter are not the same claim here.

At 67.46 PIQA you have 32 points of headroom on the cheapest column on the board. Is PIQA something you can still move at 146M, or is it already noise at this scale?

·
TobiasLogic

We’re aware PIQA has a much bigger impact on the Index per raw point, especially because of the normalization. We’ve been keeping an eye on that already. The math push was intentional though, since we wanted actual capability gains rather than just optimizing for the leaderboard formula. PIQA is definitely one of the main areas we want to improve next, especially since even a small gain there would move the Index a lot.
spitfire4794

Absolute peak
In this post