Jackson Yeskie

I tried some simple ones like a teapot and I tried something from a video game which worked better than I thought it would. I'm curious though, is the real image or synthetic toggle for training reasons (so you dont use ai generated images in training) or does it tweak the model at all based on that?

Tried the space. Its super good, especially with only 7.2gb of vram, good job!

It's still training but I'll @ you here when its ready!

@RakshithFury I'm training a model right now and by 18b tokens it was able to do division, carrying, and even some multiplication that would be difficult in your head. 1000% It just depends on your data.

@jiteshkprasad For your next model you should focus on data, if you are just training on random scrapes, or datasets that are extremely old, you should look to much more quality, newer datasets. At this scale training on large, unfiltered dumps is only worth it if you NEED data at the trillions

1,000× less training data than Qwen2.5-0.5B and accuracy almost equal. Have you benchmaxed it?

Seems so. I tried it out and it couldn't even make a script that combines a + b and returns their outputs or even reverse a string. It also, as the post says, is extremely bad at math. Even on their website the prompt 'Explain in simple words why the sky is blue.' that it tells you just incorrect stuff consistently. Its SFT is sort of good in how it looks with certain prompts but is very bad at instruction following. For example: 'Give me 9 points on a bullet point list about gardening.' produces is only actually 9 points sometimes.