@DedeProGames Why those languages?
@Banaxi-Tech YES! you should. You are the one that inspired me to start making SLMs in the first place. Just because of some controversy doesn't mean you should stop.
yeah def, it might be easier to do one launch for these though... depends i guess.
@Datdanboi25 just wondering why you would prefer the models to be launched when ready instead of in one big launch?
yeah i know, i have my priorities elsewhere
YAY!
also dang it, i was gonna do that lol. How do you build all of these so quickly?
they wont be using NSA i really only heard about NSA after the BGA stuff lol, ive already started training on most of the models to switch, i'll consider it for future models..
the largest I'm going to do for 1.5 is 125M, 150M will be later lol
actually a longer context does not take up more embedding params. though it sometimes can depending on the positional embeddings used; if you are using RoPE more or less context doesnt affect param count. Out of the most popular types only learned absolute embeddings will always create more params whereas relative learned embeddings will sometimes create more. most others dont.
