{"id":300,"date":"2024-01-13T13:30:45","date_gmt":"2024-01-13T10:00:45","guid":{"rendered":"https:\/\/haghiri75.com\/en\/?p=300"},"modified":"2024-01-15T00:00:16","modified_gmt":"2024-01-14T20:30:16","slug":"nucleus-is-the-proof-that-small-is-the-new-big","status":"publish","type":"post","link":"https:\/\/haghiri75.com\/en\/nucleus-is-the-proof-that-small-is-the-new-big\/","title":{"rendered":"Nucleus is the proof that “Small is the new Big”"},"content":{"rendered":"

\"\"<\/p>\n

No matter what you heard, size matters. Specially in the world of AI models, having a smaller and more affordable model is the key to win the competition. This is why Microsoft even invested time, GPU and money on Phi project<\/a>, which is a Small Language Model or SLM for short.<\/p>\n

In this post, I represent Nucleus<\/a>. My newest language model project, which is based on Mistral (again) and has 1.13 billion parameters. And of course, this post will have a s*it ton of reference to HBO’s Silicon Valley series \ud83d\ude01<\/p>\n

Background<\/h2>\n

If you know me, you know that I have a good background in messing around with generative AI models such as Stable Diffusion, GPT-2, LLaMa and Mistral. I even tried to do something with BLOOM (here<\/a>) before but since the 176B model is too expensive to be put in the mix, I left it behind.<\/p>\n

But later, I started my own AI image generation platform called Mann-E, and in previous weeks, my team delivered Maral, which is a 7 billion parameters language model specializing in Persian language.<\/p>\n

After observing the world of smaller but more specific language models <\/em>(should we call them SMBMLMs now?) like Phi, and also after observing the release of TinyLLaMa, I just started a journey to find how can I stay loyal to Mistral models but make them smaller.<\/p>\n

You know, since the dawn of time, the mankind tried to make things smaller. Like smaller cars, smaller homes, smaller computers and now smaller AI models!<\/p>\n

Basic Research<\/h2>\n

In my journey, I just wanted to know if someone ever bothered to make a smaller version of Mistral or we have to go through the whole coding procedure ourselves.<\/p>\n

Lucky us, I could find Mistral 1B Untrained<\/a> on HuggingFace and even asked the author a few questions about the model. As you can see, they’re not really okay with the model but I saw the potential. So I decided to keep this model in my arsenal of small models for research.<\/p>\n

Then, I searched for datasets, and sparks started in my head about how can I make the damn thing happen!<\/p>\n

\"\"<\/p>\n

The name and branding (and probably Silicon Valley references)<\/h2>\n

The name Nucleus <\/em>comes from HBO’s Silicon Valley. Which is by far my most favorite shows of all time. If you remember correctly, Hooli CEO, Gavin Belson had something to do to piss Richard off, right? So he made Nucleus. But his Nucleus was bad. I tried to make mine better at least \ud83d\ude01<\/p>\n

Since we know it’s time to pay the piper, let’s waste less time and jump right into the technical details of the project.<\/p>\n

Pre-Training<\/h2>\n

Since the model claimed to be untrained <\/em>we can understand that it only knows what the language is right? Even if now you try to infer the model on HuggingFace or locally, you may get a huge sequence of letters with no meaning at all.<\/p>\n

So our first task was to pretrain that. Pretraining the model was quiet easy using a 3090 and spending 40 hours. It was done on the one and only TinyStories<\/a> dataset.<\/p>\n

Actually this dataset is great for pre-training and giving the base models the idea <\/em>of the language and linguistic structures. It does it pretty well. Although since it only has 2 million rows, you have to expect huge over-fitting which can be easily fixed trough fine tuning <\/em>the model.<\/p>\n

Training on Tiny Textbooks<\/h2>\n

Well, the whole point of Phi 1 was that textbooks are all you need <\/em>and since Microsoft doesn’t like to share their dataset with us, we had to perform a huge research on available options.<\/p>\n

The very first option coming to my mind was using GPT-4 to generate textbooks but it could be astronomical minding that we are not funded and spending a few thousand dollars on a dataset? no thanks.<\/p>\n

So during this research procedure, we discovered Tiny Textbooks<\/a> dataset. Apparently Nam Pham did a great thing. They crawled the web and made it to textbooks. So kudos to them for letting us use their awesome dataset.<\/p>\n

Okay, fine tuning called for another 40 hours of training, and it was fine. After fine-tuning for two epochs and 420k steps each, we’ve got the best results we could get.<\/p>\n

Results<\/h2>\n

On TinyStories, the model really loved telling stories about Lily <\/em>and there was no surprise for me at least. But on Tiny Textbooks, the model did a great job. Okay, this is just the result when I asked for a pizza recipe:<\/p>\n

\"\"<\/p>\n

And as you can see, it’s basically what HuggingFace offers. With a little bit of settings, you easily can get good results out of this baby!<\/p>\n

But sadly it still sucks at two things (which are basically what make you even click on an LLM-related link). First is question-answering (or instruction following) which is not surprising and second which made me personally sad is coding <\/em>since I am a developer and I like a well-made coding assistant.<\/p>\n

But in general, it can be competing with other well known models I guess. it all depends on what we train the model on, right?<\/p>\n

But it still needs more effort and training, so we are heading to the next section!<\/p>\n

License<\/h2>\n

If you know me from the past, you know I love permissive licenses. So this model is licensed and published under MIT license. You can use it for commercial use without any permission from us.<\/p>\n

Further changes and studies<\/h2>\n