model_name_talk

I gave a follow-up talk to the AI guild, this time walking through the parts of a model’s filename, specifically Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf, with the goal of understanding the memory requirements of running models and explaining a few of the more important concepts:

A3B: dense versus mixture-of-experts architectures

The key takeaways here are that the parameter count, 30B in our running example, defines how much memory the model requires (along with the quantization, more on that below) and A3B implies that it’s a mixture-of-experts model with the speed of a (dense) 3B model.

Instruct: the effects of post-training

Specifically base versus instruct versus reasoning models. Given a question, a base model probably just asks another question in the same vein where an instruct model will give an answer. A reasoning model goes further and thinks through its response before landing on an answer.

Q4_K_M: quantization

We talked through several of the more popular data types, including FP16, BF16, INT8, and INT4. I also talked through how to approximate the memory requirements for a model given its parameter count and quantization: # of parameters * # of bits per parameter / 8 bytes.

It’s fascinating to me that we can significantly reduce the precision of the model parameters and still get very good results. Even in my toy model, TinyLLM, you can quantize down to 4 bits per parameter without losing much in the way of accuracy. However, at 2 bits it gets a little silly with phrases like “a goose who who is small”.

My next talk will be on model interpretability: revisiting TinyLLM to see if we can figure out why it takes two hops to get to “llama”.

Code: the talk, TinyLLM, and the first talk.