Hello from a 30B model

← Back to home

Hi! This is the first post in a new blog series about local AI models. The thing is, there’s a lot of confusion about local models’ capabilities and limitations right now. People are unsure whether they should invest in a computer with lots of memory to run models locally, or if the models themselves aren’t good enough yet. It might be a waste of money, and they should just stick to the cloud for now. I was unsure, and still am, but I got a new MacBook with 48 gigabytes of unified memory. It’s not the top of the line, like 128 gigabytes that could run DeepSeek Flash with the help of the DwarfStar project, but it’s enough to run 27-billion-parameter models or so.

I decided to document my experience with it for several reasons. First, it might be helpful to other people. Second, it might help me sort things out and understand them more fully. Third, local models will evolve over time, and this website can become a history log of that evolution. It would be interesting to see how their capabilities in the future compare to the current ones. I thought it would be interesting to build the site with the help of local AI. In these posts, I’ll share my experience using local AI for different tasks. The website itself is a task, and each post will be a page created by a local model.

The post texts will come from me, but the page itself will be vibe-coded to publish this human blog post text. I thought maybe it would be done entirely with local models, but they showed their limitations pretty quickly. I also reached for frontier cloud models, but I tried to consult them on the overall direction while delegating specific tasks, like creating a web page, to local models. We’ll see how that goes.

The first thing I wanted to try local models with was setting up this new laptop. In the past, before AI, when I bought a new laptop, I’d spend the first day installing applications. Obviously, I’d also tweak settings and set up terminal aliases and things like that. I know those settings can be moved from one laptop to another, maybe via a script. But I like to start fresh, get rid of the accumulated garbage from my previous laptop, and use only things that have stood the test of time and shown I really need them. Now with AI, it doesn’t seem practical to do all that by hand. At least part of the tasks can and should be delegated to AI models, sparing me manual labor that isn’t really interesting. The question is: which models? I already knew I could set up a new laptop with cloud models, and it’s an enjoyable experience. But what about local ones? How helpful might they be?

With that thought in mind, I fired up the laptop. First, I thought I was too lazy to install all these local models and set everything up myself. So I should use Claude just for that, then switch to local ones after Claude installs them. Like using Internet Explorer to download other LLMs. I tried using Claude to download other LLMs, and I also asked it what the reasonable choices are today. It installed Qwen—regular Qwen for chatting and Qwen 3 for coding. I also installed LM Studio, and it suggested a small Gemma model. It’s small, so I downloaded that too.

I’m not sure about Claude’s choices because it didn’t pick the most modern versions, maybe due to its knowledge cutoff date. I’m not really an expert on local models yet, so I’ll need to dive deeper and check out the current best options for different tasks. For now, though, this web page was built by the Qwen model listed below.

After Claude downloaded the local LLMs and LM Studio to run them, I tried switching over and continuing the setup. But I quickly ran into their limitations. I wanted to create some shell aliases to open things quickly in the terminal. I like short aliases for CLI applications I use often, along with some comments. I asked Qwen to add them, and it started. It added some of what I asked for. Then, when I requested an alias that would take a title for a new folder, create it, and immediately go inside, I was basically asking to combine the mkdir and cd terminal commands. Qwen correctly pointed out that it shouldn’t be an alias. It should be a function, because aliases can’t hold variables, and I needed one to pass the new folder’s name. It added that function to my shell configuration file, and it worked fine.

I thought, OK, great. That slightly exceeded my expectations. I prompted it the wrong way—I said “alias” instead of “function”—but the model correctly understood exactly what I needed and how it’s actually called. That was a pleasant surprise. After that came a less pleasant surprise. When I asked for aliases to open folders I use often, it implemented them as functions too, even though a simple alias would have been enough. So, I asked the model, “Don’t you think an alias might be better here?” And Qwen answered, “Oh, yes, good catch. It could be an alias. I will redo it.” And it tried to fix the situation. But it actually made it worse by leaving out the alias keyword entirely. Maybe because it was trying to redo a function, which doesn’t use that keyword. The new version didn’t work at all until I manually added the alias keyword before them. First, it created an overcomplicated version that I didn’t really need but which worked. Then, when I asked to make it better, it actually broke it.

At this point, I felt I’d need to review each small step it takes and fix things manually. In that case, it doesn’t help much, and it might be easier to just create the shell configuration file myself. There are also more abstract, general questions about this configuration file. How should I group different aliases together? They need some logical structure. It would work without it, but it’s more readable with one. I’ve spent time thinking about how to create a well-thought-out structure for it. If Qwen can’t reliably implement something simple like an alias, it’s probably too much to ask of it to help me think through this structure.

Speaking with frontier models really helps me settle on some of these questions. There’s no strict right answer; it’s up to me how I feel things should be structured. When I talk to a frontier model, it feels like talking to a peer with judgment that can actually help structure things. It suggests ideas, and I decide whether to use them or not. The ideas themselves are pretty decent. Sometimes I just say, okay, I like your plan—let’s go with it. So, I felt I really didn’t want to use the local Qwen model for that. I didn’t even try it. I don’t know what it would come up with. It would be too time-consuming to sift through its suggestions, wondering if they’re great or just random hallucinations. I’d miss out on better ideas from other models by relying solely on local ones.

The thing is, I decided to just use Claude for this. I mostly set up my laptop with Claude after that. It was a really pleasant experience that sometimes felt like someone was taking care of me thoughtfully. I wanted to transfer some configuration files from my old laptop, and Claude understood which parts were still suitable and which should be left behind. It also made me notice some subtle but useful things. I didn’t want to lose that experience just for the sake of using a local model. Setting up a computer is important—it affects how I’ll use it afterward. I want to get the most out of it and feel that it’s as convenient and efficient as possible. I don’t want to use a subpar model for that.

But does that mean my local AI journey has ended before it even started? No, far from it. I don’t want to use local models for important, complicated tasks. But first, they make a good backup plan. I like the idea that even if Claude becomes unreachable tomorrow for some reason, I’ll still have something else. It’s my backup plan. I feel much better knowing that even if Anthropic bans my account tomorrow, I don’t depend on them fully. I rely on their frontier models for their superiority, but I can live without them. That feeling of independence is important. While I have access to Claude’s frontier models, I’ll use them. If I lose that access, I’ll do without.

Second, I still think local models can be used in their current state, despite their limitations. I’d like to use them for tasks that aren’t mission-critical. One of those tasks is actually creating this website. I’m willing to let Qwen publish this post. A frontier model could probably do a better job styling it or creating cool effects. I don’t know. But I enjoy simple websites. I’ve enjoyed them for many years. I remember the old internet. I think Qwen can handle that old-school feeling—black letters over a white background. What could be simpler? That’s a starting point, and I’m okay with it. I don’t need fancy effects; people often overuse them anyway. So, I think Qwen might be useful for publishing it.

Also, the limitations and issues that arise are part of this website’s content. Even if something looks broken on the page, it’s still part of the content. It shows model limitations, which is exactly what this website is dedicated to. It’ll be especially interesting to document some problems now and see how they disappear with newer models. So, I’ll definitely use local models for things like this website. For now, I’ll stick with cloud models for tasks that affect my life and work more directly, where quality matters to me—like work-related projects. But I’ll see where I can get away from cloud models without hurting quality.

Also, something that looks promising right now is using local AI for image generation. It might not reach frontier levels like Nano Banana or ChatGPT Images 2.5/6, which now support GIFs. I won’t get Nano Banana level locally right now, but models like Z-Image seem to work half-decently. For some scenarios, they’re enough. I want to explore what they can and can’t do, figure out the scenarios where they’re good enough. It’ll be especially interesting to see how the number of those scenarios expands over time.

So, I think I’ll wrap up for now. It’s a long text. Ah! You might have already noticed: this post wasn’t written in the usual way. That would take too long, and I don’t have much time right now. I’m a writer, so I think a lot while writing. I can spend a long time chiseling at a short paragraph, trying to find the perfect words. But this website is much simpler and more direct, so I decided to take a different approach. I decided to just say out loud what I think, then run the local Whisper model to transcribe it. Then, I’ll use another local model to slightly edit the transcript—catching things Whisper missed about context and fixing its mistakes. Then I’ll see the result. If it looks publishable, I’ll just publish it that way, with a note about which models were used in its creation. If I’m too unhappy with the result and feel it needs more editing, I might edit it manually and note that in the post. Or, I might give it to Claude or another frontier model to edit and ask them to add a note at the end if they change anything. That’s how it works.

So, on the first day of using local models, I ran into some limitations pretty quickly. I don’t know all of them yet, which isn’t surprising. But I still think local models are good enough for some simple tasks today. I’m pretty sure they’ll be useful for more tasks in the future. So, I’ll use them for this website and we’ll see how it goes. That’s all for the first day, I think.