Picking a Model That Fits Your Machine (and Letting Claude Be Your Guide)

This is the post I most wanted to write, because it’s where I learned two things that changed how I approached the whole project.

The first: the biggest, smartest model that fits on your computer is not always the one you should use. The second, and the one I’ll start with because it makes everything else easier: you don’t have to figure any of this out alone. The single best tool for setting up local AI is the frontier AI you already pay for. More on that in a moment.

Let the frontier model do the hard part

Here’s the slightly ironic shortcut at the heart of this series. The fastest way to get a local model running is to ask one of the big cloud chatbots to walk you through it, step by step, for your exact computer.

I tried this properly and it was the thing that made the whole project click. I told the chatbot precisely what hardware I had and what I wanted to do, and it produced a setup plan tailored to my machine: which model to pick, which size to download, what settings to use, and what to do when something went wrong. It was like having a patient expert sitting next to me.

I tried a few. In my experience, Claude was the most accurate. It made the fewest confident-but-wrong claims about hardware and models, which matters enormously when a wrong number sends you downloading the wrong file for an hour. And a detail worth knowing if you’re watching costs: I didn’t need the most expensive, most powerful version. Claude’s mid-tier model, Sonnet, was plenty. Accurate, and noticeably faster to go back and forth with. You don’t need to burn your premium quota on this. The everyday model is more than up to the job of being your setup guide.

So before you do anything else, here’s the move: open your chatbot of choice, tell it exactly what computer you have, and ask it to recommend a local model and walk you through running it. Everything below will help you have a smarter conversation with it, but the conversation itself is the real shortcut.

The thing that decides everything: memory

To know what your machine can run, and to tell the chatbot something useful, you need to understand one concept. Just one.

Every AI model is, underneath, a gigantic file. To run, that whole file has to be loaded into fast memory that the part of your computer doing the thinking can reach instantly. Think of that memory as the size of your kitchen counter. A small recipe needs a little counter space. An elaborate one needs a lot. If the recipe needs more counter than you have, you simply can’t make it. There’s nowhere to lay everything out.

Models work exactly the same way. A model either fits in your fast memory or it doesn’t. There’s no “sort of fits.” Bigger models, which tend to be smarter, need more room. So the amount of this fast memory you have sets a hard ceiling on how capable a model you can run.

Now, and this is the part that was glossed over in my earlier thinking and genuinely deserves more attention, what counts as that fast memory depends entirely on what kind of computer you have. And the two main options work so differently that they’re almost different sports.

The great hardware fork: PC-with-a-graphics-card vs. Mac

There are two sensible roads into local AI hardware. They have very different shapes, costs, and trade-offs, and which suits you depends on how you weigh money, speed, noise, and tinkering.

Road one: a PC with a graphics card

On a Windows or Linux PC, the fast memory that runs AI lives on your graphics card: the GPU, the expensive chip gamers obsess over. It’s called VRAM, and it’s separate from your computer’s ordinary memory.

This is the road I took, and its great virtue is raw speed. A good graphics card runs models fast, noticeably faster, token for token, than the Mac equivalent. If you or someone in your house already games, you may own a capable AI machine and not know it, which makes this road close to free to start.

The catch is the ceiling. Consumer graphics cards top out at a limited amount of VRAM (commonly 8, 12, 16, or 24 gigabytes), and you can’t simply add more. Want to run a bigger model than your card allows? You’re buying another card, which is what I did (I run two cards together for about 28 gigabytes), or buying a pricier card. And graphics cards are power-hungry and can be loud; a serious rig draws several hundred watts and the fans let you know it’s working.

Rough money: if you’re building or buying fresh, a capable single-card machine runs from perhaps a thousand dollars upward depending on the card, and climbs steeply if you chase the big cards with lots of VRAM. If you already own a gaming PC, your starting cost might be zero.

Road two: a Mac (and why people happily spend a few thousand on one)

Apple’s recent Macs work on a fundamentally different principle called unified memory, and once you understand it, the appeal of those pricey Mac minis and Mac Studios suddenly makes sense.

On a PC, your computer’s ordinary memory and the graphics card’s VRAM are two separate pools, and the AI can only use the smaller VRAM pool. On an Apple Silicon Mac, there’s just one pool of memory that everything shares: the main processor, the graphics part, all of it. There’s no separate, smaller VRAM box to be trapped inside.

The practical consequence is striking: on a Mac, nearly all your memory is available to run models. A Mac with 64 gigabytes of memory can load models that would need a graphics card costing far more than the whole Mac. No consumer graphics card offers anywhere near 64 gigabytes of VRAM at any price. That single fact is why people cheerfully drop a few thousand on a Mac for this. They’re not overpaying; they’re buying access to model sizes the PC road can’t reach without serious expense.

The trade-offs, to be fair about it:

Macs are slower per answer than a strong graphics card. The memory is shared and capable, but its bandwidth (think of it as how fast ingredients can be moved on and off that big counter) is lower than a dedicated card’s, and AI speed is largely governed by that bandwidth. Expect a Mac to be meaningfully slower per word than a good PC card running the same model. It can still feel like real-time chat for mid-sized models; it’s just not the PC’s raw pace.

You choose your memory at purchase and you’re stuck with it forever. Apple’s memory is soldered in. You cannot add more later, ever. So the one hard rule everyone repeats is: buy as much memory as you can afford up front, because it’s the thing that decides which models you’ll ever be able to run.

But the upsides are real and easy to undersell. A Mac sips power (a fraction of what a graphics-card rig draws), runs essentially silently, and the software experience tends to be smoother. For an always-on assistant humming away in the corner, that quiet efficiency is genuinely lovely, and over a year of running it the electricity savings are not trivial.

Roughly how the tiers shake out

To make the Mac road concrete (since that’s the one that needed more attention), and treating these as ballpark figures rather than gospel:

• An entry Mac mini with modest memory handles small, useful models and is a fine, cheap way to experiment.
• The configuration most people land on as the sweet spot is a Mac mini with the “Pro” chip and 48 to 64 gigabytes of memory, somewhere in the rough region of two thousand dollars, which comfortably runs the mid-sized models that do most real work.
• Stepping up to a Mac Studio with 128 gigabytes or more, at meaningfully higher cost, is for people who want to run the really large models or serve several at once.

The honest summary of the fork: the PC-with-a-card road is faster and can be cheaper to start (especially if you already game), but it hits a memory ceiling that’s awkward and expensive to push past. The Mac road costs more up front and runs slower per answer, but its unified memory buys you access to bigger models, near silence, and tiny power bills. Neither is “right.” It depends on whether you value speed and tinkering or capacity and calm. And, honestly, what’s already on your desk.

And again: you do not have to adjudicate this alone. Tell the chatbot your budget and your preferences and let it argue both sides for your situation. That’s exactly the kind of question it’s good at.

The test that taught me the second lesson

With my own setup sorted (about 28 gigabytes across two graphics cards), I could fit several models, so I tried them on real tasks. Two stood out, and the contrast is the rest of this post.

The first, GPT-OSS 20B, is quick. Snappy, conversational, answers just there. It fits comfortably with room to spare and for everyday questions it’s a pleasure.

The second, Qwen3.6-27B, is bigger and uses my full setup, both cards working together. And it is noticeably smarter. On a genuinely hard, multi-step problem (the kind where you hold several things in your head and reason carefully), it was clearly better, making fewer mistakes and reasoning more soundly.

But it pays for that intelligence with speed: in my testing, roughly three times slower. Not unusable, but you feel it. With the quick model the answer is just there; with the smart one you ask, then wait, watching words appear at a patient, deliberate pace.

Patient versus snappy

That’s the trade-off, and I think of it in exactly those terms: patient versus snappy. There’s no universally right answer. It depends on what you’re doing.

For live back-and-forth (quick questions, drafting, staying in flow), the snappy model wins easily. Waiting three times longer per reply would drive me up the wall, and the extra brains aren’t worth much for simple tasks anyway.

But for a job I set running overnight, or one hard question I ask before going to make coffee, the smart model is the obvious pick. Speed is irrelevant when I’m not waiting; quality is everything. This is exactly where an agent like Hermes pays off. Handed a job to chew on, the patient-but-smarter model does work I actually trust.

So I keep both. Snappy for conversation, patient for hard problems I can walk away from. That dual setup turned out far more useful than chasing one “best” model.

	GPT-OSS 20B (snappy)	Qwen3.6-27B (patient)
Speed	Fast, conversational	About 3x slower
Reasoning on hard problems	Good	Genuinely better
Fits on	One card, room to spare	Both cards together
Best for	Live back-and-forth	Hard problems you can walk away from

There’s a newcomer worth a mention: the graphics-card maker Nvidia now releases its own family of open models, called Nemotron, built for exactly this kind of work, with small versions that fit a home setup easily. I haven’t lived with them long enough for a firm verdict, but they signal where this is going: the hardware makers themselves now want you running capable AI at home.

The takeaway

“Which model is best” is the wrong question. The right one is “best for what.” Fast for flow, smart for depth, both for free. And the smartest move of all is to stop guessing and let the frontier model (Sonnet is plenty) plan the whole thing around your specific machine.

Next post, we actually build it.