Local AI: Starting the Journey

So, I have another piece that’s coming soon about how in general I feel about capitalism and tech. This piece includes how AI is being used showing the worst aspects of both. As a technology person though, I have two things about AI usage that I navigate. One is my personal fascination about the potential of generative AI for coding assistance and tech automation (not for art, entertainment, or “reinventing the wheel” without checking the impact). Two, as a Black Tech engineer who specializes in information security, I unfortunately don’t have the privilege to not understand or limit my use of technology. Especially in the political and economic climate we’re in now. If I don’t at least know AI, it puts my career at risk. So with this lens, I’m trying to move forward with being as ethical in my usage as I possibly can. This brings me to prioritizing use of locally-hosted (or on-prem) Local Language Models (hereafter known as “LLMs”). Locally hosted LLMs are different from service-based or cloud/datacenter hosted AI like Claude, ChatGPT, or Gemini, to name a few.

For me, using local LLMs for code assistance and automation work feels less consumptive than using Cloud-Based AI, despite knowing that it’ll be less efficient overall. I have some privileges that even make this less consumptive, such as having solar-panels on my home to offset some of the electrical consumption from a machine (or so I hope, I’m not an expert on green energy, sadly). This also helps me cost wise as a consultant and researcher because not everyone can afford the $200 bucks a month (before you get into overages) for Claude.

I talk about all this, but, note that my title said starting the journey. Well, my journey begins as any great OG engineering story begins: I need a machine. Let’s go over the research I did to figure out what to prioritize in this build and then go for later (which might be surprising to some of the choices I made).

First, the reason for this build. While my Macbook Air does smaller models fine, working with the bigger models ran into issues just simply because I don’t have enough memory(and while VRAM and RAM are shared, 24GB just isn’t enough for 31B parameter models). And unfortunately my other Linux machine just would crap out at even a small model, which I should’ve known but it’s good to try things out in tech. This resulted in me searching for a machine build that:

1) I could upgrade over time as prices fluctuate
2) That gives me enough to run bigger 30B+ parameter models
3) Can fit on my desk.
4) Can use for other things outside of AI, like 3D modeling
5) 2k budget.

Number 1 obviously took out a lot more of the smaller boxes or Mac Machines due to the soldered-on parts. Point 3 and 4 also made things difficult, as you then have to make the decision on whether you want a ATX board and finding a case that fits on my desk, or going for M-ATX board that won’t be able to upgrade as much over time, or have much capacity to do so.

So, with this knowledge in mind, did some research on the internet, and here’s what I found:

So, working with this and other recommendations, I then decided what I’d like to support with all my dollars. My budget was 2k. Because I didn’t want to deal with porch pirates, wanted to support local businesses, and, I’m near a MicroCenter, decided to go with them. I was able to use their PC builder to spec out some things, but also kept my eye on deals. They recently had a deal where there was a mobo-cpu-ram combo that was 1100, with the newest AMD processor.

So, what I ended up with, though a TON of AI folks are gonna be like “Why Jam”, is this build:

Though people stressed prioritizing GPU VRAM, as an OG computer person (and seeing how CPU is still important), I wanted to have a really strong PC foundation to work with, and the deal MicroCenter gave made me feel confident in that. I’m also kind of excited to see what I can pair with the 3050 to do some interesting things. My next goal is to see if I can get a server-based GPU like the Tesla P40 or V100 (which warning, requires extra cooling and specific cabling in order to not be fried on a desktop) and really see what I can work with not doing a card that just costs as much as this base build, but, we’ll see. I’ll keep you posted!