How to Deploy Local LLMs for Embedded Software Development: Hardware Selection and Trade-offs

In the last blog post (here: https://embeddedrelated.com/showarticle/1807.php), I discussed the terminology needed to understand local LLM deployment, such as parameters, quantization, context window, KV cache, etc. In this blog post, I will map these terms to hardware selection and walk through the trade-offs I had to consider when choosing hardware for my local LLM deployment. 

Memory Sizing

Before we get to the hardware, we need to do the math. The terminology from the last post has a direct effect on how much memory (and what type of memory) you need and how fast your system will run. This is where the rubber meets the road.

• Weights at different precisions: Recall that parameters are stored as floating-point values, and quantization reduces that precision to integers to save memory.  The math here is straightforward: Bytes per parameter = Precision bits / 8. Thus, a model at 16-bit floating point ("FP16", which is a common baseline precision) equates to 2 bytes per parameter. 8-bit quantization ("INT8") costs 1 byte per parameter, and 4-bit quantization ("INT4") costs 0.5 bytes per parameter. Concretely, if we have a model with 7 billion parameters ("7B"), that would require roughly 14 GB of RAM at FP16, 7 GB at INT8, and 3.5 GB at INT4. However, the effect of quantization is that we throw away information that the model learned during training. At some point, that starts to show up as degraded output quality, such as less reliable reasoning, more factual errors, or a model that's more easily confused by longer or more complex prompts. From my personal experience, INT8 is usually a safe middle ground with minimal quality loss, while INT4 can introduce noticeable degradation on harder tasks. Generally, I treat quantization as a lever I can pull to see if a model will fit in my hardware and evaluate its efficacy.
• KV cache isn't optional: A lot of novices only consider the parameter size and quantization when choosing the appropriate hardware for their model. They forget to consider the KV cache and realize (often too late) that the model they selected will not fit in the hardware for meaningful use cases. As I mentioned in the last blog post, the KV cache scales with both context length and model size, and it has to be accounted for in addition to the weights, not instead of them. For example, for a 70B parameter model at INT4 (35GB of weights) with a 32K context window, the KV cache can easily add another 10-20GB depending on the model's architecture. If you only budget for the weights and forget the KV cache, you will run out of memory the first time you feed it a large file or a long conversation.
• Mixture-of-Experts misconception: Recall that MoE architectures activate only a subset of parameters per token, even though the total parameter count can be enormous. However, the critical detail that most novices forget is that the entire model still has to be resident in memory, even though only the active parameters are used for a given token. Thus, a 120B total parameter MoE model with 20B active parameters, at INT4, still needs about 60 GB of memory for weights. MoE doesn't save memory but saves bandwidth and compute (and can lead to faster token generation).

The takeaway is that the model's parameter count alone doesn't tell us what hardware we need. We have to account for quantization, context window, and KV cache together, and answering "how much memory do I need" changes depending on the workload and how all of these parameters affect the quality associated with your workload.

Unified Memory vs Discrete GPU

Now that we've established how much memory a given model actually needs, the next question is the type of memory. As I mentioned in the previous post, this comes down to two architectures: unified memory and discrete GPU VRAM.

• VRAM (discrete GPU): The traditional approach, where a dedicated GPU has its own high-bandwidth memory that is separate from system RAM. In this case, the advantage is bandwidth. Discrete GPUs are built for exactly this workload, and higher bandwidth translates directly to faster token generation. The disadvantage is capacity and cost. VRAM is expensive per GB, and even high-end consumer cards can max out well below what we would need for a large model with a large context window. If your model and context window fit in VRAM, a discrete GPU will ALWAYS outperform unified memory. 
• Unified memory: In this architecture, popularized by Apple Silicon, the CPU and GPU share the same physical memory, so there's no hard VRAM ceiling. A system with 128 GB of unified memory can make nearly all of it available to the GPU for weights and KV cache. This is what made unified memory options attractive to me in general. For the price of a capable GPU, I can get an entire system that could load a much larger model than on the GPU. The tradeoff is bandwidth. Unified memory is nowhere close to a discrete GPU in terms of memory bandwidth, so token speed suffers. Thus, we're trading speed for capacity, and the price per GB makes that trade-off very favorable if capacity is your bottleneck.

Making the call for Embedded Development

In the last post, I mentioned that token speed should not be our primary metric as embedded software engineers. Specifically, token speed matters a lot for chatbot-style synchronous use, where you're sitting there watching the tokens stream and want it to feel fast. But a lot of embedded software development isn't like that. If we're asking a model to review a driver, generate Devicetree binding, or work through a longer agentic task, we're not staring at the screen waiting on every token. A slower token speed is a reasonable tradeoff if it means the model and hardware can hold your codebase in context.

If we're running small, quantized models with short context windows, a discrete GPU is the better choice. We get speed without needing much capacity. However, if we want to run larger models, or need a large context window to feed the model substantial portions of a codebase, unified memory is the only economically sensible option. This is why I went with a unified memory setup rather than building a discrete GPU workstation. 

Hardware Tiers

With the memory math and unified memory vs VRAM trade-off discussion, let's get down to the actual hardware. Here is how a few real options stack up as of mid-2026:

• Discrete GPU (NVIDIA RTX 5090 with 32 GB VRAM): While NVIDIA's MSRP is $2k, realistic prices are closer to $4k. Even setting price aside, 32 GB here is a hard ceiling. Recall that a 70B model at INT4 needs roughly 35-45GB once you include KV cache, so it wouldn't fit without dropping to a lower-quality quantization (INT2/INT3) or a smaller model entirely. This is the fastest option per token for anything that fits. However, the VRAM ceiling is what rules it out for larger models and my specific workflows.
• Framework Desktop (Ryzen AI Max+ 395 with 128 GB unified memory): This was impressive when it first launched at $2k (although prices have climbed closer to $3.5k due to the RAM shortage. While memory bandwidth at 256 GB/s is far below a discrete GPU, the 128 GB capacity allows a 70B model at INT4 with room for a large KV cache to comfortably fit. For the price, this is the best capacity-per-dollar option I found. This combination is also a particularly good match for MoE models. The large total unified memory absorbs the total parameter count, while the modest bandwidth is less of a penalty since only a fraction of the parameters are active per token. I didn't end up using this for my embedded software pipeline (see the DGX Spark entry below), but I still use it to automate other, non-technical parts of my business. 
• Mac Studio (up to 96 GB unified memory): Priced from roughly $2k up to $7k depending on the configuration, with meaningfully higher bandwidth than the Framework Desktop (around 800 GB/s on the M3 Ultra). If you're not tied to Linux and can work in Apple's ecosystem this is a strong alternative considering you get the advantage of unified memory's capacity with less of the bandwidth hit. However, its lower capacity ceiling when compared to the Framework Desktop or DGX Spark rules out the largest models.
• NVIDIA DGX Spark (128 GB unified memory each): This is what I bought, in a dual configuration. At around $4.5k per unit, this is a CUDA-native unified memory box, and CUDA-native tooling was the deciding factor for me. I had trouble working out AMD's quirks when I first bought the Framework to get token speeds that were anywhere close to those advertised, which effectively made the Framework Desktop unusable. I ended up biting the bullet and spending a good chunk of money to buy 2 DGX Sparks, and pooling their memory over a high-speed interconnect to give me a whopping 256 GB of combined unified memory. 

Putting It Together

Everything above collapses into a fairly simple decision process that I typically walk through (assuming a Zephyr-based project):

1. Define your context window: As an embedded software engineer, the context window is driven by the amount of code, documentation, or log output that we need the model to hold at once. A prompt about a single file requires far less context than reviewing a driver alongside its Devicetree bindings and related headers.
2. Pick a target model size and quantization: Larger models generally reason better, but quantization lets us trade some quality for a smaller footprint. Depending on our setup, INT8 or INT4 is a reasonable starting point. However, if we find the output quality lacking for a use case, we can explore increased precision.
3. Compute a rough memory requirement: Add the weight size, based on model size and quantization, to a rough estimate of the KV cache size, based on an estimated context window. This is the total that has to fit in memory.
4. Match the number to a hardware tier: If your calculated memory from #3 is under 32 GB, a discrete GPU is worth considering for the speed advantage. If it's > 64 GB, unified memory is almost certainly the more economical choice.

In short, the overall trade-off comes down to memory capacity versus memory bandwidth, and we generally can't maximize both without spending an egregious amount of money. Unified memory systems give you a large pool of memory at a relatively low price per GB but at the cost of bandwidth, compared to a discrete GPU. Discrete GPUs give you much higher bandwidth at more expense and limited VRAM. Remember, neither option is strictly better. The right choice depends on what we're actually trying to run and how we plan to use it. For my own embedded software development pipeline, I ended up going with a dual NVIDIA DGX spark setup, pooling memory across both boxes, mainly because the CUDA-native ecosystem let me get up and running quickly.

In the next blog post, I'll cover the actual pipeline I built on my dual DGX Spark setup: the inference engine, and how I'm evaluating different local models against Claude Code to see how close I can get to that baseline for my day-to-day embedded software development workflow.

If you're trying to figure out whether local inference makes sense for your team and how to size your hardware, reach out to me about my Clarity Session offering (https://mab-labs.com/services). 

Sign in