I’ve been watching the AI hype train for a couple of years now, equal parts fascinated and concerned. The technology is genuinely impressive in some ways, but there’s always been this nagging worry at the back of my mind about what happens when we hand over our critical thinking to machines that don’t actually think.

Recent research showing that ChatGPT, Gemini, DeepSeek, and Grok are serving up Russian propaganda about the Ukraine invasion feels like that worry manifesting in real time. It’s not surprising, but it’s deeply frustrating.

Running large language models locally has been a fascinating journey, especially for those of us who’ve been tinkering with these systems on consumer-grade hardware. Recently, I’ve discovered something quite remarkable about tensor offloading that’s completely changed how I approach running these models on my setup.

The traditional approach of offloading entire layers to manage VRAM constraints turns out to be rather inefficient. Instead, selectively offloading specific tensors - particularly the larger FFN (Feed Forward Network) tensors - to the CPU while keeping the attention mechanisms on the GPU can dramatically improve performance. We’re talking about potential speed improvements of 200% or more in some cases.