A storage appliance that shipped before AVX2 existed now runs Google's Gemma 4 26B in my basement, no GPU. Here's how we got a modern MoE model onto a CPU older than the instructions its inference engine assumed were there.
TBH, smaller models run pretty well on consumer grade hardware and are good enough for minor tasks like chewing through docs, analysing bank statements or interpreting images. At least the use cases I use it for.
I am playing with olmo-3:7b and I found that my PC gives me 9 tokens per second.
When I looked up a graphics card that could handle it, I found out that it one would only cost me like 0.4 Monero, which I thought was going to be a lot worse.
TBH, smaller models run pretty well on consumer grade hardware and are good enough for minor tasks like chewing through docs, analysing bank statements or interpreting images. At least the use cases I use it for.
I am playing with olmo-3:7b and I found that my PC gives me 9 tokens per second.
When I looked up a graphics card that could handle it, I found out that it one would only cost me like 0.4 Monero, which I thought was going to be a lot worse.