Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU | Neomind
www.neomindlabs.com
A storage appliance that shipped before AVX2 existed now runs Google's Gemma 4 26B in my basement, no GPU. Here's how we got a modern MoE model onto a CPU older than the instructions its inference engine assumed were there.
  • deathmetal27@lemmy.worldEnglish
    7·
    3 days ago

    TBH, smaller models run pretty well on consumer grade hardware and are good enough for minor tasks like chewing through docs, analysing bank statements or interpreting images. At least the use cases I use it for.

    • shortwavesurfer@lemmy.zipEnglish
      11·
      3 days ago

      I am playing with olmo-3:7b and I found that my PC gives me 9 tokens per second.

      When I looked up a graphics card that could handle it, I found out that it one would only cost me like 0.4 Monero, which I thought was going to be a lot worse.