The $2,000 Fidget Spinner

I watched a man on a forum yesterday celebrate hitting 145 tokens per second on a Llama-3-8B model like he’d just witnessed the birth of his firstborn. He wasn't using the model to write a novel, solve climate change, or even generate a recipe for mediocre chili. He was just watching the text fly across the screen so fast that no human eye could actually read it. It’s the digital equivalent of buying a Ferrari just to see how fast you can make the garage door open and close.

We’ve officially entered the era of Inference Anxiety. This is a specific psychological malady where the utility of a tool is completely eclipsed by the metrics of its operation. It’s not about what the AI says anymore; it’s about how hard it screams it at you. If the model takes 1.2 seconds to think about a complex philosophical prompt, we feel a deep, soul-crushing sense of hardware inadequacy. We start checking the thermal paste. We start eyeing our savings accounts. We start wondering if our spouse really needs that kidney.

The Church of the Perpetual Benchmark

Tools like Llmspeed have turned us into the equivalent of those guys who spend $50,000 on a home theater system and then only use it to play a Blu-ray of a train crashing to test the subwoofer. You ask them, "How’s the movie?" and they say, "I don't know, but did you feel the floor vibrate during the explosion at 42:08?" We are optimizing for the vibration, not the story.

  • Phase 1: You buy a GPU because you want to "explore the frontiers of intelligence."
  • Phase 2: You realize the model is slightly sluggish when generating a 5,000-word fan fiction about Shrek.
  • Phase 3: You spend four nights in a row tweaking quantization kernels and flash-attention parameters.
  • Phase 4: You achieve 200 t/s, stare at the blurring text for three seconds, feel a fleeting moment of God-hood, and then go to sleep, exhausted and alone.

This behavior reveals a fundamental glitch in the human reward system. We are wired to love the "Number Go Up" game. Whether it’s bank accounts, Twitter followers, or tokens per second, the lizard brain doesn't care about the context. If the number is bigger today than it was yesterday, the lizard brain throws a tiny party. The fact that you’re generating 180 tokens per second of absolute gibberish about why Batman should be played by a golden retriever is irrelevant. The speed is the product.

Quantization Is the New Low-Carb Diet

We’ve become obsessed with "quantization," which is just a fancy word for making the AI stupider so it can talk faster. We will happily lobotomize a model from 16-bit to 4-bit just to shave off a few milliseconds of latency. It’s like demanding your waiter sprint to your table at 25 miles per hour, even if he drops half your steak on the floor during the dash. "Sir, your meal is ruined," the waiter pants. "I don't care," you respond, checking your stopwatch, "that was a personal best."

I saw a guy argue that a model was "unusable" because it only ran at 15 tokens per second. For context, the average human reads at about 5 to 6 tokens per second. This man is upset because the computer isn't outrunning his own consciousness by a wide enough margin. He wants the information delivered to his brain via a firehose, even though his brain is a tea strainer. It’s pure, unadulterated performance theater.

What This Actually Means

What this actually means is that we are terrified of the silence between the prompt and the answer. In that 0.8-second gap where the GPU is actually crunching numbers, we are forced to confront the fact that we don't actually have anything important to ask. The speed-running of LLMs is a distraction from the terrifying reality that we have been given the most powerful creative tool in history and most of us are using it to ask for "SpongeBob in the style of The Godfather."

We are treating AGI like a benchmark because benchmarks are easy to understand, while intelligence is scary and demanding. If the number goes up, we’re winning. If we actually have to use the output to do work, we might fail. It’s much safer to stay in the terminal, watching the green text fly by at Mach 3, convinced that because our hardware is fast, our lives are moving somewhere important.

Ultimately, the "Inference Anxiety" trap is just the latest version of the classic human hobby: polishing the hammer instead of hitting the nail. We will continue to buy RTX 5090s, we will continue to obsess over memory bandwidth, and we will continue to produce 400 tokens per second of pure, high-velocity nonsense. And honestly? As long as the graph looks like a hockey stick, we’ll be happy.

Quick Answers

Is 20 tokens per second actually slow?
Only if you are a cyborg or a very impatient speed-reader; for everyone else, it’s faster than you can actually process a coherent thought.

Why am I addicted to benchmarking my local models?
Because it provides a measurable sense of progress in a world where your actual work and creative output feel frustratingly subjective and slow.

Should I upgrade my RAM to get better t/s?
If you have to ask, you’ve already lost the battle against the lizard brain, so yes, buy the RAM and enjoy the three-second dopamine hit.