Summary

Armin argues that local AI models are held back not by model quality but by fragmentation and lack of polish in the local inference stack. The gap between hosted and local experiences comes from too many moving parts—inference engines, quantizations, templates, configs—that users must assemble themselves. He advocates for picking one model, one engine, one hardware config and polishing it end-to-end rather than spreading effort across every new release. He highlights ds4.c, Salvatore Sanfilippo's narrow inference engine for DeepSeek V4 Flash on high-RAM Macs, as the right approach. He built pi-ds4 to embed ds4.c directly into the Pi coding agent with zero configuration, treating local inference as a first-class provider rather than a pile of manual setup.

Key Insight

The local model ecosystem needs to stop optimizing for making every model runnable and instead pick one configuration and polish it end-to-end until it matches the ergonomics of hosted providers.

Spicy Quotes (click to share)

• 3

  I really, really want local models to work.

• 5

  The thought of locking all the experimentation away from the average developer really upsets me.

• 7

  A lot of local model work optimizes for making models runnable. That is necessary, but it is not the same thing as making them feel finished.

• 7

  Hosted model providers do not ship a bag of weights and ask you to figure out the rest, and we need to approach that line of thinking for local models too.

• 7

  Let's pick one winner and polish the hell out of it.

• 4

  This is a terrible way to build confidence.

• 8

  Engineers need hammers and a hammer that's locked behind a subscription in a data center in another country does not qualify.

• 8

  I'm getting big old Python packaging vibes.

Tone

opinionated, urgent, practical