Apple Silicon MacApache-2.0Early

kinfer

A single-file local inference runtime for agent fleets.

No Docker, no service to install, no separate llama.cpp — one binary, one copy, and it runs. It pulls models straight from Hugging Face and serves them over the Ollama, OpenAI, Anthropic and Responses APIs, so Claude Code and Codex can talk to it as well.

View on GitHub

Status: early but usable. The inference core is done and the CLI works: pull, list, rm, run, ps, and an HTTP server speaking four dialects. What is not done yet is the reason the project exists — health probing, automatic failover and circuit breaking. The changelog records exactly what works and what does not.

Install

On an Apple Silicon Mac:

$ curl -fsSL https://raw.githubusercontent.com/LocalKinAI/kinfer/main/scripts/install.sh | sh

It puts the latest release in ~/.local/bin, checked against its sha256: one file with llama.cpp inside. It is not signed; a firewall that is on asks once before serve or worker accepts connections from another machine.

Or with Homebrew, which keeps it upgraded with everything else:

$ brew tap localkinai/tap

$ brew trust --formula localkinai/tap/kinfer

$ brew install localkinai/tap/kinfer

To build it yourself, or on another platform, see Build in the README.

A first session

$ kinfer pull Qwen/Qwen2.5-0.5B-Instruct-GGUF:Q4_K_M
$ kinfer list
NAME                                    SIZE  MODIFIED
qwen2.5-0.5b-instruct-q4_k_m        468.6 MB  just now

$ kinfer run qwen "explain merkle trees in one sentence"
A Merkle tree is a data structure used in cryptography to store and
retrieve data in a way that is resistant to attacks…

$ kinfer serve
kinfer serving on :11500 — 1 model(s) in ~/.kinfer/models
  Ollama API : POST :11500/api/chat        GET :11500/api/tags
  OpenAI API : POST :11500/v1/chat/completions
  Anthropic  : POST :11500/v1/messages          (Claude Code)
  Responses  : POST :11500/v1/responses         (Codex)

What makes it different

Existing local runners are built for a person talking to a model. kinfer is built for a fleet of agents that must not go down.

	Ollama	Docker Model Runner	kinfer
Install	background service	Docker required	copy one file
Model source	registry + hf.co/…	Docker Hub / OCI / HF	Hugging Face directly
Models on disk	blob hashes	OCI layers	plain filenames
APIs	Ollama, OpenAI, Anthropic, Responses	Ollama, OpenAI	Ollama, OpenAI, Anthropic, Responses
Designed for	one user, one chat	container workflows	agent fleets

Plain filenames matter more than they sound: when something breaks at 2am, ls ~/.kinfer/models tells you what you have, and scp is a valid way to move a model to another machine.

Models bigger than the machine

A model the GPU cannot hold needs no flags. A mixture of experts uses only a few of its experts for each token, so kinfer keeps on the GPU what every token uses plus a cache of the experts in use, and reads the rest from the SSD as the cache misses them. Given another machine's GPU, it hands that machine only the layers that do not fit here.

Qwen3.8-Flash-Next IQ3_S — 51 GiB of weights — with no flags, on a 7,753-token prompt and three 200-token replies:

Machine	GPU memory	Prompt	Decode
Mac Studio M3 Ultra, 96 GB	88 GiB — all of it on the GPU	748 tok/s	43 tok/s
MacBook Pro M5 Max, 48 GB	40 GiB — experts of 28 layers on host memory and SSD	706–726 tok/s	27–29 tok/s
The MacBook Pro, with the Mac Studio as a peer over USB 2.0	40 + 88 GiB	258 tok/s	23 tok/s

Against Ollama on the same MacBook Pro, from the same files and the same requests:

	Prompt	Decode
Ollama 0.40.13, as it sets itself up	17 tok/s	6.9–8.4 tok/s
kinfer, no flags	716 tok/s	27.9–28.4 tok/s

The difference is kinfer's build of llama.cpp, which reads missed experts with a pool of threads and reads the next ones ahead; part of that approach is Strata's. How it works, and every measurement behind it: docs/big-models.md.

Commands

kinfer fitwhat this machine can actually run

kinfer pull <repo>[:quant]download a GGUF model from Hugging Face

kinfer listwhat is installed

kinfer rm <model>delete one

kinfer run <model> [prompt]chat — no prompt means interactive

kinfer pswhat the daemon is holding right now

kinfer serve [-addr :11500]serve over HTTP, context and slots sized to the model

kinfer installkeep serve running at login and after a crash (macOS)

kinfer worker -addr <ip:port>lend this machine's GPU to a serve -peers elsewhere

kinfer plan <model>where a model would go, without loading it

How serve sizes itself, and the rest: serving, agents.

Why it exists

LocalKin runs about 150 agents against cloud models. On 2026-07-23 one of those models stopped responding — not an error, a hang — and the whole fleet went silent for half a day. The failure was not in the model; it was in having no second leg to stand on. kinfer is that second leg: a local inference path that does not depend on any cloud provider or background service.