Local LLM Inference in Go with Kronk

I’ve been running local models through Ollama’s HTTP API for a while. It works, but it means a separate daemon, JSON over localhost, and no control over when models load or unload. Kronk, from Ardan Labs, takes the other route: it’s a Go SDK that binds llama.cpp (and whisper.cpp, via Bucky) directly into your process. No Python, no sidecar server. The model lives in your binary’s memory and you drive it with Go calls.

Why in-process

The tradeoff versus calling an HTTP API is simple. In-process you get direct control over model loading and lifetime, application-level caching and concurrency, and one fewer moving part to deploy. The HTTP route wins when multiple clients need the same model. Kronk covers both: the same SDK powers a model server with OpenAI- and Anthropic-compatible APIs (more below).

Setup

brew install ardanlabs/kronk/kronk
# or
go install github.com/ardanlabs/kronk/cmd/kronk@latest

The SDK downloads compatible native libraries and models on first run — versions are pinned per Kronk release, so don’t mix in your own llama.cpp build. Hardware acceleration is per-platform: Metal on macOS, CUDA/Vulkan/HIP/ROCm/SYCL on Linux.

Minimal chat

Condensed from examples/question in the repo. Download the native libs, init, download a model, load it, stream a chat completion:

package main

import (
	"context"
	"fmt"

	"github.com/ardanlabs/kronk/sdk/kronk"
	"github.com/ardanlabs/kronk/sdk/kronk/model"
	"github.com/ardanlabs/kronk/sdk/tools/libs"
	"github.com/ardanlabs/kronk/sdk/tools/models"
)

func run(ctx context.Context) error {
	lb, err := libs.New(libs.WithDetect(ctx, kronk.FmtLogger))
	if err != nil {
		return err
	}
	if _, err := lb.Download(ctx, kronk.FmtLogger); err != nil {
		return err
	}
	if err := kronk.Init(kronk.WithLibPath(lb.LibsPath())); err != nil {
		return err
	}

	mdls, err := models.New()
	if err != nil {
		return err
	}
	mp, err := mdls.Download(ctx, kronk.FmtLogger, "unsloth/Qwen3-0.6B-Q8_0")
	if err != nil {
		return err
	}

	krn, err := kronk.New(
		model.WithModelFiles(mp.ModelFiles),
		model.WithAutoTune(true),
	)
	if err != nil {
		return err
	}
	defer krn.Unload(context.Background())

	d := model.D{
		"messages": model.DocumentArray(
			model.TextMessage(model.RoleUser, "Why is the sky blue?"),
		),
		"temperature": 0.7,
		"max_tokens":  2048,
	}

	ch, err := krn.ChatStreaming(ctx, d)
	if err != nil {
		return err
	}
	for resp := range ch {
		if resp.Choices[0].FinishReason() == model.FinishReasonStop {
			break
		}
		fmt.Print(resp.Choices[0].Delta.Content)
	}
	return nil
}

model.D is a document type mirroring the OpenAI request shape, so the request parameters look familiar. The response deltas carry Reasoning separately from Content if the model emits thinking tokens.

The model server

When you do want HTTP, kronk server start runs a server on localhost:11435 built on the same SDK: OpenAI-compatible Chat Completions, Responses, embeddings, reranking, transcription, plus an Anthropic-style Messages API and a browser UI for model management. Existing OpenAI-client code points at it unchanged.

When to use it

Use the SDK when inference belongs inside a Go service — agents, RAG pipelines, embedding jobs — and you want lifecycle control without a daemon. Use the server (or stick with Ollama) when several clients share models or you need auth, rate limiting, and metrics out of the box. Skip Kronk if you’re not on Go or need image generation in production — Malina, its stable-diffusion.cpp SDK, is still experimental.