Models

hedos treats the models on your machine as one shelf, no matter where they came from or what they do. This page covers how it finds them, how each one is served, how to install new ones and remove old ones, and the one rule that holds through all of it: hedos never moves, copies, or rewrites your weights. It reads them where they are, and deletes them only when you ask it to with hedos rm.

Where models are found#

hedos scan looks in the places models actually live:

The machine has exactly one Hugging Face hub cache, the one the hub's own tools use: HF_HUB_CACHE or HF_HOME says where it is, replacing the default rather than adding to it. A cache elsewhere that should also be swept goes in the hf_cache_roots setting (hedos uses its hub or huggingface/hub subfolder when one exists), so what a pull writes and what a scan reads never come apart. Both settings live in [models]; see configuration.md.

For each model it finds, the scan reads the format, works out the modality and capabilities, and reconciles the result into the registry. A model that moved is migrated to its new record, a model whose weights vanished is marked missing, and duplicates are noted. Nothing is copied or relocated. The registry records point at the files where they already are, so every other tool still sees the same models.

A model on the shelf is in one of three states: ready (resolved to a runtime, weights on disk), unresolved (no runtime here can serve it), or missing (its weights are gone). Only ready models are served.

How a runtime is resolved#

Discovery also resolves each model to a runtime, so the shelf tells you not just what you have but how each model will actually be served.

1. Identify. hedos reads the model's header or config: its format, modality, capabilities, context length, chat template, senses, and tool-calling dialect.
2. Bid. Every runtime wired into this build looks at that identification and bids on the models it can serve.
3. Rank. The lowest bid wins, ties broken by runtime id. The runners-up are kept on the record as alternatives.

The bids, from most to least preferred:

In practice few runtimes bid on the same model, since each bids only on its own formats: a GGUF goes to llama.cpp, a model in the Ollama store to Ollama, an MLX checkpoint to mlx-lm or mlx-vlm. A model no runtime bids on stays unresolved, and a model whose weights are gone is not re-resolved.

Runtimes at a glance#

A model resolves to whichever of these fits it, and each serves only when its backend is present:

The Python sidecars provision their own environment through uv on first use; their runtime code ships inside the binary. A manifest runtime runs code on the host, so it neither bids on a model nor serves one until you approve it, and the approval is bound to a hash of its files: editing any of them asks for it again. hedos ships python:laya, python:zerank, python:yue2 and cli:tessera this way, and reads your own from runtimes.d in the data directory. hedos runtimes lists them and where each approval stands.

GGUF models on llama.cpp#

llama-server runs as a subprocess for .gguf files: chat models, the models that only embed, and the decision models.

• Embedders answer on /v1/embeddings and /api/embed from a server started for embeddings. A GGUF embeds when its header pools its output to one vector (mean, cls, or last), whatever its architecture: the encoders (BERT, nomic-bert, jina, and similar) and the decoders converted to embed (Qwen3-Embedding) alike, and such a model never chats. One whose header ranks (a reranker such as Qwen3-Reranker), and an encoder that names no pooling (a diffusion pipeline's text encoder, or a reranker converted without its pooling), are not embedders and stay unresolved. See gateway.md for how batches and windows are served.
• Decision models are the GGUFs whose header names a decision type ({arch}.decision.type): clef and clef-flash, OpenJev, Kev, lev, Laya and Julia-1, as ggml-org publishes them. Each is a judge, whatever its architecture or chat template says, and answers typed questions on /v1/systemone and with hedos run, never chat. They need llama.cpp 0.6.0 or newer; an older llama-server fails with a message saying so. A decision server holds its declared window up to 16384 tokens.
• Sight. A GGUF sees when a multimodal projector (an mmproj file) sits beside it, or anywhere in its snapshot, that encodes images and outputs the model's own embedding width, as llama.cpp checks it by its tensors. A projector of another width (another model's, kept in the same folder) and one that only encodes audio do not count. Its server is launched with the projector, taking the one named for the weights' quantization when there are several and the smallest otherwise, and never one cut short or unreadable, as an interrupted download leaves it. Among decision models, clef and OpenJev read images; the others read text only, projector or not.

Apple Intelligence#

Apple's on-device model is served through a Swift bridge on Macs where the model is enabled and ready. It appears on the shelf as a built-in model and serves tool calls like any other. On Linux, or on a Mac without it, it simply never shows up.

The Apple bridge is a separate library, and installers do not carry it. Unlike every other runtime, Apple Intelligence needs a companion dynamic library, libhedos_apple_shim.dylib. The build script compiles it during a source build whenever the building SDK carries the FoundationModels framework (a recent Xcode on macOS 26+). Installs that copy only the binary, such as cargo install, Homebrew, and the prebuilt release archives, leave the library behind, so a model that a previous scan recorded still lists on the shelf, but serving it fails with "Apple's model needs the Apple Intelligence bridge, which is not built into this binary."

hedos looks for the library at HEDOS_APPLE_SHIM when that is set, then next to the running binary, then at the path the build wrote it to (which only exists on the machine that built the binary). To enable it after a binary-only install, build the library from source and place it beside the binary:

shcopy

cargo build --release
cp "$(ls -t target/release/build/hedos-runtime-*/out/libhedos_apple_shim.dylib | head -1)" \
   "$(dirname "$(command -v hedos)")/"

The library and the binary must come from compatible builds (the bridge carries an ABI version the binary checks); a mismatched or absent library is skipped, and Apple Intelligence reports itself unavailable rather than crashing.

MLX-Swift#

The MLX-Swift runtime from the original macOS build is framework-bound and is not part of this headless port; its models are served by the MLX sidecars instead. A model that would need it still appears on the shelf, but hedos will tell you it cannot serve it here rather than dropping it.

Capabilities#

Each model declares what it can be asked to do. hedos ls shows them, and hedos ls --capability <name> lists only the models that have one. The gateway and the CLI refuse a request a model's capabilities do not cover, rather than guessing.

Fit and memory#

The FIT column of hedos ls says how a model will sit on this machine. hedos estimates what the model needs as its serving size plus a quarter (working memory beyond the raw weights), then compares that with what the engine it resolved to may use here:

• On Apple Silicon, the Metal engines (Ollama, llama.cpp, the MLX runtimes) may use Metal's working set, the share of memory the GPU may take, read from Metal itself (51.8 GiB of 64 on an M5 Pro). The PyTorch runtimes (diffusers, embeddings) allocate from all of memory.
• With NVIDIA or AMD cards, Ollama and llama.cpp may use the cards' memory together, and run what does not fit from system memory, slower. A model the cards cannot hold well is judged against all of memory instead whenever that reads better, so a small card never makes a model look worse than the processor alone would. A PyTorch runtime uses the largest card. A card under 1 GiB (an integrated GPU's carve-out) is not counted.
• With neither, every engine runs on the processor, from all of memory. So does a model that has not resolved to a runtime, or whose runtime is a remote endpoint or a daemon of its own.

The GPU's share is already what a model may take, so it is judged more loosely than all of memory, which the system and every other program share:

A model whose weights are gone reads gone instead. hedos ls --json carries the verdict as fit: runs_well, tight_fit, too_large, or null.

The serving size is what serving the model loads, which is not always what it takes on disk: a Hugging Face repo that holds several quantizations, or blobs from older revisions, serves one weight set (with its projector and config). The same verdict drives the "too big" count of hedos scan, the shelf screen, and the recommendations of hedos recommend, which hedos pull and the shelf's pull screen offer too.

While serving, a memory governor decides which models stay loaded. Two settings in [models] shape it: keep_warm (how long an idle model stays loaded) and eviction (strict-single keeps one heavy model resident, budgeted keeps as many as fit ram_budget_mb). See configuration.md.

Installing with hedos pull#

hedos pull <reference> resolves a reference and plans the install before anything downloads.

References#

hedos infers the provider from the shape: an org/model with no :tag is a Hugging Face repo, and anything else that reads as a tag is Ollama. --from ollama or --from hf forces it. Since a bare word is a valid Ollama tag, a model named after a pull subcommand is written hedos pull -- ls.

Run hedos pull with no reference in a terminal to search Hugging Face by keyword, or leave the search blank to pick from the models hedos recommend picks for this machine.

The plan#

Before a byte moves, hedos resolves the plan (the name, the destination, and the size) and, in a terminal, asks you to confirm it; from a script it starts without asking. Pulling a model that is already being fetched joins that download instead of starting a second one, and pulling one that stopped part-way carries on from the bytes on disk.

The download itself runs in a worker process of its own, so closing the terminal does not stop it. hedos pull ls shows what is running, hedos pull pause and resume stop and restart a pull without losing the bytes already fetched, and -d starts a download without following it. The worker scans when it finishes, so the model reaches the shelf whether anything is watching or not. The full set of pull subcommands is in cli.md.

Where it lands#

Installs write into each platform's native layout:

• Ollama models pull through the daemon's own API. If Ollama is installed but not running, hedos starts the daemon first.
• Hugging Face models download into the standard hub cache: content-addressed blobs, a snapshot directory of symlinks, and a ref pointing at the revision. Downloads resume with HTTP Range, and each file stored in LFS is verified with SHA-256 against the hash its listing names.

hedos picks which files of a Hugging Face repo to fetch. For GGUF it takes one complete quantization, preferring Q4_K_M, then Q4_0, Q5_K_M, Q6_K, Q8_0, and F16 (else the smallest), along with any mmproj projector and small companion files. Otherwise it takes the safetensors (or PyTorch) weight set and the config and tokenizer files beside it. Documentation, images, and exports nothing here runs (ONNX, OpenVINO, Core ML, Flax, TensorFlow) are skipped.

hedos owns no weights directory of its own, so the moment an install finishes, every other tool sees the model too. Installs do not touch the registry directly; the scan that follows discovers the result.

Gated repositories#

A gated Hugging Face repository needs a token with access to it. hedos reads HF_TOKEN (or HUGGING_FACE_HUB_TOKEN) from the environment, else the token file huggingface-cli login writes ($HF_TOKEN_PATH, else $HF_HOME/token). You also have to accept the model's terms on its Hugging Face page. Without access, the pull stops at the plan, before anything downloads, with a message saying what to do. A download that would not fit on the disk is refused too, naming the size it needs and the space available.

Removing#

Removal is symmetric with install. hedos rm <model> first shows a deletion preview: how many items would go and the estimated size. In a terminal it then asks for a yes/no confirmation (the default is no) and removes nothing unless you agree; in a script or pipe it removes nothing unless you pass -y.

• File-backed models are deleted from disk, permanently (they do not go to the trash).
• Ollama models delete through the daemon, which hedos starts first if Ollama is installed but not running.

The preview is honest about what remains. If duplicate copies of the same weights exist elsewhere on the machine, removing one does not remove the others, and the shelf will still show them. hedos scan lists the copies that can go because another model keeps everything they hold, with what removing each one frees. The exact rules for what counts as a copy, and how shared files, hard links, and symlinks are counted, are under hedos scan in cli.md.

Warm and unload#

hedos warm <model> loads a model with a tiny request, so the next real request starts warm, and reports whether it is resident afterwards. A model is warm where it is served: when a gateway is running on the configured port (or the one --port names), the model is loaded there rather than in the command's own process, which would exit and take the loaded model with it. The probe fits the model, so a judge is asked the smallest well-formed typed question rather than greeted. An extractor reads its model from disk for each request and holds nothing between them, so hedos warm says there is nothing to keep warm.

hedos unload <model> evicts a model from residency and reports the result, asking the Ollama daemon to unload it too when the daemon holds it. Omit the model to pick from the ones currently warm. An idle model is also unloaded on its own once keep_warm runs out.

Judges#

A judge is a model that answers typed questions about a state rather than chatting: given a state and questions of three types (choice, score, and noul, a yes-or-no), it returns probabilities. These are TypeSafe's System One questions. Judges declare the judge capability (hedos ls --capability judge), and come in two kinds:

• Decision GGUFs, served by llama.cpp 0.6.0 or newer: clef and clef-flash, OpenJev, Kev, lev, Laya, and Julia-1. Each reads its whole question in one batch of its window (its declared context, capped at 16384 tokens). clef and OpenJev read images when a projector sits beside them.
• Manifest runtimes: python:laya, an encoder with a decision head that answers in one forward pass, detected by its rl_agent_config.json; and python:zerank, a reranker that scores each option as a document against the question, detected by its chat template. Their probabilities differ in kind: zerank's are relevance, not laya's calibrated belief. Both run only once approved (hedos runtimes approve python:laya).

Ask a judge with hedos run <judge>, passing the question as JSON ({"state": ..., "questions": {...}}) or composing it in a terminal; on the shelf's try screen; or over the gateway's /v1/systemone, which is what a TypeSafe SDK calls. gateway.md has the request and answer format.

Extractors#

An extractor finds people, organizations, addresses, emails and phone numbers in text, groups them into contacts, and splits an address into its parts, with offsets into the source. Tessera is one: a 3.5 MB model with its own small runtime, which hedos runs as a command once per request through the shipped cli:tessera manifest.

hedos pull theiskaa/tessera
hedos runtimes approve cli:tessera
hedos run tessera "Jordan Lee, 123 Main St, Bismarck, ND 58501, (701) 555-0142, [email protected]"

There is nothing to install by hand. The manifest pins Tessera's release archive for each platform (Apple silicon and Intel Macs, x86-64 Linux) by its sha256, and approving it downloads this machine's archive, checks it against the pin, and keeps the binary in the data directory under commands/. The pin is part of the manifest, so the approval covers exactly the binary that runs, and a hedos that pins a newer release asks for it again. The text goes to Tessera on its standard input, never on its command line.

Extractors declare the extract capability (hedos ls --capability extract). Each request names an operation:

Use one with hedos run (see cli.md), on the shelf's try screen, or over the gateway's /v1/extract (gateway.md), which answers exactly what Tessera's own tessera json would.

Tessera's model is experimental and was trained on United States documents: it finds US phone numbers and addresses best, and a request that names no region is read as one from the US, the hint its detector expects. Its confidences say how sure it is, and a result it suggests a person look at is marked.

Audio generation#

An audio generator makes a song, music or a sound from a description, and declares the audio capability (hedos ls --capability audio). It runs as a job that ends in a file, FLAC (24-bit) or WAV at the model's own rate. Two kinds of model are served:

• Models diffusers and transformers run by themselves, on the built-in python:audio runtime: Stable Audio Open, AudioLDM and AudioLDM2, MusicLDM, ACE-Step 1.5 (its -diffusers repos, which sing from lyrics), and MusicGen (and MusicGen-Melody). Each takes the parameters its pipeline has: a duration within what it was trained on (47 s for Stable Audio, 30 s for AudioLDM and MusicGen, up to 600 s for ACE-Step), steps, guidance, a negative prompt where the pipeline takes one, lyrics only where the model sings.
• Models that ship their own code, each on a manifest runtime of its own. hedos ships python:yue2 for YuE2, which writes whole songs with vocals from a style prompt and lyrics, at 48 kHz stereo. It runs YuE2's own package, installed from the wheel in YuE2's repo at a pinned commit and hash. YuE2 decodes through a second repo, m-a-p/YuE2-Vae, which hedos treats as its companion:

hedos pull m-a-p/YuE2-3B
hedos runtimes approve python:yue2      # offers to pull m-a-p/YuE2-Vae
hedos audio YuE2-3B "warm indie pop, female vocals, acoustic guitar" --lyrics song.txt

YuE2 needs lyrics, with [verse] and [chorus] tags on their own lines (--instrumental for none); the song is as long as the lyrics make it, so it takes no duration. --set cot=melody plans only the melody first, --set abc=... gives a score for a cover or an edit, and --set cfg_scale=... sets guidance. It runs on Apple Silicon through Metal: on an M5 Pro with 64 GB, songs of 20 to 45 seconds took about one to one and a half minutes each, after a first run that also builds the runtime's environment (about four minutes in all). A fixed seed gives the same song again, sample for sample. YuE2's model card reports about 71 seconds for a 3.6-minute song on an RTX 4090.

A model that needs a companion counts the companion's size in its fit. Until the companion is on the shelf, the model is refused with the hedos pull that fetches it, and hedos ls says so; a companion on the shelf reads as a part of its owner's runtime.

Voices and generators#

Speech and audio generation stay apart. A voice reads text aloud (speak, streamed, with a voice and a speed); a generator makes a piece from a description (audio, a job, a file). No model claims both. Each route and command refuses the other's models and names the one to use instead.

hedos tells them apart from the model's files. Speech checks run first (Kokoro by its config, Bark, Parler and others by architecture), then the model type (yue2, acestep, heartmula, diffrhythm, minimax_music3, musicgen), then the model card: a card filed under text-to-speech is a voice, one filed under text-to-audio or text-to-music makes audio, and a music tag counts only when no chat template ships. A model hedos pulled before it kept model cards has none on disk, so a voice built on a plain Llama checkpoint (Orpheus, OuteTTS) still reads as a chat model there: hedos pull it again, which fetches the card and nothing it already holds. A model recognized this way never inherits chat from an architecture name such as YuE2ForCausalLM; a model that makes audio with its own code claims no capability until a runtime that drives it is approved. The speech runtime python:mlx-audio serves Kokoro; other voices (Bark, Orpheus, CSM) are recognized but wait for a runtime that can load them.