There is no lmgw SDK, and you don't need one. The gateway copies the OpenAI and Anthropic protocols, so the official SDKs, the OpenAI Agents SDK and anything else built on them work after you change the base URL. Where lmgw adds something, it does so in headers and extra fields that a client can ignore.
Connect
lmgw listens on 127.0.0.1:8787 by default. Until you turn on Require gateway API keys on /v1
in its settings, any key or none is accepted; after that, use a key from Usage → Keys &
budgets.
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8787/v1", api_key="unused")
stream = client.chat.completions.create(
model="my-model",
messages=[{"role": "user", "content": "Write a haiku about VRAM."}],
stream=True,
)
for chunk in stream:
if chunk.choices: # with stream_options.include_usage, the last chunk has only usage
print(chunk.choices[0].delta.content or "", end="", flush=True)
The Anthropic SDKs take the bare address, because they add /v1/messages themselves:
import anthropic
client = anthropic.Anthropic(base_url="http://127.0.0.1:8787", api_key="unused")
message = client.messages.create(
model="my-model",
max_tokens=512,
messages=[{"role": "user", "content": "Hello!"}],
)
print("".join(block.text for block in message.content if block.type == "text"))
Both reach the same alias. A model served by llama.cpp answers an Anthropic client, and a Claude
model answers an OpenAI client: lmgw translates streaming, tool calls, images and reasoning in
both directions. A model that reasons puts a thinking block before its text, which is why the
sample joins only the text blocks.
Find out what a model can do
GET /v1/models lists every alias with what lmgw knows about it, read from the GGUF file, the
running llama-server, a provider's catalog or your own overrides. A client can decide from this
whether to send images, offer tools or show a thinking toggle, instead of guessing from the name.
{
"id": "my-model",
"object": "model",
"owned_by": "llama-server",
"context_length": 131072,
"capabilities": {
"task": "chat",
"endpoints": ["/v1/chat/completions", "/v1/messages", "/v1/responses", "/v1/completions"],
"input_modalities": ["text", "image"],
"vision": true,
"reasoning": { "kind": "toggle", "can_disable": true, "enabled": false },
"tool_calls": { "kind": "native", "parallel": true },
"source": "gguf+config"
},
"notes": ["Reasoning is OFF by default. Per request: header x-lmgw-reasoning: on|off, ..."]
}
That is an excerpt of one entry. endpoints names the routes that accept the model, notes explains its quirks in plain
sentences, and an unknown capability is left out rather than reported as missing.
ModelCapabilities in the API reference has every field.
Reasoning on demand
Thinking models differ in how they are switched: a template flag, an effort level, a token budget. lmgw puts one set of request headers in front of all of them, on chat completions, messages and responses:
x-lmgw-reasoning: onoroffx-lmgw-reasoning-effort: low(the levels a model takes are in itscapabilities.reasoning)x-lmgw-reasoning-budget: 2048
When a route cannot express a control, the response says so in x-lmgw-reasoning-ignored.
Which model answered
A local model can have a fallback, usually a cloud alias that takes over while the GPU hold pauses
local models, or while another program holds the memory the model needs. The response body keeps the alias you asked for in its model field, so
clients that check it keep working. The headers say what really happened:
x-lmgw-fallbacknames the alias that answered instead, andx-lmgw-fallback-reasonsays why.x-lmgw-images-omittedcounts images that went to a fallback without vision as text placeholders.x-lmgw-candidatenames the local model that answered a candidate alias: one name for a primary model and alternates, answered by whichever of them is already loaded.
With no fallback, a request to a paused model fails with 503 and the code gpu_hold, so the
fallback settings decide whether a request may leave the machine. Chat models take the
gateway-wide hold fallback unless their own row says otherwise. The
headers table lists all of them. The API compatibility
page shows which scenarios the OpenAI and Anthropic routes pass against the official SDKs.
Voice
ws://127.0.0.1:8787/v1/realtime speaks OpenAI's Realtime protocol. The OpenAI Python SDK's
realtime client connects with the same base URL as above, since it turns http into ws. The
JavaScript SDK insists on wss:, so there, pass the URL to a client that takes one, such as the
Agents SDK's OpenAIRealtimeWebSocket({ url }). Behind it is a cascade of your own models: lmgw
detects the turns, transcribes them with a speech-to-text model, answers with a chat model and
speaks with a text-to-speech model. Pick the three under Settings → Realtime.
Sessions take audio or text output, PCM16 at 24 kHz, server_vad, semantic_vad and manual
turns, and barge-in. Function tools work as with OpenAI. SIP, ephemeral keys,
transcription-only sessions and G.711 are not implemented yet.
WebRTC calls
POST /v1/realtime/calls is OpenAI's WebRTC transport for the same sessions, in OpenAI's shape:
post an SDP offer (raw application/sdp, or multipart with a session part) with your key and get
the answer back. Audio is Opus over UDP, encrypted with DTLS-SRTP, and events travel on the
oai-events data channel. POST /v1/realtime/calls/{call_id}/hangup ends a call. OpenAI's browser
example and the Agents SDK's OpenAIRealtimeWebRTC work with the base URL pointed at lmgw.
lmgw answers as ICE-lite with one host candidate, the address the POST arrived on, and has no STUN or TURN: the client must reach that address directly, on the LAN or through a VPN. The UDP port is the HTTP port's number unless Settings → Realtime → WebRTC calls names another; that group also lists the live calls.
A session can also use the MCP servers lmgw serves, by label:
{"type": "session.update", "session": {"type": "realtime", "tools": [
{"type": "mcp", "server_label": "docs"}
]}}
lmgw runs the call before the response ends and reports it as an mcp_call item, and as in
OpenAI's API, the client sends response.create for the answer. The OpenAI Agents SDK for
JavaScript (hostedMcpTool) does that by itself. To offer a picker, GET /v1/mcp/servers lists the labels a key may use and
GET /v1/mcp/servers/{label} the tools behind one.
Tools over MCP
http://127.0.0.1:8787/mcp is one MCP endpoint (Streamable HTTP) for every server registered in
lmgw, each under its own tool prefix, plus built-in tools that search library documentation
(docs__*) and your knowledge bases (kb__*). For Claude Code:
claude mcp add --transport http lmgw http://127.0.0.1:8787/mcp
A key's tool scope decides which of them a client sees. The same tools can be named in a
/v1/responses request with {"type": "mcp", "server_label": ...} on any alias lmgw answers
itself, which is every local model and every upstream not set to take /v1/responses directly.
lmgw resolves the label against its own servers and never connects to a URL a client sends.
/mcp also passes the servers' resources through, with their URIs namespaced by tool prefix, so
an MCP Apps host can read a tool's view with resources/read. tools/list stamps every tool of a
registered server with _meta["lmgw/server"] (its label, its name and the tool's own name), and
app-only tools stay out of model runs. When two servers offer a tool of the same name, both move to
<server>__<tool>. When a result has both text and structuredContent, the model reads the text,
and clients and views get the structured part.
Embeddings, rerank and tokens
POST /v1/embeddingstakes OpenAI's request shape. Local embedding models appear with theembed/prefix by default.POST /v1/reranktakes aqueryanddocuments(Jina's spelling) ortexts(TEI's).POST /v1/count_tokenstakes{model, input}and counts a text for any model, with the backend's own tokenizer where it has one.POST /v1/messages/count_tokensandPOST /tokenizeare there for clients that expect Anthropic's or llama.cpp's routes.
Apps: the device link, the Chat and its feed
A desktop or phone client like Kai or Kai mobile needs more than model calls: Chat threads it shares with the dashboard, a feed of changes, an identity of its own and a way to offer its own tools. lmgw has these as general features of its API, open to any client.
- Pairing. Usage → Keys & budgets → Pair a device… on the Devices card shows a pairing link
and a QR code, of the form
lmgw-pair:?v=2&name=…&url=…&fp=sha256:…&token=…. It carries the https URLs to try (LAN, VPN), the fingerprint of lmgw's own certificate authority and a one-time token that works once and expires after ten minutes by default, so treat it as a credential. The device makes its own key pair, enrols the public half over a connection pinned to that fingerprint and gets a certificate. Both sides then show a short code, and the device opens nothing until the owner confirms it on the card. A client can register a URL scheme handler for the link. - The device listener. A paired device connects to a TLS listener of its own, off until the
first pairing turns it on, on the HTTP port's number plus one by default
(Settings → Network & access → Devices). Both sides check each other's certificate, and there
is no bearer key. The listener serves
/v1,/tokenize,/mcp, the Chat API under/chat/api/and a device's own/mcp/hostlink, and nothing else: no dashboard and no admin side. The device's key policy sets its scopes and budget as for a client key. - The feed.
GET /chat/api/feedis one Server-Sent Events stream per client. Stored events (thread.*,folder.*,profile.*,message.*,approval.*,task.*,device.revoked) carry an SSEid:, the cursor to resume from with?since=<cursor>orLast-Event-ID. Live events (turn.*,voice.*,hold,state,resync,revoked,caught_up) carry none, andhellosays what is live when the stream opens, with the device's URLs, its certificate and the approvals still open.keepalive_s(up tochat_feed_keepalive_max_s, 900 by default) sets the keep-alive a client wants, so a phone can hold the connection open in the background and get new messages and open approvals without any push service. - Ongoing conversations. A folder can be marked as one conversation that continues.
POST /chat/api/folders/{id}/currentanswers its current thread, starting a new one after the folder's idle time, or at once with{"new": true}. - Voice in a thread.
GET /v1/realtime?chat_thread=<id>binds a realtime session to a Chat thread: its turns are stored there, and its tools are the thread's. - Personality profiles.
GET /chat/api/profileslists them withdefault_profile_id, the profile new threads start with. A thread'sprofile_id(nullfor none) is set throughPOST /chat/api/threads/{id}/settings, and a folder's defaults take one too. A folder whose default isnullinherits the Chat's default, it does not mean "none".
lmgw-client is the Rust crate
Kai, the core of Kai mobile and the dashboard's voice panel use for these: sans-IO, with the feed decoder, the realtime
voice state and the wire types. The
client apps design record
has the details of the feed and approvals; the pairing and the listener have a design record of
their own in the
device link design record.
Approvals for MCP tools
An entry of a thread's mcp_tools takes require_approval in OpenAI's shapes: "never" (the
default), "always", or {"always": {"tool_names": [...]}, "never": {"tool_names": [...]}}. A
folder's defaults take it too. A turn that reaches a gated call stops there: it streams a tool
frame with "event": "approval" for each waiting call and ends with pending_approvals, and the
waiting reply in GET /chat/api/threads/{id} lists the same calls as pending_approvals.
Any Chat client decides them, with a verdict for every waiting call:
POST /chat/api/threads/42/approvals
content-type: application/json
{"decisions": [
{"approval_request_id": "mcpr_1", "approve": true},
{"approval_request_id": "mcpr_2", "approve": false, "reason": "not now"}
]}
The answer streams the resumed reply, which runs as the key that started the turn. The first
decision wins (409 approval_decided for a later one), and a new message in the thread declines
whatever still waits. The feed records approval.requested and approval.decided, each with the
call's call_id, the same id the turn's ready and result frames carry. A realtime session bound
to the thread gets OpenAI's mcp_approval_request item, answers with mcp_approval_response and
response.create, and hears lmgw.approval.decided when another client decided first.
A device can only tighten the rules: a settings write that would gate fewer calls is
403 approval_loosen_refused. It approves only calls within its own reach
(403 approval_out_of_scope), though it may decline any.
Late tool results
An MCP server can answer a tool call with a task (MCP Tasks). The turn then goes on without the
result, and the result enters the thread when it arrives, never while a turn runs. The next turn
sees it as a call of lmgw__job_result and its result, in the order things happened.
GET /chat/api/threads/{id}/taskslists the thread's tasks that still run or whose results wait to enter, each withid,tool,statusandwaiting_for.POST /chat/api/threads/{id}/tasks/{task}/cancelcancels one,{task}being itsid.POST /chat/api/threads/{id}/answerasks the model to answer results that arrived while nobody was talking. With none waiting it is409 nothing_to_answer, and while a turn runs409 turn_running.- The feed says
task.startedwhen a call becomes a task andtask.donewhen its result entered the thread. A realtime session bound to the thread getslmgw.task.done, and aresponse.createwith no new words has the model answer it aloud. device.revokedon the feed names a paired device whose key was deleted, so a client that runs work that device started can cancel it.
Tools hosted by a device
A device paired with a label under hosts tools as can serve MCP tools of its own. It opens a
WebSocket to GET /mcp/host on the device listener, with its certificate, and the roles reverse: lmgw is the MCP client on
the link, the device the server, one JSON-RPC message per text frame. lmgw's initialize carries
the link's limits in params._meta["lmgw/host_limits"], and each forwarded tools/call names who
it runs for in _meta["lmgw/caller"] and the approval that let it through in
_meta["lmgw/approval"]. The tools appear as <label>__<tool> to the owner and lmgw's own Chat.
A key reaches them only when its device_tools grant names the label, and its tool scope then
filters them like any server's tools.
A device without the grant is refused 403 host_not_granted before the upgrade, and a request with an
Origin header 403 cross_origin_refused. A newer link of the same device takes over (close
4000), a revoked key closes the link with 4003, and a device that declares the tasks capability can
run its tools as MCP tasks, as above.