Reference

API compatibility

What lmgw's OpenAI and Anthropic routes do against the official SDKs, model by model: a matrix from a live check of the running gateway, with every failure and its reason.

This page is generated from a live check of a running lmgw. The check sends real requests through the official openai 3.45.0 and anthropic 2.4.0 Python SDKs and also reads the raw bytes on the wire, because an SDK can accept what the protocol forbids. It covers lmgw's three inference routes, /v1/chat/completions (Chat Completions), /v1/responses (Responses) and /v1/messages (Anthropic Messages), in plain and streaming mode, with 19 scenarios: text, system prompts, multi-turn history, tool calls (none, nested, parallel, forced, a tool that fails), images, structured output, stop sequences, token caps, reasoning with tool calls, token counting and error shapes. Tool scenarios are multi-turn: the tool result goes back and the final answer is read. Every model is reached through the alias a client would use, so each column is also a different path through lmgw.

Run on 2026-11-06 against lmgw 0.5.0: 482 of 540 checks passed, 14 failed and 28 came out differently because of the model. The failures are shown as they are, each with a one-line reason below.

Summary

Checks per route and per way lmgw reached the model. The route-level checks (unknown model, malformed body) are not counted here; they are in the full matrix.

Route OpenAI-compatible upstreamthrough an OpenAI-compatible gateway serviceAnthropic APInativeOpenAI APInativeGemini APInativellama.cpplocal model
/v1/chat/completionsChat Completions 81 pass 2 fail 4 modelof 87 26 pass 3 modelof 29 25 pass 2 fail 2 skippedof 29 27 pass 2 modelof 29 not run yet
/v1/responsesResponses 82 pass 5 modelof 87 27 pass 2 modelof 29 27 pass 2 skippedof 29 27 pass 2 modelof 29 not run yet
/v1/messagesAnthropic Messages 77 pass 6 fail 4 model 6 skippedof 93 27 pass 4 modelof 31 25 pass 2 fail 4 skippedof 31 25 pass 2 fail 2 model 2 skippedof 31 not run yet

Full matrix

One table per route, a plain and a streaming column per model. A number next to a status points to the finding it belongs to.

  • pass every protocol rule held, in the SDK and on the wire
  • fail a protocol rule was broken, or the request failed
  • model the protocol held, but the model did not do what the scenario asked; this says nothing about lmgw
  • not supported yet lmgw refused with its typed unsupported error
  • skipped not run: the GPU was held, a rate limit, or the model has no reasoning control
  • n/a the scenario does not apply to this route

/v1/chat/completions Chat Completions

Scenario OpenAI-compatible upstreamthrough an OpenAI-compatible gateway serviceAnthropic APInativeOpenAI APInativeGemini APInativeNot run yetllama.cpp (local model)
Claudekilo/anthropic/claude-haiku-5.5GPTkilo/openai/gpt-6-lunaGeminikilo/google/gemini-3.8-flashanthropic/claude-haiku-5-5openai/gpt-6-lunaaistudio/gemini-3.8-flash
plainstreamplainstreamplainstreamplainstreamplainstreamplainstream
textOne-word answer pass pass pass pass pass pass pass pass pass pass pass pass not run yet
systemSystem prompt obeyed pass pass pass pass pass pass pass pass pass pass pass pass
multi_turnTwo text turns, the first answer replayed as history pass pass pass pass pass pass pass pass pass pass pass pass
tool_noargsTool without arguments, result sent back, final answer pass pass pass pass pass pass pass pass pass pass pass pass
tool_nestedTool with object and array arguments, result, replay pass pass pass pass pass pass pass pass pass pass pass pass
tool_parallelTwo tool calls in one turn, both results, final answer pass pass pass pass pass pass pass pass pass pass pass pass
tool_errorTool result flagged as an error pass pass pass pass pass pass model 7 pass pass pass pass pass
tool_forcedtool_choice naming one of two tools pass pass pass pass pass pass pass pass pass pass pass pass
tool_nonetool_choice none with tools present pass pass pass pass pass pass pass pass pass pass pass pass
imageImage input pass pass pass pass pass pass pass pass pass pass pass pass
json_schemaStructured output (JSON schema) pass pass pass pass pass pass pass pass pass pass pass pass
stop_seqStop sequence pass pass fail 1 fail 1 pass pass pass pass fail 2 fail 2 pass pass
max_tokensToken cap: the length finish reason pass pass pass pass pass pass pass pass pass pass pass pass
reasoning_toolReasoning, tool call, replay with reasoning, final answer model 6 model 6 model 6 model 6 pass pass model 6 model 6 skipped 12 skipped 12 model 6 model 6
prev_responseSecond turn through previous_response_id not applicable not applicable not applicable not applicable not applicable not applicable not applicable not applicable not applicable not applicable not applicable not applicable
count_tokensThe route's token counter pass not applicable pass not applicable pass not applicable pass not applicable pass not applicable pass not applicable
compactionConversation compacted on demand, then continued from the summary not applicable not applicable not applicable not applicable not applicable not applicable not applicable not applicable not applicable not applicable not applicable not applicable
error_unknown_modelUnknown model: 404 in the route's error shape pass checked once for the route, with a made-up alias
error_malformedTruncated JSON body: 400 in the route's error shape pass checked once for the route, with a made-up alias

/v1/responses Responses

Scenario OpenAI-compatible upstreamthrough an OpenAI-compatible gateway serviceAnthropic APInativeOpenAI APInativeGemini APInativeNot run yetllama.cpp (local model)
Claudekilo/anthropic/claude-haiku-5.5GPTkilo/openai/gpt-6-lunaGeminikilo/google/gemini-3.8-flashanthropic/claude-haiku-5-5openai/gpt-6-lunaaistudio/gemini-3.8-flash
plainstreamplainstreamplainstreamplainstreamplainstreamplainstream
textOne-word answer pass pass pass pass pass pass pass pass pass pass pass pass not run yet
systemSystem prompt obeyed pass pass pass pass pass pass pass pass pass pass pass pass
multi_turnTwo text turns, the first answer replayed as history pass pass pass pass pass pass pass pass pass pass pass pass
tool_noargsTool without arguments, result sent back, final answer pass pass pass pass pass pass pass pass pass pass pass pass
tool_nestedTool with object and array arguments, result, replay pass pass pass pass pass pass pass pass pass pass pass pass
tool_parallelTwo tool calls in one turn, both results, final answer pass pass pass pass pass pass pass pass pass pass pass pass
tool_errorTool result flagged as an error model 9 pass pass pass pass pass pass pass pass pass pass pass
tool_forcedtool_choice naming one of two tools pass pass pass pass pass pass pass pass pass pass pass pass
tool_nonetool_choice none with tools present pass pass pass pass pass pass pass pass pass pass pass pass
imageImage input pass pass pass pass pass pass pass pass pass pass pass pass
json_schemaStructured output (JSON schema) pass pass pass pass pass pass pass pass pass pass pass pass
stop_seqStop sequence not applicable not applicable not applicable not applicable not applicable not applicable not applicable not applicable not applicable not applicable not applicable not applicable
max_tokensToken cap: the length finish reason pass pass pass pass pass pass pass pass pass pass pass pass
reasoning_toolReasoning, tool call, replay with reasoning, final answer model 8 model 8 model 8 model 8 pass pass model 8 model 8 skipped 13 skipped 13 model 8 model 8
prev_responseSecond turn through previous_response_id pass pass pass pass pass pass pass pass pass pass pass pass
count_tokensThe route's token counter pass not applicable pass not applicable pass not applicable pass not applicable pass not applicable pass not applicable
compactionConversation compacted on demand, then continued from the summary not applicable not applicable not applicable not applicable not applicable not applicable not applicable not applicable not applicable not applicable not applicable not applicable
error_unknown_modelUnknown model: 404 in the route's error shape pass checked once for the route, with a made-up alias
error_malformedTruncated JSON body: 400 in the route's error shape pass checked once for the route, with a made-up alias

/v1/messages Anthropic Messages

Scenario OpenAI-compatible upstreamthrough an OpenAI-compatible gateway serviceAnthropic APInativeOpenAI APInativeGemini APInativeNot run yetllama.cpp (local model)
Claudekilo/anthropic/claude-haiku-5.5GPTkilo/openai/gpt-6-lunaGeminikilo/google/gemini-3.8-flashanthropic/claude-haiku-5-5openai/gpt-6-lunaaistudio/gemini-3.8-flash
plainstreamplainstreamplainstreamplainstreamplainstreamplainstream
textOne-word answer pass pass pass pass pass pass pass pass pass pass pass pass not run yet
systemSystem prompt obeyed pass pass pass pass pass pass pass pass pass pass pass pass
multi_turnTwo text turns, the first answer replayed as history pass pass pass pass pass pass pass pass pass pass pass pass
tool_noargsTool without arguments, result sent back, final answer pass pass pass pass pass pass pass pass pass pass pass pass
tool_nestedTool with object and array arguments, result, replay pass pass pass pass pass pass pass pass pass pass pass pass
tool_parallelTwo tool calls in one turn, both results, final answer pass pass pass pass pass pass pass pass pass pass pass pass
tool_errorTool result flagged as an error pass pass pass pass pass pass model 11 model 11 pass pass pass pass
tool_forcedtool_choice naming one of two tools pass pass pass pass pass pass pass pass pass pass pass pass
tool_nonetool_choice none with tools present pass pass pass pass pass pass pass pass pass pass pass pass
imageImage input pass pass pass pass pass pass pass pass pass pass pass pass
json_schemaStructured output (JSON schema) pass pass pass pass pass pass pass pass pass pass pass pass
stop_seqStop sequence fail 3 fail 3 fail 4 3 fail 4 3 fail 3 fail 3 pass pass fail 5 fail 5 fail 3 fail 3
max_tokensToken cap: the length finish reason pass pass pass pass pass pass pass pass pass pass pass pass
reasoning_toolReasoning, tool call, replay with reasoning, final answer model 10 model 10 model 10 model 10 pass pass model 10 model 10 skipped 15 skipped 15 model 10 model 10
prev_responseSecond turn through previous_response_id not applicable not applicable not applicable not applicable not applicable not applicable not applicable not applicable not applicable not applicable not applicable not applicable
count_tokensThe route's token counter pass not applicable pass not applicable pass not applicable pass not applicable pass not applicable pass not applicable
compactionConversation compacted on demand, then continued from the summary skipped 14 skipped 14 skipped 14 skipped 14 skipped 14 skipped 14 pass pass skipped 14 skipped 14 skipped 14 skipped 14
error_unknown_modelUnknown model: 404 in the route's error shape pass checked once for the route, with a made-up alias
error_malformedTruncated JSON body: 400 in the route's error shape pass checked once for the route, with a made-up alias

Failures and model notes

Each distinct finding once, with where it shows up. Statuses are as in the legend: a fail is a protocol rule lmgw did not meet, a model note is what the model did.

#StatusRouteWhat happensWhere
1 fail chat The text runs past the stop sequence instead of ending at it. 2 checks, plain and stream: stop_seqkilo/openai/gpt-6-luna
2 fail chat The model refuses the stop parameter with HTTP 400, and lmgw passes that refusal on. 2 checks, plain and stream: stop_seqopenai/gpt-6-luna
3 fail messages The stop reason is end_turn with no stop_sequence, where the matched stop sequence should be reported. 8 checks, plain and stream: stop_seqkilo/anthropic/claude-haiku-5.5, kilo/openai/gpt-6-luna, kilo/google/gemini-3.8-flash, aistudio/gemini-3.8-flash
4 fail messages The text runs past the stop sequence instead of ending at it. 2 checks, plain and stream: stop_seqkilo/openai/gpt-6-luna
5 fail messages The model refuses the stop parameter with HTTP 400, and lmgw passes that refusal on. 2 checks, plain and stream: stop_seqopenai/gpt-6-luna
6 model chat The model sent no reasoning content with its tool call, so the reasoning replay was not exercised. 8 checks, plain and stream: reasoning_toolkilo/anthropic/claude-haiku-5.5, kilo/openai/gpt-6-luna, anthropic/claude-haiku-5-5, aistudio/gemini-3.8-flash
7 model chat After the tool's error result the model called the tool again instead of answering. 1 check, plain: tool_erroranthropic/claude-haiku-5-5
8 model responses The model sent no reasoning content with its tool call, so the reasoning replay was not exercised. 8 checks, plain and stream: reasoning_toolkilo/anthropic/claude-haiku-5.5, kilo/openai/gpt-6-luna, anthropic/claude-haiku-5-5, aistudio/gemini-3.8-flash
9 model responses After the tool's error result the model called the tool again instead of answering. 1 check, plain: tool_errorkilo/anthropic/claude-haiku-5.5
10 model messages The model sent no reasoning content with its tool call, so the reasoning replay was not exercised. 8 checks, plain and stream: reasoning_toolkilo/anthropic/claude-haiku-5.5, kilo/openai/gpt-6-luna, anthropic/claude-haiku-5-5, aistudio/gemini-3.8-flash
11 model messages After the tool's error result the model called the tool again instead of answering. 2 checks, plain and stream: tool_erroranthropic/claude-haiku-5-5
12 skipped chat Not run: lmgw's model card for this model lists no reasoning control. 2 checks, plain and stream: reasoning_toolopenai/gpt-6-luna
13 skipped responses Not run: lmgw's model card for this model lists no reasoning control. 2 checks, plain and stream: reasoning_toolopenai/gpt-6-luna
14 skipped messages Not run: lmgw's model card for this model lists no on-demand compaction. 10 checks, plain and stream: compactionkilo/anthropic/claude-haiku-5.5, kilo/openai/gpt-6-luna, kilo/google/gemini-3.8-flash, openai/gpt-6-luna, aistudio/gemini-3.8-flash
15 skipped messages Not run: lmgw's model card for this model lists no reasoning control. 2 checks, plain and stream: reasoning_toolopenai/gpt-6-luna

How to read this

  • The matrix shows what one run found, on the date above, with the models named in the columns. A model that passes here may still fail on another prompt.
  • Columns marked not run yet have no result. That is a gap in the check, not a pass.
  • Only the status of each check and the reason for a failure are published, not the requests or the answers.