This page is generated from a live check of a running lmgw. The check sends real requests
through the official openai 3.45.0 and anthropic
2.4.0 Python SDKs and also reads the raw bytes on the wire, because an SDK can
accept what the protocol forbids. It covers lmgw's three inference routes,
/v1/chat/completions (Chat Completions), /v1/responses (Responses) and /v1/messages (Anthropic Messages),
in plain and streaming mode, with 19 scenarios: text, system prompts,
multi-turn history, tool calls (none, nested, parallel, forced, a tool that fails), images,
structured output, stop sequences, token caps, reasoning with tool calls, token counting and
error shapes. Tool scenarios are multi-turn: the tool result goes back and the final answer is
read. Every model is reached through the alias a client would use, so each column is also a
different path through lmgw.
Run on 2026-11-06 against lmgw 0.5.0: 482 of 540 checks passed, 14 failed and 28 came out differently because of the model. The failures are shown as they are, each with a one-line reason below.
Summary
Checks per route and per way lmgw reached the model. The route-level checks (unknown model, malformed body) are not counted here; they are in the full matrix.
| Route | OpenAI-compatible upstreamthrough an OpenAI-compatible gateway service | Anthropic APInative | OpenAI APInative | Gemini APInative | llama.cpplocal model |
|---|---|---|---|---|---|
/v1/chat/completionsChat Completions |
81 pass 2 fail 4 modelof 87 | 26 pass 3 modelof 29 | 25 pass 2 fail 2 skippedof 29 | 27 pass 2 modelof 29 | not run yet |
/v1/responsesResponses |
82 pass 5 modelof 87 | 27 pass 2 modelof 29 | 27 pass 2 skippedof 29 | 27 pass 2 modelof 29 | not run yet |
/v1/messagesAnthropic Messages |
77 pass 6 fail 4 model 6 skippedof 93 | 27 pass 4 modelof 31 | 25 pass 2 fail 4 skippedof 31 | 25 pass 2 fail 2 model 2 skippedof 31 | not run yet |
Full matrix
One table per route, a plain and a streaming column per model. A number next to a status points to the finding it belongs to.
- pass every protocol rule held, in the SDK and on the wire
- fail a protocol rule was broken, or the request failed
- model the protocol held, but the model did not do what the scenario asked; this says nothing about lmgw
- not supported yet lmgw refused with its typed unsupported error
- skipped not run: the GPU was held, a rate limit, or the model has no reasoning control
- n/a the scenario does not apply to this route
/v1/chat/completions Chat Completions
| Scenario | OpenAI-compatible upstreamthrough an OpenAI-compatible gateway service | Anthropic APInative | OpenAI APInative | Gemini APInative | Not run yetllama.cpp (local model) | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Claudekilo/anthropic/claude-haiku-5.5 | GPTkilo/openai/gpt-6-luna | Geminikilo/google/gemini-3.8-flash | anthropic/claude-haiku-5-5 | openai/gpt-6-luna | aistudio/gemini-3.8-flash | ||||||||
| plain | stream | plain | stream | plain | stream | plain | stream | plain | stream | plain | stream | ||
textOne-word answer |
pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | not run yet |
systemSystem prompt obeyed |
pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | |
multi_turnTwo text turns, the first answer replayed as history |
pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | |
tool_noargsTool without arguments, result sent back, final answer |
pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | |
tool_nestedTool with object and array arguments, result, replay |
pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | |
tool_parallelTwo tool calls in one turn, both results, final answer |
pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | |
tool_errorTool result flagged as an error |
pass | pass | pass | pass | pass | pass | model 7 | pass | pass | pass | pass | pass | |
tool_forcedtool_choice naming one of two tools |
pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | |
tool_nonetool_choice none with tools present |
pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | |
imageImage input |
pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | |
json_schemaStructured output (JSON schema) |
pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | |
stop_seqStop sequence |
pass | pass | fail 1 | fail 1 | pass | pass | pass | pass | fail 2 | fail 2 | pass | pass | |
max_tokensToken cap: the length finish reason |
pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | |
reasoning_toolReasoning, tool call, replay with reasoning, final answer |
model 6 | model 6 | model 6 | model 6 | pass | pass | model 6 | model 6 | skipped 12 | skipped 12 | model 6 | model 6 | |
prev_responseSecond turn through previous_response_id |
not applicable | not applicable | not applicable | not applicable | not applicable | not applicable | not applicable | not applicable | not applicable | not applicable | not applicable | not applicable | |
count_tokensThe route's token counter |
pass | not applicable | pass | not applicable | pass | not applicable | pass | not applicable | pass | not applicable | pass | not applicable | |
compactionConversation compacted on demand, then continued from the summary |
not applicable | not applicable | not applicable | not applicable | not applicable | not applicable | not applicable | not applicable | not applicable | not applicable | not applicable | not applicable | |
error_unknown_modelUnknown model: 404 in the route's error shape |
pass checked once for the route, with a made-up alias | ||||||||||||
error_malformedTruncated JSON body: 400 in the route's error shape |
pass checked once for the route, with a made-up alias | ||||||||||||
/v1/responses Responses
| Scenario | OpenAI-compatible upstreamthrough an OpenAI-compatible gateway service | Anthropic APInative | OpenAI APInative | Gemini APInative | Not run yetllama.cpp (local model) | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Claudekilo/anthropic/claude-haiku-5.5 | GPTkilo/openai/gpt-6-luna | Geminikilo/google/gemini-3.8-flash | anthropic/claude-haiku-5-5 | openai/gpt-6-luna | aistudio/gemini-3.8-flash | ||||||||
| plain | stream | plain | stream | plain | stream | plain | stream | plain | stream | plain | stream | ||
textOne-word answer |
pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | not run yet |
systemSystem prompt obeyed |
pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | |
multi_turnTwo text turns, the first answer replayed as history |
pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | |
tool_noargsTool without arguments, result sent back, final answer |
pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | |
tool_nestedTool with object and array arguments, result, replay |
pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | |
tool_parallelTwo tool calls in one turn, both results, final answer |
pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | |
tool_errorTool result flagged as an error |
model 9 | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | |
tool_forcedtool_choice naming one of two tools |
pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | |
tool_nonetool_choice none with tools present |
pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | |
imageImage input |
pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | |
json_schemaStructured output (JSON schema) |
pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | |
stop_seqStop sequence |
not applicable | not applicable | not applicable | not applicable | not applicable | not applicable | not applicable | not applicable | not applicable | not applicable | not applicable | not applicable | |
max_tokensToken cap: the length finish reason |
pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | |
reasoning_toolReasoning, tool call, replay with reasoning, final answer |
model 8 | model 8 | model 8 | model 8 | pass | pass | model 8 | model 8 | skipped 13 | skipped 13 | model 8 | model 8 | |
prev_responseSecond turn through previous_response_id |
pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | |
count_tokensThe route's token counter |
pass | not applicable | pass | not applicable | pass | not applicable | pass | not applicable | pass | not applicable | pass | not applicable | |
compactionConversation compacted on demand, then continued from the summary |
not applicable | not applicable | not applicable | not applicable | not applicable | not applicable | not applicable | not applicable | not applicable | not applicable | not applicable | not applicable | |
error_unknown_modelUnknown model: 404 in the route's error shape |
pass checked once for the route, with a made-up alias | ||||||||||||
error_malformedTruncated JSON body: 400 in the route's error shape |
pass checked once for the route, with a made-up alias | ||||||||||||
/v1/messages Anthropic Messages
| Scenario | OpenAI-compatible upstreamthrough an OpenAI-compatible gateway service | Anthropic APInative | OpenAI APInative | Gemini APInative | Not run yetllama.cpp (local model) | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Claudekilo/anthropic/claude-haiku-5.5 | GPTkilo/openai/gpt-6-luna | Geminikilo/google/gemini-3.8-flash | anthropic/claude-haiku-5-5 | openai/gpt-6-luna | aistudio/gemini-3.8-flash | ||||||||
| plain | stream | plain | stream | plain | stream | plain | stream | plain | stream | plain | stream | ||
textOne-word answer |
pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | not run yet |
systemSystem prompt obeyed |
pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | |
multi_turnTwo text turns, the first answer replayed as history |
pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | |
tool_noargsTool without arguments, result sent back, final answer |
pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | |
tool_nestedTool with object and array arguments, result, replay |
pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | |
tool_parallelTwo tool calls in one turn, both results, final answer |
pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | |
tool_errorTool result flagged as an error |
pass | pass | pass | pass | pass | pass | model 11 | model 11 | pass | pass | pass | pass | |
tool_forcedtool_choice naming one of two tools |
pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | |
tool_nonetool_choice none with tools present |
pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | |
imageImage input |
pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | |
json_schemaStructured output (JSON schema) |
pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | |
stop_seqStop sequence |
fail 3 | fail 3 | fail 4 3 | fail 4 3 | fail 3 | fail 3 | pass | pass | fail 5 | fail 5 | fail 3 | fail 3 | |
max_tokensToken cap: the length finish reason |
pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | |
reasoning_toolReasoning, tool call, replay with reasoning, final answer |
model 10 | model 10 | model 10 | model 10 | pass | pass | model 10 | model 10 | skipped 15 | skipped 15 | model 10 | model 10 | |
prev_responseSecond turn through previous_response_id |
not applicable | not applicable | not applicable | not applicable | not applicable | not applicable | not applicable | not applicable | not applicable | not applicable | not applicable | not applicable | |
count_tokensThe route's token counter |
pass | not applicable | pass | not applicable | pass | not applicable | pass | not applicable | pass | not applicable | pass | not applicable | |
compactionConversation compacted on demand, then continued from the summary |
skipped 14 | skipped 14 | skipped 14 | skipped 14 | skipped 14 | skipped 14 | pass | pass | skipped 14 | skipped 14 | skipped 14 | skipped 14 | |
error_unknown_modelUnknown model: 404 in the route's error shape |
pass checked once for the route, with a made-up alias | ||||||||||||
error_malformedTruncated JSON body: 400 in the route's error shape |
pass checked once for the route, with a made-up alias | ||||||||||||
Failures and model notes
Each distinct finding once, with where it shows up. Statuses are as in the legend: a fail is a protocol rule lmgw did not meet, a model note is what the model did.
| # | Status | Route | What happens | Where |
|---|---|---|---|---|
| 1 | fail | chat |
The text runs past the stop sequence instead of ending at it. | 2 checks, plain and stream: stop_seqkilo/openai/gpt-6-luna |
| 2 | fail | chat |
The model refuses the stop parameter with HTTP 400, and lmgw passes that refusal on. | 2 checks, plain and stream: stop_seqopenai/gpt-6-luna |
| 3 | fail | messages |
The stop reason is end_turn with no stop_sequence, where the matched stop sequence should be reported. | 8 checks, plain and stream: stop_seqkilo/anthropic/claude-haiku-5.5, kilo/openai/gpt-6-luna, kilo/google/gemini-3.8-flash, aistudio/gemini-3.8-flash |
| 4 | fail | messages |
The text runs past the stop sequence instead of ending at it. | 2 checks, plain and stream: stop_seqkilo/openai/gpt-6-luna |
| 5 | fail | messages |
The model refuses the stop parameter with HTTP 400, and lmgw passes that refusal on. | 2 checks, plain and stream: stop_seqopenai/gpt-6-luna |
| 6 | model | chat |
The model sent no reasoning content with its tool call, so the reasoning replay was not exercised. | 8 checks, plain and stream: reasoning_toolkilo/anthropic/claude-haiku-5.5, kilo/openai/gpt-6-luna, anthropic/claude-haiku-5-5, aistudio/gemini-3.8-flash |
| 7 | model | chat |
After the tool's error result the model called the tool again instead of answering. | 1 check, plain: tool_erroranthropic/claude-haiku-5-5 |
| 8 | model | responses |
The model sent no reasoning content with its tool call, so the reasoning replay was not exercised. | 8 checks, plain and stream: reasoning_toolkilo/anthropic/claude-haiku-5.5, kilo/openai/gpt-6-luna, anthropic/claude-haiku-5-5, aistudio/gemini-3.8-flash |
| 9 | model | responses |
After the tool's error result the model called the tool again instead of answering. | 1 check, plain: tool_errorkilo/anthropic/claude-haiku-5.5 |
| 10 | model | messages |
The model sent no reasoning content with its tool call, so the reasoning replay was not exercised. | 8 checks, plain and stream: reasoning_toolkilo/anthropic/claude-haiku-5.5, kilo/openai/gpt-6-luna, anthropic/claude-haiku-5-5, aistudio/gemini-3.8-flash |
| 11 | model | messages |
After the tool's error result the model called the tool again instead of answering. | 2 checks, plain and stream: tool_erroranthropic/claude-haiku-5-5 |
| 12 | skipped | chat |
Not run: lmgw's model card for this model lists no reasoning control. | 2 checks, plain and stream: reasoning_toolopenai/gpt-6-luna |
| 13 | skipped | responses |
Not run: lmgw's model card for this model lists no reasoning control. | 2 checks, plain and stream: reasoning_toolopenai/gpt-6-luna |
| 14 | skipped | messages |
Not run: lmgw's model card for this model lists no on-demand compaction. | 10 checks, plain and stream: compactionkilo/anthropic/claude-haiku-5.5, kilo/openai/gpt-6-luna, kilo/google/gemini-3.8-flash, openai/gpt-6-luna, aistudio/gemini-3.8-flash |
| 15 | skipped | messages |
Not run: lmgw's model card for this model lists no reasoning control. | 2 checks, plain and stream: reasoning_toolopenai/gpt-6-luna |
How to read this
- The matrix shows what one run found, on the date above, with the models named in the columns. A model that passes here may still fail on another prompt.
- Columns marked not run yet have no result. That is a gap in the check, not a pass.
- Only the status of each check and the reason for a failure are published, not the requests or the answers.