Ultralytics LLM Interface#

LLM is a small interface for calling language and vision models through an OpenAI-compatible API. It prepares text or image input, forwards request arguments to the official OpenAI Python SDK, and returns the SDK's native response object. This makes it easy to add language or vision reasoning next to a YOLO pipeline without wrapping provider-specific response types.

The default API is responses; use api="chat.completions" for providers that implement only Chat Completions.

Installation#

Install Ultralytics with the optional LLM dependency:

pip install "ultralytics[llm]"

For OpenAI, set the API key in the environment:

export OPENAI_API_KEY="your-api-key"

Responses API#

The Responses API is the default. Pass a prompt and read output_text:

from ultralytics import LLM

llm = LLM("gpt-5.6-luna")
response = llm("What is YOLO?")
print(response.output_text)

Multimodal Input#

Pass a path, HTTP URL, data URI, NumPy array, or PIL image through image:

from ultralytics import LLM

llm = LLM("gpt-5.6-luna")
response = llm("What is happening in this image?", image="https://ultralytics.com/images/bus.jpg")
print(response.output_text)

Local files and image objects are encoded as JPEG data URIs. NumPy arrays use OpenCV's BGR channel order. A string passed as source, such as llm("bus.jpg"), is treated as text; use the image argument to send an image.

Use prompt for an instruction shared by every request:

llm = LLM("gpt-5.6-luna", prompt="Answer in one sentence.")
response = llm("Describe this image.", image="bus.jpg")

Chat Completions#

Select Chat Completions when required by the endpoint, and read its native response shape:

from ultralytics import LLM

llm = LLM("gpt-5.6-luna", api="chat.completions")
response = llm("What is non-maximum suppression?")
print(response.choices[0].message.content)

The same interface accepts multimodal input:

response = llm("Describe this image.", image="bus.jpg")
print(response.choices[0].message.content)

Pass native message objects when you need conversation history. They are forwarded unchanged.

Streaming and Async Calls#

SDK request arguments pass through unchanged, including stream=True:

llm = LLM("gpt-5.6-luna")
for event in llm("Explain object detection.", stream=True):
    if event.type == "response.output_text.delta":
        print(event.delta, end="", flush=True)

Use async_call for concurrent requests:

import asyncio

from ultralytics import LLM

llm = LLM("gpt-5.6-luna")

async def main():
    response = await llm.async_call("What is YOLO?")
    print(response.output_text)

asyncio.run(main())

OpenAI-Compatible Providers#

LLM works with providers that expose an OpenAI-compatible Responses or Chat Completions API, including DeepSeek, Kimi, Z.AI GLM, and OpenRouter. Set the provider's base_url, model name, and API key:

from ultralytics import LLM

llm = LLM(
    "provider-model",
    api="chat.completions",
    base_url="https://provider.example/v1",
    api_key="your-api-key",
)
response = llm("What tasks does YOLO support?")
print(response.choices[0].message.content)

The same configuration works with compatible local servers. Model names, image support, request arguments, and supported APIs vary by provider, so check its linked documentation.

Combine YOLO and an LLM#

Run YOLO first, then call the LLM only when a person is detected:

from ultralytics import LLM, YOLO

yolo = YOLO("yolo26n.pt")
llm = LLM("gpt-5.6-luna")
image = "https://ultralytics.com/images/bus.jpg"

result = yolo(image)[0]
# Call the LLM only when YOLO detects a person.
if any(result.names[int(cls)] == "person" for cls in result.boxes.cls):
    response = llm("Describe the scene.", image=image)
    print(response.output_text)

API Reference#

source accepts text, a native message list, or an image object. image accepts an image URL, data URI, path, NumPy array, or PIL image. Calling model() with a configured prompt sends the prompt by itself.

LLM is inference-only and is not exposed through the yolo CLI. It does not implement train, val, export, track, or benchmark.

FAQ#