YOLOE: Real-Time Open-Vocabulary Detection and Segmentation#

Ultralytics YOLOE (Real-Time Seeing Anything) is an open-vocabulary detection and instance segmentation model: instead of a class list fixed at training time, it takes the categories you want at inference time, as a text prompt, a visual example, or a built-in 4,585-name vocabulary. Built on the Ultralytics YOLO architectures — YOLOv8, YOLO11 and YOLO26 — and inspired by YOLO-World, YOLOE reaches state-of-the-art zero-shot accuracy at close to closed-set YOLO speed.

Quick Start#

Name the classes you want and run. YOLOE-26 returns boxes and instance segmentation masks for categories it was never trained on.

from ultralytics import YOLOE

model = YOLOE("yoloe-26s-seg.pt")

# "double-decker bus" is not a COCO class; YOLOE resolves it from the words alone
model.set_classes(["double-decker bus", "person"])

results = model.predict("https://ultralytics.com/images/bus.jpg")
results[0].show()

The first set_classes() call downloads a text encoder; see Installation and Requirements before deploying to a machine without network access.

Choosing a Prompting Mode#

YOLOE supports three prompting modes, and the choice decides which checkpoint you load and what your class labels look like. Pick the row that matches what you can supply at inference time.

Installation and Requirements#

YOLOE ships in the main Ultralytics package:

pip install -U ultralytics

Text prompting needs a text encoder on top of that, and it is fetched on first use rather than at install time:

• The first set_classes() call installs ultralytics/CLIP from GitHub with pip (it provides the tokenizer) and downloads a TorchScript text encoder into the current working directory. YOLOE-26 pulls mobileclip2_b.ts, about 254 MB; YOLOE-11 and YOLOE-v8 pull mobileclip_blt.ts. Run the download once from the directory you will run from, or copy the file there, otherwise it is fetched again.
• Both steps need network access, so run one prompted prediction before deploying to an offline or air-gapped machine.
• Visual prompts and prompt-free checkpoints need no text encoder at all.

The YOLOE-26 checkpoints require ultralytics 8.4.0 or newer; the YOLOE-11 and YOLOE-v8 families are available in earlier releases. One text-prompted prediction exercises the whole path — checkpoint download, CLIP install, text encoder, inference:

yolo predict model=yoloe-26s-seg.pt source="https://ultralytics.com/images/bus.jpg" classes="person"

To skip the text-encoder download at inference time entirely, bake the prompts into the weights once and reuse them — see Reuse prompt embeddings.

Architecture Overview#

YOLOE keeps the standard YOLO structure — a convolutional backbone for feature extraction, a neck for multi-scale fusion, and an anchor-free, decoupled head predicting classes and boxes — and adds three modules, one per prompting mode:

• Re-parameterizable Region-Text Alignment (RepRTA) refines text embeddings from CLIP through a small auxiliary network. That network runs once per set_classes() call and is folded away at export, so it costs nothing per frame. What does run on every forward pass is the comparison of the stored prompt embeddings against region features; see Limitations for what that costs with a large prompt set.
• Semantic-Activated Visual Prompt Encoder (SAVPE) encodes semantic and activation features from an example box, conditioning the model on objects that look like it. This is the one-shot path for targets that are hard to name, such as a logo or a specific part.
• Lazy Region-Prompt Contrast (LRPC) matches region embeddings against a built-in 4,585-name vocabulary, so prompt-free checkpoints recognize objects with no external prompt and no text encoder.

Instance segmentation comes from a mask branch on the detection head, as in YOLOv8-Seg, and every prediction carries a mask on results[0].masks. Once the model is exported, the open-world modules are re-parameterized into a standard YOLO head, so the exported file runs the ordinary detect/segment path.

Available Models#

Every checkpoint below is an instance segmentation model and supports val, predict, export and track. Load a *-seg.pt file for text or visual prompting and a *-seg-pf.pt file for prompt-free inference; they are not interchangeable, see Choosing a Prompting Mode. Only the *-seg.pt files support train; a prompt-free checkpoint is produced from a trained text-prompt model, see Training the Official Models from Scratch.

YOLOE Performance on LVIS#

Zero-shot results on LVIS minival at 640 pixels, from the Ultralytics YOLO26 paper.

Text and visual prompts#

Each accuracy and parameter cell reads text prompt / visual prompt; FLOPs are given once. Parameters and FLOPs are for the detection configuration the paper evaluates. Accuracy is the paper's Non-E2E figure, the only protocol it reports for every model in the comparison; the YOLOE-26 end-to-end head trails it by at most 1.1 AP under text prompts and 2.6 AP under visual prompts.

Prompt-free#

The prompt-free checkpoints answer from their built-in vocabulary with no prompt supplied. Each accuracy cell reads end-to-end / Non-E2E, the two protocols the paper scores YOLOE-26 under; the YOLO26 page quotes the Non-E2E column.

Under text and visual prompts the YOLOE-26 models lead their YOLOE-11 and YOLOE-v8 counterparts at every matching scale on mAP_50-95, while staying below the v8 line on parameters and FLOPs. On the same split the paper reports YOLO-Worldv2 at 24.4 (S), 32.4 (M) and 35.5 (L), and the transformer-based detectors GLIP-T at 26.0, GDINO-T at 27.4 and DetCLIP-T at 34.4, each carrying 155 to 232 M parameters. The original YOLOE paper adds two results for the v8-scale models it introduced. On LVIS, YOLOE-v8s beats YOLO-Worldv2-S by 3.5 AP at a third of the training cost and 1.4× the inference speed. Transferred to COCO, YOLOE-v8l gains 0.6 box AP and 0.4 mask AP over closed-set YOLOv8-L with nearly 4× less training time.

Usage Examples#

Every YOLOE example below runs from the Python API. Text-prompt prediction, validation, export, tracking and plain training also work from the CLI; the fine-tuning recipes and visual prompting pass a trainer or predictor class as an argument, which only the Python API accepts.

Train Usage#

Fine-tune any released *-seg.pt checkpoint on your own YOLO dataset. This mostly follows the standard YOLO training procedure; the difference is which trainer you pass. YOLOEPESegTrainer fuses your class names into the head and fine-tunes from there, which is what you want on your own labels; the default trainer does not train against your class names.

from ultralytics import YOLOE
from ultralytics.models.yolo.yoloe import YOLOEPESegTrainer

model = YOLOE("yoloe-26s-seg.pt")

results = model.train(
    data="coco128-seg.yaml",
    epochs=80,
    patience=10,
    trainer=YOLOEPESegTrainer,  # <- Important: the fine-tuning trainer, not the default
)

from ultralytics import YOLOE
from ultralytics.models.yolo.yoloe import YOLOEPETrainer

model = YOLOE("yoloe-26s.yaml").load("yoloe-26s-seg.pt")

results = model.train(data="coco128.yaml", epochs=80, patience=10, trainer=YOLOEPETrainer)

Predict Usage#

The text-prompt call is the one shown in Quick Start. The remaining two modes each need an extra argument:

import numpy as np

from ultralytics import YOLOE
from ultralytics.models.yolo.yoloe import YOLOEVPSegPredictor

model = YOLOE("yoloe-26l-seg.pt")

# One example box per target, each with its own class ID
visual_prompts = {
    "bboxes": np.array([[221.52, 405.8, 344.98, 857.54], [120, 425, 160, 445]]),  # person, glasses
    "cls": np.array([0, 1]),
}

results = model.predict(
    "ultralytics/assets/bus.jpg",
    visual_prompts=visual_prompts,
    predictor=YOLOEVPSegPredictor,
)
results[0].show()

import numpy as np

from ultralytics import YOLOE
from ultralytics.models.yolo.yoloe import YOLOEVPSegPredictor

model = YOLOE("yoloe-26l-seg.pt")

visual_prompts = {"bboxes": np.array([[221.52, 405.8, 344.98, 857.54]]), "cls": np.array([0])}  # person

results = model.predict(
    "ultralytics/assets/zidane.jpg",  # Target image
    refer_image="ultralytics/assets/bus.jpg",  # Where the example boxes live
    visual_prompts=visual_prompts,
    predictor=YOLOEVPSegPredictor,
)
results[0].show()

# refer_image also sets the classes permanently, so later calls need no prompts at all
results = model("ultralytics/assets/bus.jpg")
model.export(format="onnx")  # And the export keeps them

import numpy as np
import torch

from ultralytics import YOLOE
from ultralytics.models.yolo.yoloe import YOLOEVPSegPredictor

model = YOLOE("yoloe-11l-seg.pt")

img_tensor = torch.rand(1, 3, 480, 480)  # (1, 3, H, W) float tensor in [0, 1]
visual_prompts = {"bboxes": np.array([[10, 10, 50, 50]]), "cls": np.array([0])}

results = model.predict(
    img_tensor,
    refer_image=img_tensor,
    visual_prompts=visual_prompts,
    predictor=YOLOEVPSegPredictor,
    imgsz=640,
)

import numpy as np

from ultralytics import YOLOE
from ultralytics.models.yolo.yoloe import YOLOEVPSegPredictor

model = YOLOE("yoloe-26l-seg.pt")

visual_prompts = {
    "bboxes": [
        np.array([[221.52, 405.8, 344.98, 857.54], [120, 425, 160, 445]]),  # bus.jpg: person, glasses
        np.array([[150, 200, 1150, 700]]),  # zidane.jpg: person
    ],
    "cls": [np.array([0, 1]), np.array([0])],
}

results = model.predict(
    ["ultralytics/assets/bus.jpg", "ultralytics/assets/zidane.jpg"],
    visual_prompts=visual_prompts,
    predictor=YOLOEVPSegPredictor,
)
results[0].show()

Val Usage#

Validation runs like any other model on a segmentation dataset:

from ultralytics import YOLOE

model = YOLOE("yoloe-26l-seg.pt")  # or yoloe-26s/m-seg.pt for other sizes

metrics = model.val(data="coco128-seg.yaml")

Two variants of the same call cover the other prompting modes:

• Visual prompts — model.val(data="coco128-seg.yaml", load_vp=True) extracts a visual embedding per category from the dataset itself. Add refer_data="coco.yaml" to take the embeddings from a different dataset, which must carry exactly the same categories.
• Prompt-free — load a *-seg-pf.pt checkpoint and pass single_cls=True.

Export Usage#

Prompt embeddings can be saved once and reused when producing static exports such as ONNX, OpenVINO, TensorRT, CoreML, LiteRT, and RKNN. The NPZ profile is loaded by the original PyTorch model before export; it is not an additional runtime input, and the exported model does not require the NPZ file.

from ultralytics import YOLOE

model = YOLOE("yoloe-26n-seg.pt")
model.set_classes(["person", "bus"])
model.save_prompt_embeddings("person-bus.npz")

# The profile is bound to the source checkpoint and can be reused for later exports.
model = YOLOE("yoloe-26n-seg.pt")
model.load_prompt_embeddings("person-bus.npz")
model.export(format="onnx")

The same prompt profile can also configure a detection-only model built from the matching YOLOE architecture. This removes the mask branch while retaining the prompted classes:

from ultralytics import YOLOE

model = YOLOE("yoloe-26n.yaml").load("yoloe-26n-seg.pt")
model.load_prompt_embeddings("person-bus.npz")
model.export(format="rknn", name="rk3588", quantize=16)

Track Usage#

Prompted classes carry straight into tracking, so you can follow objects the tracker was never trained on:

from ultralytics import YOLOE

model = YOLOE("yoloe-26s-seg.pt")
model.set_classes(["forklift", "pallet"])

# persist=True keeps track IDs stable across frames
for result in model.track("path/to/video.mp4", stream=True, persist=True):
    print(result.boxes.id)

How YOLOE Compares#

YOLOE sits between a closed-set detector and a heavyweight open-vocabulary model. Three comparisons decide whether it is the right choice:

• Against a closed-set YOLO. Once the prompts are set, YOLOE predicts through the ordinary detect/segment path and exports like any other model. What it adds is the ability to change the class list at inference time instead of retraining; what it costs is zero-shot accuracy well below a model trained on your own classes.
• Against the earlier YOLOE families. YOLOE-26 inherits YOLO26's NMS-free end-to-end head and covers five scales (n/s/m/l/x) against the earlier three (s/m/l), and leads at every matching scale in Performance.
• Against transformer-based open-vocabulary detectors. GLIP and OWL-ViT run a vision-language transformer at inference. YOLOE encodes the prompts once and then compares them against region features inside a convolutional head.

The nearest alternatives all take a text prompt, but only YOLOE and SAM 3 return masks, and the three answer different questions:

Coming from YOLO-World? The API is the same shape: swap YOLOWorld for YOLOE, load a *-seg.pt checkpoint, and keep your set_classes() call as it is. You gain masks and visual prompts; the export note about frozen classes applies to both.

Use Cases and Applications#

Open-vocabulary detection removes the retrain-per-class step, which matters most where the target list is not known up front:

• Open-world detection — robotics and security systems that meet objects nobody enumerated at training time.
• One-shot detection from an example — visual prompts pick up a specific part, logo or defect from a single reference box, useful in industrial inspection.
• Long-tail cataloging — the built-in 4,585-name vocabulary is broad enough for biodiversity monitoring or retail inventory sweeps.
• Dataset bootstrapping — pre-label images with boxes and masks before human review, then train a fast closed-set model on the result.
• Segmentation of arbitrary targets — the released *-seg.pt checkpoints return a mask with every prediction, so medical imaging and satellite analysis get pixel-precise output without a second model.

A common pattern combines two modes: run prompt-free once to discover what is present, then switch to text prompts for the categories that matter.

Limitations#

YOLOE trades accuracy for the ability to change classes at inference time. The consequences worth knowing before you commit:

• Zero-shot accuracy is well below a model trained on your classes. The prompted checkpoints land roughly in the 22-40 mAP band on LVIS minival; a closed-set YOLO trained on your own data will beat that on those classes. Reach for YOLOE to cover classes you cannot train for, not to replace training.
• Rare categories are the weak spot. The mAP_r column in Performance reports accuracy on LVIS's rare classes specifically, and under text prompting it sits below the common and frequent columns in every row. Check it rather than the headline mAP when your targets are unusual.
• A prompt describes appearance, not relationships. Detection works by comparing region features against the prompt embedding, so prompts that depend on state, context or comparison — "damaged", "left-most", "the one being carried" — have no reliable handle to match on. Prefer wording close to everyday category names.
• Large prompt sets cost latency. The prompt embeddings are computed once, but they are compared against region features on every forward pass. Measured on CPU with yoloe-26s-seg.pt, a forward pass grows about 19% going from 80 to 1,203 classes and about 89% at the full 4,585-name vocabulary. Reported FLOPs do not move at all, because the region-text similarity is not counted, so the profile will not warn you.
• Class names are placeholders until you prompt. A freshly loaded *-seg.pt checkpoint reports nc=80 with numeric names ("0", "1", …), so call set_classes() before reading labels. Prompt-free checkpoints ship the full vocabulary already populated.

Deployment Notes#

• Hardware. Inference needs an NVIDIA GPU with 4-8 GB of VRAM; the n and s scales run on edge GPUs such as Jetson or on CPU at reduced resolution. Fine-tuning needs a single GPU.
• NMS is class-agnostic by default. YOLOE predicts with agnostic_nms=True. On YOLOE-11 and YOLOE-v8 this suppresses lower-scoring overlapping boxes across different classes rather than only within the same class, which prevents duplicates when one object matches several categories. End-to-end YOLOE-26 models apply no IoU suppression at all; there, agnostic mode only keeps the single best class per anchor instead of letting one anchor emit several class labels. Pass agnostic_nms=False to override.
• Batching. Batch inference works directly, and visual prompts can differ per image in the same call.

Training the Official Models from Scratch#

Most readers never need this. It reproduces the published open-vocabulary checkpoints from Objects365, GQA and Flickr30k — about 1.4 M training samples on 8× RTX 4090 — and is unrelated to fine-tuning on your own data, which is covered under Train Usage above.

Training needs segment annotations. Either download the processed files below, or generate your own with the script provided by the official team, which is powered by SAM 2.1. Validation uses LVIS minival.

The text-prompt model is trained first, and the other two prompting modes are refinements of it:

from ultralytics import YOLOE
from ultralytics.models.yolo.yoloe import YOLOESegTrainerFromScratch

data = {
    "train": {
        "yolo_data": ["Objects365.yaml"],
        "grounding_data": [
            {
                "img_path": "flickr/full_images/",
                "json_file": "flickr/annotations/final_flickr_separateGT_train_segm.json",
            },
            {
                "img_path": "mixed_grounding/gqa/images",
                "json_file": "mixed_grounding/annotations/final_mixed_train_no_coco_segm.json",
            },
        ],
    },
    "val": {"yolo_data": ["lvis.yaml"]},
}

model = YOLOE("yoloe-26l-seg.yaml")
model.train(
    data=data,  # or the path to a YAML file holding the same structure
    batch=128,
    epochs=30,
    close_mosaic=2,
    optimizer="AdamW",
    lr0=2e-3,
    warmup_bias_lr=0.0,
    weight_decay=0.025,
    momentum=0.9,
    workers=4,
    trainer=YOLOESegTrainerFromScratch,
    device="0,1,2,3,4,5,6,7",
)

The visual-prompt and prompt-free checkpoints start from that trained text-prompt model and update one module each. YOLOESegVPTrainer is the visual-prompt recipe and YOLOEPEFreeTrainer the prompt-free one, but neither class freezes anything by itself: the selective training comes from the freeze list you pass alongside it, which names every head child except savpe (resp. every classification tower), and the prompt-free run additionally needs single_cls=True. The upstream YOLOE repository carries the full recipes for the v8-scale models.

A finished prompt-free run is re-parameterized into a checkpoint that reports its names with no prompt at inference, using get_vocab and set_vocab:

from ultralytics import YOLOE

# Weights written by the prompt-free run and by the text-prompt run it started from. Each
# rerun creates a new directory (train-2, train-3, ...), so take the paths the runs printed.
model = YOLOE("runs/segment/train-2/weights/best.pt")  # prompt-free run, its head is already fused
text_model = YOLOE("runs/segment/train/weights/best.pt")  # text-prompt run, its head is still unfused

names = list(YOLOE("yoloe-26l-seg-pf.pt").model.names.values())  # the 4,585-name vocabulary, or your own list
vocab = text_model.get_vocab(names)

model.set_vocab(vocab, names)
model.save("yoloe-26l-seg-pf-custom.pt")  # never overwrite the released checkpoint

Citations and Acknowledgments#

If YOLOE has contributed to your research or project, please cite the original paper by Ao Wang, Lihao Liu, Hui Chen, Zijia Lin, Jungong Han, and Guiguang Ding from Tsinghua University:

@misc{wang2025yoloerealtimeseeing,
      title={YOLOE: Real-Time Seeing Anything},
      author={Ao Wang and Lihao Liu and Hui Chen and Zijia Lin and Jungong Han and Guiguang Ding},
      year={2025},
      eprint={2503.07465},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2503.07465},
}

For further reading, the original YOLOE paper is available on arXiv. The project's source code and additional resources can be accessed via their GitHub repository.

FAQ#