Get Started

NVIDIA TensorRT Export for YOLO26 Models

TensorRT is NVIDIA's SDK for high-speed deep learning inference. Exporting an Ultralytics YOLO26 model with format="engine" builds a .engine file that is optimized for the NVIDIA GPU it is built on, in FP32, FP16, or INT8 precision.

TensorRT engines are hardware- and runtime-specific

TensorRT tunes an engine for the GPU it is built on. Build the engine on the deployment GPU, or on one of the same model, with the same TensorRT version; an .engine file is not a portable model format. For edge deployment, Ultralytics Platform offers eight Jetson target selections, with physical build and validation status documented for each, or you can export locally on the destination device.

At a Glance#

PropertyDetails
format argumentengine (aliases: tensorrt, trt)
Outputyolo26n.engine, one file that starts with a JSON metadata header (class names, task, image size). The intermediate yolo26n.onnx stays next to it
Runs onNVIDIA GPUs: data center, workstation, and Jetson, including Jetson DLA cores (not on TensorRT 11)
RuntimeTensorRT 7.0 or newer (10.2.0 is not supported)
PrecisionFP32, FP16, INT8
Export hostA machine with an NVIDIA GPU and CUDA. Build on the deployment GPU or one of the same model
Export in the cloud

Ultralytics Platform can build a TensorRT engine for you on the GPU type you select, including NVIDIA Jetson targets, so you do not need a local export environment.

Quickstart#

Install the Ultralytics package on a machine with an NVIDIA GPU and CUDA. On Linux, the first export installs the TensorRT Python package that matches your CUDA version if it is missing. On NVIDIA Jetson, use the TensorRT version that ships with JetPack. On Windows, install TensorRT yourself before you export.

pip install ultralytics

For other installation methods, see the installation guide. Then export your model and load the exported engine to run inference or validate its accuracy. Inference and validation require an NVIDIA GPU. Building a YOLO26n engine takes one to three minutes on an NVIDIA T4.

Export
from ultralytics import YOLO

# Load a YOLO26 model
model = YOLO("yolo26n.pt")

# Export the model to TensorRT format
model.export(format="engine")  # creates 'yolo26n.engine'

# Or export an FP16 engine instead
model.export(format="engine", quantize=16)  # creates 'yolo26n.engine'
Predict
from ultralytics import YOLO

# Load the exported TensorRT model
model = YOLO("yolo26n.engine")

# Run inference
results = model("https://ultralytics.com/images/bus.jpg")
Validate
from ultralytics import YOLO

# Load the exported TensorRT model
model = YOLO("yolo26n.engine")

# Validate accuracy on the COCO8 dataset
metrics = model.val(data="coco8.yaml")

Supported Tasks#

TensorRT export supports all seven Ultralytics tasks with YOLOv8, YOLO11, and YOLO26 models: detection, segmentation, semantic segmentation, depth estimation, classification, pose estimation, and OBB. Semantic segmentation and depth estimation are available only with YOLO26, the only family that ships those heads.

Export Arguments#

ArgumentTypeDefaultDescription
formatstr'engine'Target format for the exported model, defining compatibility with various deployment environments.
imgszint or tuple640Desired image size for the model input. Can be an integer for square images or a tuple (height, width) for specific dimensions.
quantizeint or strNoneQuantization precision: 16 (FP16) or 8 (INT8/PTQ; needs calibration data/fraction); 32/unset is FP32. A checkpoint trained with quantize=8 exports INT8 from the ranges it carries, without calibration, also when quantize is unset; quantize=16 or quantize=32 raises an error. Replaces the deprecated half/int8 flags.
dynamicboolFalseBuilds an engine that accepts batch sizes from 1 to batch and image sizes from 32 pixels to twice imgsz; the minimum is higher for dynamic INT8 engines on older GPUs. When False, the engine accepts only the export batch and imgsz.
simplifyboolTrueSimplifies the model graph with onnxslim, potentially improving performance and compatibility.
opsetintNoneSpecifies the ONNX opset version for the intermediate ONNX graph. If not set, uses opset 18 (lower on PyTorch < 2.4), capped at the installed onnx version.
workspacefloat or NoneNoneUpper limit in GiB for the memory TensorRT uses while it builds the engine. Leave it at None to let TensorRT allocate memory automatically.
nmsbool, optionalNoneSelect raw output (None, default), embedded NMS (True), or the NMS-free head (False). With nms=True, conf (default 0.25), iou, max_det, and agnostic_nms are fixed at export; nms=True is ignored for semantic and depth models and raises an error for classify models.
batchint1Batch size of the engine, or the maximum batch size with dynamic=True. Also the batch size used for INT8 calibration.
datastrNonePath to the dataset YAML, used for INT8 calibration; classification instead takes a dataset directory or a built-in dataset name. Calibration uses the val split unless split is set. If omitted with quantize=8, Ultralytics uses the small default dataset for the model task, such as coco8.yaml for detection; a checkpoint trained with quantize=8 needs none.
fractionfloat, int, or list1.0Calibration subset as a ratio, image count, or [train, val, test] ratios/counts. Two-item lists leave test full, while 0 skips it.
devicestrNoneSpecifies the device for exporting: GPU (device=0, used when unset), DLA for NVIDIA Jetson (device=dla:0 or device=dla:1, with quantize=16 or quantize=8).
JetPack and DLA compatibility

On NVIDIA Jetson, Ultralytics exports with the TensorRT runtime of your JetPack release. On JetPack 7 the export requires TensorRT 10.15 or newer and below 11. device=dla:0 and device=dla:1 raise an error on TensorRT 11. NVIDIA lists TensorRT 10.7 as the last 10.x release with DLA support, and TensorRT 11.4.1 adds JetPack 7.2.2 support, which Ultralytics does not use yet.

For the arguments shared by all formats, see the Export mode documentation.

Exporting TensorRT with INT8 Quantization#

quantize=8 builds an INT8 engine with post-training quantization (PTQ): the export runs calibration images through the model to measure the value range of each tensor. A checkpoint fine-tuned with quantize=8 through quantization-aware training already carries its INT8 ranges, so its export skips calibration.

The quantize and data arguments are the same on every TensorRT version, but the export of a checkpoint not trained with quantize=8 works differently underneath:

TensorRT 7-10TensorRT 11
INT8 methodImplicit quantization with the IInt8Calibrator interfaceExplicit quantization: NVIDIA ModelOpt inserts Q/DQ nodes into the ONNX graph
Calibration algorithmMINMAX_CALIBRATION on GPU, ENTROPY_CALIBRATION_2 for DLAModelOpt max, run on the CPU
Calibration images usedEvery full batch of the splitAbout 512 images in shuffled order, or every full batch of a smaller split
dynamic=True with quantize=8Not covered by tests; export without dynamic=TrueSupported
FP16 (quantize=16)TensorRT builder flagModelOpt AutoCast mixed-precision conversion

ModelOpt is installed on first use. Ultralytics does not reuse a calibration cache, so every export calibrates again. For details on both methods, see Working with Quantized Types in the TensorRT documentation.

For a checkpoint not trained with quantize=8, pass calibration images with data, for example data="my_dataset.yaml". Calibration uses the val split of that dataset unless you set split, and fraction limits the number of images. NVIDIA recommends at least 500 images that represent your deployment data, and the export logs a warning below 300. Without data, the export logs a warning and uses the small example dataset for the model task, such as coco8.yaml for detection. Its validation split has 4 images, which is enough to test the export but not to calibrate a model for deployment.

Example
from ultralytics import YOLO

model = YOLO("yolo26n.pt")

# Calibrate on 10% of the COCO validation images (500 images); replace coco.yaml with your own dataset
model.export(format="engine", quantize=8, data="coco.yaml", fraction=0.1)

# Load the exported INT8 engine and run inference
model = YOLO("yolo26n.engine")
result = model.predict("https://ultralytics.com/images/bus.jpg")

On the benchmark GPU, INT8 engines ran 1.04 to 2.3 times faster than FP32 engines, but faster than FP16 engines for only three of ten models. INT8 lost 1.7 to 2.9 mAP on YOLO26n to YOLO26l and far more on YOLO26x and YOLO26n-pose. The YOLO26n INT8 engine was larger than its FP32 engine (30 MB against 11 MB), while large models shrank (53 MB against 103 MB for YOLO26l). INT8 can also shift confidence values, so choose the confidence threshold from the INT8 engine's own F1 curve. See the benchmarks for every model.

Output Format#

The engine takes one input named images and returns the same outputs as the ONNX export it is built from, with one exception: on TensorRT versions earlier than 10, semantic models return float class scores with shape (batch, classes, height, width) in place of the uint8 class map. See the ONNX output format for the output shape of each task and each nms setting.

Load the Engine Outside Ultralytics#

An Ultralytics .engine file is not a plain TensorRT engine. It starts with a 4-byte little-endian length, followed by that many bytes of JSON metadata, followed by the serialized engine. TensorRT's deserialize_cuda_engine returns None when it is given the whole file. Skip the header to load the engine with the TensorRT API:

import json

import tensorrt as trt

with open("yolo26n.engine", "rb") as f:
    n = int.from_bytes(f.read(4), byteorder="little")  # length of the metadata header
    metadata = json.loads(f.read(n))  # class names, task, imgsz, batch, export arguments
    engine = trt.Runtime(trt.Logger(trt.Logger.WARNING)).deserialize_cuda_engine(f.read())

When another tool expects a plain TensorRT engine file, write the bytes after the header to a new file, or build the engine in that tool from the yolo26n.onnx file that the export leaves next to the engine. On TensorRT 11, FP16 and calibrated INT8 exports also leave yolo26n.fp16.onnx and yolo26n.int8.onnx; build an INT8 engine in another tool from the .int8.onnx file. For a checkpoint trained with quantize=8, yolo26n.onnx already holds the Q/DQ nodes, and no .int8.onnx file is written; on TensorRT 11 that export leaves yolo26n.fp16.onnx, the same Q/DQ graph with its other layers in FP16.

Deploy#

YOLO("yolo26n.engine") loads the engine with the TensorRT Python runtime. The engine runs on GPU 0 by default, also when you pass device="cpu". Pass device=1 to run it on another GPU of the same model. An engine exported with device=dla:0 or device=dla:1 runs on that DLA core.

An engine exported without dynamic=True has a fixed input shape. Predict and val use the export imgsz and ignore a different imgsz argument. Ultralytics splits larger batches into batches of the export size and pads smaller ones, so any number of images works. An engine exported with dynamic=True accepts every batch size and image size inside its range and raises ValueError: input size ... is outside the TensorRT optimization profile for a shape outside it. With nms=True, a batch above the range is split into batches of the export size instead.

Deployment Guides#

Limitations and Troubleshooting#

  • Engines are not portable: an engine is built for one GPU model and one TensorRT version. Rebuild it when you change the GPU, the TensorRT version, or the JetPack release. When an engine cannot be loaded, Ultralytics logs TensorRT model exported with a different version than expected.
  • GPU required: the export uses device=0 when device is unset and fails with export running on CPU but must be on GPU for device=cpu.
  • Engine does not load in other tools: the .engine file starts with a metadata header, so the TensorRT API cannot read it directly. See Load the Engine Outside Ultralytics.
  • TensorRT version: Ultralytics requires TensorRT 7.0.0 or newer. TensorRT 10.2.0 is not supported, and the automatic install skips it. On JetPack 7 and NVIDIA DGX systems, the export requires TensorRT 10.15 or newer (and below 11 on Jetson) and tries to install it when the installed version is outside that range.
  • NMS-free head on old TensorRT: TensorRT versions earlier than 8.5.0 cannot build the YOLO26 NMS-free head. The export logs a warning and writes raw predictions that need NMS. Upgrade TensorRT to 8.5.0 or newer to export with nms=False.
  • INT8 on JetPack 6: with TensorRT 10.3.0 on JetPack 6, an INT8 export with nms=True fails, and an INT8 export with nms=False falls back to raw predictions. See the Jetson guide for the fix.
  • INT8 on TensorRT 10.16: INT8 export with TensorRT 10.16.1 failed on NVIDIA RTX PRO 6000 Blackwell and RTX 6000D GPUs with Could not find any implementation for node. On the RTX PRO 6000 Blackwell, TensorRT 10.13, 10.14, 10.15, and 11.4 built the INT8 engine.
  • DLA: device=dla:0 and device=dla:1 work only on NVIDIA Jetson, and only with quantize=16 or quantize=8. The export raises an error on TensorRT 11. Layers that DLA does not support run on the GPU.
  • Dynamic shapes: dynamic=True needs a maximum batch size, so also set batch, for example batch=8. Without it the export logs a warning and the maximum batch size is 1. The engine accepts batches from 1 to batch and image sizes from 32 pixels up to twice the export imgsz. For example, an engine exported with imgsz=320 batch=4 runs at 640 pixels and rejects 704 pixels or a batch of 6 with input size ... is outside the TensorRT optimization profile. Combine dynamic=True with quantize=8 only on TensorRT 11. On GPUs with compute capability 7.5 or lower, such as the T4, the minimum size of those dynamic INT8 engines is 64 pixels (128 for P6 models).
  • Fixed input size: an engine exported without dynamic=True always runs at its export imgsz. Passing another imgsz to predict or val has no effect, so re-export to change the input size.
  • INT8 export without data: unless the checkpoint was trained with quantize=8, the export calibrates on the small default dataset for the task, which has 4 validation images for detection, and logs a warning. Pass data with representative images; the export warns below 300.
  • Slow first inferences: the first few calls with a new engine, or with a new imgsz on a dynamic engine, take longer than later calls.

For other errors, see the Common Issues guide.

Benchmarks#

The YOLO26 model page lists the TensorRT 10 speed of every model on an NVIDIA T4 GPU for all tasks except semantic segmentation.

FP32, FP16, and INT8 Compared#

The tables below compare PyTorch with FP32, FP16, and INT8 TensorRT 11.4 engines of every YOLO26 detection model, and of the YOLO26n model for five other tasks. All values were measured on one NVIDIA RTX PRO 6000 Blackwell Server Edition GPU.

Milliseconds per image, lower is better. The fastest format for each model is in bold; the YOLO26x INT8 engine is excluded because its accuracy was unstable.

ModelimgszPyTorchTensorRT FP32TensorRT FP16TensorRT INT8
YOLO26n6402.850.670.480.53
YOLO26s6402.981.150.610.66
YOLO26m6403.292.221.031.08
YOLO26l6404.703.031.491.43
YOLO26x6405.934.312.091.84
YOLO26n-seg6403.180.930.650.72
YOLO26n-sem10241.740.740.450.71
YOLO26n-cls2241.750.290.240.20
YOLO26n-pose6403.390.760.510.59
YOLO26n-obb10244.551.030.640.88
  • FP16 kept the FP32 accuracy within 0.1 points on every model. On the detection models it ran 1.4 to 2.2 times faster than the FP32 engine and 2.8 to 5.9 times faster than PyTorch.
  • INT8 ran within 12% of the FP16 latency on the detection models and lost 1.7 to 2.9 mAP on YOLO26n, YOLO26s, YOLO26m, and YOLO26l. On this GPU it was faster than FP16 only for YOLO26l, YOLO26x, and YOLO26n-cls.
  • YOLO26x INT8 accuracy fell to between 25.3 and 38.5 mAP across three builds, against 57.5 in FP32. YOLO26n-pose INT8 fell from 56.7 to 33.4 pose mAP. Validate every INT8 engine on your own data before you deploy it.
  • TensorRT 10.16 FP32 and FP16 engines of YOLO26n and YOLO26s matched the TensorRT 11.4 accuracy. Their FP16 engines were about 7% faster and their FP32 engines 4% to 9% slower than on TensorRT 11.4.

For NVIDIA Jetson and DGX Spark, see the Jetson benchmarks and the DGX Spark guide.

Measured with ultralytics 8.4.211, TensorRT 11.4.0, PyTorch 2.15.0, and CUDA 13.2 at batch=1 on November 13, 2026. Latency is the inference stage that val reports and does not include preprocessing or postprocessing.

About TensorRT#

TensorRT builds an engine by fusing layers, selecting the fastest kernel for each layer on the build GPU, and optionally lowering precision to FP16 or INT8. For the full reference, see the TensorRT documentation.

NVIDIA TensorRT connecting deep learning frameworks to NVIDIA data center GPUs, Jetson, and DLA hardware

FAQ#

  • Only on a GPU of the same model with the same TensorRT version. An engine is built for one GPU model and one TensorRT version. To share one file across different hardware, export to ONNX and build the engine on each device. For NVIDIA Jetson, Ultralytics Platform can build engines for eight Jetson targets.

  • Start with FP16 (quantize=16). It needs no calibration data and keeps FP32 accuracy. In the benchmarks on an NVIDIA RTX PRO 6000 Blackwell with TensorRT 11.4, FP16 reduced YOLO26s latency from 1.15 ms to 0.61 ms per image at the same 48.5 mAPval 50-95. INT8 (quantize=8) ran at 0.66 ms and 46.3 mAPval 50-95, so it was not faster than FP16 on that GPU, and it needs representative calibration data unless the checkpoint was trained with quantize=8. INT8 can pay off on other hardware: the Jetson benchmarks list FP16 and INT8 results for embedded devices. Measure both precisions on your deployment device before you choose INT8.

  • Export with dynamic=True and set batch to the largest batch you plan to run. The engine then accepts batches from 1 to batch and image sizes from 32 pixels up to twice the export imgsz, with a higher minimum for dynamic INT8 engines on older GPUs. Without dynamic=True the engine has one fixed input shape, and Ultralytics splits or pads each batch to fit it.

    Dynamic engine
    from ultralytics import YOLO
    
    model = YOLO("yolo26n.pt")
    model.export(format="engine", dynamic=True, batch=8, quantize=16)
    
    model = YOLO("yolo26n.engine")
    results = model(["https://ultralytics.com/images/bus.jpg"] * 4, batch=4)  # a batch of 4 images
  • Run the export on the Jetson itself with the TensorRT version from your JetPack release. To use a DLA core, add device=dla:0 with quantize=16 or quantize=8. See the NVIDIA Jetson guide for setup and benchmarks.

    Export on Jetson
    from ultralytics import YOLO
    
    model = YOLO("yolo26n.pt")
    model.export(format="engine", quantize=16)  # FP16 engine on the Jetson GPU
    
    # Or build the FP16 engine for DLA core 0 instead
    model.export(format="engine", quantize=16, device="dla:0")
  • The Triton Inference Server guide exports the model to ONNX and enables Triton's TensorRT accelerator in config.pbtxt. Triton then builds the TensorRT engine on the server GPU at first load and caches it, so the engine always matches the server's GPU and TensorRT version. The guide covers the model repository layout, the server container, and client inference with Ultralytics.

Comments116

Ask a question, share your results, or help someone else.

  1. AgbajeAyomipoGitHub

    Hi Ultralytics team, I’ve trained a YOLOv11 model on (1024x1024) images from a camera feed and now I’m trying to run real-time inference on live streams using a Jetson Orin Nano. I want to maintain full resolution for inference because of accuracy requirements, but currently I'm getting only 1-2 FPS. When I reduce the input size, the FPS improves significantly, but it's not acceptable due to missed detections at lower resolutions. I attempted to export the model to TensorRT using:

    model.export(format="engine", imgsz=(2592, 1944), half=True) but it fails silently. If I use a dynamic=true without specifying the imgsz parameter, the export succeeds and uses (1024, 1024). My questions:

    Is TensorRT export for such high-resolution input supported/recommended with YOLOv11? If not, is there any known limitation or workaround? Are there ways to improve performance with high-res inference on Jetson Orin Nano? What hardware would you recommend for real-time inference on 1-3 camera streams with 5MP inputs, assuming we need at least ~15-20 FPS per stream? Device: AAEON BOXER-8621AI Module: NVIDIA® Jetson Orin Nano™ (4GB LPDDR5, Super Mode Enabled) Memory: 4 GB LPDDR5 I/O: USB 3.2 Gen 2, M.2 Expansion Operating System: JetPack 5.1.4 (L4T 35.6.0, Ubuntu 20.04) Use Case: Real-time multi-stream 5MP video inference

    1
  2. AgbajeAyomipoGitHub

    Hi Ultralytics team, I’ve trained a YOLOv11 model on full-size 5MP (2592x1944) images from a camera feed and now I’m trying to run real-time inference on live streams using a Jetson Orin Nano. I want to maintain full resolution for inference because of accuracy requirements, but currently I'm getting only 1-2 FPS. When I reduce the input size, the FPS improves significantly, but it's not acceptable due to missed detections at lower resolutions. I attempted to export the model to TensorRT using:

    model.export(format="engine", imgsz=(2592, 1944), half=True) but it fails silently. If I use a dynamic=true, the export succeeds and uses (1024, 1024). My questions:

    1. Is TensorRT export for such high-resolution input supported/recommended with YOLOv11? If not, is there any known limitation or workaround?
    2. Are there ways to improve performance with high-res inference on Jetson Orin Nano?
    3. What hardware would you recommend for real-time inference on 1-3 camera streams with 5MP inputs, assuming we need at least ~15-20 FPS per stream? Device: AAEON BOXER-8621AI Module: NVIDIA® Jetson Orin Nano™ (4GB LPDDR5, Super Mode Enabled) Memory: 4 GB LPDDR5 I/O: USB 3.2 Gen 2, M.2 Expansion Operating System: JetPack 5.1.4 (L4T 35.6.0, Ubuntu 20.04) Use Case: Real-time multi-stream 5MP video inference
    1
  3. 020423lzlGitHub

    m的模型大小比n大的多,速度也慢了很多,输入图像尺寸越大,越吃显存,速度也越慢

    李子乐控制工程 @.***

    ---Original--- From: @.> Date: Sat, May 9, 2026 10:30 AM To: @.>; Cc: @.@.>; Subject: Re: [ultralytics/ultralytics] zh/integrations/tensorrt/ (Discussion #9868)

    现在有一个奇怪的现象,我导出的tensorrt模型的推理速度几乎和原来的.pt几乎没有差别甚至更慢,并且导出的tensorrt模型的推理结果和.pt模型的差别很大。 我的设备是i914900k+64G RAM+RTX A6000 我的环境是cuda12.4.131+tensorrt10.10.0.31+torch2.5.1+ultralytics8.3.252 导出命令: yolo export model=yolo11m.pt imgsz=1280 format='engine 推理代码为: model.predict(stream_path, imgsz=1280, classes=[14], conf=0.2, iou=0.5, stream=True, task='detect')

    另外我试了下yolo11n.pt的640输入尺寸导出,推理速度就有明显提升,当然推理结果差别仍然较大。 请问这是什么道理,有什么解决办法?

    — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you commented.Message ID: @.***>

    1. User

      能不能读完再说话,肯定是m和m比,n和n比,同样尺寸输入和同样尺寸输入的比啊。 该显卡的RT core算力是普通fp32算力的近两倍。

    2. Glenn JocherUltralytics

      是的,性能应当在同一模型、同一 imgsz、同一 batch、同一后处理参数下比较;m 比 n 更大、1280 比 640 更吃显存且更慢是正常现象。对 RTX A6000 来说,FP32 的 TensorRT 往往不会比 .pt 明显更快,建议先按 TensorRT 导出文档 用 half=True 重新导出,并在相同输入上预热后再统计平均耗时;如果 engine 与 .pt 结果仍明显不一致,请贴出 !yolo checks、完整控制台日志文本和一个最小可复现示例,我们好继续排查环境或后处理差异。

  4. User

    现在有一个奇怪的现象,我导出的tensorrt模型的推理速度几乎和原来的.pt几乎没有差别甚至更慢,并且导出的tensorrt模型的推理结果和.pt模型的差别很大。 我的设备是i914900k+64G RAM+RTX A6000 我的环境是cuda12.4.131+tensorrt10.10.0.31+torch2.5.1+ultralytics8.3.252 导出命令: yolo export model=yolo11m.pt imgsz=1280 format='engine 推理代码为:

    model.predict(stream_path, imgsz=1280, classes=[14], conf=0.2, iou=0.5, stream=True, task='detect')

    另外我试了下yolo11n.pt的640输入尺寸导出,推理速度就有明显提升,当然推理结果差别仍然较大。 请问这是什么道理,有什么解决办法?

  5. 020423lzlGitHub

    etson@yahboom:~/ultralytics/ultralytics$ yolo predict model=best.engine source=0 save=False show WARNING ⚠️ Unable to automatically guess model task, assuming 'task=detect'. Explicitly define task for your model, i.e. 'task=detect', 'segment', 'classify','pose' or 'obb'. Ultralytics 8.3.172 🚀 Python-3.10.12 torch-2.5.0a0+872d972e41.nv24.08 CUDA:0 (Orin, 7620MiB) Loading best.engine for TensorRT inference... [12/18/2025-17:11:05] [TRT] [I] Loaded engine size: 75 MiB [12/18/2025-17:11:05] [TRT] [I] [MS] Running engine with multi stream info [12/18/2025-17:11:05] [TRT] [I] [MS] Number of aux streams is 7 [12/18/2025-17:11:05] [TRT] [I] [MS] Number of total worker streams is 8 [12/18/2025-17:11:05] [TRT] [I] [MS] The main stream provided by execute/enqueue calls is the first worker stream [12/18/2025-17:11:05] [TRT] [I] [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +1, GPU +88, now: CPU 1, GPU 158 (MiB)

    [ WARN:0@2.771] global cap_gstreamer.cpp:1777 open OpenCV | GStreamer warning: Cannot query video position: status=0, value=-1, duration=-1 1/1: 0... Success ✅ (inf frames of shape 1920x1080 at 5.00 FPS)

    Traceback (most recent call last): File "/home/jetson/.local/bin/yolo", line 7, in sys.exit(entrypoint()) File "/home/jetson/Downloads/ultralytics-main/ultralytics/cfg/init.py", line 983, in entrypoint getattr(model, mode)(**overrides) # default args from model File "/home/jetson/Downloads/ultralytics-main/ultralytics/engine/model.py", line 555, in predict return self.predictor.predict_cli(source=source) if is_cli else self.predictor(source=source, stream=stream) File "/home/jetson/Downloads/ultralytics-main/ultralytics/engine/predictor.py", line 247, in predict_cli for _ in gen: # sourcery skip: remove-empty-nested-block, noqa File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 36, in generator_context response = gen.send(None) File "/home/jetson/Downloads/ultralytics-main/ultralytics/engine/predictor.py", line 351, in stream_inference s[i] += self.write_results(i, Path(paths[i]), im, s) File "/home/jetson/Downloads/ultralytics-main/ultralytics/engine/predictor.py", line 434, in write_results string += f"{result.verbose()}{result.speed['inference']:.1f}ms" File "/home/jetson/Downloads/ultralytics-main/ultralytics/engine/results.py", line 694, in verbose return "".join(f"{n} {self.names[i]}{'s' * (n > 1)}, " for i, n in enumerate(counts) if n > 0) File "/home/jetson/Downloads/ultralytics-main/ultralytics/engine/results.py", line 694, in return "".join(f"{n} {self.names[i]}{'s' * (n > 1)}, " for i, n in enumerate(counts) if n > 0) KeyError: 171 terminate called without an active exception Aborted (core dumped) jetson@yahboom:~/ultralytics/ultralytics$ How to solve it

    1. Glenn JocherUltralytics

      KeyError: 171 表示模型输出了类别 171,但当前 names 里没有这个索引(通常是自定义数据集的标签类别编号不在 0..nc-1 范围内、data.yaml 里的 names 数量与 nc 不匹配,或导出/推理用的模型与数据配置不一致导致的元数据错配);请先把所有标注 .txt 的 class id 修正为连续的 0..nc-1,同步修正 data.yaml 的 nc/names,修正后删除数据集 labels/ 下的 *.cache,再用同一套环境从 best.pt 重新导出 .engine(参考 TensorRT 导出文档),并在推理时显式加上 task=detect。另外请把下面命令的完整输出(文本)贴在这个 discussion 里,以及你当时的导出命令,方便我们确认是否有多环境/路径冲突:

      yolo checks
    2. 020423lzlGitHub

      Thank you for your reply. The issue has been resolved. I was using the RT-DETR model, but I mistakenly used "from ultralytics import YOLO" when exporting

    3. Glenn JocherUltralytics

      Glad you found it — RT-DETR models should be loaded/exported with RTDETR(...) (mixing them with YOLO(...) can produce mismatched metadata like names), see the RT-DETR model docs for details:

      from ultralytics import RTDETR
      
      model = RTDETR("best.pt")
      model.export(format="engine")  # creates best.engine
  6. fly2skyToEnjoyGitHub

    May I ask : The following command exports the file, whether the box and mask are both nmsed at the same time ???

    from ultralytics import YOLO

    Load the YOLO11 model

    model = YOLO("yolo11n.pt")

    Export the model to TensorRT format

    model.export(format="engine") # creates 'yolo11n.engine'

    Load the exported TensorRT model

    tensorrt_model = YOLO("yolo11n.engine")

    Run inference

    results = tensorrt_model("https://ultralytics.com/images/bus.jpg")

  7. dilanpatel195GitHub

    I am trying to prediction model for the YOLOv11n-seg, I have exported the .py to .engine to back it run faster but keep getting a KEY error message:


    KeyError Traceback (most recent call last) Cell In[4], line 109 106 # Inside the main loop after grabbing a frame 107 if inference_active and not inpainting_active: 108 # Perform YOLO inference --> 109 results = model(rgb_frame)[0] 110 masked_image = rgb_frame.copy() # Make a copy for masking 112 # Process YOLO masks (original functionality remains unchanged)

    File c:\ProgramData\miniconda3\Lib\site-packages\ultralytics\engine\model.py:176, in Model.call(self, source, stream, **kwargs) 147 def call( 148 self, 149 source: Union[str, Path, int, Image.Image, list, tuple, np.ndarray, torch.Tensor] = None, 150 stream: bool = False, 151 **kwargs, 152 ) -> list: 153 """ 154 Alias for the predict method, enabling the model instance to be callable for predictions. 155 (...) 174 ... print(f"Detected {len(r)} objects in image") 175 """ --> 176 return self.predict(source, stream, **kwargs)

    File c:\ProgramData\miniconda3\Lib\site-packages\ultralytics\engine\model.py:554, in Model.predict(self, source, stream, predictor, **kwargs) 552 if prompts and hasattr(self.predictor, "set_prompts"): # for SAM-type models 553 self.predictor.set_prompts(prompts) --> 554 return self.predictor.predict_cli(source=source) if is_cli else self.predictor(source=source, stream=stream)

    File c:\ProgramData\miniconda3\Lib\site-packages\ultralytics\engine\predictor.py:168, in BasePredictor.call(self, source, model, stream, *args, **kwargs) 166 return self.stream_inference(source, model, *args, **kwargs) 167 else: --> 168 return list(self.stream_inference(source, model, *args, **kwargs))

    File c:\ProgramData\miniconda3\Lib\site-packages\torch\utils_contextlib.py:36, in _wrap_generator..generator_context(*args, **kwargs) 33 try: 34 # Issuing None to a generator fires it up 35 with ctx_factory(): ---> 36 response = gen.send(None) 38 while True: 39 try: 40 # Forward the response to our caller and get its next request

    File c:\ProgramData\miniconda3\Lib\site-packages\ultralytics\engine\predictor.py:274, in BasePredictor.stream_inference(self, source, model, *args, **kwargs) 268 self.results[i].speed = { 269 "preprocess": profilers[0].dt * 1e3 / n, 270 "inference": profilers[1].dt * 1e3 / n, 271 "postprocess": profilers[2].dt * 1e3 / n, 272 } 273 if self.args.verbose or self.args.save or self.args.save_txt or self.args.show: --> 274 s[i] += self.write_results(i, Path(paths[i]), im, s) 276 # Print batch results 277 if self.args.verbose:

    File c:\ProgramData\miniconda3\Lib\site-packages\ultralytics\engine\predictor.py:334, in BasePredictor.write_results(self, i, p, im, s) 332 result = self.results[i] 333 result.save_dir = self.save_dir.str() # used in other locations --> 334 string += f"{result.verbose()}{result.speed['inference']:.1f}ms" 336 # Add predictions to image 337 if self.args.save or self.args.show:

    File c:\ProgramData\miniconda3\Lib\site-packages\ultralytics\engine\results.py:663, in Results.verbose(self) 661 for c in boxes.cls.unique(): 662 n = (boxes.cls == c).sum() # detections per class --> 663 log_string += f"{n} {self.names[int(c)]}{'s' * (n > 1)}, " 664 return log_string

    KeyError: 2

    Any help would be greatly appreciated!

  8. Ranjith3436GitHub

    Now, My TensorRT Sengment model used for track. if any detections not there the processing FPS is 75-80 in jetson. But once the person or people detected suddenly reused to 15-20. processing time going to 40ms. only when the object detected. What's the issue in model.track(), could i run all the tracking thing on GPU.

    1. Glenn JocherUltralytics(edited)

      @Ranjith3436 the performance drop when objects are detected is expected behavior, as tracking introduces additional computational overhead. When objects are detected, the model.track() method performs several CPU-bound operations including feature extraction, association, and trajectory management.

      To improve performance:

      1. Try using a smaller model variant (like YOLO26n-seg instead of larger variants)
      2. Consider reducing the image resolution for tracking
      3. Optimize tracking parameters by setting persist=True and adjusting conf thresholds higher to track fewer objects
      4. Use half=True (FP16) or int8=True when exporting your model to TensorRT for maximum GPU acceleration

      For the tracking components specifically, the BoT-SORT or ByteTrack algorithms used by Ultralytics do have some CPU-bound operations, but you can minimize their impact with the above optimizations. For more details on tracking optimization, check our tracking documentation.

      1
  9. Ranjith3436GitHub

    I convert Classification model that i retrained. But it throw the error "amax(): Expected reduction dim 1 to have non-zero size."

    1. Glenn JocherUltralytics

      @Ranjith3436 this error typically occurs when TensorRT encounters an empty dimension during model conversion. For classification models, this could happen if:

      1. Your model architecture was modified or if you're using a custom architecture
      2. There might be an issue with batch normalization layers during export
      3. The class count in your retrained model might be causing dimension mismatches

      Try these solutions:

      • Ensure your model's forward pass works correctly before export
      • Try exporting with simplify=False to preserve the original graph structure
      • Check if your classification model has the correct output dimension for your number of classes

      If the issue persists, share more details about your model architecture and export command for better assistance. For more guidance, refer to our TensorRT integration documentation.

    2. Ranjith3436GitHub

      The model trained use yolov8l-cls.pt and the error come from jetson device.

      (model): ClassificationModel( (model): Sequential( (0): Conv( (conv): Conv2d(3, 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (bn): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (1): Conv( (conv): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (bn): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (2): C2f( (cv1): Conv( (conv): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (cv2): Conv( (conv): Conv2d(320, 128, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (m): ModuleList( (0-2): 3 x Bottleneck( (cv1): Conv( (conv): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (cv2): Conv( (conv): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) ) ) (3): Conv( (conv): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (4): C2f( (cv1): Conv( (conv): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (cv2): Conv( (conv): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (m): ModuleList( (0-5): 6 x Bottleneck( (cv1): Conv( (conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (cv2): Conv( (conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) ) ) (5): Conv( (conv): Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (bn): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (6): C2f( (cv1): Conv( (conv): Conv2d(512, 512, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (cv2): Conv( (conv): Conv2d(2048, 512, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (m): ModuleList( (0-5): 6 x Bottleneck( (cv1): Conv( (conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (cv2): Conv( (conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) ) ) (7): Conv( (conv): Conv2d(512, 1024, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (bn): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (8): C2f( (cv1): Conv( (conv): Conv2d(1024, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (cv2): Conv( (conv): Conv2d(2560, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (m): ModuleList( (0-2): 3 x Bottleneck( (cv1): Conv( (conv): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (cv2): Conv( (conv): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) ) ) ) (9): Classify( (conv): Conv( (conv): Conv2d(1024, 1280, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn): BatchNorm2d(1280, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (act): SiLU(inplace=True) ) (pool): AdaptiveAvgPool2d(output_size=1) (drop): Dropout(p=0.0, inplace=True) (linear): Linear(in_features=1280, out_features=2, bias=True) ) ) ) )

      model architecture