Bắt đầu

Triton Inference Server với Ultralytics YOLO26

Triton Inference Server (trước đây có tên là TensorRT Inference Server) là một giải pháp phần mềm mã nguồn mở do NVIDIA phát triển. Giải pháp này cung cấp dịch vụ suy luận trên cloud được tối ưu cho GPU NVIDIA. Triton đơn giản hóa việc phục vụ model ở quy mô lớn trong môi trường production. Tích hợp Ultralytics YOLO26 với Triton Inference Server cho phép bạn triển khai các workload suy luận học sâu có khả năng mở rộng và hiệu suất cao. Hướng dẫn này trình bày các bước thiết lập và kiểm thử tích hợp.



Xem: Bắt đầu với NVIDIA Triton Inference Server.

Triton Inference Server là gì?#

Triton Inference Server được thiết kế để triển khai nhiều loại model AI trong môi trường production. Server này hỗ trợ nhiều framework deep learning và machine learning, bao gồm PyTorch, TensorFlow, ONNX, OpenVINO, TensorRT và nhiều framework khác. Các trường hợp sử dụng chính gồm:

  • Phục vụ nhiều model từ một phiên bản server duy nhất
  • Tải và gỡ tải model linh hoạt mà không cần khởi động lại server
  • Suy luận ensemble, cho phép kết hợp nhiều model để đạt kết quả tốt hơn
  • Quản lý phiên bản model để kiểm thử A/B và cập nhật cuốn chiếu

Lợi ích chính của Triton Inference Server#

Dùng Triton Inference Server với Ultralytics YOLO26 mang lại một số lợi ích:

  • Batch tự động: Nhóm nhiều yêu cầu AI lại với nhau trước khi xử lý, giúp cải thiện thông lượng và mức độ sử dụng GPU
  • Tích hợp Kubernetes: Thiết kế cloud-native hoạt động với Kubernetes để quản lý và mở rộng quy mô ứng dụng AI
  • Tối ưu hóa theo phần cứng: Khai thác tối đa GPU NVIDIA để đạt hiệu năng cao nhất
  • Linh hoạt về framework: Hỗ trợ nhiều framework AI, bao gồm PyTorch, TensorFlow, ONNX, OpenVINO và TensorRT
  • Mã nguồn mở và có thể tùy chỉnh: Có thể sửa đổi để đáp ứng nhu cầu cụ thể, đảm bảo tính linh hoạt cho nhiều ứng dụng AI

Điều kiện tiên quyết#

Trước khi tiếp tục, hãy đảm bảo bạn có các điều kiện tiên quyết sau:

  • Đã cài Docker (>= 28.2.0, kèm NVIDIA Container Toolkit >= 1.18 để truy cập GPU qua CDI) hoặc Podman trên máy
  • Cài đặt ultralytics:
    pip install ultralytics
  • Cài đặt tritonclient:
    pip install tritonclient[all]

Thiết lập Triton Inference Server#

Chạy toàn bộ khối thiết lập này để export Ultralytics YOLO26 sang ONNX, tạo kho model Triton và khởi động Triton Inference Server:

Lưu ý

Dùng tùy chọn runtime trong script để chọn container engine:

  • Đặt runtime = "docker" cho Docker
  • Đặt runtime = "podman" cho Podman
import contextlib
import subprocess
import time
from pathlib import Path

from tritonclient.http import InferenceServerClient

from ultralytics import YOLO

runtime = "docker"  # set to "podman" to use Podman

# 1) Exporting YOLO26 to ONNX Format

# Load a model
model = YOLO("yolo26n.pt")  # load an official model

# Retrieve metadata during export. Metadata needs to be added to config.pbtxt. See next section.
metadata = []

def export_cb(exporter):
    metadata.append(exporter.metadata)

model.add_callback("on_export_end", export_cb)

# Export the model
onnx_file = model.export(format="onnx", dynamic=True)

# 2) Setting Up Triton Model Repository

# Define paths
model_name = "yolo"
triton_repo_path = Path("tmp") / "triton_repo"
triton_model_path = triton_repo_path / model_name

# Create directories
(triton_model_path / "1").mkdir(parents=True, exist_ok=True)

# Move ONNX model to Triton Model path
Path(onnx_file).rename(triton_model_path / "1" / "model.onnx")

# Create config file
(triton_model_path / "config.pbtxt").touch()

data = """
# Add metadata
parameters {
  key: "metadata"
  value {
    string_value: "%s"
  }
}

# Enable TensorRT acceleration (requires a GPU and TensorRT-enabled Triton; remove this block for CPU-only serving)
# The first run will be slow due to TensorRT engine conversion
optimization {
  execution_accelerators {
    gpu_execution_accelerator {
      name: "tensorrt"
      parameters {
        key: "precision_mode"
        value: "FP16"
      }
      parameters {
        key: "max_workspace_size_bytes"
        value: "3221225472"
      }
      parameters {
        key: "trt_engine_cache_enable"
        value: "1"
      }
      parameters {
        key: "trt_engine_cache_path"
        value: "/models/yolo/1"
      }
    }
  }
}
""" % metadata[0]  # noqa

with open(triton_model_path / "config.pbtxt", "w") as f:
    f.write(data)

# 3) Running Triton Inference Server

# Define image https://catalog.ngc.nvidia.com/orgs/nvidia/containers/tritonserver
tag = "nvcr.io/nvidia/tritonserver:26.02-py3"  # 16.17 GB (Compressed Size)

subprocess.call(f"{runtime} pull {tag}", shell=True)

# CDI GPU request works identically on Docker and Podman
gpu_flags = "--device nvidia.com/gpu=all"

container_name = "triton_server"

# Note: The :z flag on the volume mount is necessary for systems with SELinux (like Fedora/RHEL)
subprocess.call(
    f"{runtime} run -d --rm --name {container_name} {gpu_flags} -v {triton_repo_path.absolute()}:/models:z -p 8000:8000 {tag} tritonserver --model-repository=/models",
    shell=True,
)

# Wait for the Triton server to start
triton_client = InferenceServerClient(url="127.0.0.1:8000", verbose=False, ssl=False)

# Wait until model is ready
for _ in range(10):
    with contextlib.suppress(Exception):
        assert triton_client.is_model_ready(model_name)
        break
    time.sleep(1)

Chạy suy luận#

Chạy suy luận bằng model Triton Server:

from ultralytics import YOLO

# # Tải model Triton Server
model = YOLO("http://127.0.0.1:8000/yolo", task="detect")

# # Chạy suy luận trên server
results = model("path/to/image.jpg")

Dọn dẹp container:

import subprocess

runtime = "docker"  # set to "podman" to use Podman
container_name = "triton_server"  # Kill the named container
subprocess.call(f"{runtime} kill {container_name}", shell=True)

Tối ưu hóa TensorRT (tùy chọn)#

Để đạt hiệu suất cao hơn nữa, bạn có thể dùng TensorRT với Triton Inference Server. TensorRT là trình tối ưu hóa học sâu hiệu suất cao được xây dựng riêng cho GPU NVIDIA, có thể tăng đáng kể tốc độ suy luận.

Các lợi ích chính khi dùng TensorRT với Triton bao gồm:

  • Suy luận nhanh hơn tới 36 lần so với các nền tảng chỉ dùng CPU, theo NVIDIA
  • Tối ưu hóa theo phần cứng để khai thác GPU tối đa
  • Hỗ trợ định dạng độ chính xác thấp hơn (INT8, FP16) mà vẫn duy trì độ chính xác
  • Gộp layer để giảm chi phí tính toán

config.pbtxt ở trên đã bật bộ tăng tốc TensorRT của Triton cho model ONNX. Để chạy TensorRT trực tiếp với Ultralytics, hãy export model Ultralytics YOLO26 sang định dạng TensorRT:

from ultralytics import YOLO

# Tải model YOLO26
model = YOLO("yolo26n.pt")

# Export model sang định dạng TensorRT
model.export(format="engine")  # # tạo 'yolo26n.engine'

Để biết thêm thông tin về tối ưu hóa TensorRT, hãy xem hướng dẫn tích hợp TensorRT.

Giờ đây, bạn có thể triển khai và chạy model Ultralytics YOLO26 trên Triton Inference Server để suy luận hiệu năng cao, có khả năng mở rộng. Nếu muốn dùng dịch vụ hosting được quản lý, Ultralytics Platform cung cấp endpoint triển khai chuyên dụng.

Câu hỏi thường gặp#

  • Thiết lập Ultralytics YOLO26 với NVIDIA Triton Inference Server gồm một vài bước chính:

    1. Export YOLO26 sang định dạng ONNX:

      from ultralytics import YOLO
      
      # Tải model
      model = YOLO("yolo26n.pt")  # tải model chính thức
      
      # # Xuất model sang định dạng ONNX
      onnx_file = model.export(format="onnx", dynamic=True)
    2. Thiết lập Triton Model Repository:

      from pathlib import Path
      
      # # Xác định đường dẫn
      model_name = "yolo"
      triton_repo_path = Path("tmp") / "triton_repo"
      triton_model_path = triton_repo_path / model_name
      
      # # Tạo thư mục
      (triton_model_path / "1").mkdir(parents=True, exist_ok=True)
      Path(onnx_file).rename(triton_model_path / "1" / "model.onnx")
      (triton_model_path / "config.pbtxt").touch()
    3. Chạy Triton Server:

      import contextlib
      import subprocess
      import time
      
      from tritonclient.http import InferenceServerClient
      
      # Define image https://catalog.ngc.nvidia.com/orgs/nvidia/containers/tritonserver
      tag = "nvcr.io/nvidia/tritonserver:26.02-py3"
      
      runtime = "docker"  # set to "podman" to use Podman
      subprocess.call(f"{runtime} pull {tag}", shell=True)
      
      # CDI GPU request works identically on Docker and Podman
      gpu_flags = "--device nvidia.com/gpu=all"
      
      container_name = "triton_server"
      subprocess.call(
          f"{runtime} run -d --rm --name {container_name} {gpu_flags} -v {triton_repo_path.absolute()}:/models:z -p 8000:8000 {tag} tritonserver --model-repository=/models",
          shell=True,
      )
      
      triton_client = InferenceServerClient(url="127.0.0.1:8000", verbose=False, ssl=False)
      
      for _ in range(10):
          with contextlib.suppress(Exception):
              assert triton_client.is_model_ready(model_name)
              break
          time.sleep(1)

    Thiết lập này có thể giúp bạn triển khai hiệu quả model Ultralytics YOLO26 ở quy mô lớn trên Triton Inference Server để suy luận model AI hiệu suất cao.

  • Tích hợp Ultralytics YOLO26 với NVIDIA Triton Inference Server mang lại một số lợi ích:

    • Suy luận AI có khả năng mở rộng: Triton cho phép phục vụ nhiều model từ một phiên bản server duy nhất, hỗ trợ tải và gỡ tải model linh hoạt, nhờ đó có khả năng mở rộng cao cho nhiều workload AI khác nhau.
    • Hiệu năng cao: Được tối ưu hóa cho GPU NVIDIA, Triton Inference Server đảm bảo hoạt động suy luận tốc độ cao, phù hợp với các ứng dụng thời gian thực như object detection.
    • Ensemble và quản lý phiên bản model: Chế độ ensemble của Triton cho phép kết hợp nhiều model để cải thiện kết quả, còn tính năng quản lý phiên bản model hỗ trợ kiểm thử A/B và cập nhật cuốn chiếu.
    • Batch tự động: Triton tự động nhóm nhiều yêu cầu suy luận lại với nhau, giúp cải thiện đáng kể thông lượng.
    • Triển khai đơn giản hơn: Ultralytics tải endpoint Triton như một model cục bộ, vì vậy bạn có thể chuyển sang dùng Triton mà không cần viết lại mã suy luận.
  • Dùng định dạng ONNX (Trao đổi mạng nơ-ron mở) cho model Ultralytics YOLO26 trước khi triển khai trên NVIDIA Triton Inference Server mang lại một số lợi ích chính:

    • Khả năng tương tác: Định dạng ONNX hỗ trợ chuyển đổi giữa các framework deep learning khác nhau (chẳng hạn như PyTorch, TensorFlow), đảm bảo khả năng tương thích rộng hơn.
    • Tối ưu hóa: Nhiều môi trường triển khai, bao gồm Triton, tối ưu hóa cho ONNX, giúp suy luận nhanh hơn và hiệu năng tốt hơn.
    • Triển khai dễ dàng: ONNX được hỗ trợ rộng rãi trên nhiều framework và nền tảng, giúp đơn giản hóa quá trình triển khai trên nhiều hệ điều hành và cấu hình phần cứng.
    • Độc lập với framework: Sau khi chuyển đổi sang ONNX, model không còn phụ thuộc vào framework ban đầu, giúp model dễ dàng chuyển đổi giữa các môi trường hơn.
    • Chuẩn hóa: ONNX cung cấp một biểu diễn tiêu chuẩn giúp khắc phục các vấn đề tương thích giữa những framework AI khác nhau.

    Để export model, hãy dùng:

    from ultralytics import YOLO
    
    model = YOLO("yolo26n.pt")
    onnx_file = model.export(format="onnx", dynamic=True)

    Bạn có thể làm theo các bước trong hướng dẫn tích hợp ONNX để hoàn tất quy trình.

  • Có, bạn có thể chạy suy luận bằng model Ultralytics YOLO26 trên NVIDIA Triton Inference Server. Sau khi thiết lập model trong Triton Model Repository và khởi chạy server, bạn có thể tải và chạy suy luận trên model như sau:

    from ultralytics import YOLO
    
    # # Tải model Triton Server
    model = YOLO("http://127.0.0.1:8000/yolo", task="detect")
    
    # # Chạy suy luận trên server
    results = model("path/to/image.jpg")

    Phương pháp này cho phép bạn tận dụng các tối ưu hóa của Triton trong khi vẫn sử dụng giao diện Ultralytics YOLO quen thuộc.

  • Ultralytics YOLO26 có một số ưu điểm riêng so với model TensorFlow và PyTorch khi triển khai:

    • Hiệu suất thời gian thực: Được tối ưu cho các tác vụ phát hiện đối tượng theo thời gian thực, Ultralytics YOLO26 mang lại tốc độ và độ chính xác cao, phù hợp với các ứng dụng cần phân tích video trực tiếp.
    • Dễ sử dụng: Ultralytics YOLO26 hoạt động với Triton Inference Server và hỗ trợ nhiều định dạng export (ONNX, TensorRT), giúp linh hoạt trong nhiều kịch bản triển khai.
    • Tính năng nâng cao: Phục vụ model qua Triton bổ sung khả năng tải model động, quản lý phiên bản model và inference kết hợp, những tính năng thiết yếu để triển khai AI có khả năng mở rộng và độ tin cậy cao.
    • API đơn giản hóa: API Ultralytics cung cấp giao diện nhất quán trên các môi trường triển khai khác nhau, giảm thời gian làm quen và phát triển.
    • Tối ưu cho thiết bị biên: Model Ultralytics YOLO26 được thiết kế để triển khai trên thiết bị biên, mang lại hiệu suất mạnh mẽ ngay cả trên thiết bị có tài nguyên hạn chế.

    Để biết thêm chi tiết, hãy so sánh các tùy chọn triển khai trong hướng dẫn export model.

Bình luận56

Đặt câu hỏi, chia sẻ kết quả hoặc giúp đỡ người khác.

  1. Tariq-droidGitHub

    I followed the steps and I am encountering this error: AttributeError: 'AutoBackend' object has no attribute 'task'

    can someone help? I have upgraded to the latest version

    1
    1. Glenn JocherUltralytics

      @Tariq-droid it seems like there might be an issue with the model setup. Please ensure that the model is correctly loaded and that the Triton server is running with the appropriate configuration. Double-check the model export and repository setup steps. If the problem persists, consider reviewing the Triton Inference Server documentation for further troubleshooting.

  2. Sachi9soniGitHub

    i am following the same approach and able to complete inference with above code, i did one extra step i also converted onnx model into TensorRT using ( nvcr.io/nvidia/tensorrt:22.11-py3 ) this TRT container and i ran inference with (nvcr.io/nvidia/tritonserver 22.11-py3 ) and i ran the infernce succesfully on any size of input image:

    however i want to run inference with different model size (i.e. 1024 or 800 or 1088), as i want model to be more accuratly work with certain size of images so i exported the onnx model with different size (1024) onnx_file = model.export(format='onnx', dynamic=True,imgsz=1024) converted mode to tesorRT engine using the same conatiner (22.11-py3) trtexec --onnx=yolov8l.onnx --saveEngine=model.plan

    the config.pbxt file i am using are also working fine model loaded succesfully showing status ready but at the time of inference its showing this error:: tritonclient.utils.InferenceServerException: [400] [request id: <id_unknown>] unexpected shape for input 'images' for model 'yolov8l'. Expected [1,3,1024,1024], got [1,3,640,640]

    @glenn-jocher

    1
    1. Glenn JocherUltralytics

      Hey there! It looks like you've done a great job running inference with different input sizes and converting the ONNX model to TensorRT. 👍

      For the error you're encountering, it seems like the Triton server is expecting an input size of [1,3,1024,1024] but is receiving [1,3,640,640] instead. This mismatch is likely due to the model configuration on the server still being set for the original size.

      To resolve this, you'll need to update the config.pbtxt file for your model in the Triton server to match the new input size that you've used during the export (imgsz=1024). Make sure the input section of your config.pbtxt reflects the new dimensions:

      input [
        {
          name: "images"
          data_type: TYPE_FP32
          format: FORMAT_NCHW
          dims: [3, 1024, 1024]
        }
      ]

      After updating the configuration, reload the model on the Triton server to apply the changes. If the server is set to watch the model repository, it should pick up the changes automatically. Otherwise, you may need to manually reload the model or restart the server.

      Give that a try and see if it resolves the issue! If you need more assistance, feel free to reach out. 😊

    2. Sachi9soniGitHub

      Hello, thanks for replying As I said .config.pbxt is ok for me .I have already mentioned the same .into config.pbxt For that I have used inspect model yolov8l.onnx

      https://www.google.com/url?sa=t&source=web&rct=j&opi=89978449&url=https://www.delltechnologies.com/asset/en-us/products/servers/industry-market/developing-and-deploying-vision-ai-with-dell-and-nvidia-metropolis.pdf&ved=2ahUKEwjU34X9xLCEAxW-ja8BHX_3AwAQFnoECB4QAQ&usg=AOvVaw2AsdCYUzyocT0opY0Rr8gD

      Following this pdf for reference .here in appendix they have given the way to inspect the model .after that I put the same configuration into config.pbxt.. I have problem only with different size of export . My cutomize models are also facing same problem for different sizes party from 640.. 640 is working fine

    3. Sachi9soniGitHub

      Apart from 640 , different export sizes are facing the same problem

      Min reproducible: Convert any model with different size model.export(onnx,imgsz=1024, dynamic = True)

      Use tensorRT container I am using the same tag as in this blog(23.09-py-3) Convert into .plan file using command trtexec --onnx= yolov8l.onnx --saveEngine = model.plan After this conversion we get summary of model too in which we also get input and output shape .( I put the same shape into config.pbxt) Setup the directory Update the config.pbxt according to model Start triton server

      Status is ready

      Send request to client for inference Run the inference using above code.

    4. Paula DerrengerUltralytics

      Hello! It seems like you've followed the steps correctly but are facing an issue with the input size during inference. Since you've already ensured that the config.pbtxt is set up with the correct input size and the model is loaded successfully, the problem might be with the client inference request.

      Make sure that when you send the inference request, the input data is also reshaped to the expected size [1,3,1024,1024]. If you're sending an image of size [1,3,640,640], it will not match the model's expected input shape and will result in the error you're seeing.

      Here's a quick check you can do:

      import numpy as np
      from tritonclient.utils import triton_to_np_dtype
      from tritonclient.http import InferenceServerClient, InferInput
      
      # Assuming 'input_name' is the name of your model's input tensor
      input_name = 'images'
      input_shape = (1, 3, 1024, 1024)
      input_dtype = 'FP32'
      
      # Create a dummy input data with the expected shape
      input_data = np.random.rand(*input_shape).astype(triton_to_np_dtype(input_dtype))
      
      # Initialize the Triton client
      client = InferenceServerClient(url='localhost:8000')
      
      # Create the InferInput object and set the data
      infer_input = InferInput(input_name, input_shape, input_dtype)
      infer_input.set_data_from_numpy(input_data)
      
      # Send inference request
      response = client.infer(model_name='yolov8l', inputs=[infer_input])

      Make sure to replace 'images' and 'yolov8l' with the actual input name and model name used in your Triton server setup.

      If you're still facing issues, double-check the client code to ensure the input data being sent for inference matches the model's expected input shape. If everything seems correct and the problem persists, it might be helpful to share the client code snippet that sends the inference request for further assistance. 😊

    5. Sachi9soniGitHub(đã chỉnh sửa)

      import contextlib import tritonclient.http as httpclient #pip install if needed from ultralytics import YOLO import subprocess import time import cv2 from tritonclient.http import InferenceServerClient import os import numpy as np from tritonclient.utils import triton_to_np_dtype from tritonclient.http import InferenceServerClient, InferInput,InferResult

      Assuming 'input_name' is the name of your model's input tensor

      input_name = 'images' input_shape = (1, 3, 1088, 1088) input_dtype = 'FP32'

      image=cv2.imread("1.jpg") input_data=cv2.resize(image,(1088,1088),interpolation = cv2.INTER_LINEAR) input_data=input_data.reshape(1,3,1088,1088).astype(triton_to_np_dtype(input_dtype))

      Initialize the Triton client

      client = InferenceServerClient(url='localhost:8000')

      Create the InferInput object and set the data

      infer_input = InferInput(input_name, input_shape, input_dtype) infer_input.set_data_from_numpy(input_data)

      Send inference request

      response = client.infer(model_name='v8_p2x_2_1080_50', inputs=[infer_input])

      i am able to run this succesfully by doig this but what to do with response object? now how to use YOLO

    6. Paula DerrengerUltralytics

      Hey there! Great job on successfully running inference with Triton Inference Server! 🎉 To work with the response object from your Triton client, you'll want to extract the inference results, which typically include the detected objects' bounding boxes, classes, and confidence scores.

      Here's a quick example of how you might process the response object:

      # Extract the output from the response
      output_data = response.as_numpy('<output_layer_name>')  # Replace with your actual output layer name
      
      # Now, output_data contains the results of the inference
      # You can process these results as needed, for example:
      for detection in output_data:
          x, y, w, h = detection[:4]  # Bounding box coordinates
          conf = detection[4]  # Confidence score
          class_id = detection[5]  # Class ID
          # ... Now you can use these values as needed

      Make sure to replace '<output_layer_name>' with the actual name of the output layer from your YOLO model.

      If you need further assistance or have more questions, feel free to reach out. Happy coding! 😊

    7. Sachi9soniGitHub

      hey thanks for your response , i was following this code for drwaing and visualizing the result which is availble on ultralytics:::::::: model = YOLO(f'http://localhost:8000/v8_p2x_2_1080_50', task='detect') results=model(input_image)

      Visualize the results on the frame

      annotated_frame = results[0].plot()

      Display the annotated frame

      cv2.imshow("YOLOv8 Inference", annotated_frame)

      which seems quiet simple to work with... but now i am getting tensor so should i have to write a cutomize python script to extracting the result?

    8. Glenn JocherUltralytics

      @Sachi9soni hey, thanks for reaching out! 😊 You're on the right track with the Triton Inference Server. If you've got the response tensor from Triton and you're looking to visualize the results similar to how it's done with Ultralytics directly, you're correct that you'll need to process the tensor.

      The response from Triton will give you raw output, which you'll need to decode to get the bounding boxes, classes, and scores. Here's a simplified way to do it:

      # Assuming 'output_data' is the raw tensor you got from Triton
      output_data = response.as_numpy('<output_layer_name>')  # Replace with your actual output layer name
      
      # Decode the detections (you'll need to implement the decoding based on your model's output format)
      detections = decode_output(output_data)  # This is a placeholder function
      
      # Now you can visualize the detections on your image
      for detection in detections:
          x1, y1, x2, y2, conf, class_id = detection
          cv2.rectangle(input_image, (x1, y1), (x2, y2), (255, 0, 0), 2)  # Draw rectangle
          cv2.putText(input_image, f'{class_id} {conf:.2f}', (x1, y1 - 10), cv2.FONT_HERSHEY_SIMPLEX, 0.9, (255, 0, 0), 2)  # Put class label
      
      # Show the annotated image
      cv2.imshow('YOLOv8 Inference', input_image)
      cv2.waitKey(0)
      cv2.destroyAllWindows()

      Remember to replace '<output_layer_name>' with the name of the output layer from your model and implement the decode_output function according to how your model formats its output.

      Hope this helps! If you need more detailed guidance, the Ultralytics docs and community are always here to help. Happy coding! 🚀

    9. Sachi9soniGitHub(đã chỉnh sửa)

      hey thanks for your responsee , i was trying for interepret the results i was getting (1,5,98260) size tensor . and also getting negative values and so many zero values after printing out the tensor values. i am thinking it like this : image size =1088x1088 is there 1088/32=34 x34 grids and each grids have 85 values? so 34x34x85=98260 and there are 5 boxes for each grid?? am i right? this is the output 5*98260 https://drive.google.com/file/d/1YmRey1AHUNr3lBQDYVjHJQpGu_cLqELi/view?usp=drive_link

    10. Glenn JocherUltralytics(đã chỉnh sửa)

      @Sachi9soni hey there! You're on the right track with your understanding of the output tensor. The tensor shape (1, 5, 98260) is the raw output of an anchor-free YOLOv8/YOLO11-style detector: batch 1, 4 box values plus 1 class score per candidate (a single-class model), and 98260 candidate predictions across all scales. There are no anchor boxes or 85-value layout.

      However, the negative and zero values you're seeing are likely due to raw model outputs that need to be processed through a detection post-processing step. This step typically involves applying a sigmoid function to certain outputs to get probabilities, decoding the bounding boxes, and applying Non-Maximum Suppression (NMS) to remove duplicates and keep only the most confident detections.

      The output you're seeing is not directly interpretable as detections; it's an intermediate representation that needs to be decoded. The YOLO class in Ultralytics handles this for you when you use it for inference, as in the code snippet you provided.

      If you're working with raw output tensors from Triton, you'll need to implement similar post-processing to interpret the results. If you'd like to stick with the simplicity of the Ultralytics interface, you can continue using the YOLO class as you've been doing, and it will take care of the post-processing for you.

      Hope this helps clear things up! If you have any more questions, feel free to ask. 😊👍

  3. eshine996GitHub

    How to do the pre-processing and post-processing of the images when converting the model to onnx? I hope to use Triton client to directly obtain inference results.

  4. nsaadatiGitHub

    Hello,

    I followed the standard steps to deploy a YOLO model using Triton Inference Server on a local machine. However, I noticed that inference without Triton is consistently faster. I’m looking to understand why this might be the case.

    Below is the code I used, along with system information and sample outputs from both local and Triton-based inference runs.

    '''from ultralytics import YOLO from pathlib import Path

    Local inference

    print("Processing video without Triton...") model = YOLO("yolo11n.pt") for _ in range(4): results = model("bus.jpg")

    Export to ONNX

    onnx_file = model.export(format="onnx", dynamic=True)

    Triton repo setup

    triton_repo_path = Path("tmp/triton_repo") triton_model_path = triton_repo_path / "yolo" / "1" triton_model_path.mkdir(parents=True, exist_ok=True) Path(onnx_file).rename(triton_model_path / "model.onnx")

    Triton config

    with open(triton_repo_path / "yolo" / "config.pbtxt", "w") as f: f.write("...") # contains standard TRT config with FP16

    Start Triton Server

    subprocess.call(f"docker run -d --rm --gpus 0 -v {triton_repo_path}:/models -p 8200:8000 <triton_image_tag> tritonserver --model-repository=/models", shell=True)

    Triton inference

    model = YOLO("http://localhost:8200/yolo", task="detect") print("Processing video Triton...") for _ in range(4): results = model("bus.jpg")'''

    System Info:

    GPU: NVIDIA RTX A3000 Laptop GPU

    CUDA: 12.2

    Driver: 538.78

    Python: 3.10.16

    Torch: 2.7.0+cu126

    Inference Output:

    Without Triton:

    css Copy Edit Processing video without Triton... 640x480 4 persons, 1 bus, 49.5ms 640x480 4 persons, 1 bus, 6.8ms 640x480 4 persons, 1 bus, 6.8ms 640x480 4 persons, 1 bus, 7.4ms With Triton:

    css Copy Edit Processing video Triton... 640x480 4 persons, 1 bus, 54.9ms 640x480 4 persons, 1 bus, 46.7ms 640x480 4 persons, 1 bus, 43.9ms 640x480 4 persons, 1 bus, 32.0ms As shown above, inference through Triton is noticeably slower. I expected Triton to offer better or at least similar performance, especially with TensorRT and FP16 optimizations.

    Could you please help explain why this might be happening, and if there are any recommendations for optimizing Triton inference performance?

    Best regards, Nastaran

    1. kuanyshbakytulyGitHub

      you need to optimize triton client using cuda shared memory due to constant requests to triton server. You can test request time.

      2
    2. Glenn JocherUltralytics

      @nsaadati the performance difference you're observing is expected behavior when comparing local inference to Triton server deployment. The additional latency comes from network communication overhead, even when running locally. Each inference request goes through HTTP/gRPC protocols, adding round-trip time.

      To optimize Triton performance, consider enabling dynamic batching in your config.pbtxt to process multiple requests together, and as @kuanyshbakytuly mentioned, using CUDA shared memory to reduce data transfer overhead between client and server. You can also try enabling the TensorRT execution accelerator directly in Triton's config rather than pre-converting to engine format.

      For single-image inference scenarios, local inference will typically be faster due to eliminating network overhead. Triton's benefits become apparent with concurrent requests, batching, and scaled deployments.

      1
    3. hpx502766238GitHub

      you need to optimize triton client using cuda shared memory due to constant requests to triton server. You can test request time.

      I need to optimize the Triton client using CUDA shared memory to reduce the overhead of constant requests to the Triton server. My question is: How can I enable CUDA shared memory via the Ultralytics API? Currently, Ultralytics only exposes a simple model.predict method. Although I have implemented custom preprocessing and postprocessing logic with shared memory acquisition, this approach is overly complex compared to the convenience of the standard Ultralytics API. Is there a more streamlined way to achieve this?

    4. Glenn JocherUltralytics

      At the moment, no—the high-level YOLO(...).predict() Triton path does not expose CUDA shared memory, and the current Triton backend sends im.cpu().numpy() to Triton via TritonRemoteModel, so if you need CUDA SHM/zero-copy-style client optimization you’ll need to use tritonclient directly rather than the Ultralytics wrapper; if you’d like, please open a feature request in Ultralytics Discussions so the team can track demand for adding this support.

  5. OBEsensiaGitHub

    Hello all,

    @glenn-jocher and @vladk-ml maybe a solution can be to consider OBB as a 4 points polygons and use some mask IoU function? It maybe not has efficient as the box IoU but doesn't it worth a try ?

    Best regards.

    1. Glenn JocherUltralytics

      @kuanyshbakytuly Thank you for the detailed analysis and benchmarks. The accuracy difference between direct model inference and Triton deployment is concerning. This could be due to several factors:

      1. Check if preprocessing/scaling is identical between direct inference and Triton - even small differences can impact results significantly
      2. Verify your TensorRT conversion parameters - try explicitly setting precision constraints (FP32/FP16)
      3. Examine your confidence threshold settings in both implementations
      4. Investigate if dynamic batching is causing any issues with tensor shapes

      For the most reliable conversion path, I recommend using:

      model = YOLO("best.pt")
      onnx_file = model.export(format="onnx", dynamic=True, simplify=True)

      Then convert to TensorRT within Triton using the GPU execution accelerator in your config.pbtxt rather than pre-converting to .plan files.

      @OBEsensia Your suggestion for handling OBB as 4-point polygons is interesting. Using mask IoU could indeed be a viable alternative for OBB post-processing in Triton deployments. This approach would allow for standard NMS implementations while still capturing the orientation information. The computational overhead might be worth the integration simplicity, especially when working with format constraints. If you or others implement this approach, we'd be interested in hearing about your results.

    2. kuanyshbakytulyGitHub

      @kuanyshbakytuly Thank you for the detailed analysis and benchmarks. The accuracy difference between direct model inference and Triton deployment is concerning. This could be due to several factors:

      1. Check if preprocessing/scaling is identical between direct inference and Triton - even small differences can impact results significantly
      2. Verify your TensorRT conversion parameters - try explicitly setting precision constraints (FP32/FP16)
      3. Examine your confidence threshold settings in both implementations
      4. Investigate if dynamic batching is causing any issues with tensor shapes

      For the most reliable conversion path, I recommend using:

      model = YOLO("best.pt")
      onnx_file = model.export(format="onnx", dynamic=True, simplify=True)

      Then convert to TensorRT within Triton using the GPU execution accelerator in your config.pbtxt rather than pre-converting to .plan files.

      @OBEsensia Your suggestion for handling OBB as 4-point polygons is interesting. Using mask IoU could indeed be a viable alternative for OBB post-processing in Triton deployments. This approach would allow for standard NMS implementations while still capturing the orientation information. The computational overhead might be worth the integration simplicity, especially when working with format constraints. If you or others implement this approach, we'd be interested in hearing about your results.

      Thank you, problem was on preprocessing. I rechecked and fixed.

  6. kuanyshbakytulyGitHub(đã chỉnh sửa)

    Summary: When deploying YOLOv8 models on Triton Inference Server (TIS), detection accuracy and confidence scores drop significantly compared to running the same models directly with ONNXRuntime or TensorRT.

    First, I tried to deploy by using this guide. I tested the inference performance for different model formats:

    s = time.time()
    for i in range(20):
        inference()
    print(time.time() - s)

    Results:

    Format	         Inference Time (20 runs)
    Triton (.plan)	     2.48 s
    YOLOv8 .engine.      0.56 s
    YOLOv8 .onnx	     1.29 s
    YOLOv8 .pt	     0.65 s

    So, I guess the problem due to low metrics (latency, so on) in Triton Server. Finally, I decided to deploying by handle in order to optimise client.py using cuda shared memory.

    Deployment Setup

    1. Export to ONNX:
    yolo export model=Models/best.pt format=onnx simplify=True opset=19 imgsz=640 batch=20 dynamic=True
    1. Convert ONNX to TensorRT (.plan) via polygraphy:
    polygraphy convert Models/model_det_fp32/best.onnx -o Models/model_det_fp32/model.plan \
      --trt-min-shapes images:[1,3,640,640] \
      --trt-opt-shapes images:[10,3,640,640] \
      --trt-max-shapes images:[20,3,640,640]
    1. Triton Server Config (tensorrt_plan):
    name: "model_det"
    platform: "tensorrt_plan"
    max_batch_size: 20
    
    input {
      name: "images"
      data_type: TYPE_FP32
      dims: [3, 640, 640]
    }
    
    output {
      name: "output0"
      data_type: TYPE_FP32
      dims: [5, 8400]
    }
    
    dynamic_batching {
      preferred_batch_size: [1, 2, 4, 8, 16, 20]
      max_queue_delay_microseconds: 500
    }
    
    instance_group {
      kind: KIND_GPU
      count: 1
    }
    1. Alternative Export via Ultralytics for comparing inference results:
    model = YOLO('Models/best.pt')
    model.export(format='engine', dynamic=True, batch=20, imgsz=640)

    ⚠️ Problem Comparing outputs from:

    model.engine (Ultralytics default TensorRT export)

    model.plan (via Triton)

    → The .plan version deployed on Triton loses confidence and detection count. Inference of model.engine detects true 17 detections (all conf in range(0.5, 0.96), while model.plan (via Triton) detects only 3 detections with max conf=0.6

    🧪 Experiment 2: ONNX Deployment on Triton Export Command:

    yolo export model=Models/model_det_test/best.pt format=onnx opset=19 dynamic=True
    name: "model_det"
    platform: "onnxruntime_onnx"
    max_batch_size: 0
    
    input {
      name: "images"
      data_type: TYPE_FP32
      format: FORMAT_NCHW
      dims: [3, -1, -1] 
    }
    
    output {
      name: "output0"
      data_type: TYPE_FP32
      dims: [5, -1] 
    }
    
    instance_group {
      kind: KIND_GPU
      count: 1
    }

    Comparison ONNXRuntime:

    preprocessed_imgs = preprocess(frame)
    sess = ort.InferenceSession("model.onnx")
    outputs = sess.run(["output"], {"input": preprocessed_imgs})

    Triton:

    client = InferenceServerClient(url='localhost:8000')
    infer_input, scales = preprocess(frame)
    response = client.infer(model_name='model_det', inputs=[infer_input])
    outputs = response.as_numpy('output')

    Result: ONNXRuntime returns 17 detections (as expected). Triton with .onnx returns only 3 detections, all with low confidence (conf=0.4, iou=0.4). Also, I tested convergence of the inferences (Onnxruntime vs Onnx on Triton). They are differing, I forgot saving values.

    diff = np.abs(onnxruntime_outputs - onnx_triton_outputs)

    ❗Expected Behavior Triton Server inference should match the accuracy and output quality of direct inference using ONNXRuntime or TensorRT.

    💬 Question Has anyone experienced similar issues when: Converting YOLOv8 models to .plan using TensorRT for Triton deployment? Observed loss of confidence scores and detection count? Any help or pointers on how to preserve accuracy when deploying YOLOv8 on Triton would be much appreciated!

    Versions: ultralytics 8.3.105 tensorrt 10.9.0.34 tensorrt-cu12 10.9.0.34 tensorrt_cu12_bindings 10.9.0.34 tensorrt_cu12_libs 10.9.0.34 onnx 1.17.0 onnxruntime 1.21.0 onnxruntime-gpu 1.21.0 torch 2.8.0.dev20250408+cu128 torchaudio 2.6.0.dev20250408+cu128 torchvision 0.22.0.dev20250408+cu128 triton 3.2.0 tritonclient 2.56.0

    Docker images: nvcr.io/nvidia/tritonserver 25.03-py3
    nvcr.io/nvidia/tensorrt 25.03-py3

    nvidia-smi Mon Apr 14 22:07:42 2025
    +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 570.124.06 Driver Version: 570.124.06 CUDA Version: 12.8 | |-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA RTX A4000 Off | 00000000:00:05.0 Off | Off | | 41% 34C P8 15W / 140W | 2415MiB / 16376MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+

    +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | 0 N/A N/A 42398 C ...ect_conveyer/.venv/bin/python 188MiB | | 0 N/A N/A 61556 C tritonserver 2212MiB | +-----------------------------------------------------------------------------------------+

    nvcc --version nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2025 NVIDIA Corporation Built on Fri_Feb_21_20:23:50_PST_2025 Cuda compilation tools, release 12.8, V12.8.93 Build cuda_12.8.r12.8/compiler.35583870_0

  7. vladk-mlGitHub

    Hello, I am currently developing an inference solution dependent on a YOLO OBB model for deployment as a docker container, and I have some questions with regards to the recommended development approach.

    I am using currently Triton container 25.01, with specific Nvidia pre-built container form " xx.yy-pyt-python-py3 " This container gives support for PyTorch(pytorch_libtorch) and Python backend for model inferencing and pre and post processing model ensembles.

    https://catalog.ngc.nvidia.com/orgs/nvidia/containers/tritonserver

    This was very suitable at the start due the need for python processes both in pre and post-inferencing, but actually integrating the OBB torchscript model into the Triton ensemble has been very painful, apparently the problem lies within the torchscript model itself with regards to how YOLO handles it's dynamic operations versus the static computation graph in TorchScript.

    I have been pursuing solutions and workarounds to the problem but have been met with only failure, and have begun considering alternatives. The error is at opt/pytorch/pytorch/aten/src/ATen/native/xnnpack/Convolution.cpp:188.

    Error Message: xnnpack::convolution not available! Reason: The provided (weight, bias, padding, stride, dilation, groups, transposed, output_min, output_max) parameters are either invalid individually or their combination is not supported by XNNPACK

    Despite XNNPACK and other optimizations being disabled at model load.

    My question(s) are this.

    1. This page only mentions ONNX for integration into Triton, is this the only official supported format by Ultralytics in terms of Triton integration? Or just the only documented one? If so, is this due to Triton's preference for scripted vs traced Torchscript model conversions in multi-GPU, and the preferred method for YOLO export being traced?

    2. If ONNX is the only supported format, is it known if ONNX OBB models converted from the standard yolo.pt models are compatible with Triton?

    3. There is an experimental feature with Triton where Pytorch models above 2.0 can be run with the python runtime, has this feature been looked at by Ultralytics in terms of support with OBB model support?

    Thanks, -Vlad K.

    1. Glenn JocherUltralytics

      @vladk-ml thank you for your detailed questions about Triton deployment with YOLO OBB models. Here's a concise response:

      1. ONNX is the officially documented format for Triton integration due to its broad compatibility and optimization benefits, but TorchScript exports are also supported. The Triton guide focuses on ONNX as it avoids graph staticity issues common in traced models. For TorchScript troubleshooting, see our TorchScript export documentation.

      2. YOLO OBB models exported to ONNX should work with Triton when using the dynamic export flag (dynamic=True). We recommend validating your ONNX export with the YOLO OBB model format documentation.

      3. Python backend support for PyTorch models in Triton hasn’t been formally tested by Ultralytics, but community contributions are welcome. For immediate needs, ONNX remains the recommended path.

      For further Triton configuration help, refer to NVIDIA’s Triton OBB deployment examples.

      1
    2. vladk-mlGitHub

      Thank you for the reply!

      I switched over to the recommended ONNX deployment, but have hit another snag, class mapping and NMS have been very difficult to implement in the post-processing stage of the model ensemble, since now the model is ONNX rather than Pytorch, the postprocessing must be done outside of the model, but this has been more difficult than expected. Either too many detections resulting in memory overload, no usable detections due to degenerate coordinates from output tensor channels [11], or all properly drawn bounding boxes don't actually have any objects within them. So I have a question:

      Do YOLO11 OBB models with multiple classes pre-trained on DOTA dataset have issues when exported to ONNX with nms and optimization disabled for Triton integration? If I have multiple classes of objects to detect with OBB or segmentation in the future, with Triton being the main development path for deployment, would it be best to have 1 model for each class?

      Thanks, -Vlad K.

    3. Glenn JocherUltralytics

      @vladk-ml Thanks for the update on your progress with ONNX deployment. The challenges you're facing with post-processing OBB models in Triton are not uncommon. Here are some insights to help:

      For post-processing OBB models in ONNX format, you'll need to implement custom NMS specific to oriented boxes, which differs from standard NMS. The output tensor from OBB models contains additional rotation parameters that require special handling.

      Regarding your specific questions:

      1. YOLO11 OBB models with multiple classes should work when exported to ONNX, but post-processing is more complex. The raw output needs careful decoding to extract the oriented bounding box parameters correctly.

      2. Using one model per class is generally not recommended - it increases deployment complexity and resource usage. Instead, focus on proper post-processing implementation for multi-class models.

      For a workable solution, consider:

      • Implement OBB-specific NMS in your Python backend (see our OBB task documentation for reference)
      • Ensure correct tensor parsing: OBB outputs include rotation parameters that standard NMS doesn't handle
      • Start with smaller confidence thresholds and gradually tune parameters

      You might also want to examine the ONNX model structure using tools like Netron to better understand the output format. If you continue to face issues, sharing a minimal reproducible example would help us provide more specific guidance.

  8. OBEsensiaGitHub

    Hello Ultralytics Community,

    I was facing an issue with the triton server integration when the model hasn't square input shape (for example 320640 for HW). It seems that the model object stucks to 640640 size even if the triton configuration is set to 320640. The real issue is that even if I resize the image to 320640 the pre_process reverts this change back to 640640... So running the inference with

    results = model(image)

    isn't enough and

    results = model(image, imgsz=[320,640])

    should be used instead. I just put the workaround here in any case someone else can save 5 minutes but...

    Do you think it would be great if the model (and actually the pre_process stage can use the model input configuration as a default?

    1. Glenn JocherUltralytics

      @OBEsensia thank you for sharing your experience! You're correct that when using non-square input shapes with Triton, explicitly specifying imgsz in the predict call is currently required. While the model configuration in Triton does set the server-side parameters, the client-side inference call needs to match those dimensions. We'll consider your suggestion for auto-inferring input shape from Triton config as a potential enhancement. For now, your workaround is the recommended approach. For more details on input shape management, see our Triton configuration guide: https://docs.ultralytics.com/guides/triton-inference-server

  9. faheem-khaskheliGitHub

    hello, i am trying to use .engine (tensorrt) format when using triton inference server but i get error, because triton does not recognize the backend needed for .engine file.

    1. faheem-khaskheliGitHub

      does .onnx give same speed as .engine file? also i have read the documentation of triton it does say that it support tensorrt, by doing some research i found it uses model.plan file. but i donot know how to make it. also i have tried .pt file it say constant.pkl file not found. in config.pbtxt i changes the backend: pytorch or backend: tensorrt. still cannot find any working solution.

    2. Glenn JocherUltralytics

      For Triton Inference Server, please export your YOLOv8 model to ONNX format, as Triton supports ONNX natively. The .engine format is not directly supported by Triton. Ensure your config.pbtxt is correctly set up for ONNX. For more details, refer to the Triton documentation.

    3. faheem-khaskheliGitHub

      https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/index.html

      if you look at the heading major features, first one say it support multiple DL frameworks, also github link clearly mention tensorrr, pytorch, tensorflow and all other popular frameworks.

      https://github.com/triton-inference-server/backend#where-can-i-find-all-the-backends-that-are-available-for-triton.

      also here is support matrix of OS and DL frameworks for triton. https://github.com/triton-inference-server/backend/blob/main/docs/backend_platform_support_matrix.md.

      i want to find out which format work better with triton specially on GPU.

    4. Glenn JocherUltralytics

      For optimal performance with Triton Inference Server, exporting your YOLOv8 model to ONNX format is recommended. While Triton supports TensorRT, it requires a .plan file, which can be generated using TensorRT tools outside of Triton. ONNX generally provides good performance and compatibility across various frameworks. If you encounter issues with .pt files, ensure your configuration aligns with Triton's supported backends. For further details, please consult the Triton documentation.