Get Started

OpenVINO Inference Optimization for YOLO

Intel OpenVINO toolkit ecosystem overview

Introduction#

When deploying deep learning models, particularly those for object detection such as Ultralytics YOLO models, you usually optimize inference either for latency (fast responses to single requests) or for throughput (many requests processed at once). This guide explains how to use Intel's OpenVINO toolkit for each goal, from consumer-grade applications to large-scale deployments.

Optimizing for Latency#

Latency optimization is vital for applications requiring immediate response from a single model given a single input, typical in consumer scenarios. The goal is to minimize the delay between input and inference result. However, achieving low latency involves careful consideration, especially when running concurrent inferences or managing multiple models.

Key Strategies for Latency Optimization#

  • Single Inference per Device: The simplest way to achieve low latency is by limiting to one inference at a time per device. Additional concurrency often leads to increased latency.
  • Leveraging Sub-Devices: Devices like multi-socket CPUs or multi-tile GPUs can execute multiple requests with minimal latency increase by utilizing their internal sub-devices.
  • OpenVINO Performance Hints: Utilizing OpenVINO's ov::LATENCY for the ov::performance_mode property during model compilation simplifies performance tuning, offering a device-agnostic and future-proof approach.

Managing First-Inference Latency#

  • Model Caching: To mitigate model load and compile times impacting latency, use model caching where possible. For scenarios where caching isn't viable, CPUs generally offer the fastest model load times.
  • Model Mapping vs. Reading: To reduce load times, OpenVINO replaced model reading with mapping. However, if the model is on a removable or network drive, consider using ov::enable_mmap(false) to switch back to reading.
  • AUTO Device Selection: This mode begins inference on the CPU and shifts to an accelerator once it is ready, reducing first-inference latency.

Optimizing for Throughput#

Throughput optimization is crucial for scenarios serving numerous inference requests simultaneously, maximizing resource utilization without significantly sacrificing individual request performance.

Approaches to Throughput Optimization#

  1. OpenVINO Performance Hints: A high-level, future-proof method to enhance throughput across devices using performance hints.

    import openvino as ov
    import openvino.properties.hint as hints
    
    core = ov.Core()
    model = core.read_model("yolo26n_openvino_model/yolo26n.xml")
    config = {hints.performance_mode: hints.PerformanceMode.THROUGHPUT}
    compiled_model = core.compile_model(model, "GPU", config)
  2. Explicit Batching and Streams: A more granular approach involving explicit batching and the use of streams for advanced performance tuning.

Designing Throughput-Oriented Applications#

To maximize throughput, applications should:

  • Process inputs in parallel, making full use of the device's capabilities.
  • Decompose data flow into concurrent inference requests, scheduled for parallel execution.
  • Use the Async API with callbacks to maintain efficiency and avoid device starvation.

Multi-Device Execution#

OpenVINO's multi-device mode simplifies scaling throughput by automatically balancing inference requests across devices without requiring application-level device management.

Real-World Performance Gains#

Implementing OpenVINO optimizations with Ultralytics YOLO models can yield significant performance improvements. As demonstrated in benchmarks, INT8 OpenVINO models ran 2.9 to 6.0 times faster than PyTorch on Intel Core Ultra CPUs, and the integrated GPUs of the same processors ran FP32 models 3.6 to 29.1 times faster than PyTorch on the CPU. The Intel Core Ultra Series 3 blog post covers the newest of these processors.

For example, on the CPU of an Intel Core Ultra X7 358H, the INT8 YOLO26n model ran in 9.99 ms per image against 29.28 ms for PyTorch, with an mAP50-95 of 0.4720 against 0.4765 on COCO128, a small loss of accuracy.

Practical Implementation#

To export and optimize your Ultralytics YOLO model for OpenVINO, use the export functionality:

from ultralytics import YOLO

# Load a model
model = YOLO("yolo26n.pt")

# Export the model to OpenVINO format
model.export(format="openvino", quantize=16)  # Export with FP16 precision

After exporting, you can run inference with the optimized model:

# Load the OpenVINO model
ov_model = YOLO("yolo26n_openvino_model/")

# Run inference (Ultralytics compiles OpenVINO models with the LATENCY performance hint)
results = ov_model("https://ultralytics.com/images/bus.jpg", verbose=True)

Conclusion#

Optimizing Ultralytics YOLO models for latency or throughput with OpenVINO can significantly improve your application's performance. The choice between optimizing for latency or throughput depends on your specific application needs and the characteristics of the deployment environment.

Whether you're optimizing for real-time responses or maximizing throughput for large-scale processing, start from your application's requirements, choose the matching performance hint, and benchmark on the target hardware.

FAQ#

  • Optimizing Ultralytics YOLO models for low latency involves several key strategies:

    1. Single Inference per Device: Limit inferences to one at a time per device to minimize delays.
    2. Leveraging Sub-Devices: Use devices like multi-socket CPUs or multi-tile GPUs which can handle multiple requests with minimal latency increase.
    3. OpenVINO Performance Hints: Use OpenVINO's ov::LATENCY during model compilation for simplified, device-agnostic tuning.

    For more practical tips on optimizing latency, check out the Latency Optimization section of our guide.

  • OpenVINO enhances Ultralytics YOLO model throughput by maximizing device resource utilization without sacrificing performance. Key benefits include:

    • Performance Hints: Simple, high-level performance tuning across devices.
    • Explicit Batching and Streams: Fine-tuning for advanced performance.
    • Multi-Device Execution: Automated inference load balancing, easing application-level management.

    Example configuration:

    import openvino as ov
    import openvino.properties.hint as hints
    
    core = ov.Core()
    model = core.read_model("yolo26n_openvino_model/yolo26n.xml")
    config = {hints.performance_mode: hints.PerformanceMode.THROUGHPUT}
    compiled_model = core.compile_model(model, "GPU", config)

    Learn more about throughput optimization in the Throughput Optimization section of our detailed guide.

  • To reduce first-inference latency, consider these practices:

    1. Model Caching: Use model caching to decrease load and compile times.
    2. Model Mapping vs. Reading: Use mapping (ov::enable_mmap(true)) by default but switch to reading (ov::enable_mmap(false)) if the model is on a removable or network drive.
    3. AUTO Device Selection: Use AUTO mode to start with CPU inference and transition to an accelerator once it is ready.

    For detailed strategies on managing first-inference latency, refer to the Managing First-Inference Latency section.

  • Balancing latency and throughput optimization requires understanding your application needs:

    • Latency Optimization: Ideal for real-time applications requiring immediate responses (e.g., consumer-grade apps).
    • Throughput Optimization: Best for scenarios with many concurrent inferences, maximizing resource use (e.g., large-scale deployments).

    Using OpenVINO's high-level performance hints and multi-device modes can help strike the right balance. Choose the appropriate OpenVINO performance hints based on your specific requirements.

  • Yes, Ultralytics YOLO models can be exported to many other runtimes. Options include:

    Explore more integrations on the Ultralytics Integrations page.

Comments2

Ask a question, share your results, or help someone else.

  1. KvsoniGitHub

    Is it possible to use the in built classes and functions of ultralytics to do pre-processing and post-processing and using openVINO for the inference part?

    1. Paula DerrengerUltralytics(edited)

      @Kvsoni hi there!

      Yes, that's possible, and the simplest way is to let Ultralytics do it all: export the model to OpenVINO and load the exported folder with YOLO. Ultralytics then handles the pre-processing (resizing, normalization), runs inference with OpenVINO, and does the post-processing (decoding the output into boxes):

      from ultralytics import YOLO
      
      model = YOLO("yolo26n.pt")
      model.export(format="openvino")  # creates 'yolo26n_openvino_model/'
      
      ov_model = YOLO("yolo26n_openvino_model/")
      results = ov_model("path/to/image.jpg")
      results[0].show()

      If you want to run the OpenVINO runtime yourself, note that YOLO model objects do not expose preprocess() or postprocess(); those steps live in the predictor classes (e.g. ultralytics.models.yolo.detect.DetectionPredictor).

      For more detailed guidance on optimizing your model with OpenVINO, including strategies for latency and throughput, please refer to our guide: Optimizing OpenVINO Inference for Ultralytics YOLO Models.

      Feel free to reach out if you have any more questions or need further assistance! 😊