Optimizing YOLO26 Inferences with Neural Magic's DeepSparse Engine#

When deploying object detection models like Ultralytics YOLO26 on various hardware, optimizing inference speed can be a challenge. Neural Magic's DeepSparse Engine addresses this on CPUs, using sparsity and quantization to approach GPU-class inference speeds without a GPU.

This guide shows you how to deploy YOLO26 using Neural Magic's DeepSparse, how to run inferences, and also how to benchmark performance to ensure it is optimized.

Neural Magic's DeepSparse#

Neural Magic's DeepSparse is an inference runtime designed to optimize the execution of neural networks on CPUs. It applies techniques like sparsity, pruning, and quantization to reduce computational demands while maintaining accuracy, for efficient and scalable neural network execution across various devices.

Benefits of Integrating Neural Magic's DeepSparse with YOLO26#

Key advantages of DeepSparse include:

• Enhanced Inference Speed: Neural Magic reports up to 525 FPS on YOLOv8n, significantly faster than standard CPU inference.

• Optimized Model Efficiency: Uses pruning and quantization to enhance YOLO26's efficiency, reducing model size and computational requirements while maintaining accuracy.

• High Performance on Standard CPUs: Delivers GPU-like performance on CPUs, a more accessible and cost-effective option for many applications.

• Streamlined Integration and Deployment: Provides tools for integrating YOLO models into applications, including image and video annotation features.

• Support for Various Model Types: Compatible with both standard and sparsity-optimized YOLO models, adding deployment flexibility.

• Cost-Effective and Scalable Solution: Reduces operational expenses and offers scalable deployment of advanced object detection models.

How Does Neural Magic's DeepSparse Technology Work?#

Neural Magic's DeepSparse technology is inspired by the human brain's efficiency in neural network computation. It adopts two key principles from the brain as follows:

• Sparsity: The process of sparsification involves pruning redundant information from deep learning networks, leading to smaller and faster models with minimal accuracy loss. This technique reduces the network's size and computational needs significantly.

• Locality of Reference: DeepSparse uses a unique execution method, breaking the network into Tensor Columns. These columns are executed depth-wise, fitting entirely within the CPU's cache. This approach mimics the brain's efficiency, minimizing data movement and maximizing the CPU's cache use.

Creating A Sparse Version of YOLOv8 Trained on a Custom Dataset#

SparseZoo, an open-source model repository by Neural Magic, offers a collection of pre-sparsified YOLOv8 model checkpoints. With SparseML, which integrates with Ultralytics, users can fine-tune these sparse checkpoints on their own datasets from the command line.

Check out Neural Magic's SparseML YOLOv8 documentation for more details.

Usage: Deploying YOLO26 using DeepSparse#

Deploying YOLO26 with Neural Magic's DeepSparse involves a few steps.

Step 1: Installation#

To install the required packages, run:

# Install the required packages
pip install deepsparse[yolov8]

Step 2: Exporting YOLO26 to ONNX Format#

DeepSparse Engine requires YOLO26 models in ONNX format. Use the following command to export YOLO26 models:

# Export YOLO26 model to ONNX format
yolo detect export model=yolo26n.pt format=onnx opset=13

This command will save the yolo26n.onnx model to your disk.

Step 3: Deploying and Running Inferences#

With your YOLO26 model in ONNX format, you can deploy and run inferences with the DeepSparse Python API:

from deepsparse import Pipeline

# Specify the path to your YOLO26 ONNX model
model_path = "path/to/yolo26n.onnx"

# Set up the DeepSparse Pipeline
yolo_pipeline = Pipeline.create(task="yolov8", model_path=model_path)

# Run the model on your images
images = ["path/to/image.jpg"]
pipeline_outputs = yolo_pipeline(images=images)

Step 4: Benchmarking Performance#

It's important to check that your YOLO26 model is performing optimally on DeepSparse. You can benchmark your model's performance to analyze throughput and latency:

# Benchmark performance
deepsparse.benchmark model_path="path/to/yolo26n.onnx" --scenario=sync --input_shapes="[1,3,640,640]"

Step 5: Additional Features#

DeepSparse provides additional features for practical integration of YOLO26 in applications, such as image annotation and dataset evaluation.

# For image annotation
deepsparse.yolov8.annotate --source "path/to/image.jpg" --model_filepath "path/to/yolo26n.onnx"

# For evaluating model performance on a dataset
deepsparse.yolov8.eval --model_path "path/to/yolo26n.onnx"

Running the annotate command processes your specified image, detecting objects, and saving the annotated image with bounding boxes and class labels. The annotated image will be stored in an annotation-results folder. This helps provide a visual representation of the model's detection capabilities.

After running the eval command, you will receive detailed output metrics such as precision, recall, and mAP (mean Average Precision). These metrics show how your model performs on the dataset and help guide fine-tuning and optimization for specific use cases.

Summary#

This guide explored integrating Ultralytics YOLO26 with Neural Magic's DeepSparse Engine and how its sparsity techniques speed up inference on CPU platforms.

For more detailed information and advanced usage, visit the DeepSparse repository by Neural Magic. You can also explore the YOLOv8 integration guide and watch a walkthrough session on YouTube.

FAQ#