Baidu's RT-DETR: A Vision Transformer-Based Real-Time Object Detector#

Overview#

Real-Time Detection Transformer (RT-DETR), developed by Baidu, is an end-to-end object detector that provides real-time performance while maintaining high accuracy. It builds on DETR, an NMS-free transformer detection framework, and adds a convolutional backbone and an efficient hybrid encoder to reach real-time speed. RT-DETR efficiently processes multiscale features by decoupling intra-scale interaction and cross-scale fusion. The model is highly adaptable, supporting flexible adjustment of inference speed using different decoder layers without retraining. RT-DETR performs best on accelerated backends such as CUDA with TensorRT.

Watch: How to Use Baidu's RT-DETR for Object Detection | Inference and Benchmarking with Ultralytics

Overview of Baidu's RT-DETR. The RT-DETR model architecture diagram shows the last three stages of the backbone {S3, S4, S5} as the input to the encoder. The efficient hybrid encoder transforms multiscale features into a sequence of image features through intrascale feature interaction (AIFI) and cross-scale feature-fusion module (CCFM). The IoU-aware query selection is employed to select a fixed number of image features to serve as initial object queries for the decoder. Finally, the decoder with auxiliary prediction heads iteratively optimizes object queries to generate boxes and confidence scores (source).

Key Features#

• Efficient Hybrid Encoder: Baidu's RT-DETR uses an efficient hybrid encoder that processes multiscale features by decoupling intra-scale interaction and cross-scale fusion. This transformer-based design reduces computational costs and allows for real-time object detection.
• IoU-aware Query Selection: Baidu's RT-DETR improves object query initialization with IoU-aware query selection. This allows the model to focus on the most relevant objects in the scene, enhancing the detection accuracy.
• Adaptable Inference Speed: Baidu's RT-DETR supports flexible adjustments of inference speed by using different decoder layers without the need for retraining. This adaptability facilitates practical application in various real-time object detection scenarios.
• NMS-Free Framework: Based on DETR, RT-DETR eliminates the need for non-maximum suppression post-processing, simplifying the detection pipeline and potentially improving efficiency.
• Anchor-Free Detection: As an anchor-free detector, RT-DETR simplifies the detection process and may improve generalization across different datasets.

Pretrained Models#

The Ultralytics Python API provides pretrained PaddlePaddle RT-DETR models with different scales:

• RT-DETR-L: 53.0% AP on COCO val2017, 114 FPS on T4 GPU
• RT-DETR-X: 54.8% AP on COCO val2017, 74 FPS on T4 GPU

Baidu released RTDETRv2 in July 2024, which further improves on the original architecture.

Usage Examples#

The examples below show simple RT-DETR training and inference. Models can also be trained on cloud GPUs through Ultralytics Platform.

from ultralytics import RTDETR

# Load a COCO-pretrained RT-DETR-l model
model = RTDETR("rtdetr-l.pt")

# Display model information (optional)
model.info()

# Train the model on the COCO8 example dataset for 100 epochs
results = model.train(data="coco8.yaml", epochs=100, imgsz=640)

# Run inference with the RT-DETR-l model on the 'bus.jpg' image
results = model("path/to/bus.jpg")

from ultralytics import RTDETR

rtdetr = RTDETR("rtdetr-l.pt")
head = rtdetr.model.model[-1]

# Choose one or both settings after validating the speed/accuracy trade-off.
head.decoder.eval_idx = 3  # Use 4 of 6 decoder layers.
head.num_queries = 100  # Use fewer object queries.

results = rtdetr("path/to/image.jpg")

# Export uses the same decoder and query settings, including TensorRT exports.
rtdetr.export(format="engine", device=0, quantize=16)

Supported Tasks and Modes#

This table presents the model types, the specific pretrained weights, the tasks supported by each model, and the supported modes (Train, Val, Predict, Export), indicated by ✅ emojis.

Ideal Use Cases#

RT-DETR is particularly well-suited for applications requiring both high accuracy and real-time performance:

• Autonomous Driving: For reliable environmental perception in self-driving systems where both speed and accuracy are critical.
• Advanced Robotics: Enabling robots to perform complex tasks requiring accurate object recognition and interaction in dynamic environments.
• Medical Imaging: For applications in healthcare where precision in object detection can be crucial for diagnostics.
• Surveillance Systems: For security applications requiring real-time monitoring with high detection accuracy. Learn about security alarm systems.
• Satellite Image Analysis: For detailed analysis of high-resolution imagery where global context understanding is important. Read about computer vision in satellite imagery.

Citations and Acknowledgments#

If you use Baidu's RT-DETR in your research or development work, please cite the original paper:

@misc{lv2023detrs,
      title={DETRs Beat YOLOs on Real-time Object Detection},
      author={Wenyu Lv and Shangliang Xu and Yian Zhao and Guanzhong Wang and Jinman Wei and Cheng Cui and Yuning Du and Qingqing Dang and Yi Liu},
      year={2023},
      eprint={2304.08069},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

For RTDETRv2, you can cite the 2024 paper:

@misc{lv2024rtdetrv2,
      title={RT-DETRv2: Improved Baseline with Bag-of-Freebies for Real-Time Detection Transformer},
      author={Wenyu Lv and Yian Zhao and Qinyao Chang and Kui Huang and Guanzhong Wang and Yi Liu},
      year={2024},
      eprint={2407.17140},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

We would like to acknowledge Baidu and the PaddlePaddle team for creating and maintaining this valuable resource for the computer vision community. Their development of the transformer-based real-time object detector RT-DETR is greatly appreciated.

FAQ#