SUN RGB-D Depth Dataset#
SUN RGB-D is a real-world indoor scene-understanding benchmark captured with four different RGB-D sensors: Intel RealSense, Asus Xtion, and Microsoft Kinect v1 and v2. Its multi-sensor design makes it a valuable source of real indoor depth diversity for monocular depth estimation.
Explore SUN RGB-D on Ultralytics Platform to preview its RGB-depth pairs, inspect dataset statistics, and clone it for training.
Key Features#
- Captured with four different RGB-D sensors (Intel RealSense, Asus Xtion, Microsoft Kinect v1 and v2), providing real multi-sensor variety.
- Covers a broad range of real indoor scenes for scene-understanding research.
- Depth range up to approximately 10 m, typical of consumer indoor RGB-D capture.
- Sensor-derived depth ground truth aligned to the RGB frames.
- Contributes real multi-sensor indoor diversity to the Ultralytics depth pretraining mix.
Dataset Structure#
The SUN RGB-D depth dataset is split into two subsets:
- Train: 9,245 images with paired depth maps for training.
- Val: 1,090 images with paired depth maps for validation during model training.
Each sample consists of one RGB image and one paired scaled uint16 depth PNG, following the Ultralytics depth dataset format. The built-in conversion writes millimeters, so the default depth_scale: 1000 applies.
Role in YOLO26-Depth#
SUN RGB-D is a training source in the Ultralytics YOLO26-Depth multi-dataset pretraining mix of roughly 2.19M image–depth pairs. It contributes real multi-sensor indoor diversity, exposing the model to depth captured across several different consumer RGB-D devices. The resulting models are evaluated on the standard NYU, KITTI, Make3D, ETH3D, and iBims-1 benchmarks.
Dataset YAML#
A YAML (Yet Another Markup Language) file is used to define the dataset configuration. It contains information about the dataset's paths, classes, and other relevant information.
# Ultralytics 🚀 AGPL-3.0 License - https://ultralytics.com/license
# SUN RGB-D dataset for monocular depth estimation — real indoor, multi-sensor RGB-D (RealSense/Xtion/Kinect), up to ~10 m
# Documentation: https://docs.ultralytics.com/datasets/depth/sunrgbd
# Example usage: yolo depth train data=depth-sunrgbd.yaml model=yolo26n-depth.pt
# parent
# ├── ultralytics
# └── datasets
# └── depth-sunrgbd ← downloads here (6.5 GB archive, ~14 GB converted)
# ├── images/{train,val} # RGB images
# └── depth/{train,val} # paired 16-bit *.png depth maps (images/ -> depth/)
# If an interrupted download leaves a partial dataset, delete the depth-sunrgbd dir and re-run to rebuild it.
path: depth-sunrgbd # dataset root dir (relative to Ultralytics settings 'datasets_dir')
train: images/train # train images (relative to 'path') 9245 images
val: images/val # val images (relative to 'path') 1090 images
nc: 1
names:
0: depth
channels: 3
# Download script/URL (optional)
download: |
import random
import shutil
from pathlib import Path
import cv2
import numpy as np
from ultralytics.data.utils import save_depth_png
from ultralytics.utils import TQDM
from ultralytics.utils.downloads import download
# Download and extract the official archive (~6.5 GB), then convert each of the 10335 scenes:
# refined depth_bfx PNGs decode via the SUN RGB-D bit-rotation (d>>3 | d<<13) to millimeters,
# clipped at 10 m; a deterministic random 1090-scene subset (seed 0) forms the val split
dir = Path(yaml["path"]) # dataset root dir
download(["https://rgbd.cs.princeton.edu/data/SUNRGBD.zip"], dir=dir / "source", delete=True, exist_ok=True)
for split in ("train", "val"):
(dir / "images" / split).mkdir(parents=True, exist_ok=True)
(dir / "depth" / split).mkdir(parents=True, exist_ok=True)
scenes = sorted(p.parent for p in (dir / "source" / "SUNRGBD").rglob("depth_bfx"))
names = ["_".join(s.relative_to(dir / "source" / "SUNRGBD").parts) for s in scenes]
val = set(random.Random(0).sample(names, k=1090))
for scene, name in TQDM(zip(scenes, names), total=len(scenes), desc="Converting"):
split = "val" if name in val else "train"
d = cv2.imread(str(next((scene / "depth_bfx").glob("*.png"))), cv2.IMREAD_ANYDEPTH)
d = (((d >> 3) | (d << 13)) / 1000.0).clip(max=10).astype(np.float32) # bit-rotated mm -> m
save_depth_png(dir / "depth" / split / f"{name}.png", d)
next((scene / "image").glob("*.jpg")).replace(dir / "images" / split / f"{name}.jpg")
shutil.rmtree(dir / "source")Usage#
To train a YOLO26n-depth model on the SUN RGB-D dataset with an image size of 640, you can use the following code snippets. For a comprehensive list of available arguments, refer to the model Training page.
from ultralytics import YOLO
# Load a model
model = YOLO("yolo26n-depth.pt") # load a pretrained depth model (recommended for training)
# Train the model
results = model.train(data="depth-sunrgbd.yaml", epochs=100, imgsz=640)Pretrained Models#
The YOLO26 depth family is trained on the broad multi-dataset depth pretraining mix that SUN RGB-D is part of. These models auto-download from the latest Ultralytics release, for example YOLO26x-depth from v8.4.0, and span a range of sizes for different accuracy and resource requirements.
Citations and Acknowledgments#
If you use the SUN RGB-D dataset in your research or development work, please cite the following paper:
@inproceedings{song2015sunrgbd,
title={SUN RGB-D: A RGB-D Scene Understanding Benchmark Suite},
author={Song, Shuran and Lichtenberg, Samuel P. and Xiao, Jianxiong},
booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
year={2015}
}We would like to acknowledge the authors for creating and maintaining this valuable resource for the computer vision community.