Reference for ultralytics/utils/plotting.py
This page is sourced from https://github.com/ultralytics/ultralytics/blob/main/ultralytics/utils/plotting.py. Have an improvement or example to add? Open a Pull Request — thank you! 🙏
Class ultralytics.utils.plotting.Colors#
Colors()Ultralytics annotation colors, the reference spec that every Ultralytics library and app mirrors.
Colors are RGB and are drawn as RGB; convert to BGR only when drawing into a BGR buffer (bgr=True).
- Class color (boxes, OBB, instance and semantic masks, box labels):
custom[c]when the model carriescolorsmetadata ("#rrggbb" strings indexed by class id, parsed to RGB once per model byColors.parse, which ignores the whole list if any entry is malformed) andc < len(custom), elsepalette[c % 20]. Colors never change with a light or dark theme. - Box label text (
Annotator.get_txt_color):#111f68when the background luma 0.299R + 0.587G + 0.114*B exceeds 150, else#ffffff.
Ultralytics Color Palette#
| Index | Color | HEX | RGB |
|---|---|---|---|
| 0 | #042aff | (4, 42, 255) | |
| 1 | #0bdbeb | (11, 219, 235) | |
| 2 | #f3f3f3 | (243, 243, 243) | |
| 3 | #00dfb7 | (0, 223, 183) | |
| 4 | #111f68 | (17, 31, 104) | |
| 5 | #ff6fdd | (255, 111, 221) | |
| 6 | #ff444f | (255, 68, 79) | |
| 7 | #cced00 | (204, 237, 0) | |
| 8 | #00f344 | (0, 243, 68) | |
| 9 | #bd00ff | (189, 0, 255) | |
| 10 | #00b4ff | (0, 180, 255) | |
| 11 | #dd00ba | (221, 0, 186) | |
| 12 | #00ffff | (0, 255, 255) | |
| 13 | #26c000 | (38, 192, 0) | |
| 14 | #01ffb3 | (1, 255, 179) | |
| 15 | #7d24ff | (125, 36, 255) | |
| 16 | #7b0068 | (123, 0, 104) | |
| 17 | #ff1b6c | (255, 27, 108) | |
| 18 | #fc6d2f | (252, 109, 47) | |
| 19 | #a2ff0b | (162, 255, 11) |
Pose Color Palette#
| Index | Color | HEX | RGB |
|---|---|---|---|
| 0 | #ff8000 | (255, 128, 0) | |
| 1 | #ff9933 | (255, 153, 51) | |
| 2 | #ffb266 | (255, 178, 102) | |
| 3 | #e6e600 | (230, 230, 0) | |
| 4 | #ff99ff | (255, 153, 255) | |
| 5 | #99ccff | (153, 204, 255) | |
| 6 | #ff66ff | (255, 102, 255) | |
| 7 | #ff33ff | (255, 51, 255) | |
| 8 | #66b2ff | (102, 178, 255) | |
| 9 | #3399ff | (51, 153, 255) | |
| 10 | #ff9999 | (255, 153, 153) | |
| 11 | #ff6666 | (255, 102, 102) | |
| 12 | #ff3333 | (255, 51, 51) | |
| 13 | #99ff99 | (153, 255, 153) | |
| 14 | #66ff66 | (102, 255, 102) | |
| 15 | #33ff33 | (51, 255, 51) | |
| 16 | #00ff00 | (0, 255, 0) | |
| 17 | #0000ff | (0, 0, 255) | |
| 18 | #ff0000 | (255, 0, 0) | |
| 19 | #ffffff | (255, 255, 255) |
For Ultralytics brand colors see https://www.ultralytics.com/brand. Please use the official Ultralytics colors for all marketing materials.
Attributes
| Name | Type | Description |
|---|---|---|
palette | list[tuple] | List of RGB color tuples for general use. |
n | int | The number of colors in the palette. |
pose_palette | np.ndarray | A specific color palette array for pose estimation with dtype np.uint8. |
Methods
| Name | Description |
|---|---|
__call__ | Return the color for a class index from the model's custom colors, else from the palette. |
hex2rgb | Convert hex color codes to RGB values (i.e. default PIL order). |
parse | Return colors (a list or its repr) as RGB tuples, or None with one warning if an entry is not "#rrggbb". |
Examples
>>> from ultralytics.utils.plotting import Colors
>>> colors = Colors()
>>> colors(5, True) # Returns BGR format: (221, 111, 255)
>>> colors(5, False) # Returns RGB format: (255, 111, 221)
>>> colors(1, custom=[(255, 0, 0), (0, 255, 0)]) # Model colors override the palette: (0, 255, 0)ultralytics/utils/plotting.py
class Colors:
"""Ultralytics annotation colors, the reference spec that every Ultralytics library and app mirrors.
Colors are RGB and are drawn as RGB; convert to BGR only when drawing into a BGR buffer (`bgr=True`).
- Class color (boxes, OBB, instance and semantic masks, box labels): `custom[c]` when the model carries `colors`
metadata ("#rrggbb" strings indexed by class id, parsed to RGB once per model by `Colors.parse`, which ignores the
whole list if any entry is malformed) and `c < len(custom)`, else `palette[c % 20]`. Colors never change with a
light or dark theme.
- Box label text (`Annotator.get_txt_color`): `#111f68` when the background luma 0.299*R + 0.587*G + 0.114*B
exceeds 150, else `#ffffff`.
## Ultralytics Color Palette
| Index | Color | HEX | RGB |
|-------|-------------------------------------------------------------------|-----------|-------------------|
| 0 | <i class="fa-solid fa-square fa-2xl" style="color: #042aff;"></i> | `#042aff` | (4, 42, 255) |
| 1 | <i class="fa-solid fa-square fa-2xl" style="color: #0bdbeb;"></i> | `#0bdbeb` | (11, 219, 235) |
| 2 | <i class="fa-solid fa-square fa-2xl" style="color: #f3f3f3;"></i> | `#f3f3f3` | (243, 243, 243) |
| 3 | <i class="fa-solid fa-square fa-2xl" style="color: #00dfb7;"></i> | `#00dfb7` | (0, 223, 183) |
| 4 | <i class="fa-solid fa-square fa-2xl" style="color: #111f68;"></i> | `#111f68` | (17, 31, 104) |
| 5 | <i class="fa-solid fa-square fa-2xl" style="color: #ff6fdd;"></i> | `#ff6fdd` | (255, 111, 221) |
| 6 | <i class="fa-solid fa-square fa-2xl" style="color: #ff444f;"></i> | `#ff444f` | (255, 68, 79) |
| 7 | <i class="fa-solid fa-square fa-2xl" style="color: #cced00;"></i> | `#cced00` | (204, 237, 0) |
| 8 | <i class="fa-solid fa-square fa-2xl" style="color: #00f344;"></i> | `#00f344` | (0, 243, 68) |
| 9 | <i class="fa-solid fa-square fa-2xl" style="color: #bd00ff;"></i> | `#bd00ff` | (189, 0, 255) |
| 10 | <i class="fa-solid fa-square fa-2xl" style="color: #00b4ff;"></i> | `#00b4ff` | (0, 180, 255) |
| 11 | <i class="fa-solid fa-square fa-2xl" style="color: #dd00ba;"></i> | `#dd00ba` | (221, 0, 186) |
| 12 | <i class="fa-solid fa-square fa-2xl" style="color: #00ffff;"></i> | `#00ffff` | (0, 255, 255) |
| 13 | <i class="fa-solid fa-square fa-2xl" style="color: #26c000;"></i> | `#26c000` | (38, 192, 0) |
| 14 | <i class="fa-solid fa-square fa-2xl" style="color: #01ffb3;"></i> | `#01ffb3` | (1, 255, 179) |
| 15 | <i class="fa-solid fa-square fa-2xl" style="color: #7d24ff;"></i> | `#7d24ff` | (125, 36, 255) |
| 16 | <i class="fa-solid fa-square fa-2xl" style="color: #7b0068;"></i> | `#7b0068` | (123, 0, 104) |
| 17 | <i class="fa-solid fa-square fa-2xl" style="color: #ff1b6c;"></i> | `#ff1b6c` | (255, 27, 108) |
| 18 | <i class="fa-solid fa-square fa-2xl" style="color: #fc6d2f;"></i> | `#fc6d2f` | (252, 109, 47) |
| 19 | <i class="fa-solid fa-square fa-2xl" style="color: #a2ff0b;"></i> | `#a2ff0b` | (162, 255, 11) |
## Pose Color Palette
| Index | Color | HEX | RGB |
|-------|-------------------------------------------------------------------|-----------|-------------------|
| 0 | <i class="fa-solid fa-square fa-2xl" style="color: #ff8000;"></i> | `#ff8000` | (255, 128, 0) |
| 1 | <i class="fa-solid fa-square fa-2xl" style="color: #ff9933;"></i> | `#ff9933` | (255, 153, 51) |
| 2 | <i class="fa-solid fa-square fa-2xl" style="color: #ffb266;"></i> | `#ffb266` | (255, 178, 102) |
| 3 | <i class="fa-solid fa-square fa-2xl" style="color: #e6e600;"></i> | `#e6e600` | (230, 230, 0) |
| 4 | <i class="fa-solid fa-square fa-2xl" style="color: #ff99ff;"></i> | `#ff99ff` | (255, 153, 255) |
| 5 | <i class="fa-solid fa-square fa-2xl" style="color: #99ccff;"></i> | `#99ccff` | (153, 204, 255) |
| 6 | <i class="fa-solid fa-square fa-2xl" style="color: #ff66ff;"></i> | `#ff66ff` | (255, 102, 255) |
| 7 | <i class="fa-solid fa-square fa-2xl" style="color: #ff33ff;"></i> | `#ff33ff` | (255, 51, 255) |
| 8 | <i class="fa-solid fa-square fa-2xl" style="color: #66b2ff;"></i> | `#66b2ff` | (102, 178, 255) |
| 9 | <i class="fa-solid fa-square fa-2xl" style="color: #3399ff;"></i> | `#3399ff` | (51, 153, 255) |
| 10 | <i class="fa-solid fa-square fa-2xl" style="color: #ff9999;"></i> | `#ff9999` | (255, 153, 153) |
| 11 | <i class="fa-solid fa-square fa-2xl" style="color: #ff6666;"></i> | `#ff6666` | (255, 102, 102) |
| 12 | <i class="fa-solid fa-square fa-2xl" style="color: #ff3333;"></i> | `#ff3333` | (255, 51, 51) |
| 13 | <i class="fa-solid fa-square fa-2xl" style="color: #99ff99;"></i> | `#99ff99` | (153, 255, 153) |
| 14 | <i class="fa-solid fa-square fa-2xl" style="color: #66ff66;"></i> | `#66ff66` | (102, 255, 102) |
| 15 | <i class="fa-solid fa-square fa-2xl" style="color: #33ff33;"></i> | `#33ff33` | (51, 255, 51) |
| 16 | <i class="fa-solid fa-square fa-2xl" style="color: #00ff00;"></i> | `#00ff00` | (0, 255, 0) |
| 17 | <i class="fa-solid fa-square fa-2xl" style="color: #0000ff;"></i> | `#0000ff` | (0, 0, 255) |
| 18 | <i class="fa-solid fa-square fa-2xl" style="color: #ff0000;"></i> | `#ff0000` | (255, 0, 0) |
| 19 | <i class="fa-solid fa-square fa-2xl" style="color: #ffffff;"></i> | `#ffffff` | (255, 255, 255) |
!!! note "Ultralytics Brand Colors"
For Ultralytics brand colors see [https://www.ultralytics.com/brand](https://www.ultralytics.com/brand).
Please use the official Ultralytics colors for all marketing materials.
Attributes:
palette (list[tuple]): List of RGB color tuples for general use.
n (int): The number of colors in the palette.
pose_palette (np.ndarray): A specific color palette array for pose estimation with dtype np.uint8.
Examples:
>>> from ultralytics.utils.plotting import Colors
>>> colors = Colors()
>>> colors(5, True) # Returns BGR format: (221, 111, 255)
>>> colors(5, False) # Returns RGB format: (255, 111, 221)
>>> colors(1, custom=[(255, 0, 0), (0, 255, 0)]) # Model colors override the palette: (0, 255, 0)
"""
def __init__(self):
"""Initialize the Ultralytics color palette from a fixed list of hex color codes."""
hexs = (
"042AFF",
"0BDBEB",
"F3F3F3",
"00DFB7",
"111F68",
"FF6FDD",
"FF444F",
"CCED00",
"00F344",
"BD00FF",
"00B4FF",
"DD00BA",
"00FFFF",
"26C000",
"01FFB3",
"7D24FF",
"7B0068",
"FF1B6C",
"FC6D2F",
"A2FF0B",
)
self.palette = [self.hex2rgb(f"#{c}") for c in hexs]
self.n = len(self.palette)
self.pose_palette = np.array(
[
[255, 128, 0],
[255, 153, 51],
[255, 178, 102],
[230, 230, 0],
[255, 153, 255],
[153, 204, 255],
[255, 102, 255],
[255, 51, 255],
[102, 178, 255],
[51, 153, 255],
[255, 153, 153],
[255, 102, 102],
[255, 51, 51],
[153, 255, 153],
[102, 255, 102],
[51, 255, 51],
[0, 255, 0],
[0, 0, 255],
[255, 0, 0],
[255, 255, 255],
],
dtype=np.uint8,
)Method ultralytics.utils.plotting.Colors.__call__#
def __call__(self, i: int | torch.Tensor, bgr: bool = False, custom: list[tuple] | None = None) -> tupleReturn the color for a class index from the model's custom colors, else from the palette.
Args
| Name | Type | Description | Default |
|---|---|---|---|
i | int | torch.Tensor | Color index. | required |
bgr | bool, optional | Whether to return BGR format instead of RGB. | False |
custom | list[tuple], optional | Model class colors as RGB tuples indexed by class id. | None |
Returns
| Type | Description |
|---|---|
tuple | RGB or BGR color tuple. |
ultralytics/utils/plotting.py
def __call__(self, i: int | torch.Tensor, bgr: bool = False, custom: list[tuple] | None = None) -> tuple:
"""Return the color for a class index from the model's custom colors, else from the palette.
Args:
i (int | torch.Tensor): Color index.
bgr (bool, optional): Whether to return BGR format instead of RGB.
custom (list[tuple], optional): Model class colors as RGB tuples indexed by class id.
Returns:
(tuple): RGB or BGR color tuple.
"""
i = int(i)
c = custom[i] if custom and i < len(custom) else self.palette[i % self.n]
return (c[2], c[1], c[0]) if bgr else cMethod ultralytics.utils.plotting.Colors.hex2rgb#
def hex2rgb(h: str) -> tupleConvert hex color codes to RGB values (i.e. default PIL order).
Args
| Name | Type | Description | Default |
|---|---|---|---|
h | str | required |
ultralytics/utils/plotting.py
@staticmethod
def hex2rgb(h: str) -> tuple:
"""Convert hex color codes to RGB values (i.e. default PIL order)."""
return tuple(int(h[1 + i : 1 + i + 2], 16) for i in (0, 2, 4))Method ultralytics.utils.plotting.Colors.parse#
def parse(hexs: list[str] | str | None) -> list[tuple] | NoneReturn colors (a list or its repr) as RGB tuples, or None with one warning if an entry is not "#rrggbb".
Args
| Name | Type | Description | Default |
|---|---|---|---|
hexs | list[str] | str | None | required |
ultralytics/utils/plotting.py
@staticmethod
def parse(hexs: list[str] | str | None) -> list[tuple] | None:
"""Return `colors` (a list or its repr) as RGB tuples, or None with one warning if an entry is not "#rrggbb"."""
if not hexs:
return None
if isinstance(hexs, str): # ONNX/CoreML repr; a failed eval stays a str and is rejected below
with contextlib.suppress(ValueError, SyntaxError):
hexs = ast.literal_eval(hexs)
if isinstance(hexs, (list, tuple)) and all(
isinstance(h, str) and re.fullmatch("#[0-9a-fA-F]{6}", h) for h in hexs
):
return [Colors.hex2rgb(h) for h in hexs]
LOGGER.warning(f"Ignoring class colors {hexs!r:.100}, expected a list of '#rrggbb' strings. Using the palette.")
return NoneClass ultralytics.utils.plotting.Annotator#
Annotator(
im,
line_width: int | None = None,
font_size: int | None = None,
font: str = FONT,
pil: bool = False,
example: str = "abc",
)Ultralytics Annotator for train/val mosaics and JPGs and predictions annotations.
Tensor images must be contiguous HWC BGR uint8.
Args
| Name | Type | Description | Default |
|---|---|---|---|
im | np.ndarray | Image.Image | torch.Tensor | Image to annotate. Arrays and tensors are HWC BGR uint8; grayscale is expanded to 3 channels, 2-channel images are zero-padded, and extra channels are dropped. | required |
line_width | int, optional | Line width. Defaults to ~0.3% of the mean image dimension, at least 2. | None |
font_size | int, optional | Font size for PIL text. Defaults to ~3.5% of the mean image dimension, at least 12. | None |
font | str | Font file name used for PIL text. | FONT |
pil | bool | Whether to render text with PIL instead of cv2. Forced True for PIL images and non-ASCII example. | False |
example | str | Example label text used to detect non-ASCII characters that require PIL and a Unicode font. | "abc" |
Attributes
| Name | Type | Description |
|---|---|---|
im | np.ndarray | torch.Tensor | The BGR image being annotated. |
pil | bool | Whether text is rendered with a PIL TrueType font instead of cv2; shapes always use cv2. |
font | ImageFont.FreeTypeFont | ImageFont.ImageFont | Font used for PIL text annotations. |
lw | int | Line width for drawing. |
skeleton | list[list[int]] | Skeleton structure for keypoints. |
limb_color | np.ndarray | BGR colors of the COCO skeleton limbs. |
kpt_color | np.ndarray | BGR colors of the COCO keypoints. |
Methods
| Name | Description |
|---|---|
_draw | Paste an (h, w, 3) patch, or blend a patch color through an (h, w) alpha, at (x, y) clipped to the image. |
_to_numpy | Move a tensor image to CPU only when a CPU drawing operation requires it. |
box_label | Draw a bounding box on an image with a given label. |
depth_map | Render a colorized depth map blended over the image. |
get_bbox_dimension | Calculate the dimensions and area of a bounding box. |
get_txt_color | Assign text color based on background color. |
kpts | Plot keypoints on the image. |
masks | Plot masks on image. |
rectangle | Add a rectangle with optional fill and outline colors to the image. |
result | Return annotated image as a BGR NumPy array, or as an RGB PIL image if pil is True. |
save | Save the annotated image to 'filename'. |
semantic_mask | Plot a semantic segmentation mask on the image. |
show | Show the annotated image. |
text | Add text to an image using PIL or cv2. |
Examples
>>> import cv2
>>> from ultralytics.utils.plotting import Annotator
>>> im0 = cv2.imread("test.png")
>>> annotator = Annotator(im0, line_width=10)
>>> annotator.box_label([10, 10, 100, 100], "person", (255, 0, 0))ultralytics/utils/plotting.py
class Annotator:
"""Ultralytics Annotator for train/val mosaics and JPGs and predictions annotations.
Tensor images must be contiguous HWC BGR uint8.
Attributes:
im (np.ndarray | torch.Tensor): The BGR image being annotated.
pil (bool): Whether text is rendered with a PIL TrueType font instead of cv2; shapes always use cv2.
font (ImageFont.FreeTypeFont | ImageFont.ImageFont): Font used for PIL text annotations.
lw (int): Line width for drawing.
skeleton (list[list[int]]): Skeleton structure for keypoints.
limb_color (np.ndarray): BGR colors of the COCO skeleton limbs.
kpt_color (np.ndarray): BGR colors of the COCO keypoints.
Examples:
>>> import cv2
>>> from ultralytics.utils.plotting import Annotator
>>> im0 = cv2.imread("test.png")
>>> annotator = Annotator(im0, line_width=10)
>>> annotator.box_label([10, 10, 100, 100], "person", (255, 0, 0))
"""
def __init__(
self,
im,
line_width: int | None = None,
font_size: int | None = None,
font: str = FONT,
pil: bool = False,
example: str = "abc",
):
"""Initialize the Annotator class with image and line width along with color palette for keypoints and limbs.
Args:
im (np.ndarray | Image.Image | torch.Tensor): Image to annotate. Arrays and tensors are HWC BGR uint8;
grayscale is expanded to 3 channels, 2-channel images are zero-padded, and extra channels are dropped.
line_width (int, optional): Line width. Defaults to ~0.3% of the mean image dimension, at least 2.
font_size (int, optional): Font size for PIL text. Defaults to ~3.5% of the mean image dimension, at least
12.
font (str): Font file name used for PIL text.
pil (bool): Whether to render text with PIL instead of cv2. Forced True for PIL images and non-ASCII
`example`.
example (str): Example label text used to detect non-ASCII characters that require PIL and a Unicode font.
"""
non_ascii = not is_ascii(example) # non-latin labels, i.e. asian, arabic, cyrillic
input_is_pil = isinstance(im, Image.Image)
self.lw = line_width or max(round(sum(im.size if input_is_pil else im.shape) / 2 * 0.003), 2)
if input_is_pil: # normalize PIL RGB pixels to the BGR drawing buffer
im = cv2.cvtColor(np.asarray(im.convert("RGB")), cv2.COLOR_RGB2BGR)
input_is_tensor = isinstance(im, torch.Tensor)
self.pil = pil or non_ascii or input_is_pil
if input_is_tensor:
assert im.ndim == 3 and im.shape[2] == 3 and im.dtype == torch.uint8, (
f"Expected HWC uint8 tensor image with 3 channels, but got shape {tuple(im.shape)} and dtype {im.dtype}."
)
if self.pil or im.device.type == "cpu":
im, input_is_tensor = im.cpu().numpy(), False
if im.ndim == 2 or im.shape[2] == 1: # handle grayscale
im = cv2.cvtColor(im, cv2.COLOR_GRAY2BGR)
elif im.shape[2] == 2: # handle 2-channel images
im = np.ascontiguousarray(np.dstack((im, np.zeros_like(im[..., :1]))))
elif im.shape[2] > 3: # multispectral
im = np.ascontiguousarray(im[..., :3])
if self.pil: # TrueType text on a BGR copy of the image, all other drawing uses cv2/NumPy
self.im = np.array(im, order="C")
size = font_size or max(round(sum(im.shape[:2]) / 2 * 0.035), 12)
self.font = _font(FONT_UNICODE if non_ascii else font, size)
else: # draw in place on the caller's BGR image
assert im.is_contiguous() if input_is_tensor else im.data.contiguous, (
"Image not contiguous. Apply contiguous() or np.ascontiguousarray(im) to Annotator input images."
)
self.im = im if input_is_tensor or im.flags.writeable else im.copy()
self.tf = max(self.lw - 1, 1) # font thickness
self.sf = self.lw / 3 # font scale
# Pose
self.skeleton = [
[16, 14],
[14, 12],
[17, 15],
[15, 13],
[12, 13],
[6, 12],
[7, 13],
[6, 7],
[6, 8],
[7, 9],
[8, 10],
[9, 11],
[2, 3],
[1, 2],
[1, 3],
[2, 4],
[3, 5],
[4, 6],
[5, 7],
]
# COCO limb and keypoint colors from the pose palette, RGB to BGR
self.limb_color = colors.pose_palette[[0, 0, 0, 0, 7, 7, 7, 9, 9, 9, 9, 9, 16, 16, 16, 16, 16, 16, 16]][:, ::-1]
self.kpt_color = colors.pose_palette[[16, 16, 16, 16, 16, 9, 9, 9, 9, 9, 9, 0, 0, 0, 0, 0, 0]][:, ::-1]Method ultralytics.utils.plotting.Annotator._draw#
def _draw(self, x: int, y: int, patch, alpha: np.ndarray | None = None)Paste an (h, w, 3) patch, or blend a patch color through an (h, w) alpha, at (x, y) clipped to the image.
Args
| Name | Type | Description | Default |
|---|---|---|---|
x | int | required | |
y | int | required | |
patch | required | ||
alpha | np.ndarray | None | None |
ultralytics/utils/plotting.py
def _draw(self, x: int, y: int, patch, alpha: np.ndarray | None = None):
"""Paste an (h, w, 3) patch, or blend a `patch` color through an (h, w) alpha, at (x, y) clipped to the image."""
x, y, (h, w) = int(x), int(y), (patch if alpha is None else alpha).shape[:2]
x0, y0, x1, y1 = max(x, 0), max(y, 0), min(x + w, self.im.shape[1]), min(y + h, self.im.shape[0])
if x0 < x1 and y0 < y1:
roi, crop = self.im[y0:y1, x0:x1], (slice(y0 - y, y1 - y), slice(x0 - x, x1 - x))
if alpha is None:
roi[:] = patch[crop]
else:
roi[:] = roi + (np.array(patch[:3], np.float32) - roi) * alpha[crop][..., None] + 0.5Method ultralytics.utils.plotting.Annotator._to_numpy#
def _to_numpy(self)Move a tensor image to CPU only when a CPU drawing operation requires it.
ultralytics/utils/plotting.py
def _to_numpy(self):
"""Move a tensor image to CPU only when a CPU drawing operation requires it."""
if isinstance(self.im, torch.Tensor):
self.im = self.im.cpu().numpy()Method ultralytics.utils.plotting.Annotator.box_label#
def box_label(self, box, label: str = "", color: tuple = (128, 128, 128), txt_color: tuple | None = None)Draw a bounding box on an image with a given label.
Args
| Name | Type | Description | Default |
|---|---|---|---|
box | tuple | list | torch.Tensor | np.ndarray | The bounding box coordinates (x1, y1, x2, y2), or a list of polygon points with shape (n, 2), e.g. the 4 corners of an oriented box. | required |
label | str, optional | The text label to be displayed. | "" |
color | tuple, optional | The color of the box outline and label background. | (128, 128, 128) |
txt_color | tuple, optional | The color of the text; None picks navy or white by box color luma. | None |
Examples
>>> import cv2
>>> from ultralytics.utils.plotting import Annotator
>>> im0 = cv2.imread("test.png")
>>> annotator = Annotator(im0, line_width=10)
>>> annotator.box_label(box=[10, 20, 30, 40], label="person")ultralytics/utils/plotting.py
def box_label(self, box, label: str = "", color: tuple = (128, 128, 128), txt_color: tuple | None = None):
"""Draw a bounding box on an image with a given label.
Args:
box (tuple | list | torch.Tensor | np.ndarray): The bounding box coordinates (x1, y1, x2, y2), or a list of
polygon points with shape (n, 2), e.g. the 4 corners of an oriented box.
label (str, optional): The text label to be displayed.
color (tuple, optional): The color of the box outline and label background.
txt_color (tuple, optional): The color of the text; None picks navy or white by box color luma.
Examples:
>>> import cv2
>>> from ultralytics.utils.plotting import Annotator
>>> im0 = cv2.imread("test.png")
>>> annotator = Annotator(im0, line_width=10)
>>> annotator.box_label(box=[10, 20, 30, 40], label="person")
"""
self._to_numpy()
txt_color = self.get_txt_color(color, txt_color)
if isinstance(box, (torch.Tensor, np.ndarray)):
box = box.tolist()
multi_points = isinstance(box[0], list) # multiple points with shape (n, 2)
# Polygon labels sit above the enclosing box's top-left corner, like axis-aligned box labels
p1 = tuple(max(int(v), 0) for v in np.min(box, 0)) if multi_points else (int(box[0]), int(box[1]))
cv2.polylines(self.im, [np.asarray(box, dtype=int)], True, color, self.lw) if multi_points else cv2.rectangle(
self.im, p1, (int(box[2]), int(box[3])), color, thickness=self.lw, lineType=cv2.LINE_AA
)
if label:
if self.pil:
patch = _label_patch(self.font, label, tuple(color), tuple(txt_color))
h, w = patch.shape[0] - 1, patch.shape[1] - 1 # text height, width
else:
w, h = cv2.getTextSize(label, 0, fontScale=self.sf, thickness=self.tf)[0] # text width, height
h += 3 # add pixels to pad text
outside = p1[1] >= h # label fits outside box
if p1[0] > self.im.shape[1] - w: # shape is (h, w), check if label extend beyond right side of image
p1 = self.im.shape[1] - w, p1[1]
if self.pil:
self._draw(p1[0], p1[1] - h if outside else p1[1], patch)
else:
p2 = p1[0] + w, p1[1] - h if outside else p1[1] + h
cv2.rectangle(self.im, p1, p2, color, -1, cv2.LINE_AA) # filled
cv2.putText(
self.im,
label,
(p1[0], p1[1] - 2 if outside else p1[1] + h - 1),
0,
self.sf,
txt_color,
thickness=self.tf,
lineType=cv2.LINE_AA,
)Method ultralytics.utils.plotting.Annotator.depth_map#
def depth_map(self, depth: np.ndarray, alpha: float = 0.6, cmap: str = "jet", mode: str = "disparity") -> NoneRender a colorized depth map blended over the image.
Args
| Name | Type | Description | Default |
|---|---|---|---|
depth | np.ndarray | (H, W) depth in meters. | required |
alpha | float | Blend factor for the heatmap overlay. | 0.6 |
cmap | str | Colormap, one of "inferno", "jet", "spectral". See colorize_depth. | "jet" |
mode | str | "metric" or "disparity" normalization. See colorize_depth. | "disparity" |
ultralytics/utils/plotting.py
def depth_map(
self,
depth: np.ndarray,
alpha: float = 0.6,
cmap: str = "jet",
mode: str = "disparity",
) -> None:
"""Render a colorized depth map blended over the image.
Args:
depth (np.ndarray): (H, W) depth in meters.
alpha (float): Blend factor for the heatmap overlay.
cmap (str): Colormap, one of "inferno", "jet", "spectral". See `colorize_depth`.
mode (str): "metric" or "disparity" normalization. See `colorize_depth`.
"""
self._to_numpy()
heat = colorize_depth(depth, cmap=cmap, mode=mode) # BGR, matching the Annotator buffer convention
if heat.shape[:2] != self.im.shape[:2]:
heat = cv2.resize(heat, (self.im.shape[1], self.im.shape[0]))
self.im = cv2.addWeighted(self.im, 1 - alpha, heat, alpha, 0)Method ultralytics.utils.plotting.Annotator.get_bbox_dimension#
def get_bbox_dimension(bbox: tuple | list)Calculate the dimensions and area of a bounding box.
Args
| Name | Type | Description | Default |
|---|---|---|---|
bbox | tuple | list | Bounding box coordinates in the format (x_min, y_min, x_max, y_max). | required |
Returns
| Type | Description |
|---|---|
width (float) | Width of the bounding box. |
height (float) | Height of the bounding box. |
area (float) | Area enclosed by the bounding box. |
Examples
>>> from ultralytics.utils.plotting import Annotator
>>> Annotator.get_bbox_dimension(bbox=[10, 20, 30, 40])
(20, 20, 400)ultralytics/utils/plotting.py
@staticmethod
def get_bbox_dimension(bbox: tuple | list):
"""Calculate the dimensions and area of a bounding box.
Args:
bbox (tuple | list): Bounding box coordinates in the format (x_min, y_min, x_max, y_max).
Returns:
width (float): Width of the bounding box.
height (float): Height of the bounding box.
area (float): Area enclosed by the bounding box.
Examples:
>>> from ultralytics.utils.plotting import Annotator
>>> Annotator.get_bbox_dimension(bbox=[10, 20, 30, 40])
(20, 20, 400)
"""
x_min, y_min, x_max, y_max = bbox
width = x_max - x_min
height = y_max - y_min
return width, height, width * heightMethod ultralytics.utils.plotting.Annotator.get_txt_color#
def get_txt_color(self, color: tuple = (128, 128, 128), txt_color: tuple | None = None) -> tupleAssign text color based on background color.
Args
| Name | Type | Description | Default |
|---|---|---|---|
color | tuple, optional | The BGR background color of the rectangle for text. | (128, 128, 128) |
txt_color | tuple, optional | Explicit text color; None picks navy or white by background luma. | None |
Returns
| Type | Description |
|---|---|
tuple | Text color for label. |
Examples
>>> import cv2
>>> from ultralytics.utils.plotting import Annotator
>>> im0 = cv2.imread("test.png")
>>> annotator = Annotator(im0, line_width=10)
>>> annotator.get_txt_color(color=(104, 31, 17)) # return (255, 255, 255)ultralytics/utils/plotting.py
def get_txt_color(self, color: tuple = (128, 128, 128), txt_color: tuple | None = None) -> tuple:
"""Assign text color based on background color.
Args:
color (tuple, optional): The BGR background color of the rectangle for text.
txt_color (tuple, optional): Explicit text color; None picks navy or white by background luma.
Returns:
(tuple): Text color for label.
Examples:
>>> import cv2
>>> from ultralytics.utils.plotting import Annotator
>>> im0 = cv2.imread("test.png")
>>> annotator = Annotator(im0, line_width=10)
>>> annotator.get_txt_color(color=(104, 31, 17)) # return (255, 255, 255)
"""
if txt_color is not None:
return txt_color
b, g, r = color
return (104, 31, 17) if 0.299 * r + 0.587 * g + 0.114 * b > 150 else (255, 255, 255) # navy #111f68 or whiteMethod ultralytics.utils.plotting.Annotator.kpts#
def kpts(
self,
kpts,
shape: tuple = (640, 640),
radius: int | None = None,
kpt_line: bool = True,
conf_thres: float = 0.25,
kpt_color: tuple | None = None,
)Plot keypoints on the image.
Args
| Name | Type | Description | Default |
|---|---|---|---|
kpts | torch.Tensor | np.ndarray | Keypoints with shape [nkpt, 2] (x, y) or [nkpt, 3] (x, y, confidence). | required |
shape | tuple, optional | Image shape (h, w). Currently unused. | (640, 640) |
radius | int, optional | Keypoint radius. Defaults to the annotator line width. | None |
kpt_line | bool, optional | Draw lines between keypoints. | True |
conf_thres | float, optional | Confidence threshold. | 0.25 |
kpt_color | tuple, optional | Color for all keypoints and limbs, overriding the default pose palette. | None |
kpt_line=Truecurrently only supports human pose plotting.- Modifies self.im in-place.
ultralytics/utils/plotting.py
def kpts(
self,
kpts,
shape: tuple = (640, 640),
radius: int | None = None,
kpt_line: bool = True,
conf_thres: float = 0.25,
kpt_color: tuple | None = None,
):
"""Plot keypoints on the image.
Args:
kpts (torch.Tensor | np.ndarray): Keypoints with shape [nkpt, 2] (x, y) or [nkpt, 3] (x, y, confidence).
shape (tuple, optional): Image shape (h, w). Currently unused.
radius (int, optional): Keypoint radius. Defaults to the annotator line width.
kpt_line (bool, optional): Draw lines between keypoints.
conf_thres (float, optional): Confidence threshold.
kpt_color (tuple, optional): Color for all keypoints and limbs, overriding the default pose palette.
Notes:
- `kpt_line=True` currently only supports human pose plotting.
- Modifies self.im in-place.
"""
radius = radius if radius is not None else self.lw
self._to_numpy()
nkpt, ndim = kpts.shape
is_pose = nkpt == 17 and ndim in {2, 3}
kpt_line &= is_pose # `kpt_line=True` for now only supports human pose plotting
for i, k in enumerate(kpts):
color_k = kpt_color or (self.kpt_color[i].tolist() if is_pose else colors(i, True))
x_coord, y_coord = k[0], k[1]
if len(k) == 3:
if k[2] < conf_thres:
continue
elif x_coord == 0 and y_coord == 0: # (0, 0) marks a missing keypoint when there is no confidence channel
continue
cv2.circle(self.im, (int(x_coord), int(y_coord)), radius, color_k, -1, lineType=cv2.LINE_AA)
if kpt_line:
ndim = kpts.shape[-1]
for i, sk in enumerate(self.skeleton):
pos1 = (int(kpts[(sk[0] - 1), 0]), int(kpts[(sk[0] - 1), 1]))
pos2 = (int(kpts[(sk[1] - 1), 0]), int(kpts[(sk[1] - 1), 1]))
if ndim == 3:
conf1 = kpts[(sk[0] - 1), 2]
conf2 = kpts[(sk[1] - 1), 2]
if conf1 < conf_thres or conf2 < conf_thres:
continue
elif not (kpts[sk[0] - 1, :2].any() and kpts[sk[1] - 1, :2].any()): # (0, 0) marks a missing keypoint
continue
if min(pos1 + pos2) < 0:
continue
cv2.line(
self.im,
pos1,
pos2,
kpt_color or self.limb_color[i].tolist(),
thickness=int(np.ceil(self.lw / 2)),
lineType=cv2.LINE_AA,
)Method ultralytics.utils.plotting.Annotator.masks#
def masks(self, masks, colors, alpha: float = 0.5)Plot masks on image.
Args
| Name | Type | Description | Default |
|---|---|---|---|
masks | torch.Tensor | np.ndarray | list | Predicted masks with shape [n, h, w], or a list of box-local ((x1, y1, x2, y2), mask) pairs at the image size. | required |
colors | list[list[int]] | BGR colors for predicted masks, [[b, g, r] * n], matching self.im. | required |
alpha | float, optional | Mask transparency: 0.0 fully transparent, 1.0 opaque. | 0.5 |
ultralytics/utils/plotting.py
def masks(self, masks, colors, alpha: float = 0.5):
"""Plot masks on image.
Args:
masks (torch.Tensor | np.ndarray | list): Predicted masks with shape [n, h, w], or a list of box-local ((x1,
y1, x2, y2), mask) pairs at the image size.
colors (list[list[int]]): BGR colors for predicted masks, [[b, g, r] * n], matching `self.im`.
alpha (float, optional): Mask transparency: 0.0 fully transparent, 1.0 opaque.
"""
local = isinstance(masks, list) # box-local (box, mask) pairs at the image size
if isinstance(masks, np.ndarray):
self._to_numpy()
overlay = self.im.copy()
for i, mask in enumerate(masks):
overlay[mask.astype(bool)] = colors[i]
self.im = cv2.addWeighted(self.im, 1 - alpha, overlay, alpha, 0)
elif len(masks):
tensor_image = isinstance(self.im, torch.Tensor)
device = self.im.device if tensor_image else (torch.as_tensor(masks[0][1]) if local else masks).device
colors = torch.tensor(colors, device=device, dtype=torch.float32) / 255.0 # shape(n,3)
colors = colors[:, None, None] * alpha # shape(n,1,1,3), premultiplied by alpha
mcs = torch.zeros((*self.im.shape[:2], 3), device=device, dtype=torch.float32) # shape(h,w,3)
inv_alpha_masks = torch.ones((*self.im.shape[:2], 1), device=device, dtype=torch.float32) # shape(h,w,1)
if local: # blend each mask inside its box, costing the boxes' pixels rather than n * h * w
for ((x1, y1, x2, y2), m), color in zip(masks, colors):
m = torch.as_tensor(m, device=device)[..., None]
mcs[y1:y2, x1:x2] = torch.maximum(mcs[y1:y2, x1:x2], m * color)
inv_alpha_masks[y1:y2, x1:x2] *= 1 - m * alpha
else:
# Use scale_masks to properly remove padding and upsample, convert bool to float first
masks = ops.scale_masks(masks[None].to(device).float(), self.im.shape[:2])[0] > 0.5
masks = masks.unsqueeze(3) # shape(n,h,w,1)
# Reduce in row bands so the (n,h,w,*) intermediates never span the full height
bands = max(1, masks.numel() * 12 // 2**23) # 12 bytes per mask element downstream, 8 MB per band
for m, mcs_band, inv_band in zip(masks.chunk(bands, 1), mcs.chunk(bands), inv_alpha_masks.chunk(bands)):
torch.amax(m * colors, 0, out=mcs_band)
torch.prod(1 - m * alpha, 0, out=inv_band)
im = (self.im if tensor_image else torch.from_numpy(self.im)).to(device).float() / 255.0
im = ((im * inv_alpha_masks + mcs) * 255).byte()
self.im[:] = im if tensor_image else im.cpu().numpy()Method ultralytics.utils.plotting.Annotator.rectangle#
def rectangle(self, xy, fill=None, outline=None, width: int = 1)Add a rectangle with optional fill and outline colors to the image.
Args
| Name | Type | Description | Default |
|---|---|---|---|
xy | required | ||
fill | None | ||
outline | None | ||
width | int | 1 |
ultralytics/utils/plotting.py
def rectangle(self, xy, fill=None, outline=None, width: int = 1):
"""Add a rectangle with optional fill and outline colors to the image."""
self._to_numpy()
for c, t in ((fill, -1), (outline, width)):
if c is not None:
cv2.rectangle(self.im, tuple(map(int, xy[:2])), tuple(map(int, xy[2:])), c, t)Method ultralytics.utils.plotting.Annotator.result#
def result(self, pil=False)Return annotated image as a BGR NumPy array, or as an RGB PIL image if pil is True.
ultralytics/utils/plotting.py
def result(self, pil=False):
"""Return annotated image as a BGR NumPy array, or as an RGB PIL image if `pil` is True."""
self._to_numpy()
return Image.fromarray(cv2.cvtColor(self.im, cv2.COLOR_BGR2RGB)) if pil else self.im # self.im is in BGRMethod ultralytics.utils.plotting.Annotator.save#
def save(self, filename: str = "image.jpg")Save the annotated image to 'filename'.
Args
| Name | Type | Description | Default |
|---|---|---|---|
filename | str | "image.jpg" |
ultralytics/utils/plotting.py
def save(self, filename: str = "image.jpg"):
"""Save the annotated image to 'filename'."""
Path(filename).parent.mkdir(parents=True, exist_ok=True)
cv2.imwrite(filename, self.result())Method ultralytics.utils.plotting.Annotator.semantic_mask#
def semantic_mask(self, mask, alpha: float = 0.5, ignore_index: int = 255, custom: list[tuple] | None = None)Plot a semantic segmentation mask on the image.
Args
| Name | Type | Description | Default |
|---|---|---|---|
mask | np.ndarray | Semantic mask with shape [h, w] containing integer class indices. | required |
alpha | float, optional | Mask transparency: 0.0 fully transparent, 1.0 opaque. | 0.5 |
ignore_index | int, optional | Class index to ignore (e.g., 255 for void/ignore). | 255 |
custom | list[tuple], optional | RGB class colors overriding the palette, see Colors. | None |
ultralytics/utils/plotting.py
def semantic_mask(self, mask, alpha: float = 0.5, ignore_index: int = 255, custom: list[tuple] | None = None):
"""Plot a semantic segmentation mask on the image.
Args:
mask (np.ndarray): Semantic mask with shape [h, w] containing integer class indices.
alpha (float, optional): Mask transparency: 0.0 fully transparent, 1.0 opaque.
ignore_index (int, optional): Class index to ignore (e.g., 255 for void/ignore).
custom (list[tuple], optional): RGB class colors overriding the palette, see `Colors`.
"""
self._to_numpy()
mask = np.asarray(mask)
mask = mask if mask.dtype.kind in "iu" else mask.astype(int) # class IDs index the palette directly
overlay = np.array([colors(i, True, custom) for i in range(int(mask.max(initial=0)) + 1)], self.im.dtype)[mask]
overlay[mask == ignore_index] = self.im[mask == ignore_index]
self.im = cv2.addWeighted(self.im, 1 - alpha, overlay, alpha, 0)Method ultralytics.utils.plotting.Annotator.show#
def show(self, title: str | None = None)Show the annotated image.
Args
| Name | Type | Description | Default |
|---|---|---|---|
title | str | None | None |
ultralytics/utils/plotting.py
def show(self, title: str | None = None):
"""Show the annotated image."""
im = self.result(pil=True)
if IS_COLAB or IS_KAGGLE: # cannot use IS_JUPYTER as it runs for all IPython environments
try:
display(im) # noqa - display() function only available in ipython environments
except ImportError as e:
LOGGER.warning(f"Unable to display image in Jupyter notebooks: {e}")
else:
im.show(title=title)Method ultralytics.utils.plotting.Annotator.text#
def text(self, xy, text: str, txt_color: tuple = (255, 255, 255), anchor: str = "top", box_color: tuple = ())Add text to an image using PIL or cv2.
Args
| Name | Type | Description | Default |
|---|---|---|---|
xy | tuple | list | Top-left (x, y) coordinates for text placement. | required |
text | str | Text to be drawn. | required |
txt_color | tuple, optional | Text color. | (255, 255, 255) |
anchor | str, optional | Text anchor position ('top' or 'bottom'), used by PIL drawing only. | "top" |
box_color | tuple, optional | Box background color with optional alpha. | () |
ultralytics/utils/plotting.py
def text(self, xy, text: str, txt_color: tuple = (255, 255, 255), anchor: str = "top", box_color: tuple = ()):
"""Add text to an image using PIL or cv2.
Args:
xy (tuple | list): Top-left (x, y) coordinates for text placement.
text (str): Text to be drawn.
txt_color (tuple, optional): Text color.
anchor (str, optional): Text anchor position ('top' or 'bottom'), used by PIL drawing only.
box_color (tuple, optional): Box background color with optional alpha.
"""
self._to_numpy()
if self.pil:
masks = [_text_mask(self.font, line) for line in text.split("\n")] # per line, raqm rejects newlines
x, y = xy
if anchor == "bottom": # start y from font bottom
y += 1 - max(m.shape[0] for m in masks)
for m in masks:
if box_color: # background box per line, alpha from an RGBA color
a = box_color[3] / 255 if len(box_color) > 3 else 1.0
self._draw(x, y, box_color, np.full((m.shape[0] + 2, m.shape[1] + 2), a, np.float32))
self._draw(x, y, txt_color, m)
y += m.shape[0]
else:
if box_color:
w, h = cv2.getTextSize(text, 0, fontScale=self.sf, thickness=self.tf)[0]
h += 3 # add pixels to pad text
outside = xy[1] >= h # label fits outside box
p2 = xy[0] + w, xy[1] - h if outside else xy[1] + h
cv2.rectangle(self.im, xy, p2, box_color, -1, cv2.LINE_AA) # filled
cv2.putText(self.im, text, xy, 0, self.sf, txt_color, thickness=self.tf, lineType=cv2.LINE_AA)Function ultralytics.utils.plotting._gaussian_filter1d#
def _gaussian_filter1d(y, sigma: int = 3, truncate: float = 4.0) -> np.ndarraySmooth a 1D array with a Gaussian kernel (NumPy replacement for scipy.ndimage.gaussian_filter1d).
Args
| Name | Type | Description | Default |
|---|---|---|---|
y | np.ndarray | Input 1D array to smooth. | required |
sigma | int | Standard deviation of the Gaussian kernel. | 3 |
truncate | float | Truncate the kernel at this many standard deviations. | 4.0 |
Returns
| Type | Description |
|---|---|
np.ndarray | Smoothed 1D array with the same length as the input. |
ultralytics/utils/plotting.py
def _gaussian_filter1d(y, sigma: int = 3, truncate: float = 4.0) -> np.ndarray:
"""Smooth a 1D array with a Gaussian kernel (NumPy replacement for scipy.ndimage.gaussian_filter1d).
Args:
y (np.ndarray): Input 1D array to smooth.
sigma (int): Standard deviation of the Gaussian kernel.
truncate (float): Truncate the kernel at this many standard deviations.
Returns:
(np.ndarray): Smoothed 1D array with the same length as the input.
"""
y = np.asarray(y, dtype=float)
radius = int(truncate * sigma + 0.5)
kernel = np.exp(-0.5 * (np.arange(-radius, radius + 1) / sigma) ** 2)
kernel /= kernel.sum()
# scipy 'reflect' boundary mode is equivalent to NumPy 'symmetric'
return np.convolve(np.pad(y, radius, mode="symmetric"), kernel, mode="valid")Function ultralytics.utils.plotting._spectral_lut#
def _spectral_lut() -> np.ndarrayBuild the 256x1x3 BGR uint8 Spectral_r LUT for cv2.applyColorMap by linearly interpolating the anchors.
ultralytics/utils/plotting.py
def _spectral_lut() -> np.ndarray:
"""Build the 256x1x3 BGR uint8 Spectral_r LUT for cv2.applyColorMap by linearly interpolating the anchors."""
xs = np.linspace(0.0, 10.0, 256)
i = np.clip(xs.astype(int), 0, 9)
f = (xs - i)[:, None]
rgb = _SPECTRAL_R_ANCHORS[i] * (1.0 - f) + _SPECTRAL_R_ANCHORS[i + 1] * f
return rgb.round().astype(np.uint8)[:, ::-1].reshape(256, 1, 3) # RGB→BGR for cv2 conventionFunction ultralytics.utils.plotting.colorize_depth#
def colorize_depth(
depth: np.ndarray,
vmin: float | None = None,
vmax: float | None = None,
cmap: str = "jet",
mode: str = "disparity",
) -> np.ndarrayMap a (H, W) metric-depth array to a BGR uint8 colorized image, invalid (<= 0) pixels black.
Args
| Name | Type | Description | Default |
|---|---|---|---|
depth | np.ndarray | (H, W) depth in meters. | required |
vmin | float, optional | Lower bound of the color range; defaults to the valid-pixel minimum (metric mode) or the 2nd disparity percentile (disparity mode). | None |
vmax | float, optional | Upper bound of the color range; defaults to the valid-pixel maximum (metric mode) or the 98th disparity percentile (disparity mode). | None |
cmap | str | Colormap, one of "inferno", "jet", "spectral" (matplotlib Spectral_r, near = warm). | "jet" |
mode | str | "metric" normalizes depth linearly; "disparity" normalizes inverse depth (1/d) between the 2nd and 98th percentiles for the DepthAnything look (near objects warm, robust to far outliers). | "disparity" |
Returns
| Type | Description |
|---|---|
np.ndarray | (H, W, 3) BGR uint8 colorized depth. |
ultralytics/utils/plotting.py
def colorize_depth(
depth: np.ndarray,
vmin: float | None = None,
vmax: float | None = None,
cmap: str = "jet",
mode: str = "disparity",
) -> np.ndarray:
"""Map a (H, W) metric-depth array to a BGR uint8 colorized image, invalid (<= 0) pixels black.
Args:
depth (np.ndarray): (H, W) depth in meters.
vmin (float, optional): Lower bound of the color range; defaults to the valid-pixel minimum (metric mode) or the
2nd disparity percentile (disparity mode).
vmax (float, optional): Upper bound of the color range; defaults to the valid-pixel maximum (metric mode) or the
98th disparity percentile (disparity mode).
cmap (str): Colormap, one of "inferno", "jet", "spectral" (matplotlib Spectral_r, near = warm).
mode (str): "metric" normalizes depth linearly; "disparity" normalizes inverse depth (1/d) between the 2nd and
98th percentiles for the DepthAnything look (near objects warm, robust to far outliers).
Returns:
(np.ndarray): (H, W, 3) BGR uint8 colorized depth.
"""
d = np.asarray(depth, dtype=np.float32)
valid = d > 0
v = np.where(valid, 1.0 / np.where(valid, d, 1.0), 0.0) if mode == "disparity" else d
if vmin is None or vmax is None:
pool = v[valid]
if mode == "disparity": # ~64k evenly spaced valid values estimate the percentiles, also for sparse maps
lo, hi = np.percentile(pool[:: max(1, pool.size // 2**16)], (2, 98)) if pool.size else (0.0, 1.0)
else:
lo, hi = (float(pool.min()), float(pool.max())) if pool.size else (0.0, 1.0)
vmin = lo if vmin is None else vmin
vmax = hi if vmax is None else vmax
if vmax <= vmin:
vmax = vmin + 1e-6
dn = np.clip((v - vmin) / (vmax - vmin), 0.0, 1.0)
idx = (dn * 255).astype(np.uint8)
lut = _SPECTRAL_LUT if cmap == "spectral" else None
color = cv2.applyColorMap(idx, lut) if lut is not None else cv2.applyColorMap(idx, _DEPTH_CMAPS[cmap]) # BGR
color[~valid] = 0
return colorFunction ultralytics.utils.plotting._arabic_forms#
def _arabic_forms() -> dict | NoneMap Arabic letters and lam-alef pairs to their presentation forms, or return None if Pillow shapes with Raqm.
ultralytics/utils/plotting.py
@lru_cache
def _arabic_forms() -> dict | None:
"""Map Arabic letters and lam-alef pairs to their presentation forms, or return None if Pillow shapes with Raqm."""
if features.check("raqm"):
return None
forms = {}
for c in map(chr, [*range(0xFE70, 0xFF00), *range(0xFB50, 0xFE00)]): # Arabic Presentation Forms-B, then -A
tag, *codes = unicodedata.decomposition(c).split() or [""]
if tag in {"<isolated>", "<final>", "<initial>", "<medial>"} and (len(codes) == 1 or "\ufef5" <= c <= "\ufefc"):
forms.setdefault("".join(chr(int(x, 16)) for x in codes), {}).setdefault(tag, c)
return formsFunction ultralytics.utils.plotting._rtl#
def _rtl(text: str) -> strJoin Arabic letters and reverse right-to-left text into visual order for Pillow builds without Raqm.
Args
| Name | Type | Description | Default |
|---|---|---|---|
text | str | required |
ultralytics/utils/plotting.py
def _rtl(text: str) -> str:
"""Join Arabic letters and reverse right-to-left text into visual order for Pillow builds without Raqm."""
forms = None if text.isascii() else _arabic_forms() # O(1) skip for ASCII labels
if forms is None or not any(unicodedata.bidirectional(c) in {"R", "AL"} for c in text):
return text
units = re.findall("|".join(k for k in forms if len(k) == 2) + "|.", text, re.DOTALL) # lam-alef is one glyph
joins = ["<medial>" in forms.get(u, {}) for u in units] # dual-joining letters connect to the next letter
for i, u in enumerate(units):
if f := forms.get(u):
p, n = i > 0 and joins[i - 1], joins[i] and i + 1 < len(units) and units[i + 1] in forms
units[i] = f.get(("<medial>" if n else "<final>") if p else ("<initial>" if n else "<isolated>"), u)
# Reverse into visual order, keeping left-to-right runs such as confidences in reading order
return re.sub(r"[0-9A-Za-z\u0660-\u0669\u06f0-\u06f9.,:%+-]+", lambda m: m[0][::-1], "".join(units)[::-1])Function ultralytics.utils.plotting._font#
def _font(name: str, size: int)Load a TrueType font once per name and size, falling back to PIL's default font.
Args
| Name | Type | Description | Default |
|---|---|---|---|
name | str | required | |
size | int | required |
ultralytics/utils/plotting.py
@lru_cache(maxsize=8)
def _font(name: str, size: int):
"""Load a TrueType font once per name and size, falling back to PIL's default font."""
try:
font = ImageFont.truetype(str(check_font(name)), size)
except Exception:
font = ImageFont.load_default()
if check_version(pil_version, "9.2.0"): # Pillow 10 removed getsize
font.getsize = lambda x: font.getbbox(x)[2:4] # text width, height
return fontFunction ultralytics.utils.plotting._text_mask#
def _text_mask(font, text: str) -> np.ndarrayRasterize text as an (h, w) float32 alpha mask anchored at its origin.
Args
| Name | Type | Description | Default |
|---|---|---|---|
font | required | ||
text | str | required |
ultralytics/utils/plotting.py
def _text_mask(font, text: str) -> np.ndarray:
"""Rasterize text as an (h, w) float32 alpha mask anchored at its origin."""
text = _rtl(text)
im = Image.new("L", font.getsize(text))
ImageDraw.Draw(im).text((0, 0), text, fill=255, font=font)
return np.asarray(im, np.float32) / 255Function ultralytics.utils.plotting._label_patch#
def _label_patch(font, text: str, color: tuple, txt_color: tuple) -> np.ndarrayReturn an opaque (h + 1, w + 1, 3) uint8 box label, cached so repeated labels skip FreeType and blending.
Args
| Name | Type | Description | Default |
|---|---|---|---|
font | required | ||
text | str | required | |
color | tuple | required | |
txt_color | tuple | required |
ultralytics/utils/plotting.py
@lru_cache(maxsize=256)
def _label_patch(font, text: str, color: tuple, txt_color: tuple) -> np.ndarray:
"""Return an opaque (h + 1, w + 1, 3) uint8 box label, cached so repeated labels skip FreeType and blending."""
text = _rtl(text)
w, h = font.getsize(text)
im = Image.new("RGB", (w + 1, h + 1), color) # channels pass through in the caller's (BGR) order
ImageDraw.Draw(im).text((0, 0), text, fill=txt_color, font=font)
return np.asarray(im)Function ultralytics.utils.plotting.plot_labels#
def plot_labels(boxes, cls, names: dict[int, str] | None = None, save_dir=Path(""), on_plot=None)Plot training labels including class histograms and box statistics.
Args
| Name | Type | Description | Default |
|---|---|---|---|
boxes | np.ndarray | Normalized bounding boxes with shape (N, 4) in format [x_center, y_center, width, height]. | required |
cls | np.ndarray | Class indices. | required |
names | dict[int, str], optional | Dictionary mapping class indices to class names, used for x-axis tick labels when there are fewer than 30 classes. | None |
save_dir | Path, optional | Directory to save the plot. | Path("") |
on_plot | Callable, optional | Function to call after plot is saved. | None |
ultralytics/utils/plotting.py
@TryExcept()
@plt_settings()
def plot_labels(boxes, cls, names: dict[int, str] | None = None, save_dir=Path(""), on_plot=None):
"""Plot training labels including class histograms and box statistics.
Args:
boxes (np.ndarray): Normalized bounding boxes with shape (N, 4) in format [x_center, y_center, width, height].
cls (np.ndarray): Class indices.
names (dict[int, str], optional): Dictionary mapping class indices to class names, used for x-axis tick labels
when there are fewer than 30 classes.
save_dir (Path, optional): Directory to save the plot.
on_plot (Callable, optional): Function to call after plot is saved.
"""
import matplotlib.pyplot as plt # scope for faster 'import ultralytics'
import polars
from matplotlib.colors import LinearSegmentedColormap
# Plot dataset labels
LOGGER.info(f"Plotting labels to {save_dir / 'labels.jpg'}... ")
nc = int(cls.max() + 1) # number of classes
boxes = boxes[:1000000] # limit to 1M boxes
x = polars.DataFrame(boxes, schema=["x", "y", "width", "height"])
# Matplotlib labels
subplot_3_4_color = LinearSegmentedColormap.from_list("white_blue", ["white", "blue"])
ax = plt.subplots(2, 2, figsize=(8, 8), tight_layout=True)[1].ravel()
y = ax[0].hist(cls, bins=np.linspace(0, nc, nc + 1) - 0.5, rwidth=0.8)
for i in range(nc):
y[2].patches[i].set_color([x / 255 for x in colors(i)])
ax[0].set_ylabel("instances")
if names and len(names) < 30:
ax[0].set_xticks(range(len(names)))
ax[0].set_xticklabels(list(names.values()), rotation=90, fontsize=10)
ax[0].bar_label(y[2])
else:
ax[0].set_xlabel("classes")
boxes = np.column_stack([0.5 - boxes[:, 2:4] / 2, 0.5 + boxes[:, 2:4] / 2]) * 1000
img = Image.fromarray(np.ones((1000, 1000, 3), dtype=np.uint8) * 255)
for class_id, box in zip(cls[:500], boxes[:500]):
ImageDraw.Draw(img).rectangle(box.tolist(), width=1, outline=colors(class_id)) # plot
ax[1].imshow(img)
ax[1].axis("off")
ax[2].hist2d(x["x"], x["y"], bins=50, cmap=subplot_3_4_color)
ax[2].set_xlabel("x")
ax[2].set_ylabel("y")
ax[3].hist2d(x["width"], x["height"], bins=50, cmap=subplot_3_4_color)
ax[3].set_xlabel("width")
ax[3].set_ylabel("height")
for a in (0, 1, 2, 3):
for s in ("top", "right", "left", "bottom"):
ax[a].spines[s].set_visible(False)
fname = save_dir / "labels.jpg"
plt.savefig(fname, dpi=200)
plt.close()
if on_plot:
on_plot(fname)Function ultralytics.utils.plotting.save_one_box#
def save_one_box(
xyxy,
im,
file: Path = Path("im.jpg"),
gain: float = 1.02,
pad: int = 10,
square: bool = False,
BGR: bool = False,
save: bool = True,
)Save image crop as {file} with crop size multiple {gain} and {pad} pixels. Save and/or return crop.
This function takes a bounding box and an image, and then saves a cropped portion of the image according to the bounding box. Optionally, the crop can be squared, and the function allows for gain and padding adjustments to the bounding box. Multispectral JPEGs preview the first three channels; returned crops retain all channels, in their original order with BGR=True or reversed otherwise.
Args
| Name | Type | Description | Default |
|---|---|---|---|
xyxy | torch.Tensor | list | A bounding box in xyxy format, or (4, 2) OBB corners for a rotation-aligned crop. | required |
im | np.ndarray | The input BGR image with shape (H, W, C) or (H, W). | required |
file | Path, optional | Output path, saved as JPEG with an incremented name if the JPEG path already exists. | Path("im.jpg") |
gain | float, optional | A multiplicative factor to increase the size of the bounding box. | 1.02 |
pad | int, optional | The number of pixels to add to the width and height of the bounding box. | 10 |
square | bool, optional | If True, the bounding box will be transformed into a square. | False |
BGR | bool, optional | If True, the image will be returned in BGR format, otherwise in RGB. | False |
save | bool, optional | If True, the cropped image will be saved to disk. | True |
Returns
| Type | Description |
|---|---|
np.ndarray | The cropped image. |
Examples
>>> from pathlib import Path
>>> import cv2
>>> from ultralytics.utils.plotting import save_one_box
>>> xyxy = [50, 50, 150, 150]
>>> im = cv2.imread("image.jpg")
>>> cropped_im = save_one_box(xyxy, im, file=Path("cropped.jpg"), square=True)ultralytics/utils/plotting.py
def save_one_box(
xyxy,
im,
file: Path = Path("im.jpg"),
gain: float = 1.02,
pad: int = 10,
square: bool = False,
BGR: bool = False,
save: bool = True,
):
"""Save image crop as {file} with crop size multiple {gain} and {pad} pixels. Save and/or return crop.
This function takes a bounding box and an image, and then saves a cropped portion of the image according to the
bounding box. Optionally, the crop can be squared, and the function allows for gain and padding adjustments to the
bounding box. Multispectral JPEGs preview the first three channels; returned crops retain all channels, in their
original order with BGR=True or reversed otherwise.
Args:
xyxy (torch.Tensor | list): A bounding box in xyxy format, or (4, 2) OBB corners for a rotation-aligned crop.
im (np.ndarray): The input BGR image with shape (H, W, C) or (H, W).
file (Path, optional): Output path, saved as JPEG with an incremented name if the JPEG path already exists.
gain (float, optional): A multiplicative factor to increase the size of the bounding box.
pad (int, optional): The number of pixels to add to the width and height of the bounding box.
square (bool, optional): If True, the bounding box will be transformed into a square.
BGR (bool, optional): If True, the image will be returned in BGR format, otherwise in RGB.
save (bool, optional): If True, the cropped image will be saved to disk.
Returns:
(np.ndarray): The cropped image.
Examples:
>>> from pathlib import Path
>>> import cv2
>>> from ultralytics.utils.plotting import save_one_box
>>> xyxy = [50, 50, 150, 150]
>>> im = cv2.imread("image.jpg")
>>> cropped_im = save_one_box(xyxy, im, file=Path("cropped.jpg"), square=True)
"""
xyxy = torch.as_tensor(xyxy, dtype=torch.float32) # float so integer boxes keep fractional centers and gain/pad
if xyxy.shape[-2:] == (4, 2): # OBB corners: warp the box upright at its size, then crop the whole warp below
p = xyxy.reshape(4, 2).cpu().numpy()
p = np.roll(p, 2 - int((np.roll(p[:, 0], 1) - p[:, 0]).argmax()), 0) # top edge spanning furthest right
w, h = (max(round(float(np.linalg.norm(p[0] - p[i]))), 1) for i in (3, 1))
M = cv2.getAffineTransform(p[:3], np.float32([[w, h], [w, 0], [0, 0]]))
im, xyxy = cv2.warpAffine(im, M, (w, h)).reshape(h, w, -1), torch.tensor([0.0, 0.0, w, h])
if gain == 1 and pad == 0 and not square and not save: # the crop is the whole warp: skip the box math
return im if BGR or im.shape[2] == 1 else im[..., ::-1]
b = ops.xyxy2xywh(xyxy.view(-1, 4)) # boxes
if square:
b[:, 2:] = b[:, 2:].max(1)[0].unsqueeze(1) # attempt rectangle to square
b[:, 2:] = b[:, 2:] * gain + pad # box wh * gain + pad
xyxy = ops.xywh2xyxy(b).long()
xyxy = ops.clip_boxes(xyxy, im.shape)
grayscale = im.ndim == 2 or im.shape[2] == 1 # grayscale image
crop = im[int(xyxy[0, 1]) : int(xyxy[0, 3]), int(xyxy[0, 0]) : int(xyxy[0, 2])]
if save:
file.parent.mkdir(parents=True, exist_ok=True) # make directory
f = str(increment_path(file.with_suffix(".jpg")))
# cv2.imwrite(f, crop) # save BGR, https://github.com/ultralytics/yolov5/issues/7007 chroma subsampling issue
im_save = crop.reshape(crop.shape[:2]) if grayscale else crop[..., :3][..., ::-1]
Image.fromarray(im_save).save(f, quality=95, subsampling=0) # save RGB
return crop if BGR or grayscale else crop[..., ::-1]Function ultralytics.utils.plotting.plot_images#
def plot_images(
labels: dict[str, Any],
images: torch.Tensor | np.ndarray | None = None,
paths: list[str] | None = None,
fname: str | Path = "images.jpg",
names: dict[int, str] | None = None,
class_colors: list[str] | None = None,
on_plot: Callable | None = None,
max_size: int = 1920,
max_subplots: int = 16,
save: bool = True,
conf_thres: float = 0.25,
show_labels: bool = True,
show_conf: bool = True,
) -> np.ndarray | NonePlot image grid with labels, bounding boxes, masks, and keypoints.
Args
| Name | Type | Description | Default |
|---|---|---|---|
labels | dict[str, Any] | Dictionary containing detection data with keys like 'cls', 'bboxes', 'conf', 'masks', 'keypoints', 'semantic_mask', 'depth', 'batch_idx', 'img'. 'img', if present, overrides images. | required |
images | torch.Tensor | np.ndarray | None | Batch of images to plot. Shape: (batch_size, channels, height, width). | None |
paths | list[str] | None | List of file paths for each image in the batch. | None |
fname | str | Path | Output filename for the plotted image grid. | "images.jpg" |
names | dict[int, str] | None | Dictionary mapping class indices to class names. | None |
class_colors | list[str] | None | Dataset colors, "#rrggbb" strings indexed by class id overriding the palette. | None |
on_plot | Callable | None | Callback function to be called after saving the plot. | None |
max_size | int | Maximum size of the output image grid. | 1920 |
max_subplots | int | Maximum number of subplots in the image grid. | 16 |
save | bool | Whether to save the plotted image grid to a file. | True |
conf_thres | float | Confidence threshold for displaying detections. | 0.25 |
show_labels | bool | Whether to display class labels. | True |
show_conf | bool | Whether to display confidence values. | True |
Returns
| Type | Description |
|---|---|
np.ndarray | None | Plotted image grid as a numpy array if save is False, None otherwise. Runs in a background thread and returns the threading.Thread instead unless called with threaded=False. |
This function supports both tensor and numpy array inputs. It will automatically convert tensor inputs to numpy arrays for processing.
Channel Support:
- 1 channel: Grayscale
- 2 channels: Third channel added as zeros
- 3 channels: Used as-is (standard RGB)
- 4+ channels: Cropped to first 3 channels
ultralytics/utils/plotting.py
@threaded
def plot_images(
labels: dict[str, Any],
images: torch.Tensor | np.ndarray | None = None,
paths: list[str] | None = None,
fname: str | Path = "images.jpg",
names: dict[int, str] | None = None,
class_colors: list[str] | None = None,
on_plot: Callable | None = None,
max_size: int = 1920,
max_subplots: int = 16,
save: bool = True,
conf_thres: float = 0.25,
show_labels: bool = True,
show_conf: bool = True,
) -> np.ndarray | None:
"""Plot image grid with labels, bounding boxes, masks, and keypoints.
Args:
labels (dict[str, Any]): Dictionary containing detection data with keys like 'cls', 'bboxes', 'conf', 'masks',
'keypoints', 'semantic_mask', 'depth', 'batch_idx', 'img'. 'img', if present, overrides `images`.
images (torch.Tensor | np.ndarray | None): Batch of images to plot. Shape: (batch_size, channels, height,
width).
paths (list[str] | None): List of file paths for each image in the batch.
fname (str | Path): Output filename for the plotted image grid.
names (dict[int, str] | None): Dictionary mapping class indices to class names.
class_colors (list[str] | None): Dataset `colors`, "#rrggbb" strings indexed by class id overriding the palette.
on_plot (Callable | None): Callback function to be called after saving the plot.
max_size (int): Maximum size of the output image grid.
max_subplots (int): Maximum number of subplots in the image grid.
save (bool): Whether to save the plotted image grid to a file.
conf_thres (float): Confidence threshold for displaying detections.
show_labels (bool): Whether to display class labels.
show_conf (bool): Whether to display confidence values.
Returns:
(np.ndarray | None): Plotted image grid as a numpy array if save is False, None otherwise. Runs in a background
thread and returns the `threading.Thread` instead unless called with `threaded=False`.
Notes:
This function supports both tensor and numpy array inputs. It will automatically
convert tensor inputs to numpy arrays for processing.
Channel Support:
- 1 channel: Grayscale
- 2 channels: Third channel added as zeros
- 3 channels: Used as-is (standard RGB)
- 4+ channels: Cropped to first 3 channels
"""
images = np.zeros((0, 3, 640, 640), dtype=np.float32) if images is None else images
for k in ("cls", "bboxes", "conf", "masks", "keypoints", "batch_idx", "images", "semantic_mask", "depth"):
if k not in labels:
continue
if k == "cls" and labels[k].ndim == 2:
labels[k] = labels[k].squeeze(1) # squeeze if shape is (n, 1)
if isinstance(labels[k], torch.Tensor):
labels[k] = labels[k].cpu().numpy()
cls = labels.get("cls", np.zeros(0, dtype=np.int64))
batch_idx = labels.get("batch_idx", np.zeros(cls.shape, dtype=np.int64))
bboxes = labels.get("bboxes", np.zeros(0, dtype=np.float32))
confs = labels.get("conf", None)
masks = labels.get("masks", np.zeros(0, dtype=np.uint8))
kpts = labels.get("keypoints", np.zeros(0, dtype=np.float32))
semantic_masks = labels.get("semantic_mask", np.zeros(0, dtype=np.int64))
depth_maps = labels.get("depth", np.zeros(0, dtype=np.float32))
images = labels.get("img", images) # default to input images
custom = colors.parse(class_colors) # parsed once per mosaic
images = images[:max_subplots] # convert only the plotted images
if len(images) and images[0].max() <= 1:
images = images * 255 # de-normalise a copy, the train/val batch is shared
if isinstance(images, torch.Tensor):
images = images.byte().cpu().numpy() # uint8 on device first, a quarter of the float transfer
# Handle 2-ch and n-ch images
c = images.shape[1]
if c == 2:
zero = np.zeros_like(images[:, :1])
images = np.concatenate((images, zero), axis=1) # pad 2-ch with a black channel
elif c > 3:
images = images[:, :3] # crop multispectral images to first 3 channels
images = images[:, ::-1] # RGB to BGR, the Annotator buffer order
bs, _, h, w = images.shape # batch size, _, height, width
bs = min(bs, max_subplots) # limit plot images
ns = np.ceil(bs**0.5) # number of subplots (square)
# Build Image
mosaic = np.full((int(ns * h), int(ns * w), 3), 255, dtype=np.uint8) # init
for i in range(bs):
x, y = int(w * (i // ns)), int(h * (i % ns)) # block origin
mosaic[y : y + h, x : x + w, :] = images[i].transpose(1, 2, 0)
# Resize (optional)
scale = max_size / ns / max(h, w)
if scale < 1:
h = math.ceil(scale * h)
w = math.ceil(scale * w)
mosaic = cv2.resize(mosaic, tuple(int(x * ns) for x in (w, h)))
# Annotate
fs = int((h + w) * ns * 0.01) # font size
fs = max(fs, 18) # ensure that the font size is large enough to be easily readable.
annotator = Annotator(mosaic, line_width=round(fs / 10), font_size=fs, pil=True, example=str(names))
for i in range(bs):
x, y = int(w * (i // ns)), int(h * (i % ns)) # block origin
annotator.rectangle([x, y, x + w, y + h], None, (255, 255, 255), width=2) # borders
if paths:
annotator.text([x + 5, y + 5], text=Path(paths[i]).name[:40], txt_color=(220, 220, 220)) # filenames
if len(cls) > 0:
idx = batch_idx == i
classes = cls[idx].astype("int")
labels = confs is None
conf = confs[idx] if confs is not None else None # check for confidence presence (label vs pred)
if len(bboxes):
boxes = bboxes[idx]
if len(boxes):
if boxes[:, :4].max() <= 1.1: # if normalized with tolerance 0.1
boxes[..., [0, 2]] *= w # scale to pixels
boxes[..., [1, 3]] *= h
elif scale < 1: # absolute coords need scale if image scales
boxes[..., :4] *= scale
boxes[..., 0] += x
boxes[..., 1] += y
is_obb = boxes.shape[-1] == 5 # xywhr
boxes = ops.xywhr2xyxyxyxy(boxes) if is_obb else ops.xywh2xyxy(boxes)
for j, box in enumerate(boxes.astype(np.int64).tolist()):
c = classes[j]
color = colors(c, True, custom)
c = names.get(c, c) if names else c
if labels or conf[j] > conf_thres:
conf_text = f" {conf[j]:.1f}" if conf is not None and show_conf else ""
label = f"{c}{conf_text}" if show_labels else conf_text.strip()
annotator.box_label(box, label, color=color)
elif len(classes):
for c in classes:
color = colors(c, True, custom)
c = names.get(c, c) if names else c
label = f"{c}" if labels else f"{c} {conf[0]:.1f}"
annotator.text([x, y], label, txt_color=color, box_color=(64, 64, 64, 128))
# Plot keypoints
if len(kpts):
kpts_ = kpts[idx].copy()
if len(kpts_):
if kpts_[..., 0].max() <= 1.01 or kpts_[..., 1].max() <= 1.01: # if normalized with tolerance .01
kpts_[..., 0] *= w # scale to pixels
kpts_[..., 1] *= h
elif scale < 1: # absolute coords need scale if image scales
kpts_[..., :2] *= scale
kpts_[..., 0] += x
kpts_[..., 1] += y
for j in range(len(kpts_)):
if labels or conf[j] > conf_thres:
annotator.kpts(kpts_[j], conf_thres=conf_thres)
# Plot masks as one instance-index map per image (1..n, later instances on top), resized and blended once
if len(masks):
ids = np.arange(1, len(classes) + 1) * (labels or conf > conf_thres) # 0 drops filtered instances
if idx.shape[0] == masks.shape[0] and masks.max() <= 1: # overlap_mask=False
k = np.flatnonzero(ids) # kept instances only
index = ((masks[idx][k] > 0.5) * ids[k, None, None].astype(np.uint16)).max(0, initial=0)
else: # overlap_mask=True, the mask already holds instance indices
index = np.r_[0, ids][masks[i].astype(int)]
index = cv2.resize(index.astype(np.uint16), (w, h), interpolation=cv2.INTER_NEAREST)
lut = np.array([(0, 0, 0)] + [colors(c, True, custom) for c in classes], np.uint8)
patch = annotator.im[y : y + h, x : x + w]
np.copyto(patch, cv2.addWeighted(patch, 0.4, lut[index], 0.6, 0), where=(index > 0)[..., None])
# Plot semantic masks
if len(semantic_masks) and i < len(semantic_masks):
mask = semantic_masks[i]
mh, mw = mask.shape
if mh != h or mw != w:
mask = cv2.resize(mask.astype(np.uint8), (w, h), interpolation=cv2.INTER_NEAREST)
sub_annotator = Annotator(np.ascontiguousarray(annotator.im[y : y + h, x : x + w]), line_width=1)
mask = np.where(mask == 1, 0, 255) if names and len(names) == 1 else mask
sub_annotator.semantic_mask(mask, alpha=0.4, custom=custom)
annotator.im[y : y + h, x : x + w] = sub_annotator.im
# Plot depth maps
if len(depth_maps) and i < len(depth_maps):
d = depth_maps[i]
if d.ndim == 3:
d = d.squeeze(0)
dh, dw = d.shape
if dh != h or dw != w:
d = cv2.resize(d.astype(np.float32), (w, h), interpolation=cv2.INTER_NEAREST)
sub_annotator = Annotator(np.ascontiguousarray(annotator.im[y : y + h, x : x + w]), line_width=1)
sub_annotator.depth_map(d, alpha=0.6)
annotator.im[y : y + h, x : x + w] = sub_annotator.im
im = cv2.cvtColor(annotator.im, cv2.COLOR_BGR2RGB)
if not save:
return im
Image.fromarray(im).save(fname)
if on_plot:
on_plot(fname)Function ultralytics.utils.plotting.plot_results#
def plot_results(file: str | Path = "", dir: str | Path = "", on_plot: Callable | None = None)Plot training results from a results CSV file.
The function supports various types of data including detection, instance segmentation, semantic segmentation, depth estimation, classification, and pose estimation. All 'results*.csv' files in the CSV's directory are plotted and saved as 'results.png' in that directory.
Args
| Name | Type | Description | Default |
|---|---|---|---|
file | str | Path, optional | Path to the CSV file containing the training results. | "" |
dir | str | Path, optional | Directory where the CSV file is located if 'file' is not provided. | "" |
on_plot | Callable, optional | Callback function to be executed after plotting. Takes filename as an argument. | None |
Examples
>>> from ultralytics.utils.plotting import plot_results
>>> plot_results("path/to/results.csv")ultralytics/utils/plotting.py
@plt_settings()
def plot_results(file: str | Path = "", dir: str | Path = "", on_plot: Callable | None = None):
"""Plot training results from a results CSV file.
The function supports various types of data including detection, instance segmentation, semantic segmentation, depth
estimation, classification, and pose estimation. All 'results*.csv' files in the CSV's directory are plotted and
saved as 'results.png' in that directory.
Args:
file (str | Path, optional): Path to the CSV file containing the training results.
dir (str | Path, optional): Directory where the CSV file is located if 'file' is not provided.
on_plot (Callable, optional): Callback function to be executed after plotting. Takes filename as an argument.
Examples:
>>> from ultralytics.utils.plotting import plot_results
>>> plot_results("path/to/results.csv")
"""
import matplotlib.pyplot as plt # scope for faster 'import ultralytics'
import polars as pl
save_dir = Path(file).parent if file else Path(dir)
files = list(save_dir.glob("results*.csv"))
assert len(files), f"No results.csv files found in {save_dir.resolve()}, nothing to plot."
loss_keys, metric_keys = [], []
fig, ax = None, None
for i, f in enumerate(files):
try:
data = pl.read_csv(f.read_bytes(), infer_schema_length=None)
if i == 0:
for c in data.columns:
if "loss" in c:
loss_keys.append(c)
elif "metric" in c:
metric_keys.append(c)
loss_mid, metric_mid = len(loss_keys) // 2, (len(metric_keys) + 1) // 2
columns = (
loss_keys[:loss_mid] + metric_keys[:metric_mid] + loss_keys[loss_mid:] + metric_keys[metric_mid:]
)
fig, ax = plt.subplots(2, (len(columns) + 1) // 2, figsize=(len(columns) + 2, 6), tight_layout=True)
ax = ax.ravel()
ax[-1].set_visible(len(columns) % 2 == 0)
x = data.select(data.columns[0]).to_numpy().flatten()
for i, j in enumerate(columns):
y = data.select(j).to_numpy().flatten().astype("float")
ax[i].plot(x, y, marker=".", label=f.stem, linewidth=2, markersize=8) # actual results
ax[i].plot(x, _gaussian_filter1d(y, sigma=3), ":", label="smooth", linewidth=2) # smoothing line
ax[i].set_title(j, fontsize=12)
except Exception as e:
LOGGER.error(f"Plotting error for {f}: {e}")
if ax is not None:
ax[1].legend()
fname = save_dir / "results.png"
fig.savefig(fname, dpi=200)
plt.close()
if on_plot:
on_plot(fname)Function ultralytics.utils.plotting.plot_multitrain_results#
def plot_multitrain_results(scores: dict, key: str = "fitness", save_dir=Path())Plot per-dataset metrics from a multi-dataset training run as a bar chart with the cross-dataset mean.
Args
| Name | Type | Description | Default |
|---|---|---|---|
scores | dict | Mapping of dataset name to its scalar metric value. | required |
key | str | Name of the plotted metric, used as the y-axis label. | "fitness" |
save_dir | str | Path | Directory to save the 'multitrain_results.png' figure. | Path() |
Returns
| Type | Description |
|---|---|
Path | Path to the saved figure. |
Examples
>>> from ultralytics.utils.plotting import plot_multitrain_results
>>> plot_multitrain_results({"coco8": 0.61, "dota8": 0.48}, key="metrics/mAP50-95(B)")ultralytics/utils/plotting.py
@plt_settings()
def plot_multitrain_results(scores: dict, key: str = "fitness", save_dir=Path()):
"""Plot per-dataset metrics from a multi-dataset training run as a bar chart with the cross-dataset mean.
Args:
scores (dict): Mapping of dataset name to its scalar metric value.
key (str): Name of the plotted metric, used as the y-axis label.
save_dir (str | Path): Directory to save the 'multitrain_results.png' figure.
Returns:
(Path): Path to the saved figure.
Examples:
>>> from ultralytics.utils.plotting import plot_multitrain_results
>>> plot_multitrain_results({"coco8": 0.61, "dota8": 0.48}, key="metrics/mAP50-95(B)")
"""
import matplotlib.pyplot as plt # scope for faster 'import ultralytics'
mean = sum(scores.values()) / len(scores)
fig, ax = plt.subplots(figsize=(max(6.0, len(scores) * 0.45), 5), tight_layout=True)
ax.bar(range(len(scores)), list(scores.values()), color="#042AFF")
ax.axhline(mean, color="orange", linestyle="--", label=f"mean = {mean:.3f}")
ax.set_xticks(range(len(scores)))
ax.set_xticklabels(list(scores), rotation=90)
ax.set_ylabel(key)
ax.set_title(f"MultiTrainer results across {len(scores)} datasets")
ax.legend()
fname = Path(save_dir) / "multitrain_results.png"
fig.savefig(fname, dpi=200)
plt.close(fig)
return fnameFunction ultralytics.utils.plotting.plt_color_scatter#
def plt_color_scatter(v, f, bins: int = 20, cmap: str = "viridis", alpha: float = 0.8, edgecolors: str = "none")Plot a scatter plot with points colored based on a 2D histogram.
Args
| Name | Type | Description | Default |
|---|---|---|---|
v | array-like | Values for the x-axis. | required |
f | array-like | Values for the y-axis. | required |
bins | int, optional | Number of bins for the histogram. | 20 |
cmap | str, optional | Colormap for the scatter plot. | "viridis" |
alpha | float, optional | Alpha for the scatter plot. | 0.8 |
edgecolors | str, optional | Edge colors for the scatter plot. | "none" |
Examples
>>> v = np.random.rand(100)
>>> f = np.random.rand(100)
>>> plt_color_scatter(v, f)ultralytics/utils/plotting.py
def plt_color_scatter(v, f, bins: int = 20, cmap: str = "viridis", alpha: float = 0.8, edgecolors: str = "none"):
"""Plot a scatter plot with points colored based on a 2D histogram.
Args:
v (array-like): Values for the x-axis.
f (array-like): Values for the y-axis.
bins (int, optional): Number of bins for the histogram.
cmap (str, optional): Colormap for the scatter plot.
alpha (float, optional): Alpha for the scatter plot.
edgecolors (str, optional): Edge colors for the scatter plot.
Examples:
>>> v = np.random.rand(100)
>>> f = np.random.rand(100)
>>> plt_color_scatter(v, f)
"""
import matplotlib.pyplot as plt # scope for faster 'import ultralytics'
# Calculate 2D histogram and corresponding colors
hist, xedges, yedges = np.histogram2d(v, f, bins=bins)
colors = [
hist[
np.clip(np.digitize(v[i], xedges, right=False) - 1, 0, hist.shape[0] - 1),
np.clip(np.digitize(f[i], yedges, right=False) - 1, 0, hist.shape[1] - 1),
]
for i in range(len(v))
]
# Scatter plot
plt.scatter(v, f, c=colors, cmap=cmap, alpha=alpha, edgecolors=edgecolors)Function ultralytics.utils.plotting.plot_depth_panels#
def plot_depth_panels(
imgs: torch.Tensor,
preds: list[torch.Tensor],
fname: str | Path,
gt: torch.Tensor | None = None,
titles: list[str] | None = None,
max_images: int = 4,
) -> NoneWrite a depth panel grid: one row per image, columns RGB | GT (if provided) | one per entry of preds.
With GT, all depth columns share the GT valid-pixel range per row, so a scale error between GT and any prediction shows up directly as a color mismatch; without GT, each prediction uses its own valid-pixel range. Panels are resized to the RGB image size, so predictions at head stride need no prior interpolation.
Args
| Name | Type | Description | Default |
|---|---|---|---|
imgs | torch.Tensor | (B,3,H,W) float image tensor in [0,1]. | required |
preds | list | List of (B,1,H,W) or (B,H,W) predicted depth tensors; each adds one column. | required |
fname | str | Path | Output image path. | required |
gt | torch.Tensor, optional | (B,1,H,W) or (B,H,W) ground-truth depth in meters (pixels <= 0 invalid, drawn black). Used for the GT column and to set the shared color scale. | None |
titles | list, optional | List of column labels, drawn in a 24 px header strip. None keeps the strip-free layout. | None |
max_images | int | Maximum number of rows. | 4 |
ultralytics/utils/plotting.py
def plot_depth_panels(
imgs: torch.Tensor,
preds: list[torch.Tensor],
fname: str | Path,
gt: torch.Tensor | None = None,
titles: list[str] | None = None,
max_images: int = 4,
) -> None:
"""Write a depth panel grid: one row per image, columns RGB | GT (if provided) | one per entry of `preds`.
With GT, all depth columns share the GT valid-pixel range per row, so a scale error between GT and any prediction
shows up directly as a color mismatch; without GT, each prediction uses its own valid-pixel range. Panels are
resized to the RGB image size, so predictions at head stride need no prior interpolation.
Args:
imgs (torch.Tensor): (B,3,H,W) float image tensor in [0,1].
preds (list): List of (B,1,H,W) or (B,H,W) predicted depth tensors; each adds one column.
fname (str | Path): Output image path.
gt (torch.Tensor, optional): (B,1,H,W) or (B,H,W) ground-truth depth in meters (pixels <= 0 invalid, drawn
black). Used for the GT column and to set the shared color scale.
titles (list, optional): List of column labels, drawn in a 24 px header strip. None keeps the strip-free layout.
max_images (int): Maximum number of rows.
"""
preds = [p.unsqueeze(1) if p.ndim == 3 else p for p in preds]
h, w = imgs.shape[-2:]
rows = []
for i in range(min(imgs.shape[0], max_images)):
rgb = (imgs[i].detach().float().cpu().clamp(0, 1).numpy() * 255).astype(np.uint8).transpose(1, 2, 0)
panels = [cv2.cvtColor(rgb, cv2.COLOR_RGB2BGR)]
if gt is not None:
g = gt[i, 0] if gt.ndim == 4 else gt[i]
gv = g[g > 0]
vmin = float(gv.min()) if gv.numel() else 0.0
vmax = float(gv.max()) if gv.numel() else 1.0
d = g.detach().float().cpu().numpy() if isinstance(g, torch.Tensor) else np.asarray(g, np.float32)
panels.append(
cv2.resize(colorize_depth(d, vmin, vmax, mode="metric"), (w, h), interpolation=cv2.INTER_NEAREST)
)
else:
# No GT: scale each prediction by its own valid range.
vmin = vmax = None
for p in preds:
d = p[i, 0] if p.ndim == 4 else p[i]
d = d.detach().float().cpu().numpy() if isinstance(d, torch.Tensor) else np.asarray(d, np.float32)
lo, hi = vmin, vmax
if lo is None or hi is None:
dv = d[d > 0]
lo, hi = (float(dv.min()), float(dv.max())) if dv.size else (0.0, 1.0)
panels.append(cv2.resize(colorize_depth(d, lo, hi, mode="metric"), (w, h), interpolation=cv2.INTER_NEAREST))
rows.append(np.hstack(panels))
grid = np.vstack(rows)
if titles:
strip = np.full((24, grid.shape[1], 3), 255, dtype=np.uint8)
for j, t in enumerate(titles):
cv2.putText(strip, str(t), (j * w + 4, 17), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 0), 1, cv2.LINE_AA)
grid = np.vstack([strip, grid])
cv2.imwrite(str(fname), grid)Function ultralytics.utils.plotting.plot_tune_results#
def plot_tune_results(results_file: str = "tune_results.ndjson", exclude_zero_fitness_points: bool = True)Plot the evolution results stored in a tuning NDJSON file.
Saves 'tune_scatter_plots.png' and 'tune_fitness.png' next to the results file.
Args
| Name | Type | Description | Default |
|---|---|---|---|
results_file | str, optional | Path to the NDJSON file containing the tuning results. | "tune_results.ndjson" |
exclude_zero_fitness_points | bool, optional | Don't include points with zero fitness in tuning plots. | True |
Examples
>>> from ultralytics.utils.plotting import plot_tune_results
>>> plot_tune_results("path/to/tune_results.ndjson")ultralytics/utils/plotting.py
@plt_settings()
def plot_tune_results(results_file: str = "tune_results.ndjson", exclude_zero_fitness_points: bool = True):
"""Plot the evolution results stored in a tuning NDJSON file.
Saves 'tune_scatter_plots.png' and 'tune_fitness.png' next to the results file.
Args:
results_file (str, optional): Path to the NDJSON file containing the tuning results.
exclude_zero_fitness_points (bool, optional): Don't include points with zero fitness in tuning plots.
Examples:
>>> from ultralytics.utils.plotting import plot_tune_results
>>> plot_tune_results("path/to/tune_results.ndjson")
"""
import json
import matplotlib.pyplot as plt # scope for faster 'import ultralytics'
def _save_one_file(file):
"""Save one matplotlib plot to 'file'."""
plt.savefig(file, dpi=200)
plt.close()
LOGGER.info(f"Saved {file}")
results_file = Path(results_file)
with open(results_file, encoding="utf-8") as f:
records = [json.loads(line) for line in f if line.strip()]
if not records:
return
keys = list(records[0].get("hyperparameters", {}))
x = np.array(
[[r.get("fitness", 0.0)] + [r.get("hyperparameters", {}).get(k, np.nan) for k in keys] for r in records],
dtype=float,
)
all_fitness = x[:, 0] # fitness
zero_mask = slice(None)
if exclude_zero_fitness_points:
zero_mask = all_fitness > 0 # exclude zero-fitness points
x, all_fitness = x[zero_mask], all_fitness[zero_mask]
if len(all_fitness) == 0:
LOGGER.warning("No valid fitness values to plot (all iterations may have failed)")
return
fitness = all_fitness.copy()
# Iterative sigma rejection on lower bound only
for _ in range(3): # max 3 iterations
mean, std = fitness.mean(), fitness.std()
lower_bound = mean - 3 * std
mask = fitness >= lower_bound
if mask.all(): # no more outliers
break
x, fitness = x[mask], fitness[mask]
j = np.argmax(fitness) # max fitness index
n = math.ceil(len(keys) ** 0.5) # columns and rows in plot
plt.figure(figsize=(10, 10), tight_layout=True)
for i, k in enumerate(keys):
v = x[:, i + 1]
mu = v[j] # best single result
plt.subplot(n, n, i + 1)
plt_color_scatter(v, fitness, cmap="viridis", alpha=0.8, edgecolors="none")
plt.plot(mu, fitness.max(), "k+", markersize=15)
plt.title(f"{k} = {mu:.3g}", fontdict={"size": 9})
plt.tick_params(axis="both", labelsize=8) # Set axis label size to 8
if i % n != 0:
plt.yticks([])
_save_one_file(results_file.with_name("tune_scatter_plots.png"))
# Fitness progress and per-dataset change from the initial to best result
plot_records = records if isinstance(zero_mask, slice) else [r for r, keep in zip(records, zero_mask) if keep]
datasets = sorted({k for r in plot_records for k in r.get("datasets", {})})
_, (ax1, ax2) = plt.subplots(
1,
2,
figsize=(16, max(6, len(datasets) * 0.28)),
gridspec_kw={"width_ratios": (1, 1.35)},
constrained_layout=True,
)
iterations = np.array([r.get("iteration", i) for i, r in enumerate(plot_records, 1)])
best_idx = int(all_fitness.argmax())
ax1.scatter(iterations, all_fitness, s=28, color="0.45", alpha=0.75, label="Iteration")
ax1.plot(iterations, np.maximum.accumulate(all_fitness), color="#2563eb", linewidth=2.5, label="Best so far")
ax1.axhline(all_fitness[0], color="#dc2626", linestyle="--", label=f"Initial {all_fitness[0]:.4f}")
ax1.scatter(iterations[best_idx], all_fitness[best_idx], s=90, color="#16a34a", zorder=5, label="Best")
ax1.set(title="Fitness Progress", xlabel="Iteration", ylabel="Fitness")
ax1.grid(alpha=0.2)
ax1.legend()
if datasets:
initial = np.array([plot_records[0].get("datasets", {}).get(k, {}).get("fitness", 0.0) for k in datasets])
best = np.array([plot_records[best_idx].get("datasets", {}).get(k, {}).get("fitness", 0.0) for k in datasets])
order = np.argsort(best - initial)
y = np.arange(len(datasets))
ax2.hlines(y, initial[order], best[order], color="0.8")
ax2.scatter(initial[order], y, s=24, color="#dc2626", label="Initial")
ax2.scatter(best[order], y, s=24, color="#16a34a", label=f"Best aggregate iteration {iterations[best_idx]}")
ax2.set_yticks(y)
ax2.set_yticklabels(np.array(datasets)[order], fontsize=8)
ax2.set(title="Per-Dataset Fitness: Initial vs Best Aggregate Iteration", xlabel="Fitness")
ax2.grid(axis="x", alpha=0.2)
ax2.legend()
else:
ax2.set_axis_off()
_save_one_file(results_file.with_name("tune_fitness.png"))Function ultralytics.utils.plotting.class_activation_map#
def class_activation_map(
model,
im: torch.Tensor,
paths: list[str],
save_dir: Path,
*args,
conf: float = 0.25,
classes=None,
topk: int = 16,
**kwargs,
) -> AnyRun inference and save a class activation heatmap for each image of the batch.
LayerCAM weights each head-input position by its positive gradient toward the predicted class score. Each prediction and head level is normalized independently before taking their element-wise maximum, preventing stronger predictions or levels from hiding weaker ones.
Args
| Name | Type | Description | Default |
|---|---|---|---|
model | torch.nn.Module | AutoBackend wrapping a PyTorch model. | required |
im | torch.Tensor | Preprocessed images of shape (B, 3, H, W). | required |
paths | list[str] | Source path of each image of the batch, used to name the saved overlays. | required |
save_dir | Path | Directory to save the overlays in. | required |
*args | Any | Additional positional arguments passed to the model forward. | required |
conf | float | Score threshold a prediction must pass to contribute, falling back to the single best prediction for images where nothing passes it, so that a near miss can still be inspected. | 0.25 |
classes | int | list[int], optional | Only let these class ids contribute, as in the predict classes filter. | None |
topk | int | Maximum number of predictions to explain per image, each one costing a backward pass. | 16 |
**kwargs | Any | Additional keyword arguments passed to the model forward. | required |
Returns
| Type | Description |
|---|---|
Any | Model predictions, detached from the autograd graph. |
ultralytics/utils/plotting.py
def class_activation_map(
model,
im: torch.Tensor,
paths: list[str],
save_dir: Path,
*args,
conf: float = 0.25,
classes=None,
topk: int = 16,
**kwargs,
) -> Any:
"""Run inference and save a class activation heatmap for each image of the batch.
LayerCAM weights each head-input position by its positive gradient toward the predicted class score. Each prediction
and head level is normalized independently before taking their element-wise maximum, preventing stronger predictions
or levels from hiding weaker ones.
Args:
model (torch.nn.Module): AutoBackend wrapping a PyTorch model.
im (torch.Tensor): Preprocessed images of shape (B, 3, H, W).
paths (list[str]): Source path of each image of the batch, used to name the saved overlays.
save_dir (Path): Directory to save the overlays in.
*args (Any): Additional positional arguments passed to the model forward.
conf (float): Score threshold a prediction must pass to contribute, falling back to the single best prediction
for images where nothing passes it, so that a near miss can still be inspected.
classes (int | list[int], optional): Only let these class ids contribute, as in the predict `classes` filter.
topk (int): Maximum number of predictions to explain per image, each one costing a backward pass.
**kwargs (Any): Additional keyword arguments passed to the model forward.
Returns:
(Any): Model predictions, detached from the autograd graph.
"""
acts, scores = [], []
def pre_hook(module, inputs):
"""Capture the feature maps entering the head, before heads like WorldDetect overwrite them in place."""
x = inputs[0]
acts.extend(a for a in (x if isinstance(x, (list, tuple)) else [x]) if a.ndim == 4)
def hook(module, inputs, output):
"""Capture the class logits leaving the head."""
raw = output[1] if isinstance(output, tuple) else output # heads returning (predictions, raw) keep the raw
if isinstance(raw, dict): # Detect and subclasses: follow the selected inference branch
s = raw.get("one2one" if module.end2end else "one2many", raw)["scores"] # (B, nc, anchors)
elif isinstance(raw, tuple): # RTDETRDecoder, raw = (dec_bboxes, dec_scores, ...)
s = raw[1][-1].transpose(1, 2) # last decoder layer, (B, nc, queries)
else: # Classify (B, nc), SemanticSegment (B, nc, h, w), Depth (B, 1, h, w)
s = raw
scores.append(s.reshape(*s.shape[:2], -1)) # class logits, (B, nc, predictions)
head = model.model.model[-1] # AutoBackend -> PyTorch model -> head
head.shape = head.shapes = None # rebuild the anchor caches, inference tensors in them break the autograd graph
handles = [head.register_forward_pre_hook(pre_hook), head.register_forward_hook(hook)]
# smart_inference_mode() wraps the caller in inference_mode from torch 1.10 and in no_grad below it, and only the
# former has to be left before autograd will record anything.
with torch.inference_mode(False) if TORCH_1_10 else contextlib.nullcontext(), torch.enable_grad():
try:
im = im.clone().requires_grad_(True) # model parameters have requires_grad=False, so seed the graph here
preds = model(im, *args, **kwargs)
finally:
for handle in handles:
handle.remove()
s = torch.cat(scores, 2) # (B, nc, predictions) class logits
if classes is not None:
cls = torch.as_tensor(classes, dtype=torch.long, device=s.device).flatten()
cls = cls[(cls >= 0) & (cls < s.shape[1])] # drop ids outside this model's output channels
if len(cls):
s = s[:, cls] # heatmap for the requested classes only
s = s.amax(1) # (B, predictions) best class logit of each prediction
keep = (s.sigmoid() >= conf) | (s == s.amax(1, keepdim=True)) # top prediction alone if none above conf
n = min(int(keep.sum(1).amax()), topk) # predictions to explain, one backward pass each
if int(keep.sum(1).amax()) > n:
LOGGER.warning(f"Explaining the {n} strongest predictions per image out of {int(keep.sum(1).amax())}.")
rank = torch.arange(n, device=s.device) % keep.sum(1, keepdim=True).clamp(min=1) # short images repeat
order = s.masked_fill(~keep, float("-inf")).argsort(1, descending=True).gather(1, rank) # (B, n)
cam = None
for k in range(n):
levels = []
grads = torch.autograd.grad(s.gather(1, order[:, k : k + 1]).sum(), acts, retain_graph=k < n - 1)
for a, g in zip(acts, grads):
c = (g.float().clamp(min=0) * a.float()).sum(1, keepdim=True) # LayerCAM, per-position weighting
c = c.clamp(min=0) # activations can be negative, keep only evidence for the prediction
c = torch.nn.functional.interpolate(c, im.shape[2:], mode="bilinear", align_corners=False)
levels.append(c / c.amax((2, 3), keepdim=True).clamp(min=1e-7))
# The level a prediction is made on peaks far higher than the rest, so summing raw would shrink the
# broader evidence the other levels hold down to a faint background.
level = torch.stack(levels).amax(0)
cam = level if cam is None else torch.maximum(cam, level)
cam = (cam.squeeze(1) * 255).byte().cpu().numpy() # (B, H, W), maps are already scaled to [0, 1]
ims = im.detach()[:, :3].float()
lo, hi = ims.amin((2, 3), keepdim=True), ims.amax((2, 3), keepdim=True) # classify inputs are mean-std normalized
ims = ((ims - lo) / (hi - lo).clamp(min=1e-7) * 255).byte().permute(0, 2, 3, 1).cpu().numpy()[..., ::-1] # to BGR
save_dir.mkdir(parents=True, exist_ok=True)
for c, img, p in zip(cam, ims, paths):
f = increment_path(save_dir / f"{Path(p).stem}_cam.jpg")
img = np.ascontiguousarray(img if img.shape[2] == 3 else img[..., :1].repeat(3, 2)) # grayscale to BGR
heatmap = cv2.addWeighted(cv2.applyColorMap(c, cv2.COLORMAP_JET), 0.5, img, 0.5, 0)
cv2.imwrite(str(f), heatmap)
LOGGER.info(f"Saving {f}... (LayerCAM)")
def detach(x):
"""Detach tensors in nested model outputs from the autograd graph."""
if isinstance(x, torch.Tensor):
return x.detach()
if isinstance(x, dict):
return {k: detach(v) for k, v in x.items()}
if isinstance(x, (list, tuple)):
return type(x)(detach(v) for v in x)
return x
return detach(preds)