Instance Segmentation with Ultralytics YOLO

Additionally, is there a method to fit the boundary of the ID card mask into four straight lines for locating the ID card？

@hp0716 for smoother segmentation results on ID cards, try using retina_masks=True during inference to get higher resolution masks:

from ultralytics import YOLO
model = YOLO('yolo11n-seg.pt')
results = model(image, retina_masks=True)

For post-processing to get smooth boundaries, you can apply morphological operations and contour approximation:

import cv2
import numpy as np

# Get mask from results
mask = results[0].masks.data[0].cpu().numpy()
mask = (mask * 255).astype(np.uint8)

# Apply morphological smoothing
kernel = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (5, 5))
mask = cv2.morphologyEx(mask, cv2.MORPH_CLOSE, kernel)
mask = cv2.morphologyEx(mask, cv2.MORPH_OPEN, kernel)

# Find contours and approximate to quadrilateral
contours, _ = cv2.findContours(mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
if contours:
    largest_contour = max(contours, key=cv2.contourArea)
    epsilon = 0.02 * cv2.arcLength(largest_contour, True)
    approx = cv2.approxPolyDP(largest_contour, epsilon, True)
    
    # Force to 4 points for ID card corners
    if len(approx) >= 4:
        corners = approx[:4]

This approach combines higher resolution masks with geometric fitting to achieve the straight-line boundaries you need for ID card localization.

Yes, you can annotate using custom build model but please check the annotation class id after annotation.

The metrics score with ShuffleAttention added to the tenth layer of backbone is reduced compared to the metrics score for YOLO11x-seg.yaml alone.

Trained my dataset using YOLO11x-seg.yaml+ShuffleAttention with no other parameters changed the metrics scores are as follows:

Class Images Instances Box( P R mAP₅₀ mAP_50-95) Mask( P R mAP₅₀ mAP_50-95): all 58 332 0.583 0.504 0.519 0.238 0.551 0.456 0.468 0.195

Training my dataset using the official YOLO11x-seg.yaml alone, the metrics scores are as follows:

Class Images Instances Box( P R mAP₅₀ mAP_50-95) Mask( P R mAP₅₀ mAP_50-95): all 58 332 0.669 0.465 0.489 0.257 0.648 0.446 0.455 0.217

@LwpZoe thank you for sharing these details. For medical instance segmentation with YOLO11x-seg, we recommend: 1) Verify your dataset quality and annotation consistency using the Dataset Guide, 2) Try increasing input resolution (imgsz=1280) to capture fine tissue details, 3) Implement medical-specific augmentations like CLAHE or elastic transforms, and 4) Consider using our pretrained weights with progressive fine-tuning rather than architectural modifications initially. The segmentation documentation contains additional optimization strategies for medical imaging tasks.

@LwpZoe your command is almost correct for training from scratch without pretrained weights. Ensure the resume=True argument is removed, as it is only used to resume interrupted training. The corrected command would be:

yolo segment train data=datasets/yolo_seg/data.yaml model=yolo11n-seg.yaml epochs=300 batch=16 imgsz=640 device=0,1

This will train your YOLO11 segmentation model from scratch using your custom dataset. For more details, visit the Segment Task Documentation.

Same question

To use two cameras for segmentation with a GPU, you'll need to set up a multi-threaded or multi-process pipeline to handle the video streams concurrently. Each camera stream can be processed in a separate thread or process, ensuring efficient utilization of the GPU. You can use the YOLO class from the Ultralytics library to load your segmentation model and process the frames from each camera. For detailed steps and code examples, please refer to the Ultralytics segmentation documentation. If you encounter any issues, make sure you're using the latest version of the package.

@Boaruzhanchik hey there! 👋 To display each detected car plate in separate windows, you can loop through the detected boxes and create a window for each. Here's a quick example to get you started:

for i, box in enumerate(boxes):
    x1, y1, x2, y2 = map(int, box[:4])
    plate = im0[y1:y2, x1:x2]
    cv2.imshow(f'Plate {i}', plate)
cv2.waitKey(0)
cv2.destroyAllWindows()

This code snippet extracts each detected plate based on its bounding box coordinates and displays it in a new window. Make sure to replace im0 with your frame variable if it's named differently. Hope this helps! 😊

Thanks a lot for the help, I did it this way, it seems to work too

if results[0].masks is not None:
    clss = results[0].boxes.cls.cpu().tolist()
    masks = results[0].masks.xy
    if len(masks) == 1:
         coordinates_mask = masks[0]
         auto_transformations_first(coordinates_mask)
    elif len(masks) <= 2:
          auto_transformations_first(coordinates_mask)
          coordinates_mask_2 = masks[1]
          auto_transformations_second_screen(coordinates_mask_2)
def auto_transformations_first(coordinates_mask):
     contour = coordinates_mask.reshape((-1, 1, 2)).astype(np.int32)
     epsilon = 0.04 * cv2.arcLength(contour, True)
     approx = cv2.approxPolyDP(contour, epsilon, True)
     auto_transform = four_point_transform(im0, approx.reshape(-2, 2))
     cv2.imshow('automated_first', auto_transform)

@Boaruzhanchik hey there! 👋 It looks like you've found a way to handle different numbers of masks with your approach. Nicely done! 🎉 Just a quick tip: you might want to ensure coordinates_mask is defined before using it in auto_transformations_first(coordinates_mask) for the elif len(masks) <= 2: case, to avoid any potential errors. Other than that, your method seems solid. Keep up the great work, and if you have any more questions or need further assistance, feel free to ask. Happy coding! 😊

Ultralytics training uses one task per model (detect, segment, or obb), so mixed label formats cannot be trained jointly in a standard pipeline. Use separate label directories/YAML files and train separate models, or convert box/OBB annotations to polygon labels so every object follows the segmentation format; note that a segmentation model returns masks and axis-aligned boxes, not native OBB predictions.

Ultralytics YOLO does not support mixing detection, segmentation, and OBB annotations in a single training run. Each model/task expects a specific label format and corresponding loss function.

The recommended approach is to separate the annotations and train separate models for segmentation, detection, and OBB, then combine their outputs during inference if needed. A single multi-task model would require a custom architecture and training pipeline.

@uqasha524, do you need masks, axis-aligned boxes, and OBBs as distinct outputs at inference, or would segmentation outputs alone meet your requirements? That distinction will help us suggest the appropriate dataset structure without introducing unnecessary custom training work.

That’s usually expected with standard instance segmentation: yolo26-seg learns the visible region of each instance, so the larger object can appear “punctured” where it is occluded; if you want a filled large-object mask, you’ll need labels that include the full occluded shape or apply post-processing to r.masks after inference, and the segmentation object isolation guide shows where to access those masks—if you share one image plus its label here, the Ultralytics team can help confirm whether it’s annotation or model behavior.

Yes — use a -seg model on your webcam feed, set classes=[...] if you only want one class, and max_det=1 if you want at most one instance per frame; if you share your use case here, the Ultralytics team can help refine it, and the Segment task docs cover the mask outputs you’ll use.

from ultralytics import YOLO

model = YOLO("yolo26n-seg.pt")
results = model.predict(source=0, stream=True, classes=[0], max_det=1)

Ok thx. I am actually trying to implement edge detection on android devices for rectangular objects. I have tried OpenCV but it's not enough. So I was thinking to use a segmentation model for my rectangular processor (overlay would only show on rectangular objects) but any time I try to implement this on android, I get serious lagging issues on the camera preview. Plus honestly new to this AI stuff.

If you only need a rectangle-style overlay, a segmentation model is usually heavier than necessary on Android; I’d start with a small OBB or detect model instead, export it with Export mode, and keep imgsz low or run inference every few frames to reduce preview lag. If you share your device, export format, and current FPS here, the Ultralytics team can suggest the lightest setup.

result = model.predict(os.path.join(path,file), nms = True, )这是我的预测代码是官方提供的呀

你这段 predict() 用法本身没问题；一般也不需要手动写 task="segment"，关键是确保加载的是分割权重（*-seg.pt），否则不会有 result.masks 输出；如果分割边缘出现“被框裁切/直线感强”的现象，可以在推理时尝试开启 retina_masks=True 并适当增大 imgsz（NMS 默认就会做，通常不需要单独传 nms=True），相关参数处理可参考 SegmentationPredictor 源码说明。

from ultralytics import YOLO

model = YOLO("yolo26n-seg.pt")  # 确保是 -seg 权重
results = model.predict(
    os.path.join(path, file),
    retina_masks=True,
    imgsz=1024,
    conf=0.25,
    iou=0.7,
)

您好，您的建议是有效的，但边界还是没有那么光滑，以及大目标效果更差，有时会出现正方形，您这边还有其他的解决方案吗？期待您的回复

这个“边缘直线/大目标出现方块”的现象通常是分割头的掩码分辨率与“mask 会按预测框裁剪”的机制共同导致的（框越紧越容易看出裁剪痕迹，可参考这条 社区说明：inference 时 mask 是否会被 bbox 裁剪）；如果你是自训练模型，想从根源提升大目标的边缘细节，建议在训练时把 mask_ratio 调小（更高 mask 分辨率）并配合更大的 imgsz/更大的 *-seg 模型，mask_ratio 参数说明见 配置参数文档。

from ultralytics import YOLO

model = YOLO("yolo26s-seg.pt")  # 也可以试试更大一点的 -seg 权重
model.train(
    data="path/to/your_dataset.yaml",
    imgsz=1024,
    epochs=100,
    mask_ratio=2,  # 需要更细可再降到 1（显存占用会明显上升）
)

如果你只是想“导出/显示更光滑”的轮廓（不改变模型预测本身），可以按 Isolating Segmentation Objects 的方式拿到 result.masks.data 后做一次轻量的 blur + close 再取轮廓；另外麻烦贴一张出现“正方形”的原图和对应可视化结果，并附上 yolo checks 的文本输出，我这边可以帮你确认主要瓶颈是分辨率、标注轮廓质量还是框裁剪造成的。