Instance Segmentation and Tracking using Ultralytics YOLO26
What is Instance Segmentation?#
Instance segmentation is a computer vision task that involves identifying and outlining individual objects in an image at the pixel level. Unlike semantic segmentation which only classifies pixels by category, instance segmentation uniquely labels and precisely delineates each object instance, making it crucial for applications requiring detailed spatial understanding like medical imaging, autonomous driving, and industrial automation.
Ultralytics YOLO26 delineates object boundaries precisely while keeping the speed and efficiency YOLO models are known for.
There are two types of instance segmentation tracking available in the Ultralytics package:
-
Instance Segmentation with Class Objects: Each class object is assigned a unique color for clear visual separation.
-
Instance Segmentation with Object Tracks: Every track is represented by a distinct color, facilitating easy identification and tracking across video frames.
Watch: Instance Segmentation with Object Tracking using Ultralytics YOLO26
Samples#
| Instance Segmentation | Instance Segmentation + Object Tracking |
|---|---|
![]() | ![]() |
| Ultralytics Instance Segmentation | Ultralytics Instance Segmentation with Object Tracking |
Segment and Track with YOLO26#
The InstanceSegmentation solution runs a YOLO26 segmentation model on each frame, draws per-object masks, and colors every instance separately so you can track individual objects across a video. Load a -seg model such as YOLO26n-seg, then run it over your source with the Python API or the CLI.
# Instance segmentation using Ultralytics YOLO26
yolo solutions isegment show=True
# Pass a source video
yolo solutions isegment source="path/to/video.mp4"
# Monitor the specific classes
yolo solutions isegment classes="[0, 5]"InstanceSegmentation Arguments#
Here's a table with the InstanceSegmentation arguments:
| Argument | Type | Default | Description |
|---|---|---|---|
model | str | None | Path to an Ultralytics YOLO model file. |
line_width | int | 2 | Line thickness for the boxes, keypoints and counts the solution draws. |
verbose | bool | True | Enables the solution's per-frame log of input shape, class counts and processing speed. The tracking call itself is always silent. |
You can also use track arguments within the InstanceSegmentation solution:
| Argument | Type | Default | Description |
|---|---|---|---|
tracker | str | 'tracktrack.yaml' | Specifies the tracking algorithm to use. Built-in options: botsort.yaml, bytetrack.yaml, ocsort.yaml, deepocsort.yaml, fasttrack.yaml, tracktrack.yaml. |
conf | float | 0.25 | Sets the confidence threshold for detections; lower values allow more objects to be tracked but may include false positives. |
iou | float | 0.5 | Sets the Intersection over Union (IoU) threshold for filtering overlapping detections. |
classes | list | None | Filters results by class index. For example, classes=[0, 2, 3] only tracks the specified classes. |
device | str | None | Specifies the device for inference (e.g., cpu, cuda:0 or 0). Allows users to select between CPU, a specific GPU, or other compute devices for model execution. |
imgsz | int or tuple | 640 | Target inference image size: an integer for a square or a (height, width) pair. Shapes are adjusted to the model stride. |
max_det | int | 300 | Maximum number of detections kept per frame before tracking, limiting the objects a solution processes in dense scenes. |
quantize | int or str | None | Inference precision: 16 (FP16) or 32/unset (FP32) for PyTorch and TorchScript models off CPU; other formats use the precision their artifact and runtime select. Replaces the deprecated half arg. |
Moreover, the following visualization arguments are available:
| Argument | Type | Default | Description |
|---|---|---|---|
show | bool | False | If True, displays the annotated images or videos in a window. Useful for immediate visual feedback during development or testing. |
show_conf | bool | True | Displays the confidence score for each detection alongside the label. Gives insight into the model's certainty for each detection. |
show_labels | bool | True | Displays labels for each detection in the visual output. Provides immediate understanding of detected objects. |
show_boxes | bool | True | Draws bounding boxes around detected objects. Essential for visual identification and location of objects in images or video frames. |
Applications of Instance Segmentation#
Instance segmentation with YOLO26 has numerous real-world applications across various industries:
Waste Management and Recycling#
YOLO26 can be used in waste management facilities to identify and sort different types of materials. The model can segment plastic waste, cardboard, metal, and other recyclables with high precision, enabling automated sorting systems to process waste more efficiently. This is particularly valuable considering that only about 10% of the 7 billion tonnes of plastic waste generated globally gets recycled.
Autonomous Vehicles#
In self-driving cars, instance segmentation helps identify and track pedestrians, vehicles, traffic signs, and other road elements at the pixel level. This precise understanding of the environment is crucial for navigation and safety decisions. YOLO26's real-time performance makes it well suited to these time-sensitive applications.
Medical Imaging#
Instance segmentation can identify and outline tumors, organs, or cellular structures in medical scans. YOLO26's ability to precisely delineate object boundaries makes it valuable for medical diagnostics and treatment planning.
Construction Site Monitoring#
At construction sites, instance segmentation can track heavy machinery, workers, and materials. This helps ensure safety by monitoring equipment positions and detecting when workers enter hazardous areas, while also optimizing workflow and resource allocation.
FAQ#
To perform instance segmentation using Ultralytics YOLO26, initialize the
InstanceSegmentationsolution with a YOLO26 segmentation model and process video frames through it. Here's a simplified code example:import cv2 from ultralytics import solutions cap = cv2.VideoCapture("path/to/video.mp4") assert cap.isOpened(), "Error reading video file" # Video writer w, h, fps = (int(cap.get(x)) for x in (cv2.CAP_PROP_FRAME_WIDTH, cv2.CAP_PROP_FRAME_HEIGHT, cv2.CAP_PROP_FPS)) video_writer = cv2.VideoWriter("instance-segmentation.avi", cv2.VideoWriter_fourcc(*"mp4v"), fps, (w, h)) # Init InstanceSegmentation isegment = solutions.InstanceSegmentation( show=True, # display the output model="yolo26n-seg.pt", # model="yolo26n-seg.pt" for object segmentation using YOLO26. ) # Process video while cap.isOpened(): success, im0 = cap.read() if not success: print("Video frame is empty or processing is complete.") break results = isegment(im0) video_writer.write(results.plot_im) cap.release() video_writer.release() cv2.destroyAllWindows()Instance segmentation identifies and outlines individual objects within an image, giving each object a unique label and mask. Object tracking extends this by assigning consistent IDs to objects across video frames, facilitating continuous tracking of the same objects over time. When combined, as in YOLO26's implementation, you can analyze object movement and behavior in videos while keeping precise boundary information.
Ultralytics YOLO26 offers real-time performance, competitive accuracy, and ease of use compared to two-stage models like Mask R-CNN or Faster R-CNN. YOLO26 processes images in a single pass (one-stage detection), making it much faster while maintaining high precision. It also integrates with Ultralytics Platform for managing models, datasets, and training pipelines. For applications that need both speed and accuracy, YOLO26 offers a strong balance.
Yes, Ultralytics offers several datasets suitable for training YOLO26 models for instance segmentation, including COCO-Seg, COCO8-Seg (a smaller subset for quick testing), Package-Seg, and Crack-Seg. These datasets come with pixel-level annotations needed for instance segmentation tasks. For more specialized applications, you can also create custom datasets following the Ultralytics format. Complete dataset information and usage instructions can be found in the Ultralytics Datasets documentation.
Comments99
Ask a question, share your results, or help someone else.
CatFatOw
Hello, I am trying to analyze a video and see if features grow/shrink over time. I am currently trying to use instance segmentation with object tracking; however, specifying
save_txt=Trueparameter inmodel.tracksaves every single prediction into a single huge txt file. Trying to count the frames by going through this large txt apparently doesn't work as the class ID are not in order and sorting it would lose all information.Is there any way for Ultralytics to save the txt files by frame number? For example, frame1.txt only contains the predictions of the first frame.
Any help would be greatly appreciated!
1
Glenn JocherUltralyticsFor per-frame TXT saving with tracking, you'll need to process results in your video loop and write custom frame-specific files. Here's a minimal adaptation of the Instance Segmentation and Tracking example:
frame_count = 0 while cap.isOpened(): ret, im0 = cap.read() if not ret: break results = model.track(im0, persist=True) # Save current frame's tracked objects with open(f'frame{frame_count}.txt', 'w') as f: for box, mask, track_id in zip(results[0].boxes, results[0].masks, results[0].boxes.id.int().cpu().tolist()): f.write(f"{track_id} {box.cls.item()} {box.xywhn.tolist()} {mask.xy[0].tolist()}\n") frame_count += 1This creates individual frameX.txt files with tracking IDs, class IDs, normalized boxes, and mask coordinates. The official documentation shows similar frame processing in the Instance Segmentation with Object Tracking examples.
CatFatOw
Thank you!
Glenn JocherUltralyticsFor per-frame TXT saving during tracking, you'll need to process each frame individually in your video loop as shown in the Instance Segmentation with Object Tracking example. Here's a minimal adaptation that creates frame-specific files:
frame_count = 0 while cap.isOpened(): ret, im0 = cap.read() if not ret: break results = model.track(im0, persist=True) with open(f'frame{frame_count}.txt', 'w') as f: for box, mask, track_id in zip(results[0].boxes, results[0].masks, results[0].boxes.id.int().cpu().tolist()): f.write(f"{track_id} {box.cls.item()} {box.xywhn.tolist()} {mask.xy[0].tolist()}\n") frame_count += 1This creates individual frameX.txt files with tracking IDs, class IDs, normalized boxes, and mask coordinates per frame. The approach mirrors the frame processing shown in our official tracking documentation.
pedmindset
Been trying to figure this out for a while now. I want to ask, can I use object counting in regions with instance sementation and tracking? You want to count vehicles that cross two specific regions, like so Object 001 crossed region A and region B. Is this possible to do
1
Glenn JocherUltralytics@pedmindset yes, it is possible to combine object counting in regions with instance segmentation and tracking using Ultralytics YOLO11. You can use the
trackmethod to assign consistent IDs to objects across frames and define regions to monitor their crossings. For guidance, refer to the segmentation and tracking examples in the Instance Segmentation and Tracking Guide. Let us know if you need further clarification!
DP1701
Cool, but I can't define an imgsz via solutions.InstanceSegmentation(…). Is there another option for this?
Glenn JocherUltralyticssolutions.InstanceSegmentationis a higher-level wrapper and doesn’t currently exposeimgszdirectly; if you need to control input size, run segmentation+tracking via the coreYOLO(...).track()API (supportsimgsz) as shown in the Tracking mode docs and keep using the Instance Segmentation and Tracking guide for solution-level usage.from ultralytics import YOLO model = YOLO("yolo26n-seg.pt") results = model.track(source="path/to/video.mp4", imgsz=640, show=True, tracker="botsort.yaml")
CatFatOw
Hello, I noticed that when using tracking mode compared to normal segmentation prediction, the number of features predicted decreased by quite a lot for feature dense videos even if I use the same parameters of confidence and maximum detection?
Glenn JocherUltralytics@CatFatOw great observation! This is actually expected behavior when using tracking mode compared to plain segmentation. Tracking algorithms apply additional filtering beyond confidence thresholds to maintain stable tracks, which can reduce the total number of detections.
Here are a few things that might help increase detections while maintaining tracking:
# Try adjusting these tracking-specific parameters results = model.track( frame, persist=True, conf=0.3, # Lower confidence threshold iou=0.5, # Adjust IoU threshold for NMS max_det=1000, # Increase max detections tracker="bytetrack.yaml" # Try different trackers )You can also experiment with different tracking algorithms - ByteTrack tends to be more permissive with new track creation compared to BoTSORT. The trade-off is between detection sensitivity and tracking stability, where tracking prioritizes consistent object IDs over detecting every possible instance.
If you need maximum detection sensitivity, consider running both modes and using tracking results only when you need the temporal consistency.
CatFatOw
Ok, thank you for the reply!
Mauricio566
Hello. Please, I need your help. I know exactly what segmenting an object is and how to do it. But I don't know what to do with an already segmented object or pixel mask. You can find a lot of information online about "how to segment a model or fit it to your dataset," but they don't tell you what to do with those masks in real projects. Can you give me some ideas or help, please? for r in results:
if r.masks is not None: mask = r.masks.data[0].cpu().numpy() there is the mask but honestly I don't know what to do. thankse-roe
Hello Glenn, thanks for the example! Actually, I had already figured it out—it was more of a suggestion for the documentation examples. []´s
Glenn JocherUltralyticsThank you for the suggestion! We always welcome community contributions to improve our documentation through GitHub Pull Requests. Feel free to submit any specific improvements you'd like to see in the Instance Segmentation with Object Tracking guide.
1
e-roe
Hello, Wouldn't it be good to have a complete example showing how to extract the track ID, contour points, class, and confidence? These methods that handle everything automatically end up being of little practical use, and finding this information in the documentation is quite challenging—it's often easier to find from third-party sources. []´s
Glenn JocherUltralytics(edited)@e-roe thanks for your feedback! You can access track IDs, classes, masks, and confidence scores directly from the results object. Here's a concise example showing how to extract these details:
results = isegment(im0) for i in range(len(isegment.clss)): track_id = isegment.track_ids[i] class_name = isegment.names[isegment.clss[i]] confidence = isegment.confs[i] mask = isegment.masks[i].data[0].numpy() # Binary mask array contours, _ = cv2.findContours(mask.astype(np.uint8), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)For full implementation details, see the InstanceSegmentation reference in our documentation. Let us know if you need further clarification!
expen6ve
Hi! I tried training a pretrained yolov8m-seg model on our custom 1 class dataset consisting of 1,307 images with 5 augmentations like:
Flip: Horizontal, Vertical Rotation: Between -15° and +15° Saturation: Between -25% and +25% Blur: Up to 1.4px Noise: Up to 1.49% of pixels
with these training configurations:
# Train the model results = model.train( data="data6.yaml", epochs=250, imgsz=320, batch=8, workers=4, device=device, patience=18, copy_paste=0.1 # Add this line to enable copy-paste augmentation )this is my 16th model training as i tried testing out from 200+ images that and so on. In my val plots, i always get bad results. What can i further do to make a good model? it stopped training at 128 with the results below:
epoch,time,train/box_loss,train/seg_loss,train/cls_loss,train/dfl_loss,metrics/precision(B),metrics/recall(B),metrics/mAP50(B),metrics/mAP50-95(B),metrics/precision(M),metrics/recall(M),metrics/mAP50(M),metrics/mAP50-95(M),val/box_loss,val/seg_loss,val/cls_loss,val/dfl_loss,lr/pg0,lr/pg1,lr/pg2
126,4123.96,0.47628,0.76702,0.30532,0.85667,0.9862,0.96565,0.98972,0.94445,0.9775,0.97566,0.98923,0.83189,0.35178,0.67432,0.26286,0.89191,0.00101,0.00101,0.00101 127,4156.45,0.47613,0.76458,0.30439,0.85872,0.96877,0.98339,0.98913,0.94227,0.98701,0.96962,0.99053,0.83626,0.34228,0.67744,0.25083,0.88958,0.00100208,0.00100208,0.00100208
Glenn JocherUltralytics(edited)@expen6ve thank you for sharing your training details. For improved segmentation results with YOLO26, consider these adjustments:
- Dataset Size: 1.3k images may be insufficient for robust segmentation. Aim for ≥5k high-quality annotated images.
- Augmentation Tuning: Reduce rotation range (try ±5°) and remove noise augmentation, which can degrade mask precision.
- Hyperparameters: Increase imgsz to 640 (standard for segmentation tasks) and lower learning rate to 0.0001 with cosine scheduler.
- Model Selection: Try YOLO26s-seg instead of medium-sized models for better generalization on limited data.
For detailed guidance, see the Segmentation Training Tips in our documentation.
ShreyasChhetri
Hi can we implement a Reinforcement Learning function up above the Yolo training function for more accuracy can it happen ?
Glenn JocherUltralytics@ShreyasChhetri integrating reinforcement learning (RL) with YOLO training is an interesting idea, but YOLO models are not inherently designed for RL workflows. While it is theoretically possible, it would require substantial custom development outside the scope of the provided Ultralytics framework. You might consider exploring hybrid approaches combining RL and computer vision if your use case demands it. For standard training, refer to the existing YOLO training documentation: https://docs.ultralytics.com/modes/train.
ZhangAiyi
I have a question when run the code for Instance Segmentation with Object Tracking. The pycharm said UnicodeEncodeError: 'gbk' codec can't encode character '\u274c' in position 401: illegal multibyte sequence. Does anyone have the similar question? Thank you.
Glenn JocherUltralytics@ZhangAiyi it seems like you're encountering a Unicode encoding issue. You can try setting your environment's encoding to UTF-8 in PyCharm to resolve this. If the problem persists, please check the PyCharm settings or your system's locale configuration.

