Multi-Object Tracking with Ultralytics YOLO

Hi!

Thank you for your great work on YOLO and tracking integration. We are using YOLO with ByteTrack/BoT-SORT for object counting and tracking and want to improve ID stability.

1. We notice that sometimes, even when the object visually looks the same and the scene and motion are stable, the track_id changes. What parameters or reasons should we check to understand why the ID changes unexpectedly?

2. Is the match_thresh parameter based on IoU, and is it possible to print or visualize the IoU used during matching for debugging?

3. Is there a way to output Kalman-predicted boxes of lost tracks so we can analyze them?

4. Could increasing the video FPS improve tracking and IoU consistency, or is this mainly related to detection stability and parameter tuning?

5. Does ByteTrack or BoT-SORT use a min_hits parameter before confirming tracks, or are tracks confirmed immediately after creation?

6. If a track is lost, can a new ID only be assigned on the second frame after reappearance if conf > new_track_thresh and then conf > track_high_thresh, or can it happen immediately on the first detection?

7. Is the track_buffer parameter the number of frames a track is kept using Kalman prediction after it is lost before being deleted?

8. Do you have any tips to improve tracking stability and avoid unnecessary ID switches?

Thank you for your support!

@bhaskar2443053 currently there isn't a built-in script for fine-tuning YOLO-Pose tracking parameters against ground truth. You would need to implement custom evaluation using libraries like TrackEval or MOTChallenge to compute IDF1/HOTA scores and optimize tracker parameters iteratively.

@maxexplorer great questions about tracking stability! Here are the key points:

1. ID changes typically occur due to low IoU between predicted and detected boxes, occlusions, or threshold mismatches. Check track_high_thresh and match_thresh in your tracker config.

2. match_thresh is indeed IoU-based. For debugging, you can add logging to the tracker code, but there's no built-in visualization.

3. Kalman predictions aren't directly exposed, but you can access them by modifying the tracker implementation.

4. Higher FPS generally improves tracking by reducing inter-frame motion, making associations more reliable.

5. Both trackers use confirmation logic - tracks need to meet certain criteria before being confirmed as valid tracks.

6. New tracks can be assigned immediately if detection confidence exceeds new_track_thresh.

7. Correct - track_buffer maintains lost tracks for that many frames using prediction.

8. For stability: increase track_buffer, adjust match_thresh based on your scenario, and ensure consistent detection quality with appropriate conf thresholds.

For detailed parameter tuning, refer to the tracker configuration files and our tracking documentation.

@rohith-aiunika thanks for reporting this. The memory buildup in Python scripts (but not CLI) typically occurs when not using streaming mode. To match the CLI's memory behavior, modify your Python script to process frames iteratively using stream=True:

from ultralytics import YOLO

model = YOLO("/home/best_pt_files/v1.0.2.pt")
for result in model.track("/home/video.MP4", stream=True):  # Process frames iteratively
    pass  # Add your frame processing logic here

This streams frames sequentially like the CLI, preventing memory accumulation. For more details, see the streaming implementation in our Tracking documentation. Let us know if this resolves the issue!

i have used stream = True, then the code is just executing without showing error and without tracking also

@rohith-aiunika thanks for the update. Let's ensure you're properly iterating through streaming results. Try adding a processing loop like this:

for result in model.track(source=video_path, stream=True):
    annotated_frame = result[0].plot()
    cv2.imshow('Tracking', annotated_frame)
    if cv2.waitKey(1) == ord('q'):
        break

This actively processes each frame and displays tracking annotations. If issues persist, you might need to verify your video source format using our supported inference sources documentation.

@mwasela hello,

It's great to hear that your custom model is performing well. To reset the tracking IDs in your Flask app, you can implement a mechanism to reinitialize the tracker state upon receiving a specific request. This can be done by resetting the tracker object or reloading the model within your tracking function.

For more detailed guidance on configuring and using trackers, please refer to our documentation: Multi-Object Tracking with Ultralytics YOLO.

It is kind a security program detecting who entered zone

import cv2 from ultralytics import YOLO import time

Load the YOLOv8 model

model = YOLO("yolov8n.pt") # pretrained YOLOv8n model

Initialize video capture for multiple cameras

camera_ids = [

'rtsp://admin:admin12345@172.17.67.20:554',

'rtsp://admin:admin12345@172.17.67.16:554', 'rtsp://admin:admin12345@172.17.67.18:554', 'rtsp://admin:admin12345@172.17.67.19:554', 'rtsp://admin:admin12345@172.17.67.17:554', 'rtsp://admin:admin12345@172.17.67.21:554', 'rtsp://admin:admin12345@172.17.67.17:554', 'rtsp://admin:admin12345@172.17.67.21:554', 'rtsp://admin:admin12345@172.17.67.17:554',
]

caps = [cv2.VideoCapture(camera_id) for camera_id in camera_ids]

Open the video file

roi_start = [(0, 0)] * len(caps) roi_end = [(0, 0)] * len(caps) drawing = [False] * len(caps)

tracked_persons = [[] for _ in range(len(caps))] # List to track persons in dangerous zones

def draw_rectangle(event, x, y, flags, param): global roi_start, roi_end, drawing idx = param # Use param to identify the camera index if event == cv2.EVENT_LBUTTONDOWN: drawing[idx] = True roi_start[idx] = (x, y) roi_end[idx] = (x, y) print(f"Camera {idx}: Started drawing rectangle at: {roi_start[idx]}") elif event == cv2.EVENT_MOUSEMOVE: if drawing[idx]: roi_end[idx] = (x, y) print(f"Camera {idx}: Updated rectangle end to: {roi_end[idx]}") elif event == cv2.EVENT_LBUTTONUP: drawing[idx] = False roi_end[idx] = (x, y) print(f"Camera {idx}: Finished drawing rectangle at: {roi_end[idx]}")

Create a window and set mouse callback for each camera

for idx in range(len(caps)): window_name = f'Video {idx}' cv2.namedWindow(window_name) cv2.setMouseCallback(window_name, draw_rectangle, idx)

def is_inside_roi(x, y, roi_start, roi_end): x1, y1 = roi_start x2, y2 = roi_end return min(x1, x2) <= x <= max(x1, x2) and min(y1, y2) <= y <= max(y1, y2)

while True: frames = [] for cap in caps: ret, frame = cap.read() if not ret: continue # frame = cv2.resize(frame, (640, 480)) frames.append(frame)

for idx, frame in enumerate(frames):
    window_name = f'Video {idx}'
    
    # Draw the ROI rectangle on the frame
    if drawing[idx] or roi_start[idx] != roi_end[idx]:
        cv2.rectangle(frame, roi_start[idx], roi_end[idx], (0, 255, 0), 2)
        cv2.putText(frame, "Dangerous Zone", (roi_start[idx][0], roi_start[idx][1] - 10), cv2.FONT_HERSHEY_SIMPLEX, 0.9, (0, 255, 0), 2)

    # Run YOLOv8 tracking on the frame, persisting tracks between frames
    results = model.track(frame, tracker="bytetrack.yaml", persist=True)

    for result in results:
        boxes = result.boxes

        for box in boxes:
            x1, y1, x2, y2 = box.xyxy.tolist()[0][:4]
            cls = box.cls.tolist()[0]
            if cls == 0:  # Person class in COCO dataset
                # Check if the center of the person is inside the ROI
                cx, cy = int((x1 + x2) / 2), int((y1 + y2) / 2)
                if is_inside_roi(cx, cy, roi_start[idx], roi_end[idx]):
                    # Check if this person is already tracked
                    person_id = box.id.tolist()[0] if box.id is not None else None
                    
                    if person_id not in tracked_persons[idx]:
                        tracked_persons[idx].append(person_id)
                        # Crop and save the frame
                        crop = frame[int(y1):int(y2), int(x1):int(x2)]
                        cv2.imwrite(f"cropped_camera{idx}_person{person_id}_{int(time.time())}.jpg", crop)
                        # Draw a label on the person
                        cv2.putText(frame, "Person in Dangerous Zone", (int(x1), int(y1) - 10), cv2.FONT_HERSHEY_SIMPLEX, 0.9, (0, 0, 255), 2)
                        cv2.rectangle(frame, (int(x1), int(y1)), (int(x2), int(y2)), (0, 0, 255), 2)

    cv2.imshow(window_name, frame)

if cv2.waitKey(1) & 0xFF == ord('q'):
    break

Release all video captures

for cap in caps: cap.release() cv2.destroyAllWindows()

Your approach looks solid for setting up a security program to track and save images of individuals entering a designated "dangerous zone" across multiple cameras. To enhance your implementation, ensure you're using the latest versions of the Ultralytics package for optimal performance and bug fixes. For more detailed guidance on tracking configurations and additional features, you can refer to the Ultralytics tracking documentation. If you encounter any specific issues or need further assistance, feel free to ask!

File "", line 91 results = model.track(frame, persist=True, tracker="botsort.yaml") ^ IndentationError: unexpected indent COMMENT CORRIGER CETTE ERREUR

The error you're encountering is due to an unexpected indent in your code. Ensure that the indentation is consistent throughout your script. If the issue persists, please verify it with the latest version of the Ultralytics package. For more details on tracking configurations, you can refer to the Ultralytics tracking documentation.

Can I set the parameter ''save_txt'' to True in the track function, and then write code to evaluate the tracking data?

Currently, the Ultralytics YOLO framework doesn't provide built-in functionality for computing tracking metrics like MOTA or IDF1 directly through the API or command line. However, you can utilize external libraries such as MOTChallenge eval for calculating these metrics.

After running your tracking with YOLOv8, you'd typically export your tracking results in a format compatible with these libraries (usually a .txt format with frame ID, object ID, bounding boxes, etc.). Then, use the external library to compute the desired metrics by providing it with both the ground truth and your results file.

Here's a generic example on how you might save tracking results (modify based on your actual use case):

from ultralytics import YOLO

# Initialize and load your model
model = YOLO("path/to/your/model.pt")

# Tracking
results = model.track(source='path/to/video.mp4', save_txt=True, project='runs/track', name='exp')

# results are saved in 'runs/track/exp' in a format that can be used with MOTChallenge eval tools

Afterwards, follow the documentation of the MOTChallenge eval toolkit to compute the metrics using the outputs stored in 'runs/track/exp'.

If you encounter any specific issues while implementing this, feel free to ask!🛠️

@AdadAlShabab yes, you can run two models simultaneously on one video, but you'll need to manage the inference in a way that allows both models to process the video frames concurrently. Here's a simplified example using threading to achieve this:

import threading
from ultralytics import YOLO

# Load the models
model1 = YOLO('E:/Ultralytics/best.pt')
model2 = YOLO('yolov8m-pose.pt')

# Function to run tracking
def track_video(model, source, conf, show, save):
    results = model.track(source=source, show=show, conf=conf, save=save)
    # Process results here or return them

# Create threads for each model
thread1 = threading.Thread(target=track_video, args=(model1, "file.mp4", 0.3, True, False))
thread2 = threading.Thread(target=track_video, args=(model2, "file.mp4", 0.3, True, False))

# Start the threads
thread1.start()
thread2.start()

# Wait for both threads to finish
thread1.join()
thread2.join()

This script will run both models in separate threads. If you want to combine the results from both models, you'll need to implement a mechanism to synchronize and merge the results from both threads. Keep in mind that displaying the results (show=True) from two threads might not work as expected, and you might need to handle the visualization separately.

Remember to handle the GPU memory usage carefully, as running two models simultaneously will require more memory. If you encounter any issues, consider running the models sequentially or on separate GPUs if available.

Use model.track(frame, persist=True) and inspect results[0].boxes.xyxy with results[0].boxes.id; select the box containing your OpenCV mouse-click coordinates, store its track ID, then process or draw only boxes whose boxes.id matches that ID. Feel free to continue posting questions here so the Ultralytics team can help.

Ultralytics track is single-camera MOT, so cross-camera person IDs are not built in; on DeepStream 7.1 the practical Orin Nano path is NvDeepSORT/ReID per stream plus downstream association from the exported embeddings described in the DeepStream gst-nvtracker docs, or NVIDIA’s separate Multi-Camera Tracking workflow for full MTMC. Please keep the details here in Discussions and share your current tracker config if you want a minimal setup suggestion. (docs.nvidia.com)

this is my code ..... from ultralytics import YOLO import cv2 import numpy as np

model = YOLO("runs/segment/pothole_seg_model3/weights/best.pt") pixels_per_foot = 220

video_path = "sample_video.mp4" cap = cv2.VideoCapture(video_path)

if not cap.isOpened(): print(" Cannot open video") exit()

width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH)) height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT)) fps = cap.get(cv2.CAP_PROP_FPS) or 25.0

fourcc = cv2.VideoWriter_fourcc(*'avc1') out = cv2.VideoWriter("pothole5.mp4", fourcc, fps, (width, height))

STRETCH_AMOUNT = 50

src = np.float32([ [0, height],
[width, height],
[922, 183],
[265, 183]
])

dst = np.float32([ [0, height], [width, height], [width + STRETCH_AMOUNT, 0], [0 - STRETCH_AMOUNT, 0] ])

matrix = cv2.getPerspectiveTransform(src, dst) def compute_iou(box1, box2): x1 = max(box1[0], box2[0]) y1 = max(box1[1], box2[1]) x2 = min(box1[2], box2[2]) y2 = min(box1[3], box2[3])

inter_area = max(0, x2 - x1) * max(0, y2 - y1)

box1_area = (box1[2] - box1[0]) * (box1[3] - box1[1])
box2_area = (box2[2] - box2[0]) * (box2[3] - box2[1])

union = box1_area + box2_area - inter_area

if union == 0:
    return 0

return inter_area / union

def should_merge(box1, box2, iou_thresh=0.1, dist_thresh=150): # Containment check - if one box is inside the other if (box1[0] >= box2[0] and box1[1] >= box2[1] and box1[2] <= box2[2] and box1[3] <= box2[3]): return True if (box2[0] >= box1[0] and box2[1] >= box1[1] and box2[2] <= box1[2] and box2[3] <= box1[3]): return True

iou = compute_iou(box1, box2)
if iou > iou_thresh:
    return True

cx1, cy1 = (box1[0]+box1[2])/2, (box1[1]+box1[3])/2
cx2, cy2 = (box2[0]+box2[2])/2, (box2[1]+box2[3])/2

if np.sqrt((cx1-cx2)**2 + (cy1-cy2)**2) < dist_thresh:
    return True

return False

while True: ret, frame = cap.read() if not ret: break

bird_eye = cv2.warpPerspective(frame, matrix, (width, height))

results = model(bird_eye, conf=0.45, iou=0.4)[0]
annotated = results.plot()

if results.boxes is not None:
    for box in results.boxes:
        cls_id = int(box.cls.item())

        if cls_id == 0:  # pothole only
            x1, y1, x2, y2 = map(int, box.xyxy.cpu().numpy()[0])

            w_px = x2 - x1
            h_px = y2 - y1

            w_ft = w_px / pixels_per_foot
            h_ft = h_px / pixels_per_foot

            label = f"{w_ft:.2f}ft x {h_ft:.2f}ft"

            cv2.putText(
                annotated,
                label,
                (x1, y1 - 15),
                cv2.FONT_HERSHEY_SIMPLEX,
                0.6,
                (0, 255, 255),
                2
            )

cv2.imshow("Bird Eye + Pothole Detection", annotated)
out.write(annotated)

if cv2.waitKey(1) & 0xFF == 27:
    break

cap.release() out.release() cv2.destroyAllWindows()

For (1), YOLO’s NMS will only suppress duplicate detections based on iou=...—it won’t “merge” two overlapping detections into a larger union box, and in your snippet compute_iou() / should_merge() aren’t actually applied to results.boxes, so you’re still visualizing the raw post-NMS output; since you’re using a Segment model, the most robust way to merge “split” potholes is to union the pothole masks and re-extract connected components (then box/measure those) rather than trying to union boxes.

import cv2
import numpy as np

r = model(bird_eye, conf=0.45, iou=0.4)[0]

if r.masks is not None and r.boxes is not None:
    cls = r.boxes.cls.int().cpu().numpy()
    pothole_idx = np.where(cls == 0)[0]
    if pothole_idx.size:
        m = r.masks.data[pothole_idx].cpu().numpy()  # (N,H,W), 0/1
        union = (m.sum(0) > 0).astype(np.uint8) * 255
        n, labels = cv2.connectedComponents(union)
        for k in range(1, n):
            ys, xs = np.where(labels == k)
            x1, y1, x2, y2 = xs.min(), ys.min(), xs.max(), ys.max()
            cv2.rectangle(bird_eye, (x1, y1), (x2, y2), (0, 255, 255), 2)

For (2), yes—without real-world calibration (and a valid “road plane” assumption), pixels_per_foot=220 can’t be physically meaningful, and even with a BEV warp you only get consistent scale if your homography is tied to metric coordinates; the usual monocular approach is: calibrate camera intrinsics, estimate the road plane, build a homography to a top-down plane in real units (or derive scale from at least one known road-plane measurement like lane width/stripe spacing), then measure in the BEV plane.

For (3), the best fix is usually data: train both classes explicitly with lots of “hard negatives” (zebra crossings that look like bumps and vice versa) and keep labels consistent; segmentation often helps because you can then add simple shape/location constraints on the mask (zebra = large repeated pattern region, bump = single narrow band across lane) and optionally smooth decisions over time (if you’re doing video, using model.track(..., persist=True) from the Track mode docs can help stabilize per-frame flicker, even though it won’t merge instances within a frame).

If you can paste !yolo checks output and one representative frame (plus what you consider the “correct” merged result), I can suggest concrete thresholds/post-processing for your exact outputs.

In the attached frame:

The large water-filled pothole on the left should be treated as a single instance.

The small region in the top-left corner (currently detected as separate) should be merged with the large region.

Essentially, all visually continuous damaged area should result in one bounding box / one connected component.

Currently:

Mask union + connected components splits them due to small segmentation gaps

Goal:

Treat visually continuous pothole surface as one instance for measurement stability.

That split is expected because connectedComponents() treats any 1–2 px “mask hole” as a hard break, so add a small gap-closing step on the union mask (morphological close / dilate→erode) before components, with the kernel sized to the largest gap you still consider “visually continuous” (see Ultralytics YOLO segmentation docs for more mask/seg usage details):

# union: uint8 {0,255}
union = (m.sum(0) > 0).astype(np.uint8) * 255

# bridge small gaps (tune kernel size to your expected mask discontinuities)
k_close = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (25, 25))
union_closed = cv2.morphologyEx(union, cv2.MORPH_CLOSE, k_close, iterations=1)

# optional: remove tiny speckles after closing
k_open = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (7, 7))
union_closed = cv2.morphologyEx(union_closed, cv2.MORPH_OPEN, k_open, iterations=1)

n, labels = cv2.connectedComponents(union_closed)

@glenn_jocher

@mslabdelhak29 ReID is only supported by some trackers (for example BoT-SORT), not ByteTrack, and it’s enabled via the tracker YAML you pass at runtime. Copy the default config, enable ReID explicitly, and point track() to your copy:

# custom_botsort.yaml
tracker_type: botsort
with_reid: True
model: yolo26n-cls.pt  # or another YOLO classification weight you have locally

from ultralytics import YOLO
model = YOLO("yolo26n.pt")  # any YOLO detector
results = model.track(source="path/to/video.mp4", tracker="custom_botsort.yaml", persist=True, show=False)

If it still doesn’t work, you’re likely editing a different YAML than the one being loaded; always pass the path as above. Also share the output of yolo checks and the exact custom_botsort.yaml you used so we can verify your environment and keys. See the ReID section in the tracking guide and the BoT-SORT config for details: Multi‑Object Tracking docs and botsort.yaml config.