Ultralytics YOLO27: Join the waitlist.
Get Started

Performance Metrics Deep Dive

Introduction#

Performance metrics are key tools to evaluate the accuracy and efficiency of object detection models. They shed light on how effectively a model can identify and localize objects within images. Additionally, they help in understanding the model's handling of false positives and false negatives. These insights are crucial for evaluating and enhancing the model's performance. This guide explores the performance metrics associated with YOLO26, their significance, and how to interpret them.



Watch: Ultralytics YOLO26 Performance Metrics | mAP, F1 Score, Precision, IoU & Accuracy

Object Detection Metrics#

The following metrics are important for YOLO26 and broadly applicable across different object detection models.

  • Intersection over Union (IoU): IoU is a measure that quantifies the overlap between a predicted bounding box and a ground truth bounding box. It plays a fundamental role in evaluating the accuracy of object localization.

  • Average Precision (AP): AP computes the area under the precision-recall curve, providing a single value that encapsulates the model's precision and recall performance.

  • Mean Average Precision (mAP): mAP extends the concept of AP by calculating the average AP values across multiple object classes. This is useful in multi-class object detection scenarios to provide a comprehensive evaluation of the model's performance.

  • Precision and Recall: Precision quantifies the proportion of true positives among all positive predictions, assessing the model's capability to avoid false positives. Recall calculates the proportion of true positives among all actual positives, measuring the model's ability to detect all instances of a class.

  • F1 Score: The F1 Score is the harmonic mean of precision and recall, providing a balanced assessment of a model's performance while considering both false positives and false negatives.

How to Calculate Metrics for YOLO26 Model#

YOLO26's Validation mode computes the evaluation metrics described above.

Using the validation mode is simple. Once you have a trained model, call the model.val() function. This function will then process the validation dataset and return a variety of performance metrics. But what do these metrics mean? And how should you interpret them?

Interpreting the Output#

The sections below break down each segment of the model.val() output.

Class-wise Metrics#

One of the sections of the output is the class-wise breakdown of performance metrics. This granular information is useful when you are trying to understand how well the model is doing for each specific class, especially in datasets with a diverse range of object categories. For each class in the dataset, the following is provided:

  • Class: This denotes the name of the object class, such as "person", "car", or "dog".

  • Images: This metric tells you the number of images in the validation set that contain the object class.

  • Instances: This provides the count of how many times the class appears across all images in the validation set.

  • Box(P, R, mAP50, mAP50-95): This metric provides insights into the model's performance in detecting objects:

    • P (Precision): The accuracy of the detected objects, indicating how many detections were correct.

    • R (Recall): The ability of the model to identify all instances of objects in the images.

    • mAP50: Mean average precision calculated at an intersection over union (IoU) threshold of 0.50. It's a measure of the model's accuracy considering only the "easy" detections.

    • mAP50-95: The average of the mean average precision calculated at varying IoU thresholds, ranging from 0.50 to 0.95. It gives a comprehensive view of the model's performance across different levels of detection difficulty.

Speed Metrics#

The speed of inference can be as critical as accuracy, especially in real-time object detection scenarios. This section breaks down the time taken for various stages of the validation process, from preprocessing to post-processing.

COCO Metrics Evaluation#

For users validating on the COCO dataset, additional metrics are calculated using the COCO evaluation script. These metrics give insights into precision and recall at different IoU thresholds and for objects of different sizes.

Visual Outputs#

The model.val() function, apart from producing numeric metrics, also yields visual outputs that can provide a more intuitive understanding of the model's performance. Here's a breakdown of the visual outputs you can expect:

  • F1 Score Curve (BoxF1_curve.png): This curve represents the F1 score across various thresholds. Interpreting this curve can offer insights into the model's balance between false positives and false negatives over different thresholds.

  • Precision-Recall Curve (BoxPR_curve.png): An integral visualization for any classification problem, this curve showcases the trade-offs between precision and recall at varied thresholds. It becomes especially significant when dealing with imbalanced classes.

  • Precision Curve (BoxP_curve.png): A graphical representation of precision values at different thresholds. This curve helps in understanding how precision varies as the threshold changes.

  • Recall Curve (BoxR_curve.png): Correspondingly, this graph illustrates how the recall values change across different thresholds.

  • Confusion Matrix (confusion_matrix.png): The confusion matrix provides a detailed view of the outcomes, showcasing the counts of true positives, true negatives, false positives, and false negatives for each class.

  • Normalized Confusion Matrix (confusion_matrix_normalized.png): This visualization is a normalized version of the confusion matrix. It represents the data in proportions rather than raw counts. This format makes it simpler to compare the performance across classes.

  • Validation Batch Labels (val_batchX_labels.jpg): These images depict the ground truth labels for distinct batches from the validation dataset. They provide a clear picture of what the objects are and their respective locations as per the dataset.

  • Validation Batch Predictions (val_batchX_pred.jpg): Contrasting the label images, these visuals display the predictions made by the YOLO26 model for the respective batches. By comparing these to the label images, you can easily assess how well the model detects and classifies objects visually.

For the detection, segmentation, and pose tasks, the curve plots are prefixed by metric type: detection writes Box* curves, segmentation writes both Box* and Mask* curves, and pose writes both Box* and Pose* curves.

Results Storage#

For future reference, the results are saved to a directory, typically named runs/detect/val.

Choosing the Right Metrics#

Choosing the right metrics to evaluate often depends on the specific application.

  • mAP: Suitable for a broad assessment of model performance.

  • IoU: Essential when precise object location is crucial.

  • Precision: Important when minimizing false detections is a priority.

  • Recall: Vital when it's important to detect every instance of an object.

  • F1 Score: Useful when a balance between precision and recall is needed.

For real-time applications, speed metrics like FPS (Frames Per Second) and latency are crucial to ensure timely results.

Interpretation of Results#

Here's what some commonly observed low scores might suggest:

  • Low mAP: Indicates the model may need general refinements.

  • Low IoU: The model might be struggling to pinpoint objects accurately. Different bounding box methods could help.

  • Low Precision: The model may be detecting too many non-existent objects. Adjusting confidence thresholds might reduce this.

  • Low Recall: The model could be missing real objects. Improving feature extraction or using more data might help.

  • Imbalanced F1 Score: There's a disparity between precision and recall.

  • Class-specific AP: Low scores here can highlight classes the model struggles with.

Case Studies#

Real-world examples can help clarify how these metrics work in practice.

Case 1#

  • Situation: mAP and F1 Score are suboptimal; Recall is good, but Precision isn't.

  • Interpretation & Action: There might be too many incorrect detections. Tightening confidence thresholds could reduce these, though it might also slightly decrease recall.

Case 2#

  • Situation: mAP and Recall are acceptable, but IoU is lacking.

  • Interpretation & Action: The model detects objects well but might not be localizing them precisely. Refining bounding box predictions might help.

Case 3#

  • Situation: Some classes have a much lower AP than others, even with a decent overall mAP.

  • Interpretation & Action: These classes might be more challenging for the model. Using more data for these classes or adjusting class weights with the cls_pw training argument could be beneficial.

Connect and Collaborate#

The YOLO26 community can help with learning, troubleshooting, and networking through the following channels.

Engage with the Broader Community#

  • GitHub Issues: The Ultralytics repository on GitHub has an Issues tab where you can ask questions, report bugs, and suggest new features. The community and maintainers are active here, and it's a great place to get help with specific problems.

  • Ultralytics Discord Server: Ultralytics has a Discord server where you can interact with other users and the developers.

Official Documentation and Resources#

  • Ultralytics YOLO26 Docs: The official documentation provides a comprehensive overview of YOLO26, along with guides on installation, usage, and troubleshooting.

These resources can help you resolve problems and stay current with best practices in the YOLO26 community.

Conclusion#

In this guide, we've taken a close look at the essential performance metrics for YOLO26. These metrics are key to understanding how well a model is performing and are vital for anyone aiming to fine-tune their models. They offer the necessary insights for improvements and to make sure the model works effectively in real-life situations.

The YOLO26 and Ultralytics community is also a useful resource: engaging with other developers and experts can surface solutions not covered in the documentation, and sharing your findings helps others working on object detection.

FAQ#

  • Mean Average Precision (mAP) is crucial for evaluating YOLO26 models as it provides a single metric encapsulating precision and recall across multiple classes. mAP50 measures precision at an IoU threshold of 0.50, focusing on the model's ability to detect objects correctly. mAP50-95 averages precision across a range of IoU thresholds, offering a comprehensive assessment of detection performance. High mAP scores indicate that the model effectively balances precision and recall, essential for applications like autonomous driving and surveillance systems where both accurate detection and minimal false alarms are critical.

  • Intersection over Union (IoU) measures the overlap between the predicted and ground truth bounding boxes. IoU values range from 0 to 1, where higher values indicate better localization accuracy. An IoU of 1.0 means perfect alignment. Typically, an IoU threshold of 0.50 is used to define true positives in metrics like mAP. Lower IoU values suggest that the model struggles with precise object localization, which can be improved by refining bounding box regression or increasing annotation accuracy in your training dataset.

  • The F1 Score is important for evaluating YOLO26 models because it provides a harmonic mean of precision and recall, balancing both false positives and false negatives. It is particularly valuable when dealing with imbalanced datasets or applications where either precision or recall alone is insufficient. A high F1 Score indicates that the model effectively detects objects while minimizing both missed detections and false alarms, making it suitable for critical applications like security systems and medical imaging.

  • Ultralytics YOLO26 offers multiple advantages for real-time object detection:

    This makes YOLO26 well suited to diverse applications, from autonomous vehicles to smart city solutions.

  • Validation metrics from YOLO26 like precision, recall, mAP, and IoU help diagnose and improve model performance by providing insights into different aspects of detection:

    • Precision: Helps identify and minimize false positives.
    • Recall: Ensures all relevant objects are detected.
    • mAP: Offers an overall performance snapshot, guiding general improvements.
    • IoU: Helps fine-tune object localization accuracy.

    By analyzing these metrics, specific weaknesses can be targeted, such as adjusting confidence thresholds to improve precision or gathering more diverse data to enhance recall. For detailed explanations of these metrics and how to interpret them, check Object Detection Metrics and consider implementing hyperparameter tuning to optimize your model.

Contributors
glenn-jocherglenn-jocher15raimbekovmraimbekovm2RizwanMunawarRizwanMunawar2miles-deans-ultralyticsmiles-deans-ultralytics1pderrengerpderrenger1Y-T-GY-T-G1WillieMaddoxWillieMaddox1leonnilleonnil1

Comments109

Ask a question, share your results, or help someone else.

  1. atharvahudeatharvahudeGitHub

    Question: Clarification on AP vs mAP in YOLOv8 metrics output

    Hello,

    I'm using YOLOv8 for object detection, and I have some questions about the metrics output, particularly regarding the use of map50 and map50-90 in the results. In the output, I'm seeing these metrics reported for individual classes as well as an "all" category that aggregates the performance.

    My understanding is that:

    AP (Average Precision) should be calculated for each individual class.

    mAP (mean Average Precision) is the average of the AP values across all classes.

    However, in the output, the terms map50 and map50-90 are listed for each individual class. Should these values be interpreted as AP for each class, rather than mAP, which I would expect to be the mean across all classes? Additionally, the "all" row appears to be giving the mAP value.

    Could you confirm:

    1. Should the per-class map50 and map50-90 values be treated as AP for those specific classes?

    2. Is the "all" row correctly representing the mean AP across all classes?

    Thanks in advance for your clarification!

    1
    2. Glenn JocherGlenn JocherUltralytics

      @atharvahude the per-class map50 and map50-95 values should indeed be treated as AP for those specific classes. The "all" row represents the mean AP across all classes, confirming your understanding. For more details, you can refer to our documentation on performance metrics.

  2. e-roee-roeGitHub

    Hi, Great guide on performance metrics — they're key tools to evaluate the accuracy and efficiency of object detection models. However, although the plots are shown, I think the train/box_loss, train/obj_loss, and train/cls_loss plots weren't explained — or did I miss something? Thanks!

    1. Glenn JocherGlenn JocherUltralytics(edited)

      @e-roe great observation! You're right that the training loss plots could use more explanation. The training losses shown during YOLO11 training represent:

      • train/box_loss: Localization loss measuring how well predicted bounding boxes match ground truth coordinates
      • train/cls_loss: Classification loss measuring how accurately the model predicts object classes
      • train/dfl_loss: Distribution Focal Loss, which helps improve localization accuracy by treating bounding box regression as a classification problem (YOLO26 is DFL-free, so this term is a plain L1 box-distance loss there)

      These training losses should generally decrease over epochs, indicating the model is learning. The validation losses (val/box_loss, val/cls_loss, val/dfl_loss) help monitor overfitting - if training losses decrease but validation losses increase or plateau, it suggests overfitting.

      Thanks for pointing this out! We'll consider expanding the training loss explanations in future documentation updates.

  3. Hemanth-mandava-neuropiHemanth-mandava-neuropiGitHub

    @glenn-jocher I'm working on a YOLOv8 segmentation task to detect pallets. I've annotated a dataset using Roboflow with 6000 images, split into train, test, and validation sets. After training, I obtained the following results:

    Class Images Instances Box(P R mAP50 mAP50-95) Mask(P R mAP50 mAP50-95) all 1227 2262 0.974 0.489 0.689 0.636 0.972 0.488 0.690 0.624 The precision is high, but the recall is quite low (~0.48), which suggests many true pallets are being missed.

    While inspecting the Roboflow data analytics, I noticed something interesting:

    Around 4000 images have only 1 annotated pallet

    The rest have 2–10 pallets annotated

    This leads me to the following questions:

    Does this cause class imbalance in the dataset, particularly affecting model generalization and recall?

    Should I balance the dataset between single-object and multi-object images? Is it worth filtering or augmenting to even it out?

    Since I’m combining pre-annotated data from multiple sources, is there a way to analyze the distribution of annotations/images before training, especially when not using Roboflow’s UI?

    What techniques are most effective to boost recall in segmentation tasks? Should I look into:

    Lowering confidence thresholds?

    Data augmentation (mosaic, copy-paste)?

    Label smoothing or loss function tweaks?

    Different anchor/mask threshold settings?

    Any help or insights would be greatly appreciate

  4. Fatima-khanjarFatima-khanjarGitHub(edited)

    Hello!

    I hope this message finds you well.

    I have a question regarding the model's performance. Typically, to evaluate the model's performance, we look at precision, recall, and F1 scores. However, these metrics alone may not be sufficient to confirm that the model is performing well. We should also consider the values of DFL, classification (cls) loss, and box loss. My question is: What are the values of these losses that are considered good? Should they be something like 0.00xxxx? Additionally, what is the significance of having high precision, recall, and F1 scores (above 0.8), while at the same time, the loss values are around 0.3 and 0.6, and when we look at the plots of losses (train vs. val), we see that both losses are decreasing over the epochs?

    I have another question: The plots and values for training and validation losses are saved, but how can I extract the overall values of train/val losses during the training process?

    Thank you for your help!

    Regards,

    1. Glenn JocherGlenn JocherUltralytics

      @Fatima-khanjar excellent questions about model evaluation!

      For DFL, classification, and box losses, there are no absolute "good" values as they depend on your dataset complexity, model size, and task. What's more important is the trend - consistently decreasing losses indicate proper training. Loss values of 0.3-0.6 with high precision/recall (>0.8) can be completely normal, as metrics often continue improving even after losses plateau.

      To extract training/validation loss values programmatically:

      import pandas as pd

# Load the results CSV
results_df = pd.read_csv('path/to/results.csv')

# Extract train/val losses
train_box_loss = results_df['train/box_loss']
val_box_loss = results_df['val/box_loss']
train_cls_loss = results_df['train/cls_loss']
val_cls_loss = results_df['val/cls_loss']
train_dfl_loss = results_df['train/dfl_loss']
val_dfl_loss = results_df['val/dfl_loss']

      Focus on comparing validation metrics against your application requirements rather than trying to reach arbitrary loss values. A model with slightly higher losses but better real-world performance is ultimately more valuable.

      1
    2. maomao123-gitmaomao123-gitGitHub

      Hi！how to print f1 score in model.val() ? In addition, I want to print metrics with four decimal places so that I can get percentage form. For example, the printed mAP50 was 0.8657, and I can transfer the number to 86.57%. Where and how can I revise the code? Thanks!

    3. Glenn JocherGlenn JocherUltralytics(edited)

      @maomao123-git to print the F1 score and format metrics with four decimal places when running validation, you can access these values from the results object:

      from ultralytics import YOLO

# Load your model
model = YOLO('yolo26n.pt')  # or your custom model path

# Run validation
results = model.val()

# results.box.f1 holds one F1 score per class
f1 = results.box.f1.mean()
map50 = results.box.map50

# Format and print
print(f"F1 Score: {f1:.4f} ({f1*100:.2f}%)")
print(f"mAP50: {map50:.4f} ({map50*100:.2f}%)")

# For class-specific metrics
for i, f1_val in enumerate(results.box.f1):
    class_name = model.names[results.box.ap_class_index[i]]
    print(f"Class '{class_name}' F1 Score: {f1_val:.4f} ({f1_val*100:.2f}%)")

      This gives you access to both overall and class-specific F1 scores with your desired formatting. The metrics class includes all the performance metrics like precision, recall, mAP, and F1 score, which you can format as needed.

      2
    4. maomao123-gitmaomao123-gitGitHub

      Thanks for your help. I can print f1 of per class, yet there is no f1 of class "all". How can I get it?

    5. Glenn JocherGlenn JocherUltralytics(edited)

      @maomao123-git to get the F1 score for "all" classes (mean F1 across all classes), you can calculate it from the overall precision and recall values:

      # Run validation
results = model.val()

# Get overall precision and recall
precision_all = results.box.mp  # mean precision
recall_all = results.box.mr     # mean recall

# Calculate F1 score for "all" classes
f1_all = 2 * (precision_all * recall_all) / (precision_all + recall_all) if (precision_all + recall_all) > 0 else 0

# Format and print
print(f"All Classes F1 Score: {f1_all:.4f} ({f1_all*100:.2f}%)")

      Alternatively, you can calculate the mean of class-specific F1 scores:

      # Calculate mean of all class F1 scores
mean_f1 = results.box.f1.mean()
print(f"Mean F1 Score: {mean_f1:.4f} ({mean_f1*100:.2f}%)")

      Both methods should give you a good representation of the overall F1 score that would appear in the "all" row.

      2
    7. Glenn JocherGlenn JocherUltralytics

      @Shubham77saini the precision calculation in YOLOv8 considers confidence thresholds and precision-recall curve interpolation, which can lead to different values than direct TP/(TP+FP) calculations. The reported 0.734 precision for the "car" class is likely affected by how the model handles confidence scores across the precision-recall curve.

      To understand this better, you might want to:

      1. Check the confusion matrix at different confidence thresholds
      2. Look at P-R curves in the validation output directory
      3. Verify that your IoU threshold (0.5) matches what's being used for metric calculation

      The metrics implementation computes precision across a range of confidence thresholds and may interpolate values between operating points, which explains why the reported value differs from your expected calculation. For deeper investigation, check the full metrics implementation in the ultralytics/utils/metrics.py file.

    8. maomao123-gitmaomao123-gitGitHub

      glenn-jocher Apr 18, 2025

      @glenn-jocher, if we can get the ap50-95, I think we also can get recall50-95, precision50-95, f1-score50-95. When I print metrics.box.r(), metrics.box.p()……, I just see the single value of each class. Are the values under IoU = 0.5 if IoU is in default? Is it possible to get recall50-95, precision50-95? thank you!

  5. Fatima-khanjarFatima-khanjarGitHub

    Hello,

    I hope this message finds you well. I have two questions:

    • There are plots showing different metrics. How can I use them to extract the performance values of the model, given that there are various confidence thresholds? During validation, the default IoU in YOLOv11 is 0.6. Does it remain stable when calculating precision, recall, and F1 score?
    • During validation, I see outputs like this: Class Images Instances Box(P) R mAP50 mAP50-95 all 30 240 0.917 0.911 0.973 0.817 Can I use these values to report the performance of the model? If yes, how can I extract them through coding?

    best regards,

    1. Glenn JocherGlenn JocherUltralytics(edited)

      For accessing validation metrics programmatically, you can retrieve them directly from the results object after running model.val(). Here's a concise example:

      results = model.val()
print(f"Precision: {results.box.mp}, Recall: {results.box.mr}, mAP50: {results.box.map50}, mAP50-95: {results.box.map}")

      The default iou=0.7 in val() is the NMS threshold, not the precision/recall threshold. Precision and recall are computed at IoU 0.5 at the confidence that maximizes F1, while mAP50-95 averages over IoU 0.5-0.95. The class-wise table values are valid for performance reporting - full implementation details are in the Model Evaluation Insights guide.

  6. e-roee-roeGitHub

    Hello,

    Looking at the confusion matrices, I find the normalized one strange when compared to the non-normalized version. I decided to train a model with only one class to try to understand it, but the result left me even more confused. Could you explain the meaning of this normalized confusion matrix? Ultralitics version: 8.3.49 Non-normalized: 18953 ---- 159 12 ---- 0

    Normalized: 1.00 ---- 1.00 0.00 ---- 0.00

    Ps. I don't know how to attach an image here Thank you.

    1. Glenn JocherGlenn JocherUltralytics(edited)

      @e-roe the normalized confusion matrix divides each column (a true class) by its total, so every column sums to 1. Rows are predictions and columns are true classes, including a background row/column. In your single-class case:

      • True object column: 18953 correct detections and 12 missed objects, so 18953/18965 ≈ 1.00 and 12/18965 ≈ 0.00
      • True background column: 159 false positives (background predicted as an object), so 159/159 = 1.00 and 0.00

      For deeper analysis, see our Performance Metrics Guide. Let us know if you need clarification on specific values! 🚀

    2. e-roee-roeGitHub

      @glenn-jocher Thank you for your response.

      If it is row-wise normalization, shouldn't the sum of each row be 1.0? In Scikit-learn, the normalized confusion matrix appears as I expected, in my single-class case: 0.99168062 ---- 0.00831938

      1. ---- 0.

      This result is much easier for me to compare with the non-normalized confusion matrix and interpret. []'s

    3. e-roee-roeGitHub

      Hi, Oh, I see... I think you calculate it this way: cm_normalized = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis] but I was expecting it to be like this: cm_normalized = cm.astype('float') / cm.sum(axis=0)[np.newaxis, :]

      Ok, thank you again.

    4. Glenn JocherGlenn JocherUltralytics(edited)

      Thank you for catching this nuance! Ultralytics normalizes each column of the confusion matrix, which corresponds to a true class (rows are predictions), rather than each row as in scikit-learn's default. Each column therefore shows how the instances of one true class were predicted.

  7. Keletom1Keletom1GitHub

    Problem with figuring out the IoU

    Hey there,

    i had been having some trouble with my models and finding out how they perform on new Data. I will explain, what i have been doing and hopefully someone can find out where i was going wrong :D

    What have I done so far:

    I have trained multiple models on different datasets and now i want to find out, how they perform when they are exposed to new data, some of which they have seen, some of which are completely new. The models are all single class segmentation YOLOv8 models and trained for 300 epochs, nothing too special because i just want to see how different models (trained of different datasets) perform on my test dataset. I had all my models run over said test dataset (Prüfdatensatz_2) of 62 images. I had the annotations saved as png and also as JSON. After that i went ahead and annotated all of my 62 images manually with roboflow and exported these as a JSON as well. So now i have the prediction-json files (one for each image) and one big GroundTruth-json file and i want to compare them and get a big results.csv in which the images names, the models they belong to and the Intersection over Union for each image are displayed. I have tried different approaches and the script itself does work, only the IoU-calculation does not seem to work. It's either 0.0 or 1.0 (the latter in case there is nothing to annotate and the model didn't predict anything in this image).

    My Code:
    import json
import os
import numpy as np
import pandas as pd
import re
from shapely.geometry import Polygon

# **1. COCO JSON laden**
coco_json_path = r"C:\Users\wavet\Desktop\OneDrive\Uni\Semester 9\Bachelorarbeit\Datensätze\COCO_Prüfdatensatz_2-2\train\_annotations.coco.json"
with open(coco_json_path, "r") as f:
    coco_data = json.load(f)

# **2. Mapping-Tabelle für COCO JSON erstellen (Dateiendungen und Suffixe entfernen)**
def clean_filename(filename):
    cleaned = re.sub(r"_png\.rf\..*|_jpg\.rf\..*", "", filename)  # Entfernt '.rf.<hash>' und Dateiendungen
    print(f"Original: {filename}, Bereinigt: {cleaned}")  # Debug-Ausgabe
    return cleaned

image_id_to_filename = {img["id"]: clean_filename(img["file_name"]) for img in coco_data["images"]}
gt_annotations = {}

# **3. Konvertiert Segmentierungen in Polygone für IoU-Berechnung**
def segmentation_to_polygon(segmentation):
    coordinates = [(segmentation[i], segmentation[i+1]) for i in range(0, len(segmentation), 2)]
    if len(coordinates) < 4:
        return None  # Weniger als 4 Punkte = kein gültiges Polygon
    return Polygon(coordinates)

# **4. Ground-Truth-Annotationen nach `image_id` speichern**
annotations_by_image = {}
for ann in coco_data["annotations"]:
    img_id = ann["image_id"]
    if img_id not in annotations_by_image:
        annotations_by_image[img_id] = []
    if "segmentation" in ann and len(ann["segmentation"]) > 0:
        polygon = segmentation_to_polygon(ann["segmentation"][0])
        if polygon:
            annotations_by_image[img_id].append(polygon)

# **5. Modell-Outputs einlesen & IoU berechnen**
predictions_dir = r"C:\Users\wavet\Desktop\OneDrive\Uni\Semester 9\Bachelorarbeit\Ergebnisse_BA\Datensatzstudie\e1000\output_Prüfdatensatz_2_Skaliert"
csv_output_path = r"C:\Users\wavet\Desktop\OneDrive\Uni\Semester 9\Bachelorarbeit\Ergebnisse_BA\Datensatzstudie\Auswertung_e1000\iou_results_v5.csv"

def calculate_iou(polyA, polyB):
    intersection = polyA.intersection(polyB).area
    union = polyA.union(polyB).area
    iou = intersection / union if union > 0 else 0
    print(f"Intersection: {intersection}, Union: {union}, IoU: {iou}")  # Debug-Ausgabe
    return iou

iou_results = []

for model_name in os.listdir(predictions_dir):
    model_path = os.path.join(predictions_dir, model_name)
    if not os.path.isdir(model_path):
        continue

    print(f"Verarbeite Modell: {model_name}")

    for json_file in os.listdir(model_path):
        if json_file.endswith(".json"):
            img_name = clean_filename(json_file.replace(".json", ""))

            # **6. Passende `image_id` suchen**
            matching_ids = [img_id for img_id, file in image_id_to_filename.items() if file == img_name]
            if not matching_ids:
                print(f"WARNUNG: Keine `image_id` für {img_name} gefunden!")
                continue
            image_id = matching_ids[0]  # Falls mehrere IDs vorhanden sind, nehmen wir die erste

            # **7. GT-Annotationen laden**
            gt_polygons = annotations_by_image.get(image_id, [])

            # **8. Predictions laden**
            with open(os.path.join(model_path, json_file), "r") as f:
                pred_data = json.load(f)

            if not isinstance(pred_data, list):
                pred_data = [pred_data]  # Falls nur eine Vorhersage vorliegt, in Liste umwandeln

            pred_polygons = []
            for pred in pred_data:
                if "segmentation" in pred and len(pred["segmentation"]) > 0:
                    polygon = segmentation_to_polygon(pred["segmentation"][0])
                    if polygon:
                        pred_polygons.append(polygon)

            # **9. IoU berechnen**
            if not gt_polygons and not pred_polygons:
                avg_iou = 1.0  # Keine GT und keine Prediction → Perfekte Erkennung
            elif not gt_polygons or not pred_polygons:
                avg_iou = 0.0  # Nur eine der beiden vorhanden → Fehlerkennung
            else:
                total_iou = 0.0
                matches = 0
                used_preds = set()

                for gt_polygon in gt_polygons:
                    best_iou = 0.0
                    best_pred = None

                    for i, pred_polygon in enumerate(pred_polygons):
                        if i in used_preds:
                            continue
                        iou = calculate_iou(gt_polygon, pred_polygon)
                        if iou > best_iou:
                            best_iou = iou
                            best_pred = i

                    if best_pred is not None:
                        used_preds.add(best_pred)
                        total_iou += best_iou
                        matches += 1

                missing_objects = len(gt_polygons) - matches
                avg_iou = (total_iou + missing_objects * 0.0) / len(gt_polygons)

            iou_results.append([model_name, img_name, avg_iou, np.mean([pred.get("confidence", 100) for pred in pred_data])])

# **10. Ergebnisse speichern**
iou_df = pd.DataFrame(iou_results, columns=["Model", "Image", "IoU", "Confidence"])
iou_df.to_csv(csv_output_path, index=False)
print(f"✅ IoU-Ergebnisse gespeichert unter: {csv_output_path}")
    What i checked:
    • segmentation coordinates are absolute for both JSONs
    • the coordinates seem to match, i was able to plot some of the prediction segmentations
    • alle img_ids were assigned correctly
    Questions:
    • What have I been doing wrong?
    • I have the feeling that there might be a much simpler way to find out how my models perform on new data and I just haven't found it yet. Is that so?

    Thank you all in advance!!

    Greetings, Tom

    1. Glenn JocherGlenn JocherUltralytics

      @Keletom1 hi Tom! For evaluating segmentation models, we recommend using YOLOv8's native validation mode which automatically calculates all metrics including mAP and IoU. Simply run:

      from ultralytics import YOLO
model = YOLO("your-model.pt")
metrics = model.val(data='your-dataset.yaml')

      This handles all ground-truth/prediction matching and metric calculations out-of-the-box. The validation docs show how to interpret results: https://docs.ultralytics.com/modes/val/. For custom analysis, you could extend the built-in metrics calculation from our AP implementation: https://docs.ultralytics.com/reference/utils/metrics/#ap_per_class

    2. Keletom1Keletom1GitHub

      Hi Glenn,

      thanks (again) for your help!

      That is some great information and saves quite a lot of work for me. I shoud have discovered that earlier.

      Anyway i still have some minor questions regarding the validation.

      1. The standard output for my segmentation model are plots of precision, recall, precision/recall and F1-score (for both the box and mask predictions). Since i don't need to compare the box recognition, is there any way i can disable this output?
      2. The IoU is not a metric that is calculated, is it? At least it is not in the predictions.json or the plots. Any idea on how i can display or calculate it anyway? Or would you recommend to just work with precision and recall?
      3. Could you briefly explain what exactly the JSON displays. I was expecting coordinates under "segmentation" but it just looks like a bunch of random letters and numbers....

      Here is an example of the first to images in my dataset:

      {"image_id": "-0008_1_png.rf.53cbd30c302485bebd6f33083f141f1c", "category_id": 1, "bbox": [222.615, 9.455, 270.527, 614.687], "score": 0.87661, "segmentation": {"size": [640, 640], "counts": "Uh[43Uch01dPXO3P]OJRb0:n]OMha06X^OM`a08`^OLXa08h^OLPa08P_OLg`09Y_OIb`0:^_OI\\`0:d_OKS`09m_OLj?8X@Ib?:b@FV?7TAJc>7dAHU>:QBFi=:]BE^=9kBHk<9[CJ\\<8iCIP<7WDJa;5gDLo:2ZE0\\:3jELQ:4VFKd93eFOS9OSG5c8MbG3X8NmG3l7K]H7X7IPI8f6L^I4]6MiI2Q6LYJ4`5JfJ<o4EVK;c4GcK9U4HRL;b3ChL>P3DTM=f2DaM:Z2EmM=j1A]Nd0Z1\\OjNe0Q1[OUOd0g0XOCj04TO3n0lKkM@W1h4n0eKoM]OT1U5k0[KRN]OR1`5l0mJSNBn0g5X1\\JjMKm0n5Z1SJkMLl0V6X1kIlMNh0a6^1ZIjM4d0i6h1kHdM:e0o6g1dHdM<d0W7e1[HgM<c0a7g1mGfMa0`0g7S2^G]Mj0?l7U2WG\\Ml0>T8U2lF]Mn0<_8W2]F^MR19g8Z2TF]MT18l8[2mE]MV18U9X2`EbMZ14^9Y2REdM`10c9^2iDbMc10h9]2dDcMc10P:Z2YDgMf1LY:]2mCfMj1I`:b2bCfMm1Hd:b2]CfMn1Hm:^2QCkMm?W2j_OmMT2^Od;`3ZDbLh;\\3nC^L]N2Q>W3aClLl<P3QCSMP=l2iB\\MY=a2dB[Ml=\\2nAZMhN:a?W2fAmM\\>R2cAoM`>n1^AfMnN2U`0j1j@UNe?j1X@WNk?g1U@YNn?d1a1G3M1O2N5K4L2N1O1O000O011O0O10002M2O1O1OO0100O010O10O0010O0010O01O0010O01O001O010O010O1O010O001O10O01O010O100O00100O00100O00100O010O10O0100O00100O010O010O10O01O010O1O010O010O1O010O0010O010O010O01O010O001O010O010O01O0100O00010O010O010O0010O01O01O010O010O01O01O01O000010O00010O01O1O001O00_kj2"}}, {"image_id": "-0008_1_png.rf.53cbd30c302485bebd6f33083f141f1c", "category_id": 1, "bbox": [148.255, 393.207, 71.421, 246.348], "score": 0.00426, "segmentation": {"size": [640, 640], "counts": "Wgm2b0\\c0O1O00O2OO01O010100OO0O21O00N01200OO0O111O0OO2O1100N00010000N12O10OO1N030O00OO11100OM20101OO1M311O0N00O21O00OO21O10OOO1110OOO110100N1O01101NhSV8"}}, {"image_id": "-0008_1_png.rf.53cbd30c302485bebd6f33083f141f1c", "category_id": 1, "bbox": [146.245, 394.098, 186.875, 244.714], "score": 0.00358, "segmentation": {"size": [640, 640], "counts": "`_l2?\\c02O11O1O00O2OO01O010100OO0O21O0OO1020O0O00011O0OO2O1100N0001000OO21O10OO1N030O00OO11100ON1O2010N0N311O0N00O21O00OO21O10OOO1110OO0001100N1O01101N3I7N2N[og02_PXO>F2K5L4L4K5M3K5M1ON11N30OOM30N4OO1N11M3010J40N4N10M4N12M11M1OO4N2ON3O01O1NOO112N20N2N2O101L4M21O0OO3M3O1O0M3N3O001N1N30O00M3N30O01M2N21O10M2M40mjn5"}}

      Thank you very much!

      Kind regards, Tom

    3. Glenn JocherGlenn JocherUltralytics

      Hi Tom,

      1. Disabling box metrics: Currently, segmentation validation outputs both box and mask metrics together. You can safely ignore the box-related metrics/plots if you're focused on segmentation, but there's no built-in flag to disable box outputs specifically.

      2. IoU calculation: While per-image IoU isn't directly output, mAP metrics (mAP50, mAP50-95) in validation results are calculated using IoU thresholds. For custom per-instance IoU calculations, see our mask_iou function in the metrics utils.

      3. Segmentation format: The JSON uses Run-Length Encoding (RLE) for compact storage of masks. You can decode these with pycocotools (COCO API) or use our native Results class to access polygon coordinates directly from predictions.

      For deeper analysis, consider using the COCO evaluation tools which handle these formats natively.

      1
  8. vonzovonzoGitHub

    After training I get the weights for "last" and "best", how do i know which line in the results.csv correspond to the "best" weights?

    1. Glenn JocherGlenn JocherUltralytics(edited)

      @vonzo the 'best.pt' weights correspond to the epoch with the highest validation mAP50-95 metric. In your results.csv, look for the row where the metrics/mAP50-95(B) column (or similar mAP column for your task) reaches its maximum value - this is the epoch where best.pt was last saved.

  9. Aniket1210Aniket1210GitHub

    Hello Team, How can be find latency number for TSR yolov8 model on RTX 4049 system ?

    1. Glenn JocherGlenn JocherUltralytics

      @Aniket1210 to measure latency for the YOLOv8 TSR model on an RTX 4090, you can use the model.val() or model.predict() functions with the verbose=True parameter in the Ultralytics API to retrieve timing details. Ensure your setup leverages the GPU (device='0') for accurate benchmarking. For further guidance, refer to the Ultralytics Val documentation.

  10. xiaoshuai123666xiaoshuai123666GitHub

    这是来自QQ邮箱的自动回复邮件。你好，您的邮件我已收到，感谢您的来信，我一定仔细阅读您的邮件。                                                            ——xiaoshuai