Seguimiento y visualización de experimentos de YOLO con Weights & Biases#

@TJdeWet for custom metric logging beyond default training metrics in YOLO11 with W&B, you can access the complete results dictionary during training and log additional metrics manually through W&B's API. The automatically logged metrics vary by task type (detection/classification), but you can extend them using a callback. For implementation details, refer to the Weights & Biases integration documentation.

@marcory-hub yes, you can integrate Weights & Biases (W&B) with YOLO11 in Google Colab. Please ensure the latest ultralytics and wandb packages are installed using pip install -U ultralytics wandb. If training metrics are not being logged, ensure your W&B environment is authenticated (wandb.login(key="<API_KEY>")) and that project and name parameters are correctly specified in model.train(). Detailed setup instructions are available in our documentation: https://docs.ultralytics.com/integrations/weights-biases/. Let us know if further assistance is needed!

Thanks for the suggestions, @glenn-jocker ! I tried starting from scratch with W&B with YOLO11 but unfortunately, only system information showed up in W&B.

On top of that, it seems like my Colab runtime might be banned due to a potential issue with a compromised package. Hopefully, they'll restore access soon for everyone affected.

Good news! My initial code now works for YOLO11. All metrics are automatically saved to W&B, and training is significantly faster than last week without maxing out Colab's RAM.

I'm not sure why it's significantly faster today, but I suspect the previous slowness prevented training from progressing beyond the warmup epochs, resulting in only system information being sent to W&B.

@marcory-hub glad to hear it's working well now, and thank you for sharing your experience! The faster training could indeed be related to runtime or hardware optimizations in Colab. Feel free to explore additional tips and resources in the YOLO11 W&B integration guide to further enhance your workflow. Let us know if you run into any other issues!

@Pushtogithub23 to log all training metrics and media with Weights & Biases, ensure you have the latest versions of ultralytics and wandb installed. Also, verify that your Weights & Biases account is properly authenticated. For detailed setup instructions, visit the Weights & Biases integration guide.

Hi there!

For your hyperparameter tuning in Weights & Biases, if you're aiming to maximize validation mAP₅₀ specifically, I recommend adjusting your metric settings accordingly. To focus on validation mAP₅₀, your metric configuration might look something like this:

metric:
  name: metrics/mAP50(B)
  goal: maximize

This setup ensures you're targeting the maximization of mAP₅₀ on the validation data directly.

Using val/box_loss with the goal to minimize can also be effective, especially if you prefer to focus on reducing the box prediction errors during validation:

metric:
  name: val/box_loss
  goal: minimize

Both approaches are valid and can be chosen based on your specific model performance goals. Hope this helps! If you have more questions, feel free to ask. 😊

Hi Glenn,

Thanks for getting back to me. I just tried it, and it doesn't seem that tuning to maximize val/mAP50 is working - the only metrics I see that are being logged are: cls, box, and dfl loss (both train and val), and metrics/mAP50(B), metrics/mAP50-75(B), metrics/Recall(B), and metrics/Precision(B), which I believe are the training metrics? I would like to log those same metrics for validation, and tune based on val/mAP50. Is there a way I can do this?

Thanks!

Hi there!

In the Ultralytics W&B integration, validation metrics are logged automatically under names such as metrics/mAP50(B) and metrics/mAP50-95(B) (not val/mAP50; the val/ prefix is used for validation losses like val/box_loss). Use that name in your sweep config:

metric:
  name: metrics/mAP50(B)
  goal: maximize

Enable the integration with yolo settings wandb=True; no manual wandb.log() call is needed.

@JiyongBoo hi there! 😊 RT-DETR trains with the same built-in Weights & Biases integration as other Ultralytics models, so you don't need add_wandb_callback() or manual wandb.log() calls. Enable it with yolo settings wandb=True, then train as usual:

from ultralytics import RTDETR

model = RTDETR("rtdetr-l.pt")
model.train(data="coco8.yaml", epochs=5, project="ultralytics", name="rtdetr-l")

See the Weights & Biases integration guide for details.

If I don't use wandb, the GPU memory hardly changes over each epoch. Is there something wrong with my code?

Hey there! It looks like you're encountering a memory issue related to using Weights & Biases (W&B) with YOLOv8, specifically the GPU running out of memory during training. This can happen for a few reasons, including excessive memory allocation by PyTorch or W&B logging mechanisms. Here are a couple of suggestions that might help:

1. Reduce Batch Size: Lowering the batch size can significantly reduce memory usage. Although this might increase training time, it can help avoid out-of-memory errors.

   model.train(batch=4)  # Try reducing the batch size

2. Clear CUDA Cache: Manually clearing the CUDA cache after each epoch might help release unused memory.

   import torch
   torch.cuda.empty_cache()

3. Optimize Model and DataLoader: Ensure that your DataLoader is not the source of the memory leak, and try to optimize your model's memory usage.

4. Disable W&B to isolate the cause: Run yolo settings wandb=False and train again; if the out-of-memory error persists, W&B is not the cause.

Finally, consider profiling your code to identify specific lines or operations that cause the memory spike. PyTorch and W&B offer tools and documentation on best practices to optimize memory usage. 📘

Hope this helps! If the issue persists, feel free to reach out with more details or check the W&B and YOLOv8 docs for more insights.

Hey Daniel 👋,

Great to hear you're exploring hyperparameter sweeps with WandB and YOLO! It's simpler than it might seem at first. Here's a quick guide to get you started:

1. Initialize WandB: Before starting your training, make sure to log in to WandB in your script or notebook using:

   import wandb
   wandb.login()

2. Set Up the Sweep: Define your sweep configuration in WandB, using Ultralytics training argument names for the hyperparameters. For example:

   method: random
   metric:
     name: val/box_loss
     goal: minimize
   parameters:
     lr0:
       min: 0.0001
       max: 0.001
     batch:
       values: [16, 32, 64]

3. Integrate Sweep with YOLO: Enable the built-in integration with yolo settings wandb=True. If a W&B run is already active (as in a sweep agent), Ultralytics logs into it and keeps any config values the sweep already set.

   import wandb
   from ultralytics import YOLO
   
   
   def train():
       wandb.init(project="your_project_name")
       model = YOLO("yolov8n.pt")
       model.train(data="coco128.yaml", epochs=5, imgsz=640, lr0=wandb.config.lr0, batch=wandb.config.batch)

4. Start the Sweep: Once everything is set up, you can start the sweep in WandB. This will iterate over the defined hyperparameter space, automatically logging each experiment's results.

Remember, the key parts are enabling the W&B setting and making sure your sweep parameter names match Ultralytics training arguments.

Hope this helps! If you have any more questions or run into issues, feel free to ask. Happy sweeping! 🚀

Thank you Paula, this helped a lot! I can make a sweep now but I noticed that the values of the parameters which are given from the sweep to the script dont match the parameters used. Even though the command looks like this: wandb.wandb_agent - INFO - About to run command: /usr/bin/env python training_rtdetr.py --batch=8 --epochs=6 --imgsz=1280 --learning_rate=0.0007013348811086657 where "training_rtdetr.py" is a script with the code you gave me. (I am working with the RTDETR but it shouldnt make a difference as the metrics and calls are the same if I understand correctly?) But then after engine/trainer it says its using epochs=100 and it also uses that number for training. So just to be sure about a few things, I already logged in to WandB, I wrote a script with the code from your 3., I went to wandb, created a sweep there with the code from 2. + the line program: training_rtdetr.py and in a terminal I used the resulting call wandb agent [sweepid] Also I am kinda confused because you used "learning_rate" and "batch_size" but as far as I can tell these are not the parameters I can give for the training and they should be called "lr0" and "batch" respectively (which I both tried but they didnt work) I hope I provided all the necessary information otherwise just tell me what you need to know

@danielst4 hey Daniel,

It sounds like you're on the right track with setting up your WandB sweep with YOLOv8! Regarding the mismatch in parameter values, it seems like there might be a discrepancy between the parameter names expected by your script and those defined in your WandB sweep configuration.

For YOLOv8, the correct parameter names for learning rate and batch size are indeed lr0 and batch, respectively. If you're using these parameters in your training_rtdetr.py script, ensure they're correctly parsed from the command line arguments. Here's a quick Python snippet using argparse for reference:

import argparse

# Parse command line arguments
parser = argparse.ArgumentParser()
parser.add_argument('--batch', type=int, help='batch size')
parser.add_argument('--epochs', type=int, help='number of epochs')
parser.add_argument('--imgsz', type=int, help='image size')
parser.add_argument('--learning_rate', type=float, help='learning rate')
args = parser.parse_args()

# Now you can use args.batch, args.epochs, etc., in your script

Make sure your WandB sweep configuration uses the same parameter names as those your script expects. For example:

program: training_rtdetr.py
method: bayes
metric:
  name: val/box_loss
  goal: minimize
parameters:
  learning_rate:
    min: 0.0001
    max: 0.001
  batch:
    values: [8, 16, 32]
  epochs:
    value: 6
  imgsz:
    value: 1280

If you've already done this and the issue persists, it might be helpful to double-check how your script handles these parameters and ensure they're being correctly applied to your YOLOv8 model configuration.

Feel free to share more details if you're still encountering issues. We're here to help! 🚀

^ this is because I'm done experimenting and need quick retraining options now.

Hey there! 👋 If you're looking to disable automatic logging to Weights & Biases (W&B) during your YOLOv8 training to speed things up, you can do so by setting the wandb argument to False when initiating your training command. Here's how you can do it:

yolo settings wandb=False

This is a global Ultralytics setting, not a train argument, and it stops YOLOv8 from connecting to your W&B account and logging training progress there. Happy training! 🚀