Machine Learning Best Practices and Tips for Model Training

Introduction#

One of the most important steps when working on a computer vision project is model training. Before reaching this step, you need to define your goals and collect and annotate your data. After preprocessing the data to make sure it is clean and consistent, you can move on to training your model.

Model training is the process of teaching your model to recognize visual patterns and make predictions from your data, and it directly shapes the accuracy of your application. This guide walks through best practices, optimization techniques, and troubleshooting tips to help you train your computer vision models effectively.

Watch: Model Training Tips | How to Handle Large Datasets | Batch Size, GPU Utilization and Mixed Precision

How to Train a Machine Learning Model#

A computer vision model is trained by adjusting its internal parameters to minimize errors. Initially, the model is fed a large set of labeled images. It makes predictions about what is in these images, and the predictions are compared to the actual labels or contents to calculate errors. These errors show how far off the model's predictions are from the true values.

During training, the model iteratively makes predictions, calculates errors, and updates its parameters through a process called backpropagation. In this process, the model adjusts its internal parameters (weights and biases) to reduce the errors. By repeating this cycle many times, the model gradually improves its accuracy. Over time, it learns to recognize complex patterns such as shapes, colors, and textures.

This learning process makes it possible for the computer vision model to perform various tasks, including object detection, instance segmentation, semantic segmentation, and image classification. The ultimate goal is to create a model that can generalize its learning to new, unseen images so that it can accurately understand visual data in real-world applications.

Now that we know what is happening behind the scenes when we train a model, let's look at points to consider when training a model.

Training on Large Datasets#

There are a few different aspects to think about when you are planning on using a large dataset to train a model. For example, you can adjust the batch size, control the GPU utilization, choose to use multiscale training, etc. Let's walk through each of these options in detail.

Batch Size and GPU Utilization#

When training models on large datasets, efficiently utilizing your GPU is key. Batch size is an important factor. It is the number of data samples that a machine learning model processes in a single training iteration. Using the maximum batch size supported by your GPU, you can fully take advantage of its capabilities and reduce the time model training takes. However, you want to avoid running out of GPU memory. If you encounter memory errors, reduce the batch size incrementally until the model trains smoothly.

Watch: How to Speed Up Ultralytics YOLO26 Inference by 2× with Batch Processing | Python & GPU | MLOps 🚀

With respect to YOLO26, you can set the batch parameter in the Train mode arguments to match your GPU capacity. On a single accelerator, batch=-1 profiles the model and picks a batch size targeting roughly 60% memory utilization; a fraction such as batch=0.70 targets a different share. Multi-GPU runs require an explicit global batch size of at least the device count. On CPU and Apple silicon, AutoBatch warns and falls back to batch=16.

Subset Training#

Subset training is a practical strategy that involves training your model on a smaller set of data that represents the larger dataset. It can save time and resources, especially during initial model development and testing. If you are running short on time or experimenting with different model configurations, subset training is a good option.

When it comes to YOLO26, you can implement subset training with the fraction parameter. Use a ratio such as fraction=0.1 for 10% of the training data, an integer such as fraction=300 for exactly 300 training images, or a [train, val, test] list such as fraction=[300, 100, 0] to limit each split and skip test images. Two-item lists such as [300, 100] leave the test split full. NDJSON datasets select evenly spaced records before downloading them, so repeated runs reuse the exact subset. This technique supports quick iterations and tuning before committing to the full dataset.

Multi-scale Training#

Multi-scale training is a technique that improves your model's ability to generalize by training it on images of varying sizes. Your model can learn to detect objects at different scales and distances and become more robust.

When you train YOLO26, you can enable scale augmentation by setting the scale parameter. This parameter adjusts the size of training images by a specified factor, simulating objects at different distances. For example, setting scale=0.5 randomly zooms training images by a factor between 0.5 and 1.5 during training. Configuring this parameter allows your model to experience a variety of image scales and improve its detection capabilities across different object sizes and scenarios.

Ultralytics also supports image-size multi-scale training via the multi_scale parameter. Unlike scale, which zooms images and then pads/crops back to imgsz, multi_scale changes imgsz itself each batch (rounded to the model stride). For example, with imgsz=640 and multi_scale=0.25, the training size is sampled from 480 up to 800 in stride steps (e.g., 480, 512, 544, ..., 800), while multi_scale=0.0 keeps a fixed size.

Caching#

Caching is an important technique to improve the efficiency of training machine learning models. By storing preprocessed images in memory, caching reduces the time the GPU spends waiting for data to be loaded from the disk. The model can continuously receive data without delays caused by disk I/O operations.

Caching can be controlled when training YOLO26 using the cache parameter:

• cache=True: Stores dataset images in RAM, providing the fastest access speed but at the cost of increased memory usage.
• cache='disk': Stores the images on disk, slower than RAM but faster than loading fresh data each time.
• cache=False: Disables caching, relying entirely on disk I/O, which is the slowest option.

Mixed Precision Training#

Mixed precision training uses both 16-bit (FP16) and 32-bit (FP32) floating-point types: FP16 for faster computation and FP32 to maintain precision where needed. Most of the neural network's operations are done in FP16 to benefit from faster computation and lower memory usage. However, a master copy of the model's weights is kept in FP32 to ensure accuracy during the weight update steps. You can handle larger models or larger batch sizes within the same hardware constraints.

To implement mixed precision training, you'll need to modify your training scripts and ensure your hardware (like GPUs) supports it. Many modern deep learning frameworks, such as PyTorch and TensorFlow, offer built-in support for mixed precision.

Mixed precision training is straightforward when working with YOLO26: AMP is already on by default (amp=True). On a CUDA GPU, YOLO runs a one-time capability check at the start of training and falls back to full FP32 if it fails; on CPU and Apple silicon AMP is skipped entirely. Set amp=False to force FP32, or amp="bf16" to train in bfloat16 on supported GPUs.

Pretrained Weights#

Using pretrained weights is an effective way to speed up your model's training process. Pretrained weights come from models already trained on large datasets, giving your model a head start. Transfer learning adapts pretrained models to new, related tasks. Fine-tuning a pretrained model involves starting with these weights and then continuing training on your specific dataset. This method of training results in faster training times and often better performance because the model starts with a solid understanding of basic features.

Pretrained weights come from the model argument, so model=yolo26n.pt already starts from COCO weights. The pretrained parameter controls whether those weights are kept (True, the default), discarded (False), or replaced by another checkpoint (pretrained=path/to/best.pt). Setting pretrained=True on a *.yaml model does not fetch weights on its own. See How to Fine-Tune YOLO on a Custom Dataset for the full transfer-learning workflow.

Other Techniques to Consider When Handling a Large Dataset#

There are a couple of other techniques to consider when handling a large dataset:

• Learning Rate Schedulers: Implementing learning rate schedulers dynamically adjusts the learning rate during training. A well-tuned learning rate can prevent the model from overshooting minima and improve stability. When training YOLO26, the lrf parameter helps manage learning rate scheduling by setting the final learning rate as a fraction of the initial rate. For behavior no argument exposes, such as per-layer learning rates or gradient clipping, subclass the trainer.
• Distributed Training: For handling large datasets, distributed training reduces training time by spreading the workload across multiple GPUs or machines. This approach is particularly valuable for enterprise-scale projects with substantial computational resources.
• Channels-last memory format: CUDA training automatically uses the faster NHWC memory layout on PyTorch 1.11 and newer, except on Windows, where it measured slower. Set channels_last=True to force it or channels_last=False to keep NCHW; PyTorch 1.10 and older, CPU, and MPS remain NCHW by default.

The Number of Epochs To Train For#

When training a model, an epoch refers to one complete pass through the entire training dataset. During an epoch, the model processes each example in the training set once and updates its parameters based on the learning algorithm. Multiple epochs are usually needed to allow the model to learn and refine its parameters over time.

A common question that comes up is how to determine the number of epochs to train the model for. A good starting point is 300 epochs. If the model overfits early, you can reduce the number of epochs. If overfitting does not occur after 300 epochs, you can extend the training to 600, 1200, or more epochs.

However, the ideal number of epochs can vary based on your dataset's size and project goals. Larger datasets might require more epochs for the model to learn effectively, while smaller datasets might need fewer epochs to avoid overfitting. With respect to YOLO26, you can set the epochs parameter in your training script.

Early Stopping#

Early stopping is a valuable technique for optimizing model training. By monitoring validation performance, you can halt training once the model stops improving. You can save computational resources and prevent overfitting.

The process involves setting a patience parameter that determines how many epochs to wait for an improvement in validation metrics before stopping training. If the model's performance does not improve within these epochs, training is stopped to avoid wasting time and resources.

For YOLO26, you can enable early stopping by setting the patience parameter in your training configuration. For example, patience=5 means training will stop if there's no improvement in validation metrics for 5 consecutive epochs. This keeps training efficient and avoids unnecessary computation.

Choosing Between Cloud and Local Training#

There are two options for training your model: cloud training and local training.

Cloud training offers scalability and powerful hardware and is ideal for handling large datasets and complex models. Platforms like Google Cloud, AWS, and Azure provide on-demand access to high-performance GPUs and TPUs, speeding up training times and enabling experiments with larger models. Ultralytics Platform also offers managed cloud training for YOLO models. However, cloud training can be expensive, especially for long periods, and data transfer can add to costs and latency.

Local training provides greater control and customization, letting you tailor your environment to specific needs and avoid ongoing cloud costs. It can be more economical for long-term projects, and since your data stays on-premises, it's more secure. However, local hardware may have resource limitations and require maintenance, which can lead to longer training times for large models.

Selecting an Optimizer#

An optimizer is an algorithm that adjusts the weights of your neural network to minimize the loss function, which measures how well the model is performing. In simpler terms, the optimizer helps the model learn by tweaking its parameters to reduce errors. Choosing the right optimizer directly affects how quickly and accurately the model learns.

You can also fine-tune optimizer parameters to improve model performance. Adjusting the learning rate sets the size of the steps when updating parameters. For stability, you might start with a moderate learning rate and gradually decrease it over time to improve long-term learning. Additionally, setting the momentum determines how much influence past updates have on current updates. A common value for momentum is around 0.9. It generally provides a good balance.

Common Optimizers#

Different optimizers have various strengths and weaknesses. Here are a few common optimizers.

• SGD (Stochastic Gradient Descent):

  • Updates model parameters using the gradient of the loss function with respect to the parameters.
  • Simple and efficient but can be slow to converge and might get stuck in local minima.

• Adam (Adaptive Moment Estimation):

  • Combines the benefits of both SGD with momentum and RMSProp.
  • Adjusts the learning rate for each parameter based on estimates of the first and second moments of the gradients.
  • Well-suited for noisy data and sparse gradients.
  • Efficient and generally requires less tuning. For shorter training runs, YOLO26's optimizer=auto selects the closely related AdamW rather than Adam itself.

• RMSProp (Root Mean Square Propagation):

  • Adjusts the learning rate for each parameter by dividing the gradient by a running average of the magnitudes of recent gradients.
  • Helps in handling the vanishing gradient problem and is effective for recurrent neural networks.

• MuSGD (Muon + SGD hybrid):

  • Combines SGD-style updates with Muon-inspired behavior for improved stability in large-scale training.
  • A good choice when you want SGD-like generalization but need smoother convergence than vanilla SGD.
  • Especially relevant for YOLO26 training recipes; if unsure, start with optimizer=auto and compare against MuSGD on your dataset.

For YOLO26, the optimizer parameter lets you choose from various optimizers, including SGD, MuSGD, Adam, Adamax, AdamW, NAdam, RAdam, and RMSProp, or you can set it to auto for automatic selection based on the number of training iterations (AdamW for shorter runs, MuSGD above 10,000 iterations).

yolo train model=yolo26n.pt data=coco8.yaml optimizer=MuSGD

Connecting with the Community#

Being part of a community of computer vision enthusiasts can help you solve problems and learn faster. Here are some ways to connect, get help, and share ideas.

Community Resources#

• GitHub Issues: Visit the YOLO26 GitHub repository and use the Issues tab to ask questions, report bugs, and suggest new features. The community and maintainers are very active and ready to help.
• Ultralytics Discord Server: Join the Ultralytics Discord server to chat with other users and developers, get support, and share your experiences.

Using these resources will help you solve challenges and stay up to date with the latest trends and practices in the computer vision community.

Key Takeaways#

Training computer vision models involves following good practices, optimizing your strategies, and solving problems as they arise. Techniques like adjusting batch sizes, mixed precision training, and starting with pretrained weights can make your models work better and train faster. Methods like subset training and early stopping help you save time and resources. Staying connected with the community and keeping up with new trends will help you keep improving your model training skills.

FAQ#