开始使用

Segment Anything 模型 (SAM)

SAM 的演进

这是 Meta 推出的原始 SAM 模型。如需更强大的功能,请参阅 SAM 2 进行视频分割,或参阅 SAM 3,了解如何通过文本和图像示例提示进行可提示概念分割。

How to use Segment Anything In Colab

Meta 推出的分割一切模型(SAM)引入了可提示的图像分割:给定一个点、框或掩码提示,它会返回指示对象的掩码,并采用足够轻量、可用于交互式应用的快速解码器。

SAM 简介:Segment Anything 模型#

分割一切模型(简称 SAM)是一种图像分割模型,旨在通过提示对各种图像分析任务进行分割。SAM 是 Meta「分割一切」项目的核心,该项目为图像分割引入了一个新模型、新任务和新数据集。

SAM 的设计使其无需先验知识即可适应新的图像分布和任务,这种能力称为零样本迁移。SAM 在 SA-1B 数据集上训练,该数据集包含 1100 万张经过整理的图像和超过 10 亿个掩码。SAM 展现出出色的零样本性能,通常可与以往的全监督结果相媲美,甚至表现更好。

带有自动分割掩码的 SA-1B 数据集示例 SA-1B 示例图像。这些图像叠加了来自 SA-1B 数据集的掩码。SA-1B 包含 1100 万张多样化、高分辨率、获得授权且保护隐私的图像,以及 11 亿个高质量分割掩码。这些掩码完全由 SAM 自动标注;经人工评分和大量实验验证,它们具有较高的质量和多样性。图像按每张图像的掩码数量分组以便展示(平均每张图像约有 ∼100 个掩码)。

Segment Anything 模型 (SAM) 的主要特性#

  • 可提示分割任务:SAM 专为可提示分割任务而设计,可以根据任意提示生成有效的分割掩码,例如用于识别对象的空间线索或文本线索。
  • 先进架构:分割一切模型采用大型图像编码器、提示编码器和轻量级掩码解码器。该架构支持灵活提示、实时掩码计算,并能在分割任务中识别歧义。
  • SA-1B 数据集:SA-1B 数据集由「分割一切」项目推出,包含 1100 万张图像上的超过 10 亿个掩码。作为发布时规模最大​​的分割数据集,它为 SAM 提供了多样且大规模的训练数据来源。
  • 零样本性能:SAM 在各种分割任务中都展现出出色的零样本性能,是一款可直接用于多种应用的工具,几乎无需进行提示工程。

如需深入了解 Segment Anything 模型和 SA-1B 数据集,请访问 Segment Anything GitHub,并阅读研究论文 Segment Anything。

Ultralytics Platform 上的 SAM

SAM 系列模型(SAM 2.1、SAM 3 和 SAM 3.1)为 Ultralytics Platform 上的智能标注功能提供支持,可通过点击生成掩码,从而快速标注数据集。

可用模型、支持的任务和运行模式#

此表列出了可用模型、各自的预训练权重、支持的任务,以及与训练、验证、推理和导出等不同运行模式的兼容性。支持的模式以 ✅ 表示,不支持的模式以 ❌ 表示。

模型类型预训练权重支持的任务训练验证推理导出
SAM 基础版sam_b.pt实例分割❌❌✅❌
SAM 大型版sam_l.pt实例分割❌❌✅❌

如何使用 SAM:图像分割的灵活性与强大功能#

分割一切模型可用于许多超出其训练数据范围的下游任务,包括边缘检测、对象提议生成、实例分割和初步的文本转掩码预测。借助提示工程,SAM 可以零样本方式适应新的任务和数据分布。

SAM 预测示例#

使用提示进行分割

使用给定的提示分割图像。

from ultralytics import SAM

# 加载模型
model = SAM("sam_b.pt")

# 显示模型信息(可选)
model.info()

# 使用边界框提示运行推理
results = model("ultralytics/assets/zidane.jpg", bboxes=[439, 437, 524, 709])

# 使用单个点运行推理
results = model(points=[900, 370], labels=[1])

# 使用多个点运行推理
results = model(points=[[400, 370], [900, 370]], labels=[1, 1])

# 为每个对象使用多个点提示运行推理
results = model(points=[[[400, 370], [900, 370]]], labels=[[1, 1]])

# 使用负点提示运行推理
results = model(points=[[[400, 370], [900, 370]]], labels=[[1, 0]])
分割全部内容

分割整张图像。

from ultralytics import SAM

# 加载模型
model = SAM("sam_b.pt")

# 显示模型信息(可选)
model.info()

# 运行推理
model("path/to/image.jpg")
  • 如果你没有传入任何提示(边界框/点/掩码),SAM 会分割整张图像。
SAMPredictor 示例

设置一次图像后,就可以多次运行提示推理,而无需重新运行图像编码器。

import cv2

from ultralytics.models.sam import Predictor as SAMPredictor

# 创建 SAMPredictor
overrides = {"conf": 0.25, "task": "segment", "mode": "predict", "imgsz": 1024, "model": "mobile_sam.pt"}
predictor = SAMPredictor(overrides=overrides)

# 设置图像
predictor.set_image("ultralytics/assets/zidane.jpg")  # 使用图像文件设置
predictor.set_image(cv2.imread("ultralytics/assets/zidane.jpg"))  # 使用 np.ndarray 设置
results = predictor(bboxes=[439, 437, 524, 709])

# 使用单点提示运行推理
results = predictor(points=[900, 370], labels=[1])

# 使用多个点提示运行推理
results = predictor(points=[[400, 370], [900, 370]], labels=[1, 1])

# 使用负点提示运行推理
results = predictor(points=[[[400, 370], [900, 370]]], labels=[[1, 0]])

# 重置图像
predictor.reset_image()

使用其他参数分割全部内容。

from ultralytics.models.sam import Predictor as SAMPredictor

# 创建 SAMPredictor
overrides = {"conf": 0.25, "task": "segment", "mode": "predict", "imgsz": 1024, "model": "mobile_sam.pt"}
predictor = SAMPredictor(overrides=overrides)

# 使用其他参数进行分割
results = predictor(source="ultralytics/assets/zidane.jpg", crop_n_layers=1, points_stride=64, min_mask_region_area=100)
注意

上述示例中返回的所有 results 都是 Results 对象,可用于访问预测掩码和源图像。

SAM 与 YOLO 对比#

下面我们将 Meta 的 SAM-b 模型与 Ultralytics 分割模型进行比较,其中包括 YOLO26:

模型大小
(MB)
参数
(M)
速度(CPU)
(毫秒/图像)
Meta SAM-b37593.741703
MobileSAM40.710.123802
FastSAM-s 搭配 YOLOv8 主干网络23.911.858.0
Ultralytics YOLOv8n-seg7.1(小 52.8 倍)3.4(少 27.6 倍)24.8(快 1682 倍)
Ultralytics YOLO11n-seg6.2(小 60.5 倍)2.9(少 32.3 倍)24.3(快 1716 倍)
Ultralytics YOLO26n-seg6.7(小 56.0 倍)2.7(少 34.7 倍)25.2(快 1655 倍)

这项比较展示了 SAM 各变体与 YOLO 分割模型在模型大小和速度方面的显著差异。SAM 具备独特的自动分割能力,而 YOLO 模型,尤其是 YOLOv8n-seg、YOLO11n-seg 和 YOLO26n-seg,则更小、更快,计算效率也更高。

SAM 速度使用 PyTorch 测量,YOLO 速度使用 ONNX Runtime 测量。测试在配备 16GB RAM 的 2025 款 Apple M4 Air 上进行,使用了 torch==2.10.0、ultralytics==8.4.31 和 onnxruntime==1.24.4。复现此测试的方法:

示例
from ultralytics import ASSETS, SAM, YOLO, FastSAM

# 分析 SAM-b、MobileSAM 的性能
for file in ["sam_b.pt", "mobile_sam.pt"]:
    model = SAM(file)
    model.info()
    model(ASSETS)

# 分析 FastSAM-s 的性能
model = FastSAM("FastSAM-s.pt")
model.info()
model(ASSETS)

# 分析 YOLO 模型的性能(ONNX)
for file_name in ["yolov8n-seg.pt", "yolo11n-seg.pt", "yolo26n-seg.pt"]:
    model = YOLO(file_name)
    model.info()
    onnx_path = model.export(format="onnx", dynamic=True, nms=False)  # YOLO26 的 NMS-free 头;对 YOLOv8/YOLO11 无操作
    model = YOLO(onnx_path)
    model(ASSETS)

自动标注:快速构建分割数据集#

自动标注是 SAM 的一项关键功能,用户可以使用预训练检测模型生成分割数据集。这项功能可快速标注大量图像,省去耗时的手动标注工作。

使用检测模型生成分割数据集#

要使用 Ultralytics 框架自动标注数据集,请使用 auto_annotate 函数,将 YOLO26x 等检测器与 SAM 配合使用:

示例
from ultralytics.data.annotator import auto_annotate

auto_annotate(data="path/to/images", det_model="yolo26x.pt", sam_model="sam_b.pt")
参数类型默认值说明
datastr必需包含待标注或分割目标图像的目录路径。
det_modelstr'yolo26x.pt'用于初始对象检测的 YOLO 检测模型路径。
sam_modelstr'sam_b.pt'用于分割的 SAM 模型路径(支持 SAM、SAM 2、MobileSAM 和 SAM 3 权重)。
devicestr''计算设备(例如 'cuda:0'、'cpu',或 '' 表示自动检测设备)。
conffloat0.25用于过滤低置信度检测结果的 YOLO 检测置信度阈值。
ioufloat0.5用于通过非极大值抑制过滤重叠框的 IoU 阈值。
imgszint640图像缩放输入尺寸(如有需要,会向上取整为 32 的倍数)。
max_detint300为提高内存效率,每张图像的最大检测数。
classeslist[int]None要检测的类别索引列表(例如,[0, 1] 表示人和自行车)。
output_dirstrNone标注结果的保存目录(默认:同级目录中的 <data>_auto_annotate_labels)。

auto_annotate 函数接收图像路径,并可选地指定预训练检测模型和 SAM 分割模型、模型运行设备,以及用于保存标注结果的输出目录。

使用预训练模型进行自动标注,可以大幅减少创建分割数据集所需的时间和精力,让拥有大量图像的团队专注于模型开发和评估,而非手动标注。

引用与致谢#

如果你在研究或开发工作中觉得 SAM 很有用,请考虑引用这篇论文:

引用
@misc{kirillov2023segment,
      title={Segment Anything},
      author={Alexander Kirillov and Eric Mintun and Nikhila Ravi and Hanzi Mao and Chloe Rolland and Laura Gustafson and Tete Xiao and Spencer Whitehead and Alexander C. Berg and Wan-Yen Lo and Piotr Dollár and Ross Girshick},
      year={2023},
      eprint={2304.02643},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

我们谨向 Meta AI 致谢,感谢他们为计算机视觉社区创建并维护这一宝贵资源。

常见问题#

  • Meta 的分割一切模型(SAM)是一种专为可提示分割任务设计的图像分割模型。它结合图像编码器、提示编码器和轻量级掩码解码器,根据点、框或掩码等提示生成分割掩码。SAM 在 SA-1B 数据集上训练,具有出色的零样本性能,无需先验知识即可适应新的图像分布和任务。

  • 你可以使用 Segment Anything 模型 (SAM) 进行图像分割,只需使用边界框或点等不同提示运行推理即可。下面是一个使用 Python 的示例:

    from ultralytics import SAM
    
    # 加载模型
    model = SAM("sam_b.pt")
    
    # 使用边界框提示进行分割
    model("ultralytics/assets/zidane.jpg", bboxes=[439, 437, 524, 709])
    
    # 使用点提示进行分割
    model("ultralytics/assets/zidane.jpg", points=[900, 370], labels=[1])
    
    # 使用多个点提示进行分割
    model("ultralytics/assets/zidane.jpg", points=[[400, 370], [900, 370]], labels=[1, 1])
    
    # 为每个对象使用多个点提示进行分割
    model("ultralytics/assets/zidane.jpg", points=[[[400, 370], [900, 370]]], labels=[[1, 1]])
    
    # 使用负点提示进行分割。
    model("ultralytics/assets/zidane.jpg", points=[[[400, 370], [900, 370]]], labels=[[1, 0]])

    或者,你也可以在命令行界面 (CLI) 中使用 SAM 运行推理:

    yolo predict model=sam_b.pt source=path/to/image.jpg
  • 与 YOLO 模型相比,SAM-b、MobileSAM 和 FastSAM-s 等 SAM 变体通常体积更大、速度更慢,但具备独特的零样本分割能力。例如,在 CPU 上, YOLO26n-seg 比 Meta 的原始 SAM-b 模型小 56 倍,速度快 1650 倍以上。因此,YOLO 模型适用于需要快速、轻量且计算效率高的分割应用,而 SAM 模型则擅长灵活、可提示的零样本分割任务。

  • Ultralytics 的 SAM 提供自动标注功能,可使用预训练检测模型生成分割数据集。下面是一个 Python 示例:

    from ultralytics.data.annotator import auto_annotate
    
    auto_annotate(data="path/to/images", det_model="yolo26x.pt", sam_model="sam_b.pt")

    此函数接收图像路径,并可选地指定预训练检测模型和 SAM 分割模型,以及设备和输出目录。如需完整指南,请参阅自动标注。

  • SAM 基于 SA-1B 数据集训练,该数据集包含 1100 万张图像中的 10 亿多个掩码。SA-1B 在发布时是规模最大 的分割数据集,提供了高质量且多样化的训练数据,为 SAM 在各种分割任务中的零样本性能奠定了基础。

评论27

提出问题、分享结果,或帮助其他人。

  1. kh-diabGitHub

    can we use SAM to process labeled images with bounding box to segment the object inside the box and generate new labels files like yolo or coco format if yes how please

    1
    1. Paula DerrengerUltralytics

      Yes, you can use SAM to process labeled images with bounding boxes to segment the objects inside and generate new label files in formats like YOLO or COCO. You can achieve this by using the auto_annotate function in the Ultralytics framework. This function allows you to auto-annotate your dataset using a pre-trained detection model and SAM for segmentation. For more details, you can refer to the Segment Anything Model documentation. If you need further assistance, feel free to ask!

    2. kh-diabGitHub

      Hello @pderrenger

      Thanks I tried to run the below but I face error all the time

      import os import cv2 import numpy as np from ultralytics import SAM

      Load the SAM model

      model = SAM("sam_b.pt") # Update the model path if needed

      Function to load YOLO annotations

      def load_yolo_annotations(annotation_path, image_width, image_height): bboxes = [] try: with open(annotation_path, 'r') as file: lines = file.readlines() for line in lines: parts = line.strip().split() class_id = int(parts[0]) x_center = float(parts[1]) * image_width y_center = float(parts[2]) * image_height width = float(parts[3]) * image_width height = float(parts[4]) * image_height x = x_center - width / 2 y = y_center - height / 2 bboxes.append((x, y, x + width, y + height)) # Update to [x1, y1, x2, y2] except Exception as e: print(f"Error reading annotation file {annotation_path}: {e}") return bboxes

      Paths

      images_folder = r'E:\MSc\tesi\20210901_Herrenpuent_Taenikon_Matrice_300_15m_1.6m_s\500 D' annotations_folder = r'C:\Users\Khaled\OneDrive - OST\Study Material\Msc Thesis\dataset\Group D yolo format\obj_train_data' output_folder = r'C:\Users\Khaled\OneDrive - OST\Study Material\Msc Thesis\dataset\test output sam'

      if not os.path.exists(output_folder): os.makedirs(output_folder)

      Process each image and its corresponding annotation

      total_images = len(os.listdir(images_folder)) processed_images = 0

      for image_filename in os.listdir(images_folder): if image_filename.endswith(('.JPG', '.jpg', '.png')): image_path = os.path.join(images_folder, image_filename) annotation_path = os.path.join(annotations_folder, image_filename.replace('.JPG', '.txt').replace('.jpg', '.txt').replace('.png', '.txt'))

          print(f"Processing {image_filename}...")
      
          # Check if annotation file exists
          if not os.path.exists(annotation_path):
              print(f"Annotation file not found for {image_filename}")
              continue
      
          # Load the image
          image = cv2.imread(image_path)
          if image is None:
              print(f"Failed to load image {image_path}")
              continue
      
          image_rgb = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
          height, width, _ = image.shape
      
          # Load YOLO annotations
          bboxes = load_yolo_annotations(annotation_path, width, height)
          if not bboxes:
              print(f"No bounding boxes found in {annotation_path}")
              continue
      
          # Perform segmentation using SAM model
          results = model(image_rgb, bboxes=bboxes)
      
          for idx, result in enumerate(results):
              if hasattr(result, 'masks') and result.masks is not None:
                  masks = result.masks.cpu().numpy()  # Convert to numpy array
                  for m_idx, mask in enumerate(masks):
                      chw_mask = mask.astype(np.uint8)  # Ensure mask is in uint8 format
                      hwc_mask = np.transpose(chw_mask, (1, 2, 0))  # Convert CHW to HWC
                      contours, _ = cv2.findContours(hwc_mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
                      
                      segmented_image = cv2.bitwise_and(image, image, mask=hwc_mask)
      
                      # Save the segmented image
                      output_path = os.path.join(output_folder, image_filename.replace('.JPG', f'_segmented_{idx}_{m_idx}.jpg').replace('.jpg', f'_segmented_{idx}_{m_idx}.jpg').replace('.png', f'_segmented_{idx}_{m_idx}.png'))
                      cv2.imwrite(output_path, segmented_image)
                      print(f"Saved segmented image to {output_path}")
      
          processed_images += 1
          print(f"Processed {processed_images}/{total_images} images.")

      print("Processing completed.")

      error is

      AttributeError: 'Masks' object has no attribute 'astype'. See valid attributes below.

      A class for storing and manipulating detection masks.
      
          xy (list): A list of segments in pixel coordinates.
          xyn (list): A list of normalized segments.
      
      Methods:
          cpu(): Returns the masks tensor on CPU memory.
          numpy(): Returns the masks tensor as a numpy array.
          cuda(): Returns the masks tensor on GPU memory.
          to(device, dtype): Returns the masks tensor with the specified device and dtype.

      PS C:\Users\Khaled> & C:/MSc_Workspace/anaconda3/envs/GenEnv/python.exe "c:/Users/Khaled/OneDrive - OST/Study Material/Msc Thesis/coding/sam segment.py" Processing DJI_20210901122818_0695.JPG...

      0: 1024x1024 67330.4ms Speed: 48.0ms preprocess, 67330.4ms inference, 8095.5ms postprocess per image at shape (1, 3, 1024, 1024) Traceback (most recent call last): File "c:\Users\Khaled\OneDrive - OST\Study Material\Msc Thesis\coding\sam segment.py", line 75, in chw_mask = mask.astype(np.uint8) # Ensure mask is in uint8 format File "C:\MSc_Workspace\anaconda3\envs\GenEnv\lib\site-packages\ultralytics\utils_init_.py", line 162, in getattr raise AttributeError(f"'{name}' object has no attribute '{attr}'. See valid attributes below.\n{self.doc}") AttributeError: 'Masks' object has no attribute 'astype'. See valid attributes below.

      A class for storing and manipulating detection masks.
      
      Attributes:
          xy (list): A list of segments in pixel coordinates.
          xyn (list): A list of normalized segments.
      
      Methods:
          cpu(): Returns the masks tensor on CPU memory.
          numpy(): Returns the masks tensor as a numpy array.
          cuda(): Returns the masks tensor on GPU memory.
          to(device, dtype): Returns the masks tensor with the specified device and dtype.

      do you have idea how to solve it thanks

  2. evans rojas

    Hola me gustaría usar SAM para clasificar vegetación a partir de una imagen. ¿Qué recomendaciones me sugieren?

    1. Paula DerrengerUltralytics

      Hola, Evans. SAM puede ayudarte a delimitar la vegetación, pero el SAM original no identifica por sí solo especies ni tipos de cobertura. Te sugiero comenzar con puntos positivos sobre la vegetación y negativos sobre el fondo, revisar las máscaras y asignarles tus clases manualmente. Si buscas automatizar la clasificación en más imágenes, puedes usar esas anotaciones para entrenar un modelo de segmentación por instancias para plantas individuales o de segmentación semántica para coberturas del terreno. ¿Quieres separar vegetación del fondo o distinguir especies, y trabajas con fotografías terrestres, de dron o satelitales?

  3. meenakshi71GitHub

    I've been trying to export the SAM model (which i had trained on my custom dataset) into ONNX but the output is always wrong. It is just generating a grid instead of a proper segmentation mask every time even after providing multiple kinds of prompts.

    1. Glenn JocherUltralytics

      @meenakshi71 it seems the issue may stem from the export process or configuration. Ensure you are using the correct export parameters and supported formats. SAM models are primarily designed for real-time inference and not all functionalities may transfer seamlessly to ONNX. Refer to the SAM export documentation for guidance. If the problem persists, consider testing the ONNX model with simpler prompts to isolate the issue.

  4. MeenakshiKK236GitHub

    Why are my segmentation results being automatically saved into this directory while using the SAM model to perform segmentation on my custom dataset? "Results saved to runs/segment/predict4"

    1. Glenn JocherUltralytics

      @MeenakshiKK236 the results are saved to the runs/segment/predict4 directory by default. You can change the save directory by specifying the project and name arguments in your command. For more details, please refer to the SAM documentation.

  5. NasserAlqeifiGitHub

    Correct me if I'm wrong, when I use the auto_annotate function I will only get labels for my dataset with the classes in Yolo pre-trained? What if I need classes that are not available within Yolo? For example, I want to get a label for the road, and this is not available in any version of Yolo, but it is available in SAM?

    1. Glenn JocherUltralytics

      @NasserAlqeifi but also remember SAM identifies segments but not classes, so it may find a sign or a road, but it won't be able to assign a class to them by itself.

    2. NasserAlqeifiGitHub

      Thank you for your quick response. Indeed, I tried to use SAM directly and found that it does not save the class names in the name, but the class number.

      from ultralytics import SAM

      Load a model

      model = SAM("sam2_l.pt")

      Predict results

      results = model.predict(source=r"/testSAM/image.png", stream=True, bboxes=None, points=None, labels=None, show=False)

      Iterate over results, print names, and save each result to a file

      for i, result in enumerate(results): # Print the names detected in the result print(f"Result {i}: Detected Names - {result.names}")

      # Save the result as an image file
      result.save(f"output_{i}.jpg")

      ------------- output ---------

      image 1/1 /testSAM/image.png: 1024x1024 1 0, 1 1, 1 2, 1 3, 1 4, 1 5, 1 6, 1 7, 1 8, 1 9, 1 10, 1 11, 1 12, 1 13, 1 14, 1 15, 1 16, 1 17, 1 18, 1 19, 1 20, 1 21, 1 22, 1 23, 1 24, 1 25, 1 26, 1 27, 1 28, 1 29, 1 30, 1 31, 1 32, 23758.2ms Result 0: Detected Names - {0: '0', 1: '1', 2: '2', 3: '3', 4: '4', 5: '5', 6: '6', 7: '7', 8: '8', 9: '9', 10: '10', 11: '11', 12: '12', 13: '13', 14: '14', 15: '15', 16: '16', 17: '17', 18: '18', 19: '19', 20: '20', 21: '21', 22: '22', 23: '23', 24: '24', 25: '25', 26: '26', 27: '27', 28: '28', 29: '29', 30: '30', 31: '31', 32: '32'} Speed: 20.3ms preprocess, 23758.2ms inference, 7.1ms postprocess per image at shape (1, 3, 1024, 1024)

    3. Glenn JocherUltralytics

      Thank you for your feedback. Currently, SAM does not assign class names, only class numbers. If you need specific class names, you may need to implement a custom mapping or post-processing step to assign the desired class names based on your requirements.

  6. dreambitGitHub

    I have images where a single object (such as a car) is partially covered by small trees, resulting in the object being divided into multiple visible fragments. In coco it could look like { "id": 1, "image_id": 1, "category_id": 1, "bbox": [x, y, width, height], "segmentation": [ [polygon1_x1, polygon1_y1, polygon1_x2, polygon1_y2, ...], [polygon2_x1, polygon2_y1, polygon2_x2, polygon2_y2, ...] ], "area": 500, "iscrowd": 0 } Each polygon is an array of coordinates defining a closed shape

    Is it possible with auto_annotate and yolo format?

    1. Glenn JocherUltralytics(已编辑)

      @dreambit hi there!

      Thank you for your detailed question. Yes, you can use the auto_annotate function in the Ultralytics framework to handle images where objects are partially occluded and appear as multiple fragments. This function can generate segmentation masks for such complex scenarios.

      Here's how you can achieve this:

      1. Prepare Your Dataset: Ensure your images are organized in a directory.
      2. Use the auto_annotate Function: This function will use a pre-trained YOLO detection model and the SAM segmentation model to annotate your images.

      Here's a code example to get you started:

      from ultralytics.data.annotator import auto_annotate
      
      # Path to your images
      data_path = "path/to/your/images"
      
      # Auto-annotate using YOLO26 and SAM models
      auto_annotate(data=data_path, det_model="yolo26x.pt", sam_model="sam_b.pt")
      Explanation:
      • data: Path to the folder containing your images.
      • det_model: The pre-trained YOLO detection model, which defaults to yolo26x.pt.
      • sam_model: The pre-trained SAM segmentation model, which defaults to sam_b.pt.

      This setup will generate segmentation masks for your images, even if the objects are partially occluded.

      For more detailed information on using SAM and its capabilities, you can refer to the Segment Anything Model (SAM) documentation.

      If you encounter any issues or need further assistance, please provide a reproducible example so we can better understand and address your specific scenario. You can find more information on creating a minimum reproducible example here.

      Feel free to update to the latest versions of the packages to ensure you have the latest features and fixes.

      Best of luck with your project! 🚀

    2. dreambitGitHub

      0 0.681 0.485 0.670 0.487 0.676 0.487 1 0.504 0.000 0.501 0.004 0.498 0.004 0.493 0.010 0.492 0.0104

    3. Glenn JocherUltralytics(已编辑)

      Hi @dreambit,

      Thank you for sharing the images and details. I understand the challenge you're facing with partial segmentations. The auto_annotate function should ideally handle such cases, but it seems like the segmentation results are not as expected.

      Steps to Troubleshoot and Improve Segmentation:
      1. Verify Latest Versions: Ensure you are using the latest versions of the Ultralytics and SAM models. This can help avoid any bugs that might have been fixed in recent updates.

      2. Reproducible Example: If the issue persists, providing a minimum reproducible example would be very helpful. This allows us to better understand and diagnose the problem. You can find guidance on creating a reproducible example here.

      3. Adjusting Prompts: Sometimes, adjusting the prompts (bounding boxes or points) can improve segmentation results. Here’s an example of how you can use different prompts:

        from ultralytics import SAM
        
        # Load a model
        model = SAM("sam_b.pt")
        
        # Run inference with bounding boxes prompt
        results = model("path/to/image.jpg", bboxes=[[x1, y1, x2, y2], [x3, y3, x4, y4]])
        
        # Run inference with points prompt
        results = model("path/to/image.jpg", points=[[x1, y1], [x2, y2]], labels=[1, 1])
      4. Segment Everything: If specific prompts are not yielding the desired results, you can try segmenting the entire image without prompts:

        from ultralytics import SAM
        
        # Load a model
        model = SAM("sam_b.pt")
        
        # Segment the whole image
        results = model("path/to/image.jpg")
      5. Multiple Masks Handling: If your object is divided into multiple fragments, ensure that the segmentation model is capable of handling multiple masks per object. The YOLO format typically supports one segmentation mask per object, but you can adapt the output to handle multiple masks by combining them post-processing.

      Example of Handling Multiple Masks:

      If you need to combine multiple masks into a single annotation, you can do this post-processing:

      # `results[0].masks.data` holds all masks of the image, shape (N, H, W)
      combined_mask = results[0].masks.data.cpu().numpy().max(axis=0)
      
      # Save or use the combined mask
      Documentation and Resources:

      For more detailed information on using SAM and its capabilities, please refer to the Segment Anything Model (SAM) documentation.

      If you continue to face issues, please share a reproducible example, and we’ll be happy to assist further. Your feedback is valuable, and we aim to provide the best support possible.

    1. Paula DerrengerUltralytics(已编辑)

      @kshweikeh hi there!

      Currently, the Segment Anything Model (SAM) from Ultralytics is designed primarily for inference and does not support training on custom datasets. However, you can leverage SAM's powerful zero-shot capabilities for various segmentation tasks without the need for additional training.

      If you're looking to create a segmentation dataset using SAM, you can use the auto-annotation feature. This allows you to generate segmentation masks using a pre-trained detection model. Here's a quick example:

      from ultralytics.data.annotator import auto_annotate
      
      auto_annotate(data="path/to/images", det_model="yolo26x.pt", sam_model="sam_b.pt")

      This function will help you annotate your images quickly and efficiently, bypassing the need for manual labeling.

      For more detailed information on using SAM, you can visit our Segment Anything Model (SAM) documentation.

      If you have any further questions or need additional assistance, feel free to ask! 😊

    2. Blessing988GitHub

      @pderrenger It looks like auto_annotate for SAM only gives out the segmentation masks. What if you want to generate the bounding boxes in additionto the masks to verify if the results are right. Thanks.

    3. Blessing988GitHub

      Also, how will one visualise the segmentation masks after the auto_annotate which have been saved in a directory of txt files.

    4. Paula DerrengerUltralytics(已编辑)

      Hi @Blessing988,

      auto_annotate writes one YOLO segmentation polygon per line (class_id x1 y1 x2 y2 ..., normalized) to its output folder (output_dir, by default <images folder>_auto_annotate_labels). Here is a script to draw those polygons on the images:

      from pathlib import Path
      
      import cv2
      import numpy as np
      
      
      def visualize_annotations(image_dir, label_dir, output_dir):
          Path(output_dir).mkdir(parents=True, exist_ok=True)
      
          for image_path in Path(image_dir).glob("*.jpg"):
              image = cv2.imread(str(image_path))
              h, w = image.shape[:2]
              label_path = Path(label_dir) / f"{image_path.stem}.txt"
              if not label_path.exists():
                  continue
      
              for line in label_path.read_text().splitlines():
                  values = line.split()
                  polygon = (np.array(values[1:], dtype=float).reshape(-1, 2) * [w, h]).astype(np.int32)
                  cv2.polylines(image, [polygon], isClosed=True, color=(0, 255, 0), thickness=2)
      
              cv2.imwrite(str(Path(output_dir) / image_path.name), image)
      
      
      visualize_annotations("path/to/images", "path/to/labels", "path/to/output")

      For more detailed information on using SAM and its features, you can refer to the Segment Anything Model (SAM) documentation.

      Feel free to reach out if you have any further questions or need additional assistance! 😊

    1. Paula DerrengerUltralytics

      @pabentse absolutely, the addition of an "H" size model would be a great enhancement to the range of model sizes we offer, catering to different computational needs and use cases. The YOLO community's feedback is invaluable, and suggestions like yours help guide future developments. Keep an eye on our updates! 👀🚀