开始使用

快速 Segment Anything 模型(FastSAM)

快速分割一切模型(FastSAM)是一种基于卷积神经网络(CNN)的实时解决方案,用于执行分割一切任务。该任务旨在根据各种可能的用户交互提示,对图像中的任意对象进行分割。FastSAM 在保持竞争力的同时大幅降低计算需求,是多种视觉任务的实用选择。



观看: 使用 Ultralytics 通过 FastSAM 进行目标跟踪

模型架构#

快速 Segment Anything 模型(FastSAM)架构概览

概述#

FastSAM 旨在克服分割一切模型(SAM)的局限。SAM 是一种计算资源需求很高的大型 Transformer 模型。FastSAM 将分割一切任务拆分为两个连续阶段:全图实例分割和提示引导的选择。第一阶段使用 YOLOv8-seg 生成图像中所有实例的分割掩码。第二阶段则输出与提示相对应的感兴趣区域。

主要功能#

  1. 实时解决方案:FastSAM 利用 CNN 的计算效率,为分割一切任务提供实时解决方案,因此对需要快速获得结果的工业应用很有价值。

  2. 效率与性能:FastSAM 在实现与 SAM 相当性能的同时,大幅降低了计算和资源需求,支持实时应用。

  3. 提示引导分割:FastSAM 可以根据各种用户交互提示对图像中的任意对象进行分割,在不同场景中灵活适用。

  4. 基于 YOLOv8-seg:FastSAM 基于 YOLOv8-seg 构建。YOLOv8-seg 是一种带有实例分割分支的对象检测器,因此可以有效生成图像中所有实例的分割掩码。

  5. 基准测试结果具有竞争力:在 MS COCO 对象提议任务中,FastSAM 在单块 NVIDIA RTX 3090 上以显著快于 SAM 的速度取得了高分,展现出出色的效率和能力。

  6. 实际应用:这种方法为许多视觉任务提供了实用解决方案,运行速度比早期的分割一切方法快数十倍至数百倍。

  7. 模型压缩的可行性:FastSAM 展示了一条可行路径:通过为结构引入人工先验,大幅减少计算量,从而为通用视觉任务的大型模型架构开辟新的可能性。

可用模型、支持的任务和运行模式#

此表列出了可用模型、各自的预训练权重、支持的任务,以及与训练、验证、推理和导出等不同运行模式的兼容性。支持的模式以 ✅ 表示,不支持的模式以 ❌ 表示。

模型类型预训练权重支持的任务训练验证推理导出
FastSAM-sFastSAM-s.pt实例分割❌✅✅✅
FastSAM-xFastSAM-x.pt实例分割❌✅✅✅

FastSAM 与 YOLO 的对比#

下面我们将 FastSAM-s 与 Meta 的 SAM 各版本及 Ultralytics 分割模型进行比较,其中包括 YOLO26:

模型大小
(MB)
参数
(M)
速度(CPU)
(毫秒/图像)
Meta SAM-b37593.741703
Meta SAM2-b16280.828867
Meta SAM2-t78.138.923430
MobileSAM40.710.123802
使用 YOLOv8 主干网络的 FastSAM-s23.911.858.0
Ultralytics YOLOv8n-seg7.1(小 11.0 倍)3.4(少 11.4 倍)24.8(快 945 倍)
Ultralytics YOLO11n-seg6.2(小 12.6 倍)2.9(少 13.4 倍)24.3(快 964 倍)
Ultralytics YOLO26n-seg6.7(小 11.7 倍)2.7(少 14.4 倍)25.2(快 930 倍)

这项比较展示了 SAM 各变体与 YOLO 分割模型在模型大小和速度方面的显著差异。SAM 具备独特的自动分割能力,而 YOLO 模型,尤其是 YOLOv8n-seg、YOLO11n-seg 和 YOLO26n-seg,则更小、更快,计算效率也更高。

SAM 速度使用 PyTorch 测量,YOLO 速度使用 ONNX Runtime 测量。测试在配备 16GB RAM 的 2025 款 Apple M4 Air 上进行,使用了 torch==2.10.0、ultralytics==8.4.31 和 onnxruntime==1.24.4。复现此测试的方法:

示例
from ultralytics import ASSETS, SAM, YOLO, FastSAM

# 分析 SAM2-t、SAM2-b、SAM-b、MobileSAM 的性能
for file in ["sam_b.pt", "sam2_b.pt", "sam2_t.pt", "mobile_sam.pt"]:
    model = SAM(file)
    model.info()
    model(ASSETS)

# 分析 FastSAM-s 的性能
model = FastSAM("FastSAM-s.pt")
model.info()
model(ASSETS)

# 分析 YOLO 模型的性能(ONNX)
for file_name in ["yolov8n-seg.pt", "yolo11n-seg.pt", "yolo26n-seg.pt"]:
    model = YOLO(file_name)
    model.info()
    onnx_path = model.export(format="onnx", dynamic=True, nms=False)  # YOLO26 的 NMS-free 头;对 YOLOv8/YOLO11 无操作
    model = YOLO(onnx_path)
    model(ASSETS)

使用示例#

FastSAM 模型很容易集成到你的 Python 应用中。Ultralytics 提供 Python API 和 CLI 命令。

预测用法#

要对图像进行分割,请按下方所示使用 predict 方法:

示例
from ultralytics import FastSAM

# 定义推理来源
source = "path/to/bus.jpg"

# 创建 FastSAM 模型
model = FastSAM("FastSAM-s.pt")  # 或 FastSAM-x.pt

# 对图像进行推理
everything_results = model(source, device="cpu", retina_masks=True, imgsz=1024, conf=0.4, iou=0.9)

# 使用边界框提示运行推理
results = model(source, bboxes=[439, 437, 524, 709])

# 使用点提示运行推理
results = model(source, points=[[200, 200]], labels=[1])

# 使用文本提示运行推理
results = model(source, texts="a photo of a dog")

# 同时使用边界框、点和文本提示运行推理
results = model(source, bboxes=[439, 437, 524, 709], points=[[200, 200]], labels=[1], texts="a photo of a dog")

此代码片段展示了如何轻松加载预训练模型并对图像进行预测。

FastSAMPredictor 示例

在图像上运行一次推理即可获得所有分割 results,然后可以多次对这些结果应用提示,而无需重新运行推理。

from ultralytics.models.fastsam import FastSAMPredictor

# 创建 FastSAMPredictor
overrides = {"conf": 0.25, "task": "segment", "mode": "predict", "model": "FastSAM-s.pt", "save": False, "imgsz": 1024}
predictor = FastSAMPredictor(overrides=overrides)

# 分割所有内容
everything_results = predictor("ultralytics/assets/bus.jpg")

# 提示推理
bbox_results = predictor.prompt(everything_results, bboxes=[[200, 200, 300, 300]])
point_results = predictor.prompt(everything_results, points=[200, 200])
text_results = predictor.prompt(everything_results, texts="a photo of a dog")
注意

上述示例中返回的所有 results 都是 Results 对象,可用于访问预测掩码和源图像。

验证用法#

FastSAM 仅支持检测和分割单个对象类别,因此会将所有对象识别并分割为同一类别。准备数据集时,请将所有对象类别 ID 转换为 0。

可以按如下方式在数据集上验证模型:

示例
from ultralytics import FastSAM

# 创建 FastSAM 模型
model = FastSAM("FastSAM-s.pt")  # 或 FastSAM-x.pt

# 验证模型
results = model.val(data="coco8-seg.yaml")

跟踪用法#

要在图像上执行目标跟踪,请使用 track 方法,如下所示:

示例
from ultralytics import FastSAM

# 创建 FastSAM 模型
model = FastSAM("FastSAM-s.pt")  # 或 FastSAM-x.pt

# 使用 FastSAM 模型在视频中跟踪
results = model.track(source="path/to/video.mp4", imgsz=640)

FastSAM 官方用法#

你也可以直接从 CASIA-LMC-Lab/FastSAM 仓库获取 FastSAM。下面简要介绍使用 FastSAM 时通常需要执行的步骤:

安装#

  1. 克隆 FastSAM 仓库:

    git clone https://github.com/CASIA-LMC-Lab/FastSAM.git
  2. 创建并激活包含 Python 3.9 的 Conda 环境:

    conda create -n FastSAM python=3.9
    conda activate FastSAM
  3. 进入克隆的仓库并安装所需软件包:

    cd FastSAM
    pip install -r requirements.txt
  4. 安装 CLIP 模型:

    pip install git+https://github.com/ultralytics/CLIP.git

用法示例#

  1. 下载模型检查点。

  2. 使用 FastSAM 进行推理。命令示例:

    • 分割图像中的所有对象:

      python Inference.py --model_path ./weights/FastSAM.pt --img_path ./images/dogs.jpg
    • 使用文本提示分割指定对象:

      python Inference.py --model_path ./weights/FastSAM.pt --img_path ./images/dogs.jpg --text_prompt "the yellow dog"
    • 分割边界框内的对象(以 xywh 格式提供框坐标):

      python Inference.py --model_path ./weights/FastSAM.pt --img_path ./images/dogs.jpg --box_prompt "[570,200,230,400]"
    • 分割特定点附近的对象:

      python Inference.py --model_path ./weights/FastSAM.pt --img_path ./images/dogs.jpg --point_prompt "[[520,360],[620,300]]" --point_label "[1,0]"

此外,你还可以通过 CASIA-LMC-Lab 的 Colab 演示试用 FastSAM。

引用与致谢#

我们感谢 FastSAM 作者在实时实例分割领域做出的重要贡献:

引用
@misc{zhao2023fast,
      title={Fast Segment Anything},
      author={Xu Zhao and Wenchao Ding and Yongqi An and Yinglong Du and Tao Yu and Min Li and Ming Tang and Jinqiao Wang},
      year={2023},
      eprint={2306.12156},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

FastSAM 原论文可在 arXiv 上查看。作者已公开其研究成果,代码库可在 GitHub 上访问。我们感谢他们为推动该领域发展并让更广泛的社区能够访问其成果所付出的努力。

常见问题#

  • FastSAM 是快速分割一切模型(Fast Segment Anything Model)的简称,是一种基于实时卷积神经网络(CNN)的解决方案,旨在降低计算需求,同时保持对象分割任务的高性能。与采用更大型 Transformer 架构的分割一切模型(SAM)不同,FastSAM 使用 Ultralytics YOLOv8-seg,通过两个阶段高效执行实例分割:先分割所有实例,再进行提示引导的选择。

  • FastSAM 将分割任务拆分为两个阶段,实现实时分割:先使用 YOLOv8-seg 分割所有实例,再进行提示引导的选择。借助 CNN 的计算效率,FastSAM 在保持竞争力的同时,显著降低了计算和资源需求。这种两阶段方法让 FastSAM 能够提供快速、高效的分割,适用于需要迅速获得结果的应用。

  • FastSAM 适用于各种需要实时分割性能的计算机视觉任务。应用包括:

    • 用于质量控制和保证的工业自动化
    • 用于安全防护和监控的实时视频分析
    • 用于目标检测和分割的自动驾驶汽车
    • 用于精确、快速分割任务的医学影像

    FastSAM 能够处理各种用户交互提示,因此可以灵活适应不同场景。

  • 要在 Python 中使用 FastSAM 进行推理,可以参考以下示例:

    from ultralytics import FastSAM
    
    # 定义推理来源
    source = "path/to/bus.jpg"
    
    # 创建 FastSAM 模型
    model = FastSAM("FastSAM-s.pt")  # 或 FastSAM-x.pt
    
    # 对图像进行推理
    everything_results = model(source, device="cpu", retina_masks=True, imgsz=1024, conf=0.4, iou=0.9)
    
    # 使用边界框提示运行推理
    results = model(source, bboxes=[439, 437, 524, 709])
    
    # 使用点提示运行推理
    results = model(source, points=[[200, 200]], labels=[1])
    
    # 使用文本提示运行推理
    results = model(source, texts="a photo of a dog")
    
    # 同时使用边界框、点和文本提示运行推理
    results = model(source, bboxes=[439, 437, 524, 709], points=[[200, 200]], labels=[1], texts="a photo of a dog")

    有关推理方法的更多详情,请查看文档中的预测用法部分。

  • FastSAM 支持多种提示类型,用于引导分割任务:

    • 全量提示:为所有可见对象生成分割结果。
    • 边界框 (BBox) 提示:在指定边界框内分割对象。与任何已检测掩码都不重叠的框不会选中任何内容。
    • 文本提示:使用描述性文本分割与描述相符的对象。
    • 点提示:分割用户指定点附近的对象。

    这种灵活性使 FastSAM 能够适应各种用户交互场景,提升其在不同应用中的实用性。有关如何使用这些提示的更多信息,请参阅主要功能部分。

评论26

提出问题、分享结果,或帮助其他人。

    1. PrashantDixit0GitHub(已编辑)

      @qinfendekaizhou, can you share the reproducible code for it? Or the reference from where u noticed this.

      In general, post-process time gets increased because of the post-processes involved to show the results, like

      1. NMS
      2. filtering by confidence score/ labels
      3. Scaling results to the original given inputs
      4. Number of objects to be annotated,
      5. Type of annotation, like BB or Segmentation
  1. shahedmomenzadehGitHub

    Hi, I have a question Can I use the generated bbox cordinates by a yolo detection model like yolov11 as bboxes prompt for FastSAM ?

    1. Glenn JocherUltralytics

      @shahedmomenzadeh yes, you can use YOLO detection model-generated bbox coordinates as prompts for FastSAM. The FastSAM documentation shows examples of passing bbox coordinates directly to the model using either the Python API or CLI. Just ensure your coordinates are in XYXY format as shown in the examples.

  2. ismailwaleGitHub

    How can I make an inference of a video using frame by frame in loop using FastSam ?because I tried multiple times making inference directly with video predictor I constantly running out of memory despite setting stream =True since SAM is trying to keep the memory of 2000 frames of the video at ones . I will appreciate any ideal on how to predict and segment the video as a single frame image at every instance and put everything in a loop.

    Similar question win MobileSam

    1. Glenn JocherUltralytics

      @ismailwale to process a video frame by frame and avoid memory overload with FastSAM or MobileSAM, load the video as a frame sequence using a tool like OpenCV. Loop through each frame, run inference on it individually, and clear any cached results to free up memory between frames. For reference on frame-wise usage, consult OpenCV and FastSAM integration. Let me know if you need further clarifications!

    2. llu025GitHub

      Thank you all for your great work! I am having troubles with prompting with a bounding box, it seems the FastSAM model ignores it and detects objects all over the video frames. This is how I obtain the results:

      results = model(
          input_filename,
          device=device,
          imgsz=(736, 1280),
          verbose=False,
          stream=True,
          bboxes=[0, 0, 360, 640],
          retina_masks=True,
          conf=0.4,
          iou=0.9,
      )

      I also tried with bboxes=[[0, 0, 360, 640]], without success. The model should detect objects in the top left quadrant of the video, instead it detects everywhere. Also, it oddly does not raise any error or warnings, both with bboxes as list of integers or bboxes as list of lists of integers.

    3. Glenn JocherUltralytics

      It seems the bounding box prompts aren't being applied as expected. Ensure the bboxes format is correct (e.g., [[x1, y1, x2, y2]] as a list of lists). Additionally, try using the FastSAMPredictor approach for more controlled prompt application, as shown in this example. Let me know if the issue persists!

    4. llu025GitHub

      Thank you @glenn-jocher for the "prompt" response ;-)

      I need to use stream=True since I am working with large videos. I cannot use it coupled with FastSAMPredictor, because it does not support it. My only alternative is to grab frames on my own with OpenCV, is that correct?

      By the way, in your examples in here bboxes is passed as a list of integers, I followed those in the beginning. I see in the examples with FastSAMPredictor that a list of lists is used instead. In any case, I ensured that I was using a list of lists, and it did not work anyway.

      Here's a minimal example of what I am trying to do, i.e., detecting objects in the top left part of the video:

      from ultralytics import FastSAM
      import os
      import ffmpeg
      
      input_filename = "test_video.mp4"
      base_name = os.path.splitext(os.path.basename(input_filename))[0]
      output_filename = os.path.join("output", f"{base_name}.avi")
      
      model = FastSAM("models/FastSAM-x.pt")
      
      process2 = (
          ffmpeg.input(
              "pipe:",
              format="rawvideo",
              pix_fmt="rgb24",
              s="1280x720",
              loglevel="-8",
          )
          .output(
              output_filename,
              pix_fmt="yuv420p",
              vcodec="libx264",
              loglevel="-8",
              crf=17,
          )
          .overwrite_output()
          .run_async(pipe_stdin=True)
      )
      
      results = model(
          input_filename,
          device="cuda",
          imgsz=(736, 1280),
          verbose=False,
          stream=True,
          bboxes=[[0, 0, 360, 640]],
          retina_masks=True,
          conf=0.4,
          iou=0.9,
      )
      
      for result in results:
          frame = result.plot()
          # Save on disk with process2
          process2.stdin.write(frame.tobytes())
      
      process2.stdin.close()
      process2.wait()
    5. Glenn JocherUltralytics

      Thank you for the detailed explanation and example. Yes, you're correct—using OpenCV to manually grab frames is a valid and efficient approach, especially when working with stream=True for large videos, as it avoids memory issues. Regarding bboxes, FastSAM expects a list of lists in the format [[x1, y1, x2, y2]]. If this still doesn't work as intended, please ensure your bounding box coordinates are correct and within the frame dimensions. Let us know if you encounter any further issues!

    6. llu025GitHub

      I still encounter the same issue. I used bboxes=[[20, 20, 500, 500]] just to be 100% sure the coordinates are within the the frame dimensions (1280x720), nothing changed. Since I have to grab frames with OpenCV anyway, I will just crop them myself, but I am quite sure this option is not working as intended.

    7. Glenn JocherUltralytics

      @llu025 thank you for the update. If you’re certain the bboxes parameter is properly configured and still not working as intended, cropping frames manually with OpenCV before passing them to the model is indeed a reliable workaround. It may also be worth checking the latest FastSAM updates or reporting this as a potential issue to ensure the bounding box prompt functionality is reviewed. Let us know if you need further assistance!

  3. chongkuiqiGitHub

    Thanks for your great work! I'm using FastSAM-x with bboxes prompt, the original image shape is [720, 1280], and I set 'imgsz' 1024. However, the results mask shape is [576, 1024], but I remembered in ultralytics-8.2.10, I can get masks with the same shape as the original image. May I ask why this change happened ?

    1. Glenn JocherUltralytics

      @feiwenxiong fastSAM should support multiple boxes. If you're experiencing limitations, please ensure your input format is correct. For further assistance, you might want to check the latest updates or documentation.

  4. YuechengLiuGitHub

    Thanks the authors for the great work!

    I'm using FastSAM-x for instance segmentation (bboxes as the prompt), but I find that the shape of the predicted masks mismatch with the bounding box number. (e.g. the model takes 18 bboxes as input, but the shape of the predicted mask is (17, H, W)) . I tried to set conf=0.01 and iou=0.999, but it doesn't work. How can I solve that? Thanks!

    1. Glenn JocherUltralytics

      @YuechengLiu thank you for your kind words! The issue you're experiencing might be due to a mismatch in the bounding box prompts and the segmentation results. Please ensure you're using the latest version of the FastSAM model. If the problem persists, consider checking the input format and the bounding box coordinates. For further assistance, you can refer to the FastSAM documentation: https://docs.ultralytics.com/models/fast-sam/.

    2. YuechengLiuGitHub

      Thanks for your kind reply! The bounding boxes I'm using are predicted by yolov10x (result.boxes.xyxy), so I think the input format might be right. I also tried the same bounding boxes for mobile_sam, and the output dimension is as expected (same as the input dimension). The problem does not always occur. Do you have any idea about that? Thanks!

    3. Glenn JocherUltralytics

      Thanks for confirming you're using the latest version. Given that the issue is intermittent, it might be related to the specific bounding boxes or the image content. Please try running the model with a different set of bounding boxes or images to see if the issue persists. If it does, consider opening an issue on the GitHub repository with detailed information for further investigation.

  5. nguyenvandang1998GitHub

    Hi! I am using the FastSAM model and trying to obtain information about the bounding box and mask image of the object, but I don't know how to do it. I hope you can help me. Thank you very much!!

    1. Glenn JocherUltralytics(已编辑)

      Hi there! 🙌 To extract both bounding boxes and mask images using the FastSAM model, you would typically work with the results object returned by the model's prediction method. For the bounding boxes, you can access them directly as shown in the documentation or examples provided. For the mask images, since FastSAM is designed for real-time segmentation, the results should also include segmentation masks corresponding to detected objects.

      Here's a simplified way to access both:

      from ultralytics import FastSAM
      
      # Define your image source
      source = 'path/to/your/image.jpg'
      
      # Load the FastSAM model (ensure you've chosen the correct model path)
      model = FastSAM('FastSAM-s.pt')  # or another appropriate model
      
      # Perform inference
      results = model(source)
      
      # Accessing bounding boxes (assuming XYXY format)
      boxes = results[0].boxes.xyxy
      
      # Accessing segmentation masks
      # The masks are directly associated with detected objects in the results
      masks = results[0].masks
      
      # To visualize or further process masks and boxes, you would iterate through these objects

      To visualize or save the masks, you'd typically iterate over the masks object, and each mask can be processed according to your requirements (e.g., overlaying on the original image or saving as separate mask images).

      Remember, the exact attribute names (boxes, masks) and methods might vary based on the model output specification, so it's always good to refer to the specific documentation or example codes for guidance.

      Hope this helps! Let us know if you have more questions. 😊

      2