μ½˜ν…μΈ λ‘œ κ±΄λ„ˆλ›°κΈ°

μ°Έμ‘° ultralytics/models/sam/modules/encoders.py

μ°Έκ³ 

이 νŒŒμΌμ€ https://github.com/ultralytics/ ultralytics/blob/main/ ultralytics/models/ sam/modules/encoders .pyμ—μ„œ 확인할 수 μžˆμŠ΅λ‹ˆλ‹€. 문제λ₯Ό λ°œκ²¬ν•˜λ©΄ ν’€ λ¦¬ν€˜μŠ€νŠΈ (πŸ› οΈ) λ₯Ό 톡해 문제λ₯Ό ν•΄κ²°ν•˜λ„λ‘ λ„μ™€μ£Όμ„Έμš”. κ°μ‚¬ν•©λ‹ˆλ‹€ πŸ™!



ultralytics.models.sam.modules.encoders.ImageEncoderViT

기지: Module

이미지λ₯Ό μ»΄νŒ©νŠΈν•œ μž μƒ κ³΅κ°„μœΌλ‘œ μΈμ½”λ”©ν•˜κΈ° μœ„ν•΄ λΉ„μ „ 트랜슀포머(ViT) μ•„ν‚€ν…μ²˜λ₯Ό μ‚¬μš©ν•˜λŠ” 이미지 μΈμ½”λ”μž…λ‹ˆλ‹€. The μΈμ½”λ”λŠ” 이미지λ₯Ό κ°€μ Έμ™€μ„œ 패치둜 λΆ„ν• ν•˜κ³  일련의 트랜슀포머 블둝을 톡해 μ΄λŸ¬ν•œ 패치λ₯Ό μ²˜λ¦¬ν•©λ‹ˆλ‹€. 그런 λ‹€μŒ μΈμ½”λ”©λœ νŒ¨μΉ˜λŠ” λ„₯을 톡해 μ²˜λ¦¬λ˜μ–΄ μ΅œμ’… μΈμ½”λ”©λœ ν‘œν˜„μ„ μƒμ„±ν•©λ‹ˆλ‹€.

이 ν΄λž˜μŠ€μ™€ μ•„λž˜ 지원 ν•¨μˆ˜λŠ” λ‹€μŒμ—μ„œ μ œκ³΅λ˜λŠ” ViTDet λ°±λ³Έμ—μ„œ μ•½κ°„ μˆ˜μ •λ˜μ—ˆμŠ΅λ‹ˆλ‹€. https://github.com/facebookresearch/detectron2/blob/main/detectron2/modeling/backbone/vit.py.

속성:

이름 μœ ν˜• μ„€λͺ…
img_size int

μž…λ ₯ μ΄λ―Έμ§€μ˜ 치수둜, μ •μ‚¬κ°ν˜•μœΌλ‘œ κ°€μ •ν•©λ‹ˆλ‹€.

patch_embed PatchEmbed

패치 μž„λ² λ”©μ„ μœ„ν•œ λͺ¨λ“ˆμž…λ‹ˆλ‹€.

pos_embed Parameter

패치λ₯Ό μœ„ν•œ μ ˆλŒ€ μœ„μΉ˜ μž„λ² λ”©.

blocks ModuleList

패치 μž„λ² λ”© 처리λ₯Ό μœ„ν•œ 트랜슀포머 블둝 λͺ©λ‘μž…λ‹ˆλ‹€.

neck Sequential

λ„₯ λͺ¨λ“ˆμ„ μ‚¬μš©ν•˜μ—¬ 좜λ ₯을 μΆ”κ°€λ‘œ μ²˜λ¦¬ν•©λ‹ˆλ‹€.

의 μ†ŒμŠ€ μ½”λ“œ ultralytics/models/sam/modules/encoders.py
class ImageEncoderViT(nn.Module):
    """
    An image encoder using Vision Transformer (ViT) architecture for encoding an image into a compact latent space. The
    encoder takes an image, splits it into patches, and processes these patches through a series of transformer blocks.
    The encoded patches are then processed through a neck to generate the final encoded representation.

    This class and its supporting functions below lightly adapted from the ViTDet backbone available at
    https://github.com/facebookresearch/detectron2/blob/main/detectron2/modeling/backbone/vit.py.

    Attributes:
        img_size (int): Dimension of input images, assumed to be square.
        patch_embed (PatchEmbed): Module for patch embedding.
        pos_embed (nn.Parameter, optional): Absolute positional embedding for patches.
        blocks (nn.ModuleList): List of transformer blocks for processing patch embeddings.
        neck (nn.Sequential): Neck module to further process the output.
    """

    def __init__(
        self,
        img_size: int = 1024,
        patch_size: int = 16,
        in_chans: int = 3,
        embed_dim: int = 768,
        depth: int = 12,
        num_heads: int = 12,
        mlp_ratio: float = 4.0,
        out_chans: int = 256,
        qkv_bias: bool = True,
        norm_layer: Type[nn.Module] = nn.LayerNorm,
        act_layer: Type[nn.Module] = nn.GELU,
        use_abs_pos: bool = True,
        use_rel_pos: bool = False,
        rel_pos_zero_init: bool = True,
        window_size: int = 0,
        global_attn_indexes: Tuple[int, ...] = (),
    ) -> None:
        """
        Args:
            img_size (int): Input image size.
            patch_size (int): Patch size.
            in_chans (int): Number of input image channels.
            embed_dim (int): Patch embedding dimension.
            depth (int): Depth of ViT.
            num_heads (int): Number of attention heads in each ViT block.
            mlp_ratio (float): Ratio of mlp hidden dim to embedding dim.
            qkv_bias (bool): If True, add a learnable bias to query, key, value.
            norm_layer (nn.Module): Normalization layer.
            act_layer (nn.Module): Activation layer.
            use_abs_pos (bool): If True, use absolute positional embeddings.
            use_rel_pos (bool): If True, add relative positional embeddings to the attention map.
            rel_pos_zero_init (bool): If True, zero initialize relative positional parameters.
            window_size (int): Window size for window attention blocks.
            global_attn_indexes (list): Indexes for blocks using global attention.
        """
        super().__init__()
        self.img_size = img_size

        self.patch_embed = PatchEmbed(
            kernel_size=(patch_size, patch_size),
            stride=(patch_size, patch_size),
            in_chans=in_chans,
            embed_dim=embed_dim,
        )

        self.pos_embed: Optional[nn.Parameter] = None
        if use_abs_pos:
            # Initialize absolute positional embedding with pretrain image size.
            self.pos_embed = nn.Parameter(torch.zeros(1, img_size // patch_size, img_size // patch_size, embed_dim))

        self.blocks = nn.ModuleList()
        for i in range(depth):
            block = Block(
                dim=embed_dim,
                num_heads=num_heads,
                mlp_ratio=mlp_ratio,
                qkv_bias=qkv_bias,
                norm_layer=norm_layer,
                act_layer=act_layer,
                use_rel_pos=use_rel_pos,
                rel_pos_zero_init=rel_pos_zero_init,
                window_size=window_size if i not in global_attn_indexes else 0,
                input_size=(img_size // patch_size, img_size // patch_size),
            )
            self.blocks.append(block)

        self.neck = nn.Sequential(
            nn.Conv2d(
                embed_dim,
                out_chans,
                kernel_size=1,
                bias=False,
            ),
            LayerNorm2d(out_chans),
            nn.Conv2d(
                out_chans,
                out_chans,
                kernel_size=3,
                padding=1,
                bias=False,
            ),
            LayerNorm2d(out_chans),
        )

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        """Processes input through patch embedding, applies positional embedding if present, and passes through blocks
        and neck.
        """
        x = self.patch_embed(x)
        if self.pos_embed is not None:
            x = x + self.pos_embed
        for blk in self.blocks:
            x = blk(x)
        return self.neck(x.permute(0, 3, 1, 2))

__init__(img_size=1024, patch_size=16, in_chans=3, embed_dim=768, depth=12, num_heads=12, mlp_ratio=4.0, out_chans=256, qkv_bias=True, norm_layer=nn.LayerNorm, act_layer=nn.GELU, use_abs_pos=True, use_rel_pos=False, rel_pos_zero_init=True, window_size=0, global_attn_indexes=())

λ§€κ°œλ³€μˆ˜:

이름 μœ ν˜• μ„€λͺ… κΈ°λ³Έκ°’
img_size int

이미지 크기λ₯Ό μž…λ ₯ν•©λ‹ˆλ‹€.

1024
patch_size int

패치 크기.

16
in_chans int

μž…λ ₯ 이미지 채널 μˆ˜μž…λ‹ˆλ‹€.

3
embed_dim int

패치 μž„λ² λ”© 차원.

768
depth int

ViT의 깊이.

12
num_heads int

각 ViT λΈ”λ‘μ˜ 관심 ν—€λ“œ μˆ˜μž…λ‹ˆλ‹€.

12
mlp_ratio float

MLP μˆ¨κΉ€ μ–΄λ‘‘κ²Œμ™€ μž„λ² λ”© μ–΄λ‘‘κ²Œμ˜ λΉ„μœ¨μž…λ‹ˆλ‹€.

4.0
qkv_bias bool

True인 경우 쿼리, ν‚€, 값에 ν•™μŠ΅ κ°€λŠ₯ν•œ 편ν–₯을 μΆ”κ°€ν•©λ‹ˆλ‹€.

True
norm_layer Module

μ •κ·œν™” λ ˆμ΄μ–΄μž…λ‹ˆλ‹€.

LayerNorm
act_layer Module

ν™œμ„±ν™” λ ˆμ΄μ–΄.

GELU
use_abs_pos bool

True이면 μ ˆλŒ€ μœ„μΉ˜ μž„λ² λ”©μ„ μ‚¬μš©ν•©λ‹ˆλ‹€.

True
use_rel_pos bool

True인 경우 관심도 맡에 μƒλŒ€μ  μœ„μΉ˜ μž„λ² λ”©μ„ μΆ”κ°€ν•©λ‹ˆλ‹€.

False
rel_pos_zero_init bool

True이면 μƒλŒ€ μœ„μΉ˜ 맀개 λ³€μˆ˜λ₯Ό 0으둜 μ΄ˆκΈ°ν™”ν•©λ‹ˆλ‹€.

True
window_size int

μ°½ 주의 λΈ”λ‘μ˜ μ°½ ν¬κΈ°μž…λ‹ˆλ‹€.

0
global_attn_indexes list

κΈ€λ‘œλ²Œ 관심도λ₯Ό μ‚¬μš©ν•˜μ—¬ 블둝을 μΈλ±μ‹±ν•©λ‹ˆλ‹€.

()
의 μ†ŒμŠ€ μ½”λ“œ ultralytics/models/sam/modules/encoders.py
def __init__(
    self,
    img_size: int = 1024,
    patch_size: int = 16,
    in_chans: int = 3,
    embed_dim: int = 768,
    depth: int = 12,
    num_heads: int = 12,
    mlp_ratio: float = 4.0,
    out_chans: int = 256,
    qkv_bias: bool = True,
    norm_layer: Type[nn.Module] = nn.LayerNorm,
    act_layer: Type[nn.Module] = nn.GELU,
    use_abs_pos: bool = True,
    use_rel_pos: bool = False,
    rel_pos_zero_init: bool = True,
    window_size: int = 0,
    global_attn_indexes: Tuple[int, ...] = (),
) -> None:
    """
    Args:
        img_size (int): Input image size.
        patch_size (int): Patch size.
        in_chans (int): Number of input image channels.
        embed_dim (int): Patch embedding dimension.
        depth (int): Depth of ViT.
        num_heads (int): Number of attention heads in each ViT block.
        mlp_ratio (float): Ratio of mlp hidden dim to embedding dim.
        qkv_bias (bool): If True, add a learnable bias to query, key, value.
        norm_layer (nn.Module): Normalization layer.
        act_layer (nn.Module): Activation layer.
        use_abs_pos (bool): If True, use absolute positional embeddings.
        use_rel_pos (bool): If True, add relative positional embeddings to the attention map.
        rel_pos_zero_init (bool): If True, zero initialize relative positional parameters.
        window_size (int): Window size for window attention blocks.
        global_attn_indexes (list): Indexes for blocks using global attention.
    """
    super().__init__()
    self.img_size = img_size

    self.patch_embed = PatchEmbed(
        kernel_size=(patch_size, patch_size),
        stride=(patch_size, patch_size),
        in_chans=in_chans,
        embed_dim=embed_dim,
    )

    self.pos_embed: Optional[nn.Parameter] = None
    if use_abs_pos:
        # Initialize absolute positional embedding with pretrain image size.
        self.pos_embed = nn.Parameter(torch.zeros(1, img_size // patch_size, img_size // patch_size, embed_dim))

    self.blocks = nn.ModuleList()
    for i in range(depth):
        block = Block(
            dim=embed_dim,
            num_heads=num_heads,
            mlp_ratio=mlp_ratio,
            qkv_bias=qkv_bias,
            norm_layer=norm_layer,
            act_layer=act_layer,
            use_rel_pos=use_rel_pos,
            rel_pos_zero_init=rel_pos_zero_init,
            window_size=window_size if i not in global_attn_indexes else 0,
            input_size=(img_size // patch_size, img_size // patch_size),
        )
        self.blocks.append(block)

    self.neck = nn.Sequential(
        nn.Conv2d(
            embed_dim,
            out_chans,
            kernel_size=1,
            bias=False,
        ),
        LayerNorm2d(out_chans),
        nn.Conv2d(
            out_chans,
            out_chans,
            kernel_size=3,
            padding=1,
            bias=False,
        ),
        LayerNorm2d(out_chans),
    )

forward(x)

패치 μž„λ² λ”©μ„ 톡해 μž…λ ₯을 μ²˜λ¦¬ν•˜κ³ , μœ„μΉ˜ μž„λ² λ”©μ΄ μžˆλŠ” 경우 μ μš©ν•˜λ©°, 블둝을 ν†΅κ³Όν•©λ‹ˆλ‹€. κ³Ό λͺ©μ„ ν†΅κ³Όν•©λ‹ˆλ‹€.

의 μ†ŒμŠ€ μ½”λ“œ ultralytics/models/sam/modules/encoders.py
def forward(self, x: torch.Tensor) -> torch.Tensor:
    """Processes input through patch embedding, applies positional embedding if present, and passes through blocks
    and neck.
    """
    x = self.patch_embed(x)
    if self.pos_embed is not None:
        x = x + self.pos_embed
    for blk in self.blocks:
        x = blk(x)
    return self.neck(x.permute(0, 3, 1, 2))



ultralytics.models.sam.modules.encoders.PromptEncoder

기지: Module

SAM 의 마슀크 디코더에 μž…λ ₯ν•  포인트, μƒμž, 마슀크 λ“± λ‹€μ–‘ν•œ μœ ν˜•μ˜ ν”„λ‘¬ν”„νŠΈλ₯Ό μΈμ½”λ”©ν•©λ‹ˆλ‹€. 인코더 λŠ” μž…λ ₯ ν”„λ‘¬ν”„νŠΈμ— λŒ€ν•΄ ν¬μ†Œ μž„λ² λ”©κ³Ό 고밀도 μž„λ² λ”©μ„ λͺ¨λ‘ μƒμ„±ν•©λ‹ˆλ‹€.

속성:

이름 μœ ν˜• μ„€λͺ…
embed_dim int

μž„λ² λ”©μ˜ ν¬κΈ°μž…λ‹ˆλ‹€.

input_image_size Tuple[int, int]

μž…λ ₯ μ΄λ―Έμ§€μ˜ 크기λ₯Ό (H, W)둜 μ„€μ •ν•©λ‹ˆλ‹€.

image_embedding_size Tuple[int, int]

이미지 μž„λ² λ”©μ˜ 곡간 크기λ₯Ό (H, W)둜 μ„€μ •ν•©λ‹ˆλ‹€.

pe_layer PositionEmbeddingRandom

μž„μ˜ μœ„μΉ˜ μž„λ² λ”©μ„ μœ„ν•œ λͺ¨λ“ˆμž…λ‹ˆλ‹€.

num_point_embeddings int

λ‹€μ–‘ν•œ μœ ν˜•μ˜ ν¬μΈνŠΈμ— λŒ€ν•œ 포인트 μž„λ² λ”© κ°œμˆ˜μž…λ‹ˆλ‹€.

point_embeddings ModuleList

포인트 μž„λ² λ”© λͺ©λ‘μž…λ‹ˆλ‹€.

not_a_point_embed Embedding

λ ˆμ΄λΈ”μ˜ 일뢀가 μ•„λ‹Œ ν¬μΈνŠΈμ— λŒ€ν•œ μž„λ² λ”©.

mask_input_size Tuple[int, int]

μž…λ ₯ 마슀크의 ν¬κΈ°μž…λ‹ˆλ‹€.

mask_downscaling Sequential

마슀크 λ‹€μš΄μŠ€μΌ€μΌλ§μ„ μœ„ν•œ μ‹ κ²½λ§μž…λ‹ˆλ‹€.

no_mask_embed Embedding

λ§ˆμŠ€ν¬κ°€ μ œκ³΅λ˜μ§€ μ•ŠλŠ” 경우 μž„λ² λ“œν•©λ‹ˆλ‹€.

의 μ†ŒμŠ€ μ½”λ“œ ultralytics/models/sam/modules/encoders.py
class PromptEncoder(nn.Module):
    """
    Encodes different types of prompts, including points, boxes, and masks, for input to SAM's mask decoder. The encoder
    produces both sparse and dense embeddings for the input prompts.

    Attributes:
        embed_dim (int): Dimension of the embeddings.
        input_image_size (Tuple[int, int]): Size of the input image as (H, W).
        image_embedding_size (Tuple[int, int]): Spatial size of the image embedding as (H, W).
        pe_layer (PositionEmbeddingRandom): Module for random position embedding.
        num_point_embeddings (int): Number of point embeddings for different types of points.
        point_embeddings (nn.ModuleList): List of point embeddings.
        not_a_point_embed (nn.Embedding): Embedding for points that are not a part of any label.
        mask_input_size (Tuple[int, int]): Size of the input mask.
        mask_downscaling (nn.Sequential): Neural network for downscaling the mask.
        no_mask_embed (nn.Embedding): Embedding for cases where no mask is provided.
    """

    def __init__(
        self,
        embed_dim: int,
        image_embedding_size: Tuple[int, int],
        input_image_size: Tuple[int, int],
        mask_in_chans: int,
        activation: Type[nn.Module] = nn.GELU,
    ) -> None:
        """
        Encodes prompts for input to SAM's mask decoder.

        Args:
          embed_dim (int): The prompts' embedding dimension
          image_embedding_size (tuple(int, int)): The spatial size of the
            image embedding, as (H, W).
          input_image_size (int): The padded size of the image as input
            to the image encoder, as (H, W).
          mask_in_chans (int): The number of hidden channels used for
            encoding input masks.
          activation (nn.Module): The activation to use when encoding
            input masks.
        """
        super().__init__()
        self.embed_dim = embed_dim
        self.input_image_size = input_image_size
        self.image_embedding_size = image_embedding_size
        self.pe_layer = PositionEmbeddingRandom(embed_dim // 2)

        self.num_point_embeddings: int = 4  # pos/neg point + 2 box corners
        point_embeddings = [nn.Embedding(1, embed_dim) for _ in range(self.num_point_embeddings)]
        self.point_embeddings = nn.ModuleList(point_embeddings)
        self.not_a_point_embed = nn.Embedding(1, embed_dim)

        self.mask_input_size = (4 * image_embedding_size[0], 4 * image_embedding_size[1])
        self.mask_downscaling = nn.Sequential(
            nn.Conv2d(1, mask_in_chans // 4, kernel_size=2, stride=2),
            LayerNorm2d(mask_in_chans // 4),
            activation(),
            nn.Conv2d(mask_in_chans // 4, mask_in_chans, kernel_size=2, stride=2),
            LayerNorm2d(mask_in_chans),
            activation(),
            nn.Conv2d(mask_in_chans, embed_dim, kernel_size=1),
        )
        self.no_mask_embed = nn.Embedding(1, embed_dim)

    def get_dense_pe(self) -> torch.Tensor:
        """
        Returns the positional encoding used to encode point prompts, applied to a dense set of points the shape of the
        image encoding.

        Returns:
          torch.Tensor: Positional encoding with shape 1x(embed_dim)x(embedding_h)x(embedding_w)
        """
        return self.pe_layer(self.image_embedding_size).unsqueeze(0)

    def _embed_points(self, points: torch.Tensor, labels: torch.Tensor, pad: bool) -> torch.Tensor:
        """Embeds point prompts."""
        points = points + 0.5  # Shift to center of pixel
        if pad:
            padding_point = torch.zeros((points.shape[0], 1, 2), device=points.device)
            padding_label = -torch.ones((labels.shape[0], 1), device=labels.device)
            points = torch.cat([points, padding_point], dim=1)
            labels = torch.cat([labels, padding_label], dim=1)
        point_embedding = self.pe_layer.forward_with_coords(points, self.input_image_size)
        point_embedding[labels == -1] = 0.0
        point_embedding[labels == -1] += self.not_a_point_embed.weight
        point_embedding[labels == 0] += self.point_embeddings[0].weight
        point_embedding[labels == 1] += self.point_embeddings[1].weight
        return point_embedding

    def _embed_boxes(self, boxes: torch.Tensor) -> torch.Tensor:
        """Embeds box prompts."""
        boxes = boxes + 0.5  # Shift to center of pixel
        coords = boxes.reshape(-1, 2, 2)
        corner_embedding = self.pe_layer.forward_with_coords(coords, self.input_image_size)
        corner_embedding[:, 0, :] += self.point_embeddings[2].weight
        corner_embedding[:, 1, :] += self.point_embeddings[3].weight
        return corner_embedding

    def _embed_masks(self, masks: torch.Tensor) -> torch.Tensor:
        """Embeds mask inputs."""
        return self.mask_downscaling(masks)

    def _get_batch_size(
        self,
        points: Optional[Tuple[torch.Tensor, torch.Tensor]],
        boxes: Optional[torch.Tensor],
        masks: Optional[torch.Tensor],
    ) -> int:
        """Gets the batch size of the output given the batch size of the input prompts."""
        if points is not None:
            return points[0].shape[0]
        elif boxes is not None:
            return boxes.shape[0]
        elif masks is not None:
            return masks.shape[0]
        else:
            return 1

    def _get_device(self) -> torch.device:
        """Returns the device of the first point embedding's weight tensor."""
        return self.point_embeddings[0].weight.device

    def forward(
        self,
        points: Optional[Tuple[torch.Tensor, torch.Tensor]],
        boxes: Optional[torch.Tensor],
        masks: Optional[torch.Tensor],
    ) -> Tuple[torch.Tensor, torch.Tensor]:
        """
        Embeds different types of prompts, returning both sparse and dense embeddings.

        Args:
          points (tuple(torch.Tensor, torch.Tensor), None): point coordinates and labels to embed.
          boxes (torch.Tensor, None): boxes to embed
          masks (torch.Tensor, None): masks to embed

        Returns:
          torch.Tensor: sparse embeddings for the points and boxes, with shape BxNx(embed_dim), where N is determined
            by the number of input points and boxes.
          torch.Tensor: dense embeddings for the masks, in the shape Bx(embed_dim)x(embed_H)x(embed_W)
        """
        bs = self._get_batch_size(points, boxes, masks)
        sparse_embeddings = torch.empty((bs, 0, self.embed_dim), device=self._get_device())
        if points is not None:
            coords, labels = points
            point_embeddings = self._embed_points(coords, labels, pad=(boxes is None))
            sparse_embeddings = torch.cat([sparse_embeddings, point_embeddings], dim=1)
        if boxes is not None:
            box_embeddings = self._embed_boxes(boxes)
            sparse_embeddings = torch.cat([sparse_embeddings, box_embeddings], dim=1)

        if masks is not None:
            dense_embeddings = self._embed_masks(masks)
        else:
            dense_embeddings = self.no_mask_embed.weight.reshape(1, -1, 1, 1).expand(
                bs, -1, self.image_embedding_size[0], self.image_embedding_size[1]
            )

        return sparse_embeddings, dense_embeddings

__init__(embed_dim, image_embedding_size, input_image_size, mask_in_chans, activation=nn.GELU)

μž…λ ₯ ν”„λ‘¬ν”„νŠΈλ₯Ό SAM 의 마슀크 λ””μ½”λ”λ‘œ μΈμ½”λ”©ν•©λ‹ˆλ‹€.

λ§€κ°œλ³€μˆ˜:

이름 μœ ν˜• μ„€λͺ… κΈ°λ³Έκ°’
embed_dim int

ν”„λ‘¬ν”„νŠΈμ˜ μž„λ² λ”© 차원

ν•„μˆ˜
image_embedding_size tuple(int, int

이미지 μž„λ² λ”©μ˜ 곡간 크기 이미지 μž„λ² λ”©μ˜ 곡간 크기(H, W)μž…λ‹ˆλ‹€.

ν•„μˆ˜
input_image_size int

이미지 인코더에 μž…λ ₯λ˜λŠ” μ΄λ―Έμ§€μ˜ νŒ¨λ”©λœ ν¬κΈ°μž…λ‹ˆλ‹€. 둜 이미지 인코더에 μ „λ‹¬λ˜λŠ” μ΄λ―Έμ§€μ˜ νŒ¨λ”©λœ 크기(H, W)μž…λ‹ˆλ‹€.

ν•„μˆ˜
mask_in_chans int

μž…λ ₯ 마슀크 인코딩에 μ‚¬μš©λ˜λŠ” μˆ¨κ²¨μ§„ μ±„λ„μ˜ μž…λ ₯ 마슀크 인코딩에 μ‚¬μš©λ˜λŠ” μˆ¨κ²¨μ§„ 채널 μˆ˜μž…λ‹ˆλ‹€.

ν•„μˆ˜
activation Module

인코딩할 λ•Œ μ‚¬μš©ν•  ν™œμ„±ν™” μž…λ ₯ 마슀크λ₯Ό 인코딩할 λ•Œ μ‚¬μš©ν•  ν™œμ„±ν™”μž…λ‹ˆλ‹€.

GELU
의 μ†ŒμŠ€ μ½”λ“œ ultralytics/models/sam/modules/encoders.py
def __init__(
    self,
    embed_dim: int,
    image_embedding_size: Tuple[int, int],
    input_image_size: Tuple[int, int],
    mask_in_chans: int,
    activation: Type[nn.Module] = nn.GELU,
) -> None:
    """
    Encodes prompts for input to SAM's mask decoder.

    Args:
      embed_dim (int): The prompts' embedding dimension
      image_embedding_size (tuple(int, int)): The spatial size of the
        image embedding, as (H, W).
      input_image_size (int): The padded size of the image as input
        to the image encoder, as (H, W).
      mask_in_chans (int): The number of hidden channels used for
        encoding input masks.
      activation (nn.Module): The activation to use when encoding
        input masks.
    """
    super().__init__()
    self.embed_dim = embed_dim
    self.input_image_size = input_image_size
    self.image_embedding_size = image_embedding_size
    self.pe_layer = PositionEmbeddingRandom(embed_dim // 2)

    self.num_point_embeddings: int = 4  # pos/neg point + 2 box corners
    point_embeddings = [nn.Embedding(1, embed_dim) for _ in range(self.num_point_embeddings)]
    self.point_embeddings = nn.ModuleList(point_embeddings)
    self.not_a_point_embed = nn.Embedding(1, embed_dim)

    self.mask_input_size = (4 * image_embedding_size[0], 4 * image_embedding_size[1])
    self.mask_downscaling = nn.Sequential(
        nn.Conv2d(1, mask_in_chans // 4, kernel_size=2, stride=2),
        LayerNorm2d(mask_in_chans // 4),
        activation(),
        nn.Conv2d(mask_in_chans // 4, mask_in_chans, kernel_size=2, stride=2),
        LayerNorm2d(mask_in_chans),
        activation(),
        nn.Conv2d(mask_in_chans, embed_dim, kernel_size=1),
    )
    self.no_mask_embed = nn.Embedding(1, embed_dim)

forward(points, boxes, masks)

λ‹€μ–‘ν•œ μœ ν˜•μ˜ ν”„λ‘¬ν”„νŠΈλ₯Ό μž„λ² λ“œν•˜μ—¬ ν¬λ°•ν•œ μž„λ² λ”©κ³Ό μ‘°λ°€ν•œ μž„λ² λ”©μ„ λͺ¨λ‘ λ°˜ν™˜ν•©λ‹ˆλ‹€.

λ§€κ°œλ³€μˆ˜:

이름 μœ ν˜• μ„€λͺ… κΈ°λ³Έκ°’
points (tuple(Tensor, Tensor), None)

포인트 μ’Œν‘œμ™€ λ ˆμ΄λΈ”μ„ μ‚½μž…ν•©λ‹ˆλ‹€.

ν•„μˆ˜
boxes (Tensor, None)

μ‚½μž…ν•  μƒμž

ν•„μˆ˜
masks (Tensor, None)

μ‚½μž…ν•  마슀크

ν•„μˆ˜

λ°˜ν™˜ν•©λ‹ˆλ‹€:

μœ ν˜• μ„€λͺ…
Tensor

torch.Tensor: 점과 μƒμžμ— λŒ€ν•œ ν¬μ†Œ μž„λ² λ”©, BxNx(embed_dim) λͺ¨μ–‘, μ—¬κΈ°μ„œ N은 μž…λ ₯ 점과 μƒμžμ˜ μˆ˜μ— μ˜ν•΄ κ²°μ •λ©λ‹ˆλ‹€. μž…λ ₯된 점과 μƒμžμ˜ μˆ˜μ— 따라 κ²°μ •λ©λ‹ˆλ‹€.

Tensor

torch.Tensor: λ§ˆμŠ€ν¬μ— λŒ€ν•œ 고밀도 μž„λ² λ”©, Bx(embed_dim)x(embed_H)x(embed_W) λͺ¨μ–‘.

의 μ†ŒμŠ€ μ½”λ“œ ultralytics/models/sam/modules/encoders.py
def forward(
    self,
    points: Optional[Tuple[torch.Tensor, torch.Tensor]],
    boxes: Optional[torch.Tensor],
    masks: Optional[torch.Tensor],
) -> Tuple[torch.Tensor, torch.Tensor]:
    """
    Embeds different types of prompts, returning both sparse and dense embeddings.

    Args:
      points (tuple(torch.Tensor, torch.Tensor), None): point coordinates and labels to embed.
      boxes (torch.Tensor, None): boxes to embed
      masks (torch.Tensor, None): masks to embed

    Returns:
      torch.Tensor: sparse embeddings for the points and boxes, with shape BxNx(embed_dim), where N is determined
        by the number of input points and boxes.
      torch.Tensor: dense embeddings for the masks, in the shape Bx(embed_dim)x(embed_H)x(embed_W)
    """
    bs = self._get_batch_size(points, boxes, masks)
    sparse_embeddings = torch.empty((bs, 0, self.embed_dim), device=self._get_device())
    if points is not None:
        coords, labels = points
        point_embeddings = self._embed_points(coords, labels, pad=(boxes is None))
        sparse_embeddings = torch.cat([sparse_embeddings, point_embeddings], dim=1)
    if boxes is not None:
        box_embeddings = self._embed_boxes(boxes)
        sparse_embeddings = torch.cat([sparse_embeddings, box_embeddings], dim=1)

    if masks is not None:
        dense_embeddings = self._embed_masks(masks)
    else:
        dense_embeddings = self.no_mask_embed.weight.reshape(1, -1, 1, 1).expand(
            bs, -1, self.image_embedding_size[0], self.image_embedding_size[1]
        )

    return sparse_embeddings, dense_embeddings

get_dense_pe()

포인트 ν”„λ‘¬ν”„νŠΈλ₯Ό μΈμ½”λ”©ν•˜λŠ” 데 μ‚¬μš©λ˜λŠ” μœ„μΉ˜ 인코딩을 λ°˜ν™˜ν•˜λ©°, μ‘°λ°€ν•œ 포인트 집합에 이미지 μΈμ½”λ”©μ˜ 이미지 μΈμ½”λ”©μ˜ λͺ¨μ–‘을 λ°˜ν™˜ν•©λ‹ˆλ‹€.

λ°˜ν™˜ν•©λ‹ˆλ‹€:

μœ ν˜• μ„€λͺ…
Tensor

torchTensor: λ„ν˜•μ„ μ‚¬μš©ν•œ μœ„μΉ˜ 인코딩 1x(embed_dim)x(embedding_h)x(embedding_w)

의 μ†ŒμŠ€ μ½”λ“œ ultralytics/models/sam/modules/encoders.py
def get_dense_pe(self) -> torch.Tensor:
    """
    Returns the positional encoding used to encode point prompts, applied to a dense set of points the shape of the
    image encoding.

    Returns:
      torch.Tensor: Positional encoding with shape 1x(embed_dim)x(embedding_h)x(embedding_w)
    """
    return self.pe_layer(self.image_embedding_size).unsqueeze(0)



ultralytics.models.sam.modules.encoders.PositionEmbeddingRandom

기지: Module

μž„μ˜μ˜ 곡간 주파수λ₯Ό μ‚¬μš©ν•œ μœ„μΉ˜ 인코딩.

의 μ†ŒμŠ€ μ½”λ“œ ultralytics/models/sam/modules/encoders.py
class PositionEmbeddingRandom(nn.Module):
    """Positional encoding using random spatial frequencies."""

    def __init__(self, num_pos_feats: int = 64, scale: Optional[float] = None) -> None:
        """Initializes a position embedding using random spatial frequencies."""
        super().__init__()
        if scale is None or scale <= 0.0:
            scale = 1.0
        self.register_buffer("positional_encoding_gaussian_matrix", scale * torch.randn((2, num_pos_feats)))

        # Set non-deterministic for forward() error 'cumsum_cuda_kernel does not have a deterministic implementation'
        torch.use_deterministic_algorithms(False)
        torch.backends.cudnn.deterministic = False

    def _pe_encoding(self, coords: torch.Tensor) -> torch.Tensor:
        """Positionally encode points that are normalized to [0,1]."""
        # Assuming coords are in [0, 1]^2 square and have d_1 x ... x d_n x 2 shape
        coords = 2 * coords - 1
        coords = coords @ self.positional_encoding_gaussian_matrix
        coords = 2 * np.pi * coords
        # Outputs d_1 x ... x d_n x C shape
        return torch.cat([torch.sin(coords), torch.cos(coords)], dim=-1)

    def forward(self, size: Tuple[int, int]) -> torch.Tensor:
        """Generate positional encoding for a grid of the specified size."""
        h, w = size
        device: Any = self.positional_encoding_gaussian_matrix.device
        grid = torch.ones((h, w), device=device, dtype=torch.float32)
        y_embed = grid.cumsum(dim=0) - 0.5
        x_embed = grid.cumsum(dim=1) - 0.5
        y_embed = y_embed / h
        x_embed = x_embed / w

        pe = self._pe_encoding(torch.stack([x_embed, y_embed], dim=-1))
        return pe.permute(2, 0, 1)  # C x H x W

    def forward_with_coords(self, coords_input: torch.Tensor, image_size: Tuple[int, int]) -> torch.Tensor:
        """Positionally encode points that are not normalized to [0,1]."""
        coords = coords_input.clone()
        coords[:, :, 0] = coords[:, :, 0] / image_size[1]
        coords[:, :, 1] = coords[:, :, 1] / image_size[0]
        return self._pe_encoding(coords.to(torch.float))  # B x N x C

__init__(num_pos_feats=64, scale=None)

μž„μ˜μ˜ 곡간 주파수λ₯Ό μ‚¬μš©ν•˜μ—¬ μœ„μΉ˜ μž„λ² λ”©μ„ μ΄ˆκΈ°ν™”ν•©λ‹ˆλ‹€.

의 μ†ŒμŠ€ μ½”λ“œ ultralytics/models/sam/modules/encoders.py
def __init__(self, num_pos_feats: int = 64, scale: Optional[float] = None) -> None:
    """Initializes a position embedding using random spatial frequencies."""
    super().__init__()
    if scale is None or scale <= 0.0:
        scale = 1.0
    self.register_buffer("positional_encoding_gaussian_matrix", scale * torch.randn((2, num_pos_feats)))

    # Set non-deterministic for forward() error 'cumsum_cuda_kernel does not have a deterministic implementation'
    torch.use_deterministic_algorithms(False)
    torch.backends.cudnn.deterministic = False

forward(size)

μ§€μ •λœ 크기의 κ·Έλ¦¬λ“œμ— λŒ€ν•œ μœ„μΉ˜ 인코딩을 μƒμ„±ν•©λ‹ˆλ‹€.

의 μ†ŒμŠ€ μ½”λ“œ ultralytics/models/sam/modules/encoders.py
def forward(self, size: Tuple[int, int]) -> torch.Tensor:
    """Generate positional encoding for a grid of the specified size."""
    h, w = size
    device: Any = self.positional_encoding_gaussian_matrix.device
    grid = torch.ones((h, w), device=device, dtype=torch.float32)
    y_embed = grid.cumsum(dim=0) - 0.5
    x_embed = grid.cumsum(dim=1) - 0.5
    y_embed = y_embed / h
    x_embed = x_embed / w

    pe = self._pe_encoding(torch.stack([x_embed, y_embed], dim=-1))
    return pe.permute(2, 0, 1)  # C x H x W

forward_with_coords(coords_input, image_size)

0,1]둜 μ •κ·œν™”λ˜μ§€ μ•Šμ€ 점을 μœ„μΉ˜ μΈμ½”λ”©ν•©λ‹ˆλ‹€.

의 μ†ŒμŠ€ μ½”λ“œ ultralytics/models/sam/modules/encoders.py
def forward_with_coords(self, coords_input: torch.Tensor, image_size: Tuple[int, int]) -> torch.Tensor:
    """Positionally encode points that are not normalized to [0,1]."""
    coords = coords_input.clone()
    coords[:, :, 0] = coords[:, :, 0] / image_size[1]
    coords[:, :, 1] = coords[:, :, 1] / image_size[0]
    return self._pe_encoding(coords.to(torch.float))  # B x N x C



ultralytics.models.sam.modules.encoders.Block

기지: Module

μ°½ 주의 및 μž”μ—¬ μ „νŒŒ 블둝을 μ§€μ›ν•˜λŠ” 트랜슀포머 블둝.

의 μ†ŒμŠ€ μ½”λ“œ ultralytics/models/sam/modules/encoders.py
class Block(nn.Module):
    """Transformer blocks with support of window attention and residual propagation blocks."""

    def __init__(
        self,
        dim: int,
        num_heads: int,
        mlp_ratio: float = 4.0,
        qkv_bias: bool = True,
        norm_layer: Type[nn.Module] = nn.LayerNorm,
        act_layer: Type[nn.Module] = nn.GELU,
        use_rel_pos: bool = False,
        rel_pos_zero_init: bool = True,
        window_size: int = 0,
        input_size: Optional[Tuple[int, int]] = None,
    ) -> None:
        """
        Args:
            dim (int): Number of input channels.
            num_heads (int): Number of attention heads in each ViT block.
            mlp_ratio (float): Ratio of mlp hidden dim to embedding dim.
            qkv_bias (bool): If True, add a learnable bias to query, key, value.
            norm_layer (nn.Module): Normalization layer.
            act_layer (nn.Module): Activation layer.
            use_rel_pos (bool): If True, add relative positional embeddings to the attention map.
            rel_pos_zero_init (bool): If True, zero initialize relative positional parameters.
            window_size (int): Window size for window attention blocks. If it equals 0, then
                use global attention.
            input_size (tuple(int, int), None): Input resolution for calculating the relative
                positional parameter size.
        """
        super().__init__()
        self.norm1 = norm_layer(dim)
        self.attn = Attention(
            dim,
            num_heads=num_heads,
            qkv_bias=qkv_bias,
            use_rel_pos=use_rel_pos,
            rel_pos_zero_init=rel_pos_zero_init,
            input_size=input_size if window_size == 0 else (window_size, window_size),
        )

        self.norm2 = norm_layer(dim)
        self.mlp = MLPBlock(embedding_dim=dim, mlp_dim=int(dim * mlp_ratio), act=act_layer)

        self.window_size = window_size

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        """Executes a forward pass through the transformer block with window attention and non-overlapping windows."""
        shortcut = x
        x = self.norm1(x)
        # Window partition
        if self.window_size > 0:
            H, W = x.shape[1], x.shape[2]
            x, pad_hw = window_partition(x, self.window_size)

        x = self.attn(x)
        # Reverse window partition
        if self.window_size > 0:
            x = window_unpartition(x, self.window_size, pad_hw, (H, W))

        x = shortcut + x
        return x + self.mlp(self.norm2(x))

__init__(dim, num_heads, mlp_ratio=4.0, qkv_bias=True, norm_layer=nn.LayerNorm, act_layer=nn.GELU, use_rel_pos=False, rel_pos_zero_init=True, window_size=0, input_size=None)

λ§€κ°œλ³€μˆ˜:

이름 μœ ν˜• μ„€λͺ… κΈ°λ³Έκ°’
dim int

μž…λ ₯ 채널 μˆ˜μž…λ‹ˆλ‹€.

ν•„μˆ˜
num_heads int

각 ViT λΈ”λ‘μ˜ 관심 ν—€λ“œ μˆ˜μž…λ‹ˆλ‹€.

ν•„μˆ˜
mlp_ratio float

MLP μˆ¨κΉ€ μ–΄λ‘‘κ²Œμ™€ μž„λ² λ”© μ–΄λ‘‘κ²Œμ˜ λΉ„μœ¨μž…λ‹ˆλ‹€.

4.0
qkv_bias bool

True인 경우 쿼리, ν‚€, 값에 ν•™μŠ΅ κ°€λŠ₯ν•œ 편ν–₯을 μΆ”κ°€ν•©λ‹ˆλ‹€.

True
norm_layer Module

μ •κ·œν™” λ ˆμ΄μ–΄μž…λ‹ˆλ‹€.

LayerNorm
act_layer Module

ν™œμ„±ν™” λ ˆμ΄μ–΄.

GELU
use_rel_pos bool

True인 경우 관심도 맡에 μƒλŒ€μ  μœ„μΉ˜ μž„λ² λ”©μ„ μΆ”κ°€ν•©λ‹ˆλ‹€.

False
rel_pos_zero_init bool

True이면 μƒλŒ€ μœ„μΉ˜ 맀개 λ³€μˆ˜λ₯Ό 0으둜 μ΄ˆκΈ°ν™”ν•©λ‹ˆλ‹€.

True
window_size int

μ°½ 주의 λΈ”λ‘μ˜ μ°½ ν¬κΈ°μž…λ‹ˆλ‹€. 0κ³Ό κ°™μœΌλ©΄ κΈ€λ‘œλ²Œ 주의λ₯Ό μ‚¬μš©ν•©λ‹ˆλ‹€.

0
input_size (tuple(int, int), None)

μƒλŒ€μ  μœ„μΉ˜ λ§€κ°œλ³€μˆ˜ 크기λ₯Ό κ³„μ‚°ν•˜κΈ° μœ„ν•œ μž…λ ₯ 해상도 μœ„μΉ˜ λ§€κ°œλ³€μˆ˜ 크기λ₯Ό κ³„μ‚°ν•˜κΈ° μœ„ν•œ μž…λ ₯ ν•΄μƒλ„μž…λ‹ˆλ‹€.

None
의 μ†ŒμŠ€ μ½”λ“œ ultralytics/models/sam/modules/encoders.py
def __init__(
    self,
    dim: int,
    num_heads: int,
    mlp_ratio: float = 4.0,
    qkv_bias: bool = True,
    norm_layer: Type[nn.Module] = nn.LayerNorm,
    act_layer: Type[nn.Module] = nn.GELU,
    use_rel_pos: bool = False,
    rel_pos_zero_init: bool = True,
    window_size: int = 0,
    input_size: Optional[Tuple[int, int]] = None,
) -> None:
    """
    Args:
        dim (int): Number of input channels.
        num_heads (int): Number of attention heads in each ViT block.
        mlp_ratio (float): Ratio of mlp hidden dim to embedding dim.
        qkv_bias (bool): If True, add a learnable bias to query, key, value.
        norm_layer (nn.Module): Normalization layer.
        act_layer (nn.Module): Activation layer.
        use_rel_pos (bool): If True, add relative positional embeddings to the attention map.
        rel_pos_zero_init (bool): If True, zero initialize relative positional parameters.
        window_size (int): Window size for window attention blocks. If it equals 0, then
            use global attention.
        input_size (tuple(int, int), None): Input resolution for calculating the relative
            positional parameter size.
    """
    super().__init__()
    self.norm1 = norm_layer(dim)
    self.attn = Attention(
        dim,
        num_heads=num_heads,
        qkv_bias=qkv_bias,
        use_rel_pos=use_rel_pos,
        rel_pos_zero_init=rel_pos_zero_init,
        input_size=input_size if window_size == 0 else (window_size, window_size),
    )

    self.norm2 = norm_layer(dim)
    self.mlp = MLPBlock(embedding_dim=dim, mlp_dim=int(dim * mlp_ratio), act=act_layer)

    self.window_size = window_size

forward(x)

μ°½ 주의 및 κ²ΉμΉ˜μ§€ μ•ŠλŠ” 창을 μ‚¬μš©ν•˜μ—¬ 트랜슀포머 블둝을 ν†΅κ³Όν•˜λŠ” ν¬μ›Œλ“œ 패슀λ₯Ό μ‹€ν–‰ν•©λ‹ˆλ‹€.

의 μ†ŒμŠ€ μ½”λ“œ ultralytics/models/sam/modules/encoders.py
def forward(self, x: torch.Tensor) -> torch.Tensor:
    """Executes a forward pass through the transformer block with window attention and non-overlapping windows."""
    shortcut = x
    x = self.norm1(x)
    # Window partition
    if self.window_size > 0:
        H, W = x.shape[1], x.shape[2]
        x, pad_hw = window_partition(x, self.window_size)

    x = self.attn(x)
    # Reverse window partition
    if self.window_size > 0:
        x = window_unpartition(x, self.window_size, pad_hw, (H, W))

    x = shortcut + x
    return x + self.mlp(self.norm2(x))



ultralytics.models.sam.modules.encoders.Attention

기지: Module

μƒλŒ€ μœ„μΉ˜ μž„λ² λ”© κΈ°λŠ₯이 μžˆλŠ” λ©€ν‹° ν—€λ“œ 주의 블둝.

의 μ†ŒμŠ€ μ½”λ“œ ultralytics/models/sam/modules/encoders.py
class Attention(nn.Module):
    """Multi-head Attention block with relative position embeddings."""

    def __init__(
        self,
        dim: int,
        num_heads: int = 8,
        qkv_bias: bool = True,
        use_rel_pos: bool = False,
        rel_pos_zero_init: bool = True,
        input_size: Optional[Tuple[int, int]] = None,
    ) -> None:
        """
        Initialize Attention module.

        Args:
            dim (int): Number of input channels.
            num_heads (int): Number of attention heads.
            qkv_bias (bool):  If True, add a learnable bias to query, key, value.
            rel_pos_zero_init (bool): If True, zero initialize relative positional parameters.
            input_size (tuple(int, int), None): Input resolution for calculating the relative
                positional parameter size.
        """
        super().__init__()
        self.num_heads = num_heads
        head_dim = dim // num_heads
        self.scale = head_dim**-0.5

        self.qkv = nn.Linear(dim, dim * 3, bias=qkv_bias)
        self.proj = nn.Linear(dim, dim)

        self.use_rel_pos = use_rel_pos
        if self.use_rel_pos:
            assert input_size is not None, "Input size must be provided if using relative positional encoding."
            # Initialize relative positional embeddings
            self.rel_pos_h = nn.Parameter(torch.zeros(2 * input_size[0] - 1, head_dim))
            self.rel_pos_w = nn.Parameter(torch.zeros(2 * input_size[1] - 1, head_dim))

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        """Applies the forward operation including attention, normalization, MLP, and indexing within window limits."""
        B, H, W, _ = x.shape
        # qkv with shape (3, B, nHead, H * W, C)
        qkv = self.qkv(x).reshape(B, H * W, 3, self.num_heads, -1).permute(2, 0, 3, 1, 4)
        # q, k, v with shape (B * nHead, H * W, C)
        q, k, v = qkv.reshape(3, B * self.num_heads, H * W, -1).unbind(0)

        attn = (q * self.scale) @ k.transpose(-2, -1)

        if self.use_rel_pos:
            attn = add_decomposed_rel_pos(attn, q, self.rel_pos_h, self.rel_pos_w, (H, W), (H, W))

        attn = attn.softmax(dim=-1)
        x = (attn @ v).view(B, self.num_heads, H, W, -1).permute(0, 2, 3, 1, 4).reshape(B, H, W, -1)
        return self.proj(x)

__init__(dim, num_heads=8, qkv_bias=True, use_rel_pos=False, rel_pos_zero_init=True, input_size=None)

주의 λͺ¨λ“ˆμ„ μ΄ˆκΈ°ν™”ν•©λ‹ˆλ‹€.

λ§€κ°œλ³€μˆ˜:

이름 μœ ν˜• μ„€λͺ… κΈ°λ³Έκ°’
dim int

μž…λ ₯ 채널 μˆ˜μž…λ‹ˆλ‹€.

ν•„μˆ˜
num_heads int

관심 ν—€λ“œ μˆ˜μž…λ‹ˆλ‹€.

8
qkv_bias bool

True인 경우 쿼리, ν‚€, 값에 ν•™μŠ΅ κ°€λŠ₯ν•œ 편ν–₯을 μΆ”κ°€ν•©λ‹ˆλ‹€.

True
rel_pos_zero_init bool

True이면 μƒλŒ€ μœ„μΉ˜ 맀개 λ³€μˆ˜λ₯Ό 0으둜 μ΄ˆκΈ°ν™”ν•©λ‹ˆλ‹€.

True
input_size (tuple(int, int), None)

μƒλŒ€μ  μœ„μΉ˜ λ§€κ°œλ³€μˆ˜ 크기λ₯Ό κ³„μ‚°ν•˜κΈ° μœ„ν•œ μž…λ ₯ 해상도 μœ„μΉ˜ λ§€κ°œλ³€μˆ˜ 크기λ₯Ό κ³„μ‚°ν•˜κΈ° μœ„ν•œ μž…λ ₯ ν•΄μƒλ„μž…λ‹ˆλ‹€.

None
의 μ†ŒμŠ€ μ½”λ“œ ultralytics/models/sam/modules/encoders.py
def __init__(
    self,
    dim: int,
    num_heads: int = 8,
    qkv_bias: bool = True,
    use_rel_pos: bool = False,
    rel_pos_zero_init: bool = True,
    input_size: Optional[Tuple[int, int]] = None,
) -> None:
    """
    Initialize Attention module.

    Args:
        dim (int): Number of input channels.
        num_heads (int): Number of attention heads.
        qkv_bias (bool):  If True, add a learnable bias to query, key, value.
        rel_pos_zero_init (bool): If True, zero initialize relative positional parameters.
        input_size (tuple(int, int), None): Input resolution for calculating the relative
            positional parameter size.
    """
    super().__init__()
    self.num_heads = num_heads
    head_dim = dim // num_heads
    self.scale = head_dim**-0.5

    self.qkv = nn.Linear(dim, dim * 3, bias=qkv_bias)
    self.proj = nn.Linear(dim, dim)

    self.use_rel_pos = use_rel_pos
    if self.use_rel_pos:
        assert input_size is not None, "Input size must be provided if using relative positional encoding."
        # Initialize relative positional embeddings
        self.rel_pos_h = nn.Parameter(torch.zeros(2 * input_size[0] - 1, head_dim))
        self.rel_pos_w = nn.Parameter(torch.zeros(2 * input_size[1] - 1, head_dim))

forward(x)

μœˆλ„μš° μ œν•œ λ‚΄μ—μ„œ 주의, μ •κ·œν™”, MLP, 인덱싱을 ν¬ν•¨ν•œ μ •λ°©ν–₯ 연산을 μ μš©ν•©λ‹ˆλ‹€.

의 μ†ŒμŠ€ μ½”λ“œ ultralytics/models/sam/modules/encoders.py
def forward(self, x: torch.Tensor) -> torch.Tensor:
    """Applies the forward operation including attention, normalization, MLP, and indexing within window limits."""
    B, H, W, _ = x.shape
    # qkv with shape (3, B, nHead, H * W, C)
    qkv = self.qkv(x).reshape(B, H * W, 3, self.num_heads, -1).permute(2, 0, 3, 1, 4)
    # q, k, v with shape (B * nHead, H * W, C)
    q, k, v = qkv.reshape(3, B * self.num_heads, H * W, -1).unbind(0)

    attn = (q * self.scale) @ k.transpose(-2, -1)

    if self.use_rel_pos:
        attn = add_decomposed_rel_pos(attn, q, self.rel_pos_h, self.rel_pos_w, (H, W), (H, W))

    attn = attn.softmax(dim=-1)
    x = (attn @ v).view(B, self.num_heads, H, W, -1).permute(0, 2, 3, 1, 4).reshape(B, H, W, -1)
    return self.proj(x)



ultralytics.models.sam.modules.encoders.PatchEmbed

기지: Module

패치 μž„λ² λ“œν•  이미지.

의 μ†ŒμŠ€ μ½”λ“œ ultralytics/models/sam/modules/encoders.py
class PatchEmbed(nn.Module):
    """Image to Patch Embedding."""

    def __init__(
        self,
        kernel_size: Tuple[int, int] = (16, 16),
        stride: Tuple[int, int] = (16, 16),
        padding: Tuple[int, int] = (0, 0),
        in_chans: int = 3,
        embed_dim: int = 768,
    ) -> None:
        """
        Initialize PatchEmbed module.

        Args:
            kernel_size (Tuple): kernel size of the projection layer.
            stride (Tuple): stride of the projection layer.
            padding (Tuple): padding size of the projection layer.
            in_chans (int): Number of input image channels.
            embed_dim (int): Patch embedding dimension.
        """
        super().__init__()

        self.proj = nn.Conv2d(in_chans, embed_dim, kernel_size=kernel_size, stride=stride, padding=padding)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        """Computes patch embedding by applying convolution and transposing resulting tensor."""
        return self.proj(x).permute(0, 2, 3, 1)  # B C H W -> B H W C

__init__(kernel_size=(16, 16), stride=(16, 16), padding=(0, 0), in_chans=3, embed_dim=768)

PatchEmbed λͺ¨λ“ˆμ„ μ΄ˆκΈ°ν™”ν•©λ‹ˆλ‹€.

λ§€κ°œλ³€μˆ˜:

이름 μœ ν˜• μ„€λͺ… κΈ°λ³Έκ°’
kernel_size Tuple

투영 λ ˆμ΄μ–΄μ˜ 컀널 ν¬κΈ°μž…λ‹ˆλ‹€.

(16, 16)
stride Tuple

투영 λ ˆμ΄μ–΄μ˜ λ³΄ν­μž…λ‹ˆλ‹€.

(16, 16)
padding Tuple

투영 λ ˆμ΄μ–΄μ˜ νŒ¨λ”© ν¬κΈ°μž…λ‹ˆλ‹€.

(0, 0)
in_chans int

μž…λ ₯ 이미지 채널 μˆ˜μž…λ‹ˆλ‹€.

3
embed_dim int

패치 μž„λ² λ”© 차원.

768
의 μ†ŒμŠ€ μ½”λ“œ ultralytics/models/sam/modules/encoders.py
def __init__(
    self,
    kernel_size: Tuple[int, int] = (16, 16),
    stride: Tuple[int, int] = (16, 16),
    padding: Tuple[int, int] = (0, 0),
    in_chans: int = 3,
    embed_dim: int = 768,
) -> None:
    """
    Initialize PatchEmbed module.

    Args:
        kernel_size (Tuple): kernel size of the projection layer.
        stride (Tuple): stride of the projection layer.
        padding (Tuple): padding size of the projection layer.
        in_chans (int): Number of input image channels.
        embed_dim (int): Patch embedding dimension.
    """
    super().__init__()

    self.proj = nn.Conv2d(in_chans, embed_dim, kernel_size=kernel_size, stride=stride, padding=padding)

forward(x)

μ»¨λ³Όλ£¨μ…˜μ„ μ μš©ν•˜μ—¬ 패치 μž„λ² λ”©μ„ κ³„μ‚°ν•˜κ³  κ·Έ κ²°κ³Ό tensor.

의 μ†ŒμŠ€ μ½”λ“œ ultralytics/models/sam/modules/encoders.py
def forward(self, x: torch.Tensor) -> torch.Tensor:
    """Computes patch embedding by applying convolution and transposing resulting tensor."""
    return self.proj(x).permute(0, 2, 3, 1)  # B C H W -> B H W C



ultralytics.models.sam.modules.encoders.window_partition(x, window_size)

ν•„μš”ν•œ 경우 νŒ¨λ”©μ„ μ‚¬μš©ν•˜μ—¬ κ²ΉμΉ˜μ§€ μ•ŠλŠ” 창으둜 λΆ„ν• ν•©λ‹ˆλ‹€. Args: x (tensor): [B, H, W, C]κ°€ ν¬ν•¨λœ μž…λ ₯ 토큰. window_size (int): μ°½ 크기.

λ°˜ν™˜ν•©λ‹ˆλ‹€:

이름 μœ ν˜• μ„€λͺ…
windows Tensor

B * num_windows, window_size, window_size, C]둜 νŒŒν‹°μ…˜ν•œ ν›„ 창을 λ§Œλ“­λ‹ˆλ‹€.

(Hp, Wp)

νŒŒν‹°μ…˜ μ „ νŒ¨λ”©λœ 높이와 λ„ˆλΉ„

의 μ†ŒμŠ€ μ½”λ“œ ultralytics/models/sam/modules/encoders.py
def window_partition(x: torch.Tensor, window_size: int) -> Tuple[torch.Tensor, Tuple[int, int]]:
    """
    Partition into non-overlapping windows with padding if needed.
    Args:
        x (tensor): input tokens with [B, H, W, C].
        window_size (int): window size.

    Returns:
        windows: windows after partition with [B * num_windows, window_size, window_size, C].
        (Hp, Wp): padded height and width before partition
    """
    B, H, W, C = x.shape

    pad_h = (window_size - H % window_size) % window_size
    pad_w = (window_size - W % window_size) % window_size
    if pad_h > 0 or pad_w > 0:
        x = F.pad(x, (0, 0, 0, pad_w, 0, pad_h))
    Hp, Wp = H + pad_h, W + pad_w

    x = x.view(B, Hp // window_size, window_size, Wp // window_size, window_size, C)
    windows = x.permute(0, 1, 3, 2, 4, 5).contiguous().view(-1, window_size, window_size, C)
    return windows, (Hp, Wp)



ultralytics.models.sam.modules.encoders.window_unpartition(windows, window_size, pad_hw, hw)

창을 μ›λž˜ μ‹œν€€μŠ€λ‘œ λΆ„ν•  ν•΄μ œν•˜κ³  νŒ¨λ”©μ„ μ œκ±°ν•©λ‹ˆλ‹€.

λ§€κ°œλ³€μˆ˜:

이름 μœ ν˜• μ„€λͺ… κΈ°λ³Έκ°’
windows tensor

B * num_windows, window_size, window_size, C]둜 토큰을 μž…λ ₯ν•©λ‹ˆλ‹€.

ν•„μˆ˜
window_size int

창 크기.

ν•„μˆ˜
pad_hw Tuple

νŒ¨λ”©λœ 높이와 λ„ˆλΉ„(Hp, Wp).

ν•„μˆ˜
hw Tuple

νŒ¨λ”© μ „μ˜ μ›λž˜ 높이와 λ„ˆλΉ„(H, W)λ₯Ό μž…λ ₯ν•©λ‹ˆλ‹€.

ν•„μˆ˜

λ°˜ν™˜ν•©λ‹ˆλ‹€:

이름 μœ ν˜• μ„€λͺ…
x Tensor

λΆ„ν• λ˜μ§€ μ•Šμ€ μ‹œν€€μŠ€ [B, H, W, C].

의 μ†ŒμŠ€ μ½”λ“œ ultralytics/models/sam/modules/encoders.py
def window_unpartition(
    windows: torch.Tensor, window_size: int, pad_hw: Tuple[int, int], hw: Tuple[int, int]
) -> torch.Tensor:
    """
    Window unpartition into original sequences and removing padding.

    Args:
        windows (tensor): input tokens with [B * num_windows, window_size, window_size, C].
        window_size (int): window size.
        pad_hw (Tuple): padded height and width (Hp, Wp).
        hw (Tuple): original height and width (H, W) before padding.

    Returns:
        x: unpartitioned sequences with [B, H, W, C].
    """
    Hp, Wp = pad_hw
    H, W = hw
    B = windows.shape[0] // (Hp * Wp // window_size // window_size)
    x = windows.view(B, Hp // window_size, Wp // window_size, window_size, window_size, -1)
    x = x.permute(0, 1, 3, 2, 4, 5).contiguous().view(B, Hp, Wp, -1)

    if Hp > H or Wp > W:
        x = x[:, :H, :W, :].contiguous()
    return x



ultralytics.models.sam.modules.encoders.get_rel_pos(q_size, k_size, rel_pos)

쿼리 및 ν‚€ 크기의 μƒλŒ€μ  μœ„μΉ˜μ— 따라 μƒλŒ€μ  μœ„μΉ˜ μž„λ² λ”©μ„ κ°€μ Έμ˜΅λ‹ˆλ‹€.

λ§€κ°œλ³€μˆ˜:

이름 μœ ν˜• μ„€λͺ… κΈ°λ³Έκ°’
q_size int

쿼리 크기 Q.

ν•„μˆ˜
k_size int

ν‚€ K의 크기.

ν•„μˆ˜
rel_pos Tensor

μƒλŒ€ μœ„μΉ˜ μž„λ² λ”©(L, C).

ν•„μˆ˜

λ°˜ν™˜ν•©λ‹ˆλ‹€:

μœ ν˜• μ„€λͺ…
Tensor

μƒλŒ€μ  μœ„μΉ˜μ— 따라 μœ„μΉ˜ μž„λ² λ”©μ„ μΆ”μΆœν•©λ‹ˆλ‹€.

의 μ†ŒμŠ€ μ½”λ“œ ultralytics/models/sam/modules/encoders.py
def get_rel_pos(q_size: int, k_size: int, rel_pos: torch.Tensor) -> torch.Tensor:
    """
    Get relative positional embeddings according to the relative positions of query and key sizes.

    Args:
        q_size (int): size of query q.
        k_size (int): size of key k.
        rel_pos (Tensor): relative position embeddings (L, C).

    Returns:
        Extracted positional embeddings according to relative positions.
    """
    max_rel_dist = int(2 * max(q_size, k_size) - 1)
    # Interpolate rel pos if needed.
    if rel_pos.shape[0] != max_rel_dist:
        # Interpolate rel pos.
        rel_pos_resized = F.interpolate(
            rel_pos.reshape(1, rel_pos.shape[0], -1).permute(0, 2, 1),
            size=max_rel_dist,
            mode="linear",
        )
        rel_pos_resized = rel_pos_resized.reshape(-1, max_rel_dist).permute(1, 0)
    else:
        rel_pos_resized = rel_pos

    # Scale the coords with short length if shapes for q and k are different.
    q_coords = torch.arange(q_size)[:, None] * max(k_size / q_size, 1.0)
    k_coords = torch.arange(k_size)[None, :] * max(q_size / k_size, 1.0)
    relative_coords = (q_coords - k_coords) + (k_size - 1) * max(q_size / k_size, 1.0)

    return rel_pos_resized[relative_coords.long()]



ultralytics.models.sam.modules.encoders.add_decomposed_rel_pos(attn, q, rel_pos_h, rel_pos_w, q_size, k_size)

mvitv2 λ…Όλ¬Έμ—μ„œ λΆ„ν•΄λœ μƒλŒ€ μœ„μΉ˜ μž„λ² λ”©μ„ κ³„μ‚°ν•˜μ„Έμš”. https://github.com/facebookresearch/mvit/blob/main/mvit/models/attention.py.

λ§€κ°œλ³€μˆ˜:

이름 μœ ν˜• μ„€λͺ… κΈ°λ³Έκ°’
attn Tensor

주의 지도.

ν•„μˆ˜
q Tensor

관심 λ ˆμ΄μ–΄μ—μ„œ λͺ¨μ–‘(B, q_h * q_w, C)을 μ‚¬μš©ν•˜μ—¬ qλ₯Ό μΏΌλ¦¬ν•©λ‹ˆλ‹€.

ν•„μˆ˜
rel_pos_h Tensor

높이 좕에 λŒ€ν•œ μƒλŒ€ μœ„μΉ˜ μž„λ² λ”©(Lh, C).

ν•„μˆ˜
rel_pos_w Tensor

λ„ˆλΉ„ 좕에 λŒ€ν•œ μƒλŒ€ μœ„μΉ˜ μž„λ² λ”©(Lw, C).

ν•„μˆ˜
q_size Tuple

쿼리 Q의 곡간 μ‹œν€€μŠ€ 크기 (Q_H, Q_W)λ₯Ό μ‚¬μš©ν•©λ‹ˆλ‹€.

ν•„μˆ˜
k_size Tuple

ν‚€ K의 곡간 μ‹œν€€μŠ€ 크기 (K_H, K_W).

ν•„μˆ˜

λ°˜ν™˜ν•©λ‹ˆλ‹€:

이름 μœ ν˜• μ„€λͺ…
attn Tensor

관심도 맡에 μƒλŒ€μ  μœ„μΉ˜ μž„λ² λ”©μ΄ μΆ”κ°€λ˜μ—ˆμŠ΅λ‹ˆλ‹€.

의 μ†ŒμŠ€ μ½”λ“œ ultralytics/models/sam/modules/encoders.py
def add_decomposed_rel_pos(
    attn: torch.Tensor,
    q: torch.Tensor,
    rel_pos_h: torch.Tensor,
    rel_pos_w: torch.Tensor,
    q_size: Tuple[int, int],
    k_size: Tuple[int, int],
) -> torch.Tensor:
    """
    Calculate decomposed Relative Positional Embeddings from mvitv2 paper at
    https://github.com/facebookresearch/mvit/blob/main/mvit/models/attention.py.

    Args:
        attn (Tensor): attention map.
        q (Tensor): query q in the attention layer with shape (B, q_h * q_w, C).
        rel_pos_h (Tensor): relative position embeddings (Lh, C) for height axis.
        rel_pos_w (Tensor): relative position embeddings (Lw, C) for width axis.
        q_size (Tuple): spatial sequence size of query q with (q_h, q_w).
        k_size (Tuple): spatial sequence size of key k with (k_h, k_w).

    Returns:
        attn (Tensor): attention map with added relative positional embeddings.
    """
    q_h, q_w = q_size
    k_h, k_w = k_size
    Rh = get_rel_pos(q_h, k_h, rel_pos_h)
    Rw = get_rel_pos(q_w, k_w, rel_pos_w)

    B, _, dim = q.shape
    r_q = q.reshape(B, q_h, q_w, dim)
    rel_h = torch.einsum("bhwc,hkc->bhwk", r_q, Rh)
    rel_w = torch.einsum("bhwc,wkc->bhwk", r_q, Rw)

    attn = (attn.view(B, q_h, q_w, k_h, k_w) + rel_h[:, :, :, :, None] + rel_w[:, :, :, None, :]).view(
        B, q_h * q_w, k_h * k_w
    )

    return attn





2023-11-12 생성, 2023-11-25 μ—…λ°μ΄νŠΈλ¨
μž‘μ„±μž: glenn-jocher (3), Laughing-q (1)