ROSïŒããããã»ãªãã¬ãŒãã£ã³ã°ã»ã·ã¹ãã ïŒã¯ã€ãã¯ã¹ã¿ãŒãã¬ã€ã
ROS Introduction (captioned)fromOpen RoboticsonVimeo.
ROSãšã¯äœãïŒ
ããããã»ãªãã¬ãŒãã£ã³ã°ã»ã·ã¹ãã ïŒROSïŒã¯ãããããå·¥åŠã®ç 究ãç£æ¥çã§åºã䜿ãããŠãããªãŒãã³ãœãŒã¹ã®ãã¬ãŒã ã¯ãŒã¯ã§ãããROSã¯ãéçºè ãããããã¢ããªã±ãŒã·ã§ã³ãäœæããã®ã«åœ¹ç«ã€ã©ã€ãã©ãªãããŒã«ã®ã³ã¬ã¯ã·ã§ã³ãæäŸããŸããROSã¯æ§ã ãªãããããã©ãããã©ãŒã ã§åäœããããã«èšèšãããŠãããããããéçºè ã«ãšã£ãŠæè»ã§åŒ·åãªããŒã«ãšãªã£ãŠããŸãã
ROSã®äž»ãªç¹åŸŽ
-
ã¢ãžã¥ã©ãŒã»ã¢ãŒããã¯ãã£ïŒROSã¯ã¢ãžã¥ã©ãŒã»ã¢ãŒããã¯ãã£ãŒãæ¡çšããŠãããéçºè ã¯ããŒããšåŒã°ããåå©çšå¯èœãªå°ããªã³ã³ããŒãã³ããçµã¿åãããããšã§ãè€éãªã·ã¹ãã ãæ§ç¯ããããšãã§ãããåããŒãã¯éåžžãç¹å®ã®æ©èœãå®è¡ããããŒãã¯ãããã¯ãŸãã¯ãµãŒãã¹ãä»ããŠã¡ãã»ãŒãžã䜿çšããŠçžäºã«éä¿¡ããŸãã
-
éä¿¡ããã«ãŠã§ã¢ïŒROSã¯ãããã»ã¹ééä¿¡ãšåæ£ã³ã³ãã¥ãŒãã£ã³ã°ããµããŒãããå ç¢ãªéä¿¡ã€ã³ãã©ã¹ãã©ã¯ãã£ãæäŸããŸããããã¯ãããŒã¿ã¹ããªãŒã ïŒãããã¯ïŒã®ãããªãã·ã¥ã»ãµãã¹ã¯ã©ã€ãã¢ãã«ãšããµãŒãã¹ã³ãŒã«ã®ãªã¯ãšã¹ãã»ãªãã©ã€ã¢ãã«ã«ãã£ãŠå®çŸãããŸãã
-
ããŒããŠã§ã¢ã®æœè±¡åïŒROSã¯ããŒããŠã§ã¢ãæœè±¡åããã¬ã€ã€ãŒãæäŸããéçºè ã¯ããã€ã¹ã«ãšããããªãã³ãŒããæžãããšãã§ãããããã«ãããåãã³ãŒããç°ãªãããŒããŠã§ã¢ã»ã»ããã¢ããã§äœ¿çšããããšãã§ããçµ±åãå®éšã容æã«ãªããŸãã
-
ããŒã«ãšãŠãŒãã£ãªãã£ïŒROSã«ã¯ãå¯èŠåããããã°ãã·ãã¥ã¬ãŒã·ã§ã³ã®ããã®ããŒã«ããŠãŒãã£ãªãã£ãè±å¯ã«çšæãããŠãããäŸãã°ãRVizã¯ã»ã³ãµãŒããŒã¿ãããããã®ç¶æ æ å ±ãå¯èŠåããããã«äœ¿çšãããGazeboã¯ã¢ã«ãŽãªãºã ãããããèšèšããã¹ãããããã®åŒ·åãªã·ãã¥ã¬ãŒã·ã§ã³ç°å¢ãæäŸããŸãã
-
åºç¯ãªãšã³ã·ã¹ãã ïŒROSã®ãšã³ã·ã¹ãã ã¯åºå€§ã§ç¶ç¶çã«æé·ããŠãããããã²ãŒã·ã§ã³ããããã¥ã¬ãŒã·ã§ã³ãç¥èŠãªã©ãããŸããŸãªããããã¢ããªã±ãŒã·ã§ã³ã«å©çšå¯èœãªæ°å€ãã®ããã±ãŒãžããããŸããã³ãã¥ããã£ã¯ããããã®ããã±ãŒãžã®éçºãšã¡ã³ããã³ã¹ã«ç©æ¥µçã«è²¢ç®ããŠããŸãã
ROSããŒãžã§ã³ã®é²å
2007幎ã«éçºãããŠä»¥æ¥ãROSã¯è€æ°ã®ããŒãžã§ã³ãçµãŠé²åãããã®ãã³ã«ããããå·¥åŠã³ãã¥ããã£ã®é«ãŸãããŒãºã«å¯Ÿå¿ããããã®æ°æ©èœãæ¹è¯ãå°å ¥ãããŠãããROSã®éçºã¯ãäž»ã«2ã€ã®ã·ãªãŒãºã«åé¡ããããšãã§ããŸãïŒãã®ã¬ã€ãã§ã¯ãROS Noetic NinjemysãšããŠç¥ãããROS 1ã®Long Term SupportïŒLTSïŒããŒãžã§ã³ã«çŠç¹ãåœãŠãŠããŸãããã³ãŒãã¯ä»¥åã®ããŒãžã§ã³ã§ãåäœããã¯ãã§ãã
ROS1ãšROS2ã®æ¯èŒ
ROS 1ã¯ããããéçºã®ããã®åŒ·åºãªåºç€ãæäŸããããROS 2ã¯ãã®æ¬ ç¹ã«å¯ŸåŠããŠããïŒ
- ãªã¢ã«ã¿ã€ã æ§èœïŒãªã¢ã«ã¿ã€ã ã·ã¹ãã ãšæ±ºå®è«çåäœã®ãµããŒããåäžã
- ã»ãã¥ãªãã£æ§ã ãªç°å¢äžã§å®å šãã€ä¿¡é Œæ§ã®é«ãéçšãå®çŸãããããã»ãã¥ãªãã£æ©èœã匷åã
- ã¹ã±ãŒã©ããªãã£ïŒãã«ãããããã·ã¹ãã ã倧èŠæš¡ãªå±éãããè¯ããµããŒãã
- ã¯ãã¹ãã©ãããã©ãŒã 察å¿ïŒWindowsãmacOSãªã©ãLinux以å€ã®æ§ã ãªOSãšã®äºææ§ãæ¡å€§ã
- æè»ãªéä¿¡ïŒDDSã䜿çšããããšã§ãããæè»ã§å¹ççãªããã»ã¹ééä¿¡ãå®çŸã
ROSã¡ãã»ãŒãžãšãããã¯
ROSã§ã¯ãããŒãéã®éä¿¡ã¯ã¡ãã»ãŒãžãš ãããã¯ãéããŠè¡ããããã¡ãã»ãŒãžã¯ããŒãéã§äº€æãããæ å ±ãå®çŸ©ããããŒã¿æ§é ã§ããããã¯ã¯ã¡ãã»ãŒãžãéåä¿¡ãããååä»ããã£ãã«ã§ããããŒãã¯ãããã¯ã«ã¡ãã»ãŒãžããããªãã·ã¥ãããããããã¯ããã®ã¡ãã»ãŒãžããµãã¹ã¯ã©ã€ãããããšã§ãçžäºã«éä¿¡ããããšãã§ããŸãããã®ãããªãã·ã¥ã»ãµãã¹ã¯ã©ã€ãã¢ãã«ã«ãããããŒãéã®éåæéä¿¡ãšãã«ãããªã³ã°ãå¯èœã«ãªããããããã·ã¹ãã ã®åã»ã³ãµãŒãã¢ã¯ãã¥ãšãŒã¿ãŒã¯éåžžããããã¯ã«ããŒã¿ããããªãã·ã¥ãããããä»ã®ããŒããåŠçãå¶åŸ¡ã®ããã«æ¶è²»ããããšãã§ããŸãããã®ã¬ã€ãã§ã¯ãImageãDepthãPointCloud ã¡ãã»ãŒãžãšã«ã¡ã©ãããã¯ã«çŠç¹ãåœãŠãŸãã
Ultralytics YOLO ãROSã§èšå®ãã
ãã®ã¬ã€ãã¯ãROSbot ROSãªããžããªã®ãã©ãŒã¯ã§ãããã®ROSç°å¢ã䜿çšããŠãã¹ããããŠããŸãããã®ç°å¢ã«ã¯ãUltralytics YOLO ããã±ãŒãžãã»ããã¢ãããç°¡åã«ããDockerã³ã³ãããå æ¬çãªROSããã±ãŒãžãè¿ éãªãã¹ãã®ããã®Gazeboã¯ãŒã«ããå«ãŸããŠããŸããHusarion ROSbot 2 PROã§åäœããããã«èšèšãããŠããŸããæäŸãããã³ãŒãäŸã¯ãã·ãã¥ã¬ãŒã·ã§ã³ãšå®äžçã®äž¡æ¹ãå«ããããããROS Noetic/Melodicç°å¢ã§åäœããŸãã
äŸåé¢ä¿ã®ã€ã³ã¹ããŒã«
ROSç°å¢ãšã¯å¥ã«ã以äžã®äŸåé¢ä¿ãã€ã³ã¹ããŒã«ããå¿ èŠããããŸãïŒ
-
ROS Numpy ããã±ãŒãž:ROS Imageã¡ãã»ãŒãžãšnumpyé åéã®é«éå€æã«å¿ èŠã§ãã
-
Ultralytics ããã±ãŒãžã§æäŸãããïŒ
ROSã§Ultralytics sensor_msgs/Image
ã«ã€ã㊠sensor_msgs/Image
ã¡ãã»ãŒãžã¿ã€ã ã¯ãç»åããŒã¿ãè¡šçŸããããã«ROSã§äžè¬çã«äœ¿çšãããŠããŸãããšã³ã³ãŒããé«ããå¹
ããã¯ã»ã«ããŒã¿ã®ãã£ãŒã«ããå«ãã§ãããã«ã¡ã©ãä»ã®ã»ã³ãµãŒã§æ®åœ±ãããç»åãéä¿¡ããã®ã«é©ããŠããŸããç»åã¡ãã»ãŒãžã¯ãèŠèŠèªèãªã©ã®ã¿ã¹ã¯ã®ããã®ããããã¢ããªã±ãŒã·ã§ã³ã§åºã䜿çšãããŠããŸãã ãªããžã§ã¯ãæ€åºãããŠããã²ãŒã·ã§ã³ã
ã€ã¡ãŒãžã»ã¹ãããã»ãã€ã»ã¹ããã
次ã®ã³ãŒãã»ã¹ããããã¯ãROS ã§Ultralytics YOLO ããã±ãŒãžã䜿çšããæ¹æ³ã瀺ããŠããŸãããã®äŸã§ã¯ãã«ã¡ã©ã»ãããã¯ããµãã¹ã¯ã©ã€ãããYOLO ã䜿çšããŠåä¿¡ç»åãåŠçããæ€åºããããªããžã§ã¯ããæ€åºãš ã»ã°ã¡ã³ããŒã·ã§ã³ã®ããã«æ°ãããããã¯ã«ãããªãã·ã¥ããŸãã
ãŸããå¿
èŠãªã©ã€ãã©ãªãã€ã³ããŒããã2ã€ã®ã¢ãã«ãã€ã³ã¹ã¿ã³ã¹åããã ã»ã°ã¡ã³ããŒã·ã§ã³ ãš æ€åº.ROSããŒãïŒåå㯠ultralytics
)ã䜿çšããŠROSãã¹ã¿ãŒãšã®éä¿¡ãå¯èœã«ãããå®å®ããæ¥ç¶ã確ä¿ããããã«ãçãäŒæ¢æéãèšããããŒããæ¥ç¶ã確ç«ããã®ã«ååãªæéãäžããŠããåŠçãé²ããŸãã
import time
import rospy
from ultralytics import YOLO
detection_model = YOLO("yolov8m.pt")
segmentation_model = YOLO("yolov8m-seg.pt")
rospy.init_node("ultralytics")
time.sleep(1)
2ã€ã®ROSãããã¯ãåæåããã æ€åº ãš ã»ã°ã¡ã³ããŒã·ã§ã³.ãããã®ãããã¯ã¯ã泚éä»ãç»åãå
¬éããããã«äœ¿çšããããããªãåŠçã®ããã«ã¢ã¯ã»ã¹ã§ããããã«ãããããŒãéã®é信㯠sensor_msgs/Image
ã®ã¡ãã»ãŒãžãããã
from sensor_msgs.msg import Image
det_image_pub = rospy.Publisher("/ultralytics/detection/image", Image, queue_size=5)
seg_image_pub = rospy.Publisher("/ultralytics/segmentation/image", Image, queue_size=5)
æåŸã«ã以äžã®ãµãã¹ã¯ã©ã€ããäœæããã /camera/color/image_raw
ãããã¯ãåãåããæ°ããã¡ãã»ãŒãžããšã«ã³ãŒã«ããã¯é¢æ°ãåŒã³åºããŸãããã®ã³ãŒã«ããã¯é¢æ°ã¯ sensor_msgs/Image
ã䜿ã£ãŠnumpyé
åã«å€æããã ros_numpy
ãããŠãäºåã«ã€ã³ã¹ã¿ã³ã¹åãããYOLO ã¢ãã«ã§ç»åãåŠçããç»åã«æ³šéãä»ããŠãããããããã®ãããã¯ã«æ»ãïŒ /ultralytics/detection/image
æ€åºãš /ultralytics/segmentation/image
ã»ã°ã¡ã³ããŒã·ã§ã³ã®ããã«ã
import ros_numpy
def callback(data):
"""Callback function to process image and publish annotated images."""
array = ros_numpy.numpify(data)
if det_image_pub.get_num_connections():
det_result = detection_model(array)
det_annotated = det_result[0].plot(show=False)
det_image_pub.publish(ros_numpy.msgify(Image, det_annotated, encoding="rgb8"))
if seg_image_pub.get_num_connections():
seg_result = segmentation_model(array)
seg_annotated = seg_result[0].plot(show=False)
seg_image_pub.publish(ros_numpy.msgify(Image, seg_annotated, encoding="rgb8"))
rospy.Subscriber("/camera/color/image_raw", Image, callback)
while True:
rospy.spin()
å®å šãªã³ãŒã
import time
import ros_numpy
import rospy
from sensor_msgs.msg import Image
from ultralytics import YOLO
detection_model = YOLO("yolov8m.pt")
segmentation_model = YOLO("yolov8m-seg.pt")
rospy.init_node("ultralytics")
time.sleep(1)
det_image_pub = rospy.Publisher("/ultralytics/detection/image", Image, queue_size=5)
seg_image_pub = rospy.Publisher("/ultralytics/segmentation/image", Image, queue_size=5)
def callback(data):
"""Callback function to process image and publish annotated images."""
array = ros_numpy.numpify(data)
if det_image_pub.get_num_connections():
det_result = detection_model(array)
det_annotated = det_result[0].plot(show=False)
det_image_pub.publish(ros_numpy.msgify(Image, det_annotated, encoding="rgb8"))
if seg_image_pub.get_num_connections():
seg_result = segmentation_model(array)
seg_annotated = seg_result[0].plot(show=False)
seg_image_pub.publish(ros_numpy.msgify(Image, seg_annotated, encoding="rgb8"))
rospy.Subscriber("/camera/color/image_raw", Image, callback)
while True:
rospy.spin()
ãããã°
ROSïŒããããã»ãªãã¬ãŒãã£ã³ã°ã»ã·ã¹ãã ïŒããŒãã®ãããã°ã¯ãã·ã¹ãã ãåæ£ããŠããããå°é£ãªå ŽåããããŸãããã®ããã»ã¹ãæ¯æŽããããŒã«ãããã€ããããŸãïŒ
rostopic echo <TOPIC-NAME>
:ãã®ã³ãã³ãã䜿ããšãç¹å®ã®ãããã¯ã§çºè¡šãããã¡ãã»ãŒãžãèŠãããšãã§ããããŒã¿ã®æµãã調ã¹ãã®ã«åœ¹ç«ã¡ãŸããrostopic list
:ãã®ã³ãã³ãã䜿çšãããšãROS ã·ã¹ãã ã§äœ¿çšå¯èœãªãã¹ãŠã®ãããã¯ãäžèŠ§è¡šç€ºãããã¢ã¯ãã£ããªããŒã¿ã¹ããªãŒã ã®æŠèŠã衚瀺ãããŸããrqt_graph
:ãã®å¯èŠåããŒã«ã¯ãããŒãéã®éä¿¡ã°ã©ãã衚瀺ããããŒããã©ã®ããã«çžäºæ¥ç¶ãããã©ã®ããã«çžäºäœçšããŠãããã«ã€ããŠã®æŽå¯ãæäŸããŸãã- 3Dè¡šçŸãªã©ãããè€éãªããžã¥ã¢ã©ã€ãŒãŒã·ã§ã³ã«ã¯ RViz.RViz (ROS Visualization)ã¯ãROSã®ããã®åŒ·åãª3Då¯èŠåããŒã«ã§ããããããã®ç¶æ
ãç°å¢ããªã¢ã«ã¿ã€ã ã§å¯èŠåããããšãã§ããŸããRVizã䜿çšãããšãã»ã³ãµãŒããŒã¿ïŒäŸ.
sensors_msgs/Image
)ãããããã¢ãã«ã®ç¶æ ããã®ä»æ§ã ãªæ å ±ãæäŸããããããã·ã¹ãã ã®ãããã°ãåäœã®ç解ã容æã«ããŸãã
æ€åºãããã¯ã©ã¹ã std_msgs/String
æšæºçãªROSã¡ãã»ãŒãžã«ã¯ä»¥äžãå«ãŸããã std_msgs/String
ã¡ãã»ãŒãžå€ãã®ã¢ããªã±ãŒã·ã§ã³ã§ã¯ã泚éä»ãç»åå
šäœãåå
¬éããå¿
èŠã¯ãªããããããã®ãã¥ãŒã«ååšããã¯ã©ã¹ã ããå¿
èŠã§ãã次ã®äŸã¯ std_msgs/String
ã¡ãã»ãŒãž ã§æ€åºãããã¯ã©ã¹ãåå
¬éããŸãã /ultralytics/detection/classes
ãããã¯ã§äœ¿çšãããããããã®ã¡ãã»ãŒãžã¯ãã軜éã§ãå¿
èŠäžå¯æ¬ ãªæ
å ±ãæäŸãããããããŸããŸãªçšéã§å©çšäŸ¡å€ãããã
䜿çšäŸ
ã«ã¡ã©ãšç©äœãè£
åããå庫ãããããèããŠã¿ããã æ€åºã¢ãã«.倧ããªæ³šéä»ãç»åããããã¯ãŒã¯çµç±ã§éä¿¡ãã代ããã«ãããããã¯æ€åºãããã¯ã©ã¹ã®ãªã¹ãã std_msgs/String
ã®ã¡ãã»ãŒãžãçæãããäŸãã°ãããããã "ç®±"ã"ãã¬ãã"ã"ãã©ãŒã¯ãªãã "ãšãã£ããªããžã§ã¯ããæ€åºãããšããããã®ã¯ã©ã¹ã /ultralytics/detection/classes
ã®ãããã¯ãåç
§ããŠãã ããããã®æ
å ±ã¯ãäžå€®ç£èŠã·ã¹ãã ããªã¢ã«ã¿ã€ã ã§åšåº«ã远跡ããããããããã®çµè·¯èšç»ãæé©åããŠé害ç©ãåé¿ããããæ€åºãããç®±ãããã¯ã¢ãããããªã©ã®ç¹å®ã®ã¢ã¯ã·ã§ã³ãããªã¬ãŒãããããããã«äœ¿çšã§ããããã®ã¢ãããŒãã«ãããéä¿¡ã«å¿
èŠãªåž¯åå¹
ãåæžãããéèŠãªããŒã¿ã®éä¿¡ã«éäžã§ããã
ã¹ããªã³ã°ã»ã¹ãããã»ãã€ã»ã¹ããã
ãã®äŸã§ã¯ãROS ã§Ultralytics YOLO ããã±ãŒãžã䜿çšããæ¹æ³ã瀺ããŸãããã®äŸã§ã¯ãã«ã¡ã©ã»ãããã¯ããµãã¹ã¯ã©ã€ãããYOLO ã䜿çšããŠåä¿¡ç»åãåŠçããæ€åºããããªããžã§ã¯ããæ°ããããã㯠/ultralytics/detection/classes
䜿çšã㊠std_msgs/String
ã¡ãã»ãŒãžãã® ros_numpy
ããã±ãŒãžã¯ãROS Imageã¡ãã»ãŒãžãYOLO ã§åŠçããããã®numpyé
åã«å€æããããã«äœ¿çšãããŸãã
import time
import ros_numpy
import rospy
from sensor_msgs.msg import Image
from std_msgs.msg import String
from ultralytics import YOLO
detection_model = YOLO("yolov8m.pt")
rospy.init_node("ultralytics")
time.sleep(1)
classes_pub = rospy.Publisher("/ultralytics/detection/classes", String, queue_size=5)
def callback(data):
"""Callback function to process image and publish detected classes."""
array = ros_numpy.numpify(data)
if classes_pub.get_num_connections():
det_result = detection_model(array)
classes = det_result[0].boxes.cls.cpu().numpy().astype(int)
names = [det_result[0].names[i] for i in classes]
classes_pub.publish(String(data=str(names)))
rospy.Subscriber("/camera/color/image_raw", Image, callback)
while True:
rospy.spin()
ROS深床ç»åã§Ultralytics
ROSã¯RGBç»åã«å ããŠæ·±åºŠç»åããµããŒãããŠãããã«ã¡ã©ããç©äœã®è·é¢ã«é¢ããæ å ±ãæäŸããã深床ç»åã¯ãé害ç©åé¿ã3Dãããã³ã°ãããŒã«ã©ã€ãŒãŒã·ã§ã³ãªã©ã®ããããã¢ããªã±ãŒã·ã§ã³ã«äžå¯æ¬ ã§ãã
深床ç»åã¯ãåãã¯ã»ã«ãã«ã¡ã©ããç©äœãŸã§ã®è·é¢ãè¡šãç»åã§ãããè²ãæããRGBç»åãšã¯ç°ãªãã深床ç»åã¯ç©ºéæ å ±ãæãããããããããã¯ç°å¢ã®3Dæ§é ãèªèããããšãã§ããã
深床ç»åã®ååŸ
深床ç»åã¯ãããŸããŸãªã»ã³ãµãŒã䜿ã£ãŠåŸãããšãã§ããïŒ
- ã¹ãã¬ãªã«ã¡ã©ïŒ2å°ã®ã«ã¡ã©ã䜿ã£ãŠãç»åã®èŠå·®ãã奥è¡ããèšç®ããã
- é£è¡æéïŒToFïŒã«ã¡ã©ïŒå ãç©äœããæ»ã£ãŠãããŸã§ã®æéã枬å®ããã
- æ§é åå ã»ã³ãµãŒïŒãã¿ãŒã³ãæ圱ããè¡šé¢äžã®å€åœ¢ã枬å®ããã
YOLO ã深床ç»åã䜿çšãã
ROSã§ã¯ã深床ç»å㯠sensor_msgs/Image
ã¡ãã»ãŒãžã¿ã€ãã«ã¯ããšã³ã³ãŒãã£ã³ã°ãé«ããå¹
ããã¯ã»ã«ããŒã¿ã®ãã£ãŒã«ããå«ãŸããã深床ç»åã®ãšã³ã³ãŒãã£ã³ã°ãã£ãŒã«ãã¯ã"16UC1 "ã®ãããªåœ¢åŒã䜿çšããããšãå€ãããã¯ã»ã«ããšã«16ãããã®ç¬Šå·ãªãæŽæ°ã瀺ããåå€ã¯ãªããžã§ã¯ããŸã§ã®è·é¢ãè¡šããŸãã深床ç»åã¯äžè¬çã«RGBç»åãšçµã¿åãããŠäœ¿çšãããããå
æ¬çãªç°å¢ãã¥ãŒãæäŸããã
YOLO ã䜿çšãããšãRGBç»åãšæ·±åºŠç»åã®äž¡æ¹ããæ å ±ãæœåºããçµã¿åãããããšãã§ãããäŸãã°ãYOLO ãRGBç»åå ã®ç©äœãæ€åºããããšãã§ãããã®æ€åºã䜿çšããŠæ·±åºŠç»åå ã®å¯Ÿå¿ããé åããã³ãã€ã³ãã§ç¹å®ããããšãã§ãããããã«ãããæ€åºããããªããžã§ã¯ãã®æ£ç¢ºãªå¥¥è¡ãæ å ±ãæœåºããããšãã§ããããããã®ç°å¢ã3次å ã§ç解ããèœåãåäžããŸãã
RGB-Dã«ã¡ã©
深床ç»åãæ±ãå ŽåãRGBç»åãšæ·±åºŠç»åãæ£ããäœçœ®åããããããšãäžå¯æ¬ ã§ããIntel RealSenseã·ãªãŒãºã®ãããªRGB-Dã«ã¡ã©ã¯ãåæãããRGBç»åãšæ·±åºŠç»åãæäŸããäž¡æ¹ã®ãœãŒã¹ããã®æ å ±ãç°¡åã«çµã¿åãããããšãã§ããŸããRGBã«ã¡ã©ãšæ·±åºŠã«ã¡ã©ãå¥ã ã«äœ¿çšããå Žåã¯ãæ£ç¢ºãªã¢ã©ã€ã¡ã³ãã確ä¿ããããã«ãã£ãªãã¬ãŒã·ã§ã³ãè¡ãããšãéèŠã§ãã
ããã¹ ã¹ãããã»ãã€ã»ã¹ããã
ãã®äŸã§ã¯ãYOLO ã䜿ã£ãŠç»åãã»ã°ã¡ã³ããŒã·ã§ã³ããæœåºãããã¹ã¯ãé©çšããŠæ·±åºŠç»åå ã®ãªããžã§ã¯ããã»ã°ã¡ã³ããŒã·ã§ã³ãããããã«ããã察象ç©äœã®åãã¯ã»ã«ã®ãã«ã¡ã©ã®çŠç¹äžå¿ããã®è·é¢ãæ±ããããšãã§ããããã®è·é¢æ å ±ãåŸãããšã§ãã«ã¡ã©ãšã·ãŒã³å ã®ç¹å®ã®ãªããžã§ã¯ããšã®è·é¢ãèšç®ããããšãã§ããŸããå¿ èŠãªã©ã€ãã©ãªãã€ã³ããŒãããROS ããŒããäœæããã»ã°ã¡ã³ããŒã·ã§ã³ã¢ãã«ãš ROS ãããã¯ãã€ã³ã¹ ã¿ã³ã¹åããããšããå§ããŸãã
import time
import rospy
from std_msgs.msg import String
from ultralytics import YOLO
rospy.init_node("ultralytics")
time.sleep(1)
segmentation_model = YOLO("yolov8m-seg.pt")
classes_pub = rospy.Publisher("/ultralytics/detection/distance", String, queue_size=5)
次ã«ãå
¥åããã深床ç»åã¡ãã»ãŒãžãåŠçããã³ãŒã«ããã¯é¢æ°ãå®çŸ©ããŸãããã®é¢æ°ã¯ã深床ç»åãš RGB ç»åã®ã¡ãã»ãŒãžãåŸ
ã¡ããããã numpy é
åã«å€æããRGB ç»åã«ã»ã°ã¡ã³ããŒã·ã§ã³ã¢ãã«ãé©çšããŸãã次ã«ãæ€åºãããåãªããžã§ã¯ãã®ã»ã°ã¡ã³ããŒã·ã§ã³ãã¹ã¯ãæœåºãã深床ç»åã䜿çšããŠã«ã¡ã©ããã®ãªããžã§ã¯ãã®å¹³åè·é¢ãèšç®ããŸããã»ãšãã©ã®ã»ã³ãµã«ã¯ãã¯ãªããè·é¢ãšåŒã°ããæ倧è·é¢ãããããã®å€ãè¶
ãããš inf (np.inf
).åŠçããåã«ããããã®NULLå€ããã£ã«ã¿ãªã³ã°ãããã®å€ã« 0
.æåŸã«ãæ€åºããããªããžã§ã¯ããããã®å¹³åè·é¢ãšãšãã« /ultralytics/detection/distance
ãšãããããã¯ãããã
import numpy as np
import ros_numpy
from sensor_msgs.msg import Image
def callback(data):
"""Callback function to process depth image and RGB image."""
image = rospy.wait_for_message("/camera/color/image_raw", Image)
image = ros_numpy.numpify(image)
depth = ros_numpy.numpify(data)
result = segmentation_model(image)
for index, cls in enumerate(result[0].boxes.cls):
class_index = int(cls.cpu().numpy())
name = result[0].names[class_index]
mask = result[0].masks.data.cpu().numpy()[index, :, :].astype(int)
obj = depth[mask == 1]
obj = obj[~np.isnan(obj)]
avg_distance = np.mean(obj) if len(obj) else np.inf
classes_pub.publish(String(data=str(all_objects)))
rospy.Subscriber("/camera/depth/image_raw", Image, callback)
while True:
rospy.spin()
å®å šãªã³ãŒã
import time
import numpy as np
import ros_numpy
import rospy
from sensor_msgs.msg import Image
from std_msgs.msg import String
from ultralytics import YOLO
rospy.init_node("ultralytics")
time.sleep(1)
segmentation_model = YOLO("yolov8m-seg.pt")
classes_pub = rospy.Publisher("/ultralytics/detection/distance", String, queue_size=5)
def callback(data):
"""Callback function to process depth image and RGB image."""
image = rospy.wait_for_message("/camera/color/image_raw", Image)
image = ros_numpy.numpify(image)
depth = ros_numpy.numpify(data)
result = segmentation_model(image)
for index, cls in enumerate(result[0].boxes.cls):
class_index = int(cls.cpu().numpy())
name = result[0].names[class_index]
mask = result[0].masks.data.cpu().numpy()[index, :, :].astype(int)
obj = depth[mask == 1]
obj = obj[~np.isnan(obj)]
avg_distance = np.mean(obj) if len(obj) else np.inf
classes_pub.publish(String(data=str(all_objects)))
rospy.Subscriber("/camera/depth/image_raw", Image, callback)
while True:
rospy.spin()
ROSã§Ultralytics sensor_msgs/PointCloud2
ã«ã€ã㊠sensor_msgs/PointCloud2
ã¡ãã»ãŒãžã¿ã€ã ã¯ã3Dç¹çŸ€ããŒã¿ãè¡šçŸããããã«ROSã§äœ¿çšãããããŒã¿æ§é ã§ãããã®ã¡ãã»ãŒãžã¿ã€ãã¯ããããã¢ããªã±ãŒã·ã§ã³ã«äžå¯æ¬ ã§ã3Dãããã³ã°ãç©äœèªèãããŒã«ãªãŒãŒã·ã§ã³ãªã©ã®ã¿ã¹ã¯ãå¯èœã«ããã
ç¹çŸ€ãšã¯ã3次å
座æšç³»å
ã§å®çŸ©ãããããŒã¿ç¹ã®éãŸãã§ããããããã®ããŒã¿ç¹ã¯ã3Dã¹ãã£ã³æè¡ã«ãã£ãŠãã£ããã£ããããªããžã§ã¯ããã·ãŒã³ã®å€é¢ãè¡šããŠããŸããã¯ã©ãŠãå
ã®åãã€ã³ã㯠X
, Y
ãã㊠Z
座æšã¯ç©ºéå
ã®äœçœ®ã«å¯Ÿå¿ããè²ã匷床ãªã©ã®è¿œå æ
å ±ãå«ãŸããã
åºæºãã¬ãŒã
ãšäžç·ã«ä»äºãããå Žå sensor_msgs/PointCloud2
ç¹çŸ€ããŒã¿ãååŸãããã»ã³ãµãŒã®åºæºãã¬ãŒã ãèæ
®ããããšãäžå¯æ¬ ã§ãããç¹çŸ€ã¯æåãã»ã³ãµãŒã®åç
§ãã¬ãŒã ã§ãã£ããã£ãããããã®åç
§ãã¬ãŒã 㯠/tf_static
ãããã¯ãåç
§ããŠãã ãããããããç¹å®ã®ã¢ããªã±ãŒã·ã§ã³ã®èŠä»¶ã«ãã£ãŠã¯ãç¹çŸ€ãå¥ã®åç
§ãã¬ãŒã ã«å€æããå¿
èŠããããããããŸããããã®å€æ㯠tf2_ros
ããã±ãŒãžã§ã座æšãã¬ãŒã ã管çãããããã®éã§ããŒã¿ãå€æããããã®ããŒã«ãæäŸããã
ç¹çŸ€ã®ååŸ
ãã€ã³ãã¯ã©ãŠãã¯ãããŸããŸãªã»ã³ãµãŒã䜿ã£ãŠåŸãããšãã§ããïŒ
- LIDARïŒLight Detection and RangingïŒïŒã¬ãŒã¶ãŒãã«ã¹ã䜿ã£ãŠå¯Ÿè±¡ç©ãŸã§ã®è·é¢ã枬å®ããé«ç²ŸåºŠã®3Då°å³ãäœæããã
- 深床ã«ã¡ã©ïŒåãã¯ã»ã«ã®æ·±åºŠæ å ±ããã£ããã£ããã·ãŒã³ã®3Dåæ§ç¯ãå¯èœã«ããã
- ã¹ãã¬ãªã«ã¡ã©ïŒ2å°ä»¥äžã®ã«ã¡ã©ã䜿çšããäžè§æž¬éã«ãã£ãŠæ·±åºŠæ å ±ãååŸããã
- æ§é åå ã¹ãã£ããŒïŒæ¢ç¥ã®ãã¿ãŒã³ãè¡šé¢ã«æ圱ããå€åœ¢ã枬å®ããŠæ·±ããèšç®ããã
YOLO ãç¹çŸ€ãšå ±ã«äœ¿çšãã
YOLO ãšçµ±åããã sensor_msgs/PointCloud2
ã¿ã€ãã®ã¡ãã»ãŒãžã§ã¯ã深床ãããã«äœ¿çšãããæ¹æ³ãšåæ§ã®æ¹æ³ãæ¡çšããããšãã§ãããç¹çŸ€ã«åã蟌ãŸããè²æ
å ±ã掻çšããããšã§ã2Dç»åãæœåºããYOLO ã䜿ã£ãŠãã®ç»åã«å¯ŸããŠã»ã°ã¡ã³ããŒã·ã§ã³ãè¡ãããã®çµæåŸããããã¹ã¯ã3次å
ã®ç¹ã«é©çšããããšã§ãé¢å¿ã®ãã3Dãªããžã§ã¯ããåé¢ããããšãã§ããã
ç¹çŸ€ãæ±ãã«ã¯ãOpen3D (pip install open3d
)ããŠãŒã¶ãŒãã¬ã³ããªãŒãªPython ã©ã€ãã©ãªã§ããOpen3Dã¯ãç¹çŸ€ããŒã¿æ§é ã®ç®¡çãå¯èŠåãè€éãªæäœãã·ãŒã ã¬ã¹ã«å®è¡ããããã®å
ç¢ãªããŒã«ãæäŸããŸãããã®ã©ã€ãã©ãªã䜿çšããããšã§ãYOLO-based segmentationãšé£åããç¹çŸ€ã®æäœãšè§£æã®ããã»ã¹ã倧å¹
ã«ç°¡çŽ åããèœåãåäžãããããšãã§ããŸãã
ç¹çŸ€ã¹ããããã€ã¹ãããã®äœ¿ãæ¹
å¿ èŠãªã©ã€ãã©ãªãã€ã³ããŒãããã»ã°ã¡ã³ããŒã·ã§ã³çšã®YOLO ã¢ãã«ãã€ã³ã¹ã¿ã³ã¹åããã
import time
import rospy
from ultralytics import YOLO
rospy.init_node("ultralytics")
time.sleep(1)
segmentation_model = YOLO("yolov8m-seg.pt")
é¢æ°ãäœæãã pointcloud2_to_array
ãå€æããã sensor_msgs/PointCloud2
ã¡ãã»ãŒãžã2ã€ã®numpyé
åã«æ ŒçŽãããã¡ãã»ãŒãžã¯ sensor_msgs/PointCloud2
ã¡ãã»ãŒãžã«ã¯ n
ã«åºã¥ãã width
ãã㊠height
ååŸããç»åã®äŸãã° 480 x 640
ç»åã«ã¯ 307,200
ç¹ã§ãããåãã€ã³ãã«ã¯3ã€ã®ç©ºé座æš(xyz
)ã®å¯Ÿå¿ããè²ã RGB
ãã©ãŒãããããããã¯2ã€ã®å¥ã
ã®æ
å ±ãã£ã³ãã«ãšèããããšãã§ããã
ãã®é¢æ°ã¯ xyz
座æšãš RGB
å€ãå
ã®ã«ã¡ã©ã®è§£å床 (width x height
).ã»ãšãã©ã®ã»ã³ãµãŒã«ã¯ãã¯ãªããè·é¢ãšåŒã°ããæ倧è·é¢ãããããããè¶
ãããšå€ã inf (np.inf
).åŠçããåã«ããããã®NULLå€ããã£ã«ã¿ãªã³ã°ãããã®å€ã« 0
.
import numpy as np
import ros_numpy
def pointcloud2_to_array(pointcloud2: PointCloud2) -> tuple:
"""
Convert a ROS PointCloud2 message to a numpy array.
Args:
pointcloud2 (PointCloud2): the PointCloud2 message
Returns:
(tuple): tuple containing (xyz, rgb)
"""
pc_array = ros_numpy.point_cloud2.pointcloud2_to_array(pointcloud2)
split = ros_numpy.point_cloud2.split_rgb_field(pc_array)
rgb = np.stack([split["b"], split["g"], split["r"]], axis=2)
xyz = ros_numpy.point_cloud2.get_xyz_points(pc_array, remove_nans=False)
xyz = np.array(xyz).reshape((pointcloud2.height, pointcloud2.width, 3))
nan_rows = np.isnan(xyz).all(axis=2)
xyz[nan_rows] = [0, 0, 0]
rgb[nan_rows] = [0, 0, 0]
return xyz, rgb
次㫠/camera/depth/points
ãããã¯ã§ç¹çŸ€ã¡ãã»ãŒãžãåä¿¡ã sensor_msgs/PointCloud2
ã¡ãã»ãŒãžããXYZ座æšãšRGBå€ãå«ãnumpyé
åã«å€æããã pointcloud2_to_array
é¢æ°)ãYOLO ã¢ãã«ã䜿ã£ãŠ RGB ç»åãåŠçããã»ã°ã¡ã³ããŒã·ã§ã³ããããªããžã§ã¯ããæœåºãããæ€åºãããåãªããžã§ã¯ãã«ã€ããŠãã»ã°ã¡ã³ããŒã·ã§ã³ãã¹ã¯ãæœåºããRGBç»åãšXYZ座æšã®äž¡æ¹ã«é©çšããŠã3D空éã§ãªããžã§ã¯ããåé¢ããã
ãã¹ã¯ã¯ãã€ããªå€ã§æ§æãããŠããã®ã§ãåŠçã¯ç°¡åã§ããã 1
ãªããžã§ã¯ãã®ååšã瀺ã 0
ããªãããšã瀺ãããã¹ã¯ãé©çšããã«ã¯ãå
ã®ãã£ã³ãã«ã«ãã¹ã¯ãæããã ãã§ããããã®æäœã«ãããç»åå
ã®ç®çã®ãªããžã§ã¯ããå¹æçã«åé¢ãããŸããæåŸã«ãOpen3Dç¹çŸ€ãªããžã§ã¯ããäœæããã»ã°ã¡ã³ãåããããªããžã§ã¯ãããé¢é£ããè²ãšãšãã«3D空éã§å¯èŠåããŸãã
import sys
import open3d as o3d
ros_cloud = rospy.wait_for_message("/camera/depth/points", PointCloud2)
xyz, rgb = pointcloud2_to_array(ros_cloud)
result = segmentation_model(rgb)
if not len(result[0].boxes.cls):
print("No objects detected")
sys.exit()
classes = result[0].boxes.cls.cpu().numpy().astype(int)
for index, class_id in enumerate(classes):
mask = result[0].masks.data.cpu().numpy()[index, :, :].astype(int)
mask_expanded = np.stack([mask, mask, mask], axis=2)
obj_rgb = rgb * mask_expanded
obj_xyz = xyz * mask_expanded
pcd = o3d.geometry.PointCloud()
pcd.points = o3d.utility.Vector3dVector(obj_xyz.reshape((ros_cloud.height * ros_cloud.width, 3)))
pcd.colors = o3d.utility.Vector3dVector(obj_rgb.reshape((ros_cloud.height * ros_cloud.width, 3)) / 255)
o3d.visualization.draw_geometries([pcd])
å®å šãªã³ãŒã
import sys
import time
import numpy as np
import open3d as o3d
import ros_numpy
import rospy
from ultralytics import YOLO
rospy.init_node("ultralytics")
time.sleep(1)
segmentation_model = YOLO("yolov8m-seg.pt")
def pointcloud2_to_array(pointcloud2: PointCloud2) -> tuple:
"""
Convert a ROS PointCloud2 message to a numpy array.
Args:
pointcloud2 (PointCloud2): the PointCloud2 message
Returns:
(tuple): tuple containing (xyz, rgb)
"""
pc_array = ros_numpy.point_cloud2.pointcloud2_to_array(pointcloud2)
split = ros_numpy.point_cloud2.split_rgb_field(pc_array)
rgb = np.stack([split["b"], split["g"], split["r"]], axis=2)
xyz = ros_numpy.point_cloud2.get_xyz_points(pc_array, remove_nans=False)
xyz = np.array(xyz).reshape((pointcloud2.height, pointcloud2.width, 3))
nan_rows = np.isnan(xyz).all(axis=2)
xyz[nan_rows] = [0, 0, 0]
rgb[nan_rows] = [0, 0, 0]
return xyz, rgb
ros_cloud = rospy.wait_for_message("/camera/depth/points", PointCloud2)
xyz, rgb = pointcloud2_to_array(ros_cloud)
result = segmentation_model(rgb)
if not len(result[0].boxes.cls):
print("No objects detected")
sys.exit()
classes = result[0].boxes.cls.cpu().numpy().astype(int)
for index, class_id in enumerate(classes):
mask = result[0].masks.data.cpu().numpy()[index, :, :].astype(int)
mask_expanded = np.stack([mask, mask, mask], axis=2)
obj_rgb = rgb * mask_expanded
obj_xyz = xyz * mask_expanded
pcd = o3d.geometry.PointCloud()
pcd.points = o3d.utility.Vector3dVector(obj_xyz.reshape((ros_cloud.height * ros_cloud.width, 3)))
pcd.colors = o3d.utility.Vector3dVector(obj_rgb.reshape((ros_cloud.height * ros_cloud.width, 3)) / 255)
o3d.visualization.draw_geometries([pcd])
ããããã質å
ããããã»ãªãã¬ãŒãã£ã³ã°ã»ã·ã¹ãã ïŒROSïŒãšã¯ïŒ
ãããããªãã¬ãŒãã£ã³ã°ã·ã¹ãã ïŒROSïŒã¯ãéçºè ãå ç¢ãªããããã¢ããªã±ãŒã·ã§ã³ãäœæããã®ã«åœ¹ç«ã€ãããããå·¥åŠã§äžè¬çã«äœ¿çšãããŠãããªãŒãã³ãœãŒã¹ã®ãã¬ãŒã ã¯ãŒã¯ã§ããROSã¯ãè€éãªã¢ããªã±ãŒã·ã§ã³ã®éçºã容æã«ãããããããã·ã¹ãã ã®æ§ç¯ãšã€ã³ã¿ãŒãã§ãŒã¹çšã®ã©ã€ãã©ãªãšããŒã«ã®ã³ã¬ã¯ã·ã§ã³ãæäŸããŸããROSã¯ããããã¯ãŸãã¯ãµãŒãã¹ãä»ããã¡ãã»ãŒãžã䜿çšããŠããŒãéã®éä¿¡ããµããŒãããŸãã
Ultralytics YOLO ãROSãšçµ±åããŠãªã¢ã«ã¿ã€ã ã®ç©äœæ€åºãè¡ãã«ã¯ïŒ
Ultralytics YOLO ãROSãšçµ±åããã«ã¯ãROSç°å¢ãã»ããã¢ããããã»ã³ãµãŒããŒã¿ã®åŠçã«YOLO ã以äžã®ãããªå¿
èŠãªäŸåé¢ä¿ãã€ã³ã¹ããŒã«ããããšããå§ããã ros_numpy
ãšUltralytics YOLO ïŒ
次ã«ãROSããŒããäœæããåä¿¡ããŒã¿ãåŠçããããã«ç»åãããã¯ããµãã¹ã¯ã©ã€ãããã以äžã«æå°éã®äŸã瀺ãïŒ
import ros_numpy
import rospy
from sensor_msgs.msg import Image
from ultralytics import YOLO
detection_model = YOLO("yolov8m.pt")
rospy.init_node("ultralytics")
det_image_pub = rospy.Publisher("/ultralytics/detection/image", Image, queue_size=5)
def callback(data):
array = ros_numpy.numpify(data)
det_result = detection_model(array)
det_annotated = det_result[0].plot(show=False)
det_image_pub.publish(ros_numpy.msgify(Image, det_annotated, encoding="rgb8"))
rospy.Subscriber("/camera/color/image_raw", Image, callback)
rospy.spin()
ROSã®ãããã¯ãšã¯äœãããããŠãããã¯Ultralytics YOLO ã§ã©ã®ããã«äœ¿ãããŠããã®ãïŒ
ROSãããã¯ã¯ããããªãã·ã¥ã»ãµãã¹ã¯ã©ã€ãã»ã¢ãã«ã䜿çšããããšã§ãROSãããã¯ãŒã¯å ã®ããŒãéã®éä¿¡ã容æã«ããŸãããããã¯ã¯ãããŒããéåæã«ã¡ãã»ãŒãžãéåä¿¡ããããã«äœ¿çšããååä»ããã£ãã«ã§ããUltralytics YOLO ã®ã³ã³ããã¹ãã§ã¯ãããŒããç»åãããã¯ã«ãµãã¹ã¯ã©ã€ããããæ€åºãã»ã°ã¡ã³ããŒã·ã§ã³ãªã©ã®ã¿ã¹ã¯ã®ããã«YOLO ã䜿çšããŠç»åãåŠçããçµæãæ°ãããããã¯ã«ãããªãã·ã¥ããããšãã§ããŸãã
äŸãã°ãã«ã¡ã©ã»ãããã¯ããµãã¹ã¯ã©ã€ãããåä¿¡ç»åãåŠçããŠæ€åºããïŒ
ãªãROSã®Ultralytics YOLO ã深床ç»åã䜿ãã®ãïŒ
ã§è¡šãããROSã®æ·±åºŠç»åã sensor_msgs/Image
ããã¯ãé害ç©åé¿ã3Dãããã³ã°ãããŒã«ã©ã€ãŒãŒã·ã§ã³ãªã©ã®ã¿ã¹ã¯ã«äžå¯æ¬ ã§ããã以äžã«ãã 奥è¡ãæ
å ±ã䜿ã£ãŠ RGBç»åãšãšãã«ãããããã¯3Dç°å¢ãããããç解ããããšãã§ããã
YOLO ã䜿ãã°ãRGBç»åããã»ã°ã¡ã³ããŒã·ã§ã³ã»ãã¹ã¯ãæœåºãããã®ãã¹ã¯ã深床ç»åã«é©çšããããšã§ãæ£ç¢ºãª3Dãªããžã§ã¯ãæ å ±ãåŸãããšãã§ããããããã®ããã²ãŒã·ã§ã³èœåãåšå²ãšã®ã€ã³ã¿ã©ã¯ã·ã§ã³èœåãåäžãããããšãã§ããŸãã
ROSã§YOLO ã3Dç¹çŸ€ãå¯èŠåããã«ã¯ïŒ
YOLO ã䜿ã£ãŠROSã§3Dç¹çŸ€ãå¯èŠåããïŒ
- ã³ã³ããŒã
sensor_msgs/PointCloud2
ã¡ãã»ãŒãžãnumpyé åã«å€æããã - YOLO ãRGBç»åãåå²ããã
- ç¹çŸ€ã«ã»ã°ã¡ã³ããŒã·ã§ã³ãã¹ã¯ãé©çšããã
Open3Dã䜿ã£ãããžã¥ã¢ã©ã€ãŒãŒã·ã§ã³ã®äŸã§ãïŒ
import sys
import open3d as o3d
import ros_numpy
import rospy
from sensor_msgs.msg import PointCloud2
from ultralytics import YOLO
rospy.init_node("ultralytics")
segmentation_model = YOLO("yolov8m-seg.pt")
def pointcloud2_to_array(pointcloud2):
pc_array = ros_numpy.point_cloud2.pointcloud2_to_array(pointcloud2)
split = ros_numpy.point_cloud2.split_rgb_field(pc_array)
rgb = np.stack([split["b"], split["g"], split["r"]], axis=2)
xyz = ros_numpy.point_cloud2.get_xyz_points(pc_array, remove_nans=False)
xyz = np.array(xyz).reshape((pointcloud2.height, pointcloud2.width, 3))
return xyz, rgb
ros_cloud = rospy.wait_for_message("/camera/depth/points", PointCloud2)
xyz, rgb = pointcloud2_to_array(ros_cloud)
result = segmentation_model(rgb)
if not len(result[0].boxes.cls):
print("No objects detected")
sys.exit()
classes = result[0].boxes.cls.cpu().numpy().astype(int)
for index, class_id in enumerate(classes):
mask = result[0].masks.data.cpu().numpy()[index, :, :].astype(int)
mask_expanded = np.stack([mask, mask, mask], axis=2)
obj_rgb = rgb * mask_expanded
obj_xyz = xyz * mask_expanded
pcd = o3d.geometry.PointCloud()
pcd.points = o3d.utility.Vector3dVector(obj_xyz.reshape((-1, 3)))
pcd.colors = o3d.utility.Vector3dVector(obj_rgb.reshape((-1, 3)) / 255)
o3d.visualization.draw_geometries([pcd])
ãã®ã¢ãããŒãã¯ãã»ã°ã¡ã³ãåããããªããžã§ã¯ãã®3Dããžã¥ã¢ã©ã€ãŒãŒã·ã§ã³ãæäŸããããã²ãŒã·ã§ã³ãæäœãªã©ã®ã¿ã¹ã¯ã«åœ¹ç«ã€ã