This document provides a brief intro of the usage of builtin functions and detectors designed for each App. Robotplus ------------ 1. Copper Detection """""""""""""""""" Create and Load Detector =========================== * Using the default Kpick's detector :: from kpick.apps.robotplus.copper_detector import create_and_load_copper_detector detector = create_and_load_copper_detector(cfg_path=cfg_path) .. note:: **cfg_path** is default **kpick/apps/robotplus/configs/copper_grip.cfg when cfg_path** is None or not given Modify and Load Detector ============================ * Extending the default Kpick's detector :: from kpick.apps.robotplus.copper_detector import CopperDetector, create_and_load_copper_detector class AppDector(CopperDetector): def new_function(self): print('new function') detector = create_and_load_copper_detector(Detector=AppDector,cfg_path=cfg_path) Demo on single RGB-D image ============================== :: def demo_detect_copper_single(): import cv2 import kpick import os from ketisdk.vision.utils.rgbd_utils_v2 import RGBD KPICK_DIR = os.path.split(kpick.__file__)[0] # configs depth_min = 900 depth_max = 1000 workspace = [(530, 278), (1252, 286), (1248, 714), (536, 704)] # load image rgb = cv2.imread(os.path.join(KPICK_DIR, 'apps/robotplus/test_images/copper01_rgb.png'))[:, :, ::-1] depth = cv2.imread(os.path.join(KPICK_DIR, 'apps/robotplus/test_images/copper01_depth.png'), cv2.IMREAD_UNCHANGED) rgbd = RGBD(rgb=rgb, depth=depth, depth_min=depth_min, depth_max=depth_max) rgbd.set_workspace(pts=workspace) # load model detector = create_and_load_copper_detector() # predict detector.args.flag.show_steps = True ret = detector.detect_and_show_grips(rgbd=rgbd, net_args=detector.args.grip_net, args=detector.args) # show cv2.imshow('reviewer', ret['im'][:, :, ::-1]) cv2.waitKey() 2. Cylind Detection """""""""""""""""""" 3. OI Detection """""""""""""""" FPCB ----- Mujin ---- Order Picking --------------- .. image:: ../images/fig_orderpicking_diagram.png :width: 100 % 1. Output format """"""""""""""""" .. list-table:: :widths: 10 90 :header-rows: 0 * - **Field** - **Description** * - suction - all suction candidates. Num_suction x 4, where each row presents a suction with format is [x,y,z,score] * - best_n_inds - top n best suction grasp indexes * - bes_ind - best suction grasp index * - best - target suction grasp * - im - display suctions on the input image 2. Create and Load Detector on Custom Project """"""""""""""""" * Extending the default Kpick's detector :: from kpick.apps.orderpicking.detector import create_and_load_order_detector, OrderDetector class AppDetector(OrderDetector): def new_function(self): print('new function') detector = create_and_load_order_detector(Detector=AppDetector, cfg_path=cfg_path) 3. Demo on single RGB-D image """"""""""""""""" :: import cv2 import kpick import os from ketisdk.vision.utils.rgbd_utils_v2 import RGBD KPICK_DIR = os.path.split(kpick.__file__)[0] # configs depth_min = 700 depth_max = 900 workspace = [(398, 172), (949, 174), (949, 580), (398, 572)] # load image rgb = cv2.imread(os.path.join(KPICK_DIR, 'apps/orderpicking/test_images/1_rgb.png'))[:, :, ::-1] depth = cv2.imread(os.path.join(KPICK_DIR, 'apps/orderpicking/test_images/1_depth.png'), cv2.IMREAD_UNCHANGED) rgbd = RGBD(rgb=rgb, depth=depth, depth_min=depth_min, depth_max=depth_max) rgbd.set_workspace(pts=workspace) # load model detector = create_and_load_order_detector() # predict detector.args.flag.show_steps = False ret = detector.detect_and_show_suctions(rgbd=rgbd, net_args=detector.args.suction_net, rpn_args=detector.args.rpn, args=detector.args, remove_bg=False) # show cv2.imshow('reviewer', ret['im'][:, :, ::-1]) cv2.waitKey() .. image:: ../images/fig_orderpicking_sample_result.png :width: 70 % :align: center 4. Parameters tuning """"""""""""""""" .. list-table:: :widths: 10 90 :header-rows: 0 * - **Parameter** - **Description** * - suction_net.pad_sizes - Patch sizes of suction pose scanning process. Larger value should be set for larger object * - suction_net.stride - Stride of suction scanning process. Smaller value makes more suction candidates, but computational heavier. * - rpn.enable - Enable RPN for suction detector or not * - rpn.score_thresh - Score threshold of RPN * - rpn.min_size - Minimum size of larger size of a detected object Kimchi ------ Smart Gripper -------------- Random Piece Picking ------------ .. image:: ../images/fig_multimode_grasp_diagram.png :width: 100 % 1. Output format """"""""""""""""" .. list-table:: :widths: 10 90 :header-rows: 0 * - **Field** - **Description** * - target_grasp - target grasp selected among all detectors' results * - target_grasp_name - name of target grasp, can be: grip, suction, inner_grip, suction_2pts * - im - display grip on the input image 2. Create and Load Detector """"""""""""""""" * Using the default Kpick's detector :: from kpick.apps.randompicking.detector import create_and_load_random_picking_detector detector = create_and_load_random_picking_detector(detector_type='suction', cfg_path=cfg_path) .. note:: Please refer **kpick/apps/randompicking/configs/*.cfg** 3. Modify and Load Detector """"""""""""""""" * Extending the default Kpick's detector :: from kpick.apps.randompicking.detector import create_and_load_random_picking_detector, OnePointDualGraspDetector, OnePointDualGraspDetector, RandomGripDetector, RandomSuctionDetector class AppDetector(RandomSuctionDetector): def new_function(self): print('new function') detector = RandomGripDetector(Detector=AppDetector, cfg_path=cfg_path, detector_type='suction') 4. Demo on single RGB-D image """"""""""""""""" :: import cv2 from ketisdk.vision.utils.rgbd_utils_v2 import RGBD import os from ketisdk.vision.utils.rgbd_utils_v2 import RGBD import kpick KPICK_DIR = os.path.split(kpick.__file__)[0] # load image rgb = cv2.imread(os.path.join(KPICK_DIR, 'apps/randompicking/test_images/01_rgb.png'))[:, :, ::-1] depth = cv2.imread(os.path.join(KPICK_DIR, 'apps/randompicking/test_images/01_depth.png'), cv2.IMREAD_UNCHANGED) rgbd = RGBD(rgb=rgb, depth=depth, depth_min=600, depth_max=800) # set crop roi rgbd.set_workspace(pts=[(348, 193), (894, 194), (894, 594), (345, 590)]) # predict detector = create_and_load_random_picking_detector(detector_type='grip') detector.args.flag.show_steps = True ret = detector.detect_and_show_grips(rgbd=rgbd, net_args=detector.args.grip_net, args=detector.args, remove_bg=detector.args.grip_net.remove_bg) # show cv2.imshow('grip', ret['im'][:, :, ::-1]) # predict detector = create_and_load_random_picking_detector(detector_type='suction') detector.args.flag.show_steps = True ret = detector.detect_and_show_suctions(rgbd=rgbd, net_args=detector.args.suction_net, args=detector.args, remove_bg=detector.args.suction_net.remove_bg, rpn_args=detector.args.rpn) # show cv2.imshow('suction', ret['im'][:, :, ::-1]) # predict detector = create_and_load_random_picking_detector(detector_type='one_point_dual') detector.args.flag.show_steps = True ret = detector.detect_and_show_multimode_grasp(rgbd=rgbd, grip_args=detector.args.grip_net, suction_args=detector.args.suction_net, inner_grip_args=detector.args.inner_grip_net, args=detector.args) # show cv2.imshow('one_point_dual', ret['im'][:, :, ::-1]) # predict detector = create_and_load_random_picking_detector(detector_type='two_points_dual') detector.args.flag.show_steps = True ret = detector.detect_and_show_multimode_grasp(rgbd=rgbd, grip_args=detector.args.grip_net, suction_args=detector.args.suction_net, inner_grip_args=detector.args.inner_grip_net, args=detector.args) # show cv2.imshow('two_points_dual', ret['im'][:, :, ::-1]) cv2.waitKey() .. image:: ../images/fig_randompicking_sample_result.png :width: 70 % :align: center 5. Parameters tuning """"""""""""""""" .. list-table:: :widths: 10 90 :header-rows: 0 * - **Parameter** - **Description** * - grip_net.erode_h - gripper plate's width * - grip_net.grip_w_ranges - ranges of gripper's width, which are tuned for each target object * - grip_net.grad_thresh - gradient threshold. Smaller value makes more grip candidates but noisy, and vice versa * - grip_net.remove_bg - remove detected grips on the background or not * - grip_net.top_n - top n target grips * - grip_net.dy - Smaller value makes more dense grip candidates. * - grip_net.ellipse_axes - Dimensions of ellipse that examines neighbor grips. For bigger object, dimension should be larger * - suction_net.pad_sizes - Patch sizes of suction pose scanning process. Larger value should be set for larger object * - suction_net.stride - Stride of suction scanning process. Smaller value makes more suction candidates, but computational heavier. * - rpn.enable - Enable RPN for suction detector or not