This document provides a brief intro of the usage of builtin functions and detectors designed for each App.

Robotplus

1. Copper Detection

Create and Load Detector

Using the default Kpick’s detector

from kpick.apps.robotplus.copper_detector import create_and_load_copper_detector
detector = create_and_load_copper_detector(cfg_path=cfg_path)

Note

cfg_path is default kpick/apps/robotplus/configs/copper_grip.cfg when cfg_path is None or not given

Modify and Load Detector

Extending the default Kpick’s detector

from kpick.apps.robotplus.copper_detector import  CopperDetector, create_and_load_copper_detector
class AppDector(CopperDetector):
    def new_function(self):
        print('new function')

detector = create_and_load_copper_detector(Detector=AppDector,cfg_path=cfg_path)

Demo on single RGB-D image

def demo_detect_copper_single():
    import cv2
    import kpick
    import os
    from ketisdk.vision.utils.rgbd_utils_v2 import RGBD

    KPICK_DIR = os.path.split(kpick.__file__)[0]

    # configs
    depth_min = 900
    depth_max = 1000
    workspace = [(530, 278), (1252, 286), (1248, 714), (536, 704)]

    # load image
    rgb = cv2.imread(os.path.join(KPICK_DIR, 'apps/robotplus/test_images/copper01_rgb.png'))[:, :, ::-1]
    depth = cv2.imread(os.path.join(KPICK_DIR, 'apps/robotplus/test_images/copper01_depth.png'), cv2.IMREAD_UNCHANGED)
    rgbd = RGBD(rgb=rgb, depth=depth, depth_min=depth_min, depth_max=depth_max)
    rgbd.set_workspace(pts=workspace)

    # load model
    detector = create_and_load_copper_detector()

    # predict
    detector.args.flag.show_steps = True
    ret = detector.detect_and_show_grips(rgbd=rgbd, net_args=detector.args.grip_net, args=detector.args)

    # show
    cv2.imshow('reviewer', ret['im'][:, :, ::-1])
    cv2.waitKey()

2. Cylind Detection

3. OI Detection

FPCB

Mujin

Order Picking

1. Output format

Field	Description
suction	all suction candidates. Num_suction x 4, where each row presents a suction with format is [x,y,z,score]
best_n_inds	top n best suction grasp indexes
bes_ind	best suction grasp index
best	target suction grasp
im	display suctions on the input image

2. Create and Load Detector on Custom Project

Extending the default Kpick’s detector

from kpick.apps.orderpicking.detector import create_and_load_order_detector, OrderDetector
class AppDetector(OrderDetector):
    def new_function(self):
        print('new function')

detector = create_and_load_order_detector(Detector=AppDetector, cfg_path=cfg_path)

3. Demo on single RGB-D image

import cv2
import kpick
import os
from ketisdk.vision.utils.rgbd_utils_v2 import RGBD

KPICK_DIR = os.path.split(kpick.__file__)[0]

# configs
depth_min = 700
depth_max = 900
workspace = [(398, 172), (949, 174), (949, 580), (398, 572)]

# load image
rgb = cv2.imread(os.path.join(KPICK_DIR, 'apps/orderpicking/test_images/1_rgb.png'))[:, :, ::-1]
depth = cv2.imread(os.path.join(KPICK_DIR, 'apps/orderpicking/test_images/1_depth.png'), cv2.IMREAD_UNCHANGED)
rgbd = RGBD(rgb=rgb, depth=depth, depth_min=depth_min, depth_max=depth_max)
rgbd.set_workspace(pts=workspace)

# load model
detector = create_and_load_order_detector()

# predict
detector.args.flag.show_steps = False
ret = detector.detect_and_show_suctions(rgbd=rgbd, net_args=detector.args.suction_net, rpn_args=detector.args.rpn,
                                        args=detector.args, remove_bg=False)

# show
cv2.imshow('reviewer', ret['im'][:, :, ::-1])
cv2.waitKey()

../_images/fig_orderpicking_sample_result.png

4. Parameters tuning

Parameter	Description
suction_net.pad_sizes	Patch sizes of suction pose scanning process. Larger value should be set for larger object
suction_net.stride	Stride of suction scanning process. Smaller value makes more suction candidates, but computational heavier.
rpn.enable	Enable RPN for suction detector or not
rpn.score_thresh	Score threshold of RPN
rpn.min_size	Minimum size of larger size of a detected object

Kimchi

Smart Gripper

Random Piece Picking

../_images/fig_multimode_grasp_diagram.png

1. Output format

Field	Description
target_grasp	target grasp selected among all detectors’ results
target_grasp_name	name of target grasp, can be: grip, suction, inner_grip, suction_2pts
im	display grip on the input image

2. Create and Load Detector

Using the default Kpick’s detector

from kpick.apps.randompicking.detector import create_and_load_random_picking_detector
detector = create_and_load_random_picking_detector(detector_type='suction', cfg_path=cfg_path)

Note

Please refer kpick/apps/randompicking/configs/*.cfg

3. Modify and Load Detector

Extending the default Kpick’s detector

from kpick.apps.randompicking.detector import create_and_load_random_picking_detector,
OnePointDualGraspDetector, OnePointDualGraspDetector, RandomGripDetector, RandomSuctionDetector
class AppDetector(RandomSuctionDetector):
    def new_function(self):
        print('new function')

detector = RandomGripDetector(Detector=AppDetector, cfg_path=cfg_path, detector_type='suction')

4. Demo on single RGB-D image

import cv2
from ketisdk.vision.utils.rgbd_utils_v2 import RGBD
import os
from ketisdk.vision.utils.rgbd_utils_v2 import RGBD
import kpick
KPICK_DIR = os.path.split(kpick.__file__)[0]

# load image
rgb = cv2.imread(os.path.join(KPICK_DIR, 'apps/randompicking/test_images/01_rgb.png'))[:, :, ::-1]
depth = cv2.imread(os.path.join(KPICK_DIR, 'apps/randompicking/test_images/01_depth.png'), cv2.IMREAD_UNCHANGED)
rgbd = RGBD(rgb=rgb, depth=depth, depth_min=600, depth_max=800)

# set crop roi
rgbd.set_workspace(pts=[(348, 193), (894, 194), (894, 594), (345, 590)])

# predict
detector = create_and_load_random_picking_detector(detector_type='grip')
detector.args.flag.show_steps = True
ret = detector.detect_and_show_grips(rgbd=rgbd, net_args=detector.args.grip_net,
                                     args=detector.args, remove_bg=detector.args.grip_net.remove_bg)

# show
cv2.imshow('grip', ret['im'][:, :, ::-1])

# predict
detector = create_and_load_random_picking_detector(detector_type='suction')
detector.args.flag.show_steps = True
ret = detector.detect_and_show_suctions(rgbd=rgbd, net_args=detector.args.suction_net, args=detector.args,
                                        remove_bg=detector.args.suction_net.remove_bg, rpn_args=detector.args.rpn)

# show
cv2.imshow('suction', ret['im'][:, :, ::-1])

# predict
detector = create_and_load_random_picking_detector(detector_type='one_point_dual')
detector.args.flag.show_steps = True
ret = detector.detect_and_show_multimode_grasp(rgbd=rgbd, grip_args=detector.args.grip_net,
                                               suction_args=detector.args.suction_net,
                                               inner_grip_args=detector.args.inner_grip_net,
                                               args=detector.args)

# show
cv2.imshow('one_point_dual', ret['im'][:, :, ::-1])

# predict
detector = create_and_load_random_picking_detector(detector_type='two_points_dual')
detector.args.flag.show_steps = True
ret = detector.detect_and_show_multimode_grasp(rgbd=rgbd, grip_args=detector.args.grip_net,
                                               suction_args=detector.args.suction_net,
                                               inner_grip_args=detector.args.inner_grip_net,
                                               args=detector.args)

# show
cv2.imshow('two_points_dual', ret['im'][:, :, ::-1])
cv2.waitKey()

../_images/fig_randompicking_sample_result.png

5. Parameters tuning

Parameter	Description
grip_net.erode_h	gripper plate’s width
grip_net.grip_w_ranges	ranges of gripper’s width, which are tuned for each target object
grip_net.grad_thresh	gradient threshold. Smaller value makes more grip candidates but noisy, and vice versa
grip_net.remove_bg	remove detected grips on the background or not
grip_net.top_n	top n target grips
grip_net.dy	Smaller value makes more dense grip candidates.
grip_net.ellipse_axes	Dimensions of ellipse that examines neighbor grips. For bigger object, dimension should be larger
suction_net.pad_sizes	Patch sizes of suction pose scanning process. Larger value should be set for larger object
suction_net.stride	Stride of suction scanning process. Smaller value makes more suction candidates, but computational heavier.
rpn.enable	Enable RPN for suction detector or not