UR5 Robot Vision Guided Grasp

Overview

Vision based grasping is mainly achieved by changing the tool coordinate system or base coordinate system of the robot. The change of the position of the object is mainly the change of the X, Y, Z direction of its position. The base coordinate system can define the user coordinate system as needed. When the robot is equipped with multiple tables, selecting the user coordinate system can make the operation easier. In real world applications, we usually need to convert the pose of objects in the external environment observed by the camera from the camera coordinate system to the coordinate system of the robotic arm to assist the robotic arm to plan some subsequent actions (such as grasping). In order to get the transformation matrix of the robot coordinate system and the camera coordinate system, we also need to perform hand-eye calibration on the robot.

Hand-eye calibration

The camera is fixed at the end of the robot. In this case, the calibration plate needs to be fixed on the ground, and the transformation relationship between the robot coordinate system and the calibration plate coordinate system remains unchanged. The amount to be solved is the pose relationship between the camera coordinates and the robot end coordinate system

Object recognition and localization

A very important part of visual grasping is the recognition of grasped objects. Whether it is a two-dimensional image or a three-dimensional point cloud, the corresponding function package can be found in ROS. If machine learning is required, it can also be easily implemented by integrating Tensorflow.

path planning

Moveit! provides developers with an easy-to-use integrated development platform. It consists of a series of functional packages for mobile operations, including motion planning, operational control, 3D perception, kinematics, control and navigation algorithms, etc. The implementation of these functional algorithms is integrated in Moveit! by means of plug-ins, and provides a friendly GUI, which can be widely used in industry, commerce, R&D and other fields.

The flow diagram is

Python UR5 demo

Contents