Skip to content

Visual_Servoing_Demo_en

Jump to bottom
Juan Rojas edited this page Dec 2, 2016 · 1 revision

<< back to Mainpage

EN | 中文

#Demo 6 - Visual Servoing Demo

#####Navigation

  1. Summary
  2. Quickstart
  3. Overview
  4. Structure
  5. Video
  6. FAQ

##Summary

The project is based on an worked example from WPI University at http://sdk.rethinkrobotics.com/wiki/Worked_Example_Visual_Servoing.

In this project, baxter is assigned the task of identifying a ball tray and some balls on the table, picking one of the balls and moving it to the ball tray. Baxter will cease the process after a maximum times or when it could not find any ball left on the table.

Files Location:

  • sysu_baxter/demos/visual_servoing/src/visual_servoing_setup.py
  • sysu_baxter/demos/visual_servoing/src/visual_servoing.py

Generated Files:

  • HOME/visual_servoing

##Quickstart

  1. Git pull the sysu_baxter repository and make sure the package visual_servoing with files visual_servoing_setup.py, visual_servoing.py are in sysu_baxter/demos.
  2. Run catkin_make in the workspace and source devel/setup.bash.
  3. Beofore running this node, make sure your computer have been connected to the baxter.
  4. Setting up the arm to be used and measure the base distance to the table: $ rosrun visual_servoing visual_servoing_setup.py
  5. Starting the main process: $ rosrun visual_servoing visual_servoing.py

Prerequisites:

  • ROS Indigo
  • Baxter Research Robot [Firmware >= v0.7.0]
  • Workstation [RSDK >= v1.1.1]
  • Rethink Robotics Electric Parallel Gripper
  • Table
  • Ping Pong Balls
  • A White Sheet Of Paper With Its Border Lines Marked in Black

##Overview

The general idea of the Visual Servoing Demo source code:

In visual_servoing_setup.py:

  1. Set the baxter arm to be used (using one arm).
  2. Use the IR sensor of the arm to record the distance from the hand camera to the table.

In visual_servoing.py:

  1. First, move the arm to the a fixed position using IK server.
  2. Take images of the ball tray using the camera of the hand and convert them to grayscale images. Use the Canny-Edge-Detect algorithm from openCV to analyze the grayscale image. Locate the ball tray and its center then convert the corners of the tray to pixels, or else adjust the position of the arm using IK server and repeat this step until a tray is found.
  3. Move the arm to a fixed position where the ball is supposed to be using IK server.
  4. Take images using the camera of the hand and convert them to grayscale image. Analyze the image using Hough-Transform algorithm to detect the balls. The program terminates when no ball is found.
  5. Select one of the balls, then calculate the distance between it and the hand camera in x-y plane. Adjust the position of the arm using IK server so that the camera is right above the ball.
  6. Move the arm downward, close the gripper to grasp the ball, and move the arm upward. Move the arm to where the ball tray is and open the gripper to place the ball in the tray.
  7. Repeat step 3 unless no ball is found or the loop has reached its limit (100 times).

##Structure

  • visual_servoing_setup.py
  • visual_servoing.py

###visaul_servoing_setup.py

####Primary Parameters

  • self.pose(self.x, self.y, self.z, self.roll, self.pitch, self.yaw): start position of the arm
  • self.distance: distance to the table

####Primary Structures

  • (class) golf_setup:
    • (function) __init__: initialize the parameters
    • (function) baxter_ik_move: move the arm according to self.pose
    • (function) get_distance: find the distance from the hand camera to the table using IR sensor
    • (function) save: save the configuration into a file
  • (function) main: setup

###visual_servoing.py

####Primary Parameters

  • positions:
    • self.ball_tray_x/y/z: position of the ball tray
    • self.golf_ball_x/y/z: position of the balls
    • self.roll/pitch/yaw: pose of the arm
  • tolerance:
    • self.ball_tolerance: ball position accuracy
    • self.tray_tolerance: ball tray position accuracy
  • camera parameters:
    • self.cam_calib: hand camera calibration
    • self.cam_x_offset, self.cam_y_offset: offset from the camera to the gripper
    • self.width/height: camera resolution
  • hough transform parameters:
    • self.hough_accumulator
    • self.hough_min_radius
    • self.hough_max_radius
  • canny transform parameters:
    • self.canny_low
    • self.canny_high
    • self.min_area: minimum ball tray area

####Primary Functions

  • (function) get_setup: read the setup configuration file saved in visual_servoing_setup.py
  • (function) main: get setup parameters and launch the visual servoing process
  • (class) locate: baxter's main manipulator
    • (function) __init__: initialize the parameters
    • (function) baxter_ik_move: move the arm to the specified place
    • (function) cv2array: convert cv image to a numpy array
    • (function) get_distance: find distance of limb from nearest line of sight object
    • (function) update_pose: update pose in x and y direction
    • (function) print_arm_pose: print all 6 arm coordinates
    • (function) splash_screen: change the information displayed on baxter's head display
    • locate the ball tray:
      • (function) find_ball_tray: find the tray using Canny Edge Detect Algorithm and find the corners of the ball tray
      • (function) ball_tray_iterate: used to place camera over the ball tray
      • (function) dither: used to prevent stalemate when looking for ball tray
      • (function) canny_it: Canny Edge Detect function
      • (function) find_corners: find four corners of the ball tray
      • (function) find_places: find places for balls
      • (function) look_for_ball_tray: remove artifacts and find largest object
      • (functino) flood_fill_edge: flood fill edge of image to leave objects
    • find ball, pick all and place ball:
      • (function) pick_and_place: find all the balls and place them in the ball tray
      • (function) hough_it: use Hough circles to find ball centres
      • (function) golf_ball_iterate: used to place camera over golf ball
      • (function) find_next_golf_ball: find next object of interest
      • (function) find_gripper_angle: find best gripper angle to avoid touching neighbouring ball
      • (function) is_near_ball_tray: if ball near any of the ball tray places
    • camera functions:
      • (function) reset_cameras
      • (funciton) open_camera
      • (function) close_camera
      • (function) camera_callback
      • (function) left/right/head_camera_callback
      • (function) subscribe_to_camera
      • (function) baxter_to_pixel: convert Baxter point to image pixel
      • (function) pixel_to_baxter: convert image pixel to Baxter point

##Video

IMAGE ALT TEXT

##FAQ

Q1: Cannot run the visual_servoing_setup.py and visual_servoing.py
A1: Please make sure you have been connected to the baxter using rostopic list. If you have properly connected to the baxter, then enter visual_servoing/src in command line and run chmod +x visual_servoing_setup.py and chmod +x visual_servoing.py.

Q2: When running the program in gazebo simulator, it shows [WARN] [WallTime: 1436414658.281419] [20.609000] left_gripper custom - not capable of 'calibrate' command
A2: Since the Baxter gazebo model does not have gripper, the program currently could only run on real baxter.

https://github.com/birlrobotics

Clone this wiki locally