Group05 Project
From Experimental Robotics
Contents |
Group05 Project - Robot Wars
 |
 |
 |
Group Members
- Matthew Nigl (mjni765)
- Jeremy de Oliveira-Kumar (jdok706)
Introduction
Goals
The original aim of the project was to develop robots that could search, aim and fire at each other autonomously using a USB missile launcher and the pioneer robots. The key goals were to:
- Make the robots capable of simple searching
- Be able to define where in the room the target robot is located
- If necessary, move to, and then take aim at the robot
- Fire accurately at the target robot
Motives for the task
The key motivations for tackling this project were:
- Learn about computer vision and techniques that could be used to identify a target robot
- Learn and understand some low level programming for different types of hardware
- Attempt to find a way of translating from a 2D image of a 3D landscape and identify the location of an object based on the data provided in the 2D image
- Fire at targets for the fun of it
Plan of Attack
Due to the fact that our task was quite ambitious given the restricted time within which we could complete our goals, we decided to try and break-down the task into a set of clear cut modules. The original plan was that we would tackle each module, one at a time in sequential order and try and get as far as possible within the session. Our original hopes were that we could tackle the first three tasks as best as possible.
The modules, written near the beginning of the session, were divided as follows:
1. Observe an environment and be able to quickly detect all friends, enemies, obstacles and possible locations for enemies. This process should continue during movement where possible to create a mapped environment.
2. Follow enemies (targets) or friends (if team play) and get to an enemy target (moving and non-moving) as fast as possible. May be able to use different techniques such as the Voronoi diagrams to create obstacle free paths that are efficient. (This module changed slightly from actual movement to use of the camera and launcher to follow)
3. Using the rocket launcher, accurately fire missiles at particular targets in the following stages:
- Stationary fire at stationary objects
- Moving in a line fire at stationary targets on a wall
- Stationary fire at moving targets
- Moving, fire at moving targets
There are plenty of unix/linux drivers using usbhid to control the missile launcher. Further to this, the missile launcher is capable of providing feedback for when the launcher reaches its maximum rotation/panning limits.
From this, if time permits, we will:
4. Develop strategies based on general military tactics for
- Defensive manouveurs
- Avoiding missiles
- Offensive approaches
- Choice of targets
Other Ideas
If we had time and managed to tackle many of our goals quickly, we also were looking at the following options:
- We may use SLAM if it is easy enough to implement without much coding.
- We may employ supervised re-inforcement learning to improve the behaviour of the robots.
- We may perform two-on-two tag team style war games that will therefore require communication and as a result tactics to reflect this.
We identified some basic knowledge i.e. we know that:
- The Pioneer robots have a camera, laser sensor and sonar sensors
- Movement is either forwards or backwards, turning is simple although not necessarily possible on the spot - need to find out if wheels are independent of each other as far as movement is concerned.
- We will use some form of targets (foam balls or pom-pons?) to identify robots
If we were not able to tackle these further problems, we thought that these would be good for suggestions for future work.
Basics
Setting up player
In order to carry out any of the work required for the project, we needed to establish a basic configuration for player that could be used to take advantage of both the camera (in terms of control and feedback) as well as movement and sonar.
The use of a laser was not available to us due to the fact that the particular robot we would be using (mawson) had its laser connected to a mini-usb cable that we would need to use for the rocket lanucher. As a result, our options for movement were significantly hampered with respect to sensing obstacles.
Nonetheless, we used the following configuration to carry out the basic tasks we would require:
driver ( name "p2os" provides ["odometry:::position2d:0" "sonar:0" "aio:0" "dio:0" "power:0" "bumper:0" "gripper:::gripper:0"] port "/dev/ttyS0" ) driver ( name "sonyevid30" provides ["ptz:0"] port "/dev/ttyS1" ) driver ( name "camerav4l" provides ["camera:0"] port "/dev/video0" source 0 size [320 240] norm "pal" ) driver ( name "cameracompress" provides ["camera:1"] requires ["camera:0"] ) driver ( name "vfh" provides ["position2d:1"] requires ["position2d:0" "laser:0"] safety_dist_0ms 0.01 safety_dist_1ms 0.01 distance_epsilon 0.05 # angle_epsilon 5 ) #driver #( # name "urglaser" # provides ["laser:1"] # port "/dev/ttyACM0" #)
Hardware, Adapters and Drivers
USB Missile Launcher
We used the dream cheeky USB standard missile launcher in this project. There were several options available to us, however, it was felt that this was the best both with respect to knowledge about how the launcher works at the lower level but also because of the availability of open source sample drivers by people who had attempted to develop a means of using the launcher on other platforms apart from Windows XP for which the original software is written.
The Dream Cheeky USB Missile Launcher MK II (Jaycar GE4074) fortunately has easy to use linux drivers and a python interface that someone has already put toether (pyrocket - http://code.google.com/p/pyrocket). However, we will need to modify or write some code from scratch using the knowledge from these drivers as many of them are quite simple and function based on human control rather than utilising function calls for movement, firing, end-point feedback and so on.
One of the best sources of information with respect to understanding how the launcher worked was the project developed for MAC OS X, namely the NZ missile lanucher software. This provided us with several ideas for optimising the performance and use of the launcher, particularly priming the launcher as will be discussed further on.
USB Adapter
One of our greatest problems was finding a way to take the mini usb female socket provided on mawson and finding a way to connect it to the female plug given as a connection from the USB missile launcher. Unfortunately, our first attempts at using a USB hub or similar device were hampered as the output (feedback) and behaviour of the launcher seemed to be erratic whenever a hub was placed between the launcher and the motherboard's USB inputs. In fact, many people had commented on various blogs on the Internet about problems with the launcher's behaviour when a USB hub was used as the medium for connecting the launcher to a computer's USB input.
We therefore decided to investigate options for direct connections between the usb socket and our rocket launcher. The best option that we could find was to have some sort of connector or adaptor that would allow mini-usb female to usb female connections. No manufacturer in Australia stocked such items to our great disappoint. However, a bit of Internet searching, particular on ebay led us to a few manufacturers overseas in Hong Kong and the USA that supplied just the adaptor we were looking for:
http://www.expansys-usa.com/p.aspx?i=160989
The device we ended up obtaining from Hong Kong via ebay took about 1 week for delivery and looked something similar to this:
This device successfully achieved a connection between the launcher and the Pioneer motherboard by way of the mini-usb cable that had been previously installed, therefore allowing us to write and modify drivers to acommodate the usage of the launcher.
Observing and Identification
Initial Work: OpenCV
Initial Thoughts and Aims
In our attempts to find a way to identify a robot within a busy field, we decided that we might try and take advantage of the OpenCV framework and in particular the Haar Classifier available as part of the framework's many features. In order to help identify the shape and features of the robot, particularly the camera "head" of a Pioneer robot, we decided to place a can covered in coloured paper as a kind of "marker" or "feature" of the item we would be searching for within the image space.
As with most machine learning problems, we needed to commence with sufficient training and testing data using a broad range of possible identifiable images. To this end we collected over 2000 images of positive and negative training images to begin training and testing the classifier. Many of these images contained distant shots as well as near close up shots of the head of the robot (our positives) as well as images where there were simply cans or no robot visible within the image (negatives). We believed that with this initial training set and further training down the track we would be able to build a sufficient amount of confidence into the identification/classification of true positive images.
Some example images and pre-processing our shown below:
 A sample positive image |
 A sample negative image |
A copy of the sample code for use in classification (particularly face detection) can be found on the OpenCV site: Sample Face Detection
Guides for training are also available online:
These guides were used for the initial process of training a classifier to carry out the task of identifying robots in our sample training images. We did commence training of the Haar classifier, however, when discussing our approach with the supervisors, we came to the conclusion that this might not be the most effective way to go about identifying a target robot for 3 key reasons:
- colour and shape recognition could be significantly difficult given the noise that could be possibly found in an image
- it would be difficult to carry out training on a complex shape within the small time frame
- tracking could be difficult even once we had identified a robot, particular during movement of either the target or our own robot
There were some possible alternatives to combat this problem such as:
- using OpenCV to identify specific colours
- using a grayscaled image or black and white image to look at simply identifying a rough shape of the robot or
- using a more specific shape such as target image attached to the robot
Some of our initial experimentation and ideas with colours and lighting are shown below, the first image isolates yellows, the second and third look at isolating different lighting and contrast within the full spectrum of the image (it should be noted that most of this experimentation was done outside OpenCV during the initial phases of the project):
Approach number three looked attractive as it would simplify our task significantly and would also help with initial testing of our ability to locate the target effectively. With the advice provided to us by the supervisors, we decided to take this approach and make use of a recommended toolkit for target locating and tracking. More on this later, but first we will examine how we went about image collection, which played an important part in setting up a simple interface for testing the reliability of our camera feedback.
For the Haar classifier, we had followed some of the work from the OpenCV Guide on the wiki for the Haar classifier. We put a green-coloured can on the top of the robot (on top of the camera) in order to recognise the target. However, Claude quickly advised us to use ARToolkit instead due to the amount of time it takes to train the classifier, and the difficulty in getting an accurate prediction. In hindsight, we aren't confident with how it would have worked trying to recognise a green can in the lab with a green soccer field and a fairly low quality camera. We captured approximately 2000 images from the camera (1000 positive and 1000 negative) before abandoning the idea. However, this is not to say that the method would not be as good (see Urban Challenge 2008 for one implementation) - we just didn't have enough time and ARToolkit was easy to use once issues with the camera and software were figured out.
Obtaining training data and getting visual feedback
One of the critical elements in the initial phase of training data collection as part of our original plan to classify data and additionally an important step in the process of obtaining visual feedback to see whether we were getting the desired results from the robot was to use a small visual interface that utilised OpenCV.
Our work for this part of the project was based on the sample code from OpenCV on how to create a basic interface and we further developed this to deal with certain circumstances such as pulling data rather than pushing, performing basic searching with the camera concurrently and additionally saving images every nth frame.
The code for our program, conveniently named capture.cc is shown below:
#include <stdio.h>
#include <math.h>
#include <libplayerc++/playerc++.h>
#include <time.h>
#include "cv.h"
#include "highgui.h"
#include <stdio.h>
using namespace PlayerCc;
int direction;
double currentPan;
double currentTilt;
double currentZoom;
void cameraSearch(PtzProxy* camera);
int main(int argc, char *argv[])
{
PlayerClient* robot = new PlayerClient("localhost");
CameraProxy* camera = new CameraProxy(robot, 1);
PtzProxy* camcontrol = new PtzProxy(robot, 0);
// Change the data mode to pull
robot->SetDataMode(PLAYER_DATAMODE_PULL);
robot->SetReplaceRule(1, -1, -1, -1);
direction = 0;
//Allocate enough memory for the image data buffer
// uint8_t* imgBuffer = new uint8_t[cam_width * cam_height * cam_depth];
uint8_t* imgBuffer;
IplImage* frame;
// Create a window in which the captured images will be presented
cvNamedWindow( "mywindow", CV_WINDOW_AUTOSIZE );
camcontrol->SetCam(0.0, 0.0, 0.0);
int counter = 0;
// Show the image captured from the camera in the window and repeat
while(1) {
//while (!robot->Peek(0))
// printf("No data\n");
cameraSearch(camcontrol);
//Read from robot
robot->Read();
// Decompress image and copy it into the image buffer
camera->Decompress();
//Get the capture information
uint cam_width = camera->GetWidth();
uint cam_height = camera->GetHeight();
uint cam_depth = camera->GetDepth();
counter++;
printf("Image %d\n", counter);
if (cam_width != 0 && cam_height != 0) {
// Assign buffer and obtain image
imgBuffer = new uint8_t[camera->GetImageSize()];
camera->GetImage(imgBuffer);
//Create an OpenGL image structure to hold image
frame = cvCreateImage(cvSize(cam_width, cam_height), IPL_DEPTH_8U, 3);
// Copy the data from the image buffer into the frame
for (uint i = 0; i < cam_width; i++) {
for (uint j = 0; j < cam_height; j++) {
frame->imageData[cam_width * j*3 + i*3 + 0] = (char)imgBuffer[cam_width * j*3 + i*3 + 2];
frame->imageData[cam_width * j*3 + i*3 + 1] = (char)imgBuffer[cam_width * j*3 + i*3 + 1];
frame->imageData[cam_width * j*3 + i*3 + 2] = (char)imgBuffer[cam_width * j*3 + i*3 + 0];
}
}
cvShowImage( "mywindow", frame );
// Do not release the frame!
}
delete[] imgBuffer;
if (counter % 3 == 0)
{
char* outFileName = new char[50];
sprintf(outFileName, "./images/%d.jpg", counter);
if(!cvSaveImage(outFileName,frame)) printf("Could not save: %s\n", outFileName);
delete[] outFileName;
}
}
//Release memory when finished processing
cvReleaseImage(&frame);
// Release the capture device housekeeping
cvDestroyWindow( "mywindow" );
return 0;
}
Below is a screenshot of the code in action:
An Alternative Approach: ARToolkit
Following some further research on possible approaches to solving our identification and tracking problem, including looking at the possibility of using the blob detection to identify key elements such as coloured cans or targets (OpenCV Blobs), we decided to move forward with the ARToolkit (ARToolkit) as it seemed this approach would be quite simple to implement.
We felt it would be effective in achieving the results we were looking for and be an excellent basis upon which we could then focus on the targeting and aiming with the launcher based on camera feedback data without the added complication of expensive and difficult image processing.
Choosing a suitable target image
The ARToolkit identifies a target "marker" based on a pattern file that is created by training the classifying program that comes with the toolkit (mk_patt). Training and testing took a number of forms and was carried out using the following methodologies. First of all we took a sample image from the toolkit itself (the typical "hiro" example). Using this example we experimented with how the training program works and how we could utilise the resulting pattern within our targeting software.
Once the basics were established with respect to training and testing, we began experimenting with different patterns in the following manner, beginning with an arbitrary pattern (bulls eye target image):
- Train the ARToolkit classifier to identify the pattern and create a pattern file (this might be done several times based on initial testing of marker identification using the simpleTest program)
- Load the pattern file onto our live robot using the base software written for tracking, following and eventual aiming/firing
- Experiment with identification in different lighting environments and different backgrounds (particular noisy backgrounds)
- Modify our choice of target based on the average confidence achieved, the number of false positives we identified and our assumptions about the results
The following images were used in the process of training and testing to find a high performing target image:
The best 2 performing images (the eventual choice coincidentally being that on the left) were:
The following table outlines the average confidence achieved from our testing and also provides the number of false positives during our standard 5 trials (each being a total of 5 minutes of movement, experimentation, hiding targets, partial target visibility etc.). We also devised a failure rate (similar to an error rate), basically being a situation where the target should most likely be identified but is not picked up by the classifier/tracker. The failure rate is measured as a percentage based on the number of total possible frames identifiable and the number of frames where identification of the marker did not take place:
| Original Target | Square Target | Diamond Target | Pattern Target | Bulls Eye Target | |
|---|---|---|---|---|---|
| ~Avg. Confidence (0 - 1) | 0.6 | 0.95 | 0.7 | 0.45 | 0.78 |
| Total False Positives | 72 | 212 | 56 | 43 | 46 |
| Failure Rate | 23% | 6% | 37% | 18% | 7% |
Training and Testing ARToolkit
Training using ARToolkit is quite simple and inexpensive in terms of both time and resources.
Key things to note when training however:
- You must get the red corner to be in the top left-hand corner if you are using an asymmetric image
- It is best to hold the image at a slight angle to take into account possible lighting differences
- Several training runs should be able as the resulting pattern data can differ across different training trials
The images below show some of the training and testing that took place:
This image presents the use of mk_patt to train the ARToolkit classifier to identify an image
This image shows some of the testing we used based on the simpleTest and simpleLite programs as well as our own code to test the viability of the chosen target image:
Of Importance
As a result of the fact that the particular version of ARToolkit that we were able to get a hold of was only is C (rather than C++ which was the original language used for our initial work in player and OpenCV), we had to convert all our existing code to C. This meant learning and understanding player's interfaces and framework for libplayerc. It also meant that all our libusb coding would need to take place in C as opposed to C++ (although this was not as big an issue as originally thought).
Camera Searching
Once we were happy that we had established a target good enough for identification and tracking using the ARToolkit, we went about testing, tuning and refining our camera search code from the original, basic left to right, right to left scan, to a scan based on some of our knowledge about our requirements and the total range we wished to work within.
Presented below is the original search code that we compiled that performs a simple scanning pan search:
void cameraSearch(PtzProxy* camera) {
// Select Control Mode of Camera
// camera->SelectControlMode(PLAYER_PTZ_POSITION_CONTROL);
double pan = camera->GetPan();
double tilt = camera->GetTilt();
double zoom = camera->GetZoom();
// camera->SetCam(pan, tilt, zoom);
if (direction == 0)
camera->SetCam(pan+=0.1, 0, zoom);
else if (direction == 1)
camera->SetCam(pan-=0.1, 0, zoom);
printf("Pan: %f\n", currentPan);
printf("Direction: %d\n", direction);
if (direction == 0 && currentPan >= 1.5f)
direction = 1;
else if (direction == 1 && currentPan <= -1.5f)
direction = 0;
currentPan = camera->GetPan();
currentTilt = camera->GetTilt();
currentZoom = camera->GetZoom();
}
Taking this code that we had written, we then rewrote it to work in C using the libplayerc libraries and added the following features that would assist in better search and tracking:
- Have a range restricted to 180 degrees to allow for accurate movement in line with the missile launcher code to be written
- Additional code to change search direction if a possible positive was identified based on weaker confidence values given by ARToolkit tracker or to deal with larger than expected movements by the camera or overreaching by our own code
In addition as can be seen in the code provided below that is basis for the final version we used:
- Keep track of the current Pan position for use in calculations of marker positioning in real 3D space
- Adjustment of launcher according to estimated translation between degrees and "clicks" - the number of individual motor movements in the total range of the launcher
The resulting code is found below:
void cameraSearch(playerc_ptz_t* camcontrol) {
previousPan = currentPan;
currentPan = camcontrol->pan;
currentTilt = camcontrol->tilt;
currentZoom = camcontrol->zoom;
// printf("Pan: %f\n", currentPan);
// printf("Direction: %d\n", direction);
if (direction == 0 && currentPan >= 1.5f)
direction = 1;
else if (direction == 1 && currentPan <= -1.5f)
direction = 0;
if (direction == 0)
playerc_ptz_set(camcontrol, currentPan+=0.2, 0, currentZoom);
else if (direction == 1)
playerc_ptz_set(camcontrol, currentPan-=0.2, 0, currentZoom);
campanDegrees = -1 * camcontrol->pan * RAD_TO_DEG;
if (launcherHoriz / (float)CLICKS_PER_DEGREE > campanDegrees)
move(LEFT, launcherHoriz - campanDegrees * CLICKS_PER_DEGREE);
if (launcherHoriz / (float)CLICKS_PER_DEGREE < campanDegrees)
move(RIGHT, campanDegrees * CLICKS_PER_DEGREE - launcherHoriz);
pauseTemp(50000000);
}
Following targets
Centreing on the marker and making aim
Much of work on centreing on the marker was also based on assumptions made about rocket launcher positioning, positing from the 2D view to the 3D world as well as of course, simple positioning within the actual camera view itself. The following section examines some of the work we did to try and translate between the 2D image and the 3D world.
Horizontal Axis
Before attempting to centre the camera on the target, we had originally estimated the angle of the target relative to the bottom-centre of the camera. This gave us the angle of the target as if it were at the same height as the bottom of the camera lens. The math for the calculation is represented in the image below.
However, we needed to know the angle of the target as if it were on the ground (which it is). In order to do this, we came up with the idea of 'extending' the image so that it appeared taller (though not wider). This was to simulate the camera taking images from the perspective of the ground. The logic behind this idea is that if the target is at the bottom of the image, then it would move towards the top as the camera is lowered towards the ground. This affects the angle alpha calculated above, as it will always predict that the angle is larger than it should be. If the image is 'extended' by a certain number of pixels, delta y, then this should make the angle more acute, and hence closer to the real value. This is represented in the image below.
The value for delta y could then be determined experimentally, as it should be proportional to the height that the camera is off the ground. This is represented in the image below, where h is the distance between the bottom of the camera and the ground.
This method had worked somewhat for a while, and then something changed which caused it to stop working. Although there is likely something wrong with this method, it did seem to work for some time, and may prove useful for other students in future. Upon talking with some of the students from Robocup 2009, we discovered that they use a similar method. Unfortunately, it was already too late in this project so we were not able to communicate with them to find out how they solved this problem. We highly recommend consulting with the Robocup team in the future in order to find out more information. We did find out that our calculations need to involve the aperture of the camera, though we could not find any literature advising how to do this.
Vertical Axis
In the final implementation, the vertical axis of the missile launcher was not changed from our default value. However, changing the value would have improved accuracy particularly when the target robot was closer; often the missile would fire over the top of the target robot. Due to the inaccuracy of the foam missiles and the inaccuracy of marker detection from ARToolkit, it seemed futile to handle cases where the target robot was far away. Regardless, we had thought about how to modify the angle of the missile launcher on the vertical axis, and our ideas are presented below.
Modifying the angle of inclination of the rocket launcher is a simple application of projectile motion. In order to predict the distance a projectile would fly, you must know the angle of fire, the initial velocity of the projectile, the distance the projectile travels, and/or the time it takes for the projectile to land. In our case, the specifications for the initial velocity of the missiles were not provided. This could be determined experimentally by setting the angle of fire to a known value, firing the missile, timing how long it takes to land and how far it lands from the robot, and then working backwards to find the initial velocity. The initial velocity should be relatively constant depending on the design of the missile launcher. Once this value is known, the angle of fire can be determined based on how far away the target robot is estimated to be. The idea is illustrated in the picture below, where gamma is the angle of fire, and d is the estimated distance between the hunter and the target.
Some methods for estimating the distance to the target robot are given below:
- Find the distance to the robot using sonar and/or laser (the laser on our robot required USB, though we needed the USB for the rocket launcher, so we only had sonar). The target robot may be detected by observing the images from the camera, estimating the angle the target robot is relative to the camera, and taking the sonar/laser value from the corresponding sensor at that angle. e.g. supposing sonar was all around the robot, then divide 360 degrees by the number of sonar sensors, find the estimated angle of the target relative to the hunter (e.g. 40 degrees), and read the distance value from the sensor which contains that angle within its range (e.g. if sensor 1 can detect 0-15 degrees, sensor 2 can detect 15-30 degrees, and sensor 3 can detect 30-45 degrees, then read sensor 3 data). Note that this will only work if the camera is sufficiently close to the centre of the robot. Looking at the image below, if beta is the estimated angle from the camera (which is NOT from the centre of the robot), and the red dots represent various positions of the target robot, then it is obvious that the angle between the green lines and the centre of the robot vary depending on the distance of the target, even though the angle is always correct for the camera. This means it is not possible to know which sonar sensor will detect the target robot unless the coordinates are able to be converted to the centre of the robot (e.g. if the camera was located in the direct centre). It may be possible to read values from the sensors to detect which is sensing movement, but this is of limited value once the hunter robot also begins to move.
- Use the fact that the size of the marker in the camera's image is a function of how far away the marker is. i.e. area = function(distance). Obviously if the area of the marker appears large in the image, then it must be close. Similarly, if the area of the marker appears small in the image, then it must be further away. There is a variable called 'area' in the ARMarkerInfo structure within ARToolkit, which provides an estimate of how large the marker appears within the image (in pixels). The distance function could then be determined experimentally by looking at the area values when the marker is set at different distances. However, this may be affected by the orientation of the robots. i.e. if a measurement of the area of the marker is taken whilst the marker is directly facing the camera, it may be larger than if the marker is slightly on an angle - the area the marker would take up would then appear smaller since it is not head-on, even if it is the same distance away in both measurements. This could possibly be solved by using the 'vertex' or 'line' variables within the ARMarkerInfo data structure. If the data does not show that the marker is a rectangle, then it must be on an angle to the camera. It may be possible to estimate what the size of the marker would be if it was facing the camera, and use this to estimate how far away the marker appears to be.
Following target position using the camera
Once we had established a method for centring in our target, we needed to look at ways of dealing with moving targets. For this we took several approaches to the problem. The first of these was to simply adjust east or west in relation to our current view based on movement of our target when the camera was stationary.
However, it soon became clear that this would not be sufficient as we would need to take into account any movement of the camera whilst tracking our moving target. In order to resolve this problem, we made use the knowledge of our movement combined with the knowledge of the previous position of target to attempt to readjust the camera as appropriate. For example if movement of the camera was one direction but it was clear that movement of the target appeared not be closer to the centre, then a direction change would occur.
Conversly, if movement was in a certain direction and the distance from the centre was the same or slightly less, then it was obvious that further movement in the same direction would be required to reach the target. This did result in overshooting and plenty of readjustments that we had to compensate for. We did this by creating a restricted range (5-10 degrees either way during testing) in which small movements would be made to compensate for readjustments of the camera. The idea being that little adjustment would need to be made within a 10 degree range of the notional centre of the image with respect to the target position.
It is important to note here that we make use of the confidence levels of the ARToolkit classifier/tracker in our most recent version. The reasoning behind this is we do not want to make too many adjustments if we are not sure that we have actually found the target. When dealing with very noisy spaces (as one such example) we found a lot of compensation happening in our initial versions and could not put our finger on the reasoning behind this. With further testing and outputting of debug data, we realised that the confidence value plays a very important role in identifying real targets (positives) from false positives.
Presented below is some of the source code behind the ideas handled here:
printf("Found marker camera at: %d launcher at: %d with xoffset: %f with confidence: %f\n", (int)campanDegrees,
(int)(round(launcherHoriz / (double)CLICKS_PER_DEGREE)),
xoffset, (marker_info->cf));
if (marker_info->cf >= 0.8f) {
// printf("Found marker x pos: %f x offset: %f y pos: %f y offset %f angle: %f camera at: %f\n", x, xoffset,
// y, yoffset, alphaDegrees, campanDegrees);
if (fabs((int)round(launcherHoriz / (double)CLICKS_PER_DEGREE) - (int)campanDegrees) <= 0 &&
xoffset >= -10.0 && xoffset <= 10.0) {
printf("Fire\n");
fire();
prime();
}
}
if (marker_info->cf > 0.1f) {
// Found but want to centre
if (xoffset < -10.0) {
playerc_ptz_set(camcontrol, currentPan+=0.015 + fabs((xoffset + deltaxoffset) / 320 * 0.07), 0, currentZoom);
direction = 0;
}
if (xoffset > 10.0) {
playerc_ptz_set(camcontrol, currentPan-=0.015 + fabs((xoffset + deltaxoffset) / 320 * 0.07), 0, currentZoom);
direction = 1;
}
if (launcherHoriz / (float)CLICKS_PER_DEGREE > campanDegrees)
move(LEFT, launcherHoriz - campanDegrees * CLICKS_PER_DEGREE);
if (launcherHoriz / (float)CLICKS_PER_DEGREE < campanDegrees)
move(RIGHT, campanDegrees * CLICKS_PER_DEGREE - launcherHoriz);
}
else {
prevConfidence = marker_info->cf;
cameraSearch(camcontrol);
}
Controlling the launcher and Firing
This section covers some of the more gruelling but also some of the best fun out of the work that we had to cover as we got to actually see our work in action. In figuring out how to move the launcher and fire, a lot of research was required on existing linux drivers, most of which took the following paths:
- inserting a module into the kernel
- utilising python and libusb
- utilising c/c++ and libusb
Since most of our code was now in C, it seemed most appropriate to take a look at libusb programming in C and try to understand how we could send and receive messages with the launcher.
Understanding the USB Launcher device
Most of the important work was about finding out what sort of feedback we could get from the launcher. There was already plenty of information on the Internet about determining how to control the missile launcher but very little information about the actual feedback that we could get back from the launcher. From early testing with the launcher and writing a basic launcher program whilst reading up on work that other people had done with the launcher, it became clear that the only feedback we could receive from the launcher was the following:
- When the launcher hit a limit point in its range (far left, far bottom etc.)
- When the launcher was in the process of firing
Given that we had this restricted amount of feedback, it became clear that it would be very difficult to establish a method of firing accurately. However, we decided that given the fact that we knew there was a determinable range of the launcher and that the speed of movement of the launcher's motor was usually about the same, we decided to try and run some tests moving the launcher through its full range left and right as well as up and down with some range testing programs.
We determined range as the number of feedback "clicks" that the launcher gave back to us during a particular session. The results were positive with the number of clicks generally being with a range of about 50-100 clicks in left-right motion and 10-20 clicks moving up and down. One thing we did notice that there was variance dependent upon the direction you were moving as well as based on the amount of usage of the motor (as the motor heated up, the performance of the launcher improved). This caused some difficult but interesting variables to take into account. We took a general average of clicks in both ranges and came up with the following numbers:
#define FULL_RANGE_LEFT 2230 #define FULL_RANGE_RIGHT 2130 #define FULL_RANGE_UP 220 #define FULL_RANGE_DOWN 215 #define FULL_RANGE_CLICKS 2230 #define FULL_RANGE_CLICKS_VERT 220 #define FULL_RANGE_DEGREES_HORIZ 315 #define CLICKS_PER_DEGREE 7
Notice that we chose to take the left and up ranges as our given ranges for the robot. This is because we decided we would need to initialise the robot's position as there was no way to find out when we were in the centre of the range. To do this, we would send the launcher to the bottom left-hand corner and then based on the ranges, we would move the launcher half way in each axis. The end result was that the launcher would generally hit close to or on centre. There was usually an error for about 2-3 degrees when the motor was first used and after excessive use.
To write the actual interface for controlling the launcher, we needed to take advantage of libusb and busses. One of our initial problems with this was that we did not have permission to write to the usb devices, nor read from them either. This was given to us by way of user access through the "plugdev" group.
We could then establish a connection to the launcher in the following manner:
- Initialise the libusb interface for our use
- Establish which busses exist (similar to running lsusb)
- Establish what devices are currently connected
- Loop through busses and devices until we find our device
- Ensure that the kernel has not grabbed our device by perforing a detach (we had this problem several times particular on the robots)
- Claim the interface for our use and check that we have a valid descriptor
Notice in the code below that we were able to determine the device descriptor and vendor as being the hex code 0x6465. The code below demonstrates how you can achieve this using libusb:
struct usb_bus *busses;
usb_init();
usb_find_busses();
usb_find_devices();
busses = usb_get_busses();
struct usb_bus *bus;
int c, i, a;
/* ... */
for (bus = busses; bus; bus = bus->next)
{
struct usb_device *dev;
for (dev = bus->devices; dev; dev = dev->next)
{
/* Check if this device is a printer */
if (dev->descriptor.idVendor == 6465)
{
launcher = usb_open(dev);
printf("Launcher: %x\n", launcher);
//do stuff
usb_detach_kernel_driver_np(launcher, 0);
int claimed = usb_claim_interface(launcher, 0);
printf("Claimed Launcher?: %d\n", claimed);
...
The results of our research and testing for feedback using libusb generated the following directional information and feedback information that we could get back from the launcher, notice that feedback was mixed aross two result chunks:
int up = 1; int down = 2; int left = 4; int right = 8; //result[0] int upf = 0xffffff80; int downf = 0x40; //result[1] int firef = 0xffffff80; int leftf = 0x04; int rightf = 0x08;
Adjusting the launcher with the camera
One strange anomaly that we came across was that the launcher required different behaviour on different devices. For example, on a standard PC or Mac, data could be obtained by grabbing data chunks from the device through libusb. By contrast, the Pioneer robots required that we obtain feedback by way of interrupt calls to the device.
The actual process of sending messages and handling movement was covered by the following two functions that we derived from a sample program appropriately named "rocket" found on the Internet. Essentially, simple calls are made to libusb to send data in a particular format as governed below:
int send_message(char* msg, int index)
{
int i = 0;
int j = usb_control_msg(launcher, 0x21, 0x9, 0x200, index, msg, 8, 1000);
//be sure that msg is all zeroes again
msg[0] = 0x0;
msg[1] = 0x0;
msg[2] = 0x0;
msg[3] = 0x0;
msg[4] = 0x0;
msg[5] = 0x0;
msg[6] = 0x0;
msg[7] = 0x0;
return j;
}
The actual movement function performs sending of padding data and the actual "control" message. It is important to note that once you send a message to the launcher, it will continue to follow that message until you send a stop command to it. If you are not careful and do not use the feedback from the launcher, you can cause damage to the motor when it hits its endpoints in the range.
void movement_handler(char control)
{
char msg[8];
//reset
msg[0] = 0x0;
msg[1] = 0x0;
msg[2] = 0x0;
msg[3] = 0x0;
msg[4] = 0x0;
msg[5] = 0x0;
msg[6] = 0x0;
msg[7] = 0x0;
//send 0s
//int deally = send_message(msg, 1);
//send control
msg[0] = control;
int deally = send_message(msg, 0);
//and more zeroes
// deally = send_message(msg, 1);
}
The move function presented here is used in combination with movement of the camera. Essentially, we were able to determine that there were approximately 7 feedback "clicks" for every degree in the launcher's full range (315 degrees). Therefore, we decided that everytime we moved the camera, it was appropriate to move the launcher by the same number of degrees. This would minimise the amount of movement needed when we did find the target, would help warm up the motor and would also help us determine the accuracy of our testing. The results were positive and we ended up with the code below:
void move(int direction, int clicks) {
int up = 1;
int down = 2;
int left = 4;
int right = 8;
//result[0]
int upf = 0xffffff80;
int downf = 0x40;
//result[1]
int firef = 0xffffff80;
int leftf = 0x04;
int rightf = 0x08;
char result[8];
int countClicks = 0;
while (countClicks < clicks) {
movement_handler(direction);
if (result[0] == downf || result[0] == upf)
break;
else if (result[1] == leftf || result[1] == rightf || result[1] == leftf+firef || result[1] == rightf+firef)
break;
usb_interrupt_read(launcher, 1, result, sizeof(result), 60000);
countClicks++;
}
movement_handler(0);
if (direction == down)
launcherVert -= countClicks;
if (direction == up)
launcherVert += countClicks;
if (direction == left)
launcherHoriz -= countClicks;
if (direction == right)
launcherHoriz += countClicks;
}
Priming and Firing
Firing was probably the best part of the process. We decided that we would try and carry out a fire when we were confident enough that we had found a marker (based on ARToolkit's confidence variable) and when the range for firing was within a set number of degrees (during testing this varied between 5 and 15 degrees).
We also came up with a trick called priming that had been used by the NZ launcher software. We were able to determine that the launcher began giving firing feedback just before it actually fired the launcher. A 3 second process of "priming" the launcher is required before the launcher can actually fire as the mechanism used to do this needs to be reset. We were able to write some code that could prime the launcher on initialisation and after every fire such that the launcher is immediately read to fire on demand.
The code firing and priming is given below:
void prime()
{
char initresult[8];
usb_interrupt_read(launcher, 1, initresult, sizeof(initresult), 60000);
char state = initresult[1];
if (state >= 0xffffff80)
return;
movement_handler(16);
char result[8];
while (result[1] < 0xffffff80)
usb_interrupt_read(launcher, 1, result, sizeof(result), 60000);
movement_handler(0);
}
void fire()
{
char initresult[8];
usb_interrupt_read(launcher, 1, initresult, sizeof(initresult), 60000);
char state = initresult[1];
if (state < 0xffffff80) {
prime();
}
movement_handler(16);
char result[8];
usb_interrupt_read(launcher, 1, result, sizeof(result), 60000);
while (result[1] >= 0xffffff80)
usb_interrupt_read(launcher, 1, result, sizeof(result), 60000);
movement_handler(0);
prime();
pauseTemp(1000000000);
}
Simple Strategies
We did get the opportunity to try and work on one simple strategy which had mixed success.
Guessing Ahead to fire at moving targets
We did some minor work on an algorithm to attempt to guess the position of a robot in the next frame for more efficient tracking based on the change of displacement from the centre of the image, taking into account camera movement. This was somewhat effective but needs further work and could be part of the future work for another group. We took confidence into account as well because guesses needed to be made based on a decent amount of confidence that we had in fact found a target in the first place.
Here is some of the sample code for our guess ahead algorithm:
// Deal with moving
// printf("Use guess ahead? xoffset: %f prevxoffset: %f with confidence \n", xoffset, prevxoffset, marker_info->cf);
if (marker_info->cf >= 0.5f && prevConfidence >= 0.5f && marker_info->cf == prevConfidence &&
((xoffset > prevxoffset && campanDegrees > previousPanDegrees) || (xoffset < prevxoffset && campanDegrees < previousPanDegrees))) {
if (((xoffset < 0 && prevxoffset < 0) || (xoffset > 0 && prevxoffset > 0)) && abs(xoffset) > abs(prevxoffset)) {
deltaxoffset = xoffset - prevxoffset;
deltaEnabled = 1;
double deltaangle = yoffset / deltaxoffset;
double deltaalpha = (M_PI / 2) - atan(deltaangle);
deltaAlphaDegrees = deltaalpha * RAD_TO_DEG;
printf("Guessing Ahead with delta alpha degrees %f delta offset %f\n", deltaAlphaDegrees, deltaxoffset);
}
}
Conclusions and Future Work
Overall Conclusions
We believe we can make the following conclusions about our results and what was achieved:
- We managed to get the ARToolkit to work together with player and our rocket launcher to achieve static aiming and firing with quite accurate precision, provided that there was not excessive noise for ARToolkit to deal with and the launcher's motor was relatively warm
- We managed to get a fair amount of the work we desired to complete within the restricted time frame, although with more time, a lot more can be done and we hope that groups continue with this basis work in the future
- Vision recognition and image recognition still leaves a lot to be desired and with further advances, we believe the accuracy and feedback produced could improve the performance of our program
- One of key problems was the poor feedback produced by launcher, with an improved hardware interface, we feel better results could be achieved with respect to aiming accuracy and firing
- We managed to have some success with tracking a target and having the launcher follow and attempt to fire at a target. With further work on our look ahead function and some other tactics such as moving with the target, better results might be achieved.
- The supply of usb interfaces for all the Pioneer robots would be handy and would allow future groups to work on networking and team play
 The hunter robot |
 The target robot |
 A sample marker tested with ARToolkit |
 The final marker used with ARToolkit |
Possible avenues to explore
The following short sections explore areas we did not have enough time to cover in greater detail and would be interesting to look at further in the future if other students are interested.
Target Detection
There are many ways in which the recognition of the target robot could be improved. Some suggested improvements to the method we used (ARToolkit) are provided below, along with a mention of our original idea for a classifier.
ARToolkit
- The robot appeared to have issues recognising the marker in certain lighting conditions, particularly when the marker was not directly facing the camera. Unfortunately, we didn't get to spend time fine-tuning this. However, ARToolkit contains a lighting threshold value for the image processing routines. Changing this value should make it easier to identify the marker for when the image is thresholded to black and white. It might also be possible to use more advanced custom techniques to convert the image to black and white before feeding the image to ARToolkit. See the images below for an example of how thresholding is performed in ARToolkit to recognise the marker.
 Sample image before thresholding (source ARToolkit Documentation) |
 Sample image after thresholding (source ARToolkit Documentation) |
- When the marker was moved, there was also some visible jitter when running simpleTest (the program which displays a cube on the marker to test that it is working), which obviously would have carried through to our code. ARToolkit possesses history functions, arGetTransMatCont() and arDetectMarker(), which use prior frames to track and detect the marker - this will enhance performance at the cost of reducing accuracy. However, it may be worth checking out these functions since it should make the detection 'smoother'. i.e. issues where the marker would be detected as moving around the screen randomly could be minimised when using historical information. Since our robot target does not move very fast, there should be minimal (if any) loss of accuracy due to movement, and a potentially significant gain from having more confidence in the target's position. If this approach is fruitless, it would not be too difficult to make a custom function which takes confidence or previous frames into account, such as a particle filter, or more appropriately, a Kalman filter. This would almost certainly improve performance in tracking.
Haar Classifier
It would be interesting to work on using the Haar classifier to identify a Pioneer robot and this is something we believe would be interesting to pursue even as a project by itself. Unfortunately, due to the time restrictions that we had, there was not enough time to focus on this particular goal and the ARToolkit satisfied our needs quite sufficiently.
Mapping
To make the project a little more interesting, it would be useful to map the area in which the robots are operating. This would then allow for the robots to employ strategy in their movement, such as hiding in small gaps, taking cover behind obstacles, and finding efficient paths to the target. We had planned to use Voronoi diagrams to avoid obstacles (as well as VFH to avoid close objects from assignment 1) and assist in finding shortcuts. We decided this was a lower priority as we could control the target robot manually, and the robotics lab did not have sufficient space which we thought would be easy to map (e.g. the maze would have been easy to map but it was too small to make seeking interesting).
We had planned to move the robots around the lab using a reactive strategy (i.e. by only avoiding obstacles), but we decided that the lab was a little too complex and it could take a while for the robot to find the target in a large open space (apart from the fact that they run out of battery quickly).
Networking
The project could also be enhanced by having multiple hunters and/or multiple targets. The robots would then be able to employ team strategies to find or avoid the opposing side. The robots may be able to share their location or positions of beacons, share marker information to increase confidence, provide sections they have mapped out, communicate the position of hunters or targets, or they may be able to pick their routes strategically (so as to box in a target for example).
The project was originally going to be a competition against an opposing group of students (but they chickened out), which would have made this idea more appealing. We were also limited since we could only install the missile launcher on one of the robots due to a lack of USB ports.
Aiming and Calculating Position
More work could be done on aiming, particularly dealing with distance, which itself is not a simple problem. We believe that with further work and the capability of using a laser on the Pioneer robot, adjustments could be made to the vertical range of the launcher to fire at targets, both moving and stationary.
Physical Movement
If the laser could be enabled on a robot with a USB interface, then searching, movement and tracking could be further enhanced. We were significantly hampered by the fact that we only had sonar to work with and believe that the combination of the laser with the camera and rocket launcher could be very handy in tackling the problem of searching a space and tracking more efficiently.
Further References
Links and Resources
- Player/Stage Manual: [1]
- Haar Classifier: [2]
- Training a Haar Classifier: [3]
- Blob Detection: [4]
- ARToolkit API Reference: [5]
- Writing code for ARToolkit: [6]
- NZ Missile Launcher Software: [7]
- Python Rocket interface [8]
- Using the missile launcher [9]
Journal Articles
- Kato, H., Billinghurst, M. (1999) Marker Tracking and HMD Calibration for a video-based Augmented Reality Conferencing System. In Proceedings of the 2nd International Workshop on Augmented Reality (IWAR 99). October, San Francisco, USA. [10]
- E. Jauregi1, E. Lazkano1 Contact Information, J. M. Martínez-Otzeta1 and B. Sierra1, Visual Approaches for Handle Recognition, In European Robotics Symposium 2008, Volume 44, 2008 [11]
- R. Lienhart and J. Maydt, “An Extended Set of Haar-Like Features for Rapid Object Detection,” Proc. IEEE Int’l Conf. Image Processing, Volume 1, 2002 [12]
- Brian P. Gerkey, Richard T. Vaughan, Andrew Howard, Tools for Multi-Robot and Distributed Sensor Systems, In Proceedings of the International Conference on Advanced Robotics (ICAR 2003) [13]
Code Locations
Development code, environments, sample libraries examined:
- /home/group05/proj_missile/development/
Actual code used for testing and final product
- /home/group05/online/
Finder program that performs search and firing
- /home/group05/online/ARToolkit/examples/finder/ (placed in the examples folder for simplicity of coding and making)
- /home/group05/online/finder/
Binary:
- /home/group05/online/ARToolkit/bin/finder
The OpenCV interface, testing and capture code:
- /home/group05/online/camera_capture/
The launcher testing and experimentation code:
- /home/group05/online/usb_operation/rocket/
Here is a zip file of most of the code:
This includes all files except ARToolkit. The finder folder can be inserted into the examples folder of ARToolkit and then compiled with the resulting executable residing in the bin file. Don't forget to copy over the pattern file into the Data directory too.
