Raspberry Pi Vision Processing – Robotics Programming

We are using a Raspberry Pi 4 along with a Raspberry Pi 5 MP camera for our vision processing. This allows us to process up to 90 frames per second at a resolution of 640×480 or 50 frames per second at a resolution of 1280×720.

The Raspberry Pi system uses the Raspberry Pi camera. Both the Pi and the camera can be purchased for a total of around $70. At some point I would like to put together a tutorial describing how to set up the Raspberry Pi, but that involves a large number of steps, some of which I don’t really remember at this point. To get around this problem, we are providing complete Raspberry Pi images which are already pre-configured:

Pi4-xrdp-FRC-2020.zip – This image automatically connects to a specified existing WiFi hotspot. It is currently configured to connect to a hotspot named robotics with the password ‘raspberry’, and has a fixed IP of 192.168.0.12. You can, however, easily configure it to connect to any network. It also has the wired IP fixed to 10.21.2.12 which you will need to change to use your team number.

Remote desktop server installed which allows you to connect to the Raspberry Pi using Window’s remote desktop application

There is also an Eclipse project with the source for the image processing software called NetworkVision2020. Unfortunately, it is not currently possible to install Eclipse on a Raspberry Pi 4 so if you wish to build the project, you will probably need to set up cross compile tools on your PC.

The vision processing software implements a image processing server which runs on the Raspberry Pi and communicates with the RoboRio and the Driver Station via a network connection.

To help people understand how to use this camera, we have created a example Java project which can be found here. This example implements a simplified version of the program we used last year to automatically place hatches using the vision targets.

Complete documentation for the Java camera classes can be found here. If your robot is C++ based, you will have to roll your own, but looking at the Java code should tell you what you need to know.

In addition to the server that provides the region data to the RoboRio, a second server is started which provides information and images to the driver station computer and allows the configuration of the camera. This server listens on port 5801 and when connected sends data packets consisting of a header which contains information about the image processing, followed by a jpeg data stream which encodes one of the frames. This can be used for display of the camera image. In addition, commands can be send to the server which allows the driver station computer to control the capture parameters.

The program used to view the images and control the server is a Java program which can be launched using the ImageViewer2020.cmd file that can be found in the FRCTools2020 folder which can be downloaded here. Complete source for this program is provided as an Eclipse project.

ImageViewer2020.cmd is a Windows command file which starts the Image Viewer. Note that the last parameter specifies the IP address of the Raspberry Pi. You will need to set the IP to the one you assign to the Raspberry Pi.

java -jar ImageViewer2020.jar 192.168.0.12

The image processing software will automatically start when the Raspberry Pi is booted. By default it will run at 90 FPS at a resolution of 640×480. If you look in the pi home directory you will find the file ‘run’ which starts the processing software. Currently you will see the line

./NetworkVision2020 -f90

Which runs at the default resolution (640×480) at 90 FPS.

The available arguments are

-f# – Specifies the frame rate. 90 is the maximum for all resolutions, 30 is the default.
-s[012] Specifies the frame size as 320×240, 640×480, or 1280×720 respectively, 640×480 is the default.

Now that you have the server running, you should be able to start the Image Viewer on your computer and connect to the pi. This should result is a display similar to the following:

You can change the parameters that control the processing of the images. The images are recognized by color and you specify a range for the Hue, Saturation, and Value components. Note that you can click anywhere on the image and the HSV values for the pixel at that location will be displayed below the image. This version of the Image Processor is capable of handling multiple regions and up to four simultaneous colors.

You can also control camera settings such as shutter speed, brightness. saturation and ISO. Finally there are horizontal and vertical target parameters that can be set. These numbers will be sent to the RoboRio as part of the region data.

Up to 4 different camera profiles can be set and these profiles can be selected via commands from the RoboRio. This allows you to have a profile which uses the green light and reflective tape and another which will let you use ambient light. For example, the second profile might produce an image like this:

A few notes on setting the camera parameters:

If the color you wish to match is red, the target hue will be zero. You can still bracket this by setting the min hue to 245 and the max hue to 10, for example.
We have found that the Brightness, Saturation and Contrast settings are of little use and should probably remain their default value.
You control the light level using the ISO and Shutter Speed fields. When using lights and reflective tape you should set the ISO and Shutter Speed to as low a value that still gives you a good target image. The idea is to exclude all but light sources. If you wish to process images using ambient light, then set the Shutter Speed to zero. You can then adjust the ISO as needed for the lighting conditions.
Auto Exposure and Auto Color should be enabled when using ambient light and disabled when using the reflective tape.

For lighting your camera, we have, in the past, used the green LED rings but have recently found these LEDs which will allow for a smaller footprint. Each light is rated at 10 watts and can, in principle, handle 900 ma. However we are running them at 250 ma each and find that produces sufficient light. They still get quite hot, however, and require a heat sink. This is the one we are using (cut in half). We also added a step-up/down voltage regulator for the lights so that changes in the battery voltage does not affect the light output.

The Raspberry Pi vision software is configured to use it’s GPIO pins to control the lights using a relay so the RoboRio can turn the lights on or off.

The entire package can be configured into a relatively small form factor about the width and height of the Raspberry Pi itself, as shown below. The case is 3d printed and the STL files needed are available in the FTCTools2020 download.