Computer Vision#

This article talks about the rover’s camera and computer vision systems. Cameras are extremely important to rover operation. We use cameras for things as simple as streaming a video feed back from the rover to show us what it’s doing to things as complex as determining where the rover should go during the autonomy task.

Camera Nodes#

main(argc, argv) creates a camera ROS node using the CameraProvider.

Streaming cameras for user view are implemented using cmr_cv/camera_node. We use an RQT plugin for cameras in order to simplify camera view for the user.

This can be enabled for a camera with the following steps:

  • cmr rover

  • cmr shell

  • cmr enable cam/camx (where x is the camera number you want i.e. cam7)

  • rqt

  • If not already up, select it from Plugins>CMR>camcontrol

As long as the selected camera is valid, there should be an output. After setting the necessary resolution and setting the q value to the desired quality (0 being perfect and a higher value indicating more loss), pushing the button should cause the image to be shown.

CameraProvider#

parseConfig(node) initializes dev_, default_res_, endpoints_, default_q_, and default_mjpeg_ with the node’s device, resolution, endpoints, q value, and mjpeg. The function returns true.

setUpNodeFunctions(handle, deps) initializes the camera, publishes the resolutions of the camera, starts the resolution and q servers, and starts the stream. The function waits a second and returns ok().

CameraStreamer#

initializeStream(device) begins a stream with --device as device. Each resolution in the stream is pushed on to the resolutions stream. The function returns true if the stream begins with ioctl: VIDIOC_ENUM_FRAMESIZES.

setResolution(resolution) sets the resolution and restarts the stream with the current parameters.

resolutions() returns the supported resolutions for the camera if and only if called before initialize().

setMjpeg(mjpeg) forces the camera to be in mjpeg mode if mjpeg==true and forces mjpeg mode off if mjpeg==false. The camera is in mjpeg mode by default to start.

inputSpec() returns a vector containing the specs of the camera including the resolution and device as well as mjpeg details if enabled.

VideoStreamer#

~VideoStreamer() stops all streams. More streams may be started after this.

initialize(device) initializes the video stream with the given device and default q value.

stop() kills the buffer and closes it if the encoder is open.

request(endpoints) opens a new stream provided the endpoint is not in use yet. Must be called after initialize().

release(endpoints) stops broadcasting to a specific endpoint.

process(f, endpoints) applies f to each endpoint and returns true if f is true for at least one endpoint.

setQ(q) sets the q value to q and restarts the stream if the q value changed.

restart() stops all streams and restarts them.

streamSpec(endpoint) gets the stream specified by the string endpoint containing the IP address and port of the stream. For example: streamSpec("1.2.3.4:5678") returns the FFMPEG rtp stream "[f=rtp]rtp://1.2.3.4:5678/".

generateCommand() generates the ffmpeg command to stream all the endpoints.

Image Capture Node#

main(argc, argv) creates a ROS node using the ImageCaptureProvider.

ImageCaptureProvider#

parseConfig(node) initializes the node using the config provided in the YAML node node. The function returns true.

setUpNodeFunctions(handle, deps) initializes and starts the server(). The function returns true.

ImageStreamer#

initializeStream(device) initializes the stream with the provided device and returns true if the input file stream was created.

inputSpec() returns a string vector with the streamer’s specifications, including the device.

FusionImageCapture#

FusionImageCapture(device) creates a new instance of FusionImageCapture with the device specified by the given path.

capture(exposure, width, height) captures an image at the given exposure, width, and height and returns it.

Ball Detector Node#

main(argc, argv) creates a ROS node using the BallDetectorProvider. The ball detector was used to tell the rover where to go during the autonomy task, but the competition has since changed the tennis balls to AR tags, so this node is now defunct.

BallDetectorProvider#

parseConfig(node) initializes the node using the config provided in the YAML node node. The function returns true.

setUpNodeFunctions(handle, deps) initializes the server and starts it. It also creates a BallDetectorProvider image with all values of 0 and publishes 1 latched ball message to be seen in rviz. The function returns true.

computeBallLocation(ball, image_width, d, theta, x, y) does the following:

  • sets d to the distance away from the ball.

  • sets theta to the angle made to the ball along the distance d.

  • sets x and y to the respective Cartesian x and y coordinates made by d and theta.

radiusToDistance(image_width, image_radius) calculates the distance from the camera to the located object using the width of the image and the radius of the object

BallDetector#

show(i, wait) displays i in the bob window (if wait is false) and then waits until the ESC key is pressed

detect(image, scale_back) attempts to detect a ball in the image. If it finds one, it draws a purple circle around it on image. The image’s scale is preserved if scale_back is true. The ball is returned if it is found, otherwise NO_BALL is returned.

detect(image, out, scale_back) is an auxillary function for detect(image, scale_back).

resize(src, max, dst) scales src to max.width or max.height depending on if size.width>size.height. The function returns the amount the image was scaled by.

search(ball_color, image) searches image for a ball colored ball_color. All circles detected are circled in red and shown. Scores are assigned to each circle. The ball with the lowest score is returned in a pair where the first element is the ball (or NO_BALL if not found) and the second element is the score of the ball.

configFromYaml(node) initializes the node using the YAML node node.

Detector#

main(argc, argv) initializes a BallDetector and configurates it with the YAML file at “/ws/src/Superproject/cmr_rover/config/__ball_detector__/real.yaml”. The detector is set to have a size of 640x640. The function must be called with at least 2 args in argv. There may be a first argument --testformat. The remaining arguments must be image filenames. The function tries to detect a ball in these images using BallDetector::detect(). If --testformat is used, the locations and radii of the balls detected are printed to the console. Otherwise, the balls are printed.

AR Tag Detection via ZED Camera#

For the Autonomy task in 2020, it became necessary to use ARTag detection to find markers in the task.

Usage cmr enable ar_track

  • Requires ZED to be plugged in and a GPU to be present

  • This will publish to the topic /ar_track/ar_pose_marker

  • Publishes a message of type AlvarMessage.

Running ZED Camera#

In order to run the ZED camera, the following is necessary:

  • ZED Camera (Camera box in CMR Workspace)

  • Team Laptop (to run the rover or mock rover)

In order to run the mock rover with the ZED Camera:

  • Change root directory to Superproject

  • Enter the autonomy_zed_refactor branch

  • cmr dclean

  • cmr make

  • cmr rover –mock

  • cmr base

  • cmr shell

  • cmr enable hal

  • cmr enable ar_track

Alvar#

Because these tags are commonly used, it was not necessary to implement our own detection algorithm. We used the alvar detection library, already integrated into ROS.

This entailed the following changes to integrate the Alvar library:

High Quality Image Capture (Filter Wheel)#

High quality capture ensures that images can be captured at varying exposures. This is necessary in order for the filter wheel to work well.

This is done by the image capture node, which creates a node and a provider that expose a capture action that can be used. This allows an image to be captured and a image file to be created in teh /data/ directory.

The default parameters for the image capture are provided in the image_capture.yaml file and are as follows:

device: "/dev/cmr/cam8/"
centerstep: .003
stepsize: 2.0
stepsbelow: 2
stepsabove: 2
width: 480
height: 480

Additionally, if no filename is provided, a default file name of the format “%d-%m-%Y %H-%M-%Simage.jpg” will be used to save the image.

The node can be run by typing cmr enable image_capture and, after enabling the node, the action can be run by typing ac /cmr_cv/capture.