Deep learning based self driving on modified MIT racecar platform

MIT racecar is an excellent and open platform by MIT to test self driving technology and algorithms.

In this article I would summarize my experience in building deep learning based driving solution on this platform. This is inspired by driving approach suggested in

Basically we train the vehicle to follow a track at constant (or possibly varied) speed by training it on data collected by human driving. This approach is radically different from the traditional way of achieving autonomous driving which involves following basic steps.

i) Creating (or acquiring) the map of environment. Various type of SLAM algorithms exist to create maps of environment. The maps created in this way are more detailed than normal road maps from Google etc. They have much detailed , usually 3D, information of the environment with position of landmarks like trees, traffic lights etc. For simpler applications like indoor AGVs or robo vacuum cleaners simple 2D maps also suffice.

ii) Implementing ability to localize the position of vehicle in the given map. This is done using data of one or more sensors and comparing it with map landmarks. GPS data can also be used in outdoor environments but it is usually not accurate enough and is fused with sensor data to achieve higher accuracy.

iii) Planning the lowest cost path in the map to reach the destination and using some control algorithm to make sure vehicle doesn’t stray from the planned path.

iv) Detecting obstacles and recalculating the path in real time.

Deep learning approach also known as end-to-end approach on the other forgoes all the above mentioned steps. Rather it trains a neural network using driving data obtained by manual(human) driving. This brings us to the question regarding nature of the driving data to be collected. What would be the input and what would be the output of the neural network? It turns out that driving data to be used required for training is fairly simple. It consists of series of images taken by camera(s) attached to front of vehicle and value of steering angle and acceleration (throttle) of vehicle corresponding to time when those images were taken. The input to neural network becomes the image and output becomes steering angle and throttle.

Note that Nvidia’s experiment only tries to predict the steering angle of the car and not the car acceleration/throttle.

Hardware

Note that this camera cannot be used as stand alone as its functionality is deeply integrated with Raspberry pi on-board GPU. I decided to hookup Raspberry pi with TX2 using ethernet. It was my hope that this will also enable me to move some of load of image pre-processing to Raspberry pi. Connecting different boards to handle different functionalities is quite simple and straight forward using ROS which was used as software platform as described below.

Software

With this basic platform setup lets jump into the deep end i-e Deep learning for track following.

Multiple tracks were used for testing. I used white tape to draw track on floor. Final testing was done on custom track shown below. As is obvious the goal is for racecar to continuously go on a loop on the track which has some slight curves, some steep curves as well as straight path.

For data collection I implemented a ROS node which subscribes to camera images and joystick input. Camera runs at around 40fps and every incoming image is stored with corresponding value of steering angle and throttle from joystick.

Following experiment which uses Donkey car platform

has excellent source for data collection. Note that although this source is in C++, I found it simpler to implement it in Python. The images do not have to be of high resolution as lower resolution images are faster to train on and give similar performance. I ended up using 348x410 image size in RGB.

To make data collection easier, joystick buttons were programmed as such that it was easy to turn off/on the recording of data. A speaker was attached to car so that car announced when it was recording and when it wasn’t recording.

Data Collection Strategy

First we have to focus on gathering data which tries to stay in middle of track as much as possible.

Not only do we have to collect data for normal driving but also we have to collect data for recovering in case the car veers off the center of track or even goes out of the track. This is done by collecting data only for car moving from just outside of track to center. This data is required to make the driving more robust. Even if car doesn’t succeed to stay in middle of track it has enough ‘knowledge’ to come back to center of node.

For final experiment for above track I collected around 30,000 images which then have to be pre-processed. Two most obvious things that need to be done are i) Crop the top part of image which does not contain information about track. For my case I ended up using only lower 160 pixels and cropping the rest i-e 148 pixels.

ii) To make the steering angle data balanced every image is flipped and steering angle multplied by -1. This immediately doubles the number of images available and balances out steering angle so that mean is always 0.

For training I used the network suggested by Nvidia Formula Epoch as it is. Its a Keras network with basically 3 convolutional layers all using 3x3 filters and 1 fully connected dense layer. I experimented with different modifications but only useful modifications with better results was increasing dropout to 0.5

Once network was trained I used Visualization library to see which pixels in image were providing strongest signal for activation

This helped a lot as my intention was for the network to focus on track lines and track and not on any other object. I trained the network to reduce validation loss and always verified through visualization that focus was on lines

The following video shows the performance of car. It stays in track although within track it does waver a lot within track.

Tests showed that performance is directly related

Speed

At lower speeds car performs better, as it has much more time to react to situations.

Curve Steepness

With tighter curve the car has more chance to go out of track. This is partly due to fact that car has less room to make mistake and recover. Unlike middle of track where even if the car makes wrong prediction for angle it can reover, the car has to predict the right angle to turn during the duration of curve.

With this observation I use 2 different speeds for curves and for straight lines. To decide if car is turning average of steering angle is calculate for a window of time and if its absolute value is higher than certain threshold speed is lowered to curve speed and vice versa.

Actuator Performance

The car wheels have to be calibrated properly and Vesc motor controling the wheels have to work properly, any error there can cause huge effect on performance.

Another interesting observation was that with increasing load of car the accuracy of actuators decrease. So load of car has to be managed properly.

Conclusion

This project checked feasibility of using deep learning to teach car to drive on its own. Results are quite promising although utility of only track following can be limited but useful in certain situations.

In next step tests will be done for larger vehicles.

Related posts:

In the world of sports, there are athletes who stand out for their exceptional talent, skill, and dedication. These individuals have not only achieved greatness in their respective sports but have…

Look down the hallway at your past.. “retrospect” is published by beth buzz in Poets Unlimited.

The Saut-d-Eau waterfall which is often referred to as “the city of happiness” is currently the tallest in Haiti. Approximately 100ft high, the waterfall owes its name to another large waterfall in…