mind_the_bend 🏎️

Vision based deep learning for racing games.

Does the car stay on track by itself ? yes

Should you trust your life on it ? no

Note: The model is trained and tested on an open source racing sim called speed dreams, you can install it from here

Ensemble model (under progresss)

.

.

data collection

For the ensemble model, data was collected from 2 perspectives within the game:

• one of them would be responsible for the "colors" with a simple RGB image from the bonnet of the car
• another one would be for the "edges" from a third person perspective

.

Vanilla model:

.

Here's the Project in action

🎥 Collection of training data -

• Rapidly takes screenshots of the game and saves them into a folder
• The label in this case is the x-co-ordinate of the mouse which is captured by pyautogui and is stored in the formatted filename of each collected image

image fileame formatting is done as (x-value)_(unique_ID).png

🔎 Processing images

• Converts all images to numpy arrays with a depth of 3 for R,G and B color channels
• The shape gets changed from [ width, height, depth ] to [ depth, width, height] for it to be of the right size for the CNN input channel
• Augmentation techniques like RandomAffine() get applied upon the images before the forward pass through the CNN for training

📈 Data preprocessing and augmentation

• First things first, plotted the frequency distribution of each steering value hen
• Roughly doubled the amount of training data by generating mirror images of existing images and vstacking them with reversed steer value.
• Flattened the frequency distribution by oversampling some labels with the Imbalanced-learn library and torch transforms
• Normalised the steering values by replacing the x-co-ordinates with steering values. In my case the "straight" steer value was at x = 400, for normalised_value = 400 - x_value.

note : Right is negative steer value and left is positive

🚗 Self driving

.

• Rapidly taken screenshots are prerpocessed and fed to the trained CNN drunk_driver()
• drunk_driver() returns a steer value
• Returned value is taken care of by pyautogui which moves the mouse accordingly

.

Stuff that's under way right now -

• take screenshots at a higher framerate both while collecting training data and testing trained models
• Figuring out a better data augmentation technique, instead of just making copies
• Create and train an ensemble model with frames from different perspectives (forward, top down)

❗ need help

• need a proper way to terminate/pause the loop in wear_your_seatbelts.ipynb when the CNN takes over mouse control, currently its done by moving the mouse to the corner of the screen manually
• kernel dies after about 5 mins of data collection in collect_training_data.ipynb