As part of the project Homecoming VR, there was a need to come up with a way the players would be able to navigate the narrative and "unlock" specific memories (360 videos) by performing certain shapes with their hands using the VR controllers.

A custom Image Classification model was trained using 100000 synthetic images of hand drawn shapes. The dataset was generated from an an automated system that was designed in a way that would allow enough variation to train a confident image classifier. As a first step, the main shapes were converted to splines that were further processed by a set of calibrated curve distortions and simplifications to resemble shapes that could have been written by hand.

The distortion effects were applied in different orders and strengths to create a wide range of hand-written shapes. The final model was used for inference directly within the VR app with Barracuda.