Contextual learning with Mixed Reality
Thesis (Work in Progress)
Context
Since I arrived in Canada about 6 months ago, I’ve been trying to learn French through different techniques. I’ve tried some of the most popular apps such as Duolingo, completed certain French grammar exercises, and watched a bunch of French movies as well. Although I’ve caught some bits of grammar and vocabulary along the way, I feel that these techniques are not efficient. After a bit of research, I realized that conventional technological tools to learn languages are inefficient as they lack a real context. Immersion can significantly accelerate the learning process and I was curious to build an app that could make learning languages fun and highly efficient for me and all the other language learners.
Aim
Reduce Integration Cycle time and improve user onboarding experience
Tools
Unity, Hololens MRTK3, Node, PyTorch
Timeline
2022 - Present
Reimagining the way we learn
Can we bring learning to life?
Context + Mixed Reality = Magic
Research Question
How can adaptive context-aware systems support interactive mixed reality learning experiences?
How Augmented Reality Will Change Education Completely | Florian Radke | TEDxGateway
# Prototype 1: Language Learning
Things is an app built for Android smart glasses with a camera and a microphone. The app features real-time intelligence to help you learn languages in an immersive context. When users wearing smart glasses look at any object around them, they see what’s that object called in the target language on the HUD(heads up display) screen. It currently supports 3 languages (French, Italian, and Japanese) and allows the users can switch between languages by tapping on the capacitive touch panel on the glasses. The smart glasses display the translation of the object’s name and speak out the pronunciation as well.
# Prototype 2: Surface area and Volumes
Further development
The app is being ported to Hololens 2 and would use its RGB cameras and depth sensors for spatial anchoring. Google Cloud Vision APIs will also be used to support many more object categories and other language APIs to support sentence construction are being added.
