Trial by virtual fire: Augmented reality project tailors training for firefighters

For firefighters, even training is a risk. According to the United States Fire Administration, 16.6% of all firefighter fatalities in 2024 took place during training exercises. George Mason University researcher Craig Yu is hoping to change that statistic.

Yu, associate professor of computer science, has been developing augmented and virtual reality training for most of his career, and he has a particular interest in exploring how artificial intelligence (AI) and emerging technologies could help research on human development and sports science. Through National Science Foundation’s EArly-concept Grants for Exploratory Research (EAGER) program, Yu and Joel Martin, associate professor of kinesiology, are developing just that: an augmented reality program that could help train firefighters while minimizing risk.

“Firefighters have high rates of injuries and many injuries happen during training. There is a need for training technology that can replicate aspects of emergencies while minimizing the injury risk and physical burn,” said Martin. Martin has spent the past few years working with Fairfax County Fire and Rescue to determine how they can reduce physical strain to firefighters during training exercises. “I think that technology such as immersive reality training can help to meet training needs while minimizing the physical risks of performing certain drills."

Augmented—or mixed—reality uses devices such as phones or a headset to overlay 3D computer-generated content onto the real world. An increasingly accessible tool, you can find augmented reality in applications like Ikea furniture previews in your house, dinosaur encounters in the Smithsonian, and catching Pokémon in your neighborhood.

As described in their project proposal, “Adapting Mixed Reality Training Programs to Real-World Scenes to Enhance Human-AI Teaming in Emergency Responses,” the new training would utilize augmented reality to create realistic training scenarios for firefighters.

Based on a 3D digital twin of the room, the program will use AI to analyze what's in the room and create a realistic outbreak of fire: where it is most likely to start and spread, and where civilians are most likely to be.

“The program understands the environment, and it will be refined with additional input from professional firefighters,” said Yu.

Yu also notes that the scenario can be adapted for different experience levels. Simpler scenarios can be developed for new firefighters, which can then be adjusted for increasing sophistication and complexity as the trainees learn and improve. It also allows them to practice scenarios that would be difficult or impossible to replicate in a live burn, such as how to best evacuate and rescue civilians.

Most importantly, it prevents firefighters from being exposed to hazardous conditions while in training.

“I don’t ever see us replacing real fire training,” said Yu, “but this is a means to augment the training to improve the health and safety of the firefighters, as well as make training easier to deploy in a variety of buildings and circumstances. AI allows us to rapidly analyze human performance and then adaptively synthesize training programs to address weakness we observe.”

Martin and Yu are in the early phases of development but hope to deploy their initial training designs with Fairfax County Fire and Rescue soon. They both have hopes for further incorporation of AI and augmented reality technology in firefighting tools, such as headsets that can scan the environment and offer real-time feedback about potential dangers, exit strategies, and health metrics as firefighters enter hazardous zones.

“More and more virtual and immersive reality training is being used [in kinesiology],” said Martin. “I think this project demonstrates the importance of collaboration and interdisciplinary teams to solve challenges in the field.”