Research

Research Projects

Mobile Augmented Reality Applications (MARA) 

This research develops the science needed to enhance mobile augmented reality applications with (a) Spatial Analysis at UNT, (b)HoloLens, (c) Navigation, (d) Geospatial  Analysis, (e) Situational awareness, (f) Intelligent Signs, (g) Evacuation, and (h) Emergency response. The project aims to develop innovative visualization techniques that enable mobile AR systems to enrich users’ perception of the physical environment. By delivering context-aware 3D visualizations, the research supports spatial knowledge acquisition and cognitive mapping, ultimately improving situational awareness in complex environments. The MARA platform is designed to provide critical, real-time information that supports effective decision-making during emergencies for both building occupants and first responders. A primary objective of this effort is to leverage AR displays on mobile devices and head-mounted systems to guide users through virtual overlays, navigation paths, and waypoints. Additionally, the research includes the development of advanced algorithms for data layering, real-time situational awareness, and precise location sensing, ensuring accurate and adaptive guidance in dynamic scenarios.

Digital Twin: MARA for real-time Indoor Navigation, Spatial Analysis and Situational Awareness 

We developed a Mobile Augmented Reality Application (MARA) for indoor navigation that utilizes augmented reality for both localization and route guidance. The user’s position is determined through the device camera and mapped onto a digital twin of the Discovery Park building at UNT. Localization is achieved using AI-generated navigation meshes combined with AR Foundation image tracking. The system enables navigation from any location within the supported area, covering both the first and second floors, and can be extended to larger environments by incorporating additional area targets. Key features include a detailed digital twin of Discovery Park, a mini-map with real-time position and orientation indicators, AI-driven navigation meshes, support for multiple destination points, seamless multi-floor navigation, and AI Assistant. We are also exploring integration with hardware devices such as Hololens2, Magic leap 2, Apple Vision Pro, and Meta Quest 3.

HoloLens Applications for Building Evacuation and Crime Analysis

Early hands-on experiences with the Microsoft HoloLens augmented/mixed reality device have given promising results for building evacuation & crime analysis applications. A range of use cases are tested, including data visualization and immersive data spaces, in-situ visualization of 3D models and full-scale architectural form visualization. We present how the mixed reality technology can provide spatial contextualized 3D visualization that promotes knowledge acquisition and support cognitive mapping.

Active Shooter Response VR Modules for Training and Decision Making

The goal of this NSF funded project is to develop and evaluate a collaborative immersive environment in VR for active shooter response for UNT campus and BSU Campus. Immersive collaborative virtual reality environment also offers a unique way for training in the emergencies for campus safety. The contribution lies in our approach to combining computer simulated agents (AI agents) and user-controlled autonomous agents in a collaborative virtual environment for conducting emergency response training for civilians and security personnel's. The immersive collaborative VR environment offers a unique method for training in emergencies for campus safety.

VR Fire Evacuation using BCI Devices

This study presents a VR-assisted fire evacuation training system that integrates Digital Twin (DT) technology and Brain–Computer Interface (BCI) sensing to improve user navigation, decision-making, and emergency response performance in hazardous building-fire scenarios. The incorporation of BCI devices, including Emotiv and Galea, facilitates real-time monitoring of cognitive and affective states such as stress, mental workload, and attentional shifts during evacuation tasks. These physiological signals provide an additional analytical layer for assessing user performance and behavioral adaptation under stress.

MindYoga: Mobile & VR Solution for Mental Health Management

The primary goal is to create an accessible, immersive, and data-driven digital platform that leverages yoga practices and mindfulness techniques to enhance users’ mental well-being. The solution aims to reduce symptoms of anxiety, depression, and stress while promoting emotional balance and self-awareness through interactive mobile and virtual reality experiences. The specific objectives include: Promote Mental Health Accessibility, Integrate Yoga-Based Therapeutic Techniques, Employ Mental Health Assessment, Enhance User Engagement Through VR Immersion, and Encourage Long-Term Well-Being.

Enhancing Mental Health Diagnostics using VR/AR

This project leverages state-of-the-art neurotechnology to better understand and mitigate the impact of stress, anxiety, and depression. The integration of EEG data from Galea and EMOTIV HMDs will facilitate richer insights and potentially transform mental health care through personalized, real-time interventions. The goal of this project is to develop real-time, personalized interventions for managing stress, anxiety, and depression. Leveraging neurotechnology and machine learning, the study aims to map neural activity into emotional and cognitive states, thereby facilitating responsive mental health support systems.

Managing Stress and Anxiety using Generative AI 

The proposed research aims to design and develop a Virtual Reality Instructional (VRI) training environment using Unity, with interactive avatars powered by generative AI. This environment will focus on mental health support by enabling non-player characters (NPCs) to simulate supportive conversations, deliver therapeutic interventions, and guide users through mindfulness and cognitive behavioral exercises. This innovation can enhance the realism, personalization, and accessibility of mental health training and interventions. 

iHARP- NSF HDR institute: Annotation and Visualization of Heterogeneous Data

The goal of this project is to develop intelligent data-driven techniques, accurate labeled data is needed for training and evaluation of deep learning techniques. To better engage scientists in the labeling and to facilitate the labeling we will use AR/VR tools. This work seeks to build upon the previous efforts by creating an integrated tool suite for enhancing the value, situational awareness, accessibility, and understanding of science data connected with Arctic science using virtual reality (VR) and augmented reality (AR) technologies. This work designs, develops, and evaluates AR/VR tools to explore and annotate, using tablets/mobile devices, and HoloLens. The objective of the effort is to use the AR displays on mobile devices and headsets to guide the users with virtual overlays, paths, and way-points. It will also involve development of algorithms for layering, situational awareness, and location sensing.

Virtual AI Tutor using Generative AI for Education

The goal of this project is to develop and evaluate an AI virtual tutor using generative AI. We are creating a realistic VR environment to provide offline tutoring assistance to students for courses such as Introduction to Data Science and Introduction to Computation with Python. The AI tutor integrates Chat GPT and provides interactivity capability of lip synchronization and hand movement. The user can ask the AI virtual tutor any question. Our research will allow artificial intelligence-powered tutors to become more versatile and accessible through voice interaction. Our aim is to create a multi-user VR environment with tutorial mode and interacting mode. We are also exploring the VRI module to work on other hardware devices such as HTC Vive, Hololens2, Magic leap 2, Apple Vision Pro, and Meta Quest 3.

Virtual AI Health Assistant 

The goal of research in Virtual AI Health Assistants is to develop intelligent systems that can support, enhance, or automate healthcare services through natural and effective interaction with patients and healthcare providers. These assistants aim to provide accessible, personalized, and efficient care using artificial intelligence. Our vision is to create AI-powered assistants that are safe, reliable, empathetic, and medically competent, serving as partners in health management for both patients and professionals.

Virtual AI Assistant for Navigation & Evacuation

The goal of research in Virtual AI Assistants for Navigation and Evacuation is to develop intelligent, real-time systems that guide individuals safely and efficiently during normal navigation and emergency evacuation scenarios using generative artificial intelligence. AI Assistant provides real-time navigation & evacuation assistance with dynamic, context-aware directions in complex environments (e.g. hospitals, campuses). It adapt routes based on user location, accessibility needs, or environmental changes. 

Digital Twin: Human Biomechanics and Squat Analysis 

The goal of this project is to make human biomechanics more accessible for a wider audience through the use of Unity for data visualization. Numerous sensors are still being used, however these sensors do not require cameras. This radically improves their utility for populations that cannot dedicate the space and funding to video motion capture endeavors. The objectives of this study is to assess the validity and reliability of machine learning models’ ability to correctly recognize squat exercise movements.

Digital Twin: Geospatial Mobile Application for Navigation and Emergency Response

The goal of this project is to develop and evaluate a Geospatial Mobile Application for Navigation and Emergency Response using Google Photorealistic 3D Tiles and Cesium for Unity. The Mobile AR Application can be used for navigation and emergency response at UNT campus. Cesium for Unity combines the 3D geospatial capability of Cesium and 3D Tiles with the Unity ecosystem. The following objectives are being explored: 1) Location-based augmented reality (AR) applications using Google Photorealistic 3D Tiles and Cesium for Unity to provide more immersive navigation experiences and emergency information to users on campus. 2) Geospatial AR Navigation: Develop mobile AR phone application that makes navigation more intuitive with a 3D model of the user’s location at UNT campus by displaying location-based content and integrating Google Photorealistic 3D Tiles perfectly with the real world. We are also exploring integration with hardware devices such as Hololens2, Magic leap 2, Apple Vision Pro, and Meta Quest 3.

Data Science and Data Visualization

Data analysis and crime data visualization offer a powerful approach to unraveling the complex dynamics of criminal behavior. Analyzing crime data involves a multifaceted study of different dimensions of crime. This involves examining the types of crimes committed, their frequency, and distribution in different geographic areas. By analyzing these factors, we can find patterns and hot spots that can show areas of criminal activity. The projects include: 1) Analysis of Crime 2) Common links between COVID-19 data and crime data in Baltimore. 3) COVID-19 Data Visualization, 4) Crime Data in Baltimore Visualization, 5) Scientific Data Visualization , 6) Data Analytics: Improve the Quality of Life in Urban Areas.

Multi‐User Virtual Reality (MUVR) for Evacuation Drills

MUVR environments for emergency evacuation drills are developed that include: Subway evacuation, airplane evacuation, school bus evacuation, VR city, night club disaster evacuation, building evacuation, and university campus evacuation. Our developed applications show an immersive collaborative virtual reality environment for performing virtual evacuation drills using head displays. Immersive collaborative virtual reality environment offers a unique way for training for emergencies situations. The participant can enter the collaborative virtual reality environment setup on the cloud and participate in the evacuation drill or a tour which leads to considerable cost advantages over large scale real life exercises.

Virtual Reality Instructional (VRI) modules for Teaching Complex Topics and for Training for Improving Quality of Care and Patient Safety

The goal of this research work is to develop virtual reality instructional (VRI) modules for Teaching, Health Care, Training, and Manufacturing. The projects include 1) create instructional course curriculum modules with more inquiry based problem-solving activities and hand-on experiences based on Gaming and Virtual Reality for teaching complex topics, 2) train integrated care team members to engage patients from vulnerable populations safely and efficiently. 3) development of training modules geared for COVID-19 testing.

Multi-Agent System (MAS)

Two MAS and models are developed and evaluated namely AvatarSim and AvatarSim2. AvatarSim was developed in Java and AvatarSim2 was developed in C# language. The AvatarSim model comprises of three models which are: a) Geometrical Model, b) Social Force Model, and c) Fuzzy behavioral Model. AvatarSim2 model further combines genetic algorithm (GA) with neural networks (NNs) and fuzzy logic (FL) to explore how intelligent agents can learn and adapt their behavior during an evacuation. The adaptive behavior focuses on the specific agents changing their behavior in the environment. The shared behavior of the agent places an emphasis on the crowd-modeling and emergency behavior in the multi-agent system. The result of this simulation was very promising as we are able to observe the agents use GA and NN to learn how to find the various exits.

Past Work

Megacity: A CVE for Emergency Response, Training, and Decision Making

The goal of this research is to use game creation as a metaphor for creating an experimental setup to study human behavior in a megacity for emergency response, decision-making strategies, and what-if scenarios. It incorporates user-controlled characters as avatars and computer-controlled characters as agents in the megacity CVE (Collaborative Virtual Reality Environment). Virtual crowds for non-combative environments play an important role in modern military operations and often create complications for the combatant forces involved. To address this problem, we are developing crowd simulation capable of generating crowds of non-combative civilians that exhibit a variety of individual and group behaviors at a different level of fidelity.

Human Centric Cyber Situation Awareness and Data Visualization

The goal of this project is to explore ways to visualize the Cyber Situational Awareness capability of an enterprise to the next level by developing holistic human centric situational awareness approaches into new systems that can achieve self-awareness. This research effort aims to identify how graphical objects (such as data-shapes) developed in accordance with an analyst's mental model can enhance analyst's situation awareness. The humans are more adept at inferring meaning from graphical objects, links and associations in a data element. The project aims to use virtual reality techniques to visualize the XML data through the use of a Force Directed Node Graph in 3D which renders and updates in real-time. It can be used to visualize computer networks for cyber-attacks.

Augmented Reality Instructional (ARI) modules for Evacuation and Training

This work presents cutting edge Augmented Reality Instructional (ARI) modules that overcome the visual limitations associated with the traditional, static 2D methods of communicating evacuation plans for multilevel buildings. Using existing building features, we demonstrate how the ARI modules provide contextualized 3D visualizations that promote and support spatial knowledge acquisition and cognitive mapping thereby enhancing situational awareness. These ARI visualizations are developed for first responders and building occupants to help increase emergency preparedness and mitigate the evacuation related risks in multilevel building rescues and safety management.