Publications

2026

1. Stacking-Based Heterogeneous Genetic Programming for Interpretable Credit Risk Evaluation

   Zixue Zhao, Qiao Lin, Yiran Li, Yue Li, and Tianxiang Cui

   Applied Soft Computing, Jan 2026

   Abs DOI PDF

   The development of advanced ensemble models to handle complex and large-scale datasets has become a central focus in credit risk prediction. Although ensemble methods offer strong predictive performance, stacking lacks a standardized construction pipeline and its complex structure often reduces transparency and robustness. To address these challenges, this study proposes a stacking based heterogeneous genetic programming classifier (SH-GPC), an end to end pipeline in which genetic programming serves as the meta-classifier to enhance interpretability and generalization. Through experiments on a large-scale credit risk dataset comprising millions of records, SH-GPC is shown to significantly outperform conventional homogeneous ensemble methods and several emerging models, including XGBoost, LightGBM, NGBoost, and TabNet, in terms of AUC. Compared to stacking frameworks with logistic regression or XGBoost as the meta-classifier, SH-GPC achieves better predictive accuracy while relying on a smaller number of base classifiers, thereby improving simplicity and interpretability. Transparency is further enhanced by representing the GP meta-classifier as evolved symbolic expressions and syntax trees. Additionally, the incorporation of the Shapley additive explanations (SHAP) technique enables visualization and attribution of each base classifier’s contribution, offering insights into the model’s internal decision logic. This study demonstrates the applicability of evolutionary algorithms in ensemble learning and introduces a new framework for credit risk modeling that achieves a balance between accuracy, stability, and interpretability.

2. Parametric H-BIM for Chinese Historical Architectures Based on Ancient Design Principles and 3D Reconstruction Technologies

   Xinyu Tong, Cheng Zhang, Yiping Dong, and Yue Li

   Journal of Green Building, 2026

   Abs DOI PDF

   Heritage Building Information Modelling (H-BIM) integrates the physical and functional characteristics of historical architecture into a digital representation, promoting sustainable management of the protection, restoration, and dissemination processes. However, due to the unique and complex geometry of Chinese historical architectures, creating an H-BIM model with comprehensive information is challenging. Conventional “3D reconstruction-only” models only contain surface information, and the current parametric modelling cannot reflect existing historical architectures in reality. To address the challenges, the present paper integrates parametric modelling and 3D reconstruction to fit the existing Chinese historical architectures. Based on the ancient design principles in Ying-zao fa-shi, a procedure is proposed for the efficient generation of various parametric components and the whole model. The obtained model encompasses both exterior and interior information of the architecture with a very small deviation from the existing architecture. Furthermore, high-resolution images are mapped onto the model as texture, allowing the model to reflect the defect information. Additionally, the strengths of photogrammetry and laser scanning are combined to achieve complete and accurate data acquisition. A case study was carried out at Pan Gate, China, where a complete parametric model was generated through the fusion of multiple data sources and parametric modelling. The model accurately reflects the actual architecture and enables further site management tasks, such as structural analysis and defect inspection.

3. What Does It Sound like? Reconstructing and Reimagining Cultural Soundscapes

   Bingqing Chen, Xuehua Fu, Yue Li, and Marcel Zaes Sagesser

   In International Conference on AI-Generated Content (AIGC 2025), Apr 2026

   Abs DOI PDF

   This study explores how generative artificial intelligence (GenAI) can support human co-creation in reconstructing and reimagining cultural soundscapes. Participants were invited to a local heritage site, Xiyuan Temple (Suzhou), where they collected field recordings and photographs, then collaboratively generated new ambient sounds through AI using text-based prompts. The process revealed that participants did not merely reproduce the existing auditory environment but actively idealized it - introducing imagined elements such as temple bells or animal sounds to express a personal sense of place. This phenomenon highlights the interplay between reconstruction and idealization in human-AI collaboration and reveals how generative AI can serve as a tool for cultural reinterpretation. Drawing on soundscape theory and human-AI co-creation, this work contributes to the field of AI-mediated sonic co-creation for cultural contexts, where human perception, imagination, and algorithmic models interact to reshape cultural and sensory heritage. The findings highlight the potential of GenAI as a tool for creative cultural engagement, expanding the AI4Culture discourse through embodied listening and participatory imagination.

4. Read and Reflect: Exploring How the Display of User-Generated Content Shapes In-Situ Museum Expression

   Wenzhe Hu, Shuran Fan, and Yue Li

   In Proceedings of the Extended Abstracts of the 2026 CHI Conference on Human Factors in Computing Systems, Apr 2026

   Abs DOI PDF

   Museums increasingly collect user-generated content (UGC), yet such materials are often presented as fragmented messages of visitors’ voluntary expressions. Prior work suggests that encountering others’ perspectives can prompt reflection and dialogue, but it remains unclear how different UGC display approaches shape visitors’ experiences and in-situ expression during a visit. In this work, we distill strategies from prior research into four UGC display modes and design corresponding prototypes as design probes. Through a field study conducted in a museum, we compare how different display approaches influence visitors’ meaning-making and expressive behaviors. Based on our findings, we derive design implications for ∗Authors contributed equally to this work.

2025

1. Exploring User Preferences for Museum Guides: The Role of Chatbots in Shaping Interactive Experiences

   Bingqing Chen, Ruoyu Wen, Shufang Tan, and Yue Li^*

   In Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, Apr 2025

   Abs DOI PDF

   Museums are increasingly using chatbots to transform passive visits into interactive experiences, leveraging advancements in Large Language Models (LLMs) for more engaging interactions. However, design guidelines for chatbot roles and interactions tailored to user preferences in museum contexts remain underexplored. To address this, we conducted an online survey with 65 participants, examining preferred chatbot roles and their relationship to artifact characteristics. Participants strongly favored chatbots using a first-person narrative as artifact creators, appreciating their empathetic, immersive, and novel perspectives. The user perceptions of chatbot roles are also found to be influenced by artifact characteristics, including artifact category, its popularity, and whether it depicts human or animal figures. However, concerns about the authenticity and ethical representation of historical figures emerged. These findings provide valuable insights for designing engaging and culturally sensitive chatbot interactions in museums.

2. Exploring and Modeling Gaze-Based Steering Behavior in Virtual Reality

   Xuning Hu, Yichuan Zhang, Yushi Wei, Yue Li, Wolfgang Stuerzlinger, and Hai-Ning Liang

   In Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, Apr 2025

   Abs DOI PDF

   Gaze-based interaction is a common input method in virtual reality (VR). Eye movements, such as fixations and saccades, result in different behaviors compared to other input methods. Previous studies on selection tasks showed that, unlike the mouse, the human gaze is insensitive to target distance and does not fully utilize target width due to the characteristics of saccades and micro-saccades of the eyes. However, its application in steering tasks remains unexplored. Since steering tasks are widely used in VR for menu adjustments and object manipulation, this study examines whether the findings from selection tasks apply to steering tasks. We also model and compare the Steering Law based on eye movement characteristics. To do this, we use data on movement time, average speed, and re-entry count. Our analysis investigates the impact of path width and length on performance. This work proposes three candidate models that incorporate gaze characteristics, which achieve a superior fit (R2 > 0.964) compared to the original Steering Law, improving the accuracy of time prediction, AIC, and BIC by 7%, 26%, and 10%, respectively. These models offer valuable insights for game and interface designers who implement gaze-based controls in VR environments.

3. Creating Panorama Virtual Tour Systems for the Built Environment: A Practitioner Perspective

   Liang Li and Yue Li^*

   In Advances in the Integration of Technology and the Built Environment, 2025

   Abs DOI PDF

   Virtual tour systems have become an integral aspect of modern technology, revolutionizing the way individuals interact with physical spaces. In the context of the built environment, virtual tour systems offer applications ranging from architectural visualization and urban planning to educational simulations and interactive tourism experiences. The design and development of these systems involve intricate considerations, such as path planning, data collection, tour design, and the integration of multimedia, to create a cohesive and engaging virtual experience of the built environments. In this paper, we present the development of a virtual tour system and discuss the practical constraints. Our work has practical implications for employing digital approaches to present the built environment, shaping the way we interact with and experience physical spaces.

4. Freestanding Transparent Organic–Inorganic Mesh E-Tattoo for Breathable Bioelectrical Membranes with Enhanced Capillary-Driven Adhesion

   Xiang Li, Junyi Zhang, Bo Shi, Yawen Li, Yanan Wang, Kexiang Shuai, Yue Li, Gege Ming, Tao Song, Weihua Pei, and Baoquan Sun

   ACS Applied Materials & Interfaces, Apr 2025

   Abs DOI PDF

   The electronic tattoo (e-tattoo), a cutting-edge wearable sensor technology adhered to human skin, has garnered significant attention for its potential in brain-computer interfaces (BCIs) and routine health monitoring. Conventionally, flexible substrates with adhesion force on dewy surfaces pursue seamless contact with skin, employing compact airtight substrates, hindering air circulation between skin and the surrounding environment, and compromising long-term wearing comfort. To address these challenges, we have developed a freestanding transparent e-tattoo featuring flexible serpentine mesh bridges with a unique full-breathable multilayer structure. The mesh e-tattoo demonstrates remarkable ductility and air permeability while maintaining robust electronic properties, even after significant mechanical deformation. Furthermore, it exhibits an impressive visible-light transmittance of up to 95%, coupled with a low sheet resistance of 0.268 \textohm sq-1, ensuring both optical clarity and electrical efficiency. By increasing the number of menisci between the mesh e-tattoo and the skin, the total adhesion force increases due to the cumulative capillary-driven effect. We also successfully demonstrated high-quality bioelectric signal collections. In particular, the controlling virtual reality (VR) objects using electrooculogram (EOG) signals collected by mesh e-tattoos were achieved to demonstrate their potential for human-computer interactions (HCIs). This freestanding transparent e-tattoo with a fully breathable mesh structure represents a significant advancement in flexible electrodes for bioelectrical signal monitoring applications.

5. ARTimeTravel: Understanding Spatial Changes in Heritage Sites Over Time through Web-Based Augmented Reality Serious Games

   Jiachen Liang, Gengyuan Zeng, Yue Li^*, and Yiping Dong

   In Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, Apr 2025

   DOI PDF
6. Augmented Reality Continuum: Categorising On-Site Digital Heritage Experiences

   Anna Vichnevetskaia, Yi-Wen Wang, Yue Li, and Nicholas Webb

   In Advances in the Integration of Technology and the Built Environment, 2025

   Abs DOI PDF

   Heritage digitalization has garnered considerable attention in academic research, yet a discernible gap exists in comprehensive studies exploring largescale commercial Augmented Reality (AR) projects. This research seeks to distil key insights and build a preliminary framework to look at both academic and commercial on-site AR heritage experiences to better understand the pragmatic implementations witnessed in the latest AR-driven heritage apps. The paper navigates the use of immersive technologies at heritage sites and uses the AR Continuum to categorize various types of in-situ AR heritage experiences. The goal is to understand the current trends and look for new research directions that will be able to support the rapidly evolving commercialisation of digital heritage.

7. Grand Challenges in Immersive Technologies for Cultural Heritage

   Hanbing Wang, Junyan Du, Yue Li, Lie Zhang, and Xiang Li

   International Journal of Human–Computer Interaction, Mar 2025

   Abs DOI PDF

   Cultural heritage, a testament to human history and civilization, has gained increasing recognition for its significance in preservation and dissemination. The integration of immersive technologies has transformed how cultural heritage is presented, enabling audiences to engage with it in more vivid, intuitive, and interactive ways. However, the adoption of these technologies also brings a range of challenges and potential risks. This paper presents a systematic review, with an in-depth analysis of 177 selected papers. We comprehensively examine and categorize current applications, technological approaches, and user devices in immersive cultural heritage presentations, while also highlighting the associated risks and challenges. Furthermore, we identify areas for future research in the immersive presentation of cultural heritage. Our goal is to provide a comprehensive reference for researchers and practitioners, enhancing understanding of the technological applications, risks, and challenges in this field, and encouraging further innovation and development.

8. ResponsiveView: Enhancing 3D Artifact Viewing Experience in VR Museums

   Xueqi Wang, Yue Li^*, Boge Ling, Han-Mei Chen, and Hai-Ning Liang

   IEEE Transactions on Visualization and Computer Graphics, May 2025

   Abs DOI PDF

   The viewing experience of 3D artifacts in Virtual Reality (VR) museums is constrained and affected by various factors, such as pedestal height, viewing distance, and object scale. User experiences regarding these factors can vary subjectively, making it difficult to identify a universal optimal solution. In this paper, we collect empirical data on user-determined parameters for the optimal viewing experience in VR museums. By modeling users’ viewing behaviors in VR museums, we derive predictive functions that configure the pedestal height, calculate the optimal viewing distance, and adjust the appropriate handheld scale for the optimal viewing experience. This led to our novel 3D responsive design, ResponsiveView. Similar to the responsive web design that automatically adjusts for different screen sizes, ResponsiveView automatically adjusts the parameters in the VR environment to facilitate users’ viewing experience. The design has been validated with two popular inputs available in current commercial VR devices: controller-based interactions and hand tracking, demonstrating enhanced viewing experience in VR museums.

9. The Effects of LLM-Empowered Chatbots and Avatar Guides on the Engagement, Experience, and Learning in Virtual Museums

   Shuhao Zhang, Mingge Ma, Yue Li^*, Ka Lok Man, Jeremy Smith, and Yong Yue

   International Journal of Human–Computer Interaction, May 2025

   Abs DOI PDF

   Virtual museums, as a form of serious game, transcend spatiotemporal constraints of traditional museums through advanced computer technologies, enabling immersive and educational user experiences. With the support of large language models (LLMs) in particular, users can have exploratory interactions with conversational agents. This paper presents a comparative study examining user engagement, experience, and learning across three interaction methods: text labels, chatbots, and avatar guides in a virtual museum. Our study revealed that users were more engaged with the LLM-empowered avatar guide than chatbots and text labels. Additionally, the overall user experience with avatar guides was superior to that of chatbots and text labels. However, no significant differences were observed in users’ learning outcomes across the three interaction methods. We discussed the characteristics of these interaction methods and proposed design recommendations for virtual museums and serious games.

10. HeyThere: Design and Evaluation of an Augmented Reality Multiplayer Social Game:

    Wenxuan Zheng, Yuzheng Wu, Ziming Li, Yue Li, Diego Monteiro, and Hai-Ning Liang

    In Proceedings of the 20th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, 2025

    DOI PDF
11. Voice of Artifacts: Evaluating User Preferences for Artifact Voice in VR Museums

    Bingqing Chen, Wenqi Chu, Xubo Yang, and Yue Li^*

    Computers & Graphics, Dec 2025

    Abs DOI PDF

    Voice is a powerful medium for conveying personality, emotion, and social presence, yet its role in cultural contexts such as virtual museums remains underexplored. While prior research in virtual reality (VR) has focused on ambient soundscapes or system-driven narration, little is known about what kinds of artifact voices users actually prefer, or if customized voices influence their experience. In this study, we designed a virtual museum and examined user perceptions of three types of voices for artifact chatbots, including a neutral synthetic voice (default ), a socially relatable voice (familiar), and a user-customized voice with adjustable elements (customized). Through a within-subjects experiment, we measured user experience with established scales and a semi-structured interview. Results showed a strong user preference for the customized voice, which significantly outperformed the other two conditions. These findings suggest that users not only expect artifacts to speak, but also prefer to have control over the voices, which can enhance their experience and engagement. Our findings provide empirical evidence for the importance of voice customization in virtual museums and lay the groundwork for future design of interactive, user-centered sound and vocal experiences in VR environments.

12. Pingjiang Mystery: A Map-based Board Game for Navigating the Ancient City of Suzhou

    Yanli Jiang, Jiachen Liang, Yingping Zhang, and Yue Li^*

    2025

    Abs PDF

    Globalization and multicultural influences pose significant challenges to the preservation and dissemination of intangible cultural heritage (ICH). Suzhou, a renowned historical city in China, is rich in historical and cultural artifacts. This study employs an interaction design approach, focusing on the ancient map of Pingjiang (平江图), to explore solutions for enhancing the learning and dissemination of Suzhou’s ICH. Through three participatory design workshops involving target users and design experts, we identified essential cultural content, developed relevant game mechanics, and designed an intuitive gameplay interface. The resulting board game, Pingjiang Mystery, effectively supports ICH learning, as evidenced by the positive user experiences, engagement, and educational outcomes observed in evaluation studies. This research contributes to the field of human-computer interaction by presenting an comprehensive case study of a board game design aiming at preserving ICH.

13. User Acceptance of 720^∘ Virtual Tour Systems for Online Museum Experiences: Effects of Immersion, Interaction, and Personal Innovativeness

    Yue Li^*, Jiachen Liang, Enhao Yang, and Hai-Ning Liang

    In Interactive Media for Cultural Heritage, 2025

    DOI PDF
14. ArtifactVM: Exploring Culturally Meaningful Presentations and User Interactions in Virtual Museums

    Jiachen Liang, Yue Li^*, Xueqi Wang, Yuexin Yao, Richard Koeck, and Hai-Ning Liang

    Digital Heritage, 2025

    Abs DOI PDF

    Virtual Museums (VMs) serve as an extension of physical museums, delivering content in digital formats. Virtual Reality (VR) technologies afford the creation of interactive virtual museum experiences with 3D artifacts, enhancing cultural dissemination by narratives based on historical artifacts for educational and entertainment purposes. However, there are notable gaps in exploring the ways to interact with virtual artifacts in VMs. In this work, we conducted workshop studies with non-expert audiences and interviewed domain experts to gather insights into 3D artifacts presentation in VMs. In addition, we investigated the digital affordances of historical artifacts in VMs and discussed opportunities for interaction design. The results provide design guidelines for the forms of presentation and outline interaction possibilities. Our findings provide insights into future forms of content curation and artifact interaction in virtual museums.

15. Enhancing Authenticity in XR Heritage: Practitioner Insights and Preliminary Recommendations

    Anna Vichnevetskaia, Yi-Wen Wang, Nicholas Webb, and Yue Li

    The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Oct 2025

    Abs DOI PDF

    Extended Reality (XR) applications are increasingly popular in heritage interpretation, leading to significant changes in the emergent digital heritage field. One major challenge is ensuring the objective, constructive, and subjective authenticity of XR experiences deployed in a variety of heritage settings. Through in-depth interviews with practitioners working directly on XR projects for cultural heritage, this paper discusses the theoretical and practical implications of authenticity in such projects. It proposes six preliminary recommendations to help practitioners design more authentic, meaningful XR experiences. Firstly, to enhance objective authenticity, the study suggests rigorous research and appropriate renderings based on historically accurate materials. Secondly, for constructive authenticity, the research emphasizes narrative design that incorporates diverse perspectives, including those of local communities and underrepresented voices, to reflect the social and historical significance of heritage sites more accurately. Thirdly, for subjective authenticity, the recommendations highlight the importance of interactive and immersive design that allows for personalized meaningmaking. Finally, the recommendations reinforce the importance of ethics, inclusion, and sustainability across all authenticity concepts.

16. KaiBiLi: Gesture-Based Immersive Virtual Reality Ceremony for Traditional Chinese Cultural Activities

    Yiping Wu, Yue Li^*, Eugene Ch’ng, Jiaxin Gao, and Tao Hong

    Visual Computing for Industry, Biomedicine, and Art, Oct 2025

    Abs DOI PDF

    Gesture-based interactions in a virtual reality (VR) setting can enhance our experience of traditional practices as part of preserving and communicating heritage. Cultural experiences embodied within VR environments are suggested to be an effective approach for experiencing intangible cultural heritage. Ceremonies, rituals, and related ancestral enactments are important for preserving cultural heritage. Kāi Bǐ Lǐ, also known as the First Writing Ceremony, is traditionally held for Chinese children before their first year of elementary school. However, gesture-based immersive VR for learning this tradition is new, and have not been developed within the community. This study focused on how users experienced learning cultural practices using gesture-based interactive VR across different age groups and hardware platforms. We first conducted an experiment with 60 participants (30 young adults and 30 children) using the First Writing Ceremony as a case study in which gestural interactions were elicited, designed, implemented, and evaluated. The study showed significant differences in play time and presence between the head-mounted display VR and desktop VR. In addition, children were less likely to experience fatigue than young adults. Following this, we conducted another study after eight months to investigate the VR systems’ long-term learning effectiveness. This showed that children outperformed young adults in demonstrating greater knowledge retention. Our results and findings contribute to the design of gesture-based VR for different age groups across different platforms for experiencing, learning, and practicing cultural activities.

17. NanyinVR: Gesture-based Immersive Virtual Reality Game for Enhancing Learning of Ancient Chinese Music and Instruments

    Yiping Wu, Kai Chen, and Yue Li^*

    2025

    Abs PDF

    Nanyin is a traditional genre of Chinese music originating from Quanzhou, Fujian, characterized by its distinctive melodies, use of traditional instruments, and rich cultural heritage. Virtual reality has made it more accessible for people to experience, learn, and transmit intangible cultural heritage, contributing to its presentation, preservation, and dissemination. In this paper, we present the design and implementation of NanyinVR, a gesture-based immersive virtual reality game. It consists of two parts: Nanyin learning and instrument playing. We systematically evaluated the usability, user experience, and game experience of NanyinVR with both children and young adults. The results showed that children found it more challenging and showed less competence and flow of game experience; adults reported greater ease of use and ease of learning, and rated higher on usability, pragmatic quality, and overall user experience than children. Our research revealed design implication of demographic difference between age groups in music learning and instrument playing in gesture-based virtual reality.

18. XR Exhibit+: An AI-Enhanced XR Museum Guide Based on Constructivism and Connectivism Theories

    Gengyuan Zeng, Yuchen Yan, Wenqi Chu, Ningjia Duan, Qisheng Huang, Fanpei Liu, and Yue Li^*

    2025

    Abs DOI PDF

    In response to the growing demand for engaging, personalized, and pedagogically grounded museum experiences, we present XR Exhibit+, an intelligent XR museum guide that integrates Extended Reality (XR) and Large Language Models (LLMs), grounded in constructivism and connectivism learning theories. XR Exhibit+ features four core components: annotated tooltips, an LLM-powered conversational interface, a visitor message board, and a semantic exhibit graph, fostering contextual learning, active meaning-making, and knowledge networking. We developed both Augmented Reality (AR) and Virtual Reality (VR) museum guides and conducted a mixed-method study (N = 28) to evaluate their effectiveness.

19. Four-Dimensional Adjustable Electroencephalography Cap for Solid–Gel Electrode

    Junyi Zhang, Deyu Zhao, Yue Li, Gege Ming, and Weihua Pei

    Sensors, Jun 2025

    Abs DOI PDF

    Currently, the electroencephalogram (EEG) cap is limited to a finite number of sizes based on head circumference, lacking the mechanical flexibility to accommodate the full range of skull dimensions. This reliance on head circumference data alone often results in a poor fit between the EEG cap and the user’s head shape. To address these limitations, we have developed a four-dimensional (4D) adjustable EEG cap. This cap features an adjustable mechanism that covers the entire cranial area in four dimensions, allowing it to fit the head shapes of nearly all adults. The system is compatible with 64 channels or lower electrode counts. We conducted a study with numerous volunteers to compare the performance characteristics of the 4D caps with the commercial (COML) caps in terms of contact pressure, preparation time, wearing impedance, and performance in brain–computer interface (BCI) applications. The 4D cap demonstrated the ability to adapt to various head shapes more quickly, reduce impedance during testing, and enhance measurement accuracy, signal-to-noise ratio (SNR), and comfort. These improvements suggest its potential for broader application in both laboratory settings and daily life.

20. Comparing VR and AR in Cultural Heritage Active Learning: A Study Based on the Stimulus-Organism-Response Model and the Engagement Theory

    Jiayi Zhao, Jiachen Liang, Yue Li^*, Cheng Zhang, and Yiping Dong

    Digital Heritage, 2025

    Abs DOI PDF

    With the rapid advancement of immersive technologies, Virtual Reality (VR) and Augmented Reality (AR) have emerged as powerful tools for enhancing cultural heritage experiences. These technologies offer opportunities to engage learners through interactive and rich content, transforming how cultural knowledge is disseminated and understood. In this study, we developed two systems - one in VR with a head-mounted display and the other in AR with a mobile device for cultural heritage exploration. By employing a comprehensive survey with 70 responses, we explored the determinants of active learning within two systems. The results showed that the effects of interactivity and content richness on active learning are mediated by cognitive engagement toward both VR and AR systems. Our findings contribute valuable insights to the field of technology-mediated learning and provide practical guidelines for optimizing immersive cultural heritage experiences through targeted design strategies, highlighting the potential gaps in content design optimization.

21. Towards Immersive Mixed Reality Street Play: Understanding Co-located Bodily Play with See-through Head-Mounted Displays in Public Spaces

    Botao Amber Hu, Rem RunGu Lin, Yilan Elan Tao, Samuli Laato, and Yue Li

    Proceedings of the ACM on Human-Computer Interaction, Oct 2025

    Abs DOI PDF

    As see-through Mixed Reality Head-Mounted Displays (MRHMDs) proliferate, their usage is gradually shifting from controlled, private settings to spontaneous, public contexts. While location-based augmented reality mobile games such as Pokémon GO have been successful, the embodied interaction afforded by MRHMDs moves play beyond phone-based screen-tapping toward co-located, bodily, movement-based play. In anticipation of widespread MRHMD adoption, major technology companies have teased concept videos envisioning urban streets as vast mixed reality playgrounds-imagine Harry Potter-style wizard duels in city streets-which we term Immersive Mixed Reality Street Play (IMRSP). However, few real-world studies examine such scenarios. Through empirical, in-the-wild studies of our research-through-design game probe, Multiplayer Omnipresent Fighting Arena (MOFA), deployed across diverse public venues, we offer initial insights into the social implications, challenges, opportunities, and design recommendations of IMRSP. The MOFA framework, which includes three gameplay modes-”The Training”, ”The Duel”, and ”The Dragon”-is open-sourced at https://github.com/realitydeslab/mofa.

22. MemoryVR: Collecting and Sharing Memories in Personal Virtual Museums

    Jiachen Liang, Yue Li, Xueqi Wang, Ziyue Zhao, and Hai-Ning Liang

    International Journal of Human–Computer Interaction, Sep 2025

    DOI PDF

2024

1. Awkward or Acceptable? Understanding the Bystander Perspective on the Ubiquity of Cross Reality in Ambiguous Social Situations

   Bingqing Chen, Yue Li^*, Botao Amber Hu, and Yilan Elan Tao

   In 2024 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), Oct 2024

   DOI PDF
2. Data Cubes in Hand: A Design Space of Tangible Cubes for Visualizing 3D Spatio-Temporal Data in Mixed Reality

   Shuqi He, Haonan Yao, Luyan Jiang, Kaiwen Li, Nan Xiang, Yue Li, Hai-Ning Liang, and Lingyun Yu

   In Proceedings of the CHI Conference on Human Factors in Computing Systems, May 2024

   DOI PDF
3. Exploring the Effects of Spatial Constraints and Curvature for 3D Piloting in Virtual Environments

   Xuning Hu, Xinan Yan, Yushi Wei, Wenxuan Xu, Yue Li, Yue Liu, and Hai-Ning Liang

   In 2024 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Oct 2024

   DOI PDF
4. On Intent Inclusivity in Spontaneous Cross Realities

   Botao Amber Hu, Yilan Elan Tao, Rem RunGu Lin, and Yue Li

   In 2024 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), Oct 2024

   DOI PDF
5. Blending Social Interaction Realms: Harmonizing Online and Offline Interactions through Augmented Reality

   Guanxuan Jiang, Yuyang Wang, Yue Li, Nafise Sadat Moosavi, and Pan Hui

   In Proceedings of the 17th International Symposium on Visual Information Communication and Interaction, Dec 2024

   Abs DOI PDF

   Online social media has revolutionized human interaction by fostering unparalleled cooperation and connectivity, surpassing the bounds of conventional, location-based methods. Despite their inherent limitations—such as physical boundaries, ongoing maintenance expenses, and rigidity—traditional methods impart a vital local context often neglected by digital platforms, potentially overshadowing local environmental engagement in favor of broader online networks. To mitigate this imbalance, it is essential to revitalize the significance of location-specific interactions. Augmented Reality (AR) stands out as a powerful means to enhance the accessibility and allure of such engagements. In this context, our research involved a detailed review and synthesis of existing location-based interactive services, pinpointing prevalent obstacles as well as offering strategic recommendations. Building upon these findings, we innovated ARMessageBoard, a prototype fusing AR with users’ immediate surroundings to craft virtual message boards. Our within-subjects study comprised 15 participants with an average age of 21.2 years (SD=5.5), systematically comparing ARMessageBoard with standard location-based mechanisms. Furthermore, we deliberated how blending AR with online social media could positively influence the convergence of virtual and real-world interaction landscapes, potentially enriching the individual’s role in shaping socio-digital exchanges.

6. ChemistryVR: Enhancing Educational Experiences through Virtual Chemistry Lab Simulations

   Guanxuan Jiang, Xuansheng Xia, Yue Li^*, Hai-Ning Liang, and Pan Hui

   In SIGGRAPH Asia 2024 Educator’s Forum, Dec 2024

   DOI PDF
7. Is VR Always a Better Choice? Investigating the Effects of Game Modes and Role-Playing on Fire Escape Simulation Training

   Zelin Jiang, Shuhao Zhang, Yue Li^*, Ka Lok Man, Yong Yue, and Jeremy Smith

   In 2024 10th International Conference on Virtual Reality (ICVR), Jul 2024

   Abs DOI PDF

   In this paper, we present a multi-user fire escape simulation training system that involves an actionist in Virtual Reality (VR) and a strategist using a desktop. We implemented two game modes (collaboration and competition) and conducted a comparative study to investigate how user experiences and learning outcomes vary between the two game modes, and between the two roles in the gameplay. The learning outcomes using the simulation training were compared against a baseline condition, where participants learned the fire escape knowledge by reading paper instructions. Our results revealed that users reported higher perceived usability and lower workload in the collaboration mode than in the competition mode. In addition, actionists (VR users) reported greater performance but also greater mental workload than strategists (desktop users). In terms of learning outcomes, strategists showed greater improvement than actionists. However, the improvement in learning outcomes did not vary significantly from the baseline condition. We discussed the effects of game modes and role-playing on user experience and learning outcomes and the implications for future interactive educational systems.

8. Examining the Use of \emphDanMu for Crowdsourcing Control in Virtual Gatherings

   Yue Li, Teng Ma, Ziming Li, and Hai-Ning Liang

   International Journal of Human–Computer Interaction, Jul 2024

   Abs DOI PDF

   The advent of web-based interactive technologies has opened up new possibilities for virtual gatherings in 3D environments. Live-streaming, in particular, has gained increasing attention due to its effectiveness in engaging a large number of users in collective online activities. With an emphasis on audience participation, live-streaming shares common characteristics of the outlook of the metaverse and is driving new waves of interaction in virtual gatherings, such as engaging users through crowdsourcing control. However, this type of social interaction has not been exam\-ined in the Asian context, and it lacks systematic investigation of user experience with different crowdsourcing control methods. In this paper, we present a novel crowdsourcing control method based on DanMu, the subtitle system of Bilibili, one of the most successful and prevalent livestreaming platforms in Asia. We organized virtual gatherings by live-streaming a Minecraft virtual campus and examined the use of DanMu for crowdsourcing control. Our first study investigated the influence of three crowdsourcing control methods (First Come First Served, Vote, and Super Command) on collective navigation task efficiency and user experience. These influences were fur\-ther discussed with user activeness and group sizes in a follow-up study. The results showed that Super Command, a representative mode on top of the democratic voting mechanism, offers better user experiences and social richness in large groups. Participants also rated its usability higher in small groups. Besides, virtual gathering in small groups allows greater pragmatic quality, usability, and a sense of agency than in large groups. Our work provides design guidelines for developers and HCI practitioners to develop crowdsourcing control methods and improve novel virtual gath\-ering experiences in virtual worlds and the future metaverse.

9. MemoryVR: Collecting and Sharing Memories in Personal Virtual Museums

   Jiachen Liang, Yue Li^*, Xueqi Wang, Ziyue Zhao, and Hai-Ning Liang

   In 2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), Mar 2024

   Abs DOI PDF

   We present MemoryVR, a virtual museum system designed to preserve and share personal memories. This system enables users to create customized virtual museums within a spatial enclosure, providing an immersive and enriched way to experience personal memories. We invited participants to use MemoryVR to create their own personal virtual museums and visit those created by others. Results from evaluation studies showed a positive impact of MemoryVR on their experience of memories. Participants reported that their experiences within the personal virtual museums were fulfilling, invoking a sense of ritual, ownership, curiosity, and engagement.

10. User-Defined Gesture Interactions for VR Museums: An Elicitation Study

    Qianru Liu, Yue Li^*, Bingqing Chen, Huiyue Wu, and Hai-Ning Liang

    In 2024 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Oct 2024

    DOI PDF
11. Exploring Controller-based Techniques for Precise and Rapid Text Selection in Virtual Reality

    Jianbin Song, Rongkai Shi, Yue Li, BoYu Gao, and Hai-Ning Liang

    In 2024 IEEE Conference Virtual Reality and 3D User Interfaces (VR), Mar 2024

    Abs DOI PDF

    Text selection is a common task in interactive systems. Often, it can be difficult because the letters and words are too small and clustered together to allow precise selection. Compared to traditional 2D interfaces, text selection is more challenging in virtual reality (VR) head-mounted displays (HMDs) because users interact with the immersive 3D space via mid-air interaction, which has higher degrees of freedom but becomes more imprecise and involves a higher workload due to the lack of support from a fixed structure like a desk. There has been limited exploration of techniques that support precise and rapid text selection at the character, word, sentence, or paragraph levels in VR HMDs. To fill this gap, we propose three controller-based text selection methods: Joystick Movement, Depth Movement, and Wrist Orientation. They are evaluated against a baseline selection method via a user study with 24 participants. Results show that the three proposed techniques significantly improved the performance and user experience over the baseline, especially for the selection beyond the character level.

12. Hands-Free Multi-type Character Text Entry in Virtual Reality

    Tingjie Wan, Rongkai Shi, Wenge Xu, Yue Li, Katie Atkinson, Lingyun Yu, and Hai-Ning Liang

    Virtual Reality, Mar 2024

    Abs DOI PDF

    Multi-type characters, including uppercase and lowercase letters, symbols, and numbers, are essential in text entry activities. Although multi-type characters are used in passwords, instant messages, and document composition, there has been limited exploration of multi-character text entry for virtual reality head-mounted displays (VR HMDs). Typically, multi-type character entry requires four kinds of keyboards between which users need to switch. In this research, we explore hands-free approaches for rapid multi-type character entry. Our work explores two efficient and usable hands-free approaches for character selection: eye blinks and dwell. To enable quick switching between keyboards, we leverage the usability and efficiency of continuous head motions in the form of cross-based activation. In a pilot study, we explored the usability and efficiency of four locations of the switch keys, the two hands-free selection mechanisms, and crossing-based switching. In the main experiment, we evaluated four user-inspired layouts designed according to the findings from the pilot study. Results show that both blinking and dwell can work well with crossing-based switching and could lead to a relatively fast text entry rate (5.64 words-per-minute (WPM) with blinking and 5.42 WPM with dwell) with low errors (lower than 3% not corrected error rate (NCER)) for complex 8-digit passwords with upper/lowercase letters, symbols, and numbers. For sentences derived from the Brown Corpus, participants can reach 8.48 WPM with blinking and 7.78 WPM with dwell. Overall, as a first exploration, our results show that it is usable and efficient to perform hands-free text entry in VR using either eye blinks or dwell for character selection and crossing for mode switching.

13. MagicMap: Enhancing Indoor Navigation Experience in VR Museums

    Xueqi Wang, Yue Li^*, and Hai-Ning Liang

    In 2024 IEEE Conference Virtual Reality and 3D User Interfaces (VR), Mar 2024

    Abs DOI PDF

    Museum visitors are typically advised to follow trajectories planned by curators. Nevertheless, the diverse locomotion techniques available in Virtual Reality (VR) offer various navigation methods that are unattainable within physical museum spaces. Interestingly, these techniques have rarely been explored within museum settings. Our study aims to investigate appropriate navigation methods in VR museums. We first conducted a study in a virtual reconstruction of a local museum with the following navigation methods: a 2D minimap, a World-in-Miniature (WiM) system, and a WiM map. Our results showed that the WiM map with a point-and-select interaction technique outperformed the other two regarding ease of learning, reduced workload, lessened motion sickness, and greater user preferences. Based on the findings, we improved the WiM map and introduced MagicMap. It builds upon the WiM map and translates the curatorial principles of museum visiting into a hierarchical menu layout. Our further evaluation showed that MagicMap supported prolonged engagement in VR museums, enhanced system usability and overall user experience, and reduced users’ perceived workload. Our findings have implications for the future design of navigation systems in VR museums and complex indoor environments.

14. The Second Joint Workshop on Cross Reality

    Nanjia Wang, Yue Li, Francesco Chiossi, Fabian Pointecker, Lixiang Zhao, and Daniel Zielasko

    2024

    Abs DOI PDF

    The 2nd Joint Workshop on Cross Reality (JWCR’24), organized as part of ISMAR 2024, seeks to explore the burgeoning field of Cross Reality (CR), which encompasses the seamless integration and transition between various points on the reality-virtuality continuum (RVC) such as Virtual Reality (VR), Augmented Virtuality (AV), and Augmented Reality (AR). This hybrid workshop aims to build upon the foundation laid by the inaugural JWCR at ISMAR 2023, which successfully unified diverse CR research communities. The workshop will address key themes including CR visualization, interaction, user behavior, design, development, engineering, and collaboration. CR Visualization focuses on creating and displaying spatial data across the RVC, enabling users to navigate and interpret information fluidly. CR Interaction delves into natural user engagements using gestures, voice commands, and other advanced techniques to enhance immersion. The study of CR User Behavior and Experience investigates how users perceive and interact within these hybrid environments. Furthermore, CR Design and Development emphasizes creating effective CR applications using innovative processes and tools, while CR Collaboration examines methods for fostering teamwork in mixed reality settings.

15. Vanti: A Novel Interaction Design for Immersive VR Walking Tours

    Xiaojie Wang and Yue Li^*

    In 2024 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), Oct 2024

    DOI PDF
16. Evaluating and Modeling the Effect of Frame Rate on Steering Performance in Virtual Reality

    Yushi Wei, Rongkai Shi, Anil Ufuk Batmaz, Yue Li, Mengjie Huang, Rui Yang, and Hai-Ning Liang

    IEEE Transactions on Visualization and Computer Graphics, 2024

    Abs DOI PDF

    Prior work has shown that frame rate significantly influences user behavior in fast-response tasks in 2D and 3D contexts. However, its impact on a steering task, which involves navigating an object along a path from the start to the end, remains relatively unexplored, especially in the context of virtual reality (VR). This task is considered a typical non-fast-response activity, as it does not demand rapid reactions within a limited time frame. Our work aims to understand and model users’ steering behavior and predict movement time with different task complexities and frame rates in VR environments. We first conducted a user study to collect user behavior in a steering task with four factors: frame rate, path length, width, and radius of curvature. Based on the results, we then quantified the effects of frame rate and built two predictive models. Our models exhibited the best fit (r2 > 0.957) and over 17% improvement in prediction accuracy for movement time compared to existing models. Our models’ robustness was further validated by applying them to predict steering performance with different VR tasks and frame rates. The two models keep the best predictability for both movement time and speed.

17. Exploring and Modeling Directional Effects on Steering Behavior in Virtual Reality

    Yushi Wei, Kemu Xu, Yue Li, Lingyun Yu, and Hai-Ning Liang

    IEEE Transactions on Visualization and Computer Graphics, Nov 2024

    Abs DOI PDF

    Steering is a fundamental task in interactive Virtual Reality (VR) systems. Prior work has demonstrated that movement direction can significantly influence user behavior in the steering task, and different interactive environments (VEs) can lead to various behavioral patterns, such as tablets and PCs. However, its impact on VR environments remains unexplored. Given the widespread use of steering tasks in VEs, including menu adjustment and object manipulation, this work seeks to understand and model the directional effect with a focus on barehand interaction, which is typical in VEs. This paper presents the results of two studies. The first study was conducted to collect behavioral data with four categories: movement time, average movement speed, success rate, and reenter times. According to the results, we examined the effect of movement direction and built the S\texttheta Model. We then empirically evaluated the model through the data collected from the first study. The results proved that our proposed model achieved the best performance across all the metrics (r2 > 0.95), with more than 15% improvement over the original Steering Law in terms of prediction accuracy. Next, we further validated the S\texttheta Model by another study with the change of device and steering direction. Consistent with previous assessments, the model continues to exhibit optimal performance in both predicting movement time and speed. Finally, based on the results, we formulated design recommendations for steering tasks in VEs to enhance user experience and interaction efficiency.

18. Covisit\textsuperscriptVM : Cross-Reality Virtual Museum Visiting

    Xuansheng Xia, Yue Li^*, and Hai-Ning Liang

    In 2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), Mar 2024

    Abs DOI PDF

    Virtual Reality Head-Mounted Displays (VR HMDs) are the main ways for users to immerse in a virtual environment and interact with its virtual objects. The experiences of those around the VR HMD users and their effects on HMD users’ experiences have not been well studied. In this work, we invite participants to engage in a cross-reality virtual museum visit. With low, medium, and high degrees of non-HMD user involvement, they could incrementally observe, navigate within, and interact with the virtual museum. Our study provides insights into the design of engaging multiuser VR experiences and cross-reality collaborations.

19. HeritageSite AR: Design and Evaluation of a Mobile Augmented Reality Exploration Game for a Chinese Heritage Site

    Ningning Xu, Yue Li^*, Jiachen Liang, Kexiang Shuai, Yuwen Li, Jiaqi Yan, Cheng Zhang, and Yiping Dong

    Journal on Computing and Cultural Heritage, Oct 2024

    Abs DOI PDF

    This paper explores the use of a mobile Augmented Reality (AR) exploration game to enhance immersive storytelling and enrich cultural experiences. Specifically, we present the prototype design and evaluation of HeritageSite AR, an AR exploration game for a Chinese heritage site known as the Relics of Arhat Monastery and Twin Pagoda , or Shuangta . To develop the AR game for use in heritage sites, we employed a holistic approach, beginning with a review of technical means for cultural application development. We then conducted semi-structured interviews with domain experts and administered an online survey to identify user requirements and design goals, which informed our prototype design. An evaluation study showed positive feedback regarding the impact of game design on meaningful and playful CH site experience, and identified areas for potential refinements in future iterations. We discuss the implications and lessons learned for our future work, which may also interest researchers and practitioners exploring the use of AR technologies and game design in heritage site contexts.

20. Exploring Embodied Asymmetric Two-Handed Interactions for Immersive Data Exploration

    Haonan Yao, Lixiang Zhao, Hai-Ning Liang, Yu Liu, Yue Li, and Lingyun Yu

    In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, May 2024

    Abs DOI PDF

    Embodied interaction plays a crucial role in facilitating effective data exploration within immersive environments, enhancing user experience, understanding, and exploring complex data presented in the virtual space. While embodied two-handed interaction has demonstrated considerable potential, there remains a gap in understanding how varying levels of embodiment impact asymmetric two-hand interactions for immersive data exploration. In this study, we systematically investigate this aspect by combining three settings (direct, indirect, and fixed) on the visualization control hand and two settings (direct, indirect) on the action hand. This combination results in six conditions that span varying levels of embodiment. We compared these conditions under two fundamental visualization tasks, focusing on curve brushing and object manipulation. Our discussion revolves around the use of techniques related to the specific requirements of the tasks, the characteristics of each condition, and users’ experience and expertise in the VR environment. Building upon these discussions, we offer suggestions for designing embodied two-handed interactions for immersive data exploration.

21. Textual Information Presentation in Virtual Museums: Exploring Environment-, Object-, and User-based Approaches

    Yuexin Yao and Yue Li^*

    In 2024 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Oct 2024

    DOI PDF
22. Understanding User Experience, Task Performance, and Task Interdependence in Symmetric and Asymmetric VR Collaborations

    Shuhao Zhang, Yue Li^*, Ka Lok Man, Jeremy S. Smith, and Yong Yue

    Virtual Reality, Dec 2024

    Abs DOI PDF

    Asymmetric collaboration is an important topic for the research of multiuser collaborative systems. Previous works have shown that by providing different abilities, devices or content to different users, users can take advantage of the unique features of each side and collaborate effectively with each other. However, there is limited work comparing the differences between asymmetric and symmetric Virtual Reality (VR) collaboration systems. How task complexity may affect symmetric and asymmetric VR collaboration is also unclear. In this paper, we present a comparative study that investigated how user experiences and task performance vary in symmetric and asymmetric VR collaboration. In addition, we also explored how task interdependence correlates with user experience and task performance. Participants were asked to collaboratively perform 3D object selection and manipulation tasks in pairs. A within-subjects study was conducted, where participants used PC and PC, VR and VR, and PC and VR, respectively in three conditions. Our results revealed that the asymmetric collaboration using both PC and VR showed the best results in closeness of relationship, social presence and task performance; the PC symmetric collaborative system showed the worst user experiences and task performance. Both user experience and task performance showed a positive correlation with task interdependence. We discussed the effects of the collaborativ mode and device on the user experience and task performance, and the implications for future symmetric and asymmetric VR collaboration systems.

23. AirWhisper: Enhancing Virtual Reality Experience via Visual-Airflow Multimodal Feedback

    Fangtao Zhao, Ziming Li, Yiming Luo, Yue Li, and Hai-Ning Liang

    Journal on Multimodal User Interfaces, Aug 2024

    Abs DOI PDF

    Virtual reality (VR) technology has been increasingly focusing on incorporating multimodal outputs to enhance the sense of immersion and realism. In this work, we developed AirWhisper, a modular wearable device that provides dynamic airflow feedback to enhance VR experiences. AirWhisper simulates wind from multiple directions around the user’s head via four micro fans and 3D-printed attachments. We applied a Just Noticeable Difference study to support the design of the control system and explore the user’s perception of the characteristics of the airflow in different directions. Through multimodal comparison experiments, we find that vision-airflow multimodality output can improve the user’s VR experience from several perspectives. Finally, we designed scenarios with different airflow change patterns and different levels of interaction to test AirWhisper’s performance in various contexts and explore the differences in users’ perception of airflow under different virtual environment conditions. Our work shows the importance of developing human-centered multimodal feedback adaptive learning models that can make real-time dynamic changes based on the user’s perceptual characteristics and environmental features.

24. Enhancing Virtual Reality Experience: Cognitive Fulfilment’s Impact on Customer Attitude

    Weihan Liang, Yue Li, and Jiyao Xun

    In International Conference on Entrepreneurship, Technology and Social Sciences 2024, 2024
25. Towards Modelling Distracted Human Driving: Sensor Fusion and Driver Status Monitoring in a VR Simulator

    Jiacheng Liu, Yue Li, and Fan Zhang

    In Proceedings of the Twelfth International Symposium of Chinese CHI, Nov 2024

    DOI PDF
26. Digital Corpse Donator: A Caring Digital Burial Process

    Fangze Qiu and Yue Li^*

    In Proceedings of the Twelveth International Symposium of Chinese CHI, 2024
27. Eyes in the Sky

    Xueqi Wang and Yue Li^*

    In Proceedings of the Twelveth International Symposium of Chinese CHI, 2024
28. MemoryVR: Collecting and Sharing Memories in Personal Virtual Museums

    Jiachen Liang, Yue Li^*, Xueqi Wang, Ziyue Zhao, and Hai-Ning Liang

    In 2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), Mar 2024

    Abs DOI PDF

    The development of virtual reality (VR) technology has allowed virtual museums to enrich experiences of personal memories. In this work, we first conducted a survey study to understand the way people use to record, store, and share their digital memories. Informed by the results, we present MemoryVR, a personal virtual museum system designed to preserve and share digital memories. It allows individuals to organize and share their own memories in a virtual museum environment and to visit the memory spaces of others in an immersive way. We invited participants to experience MemoryVR and analyzed their behavior and expectations. The evaluation results showed that users perceived excellent pragmatic and hedonic qualities of MemoryVR. Participants found their experiences of memories in personal virtual museums to be fulfilling. Additionally, we gathered suggestions from participants about MemoryVR, providing design recommendations for customized personal virtual museums.

2023

1. HapticBox: Designing Hand-Held Thermal, Wetness, and Wind Stimuli for Virtual Reality

   Kedong Chai, Yue Li^*, Lingyun Yu, and Hai-Ning Liang

   In 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), Mar 2023

   Abs DOI PDF

   Experiences in virtual reality (VR) through multiple sensory modalities can be as rich as real-world experiences. However, many VR systems offer only visual and auditory stimuli. In this paper, we present HapticBox, a small, portable, and highly adaptable haptic device that can provide hand-held thermal, wetness, and wind haptics. We evaluated user perception of wetness and wind stimuli in the hand and to the face. The results showed that users had a stronger perception of the stimuli and a higher level of comfort with haptics in the hand. While increasing the voltage enhanced the wind perception, the results suggested that noise is an important side effect. We present our design details and discuss the future work.

2. AR.S.Space: An AR Casual Game for Social Engagement in Work Environments

   Boyuan Chen, Junkun Long, Wenxuan Zheng, Yuzheng Wu, Ziming Li, Yue Li, and Hai-Ning Liang

   In 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), Oct 2023

   Abs DOI PDF

   In social situations, individuals often encounter communication challenges, particularly when adapting to new environments. While some studies have acknowledged the potential of AR social games to aid in effective socialization to some extent, little attention has been given to AR HMD-based games specifically designed to facilitate social interactions. In response, we propose AR.S.Space, an AR HMD-based social game that employs augmented reality features to engage users with virtual social agents through asynchronous communication. The game aims to mitigate the unease associated with initial social interactions and foster long-term connections. To assess its efficacy, a user study was conducted within a specific scenario (an office space), gathering quantitative data and qualitative feedback through questionnaires and interviews. The findings highlight the game’s potential to enhance socialization in small-scale environments. Moreover, the study offers valuable design guidelines for future research and the application of AR social games in similar settings.

3. Hanfu AR: Digital Twins of Traditional Chinese Costumes for Augmented Reality Try-On Systems

   Yukun Fu and Yue Li^*

   In 2023 IEEE 47th Annual Computers, Software, and Applications Conference (COMPSAC), Jun 2023

   Abs DOI PDF

   We present Hanfu AR, an Augmented Reality (AR) try-on system that presents digital twins of traditional Chinese costumes based on Kinect. The system allows users to virtually try on 3D clothing with real-time interactions and realistic cloth simulation. Specifically, we present an optimized framework that addresses four aspects of the development of digital twins of virtual clothing and try-on systems: calibration, cloth simulation, control, and configuration. Our work contributes to the development of realistic digital twins of virtual clothing and interactive try-on systems. The system can be applied in various areas and has great values in design, culture, education, and marketing. The proposed framework will benefit the future development of digital twins of virtual clothing for applications in the Metaverse.

4. Avatar Type, Self-Congruence, and Presence in Virtual Reality

   Tianqi Huang, Yue Li^*, and Hai-Ning Liang

   In Proceedings of the Eleventh International Symposium of Chinese CHI, Nov 2023

   Abs DOI PDF

   Avatars serve as users’ virtual identities and hold a significant role in shaping the user experience within the realm of Virtual Reality (VR). The appearance of individual avatars and the perceived selfcongruence within the environment are likely to influence users’ perceived presence in VR. In this paper, we present a study that investigates four types of avatars in VR: anime, human, animal, and item. Participants were asked to choose an avatar before entering a virtual environment (classroom, gallery, café, street, and forest) populated with avatars of different types and to evaluate their perceived self-congruence within the environment and the perceived presence. Our study results showed no significant difference in presence when users use different avatars. However, there is a correlation between users’ perceived self-congruence and social presence. We discuss the findings and provide suggestions for the future use of avatars in VR.

5. Easy Induction: A Serious Game Using Participatory Design

   Yuwen Li, Yue Li^*, Jiachen Liang, and Hai-Ning Liang

   In Computer-Human Interaction Research and Applications, 2023

   Abs DOI PDF

   College freshmen often face difficulties adjusting to the new academic and social environment of university life. It is critical to help them adapt to academic and personal life, while also improving their sense of belonging and engagement with the university. In this paper, we focus on the context of an international joint venture university, Xi’an Jiaotong-Liverpool University (XJTLU), and present a participatory design approach to identify potential solutions collaboratively. We conducted three participatory design workshops with freshmen in undergraduate and postgraduate studies, where we discovered specific challenges, developed serious game content and design alternatives, and delivered a board game that supports academic and social integration at XJTLU. To evaluate the effectiveness of the board game, we collected both quantitative and qualitative data. The quantitative analysis revealed that the board game is effective in improving freshmen’s knowledge acquisition of academic affairs, increasing their familiarity with the environment and resources, and enhancing their ability to access information and resources. The board game also received high scores in system usability and user experience. The qualitative analysis indicated that the board game was engaging, interesting, and well-received by students. They found the board game helpful in their academic and social integration and expressed a desire to play it again in the future. Our participatory design approach and the resulting board game provide a promising avenue for universities to support freshmen’s transition to university life.

6. Factors Influencing Engagement in Hybrid Virtual and Augmented Reality

   Yue Li^*, Eugene Ch’ng, and Sue Cobb

   ACM Transactions on Computer-Human Interaction, Aug 2023

   Abs DOI PDF

   Hybridity in immersive technologies has not been studied for factors that are likely to influence engagement. A noticeable factor is the spatial enclosure that defines where users meet. This involves a mutual object of interest, contents that the users may generate around the object, and the proximity between users. This study examines these factors, namely how object interactivity, user-generated contents (UGC) and avatar proximity influence engagement. We designed a Hybrid Virtual and Augmented Reality (HVAR) environment that supports paired users to experience cultural heritage in both Virtual Reality (VR) and Augmented Reality (AR) . A user study was conducted with 60 participants, providing assessments of engagement and presence via questionnaires, together with mobile electroencephalogram (mEEG) and user activity data that measures VR user engagement in real-time. Our findings provide insights into how engagement between users can occur in HVAR environments for the future hybrid reality with multi-device connectivity.

7. Understanding the Needs of Virtual Reality for Learning and Teaching: A User-Centered Approach

   Yue Li^*, Jiachen Liang, Yuang Zhou, Cheng Zhang, Yong Yue, and Hai-Ning Liang

   In 2023 3rd International Conference on Educational Technology (ICET), Sep 2023

   Abs DOI PDF

   The emergence of COVID-19 has had a significant impact on the education field, leading to a surge in the adoption of online learning and teaching. The recent development in Virtual Reality (VR) and metaverse has witnessed an increasing number of online platforms being utilized in online education. In this study, we took a user-centered approach and conducted a series of survey and interview studies with students and teachers to understand their needs of VR for learning and teaching. Additionally, we evaluated existing online platforms that can serve as virtual classrooms to host teaching materials and support students in online learning. The comparison results together with the requirements we summarized offer valuable takeaways and guides for the future adoption and creation of virtual classrooms for VR-enhanced learning and teaching.

8. A Study of Zooming, Interactive Lenses and Overview+Detail Techniques in Collaborative Map-based Tasks

   Yu Liu, Zhichao Zhang, Yushan Pan, Yue Li, Hai-Ning Liang, Paul Craig, and Lingyun Yu

   In 2023 IEEE 16th Pacific Visualization Symposium (PacificVis), Apr 2023

   DOI PDF
9. Interactive Visualization of Sport Climbing Data

   Fangze Qiu and Yue Li^*

   In Human-Computer Interaction – INTERACT 2023, 2023

   Abs DOI PDF

   The official website of the International Federation of Sport Climbing (IFSC) stores information about sport climbing competitions and athletes. While the website shows comprehensive data, it was mainly static and there was limited interaction or effective visualization, impeding the attempts to understand the performance of the athletes. To address this problem, we developed IFSC+, an interactive visualization system for sport climbing data from the IFSC official website. This paper details the design of the interactive visualizations, highlighting how they can be used to compare athlete performance and identify promising candidates in future competitions. Our work demonstrates the value of interactive visualizations in supporting effective meaning-making and informed decision-making in sports.

10. Expanding Targets in Virtual Reality Environments: A Fitts’ Law Study

    Rongkai Shi, Yushi Wei, Yue Li, Lingyun Yu, and Hai-Ning Liang

    In 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), Oct 2023

    DOI PDF
11. Effects of Object Complexity in Occlusion, Structure, and Texture on 3D Virtual Object Observation in Virtual Reality

    Kexiang Shuai, Yue Li^*, and Hai-Ning Liang

    In 2023 Asia Conference on Cognitive Engineering and Intelligent Interaction (CEII), Dec 2023

    Abs DOI PDF

    Virtual Reality (VR) environments involve users in 3D virtual object interactions and manipulation tasks. Many of these are for the purpose of 3D virtual object observation, such as viewing a reconstructed museum artifact in a virtual museum. In this paper, we present a study that investigated the effects of object complexity in occlusion, structure, and texture on 3D virtual object observation in VR. We implemented a direct manipulation technique that allows users to grab, move, rotate, and scale an object for close-up observations. Twenty participants used the technique to manipulate virtual objects of various levels of complexity in occlusion, structure, and texture, to complete observation tasks (search and classify marks). The results showed that among the three dimensions of object complexity, occlusion and texture have significant impacts on users’ observation task completion time, but structure showed no significant impact. Our work contributes to the understanding of object complexity for 3D object observation in VR environments.

12. Comparative Analysis of Artefact Interaction and Manipulation Techniques in VR Museums: A Study of Performance and User Experience

    Yifan Wang, Yue Li^*, and Hai-Ning Liang

    In 2023 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Oct 2023

    Abs DOI PDF

    For museums in Virtual Reality (VR), various interaction and manipulation techniques could be employed for users to engage with artefact interactions. This study examined four combinations of interaction (controller-based and hand-tracking) and manipulation (direct and indirect) techniques, assessing user performance and experience with these interaction techniques in a virtual museum environment. We conducted a within-subjects experiment and asked participants to perform a series of transform manipulation tasks using the four techniques. Participants’ task completion time was measured. They also provided feedback on acceptance, learnability, presence, sickness, and fatigue, and gave an overall ranking through post-experiment questionnaires and interviews. The results revealed that controller-based direct manipulation outperformed the other techniques in terms of task performance and user experience, with hand-tracking indirect manipulation being the least efficient and the least preferred option. The study offers insights for future research and development in refining interaction and manipulation techniques and designing more user-friendly VR museum experiences.

13. Evaluating User Performance, Workload, and Presence of Virtual Reality Questionnaires Using Joystick and Raycasting Selection Techniques

    Xingbo Wei and Yue Li^*

    In Proceedings of the 2023 7th International Conference on Virtual and Augmented Reality Simulations, Mar 2023

    Abs DOI PDF

    Understanding users’ subjective feelings is vital for Virtual Reality (VR) research, and questionnaire is one of the common used approaches to obtain subjective feedback. Embedding questionnaires into VR systems has been shown effective in reducing the break in presence (BIP) and systematic bias compared to filling out questionnaire outside VR. However, it is not clear how users perform and perceive workload and presence of VR questionnaires, and there is no clear guideline for choosing appropriate selection techniques. In this paper, we present an experimental study that examined user performance, workload, and presence of VR questionnaires, and compared them to the use of PC. We investigated two commonlyused selection techniques in VR (joystick selection and raycasting selection) and three question types (radio, block, and slider). Our results showed that despite the benefits of in-VR questionnaires, user performance was better and workload was lower outside VR using a PC. Comparing joystick and raycasting, user workload is slightly lower using raycasting selection, whereas joystick better supports precise selections. There is room for optimizing existing VR questionnaire design and developing novel selection techniques for VR questionnaires.

14. Predicting Gaze-based Target Selection in Augmented Reality Headsets Based on Eye and Head Endpoint Distributions

    Yushi Wei, Rongkai Shi, Difeng Yu, Yihong Wang, Yue Li, Lingyun Yu, and Hai-Ning Liang

    In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, Apr 2023

    Abs DOI PDF

    Target selection is a fundamental task in interactive Augmented Reality (AR) systems. Predicting the intended target of selection in such systems can provide users with a smooth, low-friction interaction experience. Our work aims to predict gaze-based target selection in AR headsets with eye and head endpoint distributions, which describe the probability distribution of eye and head 3D orientation when a user triggers a selection input. We first conducted a user study to collect users’ eye and head behavior in a gaze-based pointing selection task with two confirmation mechanisms (air tap and blinking). Based on the study results, we then built two models: a unimodal model using only eye endpoints and a multimodal model using both eye and head endpoints. Results from a second user study showed that the pointing accuracy is improved by approximately 32% after integrating our models into gaze-based selection techniques.

15. Cross-Reality Interaction and Collaboration in Museums, Education, and Rehabilitation

    Xuansheng Xia, Jiachen Liang, Ruixiang Zhao, Ziyue Zhao, Mingze Wu, Yue Li^*, and Hai-Ning Liang

    In 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), Oct 2023

    Abs DOI PDF

    With Virtual Reality Head-Mounted Displays (VR HMDs) establishing themselves as a potent platform for collaborative tasks, their cross-reality capability and cross-domain applicability remain largely unexplored. This study intends to assess the effectiveness of cross-reality collaboration systems using a VR HMD and a desktop PC across three disparate sectors: museum visiting, chemical education, and assisted rehabilitation. The systems were designed to support social interactions and scenario-specific collaborative tasks. Evaluation of the systems showed above-average system usability and user experience. By probing into these varied environments, our study offers a comprehensive understanding of the applicability of such collaborative cross-reality systems in real scenarios, potentially fostering more immersive, efficient, and enriching multi-field applications of cross-reality technologies.

16. CubeMuseum AR: A Tangible Augmented Reality Interface for Cultural Heritage Learning and Museum Gifting

    Ningning Xu, Yue Li^*, Xingbo Wei, Letian Xie, Lingyun Yu, and Hai-Ning Liang

    International Journal of Human–Computer Interaction, 2023

    Abs DOI PDF

    Museum artifacts are the main way for visitors to experience and learn about cultural heritage. Augmented reality (AR) allows for high interactivity and is increasingly applied in museums to improve tourists’ experience and learning. It also supports the extension of museum experience to outside of the physical museum space, contributing to the visiting trajectory and takeaway experience. In this paper, we present our design of two tangible AR interfaces for cultural artifacts: Postcard AR and CubeMuseum AR, followed by three user studies that evaluate and optimize the design. In Study 1, we conducted a within-subjects study (N \frac14 24) that compares the two AR interfaces with a baseline condition (Leaflet). Our results demonstrate the positive effects of tangible AR interfaces on users’ motivation and engagement in learning cultural heritage. In Study 2, we further explored how to optimize CubeMuseum AR by adopting a user-centered design approach. Through the analysis of expert interviews (N \frac14 7) and an online survey (N \frac14 207), the results specify a series of requirements and design guidelines for tangible AR interfaces to be used as a learning tool and a hybrid gift. Based on the findings, the design of the CubeMuseum AR was optimized and evaluated in Study 3. A between-subjects user study was conducted (N \frac14 32) to compare the optimized design with the initial design. The results verified the positive effects of gamified tangible AR interfaces on users’ motivation, engagement, and performance in learning cultural heritage. We present our design and evaluation results, and discuss the implications of designing tangible AR interfaces for cultural heritage learning and museum gifting.

17. Enabling Transcranial Electrical Stimulation via STI01: Experimental Simulations and Hardware Circuit Implementation

    Guanjie Xu, Gaomin Su, Hao Fang, and Yue Li^*

    In 2023 5th International Conference on Electronic Engineering and Informatics (EEI), Jun 2023

    Abs DOI PDF

    This paper presents the novel utilization of a chip, STI01, developed in China and traditionally used for muscle stimulation, to meet the demands of transcranial electrical stimulation (tES). We present the principles behind the design of the external circuit’s filter, along with the simulation results. Notably, we successfully accomplished tES utilizing the STI01 chip. This design could be enhanced further by enriching the MCU (STM32F030) program and introducing a closed-loop Brain-Computer Interface (BCI). Given the compact nature of the circuit design, it has potential for integration as a core circuit in portable wearable devices.

18. HeritageSite AR: An Exploration Game for Quality Education and Sustainable Cultural Heritage\ding81

    Ningning Xu, Jiachen Liang, Kexiang Shuai, Yuwen Li, and Jiaqi Yan

    In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems, Apr 2023

    Abs DOI PDF

    Cultural heritage (CH) plays an important role in realizing the Sustainable Development Goals (SDGs). In this paper, we focus on emerging technologies such as Augmented Reality (AR) and

19. Towards Cross-Reality Interaction and Collaboration: A Comparative Study of Object Selection and Manipulation in Reality and Virtuality

    Shuhao Zhang, Yue Li^*, Ka Lok Man, Yong Yue, and Jeremy Smith

    In 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), Mar 2023

    Abs DOI PDF

    Cross-Reality (CR) is an important topic for the research of multiuser collaborative systems. It allows users to participate in the realityvirtuality continuum and select appropriate interactive systems to work with, such as Virtual Reality Head-Mounted Displays (VR HMDs). However, there is limited work showing how interaction in VR differs from the more commonly used Personal Computers (PCs) and tablet devices in terms of object selection and manipulation. In this paper, we present a comparative study that investigated how users perform and perceive workload on 3D object selection and manipulation tasks using different devices (e.g. PC, tablet, and VR). We recorded the time and accuracy as objective task performance measures, and users’ self-reported workload as a subjective measure. Our results revealed that unlike the biased performances of PC and tablet, VR has a balanced performance and great potentials in complex tasks.

20. LeanOn: Simulating Balance Vehicle Locomotion in Virtual Reality

    Ziyue Zhao, Yue Li^*, and Hai-Ning Liang

    In 2023 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Oct 2023

    Abs DOI PDF

    Locomotion plays a critical role in user experience in Virtual Reality (VR). This work presents a novel locomotion device, LeanOn, which aims to enhance immersion and feedback experience in VR. Inspired by balance vehicles, LeanOn is a leaning-based locomotion device that allows users to control their location by tilting a board on two balance wheels, with rotation enabled by two buttons near users’ feet. To create a more realistic riding experience, LeanOn is equipped with a terrain vibration system that generates varying levels of vibration based on the roughness of the terrain. We conducted a within-subjects experiment (N=24) and compared the use of LeanOn and joystick steering in four aspects: cybersickness, spatial presence, feedback experience, and task performance. Participants used LeanOn with and without the vibration system to investigate the necessity of tactile feedback. The results showed that LeanOn significantly improved users’ feedback experience, including autotelic, expressivity, harmony, and immersion, and maintained similar levels of cybersickness and spatial presence, compared to joystick steering. Our work contributes to the field of VR locomotion by validating a leaning-based steering prototype and showing its positive effect on improving users’ feedback experience in VR. We also showed that tactile feedback in locomotion is necessary to further enhance immersion in VR.

21. TeleSteer: Combining Discrete and Continuous Locomotion Techniques in Virtual Reality

    Ziyue Zhao, Yue Li^*, Lingyun Yu, and Hai-Ning Lianq

    In 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), Mar 2023

    Abs DOI PDF

    Steering and teleporting are two common locomotion techniques in virtual reality (VR). Steering generates a great sense of spatial awareness and immersion but tends to lead to cybersickness; teleporting performs better in mitigating cybersickness but may lead to the loss of spatial awareness. Hence, we combined these two techniques and designed TeleSteer. This technique allows users to perform both steering and teleporting and customize the control. We discuss that a combined use of discrete (e.g. teleporting) and continuous (e.g. steering) locomotion techniques is necessary for scenarios that require both free explorations and close-range interaction tasks, making TeleSteer a suitable alternative.

2022

1. Cultural Heritage Assets Optimization Workflow for Interactive System Development

   Kedong Chai and Yue Li^*

   In 2022 IEEE 46th Annual Computers, Software, and Applications Conference (COMPSAC), Jun 2022

   Abs DOI PDF

   An increasing number of reconstructed digital assets are being created worldwide to preserve cultural heritage. These assets can be used in interactive systems such as augmented reality (AR) and virtual reality (VR) to provide effective ways to access and learn about cultural heritage. One of the widely adopted reconstruction techniques is close-range photogrammetry. However, scanned models need to be processed and optimized before they can be used in interactive systems, which requires a series of retopology and baking work to reduce the size of models while maintaining visual fidelity. Nevertheless, manual retopology and baking are complex processes. An efficient optimization workflow is essential for the use of cultural heritage assets in interactive systems. This paper presents an optimization workflow for retopology and texture baking using free and opensource software. Evaluations show that the workflow demonstrates its strengths in its high efficiency, versatility, learnability, and low cost. This work contributes insights to researchers and practitioners in the field of cultural heritage.

2. DARC: A Visual Analytics System for Multivariate Applicant Data Aggregation, Reasoning and Comparison

   Yihan Hou, Yu Liu, He Wang, Zhichao Zhang, Yue Li, Hai-Ning Liang, and Lingyun Yu

   Pacific Graphics Short Papers, Posters, and Work-in-Progress Papers, 2022

   Abs DOI PDF

   People often make decisions based on their comprehensive understanding of various materials, judgement of reasons, and comparison among choices. For instance, when hiring committees review multivariate applicant data, they need to consider and compare different aspects of the applicants’ materials. However, the amount and complexity of multivariate data increase the difficulty to analyze the data, extract the most salient information, and then rapidly form opinions based on the extracted information. Thus, a fast and comprehensive understanding of multivariate data sets is a pressing need in many fields, such as business and education. In this work, we had in-depth interviews with stakeholders and characterized user requirements involved in data-driven decision making in reviewing school applications. Based on these requirements, we propose DARC, a visual analytics system for facilitating decision making on multivariate applicant data. Through the system, users are supported to gain insights of the multivariate data, picture an overview of all data cases, and retrieve original data in a quick and intuitive manner. The effectiveness of DARC is validated through observational user evaluations and interviews.

3. A Framework for Sharing Cultural Heritage Objects in Hybrid Virtual and Augmented Reality Environments

   Yue Li^* and Eugene Ch’ng

   In Visual Heritage: Digital Approaches in Heritage Science, 2022

   Abs DOI PDF

   The emulation of social environments within which ideas, knowledge and interpretation are exchanged is a challenge for Extended Reality (XR) technologies. One aspect of the challenge is the concept of Extended Reality itself, and this within the broad spectrum of the physical and virtual reality continuum. As users settle down into the spectrum via their preferred devices, so must we investigate the viability of communication between users adopting different modes of XR. In this chapter, we discuss three attributes of virtual objects and explore the concept of a Hybrid Virtual and Augmented Reality (HVAR) environment. We look at how users from different realities could interact, engage and communicate in a shared space via objects. We believe that the use of HVAR environments is the way forward for connecting worlds, and that it will facilitate future communications around virtual objects, developing and flourishing across time, space and devices, much like how social media has facilitated user-generated contents, empowering individual interpretations and the formation of collective meanings. The concept of a hybrid space aims to gather communities from disparate backgrounds and cultures, and to facilitate discussions around objects of interest.

4. Student Engagement in Software Engineering Group Projects: An Action Research Study

   Yue Li^*, Soon Phei Tin, and Charlie Reis

   In Proceedings of the 2022 5th International Conference on Education Technology, Jun 2022

   DOI PDF
5. Student Engagement in Software Engineering Group Projects: An Action Research Study

   Yue Li^*, Soon Phei Tin, and Charlie Reis

   In Proceedings of the 2022 5th International Conference on Education Technology Management, Dec 2022

   Abs DOI PDF

   We present an action research study on student engagement in group work. The study was carried out within CPT202 Software Engineering Group Projects, a UK Level Two module with 370 students enrolled during the 2020-2021 academic year. The primary finding of our action research is that peer evaluation could encourage student engagement in group work. In addition, student engagement in group work positively correlates with their academic performances. We also discuss several effective strategies in supporting student engagement in the group work of software engineering projects. The results and findings have pedagogical implications in encouraging student engagement not only in software engineering group projects, but also in general activities that involve students working in a group.

6. Cross-Cultural Factors Influencing the Adoption of Virtual Reality for Practical Learning

   Diego Monteiro, Teng Ma, Yue Li, Zhigeng Pan, and Hai-Ning Liang

   Universal Access in the Information Society, Nov 2022

   Abs DOI PDF

   Education is one area that was significantly affected by the COVID-19 pandemic with much of the education being transferred online. Many subjects that require hands-on experimental experience suffer when taught online. Education is also one area that many believe can benefit from the advances in virtual reality (VR) technology, particularly for remote, online learning. Furthermore, because the technology shows overall good results with hands-on experiential learning education, one possible way to overcome online education barriers is with the use of VR applications. Given that VR has yet to make significant inroads in education, it is essential to understand what factors will influence this technology’s adoption and acceptance. In this work, we explore factors influencing the adoption of VR for hands-on practical learning around the world based on the Unified Theory of Acceptance and Use of Technology and three additional constructs. We also performed a cross-cultural analysis to examine the model fit for developed and developing countries and regions. Moreover, through open-ended questions, we gauge the overall feeling people in these countries have regarding VR for practical learning and how it compares with regular online learning.

7. Examining Cross-modal Correspondence between Ambient Color and Taste Perception in Virtual Reality

   Zhen Wu, Rongkai Shi, Ziming Li, Mengqi Jiang, Yue Li, Lingyun Yu, and Hai-Ning Liang

   Frontiers in Virtual Reality, Dec 2022

   Abs DOI PDF

   This research explores the cross-modal correspondence effect of ambient color on people’s taste perception in virtual reality (VR). To this end, we designed and conducted two experiments to investigate whether and how taste-congruent ambient colors in VR influence taste perception measured by four aspects: 1) taste ratings of a neutral drink; 2) taste association with virtual environments; 3) associated scenarios when immersed in these virtual environments; and 4) participants’ liking of these environments. In Experiment 1, participants adjusted the ambient light with different cross-modal-related colors in the immersive environments and reported their scaling of the Virtual Reality Sickness Questionnaire (VRSQ). Comfortable light intensity for each ambient color was obtained and color recognition problems were observed. In Experiment 2, participants tasted black tea (as the neutral drink), after being exposed to eight different virtual environments with different ambient colors. Results showed that the pink ambient color significantly increased the sweetness ratings. Differences in the color-taste association and environment liking were also observed in the ambient color conditions. Our results provide new insights into the cross-modal correspondence effect on ambient color and taste perception not found in prior work in VR scenarios.

8. ChemistryVR: Simulating Chemistry Experiments in Virtual Laboratories

   Xuansheng Xia, Guanxuan Jiang, Yue Li^*, and Hai-Ning Liang

   In 2022 IEEE International Conference on Virtual Reality and Visualization (ICVRV), 2022

   Abs PDF

   Experiment is an essential part of chemistry education. However, it is equipment- and space-demanding, and sometimes risky. Immersive and interactive simulations in Virtual Reality (VR) can address these issues. In this project, we developed a virtual laboratory based on China’s ninth grade chemistry textbook published by People’s Education Press. The system provides safety training and step-bystep tutorials so that students can learn from interactive simulations and observations of realistic experimental phenomena in a safe condition. Our system provides a risk-free approach and effectively supports practice-led and experiential learning of chemistry.

9. User Retention of Mobile Augmented Reality for Cultural Heritage Learning

   Ningning Xu, Yue Li^*, Jie Lin, Lingyun Yu, and Hai-Ning Liang

   In 2022 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), Oct 2022

   Abs DOI PDF

   Mobile Augmented Reality (AR) is becoming increasingly affordable and popular with the constantly improving computing power of mobile devices and the popularity of smartphones and tablets. In this paper, we present a user study that investigates user retention of mobile AR in cultural heritage learning. We developed a mobile AR application that allows users to observe 3D models of museum artifacts and learn about their culture and history. Participants achieved a knowledge retention rate of 78.21%, indicating the positive effects of mobile AR on cultural heritage learning. We performed a structural equation modeling analysis (N=50) to investigate the effects of usability, satisfaction, emotional attachment, focus of attention, and flow experience on user retention of mobile AR. The analysis results confirmed that user satisfaction and flow experience positively affect user retention. Usability and focus of attention contribute positively to user satisfaction and flow experience respectively.

10. Exploratory 3D Virtual Classrooms for Online Learning and Teaching

    Yuang Zhou, Jiarui Chen, Yinuo Wang, Yue Li^*, Cheng Zhang, Yong Yue, and Hai-Ning Liang

    In 2022 IEEE International Conference on Virtual Reality and Visualization (ICVRV), 2022

    PDF

2021

1. Effect of Visual Cues on Pointing Tasks in Co-located Augmented Reality Collaboration

   Lei Chen, Yilin Liu, Yue Li, Lingyun Yu, BoYu Gao, Maurizio Caon, Yong Yue, and Hai-Ning Liang

   In Symposium on Spatial User Interaction, Nov 2021

   DOI PDF
2. CubeMuseum: An Augmented Reality Prototype of Embodied Virtual Museum

   Yue Li^*, Lingyun Yu, and Hai-Ning Liang

   In 2021 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), Oct 2021

   Abs DOI PDF

   An Augmented Reality (AR) prototype, CubeMuseum, is proposed in this paper to present an embodied experience with virtual museum collections. With a cost-effective cube and a smartphone application, users can view and interact with 3D museum objects embodied on the cube. Detailed design of the prototype is presented to illustrate the approaches to visualize, present, and interact with virtual objects. CubeMuseum has been evaluated by hundreds of users in both laboratory studies and public exhibitions. The results indicated that the prototype is simple yet effective. It demonstrates several benefits and potential implications in supporting user engagement and learning experience. This research provides insights to researchers and practitioners in designing interactive cultural heritage experiences using a cost-effective approach.

3. Presence and Communication in Hybrid Virtual and Augmented Reality Environments

   Yue Li^*, Eugene Ch’ng, Sue Cobb, and Simon See

   PRESENCE: Virtual and Augmented Reality, Dec 2021

   Abs DOI PDF

   Abstract The use of virtual reality (VR) and augmented reality (AR) in connected environments is rarely explored but may become a necessary channel of communication in the future. Such environments would allow multiple users to interact, engage, and share multidimensional data across devices and between the spectrum of realities. However, communication between the two realities within a hybrid environment is barely understood. We carried out an experiment with 52 participants in 26 pairs, within two environments of 3D cultural artifacts: (1) a Hybrid VR and AR environment (HVAR) and (2) a Shared VR environment (SVR). We explored the differences in perceived spatial presence, copresence, and social presence between the environments and between users. We demonstrated that greater presence is perceived in SVR when compared with HVAR, and greater spatial presence is perceived for VR users. Social presence is perceived greater for AR users, possibly because they have line of sight of their partners within HVAR. We found positive correlations between shared activity time and perceived social presence. While acquainted pairs reported significantly greater presence than unacquainted pairs in SVR, there were no significant differences in perceived presence between them in HVAR.

2020

1. The Effects of VR Environments on the Acceptance, Experience, and Expectations of Cultural Heritage Learning

   Eugene Ch’ng, Yue Li, Shengdan Cai, and Fui-Theng Leow

   Journal on Computing and Cultural Heritage, Feb 2020

   Abs DOI PDF

   This article attempts to understand how present Virtual Reality (VR) environments can contribute to enhancing the communication of cultural heritage by providing an experience of the past that is acceptable for the younger generation and how museums and cultural institutions should adopt and use such technologies. Aspects of acceptance, experience, and expectation of VR with the underlying values are not well understood but are important for the sustainability of the communication of cultural heritage as a bequest to future generations. We conducted a combined quantitative–qualitative study on the participants who have various prior experience with gaming and VR, and different levels of knowledge on the history presented within the virtual environment. This study investigates how participants accept and are stimulated in terms of personal experience and their expectations and ideas for the future of museums if VR is used for enhancing the learning of cultural heritage. Prior gaming and VR experience were investigated to see whether they do indeed influence the preference for using VR for learning cultural heritage. We demonstrated that particular age groups and background are especially agreeable to virtual reality as environments for learning and experiencing cultural heritage, regardless of their knowledge of the historical context of the virtually reconstructed site. Our findings also revealed important behaviours in our demographics group with regards to user preferred length of time and the believability of the virtual environment and how it influences aspects of their experience such as the exploration of the heritage site, familiarity, and meaning making. The study has implications for the use of VR for enhancing the experience of cultural heritage in museums and cultural institutions.

2. Digital Campus with Virtual Reality: From Immersive Visualization to Interactive Experience

   Yue Li^*, Qilei Sun, Cheng Zhang, Eng Gee Lim, Hai-Ning Liang, and Yong Yue

   In CCSAT’20, 2020

   Abs PDF

   Constructing a digital campus with virtual reality (VR) allows users to experience an immersive campus life, access campus services remotely, and have social interaction with others. However, most existing digital campus systems only present users with immersive visualizations of the digital campus scenes only. We argue that digital campus with VR should move a step forward from the immersive visualization to supporting users’ interactive experience. Hence, we propose a framework of digital campus with VR and present three scenarios at Xi’an Jiaotong-Liverpool University, which include the building evacuation drills, the digital library system, and the VR experiential learning. Our research provides practical insights into the design and application of interactive experience of digital campus with VR.

2019

1. Appropriate Control Methods for Mobile Virtual Exhibitions

   Yue Li^*, Paul Tennent, and Sue Cobb

   In VR Technologies in Cultural Heritage, 2019

   Abs DOI PDF

   It is becoming popular to render art exhibitions in Virtual Reality (VR). Many of these are used to deliver at-home experiences on peoples’ own mobile devices, however, control options on mobile VR systems are necessarily less flexible than those of situated VR fixtures. In this paper, we present a study that explores aspects of control in such VR exhibitions - specifically comparing ‘on rails’ movement with ‘free’ movement. We also expand the concept of museum audio guides to better suit the VR medium, exploring the possibility of embodied characterguides. We compare these controllable guides with a more traditional audio-guide. The study uses interviews to explore users’ experience qualitatively, as well as questionnaires addressing both user experience and simulator sickness. The results suggest that users generally prefer to have control over both their movement and the guide, however, if relinquishing movement control, they prefer the uncontrolled guide. The paper presents three key findings: (1) users prefer to be able to directly control their movement; (2) this does not make a notable difference to simulator sickness; (3) embodied guides are potentially a good way to deliver additional information in VR exhibition settings.

2. Enhancing VR Experiential Learning through the Design of Embodied Interaction in a Shared Virtual Environment

   Yue Li^*, Teng Ma, and Eugene Ch’ng

   2019

   Abs PDF

   Virtual Reality (VR) has great potentials for experiential learning, especially in a shared environment with multiple users. However, the factors influencing such experience are not well understood. This research considers one specific feature of VR, the embodiment richness. We consider embodiment richness a powerful feature because VR supports various media and the immersion with the entire physical body, which could facilitate the design of dynamic interactions, embodying both physical information and social activities. We propose a conceptual framework for its influences on VR experiential learning through engagement and communication. Aside from retrospective questionnaires, our methods also incorporate a physiological measure with brain sensing technology, reflecting the cognitive process. This will be a contribution of our work. This research will also contribute to the theoretical framework to understand the effect of embodiment in VR experiential learning, informing the future design and application of VR for experiential learning in practice.

2018

1. A Comparison of the Capacities of VR and 360-Degree Video for Coordinating Memory in the Experience of Cultural Heritage

   Shengdan Cai, Eugene Ch’ng, and Yue Li

   In 2018 3rd Digital Heritage International Congress (DigitalHERITAGE) Held Jointly with 2018 24th International Conference on Virtual Systems & Multimedia (VSMM 2018), Oct 2018

   Abs DOI PDF

   Virtual Reality (VR), a medium which can create alternate or representations of reality, could potentially be used for triggering memory recollections by connecting users with their past. Comparing to commonly-used media within museum such as photos and videos, VR is distinct because of its ability to move beyond the confines of time and space, by enabling users to be immersed in the reconstructed context and allowing them to take charge of the environment by interacting with objects, navigating the environment, and evolving the narratives. In this paper, we compared audience experiences of cultural heritage (CH) between 360-degree video recordings and Virtual Environments to investigate the capacity of these two types of media for coordinating the audience’s memory of the past. The findings will help guide the future design and evaluation of VR as a medium for communicating CH.

2. Multiuser Interaction with Hybrid VR and AR for Cultural Heritage Objects

   Yue Li^*, Eugene Ch’ng, Shengdan Cai, and Simon See

   In 2018 3rd Digital Heritage International Congress (DigitalHERITAGE) Held Jointly with 2018 24th International Conference on Virtual Systems & Multimedia (VSMM 2018), Oct 2018

   Abs DOI PDF

   This research investigates the factors and ways in which users initiate conversations and engage in interactions in a hybrid virtual environment using a combination of Virtual Reality (VR) and Augmented Reality (AR) devices. The research was done in the ‘spirit of the ancient Silk Road’ where trade brought in exchange of ideas, cultural influence and cross-border communications. The notion of a 21st century Silk Road is necessarily digital, over the Internet and based around 3D cultural heritage objects. Digi-Capital’s Report forecasts the revenue of AR and VR to be US$150b by 2020. We projected that VR and AR will become pervasive, much like the Social Web and the universal ubiquity of mobile devices such as smartphones and wearables. Here, we conducted a user study exploring users’ acceptance of the use of hybrid VR and AR for cultural heritage, and investigated the social nature of multiple co-located user interaction. We adapted the UTAUT questionnaire in our experiment and found that social influence has positive effects on performance expectancy and effort expectancy, which generate positive effects on user behavioural intention. This study pioneers the future design and use of hybrid VR and AR technology in cultural heritage specifically, and in other application areas generally by highlighting the significant role that social influence plays in enhancing users’ behavioural intention facilitated by different immersive devices.