Bachelor's project

Overview

In my Bachelor's project, I developed a prototype for an immersive authoring tool that leverages generative AI to streamline the creation of mixed reality experiences. The tool reimagines how designers can prototype in MR environments by enabling natural interactions through speech, gestures, and gaze rather than traditional programming interfaces. The project explored the intersection of GenAI and MR technologies while addressing current limitations in XR prototyping.

“Towards a Generative Artificial Intelligence-Powered In-Situ Authoring Tool for Immersive Experiences” was conceived as part of the Bachelor's programme in Digital Design at Aarhus University and assessed with a final grade of 12.

The problem

Current tools for prototyping MR experiences require designers to be well-versed in multiple software environments and technical skills like programming and 3D modeling. The process involves constant switching between conventional computers and head-mounted displays, creating friction in the design workflow. This limits rapid prototyping and iteration capabilities.

How can generative artificial intelligence optimize the user flow of creating and manipulating original content for authoring mixed reality experiences?

The approach

Combined theoretical research with practical exploration to envision a natural, in-situ authoring environment.

To build a theoretical foundation of the subject at hand, I explored three key areas:

Mixed Reality

According to Paul Milgrams Reality-Virtuality Continuum, MR encompasses environments where real and virtual objects are presented together. Modern MR systems support multiple interaction techniques including gesture-based interaction, speech input, and eye tracking creating the foundation for natural user interfaces that require minimal training.

Generative AI

GenAI models generate synthetic data resembling real-world data, with outputs spanning different modalities, including text, images, 3D models, audio, and more. Public models like GPT, Stable Diffusion, and Midjourney rely on natural language prompts, making them highly usable but sometimes requiring specialized prompt engineering to achieve desired results.

Prototyping for MR

Creating MR applications involves generating graphics/audio content and defining object behaviors and interactions. Current XR prototyping tools often fail to adequately realize design ideas, with researchers noting that designs appropriate in conventional settings may be difficult to translate to immersive environments.

For this project, I produced a scenario in which a designer assisted by GenAI aimed to construct an educative interactive immersive experience more specifically, an environment that could educate users about the tyrannosaurus rex. To do so, several steps are needed, including generating assets, describing the object behavior and interactivity using only GenAI platforms, tools, models, etc., and importing them into an immersive environment.

To better understand the current state of the art of GenAI-assisted MR prototyping, I conducted a study of the process of generating, importing, and manipulating both 3D models and 2D UI components. The aim was in part to gain a deeper understanding of different types of GenAI models through an evaluation of their respective capabilities, limitations, and use cases, and in part to map out the different steps involved in carrying out such a task.

Though advanced models like DreamFusion and Magic3D showed promise, I tested available software solutions instead due to project constraints, including scope and technical limitations.

Most text-to-3D services had critical limitations. Luma Labs Genie 1.0 performed best, generating four T-rex variations quickly, though refinement for higher quality required significant processing time.

I also tested an alternative: using Dall-E to create reference images, then converting them with Cube by Common Sense Machines. This allowed better control of specific details but produced models with skewed proportions and took hours compared to direct text-to-3D generation.

For animation, I tested Anything World s platform with a 3D model of a quadruped animal. While it succeeded to identify body parts and apply rig joints, many of those required manual adjustment and the final animations were often miscalculated.

I finally uploaded assets to ShapesXR for assessment in an immersive environment but found no way to use GenAI within the MR environment to describe object interactivity, a critical limitation for my workflow.

Findings

Current GenAI models for 3D content suffer from slow processing, inconsistent outputs, and fragmented workflows that require constant platform switching, with no way to describe interactivity within the immersive environment.

In design research, Fallman (2008) categorizes knowledge-gathering activities as design practice, design research, and design exploration. Design exploration focuses on asking "What if?" without concern for market fit or user needs, instead showing what is possible, what would be desirable or ideal.

Based on my findings, I created a design exploration of a future scenario where GenAI seamlessly integrates into immersive in-situ prototyping. I visualized this through a storyboard and flowchart that show the user experience and workflow. Storyboards effectively support design exploration by helping generate what if scenarios and communicating ideas visually.

My system combines multiple input modalities gaze, speech, and gestures with natural language processing to create a natural user interface. Speech enables hands-free operation and natural communication with an AI agent, similar to Bolt s Put-That-There system but enhanced by modern GenAI capabilities. For 3D manipulation, I primarily use freehand gestures, addressing haptic feedback limitations by allowing users to draw on physical surfaces what Henderson & Feiner call opportunistic controls.

This approach eliminates programming needs through voice commands and text prompts. In this no-code future, rapid prototyping extends beyond lo-fi sketching, potentially shortening development cycles with higher-fidelity prototypes created using fewer resources.