Immersive Rail.

Researcher and creative writer - Dr Zelda Hannay of the University of Sheffield Faculty of Arts and Humanities wrote a compelling, interactive, mystery/detective story based on the National Railway Museum’s archive.

Our intention was to use the archives as a foundation for the story and so Dr Zelda Hannay was supported by two History MA students, from the University of Sheffield, to help with research into the archives.

Using the story written by Dr Zelda Hannay, we drew storyboards to start to gain a clearer vision for how the story could be realised through an app.

We were grateful that someone had already created a highly accurate model of a British Railways Mark 1 railway carriage. This allowed us to create an accurate and convincing stage for our Mid 20th Century setting.

An Illustrator was commissioned for loading and intermediate screens taking place outside of the train carriage. This meant that we did not have to create 3D models for the whole world within the game and therefore saved us a significant amount of time.

Initially, our aim was to create characters in Character Creator and export them to Unity. However, sourcing or designing period clothing was challenging using Character Creator’s library alone.

Therefore we developed a pipeline from DAZ 3D (where there were more clothing options) to Character Creator to Unity, which in theory should have worked.

In reality, differing approaches to how clothing moves and interacts with the characters underneath caused significant issues. Furthermore, the number of polygons used in the export process caused problems in development, requiring a good deal of trial and error to reduce.

Throughout the process of trying to source library material, we ran into problems with the commonly commercially available models being focused on young, unfeasibly attractive people with rather exaggerated physical attributes, making the creation of, for instance, a nun in her 70s extremely challenging.

We were able to source an actor for the main character, but we relied on staff members for the other voices. As we did not have access to a recording studio, we recorded in a number of different places with different equipment.

The NRM had access to an extensive archive of period sound recordings and supported us with an archivist, who searched through to find the relevant audio required and obtained permissions for us.

These sound recordings allowed us to successfully create an immersive soundscape, even with limited time and sound editing experience. We used Adobe Audition as our audio workstation and used the asset "LipSync Pro" to sync the lips of the characters to the words.

Initially, we spoke to the Science Museum to discuss the possibility of scanning artefacts in order to create the 3D models within the story. This proved to be a worthwhile conversation but the outcome was that because of software costs and the constraints around moving artefacts, this was no longer a viable option.

Instead, the game and artefacts were all programmed in Unity. To build the game, we began by using a system called "Animation Controllers". During implementation, a previously unknown system called "Timelines" was discovered and development switched to this.

We created and manipulated the Timeline and the underlying code to create custom objects. This included a custom Timeline object upon which the script could be placed. This made it easier to synchronise the narration visuals with the narration audio, by just dragging it around the timeline.

Development was an iterative process and although we had to change our methods of working, we did learn a tremendous amount about creating an Augmented Reality Experience.

We used the following tools throughout this project:

• Character Creator and DAZ 3D- Build 3D characters
• Unity - 3D Modelling of Interactions
• Adobe Audition - Audio Workstation
• LipSync Pro - Audio Syncing
• Timelines - Building the game

We successfully deployed a prototype that gives a flavour of our goals and a brief glimpse into the story we have worked to develop, though it is and should be considered a proof of concept.

It should be noted that due to the Covid-19 Government advised lockdown, our plans to visit the NRM and user test the game with visitors at the site was not possible. It is also worth pointing out here that although we were unable to visit the NRM for user testing, everyone we worked with from the NRM were extremely useful throughout this project. They helped significantly, by providing us with a vast amount of information and resources, which enabled us to accurately depict the Mid 20th Century throughout the interactive story.

In addition to being supported by XR stories, we are grateful to have met more people in the XR field by attending several Immerse Sheffield and Sheffield Digital events. Thanks to connections via the University of Sheffield Faculty of Arts and Humanities, the project team visited the Confetti Institute of Creative Technologies which was also a useful learning experience.

During the course of this project, we have expanded our knowledge, skills, and expertise in the AR, modelling, and game development sphere. We have learned a great deal about the Archives and processes at the National Rail Museum and have enjoyed collaborating with the University to bring the concept of “Archive to AR” closer to fruition.

You can read more about the project here: Immersive Rail XR Stories.