How ‘And here happened a murder’ gave a special touch to storytelling

The story

The person responsible for this project at Yle was Wesa Aapro. He is a creative technologist and has done several cool projects before. Wesa knew from the beginning that he wanted to make a documentary in an interactive way for this project. But the big question revolved around what kind of story Yle would bring to life, that’s where creative partner and screenwriter Laura Majava pulled her weight. For an earlier project that didn’t take place, they had the idea to set up the lines of a body of a murder victim in a public space, and let people discover what happened, using their phone and augmented reality that enabled them to find clues and evidence. This concept got stuck into their heads, so therefore the basis of this idea was used as the starting point for the interactive documentary.

They decided they wanted to bring a murder story, through the point of view of the murderer. The idea that you’re able to play a game, receive information about a sinister event and get the sense that you are in the same room as the murderer was really appealing to them. At first, they had contact with someone that actually committed a homicide, but unfortunately they lost track of this person. They were forced to find another story to tell, and stumbled upon a civil war related crime that happened a hundred years ago in Finland. It was only a matter of finding out more about this event, and the storyline of ‘And here happened a murder’ was born.

The technology

The project was of course more than the story alone. Yle focused on incorporating the newest technologies into the game as well. The particular technology they used is called volumetric video. This makes it possible to use real people and add them with all their details into the game world, in a three dimensional way. The result is an avatar that you can barely separate from its real world twin. The whole idea of volumetric video is comparable with 3D animations, but without the idea of skeletons. They worked together with 4DR Studios from Eindhoven to capture all the movements from the game avatars and make them come alive using volumetric video. Ruud op den Kelder, independent AR developer, was responsible for this. Although technologies such as volumetric video aren’t that new for people like Ruud anymore, he still saw a challenge in this project and he was really excited to be a part of it. According to Ruud, the biggest challenge was setting the stage and mood in a way that matched the scene of the Nordic-Noir. He really wanted to make sure that people got drawn into the story by setting the mood right.

As a consequence of the volumetric capturing, there was too much large data and it became  impossible to make an AR based game out of it. Even the latest iPhone couldn’t handle the data of the volumetric video. Because of this, they ended up making a computer game that can be played on both computer and game pad.

The experiment

But why exactly would they want to use volumetric video if this complicated the whole game set-up? First of all, it’s about getting familiar with a new technology and learning new things. A real innovator never stops challenging himself. Besides that, Wesa from Yle was over the moon about the quality and experience you get with volumetric video. For him, volumetric video was able to capture the stillness of characters and make them look alive at the same time. This makes it the ideal technology for telling documentaries, since characters are typically depicted in a rather still way in these kinds of stories.

The degree of aliveness present in the volumetric video setup and the reception of it was tested by means of an experiment, conducted by Chronosphere. This is an R&D program by ‘Effenaar Smart Venues’ and 4DR Studios. They focus on projects using virtual humans and virtual video, such as the murder story from Yle. They saw this as a very interesting opportunity to explore through an experiment how volumetric video and immersive tech can contribute to the world of the famous Nordic-Noir.

The experiment tested the difference between a normal game and a game using volumetric video. Half of the participants played the volumetric video version, and the results differed greatly between both groups. There is indeed a clear difference in terms of the quality of the experience, which was way better in the volumetric video condition. Their experience was more lifelike, they felt more involved and they had the feeling they were really there. But also other topics were impacted by the volumetric video. For example, there were fewer complaints about feeling dizzy, nauseous or having a headache. Thanks to this experiment, it seems like we can conclude that there is a lot of positive impact of volumetric video.

Although the outcome of the interactive documentary looks very cool, and the use of volumetric video betters the experience of the audience, not every step in the process ran very smoothly. The whole making-off actually took place during the covid pandemic. Imagine having to design a game, collaborating with a partner that lives almost 2000 kilometres away from you…

The future

The future of telling documentaries using volumetric video looks bright. Now the technology is figured out, it will be easier to start new projects in the future. Yle definitely wants to keep using this format to tell more linear documentaries. They would also like to give workshops to tell other media companies about their experiences with the format and the technology. We are definitely looking forward to new gaming documentaries with interesting storylines and an interactive and immersive experience.

This project was produced by Wesa Aapro, Tomi Susanov, Laura Majava and Ruud op den Kelder

Article written by Muze Parmentier