Behind the Scenes

AR app for the ZDF series 'Der Bergdoktor'

About

It’s a prototype augmented reality app that was developed in cooperation with the IRT (Institut für Rundfunktechnik) and the German TV station ZDF. The idea is to look behind the scenes of the ZDF series ‚Der Bergdoktor‘: as a visitor to the film location, you can use your smartphone/tablet to scan the farm shop or a seating area that serve as image targets and then be surrounded by the film crew as 3D objects. Also, the app provides 360° recordings, 2D video clips and sound recordings recorded live on site. Of course, the AR content can also be viewed at home, but then it is less immersive.

Previous image

Next image

Project Details

Period
Team Size
Platform
Language
Game Engine
AR Framework
Video-Editor

07/2019 -08/2019
4
Android und iOS
C#
Unity 3D (2018.4.6f1)
Vuforia (8.3.8)
DaVinci Resolve (16)

Project Description

The app was created in the university course Development of media systems: AR beyond the hype – augmented reality in TV (practical training) at LMU Munich. The aim was to develop future-oriented solutions for the use of augmented reality on television and to implement the ideas as prototypes.

During the summer semester we held several presentations and worked on different documents like an exposé and proposal. We also did a lot of planning and communicated with our project partners. The app was created in a sprint week in the semester break.

One day on set

Our team was invited to be on the film set (in Austria) for a day and collect video footage from the shooting. While two of our team took 2D shots, another and I filmed with 360° cameras provided to us by ZDF. Most of the time, the two 360° cameras just stood around on their tripods, but when the film crew was shooting indoors, we changed rooms more often to always be live on the spot.

In our app, we then tried to exactly reproduce the positions of the 3D models of the film crew, actors and vehicles (a car and a camper).

On the set — We chose the seating area as an image target

Previous image

Next image

Features

On site on the film set, the farm shop and a seating area can be scanned with a smartphone/tablet camera to access the respective AR scenes. Alternatively, an image of the image targets can be scanned at home. You can then see what it looked like on set.

2 different AR scenes
3D models of the film crew, actors and vehicles
2D videos
360° videos
Sound recordings

My Contribution

I was primarily responsible for communicating with people from the television station ZDF, whom we first had to win over as partners. Then I negotiated that we could be on the film set with two ZDF employees and be allowed to film with their 360° and other 2D cameras.
I made sure that we received the video material in a timely manner and in a compressed form.
During the sprint week, I selected suitable videos for our app from the 360° and 2D video material from our shooting day, cut them into smaller video clips using DaVinci Resolve and integrated them into the app.
Since I noticed when watching the videos that the noises on the film set and the instructions of the film crew can also be heard clearly, I created sound files and integrated them into the app.
I placed animated buttons over certain 3D objects that can be tapped to play the video and sound material.

Screenshot Unity — Scene 1: positions of the vehicles and film crew

Previous image

Next image

What went wrong...

The videos from the set, especially the 360° videos, had an enormous memory size, which is why we could only use them in a very compressed form. As a result, the 360° recordings in the app are very blurry. Despite the strong compression, there are enormous stability problems while playing the 360° videos on the smartphone.
Also, while using the app, it often happened that 3D objects briefly disappeared and reappeared or even slowly slipped out of view.

-> But since we only had 5 days of the sprint week to implement the project, we couldn’t find a way to improve the stability issues.