Augmented Reality Movies

We want to design and implement a new kind of cinematographic experience.

Classic Cinema proposes a linear storyline (even in those cases in which directors and screenwriters experiment, as in movies such as Memento or Inception, the experience offered by Cinema is a linear one) which is the product of a single point of view: the movie camera is a single eye onto a time/space using characters, locations and events to propose a single perspective.

The idea of an Augmented Reality movie proposes a model which is radically different from this.

In an AR movie the storyline unfolds in non-linear ways, across a series of different points of view, in a time dis-continuum which can include synchronicity, atemporality, and simultaneity. 

Many examples of AR Movie have already been tried out and produced.


In this example above, an AR Movie experience has been designed for the Netherlands Film Festival 2011 held in Utrecht.

From the project’s website:

“Storylines” turns a citycentre into a 3D surround film-set of a ‘movie’ happening in augmented reality. Using the smartphone app “Layar” a world of scenes and storylines can be explored. Dialogues are still hanging in the air, as the geo-located textual remains of movie scenes which occured throughout the city.

This is an interesting concept: several paths are designed throughout the city and along them, a series of “speech bubbles” are disseminated to hold the dialogues of the movie narrative.

In this way not only there are multiple storylines disseminated throughout the city, ready to be “walked” and experienced, but the interesting side effect of turning passer-bys into unsuspecting actors (whenever we frame them in our mobile phone with the bubbles on their head) is also achieved.

This interesting concept has:

  • some positive aspects
    • stories are easy to follow
    • people can develop an interaction with the story (by “wearing” the bubbles)
    • the product stimulates the production of further content (“take a picture of us with the speech bubbles on our head!”)
    • the dissemination patterns of the parts of the story can be used as a tool to let people traverse the city to go from one place to another (for example from one event venue to the other) transforming city traversal into a narrative experience
  • some limitations
    • stories are linear
    • stories are predefined, as well as the paths along which they are disseminated
    • speech bubbles float in mid-air by themselves, even if no-one is there, partially ruining the experience

As part of a relaunch for Germany’s Universal 13, a revolutionary new film, The Witness, has been created in which you play an interactive, traveling roll. Using augmented reality on an iPhone, participants begin their adventure by witnessing a kidnapping by the Russian mafia. It’s then their job to track down the victim, finding clues, communicating with other participants and viewing other scenes on location around the city of Berlin.

This experiment in AR Movies is a bit more advanced over the previous one.

Parts of a movie are disseminated throughout the city of Berlin and people joining in to the adventure are able to traverse the city and experience them, also being able to interact with the storyline, turning it into a game much like the old Dragon’s Lair videogame, in which user interactions basically were limited to the choice of direction at certain hotspots of the timeline.

This idea is closer to the objective to which we st forth, but still offers what is substantially a single point of view.

Man pros and cons:

  • pros
    • the product offers a balanced mix between a movie experience and an adventure game, in which the locations of the parts of the narrative are used in a cinematographic setting providing good quality and narrative experiences which people expect from movie theaters
    • the visual helps which are disseminated throughout the experience implement in novel ways what we will call “Augmented Context” in the next parts of the workshop, providing tools which help users navigate physical space in search of the next part of the experience
  • cons
    • the product offers what is a substantially linear narrative (or multi-linear, if we take into account the possibilities for choice)
    • the storyline is fixed and there is no space for user contributions
    • the product suffers from its “marketing” approach: it bears great quality in the content but the structure of the product itself is basic, and resembles the standard treasure-hunt scheme, but without the freedom of action which characterizes that kind of game
Even in its simplicity, we can gather useful insights from this product, specifically in the idea of providing alternative versions of “reality” for physical spaces. The possibility to overlay the physical world with digital narratives provides an essential tool that we will use in the next step of our design and production process.

This promo video of Augmented Reality Cinema does not correspond to any “finished” product/service, and is basically done using post production. It is a design narrative of an implementation scenario. Nonetheless it is worthwhile examining it, as it provides interesting design and implementation scenarios on which to reflect upon.

The first is the idea to disseminate parts of a movie experience around urban spaces.

While the images shown in the video describe levels of interactivity and interface responsiveness which will not be achieved by current mobile devices (for example the movie frame which users seem to be able to pan/zoom through by simply moving their phone while maintaining visual coherence with the surrounding environment: it will simply *not* be done) the technique used in the demo appears to be rather interesting, for the reasons we already described. Apart from being able to see famous actors superimposed on the landscape, this possibility allows to imagine and develop interesting concepts which are valuable for multiple scenarios: imagine an AR documentary onto the location you are looking at, explaining the historic and other values of the place you are traversing; or even the idea of AR Guides which directly assist you with reference to the place you are pointing your device to.

At around 50 seconds into the video, the actors are seen to integrate with the storyline: by pointing the device onto the specific location, an actor “enters the scene”. This possibility allows for a series of opportunities, from the silliest ones to the most serious ones. For example simulations could be organized, transforming this functionality in a testbed for architectures that are yet-to-be-built, for disaster scenarios, for scenographies. Or it could be used to research the history of places, overlaying the current landscape with the buildings and life activity of several thousands years earlier. Or it could be used to see what it would be like if the Colosseum in Rome was filled with 10 million red balloons, and to walk through it, and to take a picture of it.

At 1:30 of the video, the bridge in front of the Big Ben in London is seen replaced with a movie scene which describes a disaster scenario. The cognitive effects of being able to experience an alternative reality for the place we are currently present in, are great. Fundamentally, a possibilistic approach generates, in which additional degrees in perceptive freedoms are felt, and the possibility to “see”  an alternative to the physical world which we are immersed in collaborates in dismantling the inhibitions by which we seldom feel able to change reality around us. We, thus, feel a bit more empowered to change things, as we can see there, in out hands, in front of us, the space we are walking through changed, thus confirming in a visual way the possibility and feasibility of such an opportunity.

While multiple other examples exist about the possibility to create Augmented Reality movies we will now focus on the design of a product which will comply with the strategies which we described at the beginning of this section.

The design of an AR Movie will be substantially different from the one of a “classical”movie.

Some roles which are commonly found in the production of a movie are (an informal, unordered, incomplete list):

  • concept designer
  • screenplay writer
  • director
  • actors
  • photography
  • operators
  • post-production operators
Even in this incomplete list, it is immediately evident how each one of these role changes in the production of an AR Movie.
The role of the Concept Designer, for example, is subject to a radical change: from a linear incipit for a storyline and its evolution, an environment needs to be designed, in which actors, sequences and story nodes are transformed into qualitative descriptions of the characteristics of an environment, on the ways in which a certain subject or issue can be tied to the parts of a physical or architectural space, and into the degrees of freedom which are allowed to the people which will traverse this space, to take part in the story.
A similar destiny is  lived by the screenplay writer, who will not design the sequence of events in the storyline, but the way in which the dynamic of the story will unfold across space and time, and without focusing on the single characters (as the characters are “undetermined”, since they will be impersonated by passer-bys and by digital information), but on the experience which they will live, their possibility for interaction with it and the types  of transformation that the “story” will present as a result of such interactions.
The director will substantially transform into an interaction designer, designing what in the end is an interface onto a physical space/time, and the process behind it, the experience which people will live while traversing the space/time of the AR Movie.
Actors will multiply, in both type and quantity. Actors will be regular actors, as they might be used to create the multimedia content which will be at the base of the parts of the movie. But they will also be the people traversing the space of the Augmented Reality Movie, as they will have direct impact on the story: wether by traversing space/time in their own unique way, thus mutating the experience and producing each time one of possibly infinite numbers of combinations of the parts of the movie;  or by interacting with the movie, expressing their points of view through the mechanism of commenting or rating; or even by mutating the content of the movie itself, should they decide to add a scene or sound or other intervention using their device.
Thus actors become viewers, directors, video operators, post producers, audience, performers or producers according to what they do, to how they perform in the space/time of the AR Movie.
These kinds of considerations might be applied to the whole production line of an AR Movie. And there also are some completely new roles which are absent in the production of “classic” movies:
  • cross-mediality: an AR Movie will be experienced by using different devices and technologies and, thus the experiences will have to be designed and implemented in ways which will guarantee coherent characteristics or, at least, in ways which are optimized for the specific device, and specific roles, skills and competencies will be required for this;
  • interactivity: the AR Movie will not be a “passive” experience, and it will require the active contribution of the users, becoming an interface to the dynamic storyline;
  • accessibility/usability: this point is directly connected to the previous one, and it deals with the need to plan and verify the effective possibility to use the product by the target which is imagined to be experiencing the movie.
On top of that, the AR Movie will need to address the strategies which have been briefly introduced in the previous sections:
  • non-linearity
  • the creation of an emergent, environment-based narrative
  • the ideas of simultaneity, synchronicity, atemporality, realtime
  • the possibility to host multiple points of view, even dissonant ones
  • the possibility to integrate into the movie data and interaction
All this will enable us to create a product which will implement a series of objectives:
  • the creation of a story which “you can walk through”
  • the creation of an environment for expression, in which multiple voices can freely and autonomously express their points of view, across cultures, values, contexts, backgrounds
  • the creation of a framework which is able to enable an ecosystem for auto-representation (natural, cultural, social, political, aesthetic…)
  • the creation of a cognitive landscape, defined as the sum of the information produced and experienced by all the actors involved in a certain environment, be them human, informational, sensorial, synthetic, generative.
We will proceed in our design process by describing and creating the two main parts of the AR Movie: