Virtual reality and digital storytelling

 

 

     Storytelling is the most important part of every movie. There is no movie without a story. Stories are everywhere. People are always telling stories. Even a simple one like what somebody did yesterday: "Yesterday we went to the park and she said…". This is the beginning of the story. It might be the beginning of the book. Or a movie. One of the most famous movies starts a similar way:

 

… "My mom always said life was like a box of chocolates. You never know what you're gonna get."[1]

 

And every story has a beginning, middle part, and end. Something started, something happened, something is finished. This is the definition Aristotle described in "Poetics":

 

"Now, according to our definition Tragedy is an imitation of an action

that is complete, and whole, and of a certain magnitude; for there

may be a whole that is wanting in magnitude. A whole is that which

has a beginning, a middle, and an end."[2]

 

The topic of this research is narrative in the virtual reality /VR/ environment.  Ways of interacting and ways of telling a story. This is a huge field for research so I will keep the focus on some of the aspects of VR which are already in use or will be soon. The impact of virtual reality /VR/ to VFX industry, new possibilities of VR as a "new media". The ways to deliver the story through the VR and the impact of VR media to the audience. This are the goals of this research.

 

What virtual reality is?!

 

Virtual reality is "the computer-generated simulation of a three-dimensional image or environment that can be interacted with in a seemingly real or physical way by a person using special electronic equipment, such as a helmet with a screen inside or gloves fitted with sensors". [3]

In "The VR Book"[4], Jason Jerald describes VR as "a computer-generated digital environment that can be experienced and interacted with as if that environment were real". According to Jerald, "VR is about psychologically being in a place different than where one is physically located, where that place may be a replica of the real world or maybe an imaginary world that does not exist and could never exist." [4]. 

 

Digital environment can be live-action 360-degree footage or animated CGI or a combination of the two. For just a couple of years there is a huge integration of VR almost in every aspect of social live – business, healthcare, architectural visualisation, art, entertainment, etc. This topic is focused on two main aspects of VR – 360-degree live action films and tree dimensional, fully CG environment. It is not going to focus on specifics or differences between “two types” of VR. This topic is about ways of storytelling in both of them methods and technics of VFX like main tool of creating VR, the new methods to create photorealism but now in 360-degree environment. And also, and this is important – in the context of this article - virtual reality IS environment already created or defined by author or designer or creator. It IS existing VR with some basic parameters of the environment. It IS a world partially or fully created by a director, producer, storyteller, designer, VFX artists, etc. For instance -  the world of Star Wars, or Avatar, but with the freedom of /user/ viewer to observe, make changes and decisions. It IS NOT hypothetical VR. VR like "tabula rasa", which the user/viewer/ must define from scratch. 

 

                                                            Theatre

 

Theatres and religious rituals are the first manifestations of creating the fictional realities. Theatre /as we know it/ was born in ancient Greek. During the “Dionysus days” the followers of Dionysus, the god of wine, have been drinking, dancing, screaming and singing (chorus) in a spectacular, orgy type masquerade. An magnificent performance of the joy of life and freedom. And this is the reason the Greek theatre to be born. In 6th century BC, during the “Days of Dionysus”, Thespis /Dionysus priest/ began to interact with Chorus in the improvised dialogue. And because of this act, he becomes the first “official” actor in history. The dialog became the base of ancient Greek theatre. First performances in Greek theatre are mostly dialogs between actors, and singing of chorus, without any other action or without even movement. How does the ancient Greek celebration and theatre are involved with virtual reality (VR)?!

During the rituals and celebrations, the ancient people did create specific performances, mixing fiction and a reality beyond the daily basis life experience. Representing gods and fauns, satyrs, etc. in their dances and theatre performances, actors and audiences diving themselves into an /I will call it/ "existing un-existing world" of another reality. In ancient Greeks theatre for the first time we talk about stage, actors, and audience. Living with the passion of the performance on the stage, laughing or suffering with the narrative and the conflicts and situations on the stage, listening to the singing of chorus - like the voices of gods, is like a healing process for minds and souls of the audience. Aristotle called this "catharsis". The story, performed by actors on the stage in realistic /for this ages/ way can "touch" the viewer`s emotions. Make audience to suffering or laughing.

 

Lost in reality

 

If we think about VR as “psychologically being in a place different than where one is physically located”, then we can consider the 360 panoramic paintings /really popular 19th century/ as kind of visual virtual reality. Being in “rotunda” – the place /most often a round building or a hall/where the panoramic paintings have been exhibited, the viewers has an illusion of “realism” and get lost. From a moment on people starts feeling like they are in the place painted on the panoramic pictures surrounding them.

 

      

                                                            Another reality

 

 

In 1930 the science fiction writer Stanley G. Weinbaum write "Pygmalion's Spectacles", which is a pretty accurate description of a virtual reality:

 

"But listen—a movie that gives one sight and sound. Suppose now I add taste, smell, even touch, if your interest is taken by the story. Suppose I make it so that you are in the story, you speak to the shadows, and the shadows reply, and instead of being on a screen, the story is all about you, and you are in it. Would that be to make real a dream?"[5]

 

In the 1950`s – Morton Heilig present “Sensorama”. “Sensorama” is a complicated machine, which is built to simulate all the senses. There was not only sight and sound but a smell generator, stereoscopic display, and vibrating chair.

 

Morton Heilig's next invention in 1960 is the "Telesphere Mask" and this is the first head-mounted display providing stereoscopic 3D and also stereo sound. 

 

In 1965 Ivan Sutherland [6] described the "Ultimate Display" concept, /which is the most scientific description of "Pygmalion's Spectacles"/:

 

“The ultimate display would, of course, be a room within which the computer can control the existence of matter. A chair displayed in such a room would be good enough to sit in. Handcuffs displayed in such a room would be confining, and a bullet displayed in such a room would be fatal. With appropriate programming, such a display could literally be the Wonderland into which Alice walked.”[7] 

 

1999 – “The Matrix”[8] The concept of the film is that we all live in the simulated world, make no difference from the "real" reality. It is a "digital version" of Plato's cave. And a pretty accurate visualization of Sutherland “Ultimate display”.

 

 

 

Language of the film

 

Cinema is a relatively "new" art but yet we can`t clearly define "the beginning" of the "filmmaking". For the first "moving pictures" is accepted "The Horse In Motion" (1878).[9], which is created with a series of photographs combined in the "moving pictures" or film. The Lumiere Brothers "Arriving of the train" /1896/ was so scary for the audience and the people start to run away from the coming train. True or a legend but this is the first sign of the impact of the “moving pictures” upon the viewers. The moving pictures can "generate" emotions. Be a reason for emotions. Have impact upon the audience. Now storytellers /moviemakers/ deliver stories to the audience trough the white screen. The moviemakers starts to think about the way to evoke the emotions in the most controlled way. Great example is the montage theory of Sergei Eisenstein*. 

He defines theory of "montage of attractions" in five methods of montage:

Metric – length of the shot

Rhythmic – what is – what happens in the frame

Tonal – lighting, shadows, framing, the composition of the shot

Over tonal – combination of all four in the meaning of shot sequence and scenes inside the movie

And overall this is "intellectual montage". While all five are focused on inducing emotional response. A "global idea" of the movie. 

For Eisenstein the basic and most important “material” “for theatre and cinema is the audience":

"An attraction is any aggressive moment in theatre, i.e. any element of it that subjects the audience to emotional or psychological influence, verified by experience and mathematically calculated to produce specific emotional shocks in the spectator in their proper order within the whole." [10]

 

Eisenstein talking about “controlled influence” of the story upon audience. Using the language of moviemaking the author can predict and controlled the reaction of audience. 

After decades of experimentation, filmmakers discovered more and more ways to influence and keep audience attention in the specific direction – sound, camera angles, lighting, editing, special and visual effects, etc. And as the result the story could be better told. And better visualized. Moviemakers know how to involved viewers deeper and deeper into fictional worlds, and make this world real for the movie length time in the cinema.  From fires of Mordor to galaxy far away, audience already has experience of enjoying the stunning views and fiction worlds. Cinema becames the way to combain the emotions of theatre performances with visions of panoramic paintings and even beyond all this

 But still…we` re looking "through the frame".

As Eisenstein says the important is audience. And the reaction of audience. But the reactions of the audience are different. There are very few films which are "masterpieces" in terms of Eisenstein's definition of "impact the human emotions ". Mostly reactions of the audience are ambivalent. Especially in the cinema. Some of the people could laugh at …not so funny moments of the story. Some are focused, inside the story. It is different in the theatres. Live performance can "grab" attention of the viewer, make viewers "breathe together" with the actors. Main reasons is: theatre`s "world", no matter how unreal or fantastic it is, is just in front of you". You can touch it. It is a physically existing world. Realism is first of all physical. There are no CG elements, or postproduction. It is real time performance. While in cinema the story is projected. The story is already told and filmed. There is a distance in the cinema between the audience and the white screen, no matter how stunning the movie is. Magic is so thin. Single cough in the hall and… magic is gone. On the other hand, moviemakers have a bunch of tools (3d models, compositing technics, motion capture, reality capture or 3D scans, interesting and stunning camera angles and views including virtual camera freedom, etc. for producing a photorealistic vision of the movies. And for a bit more of hundred years movie makers did a lot. Especially nowa days with all new compositing technics, powerful computers and render farms, calculating faster a huge 3d scenes and characters, etc. Combined with a good story and stunning visual effects movies can be – and THEY ARE worlds of wonder…but still… this feeling of the frame of the “white screen”.

 

 

                                                VR integration and movie making

 

Visual effects are a really powerful tool in moviemaking for achieving the realistic vision and involved viewers emotionally in the story of the movie. This is a fact already. These days not only high budget productions, but almost every movie has a bunch of VFX. In a lot of productions, we can see now computer-generated characters /Star Wars, Blade runner 2, etc./ Cinema himself is still "young" art. And there is still a long way to go. Compare to the cinema, VR technology is a baby in arts/not in the meaning of idea but in the meaning of technical realization/. The difference between cinema and VR isn’t so big. We can easily think about greenscreen shots as a kind of virtual reality. In fact, greenscreen is a fantastic, imaginary world and actors interact with this world, using their imaginations. They are interacting with fiction character and environment. On the picture bellow is the romantic scene between Anakin and Padme in "Episode I" during the filming.

 

 

Of course, after the postproduction, scene looks much different. What is important here is - in the movies – creators work "for the shot". There is still something to be done, to be added after the filming. Various elements will be added in the "post". The environment /especially in high budget VFX movies last decade/ is entirely greenscreen - fictional, virtual. Tim Burton`s "Alice in Wonderland" is an entirely greenscreen based movie. The entire set is green screens everywhere and fictional reality. And actors interacting with … their imagination. 

But what if...?! 

What if the romantic scene can be filmed with the actors being not on the greenscreen – but in the real "galaxy far away" environment?! Looks impossible.  The "galaxy far away" is far away and even we have space ships to get there – it will be so expensive to film the scene that way. That's why we have all this VFX knowledge and tools. To "filming the impossible". And it is possible. With the help of VR technology. Already exists a possibility to "project a "real" virtual set on top of the green screen. In real time. Unfortunately, it is only on the director`s monitor. But this is a start. And, yes, directors can see a complete shot on the monitor. With real-time camera tracking and real time CGI simulations and environment. And it is opportunity to plan and compose the shot better on set, not in postproduction. Building a virtual reality set on top the green screen is in a great help for the directors. That is what technology like N cam offers.

N cam's is quite innovative technology. And that’s what it does. Provides for directors ability  to “preview”,  how the complete VFX shot will looks like. Is is much easier to compose the shot “on set”, doing all necessarily corrections right before filming.  And it is much easier for postproduction and for movie making process. It is a huge timesaver. And it s just a matter of time to allow on set projection of real-time visual effects, as they promised from the company. Ncam technology have being used in several productions already: “Aquaman” (Warner Bros), “Fantastic Beasts 2” (Warner Bros), etc.

 

"Ncam's software includes 3D point-cloud generation, which makes the camera tracking marker-free. The company also has a depth-sensing option called Real Depth which can extract actors from greenscreens with interactive depth-based compositing." - says Ncam CEO Nick Hatch. "The system figures out where the real-world lights are in terms of direction, brightness, and colour, and recreates those in real-time," explains Hatch. [11]

 

And this is amazing. Combining the technology with the knowledge of compositing, modelling, lighting, etc. All skills of the VFX artists then we literally can achieve everything. “The limit is the sky” will not be a metaphor anymore. And the productions are halfway there. There are a lot of "virtual" news sets on many TV channels. Forecast studious are kind of VR environment also. It is just a matter of time to get rid of green and blue screens. The VR environment will be projected to “surround” the actors. Technology is innovative but simple. It is a reversed camera projection method, used in compositing. For instance - camera projection in Foundry NUKE. VFX composer “project” on the geometry /cards or cubes, spheres or real 3d models/ an image. It can be from single texture to entire environment. VFX compositors are able to remove and replace background of the live action shot /even in 360 degrees panorama/ with the projected one. Camera Projection technic is used in the compositing for sky replacement or for projecting a matte painting backgrounds. So the difference between camera projection technics and N cam technology is not so big. It is just reversed process and with, already existing deep compositing, laser scanning and 3d point clouds, very soon will talk about VR set instead of green screen. Actors will act and INTERACT in and with VR environment, instead of green or bluescreens. It is not so far the time when some of the traditional compositing practices like green screen keying and rotoscoping will be a history. And the movies will be filmed entirely in the virtual reality sets. With real-time simulations and computer graphics. And this process will be mostly capturing the environment than filming an image sequence. The compositing post process will be no more "adding a background behind the actors" but putting the actor/s/ INTO environment. The montage will be an entirely different concept. Working "for the shot", will be an entirely different meaning. Movie format will be "full 360". This is a new way of filming and a new way of montage. It will be not necessary to stop the camera just for changing the set or point of view, once you are in 360 degrees, controlled VR. Changing the environment will be as easy as just press a button. "Set extension's on postproductions" will be history. We can have an entire scene like 3d projection or scan straight in the studio. And this will be not studio anymore but VR set. And this is a fread opportunity for 3d modeleres. The industry will need more and more artists to build models of 3d environment. VR technology already is and will be really powerful tool for moviemakers, but what about the

 

 

 

 

VR ways of storytelling

 

            For more than 100 years of history, cinematography has a bunch of tools and specific “movie language”. Cinema has masters of art like Antonioni, Kubrick, Tarkovsky, etc. Filmmakers have a good known ways to tell a story – to focus attention on the specific object or subject wit composition of the shot, lights on the scene, camera zooms, visual effects, etc. 

VR is on the way to discovering methods of own storytelling. Own language. And this will going be a long way. Most of the principals of cinematography are irrelevant in VR environment. For example - there is no more "rule of thirds" in the 360 environments. The way of compositing environment/shot/ will be different. The rules for perspective will still works, as much as 360 VR is much closer to the real world than the movies. And this will be a major challenge. The experience we already have in games and cinema practice is irrelevant in VR visual product.

 

"We're moving into a dangerous medium with virtual reality," Steven Spielberg said at a press conference at the 2016 Cannes Film Festival. "The only reason I say it is dangerous is because it gives the viewer a lot of latitudes not to take direction from the storytellers but make their own choices of where to look."

 

When the viewer is given this power, as James Cameron points out, the experience becomes a gaming environment: "If you want to move through a virtual reality it's called a video game, it's been around forever." - he said. And he is right, as a moviemaker. He is right until we still talk about movies. Yes – the language of the cinema doesn't work in VR. Yes, game environment is VR. But we do not talk about the cinema, we talk about entire new media – different from the movies and games. Compare games and movies to VR – games and movies have a closer relationship like methods of storytelling between each other, than with the VR. With all cinematography methods, the author /director/ leads audience trough the story shot after shot, scene after scene and action after action. No freedom of any choices for the viewer. It is predetermined process. It is the same in the games. To get in the higher level you have to achieve something in the previous one. You have freedom of choice and movement but… not exactly. The designers of the game don't give you so much choice. You cannot pass if you do not defeat the Boss for instance. But in the VR there might be or might be not a Boss. You might fight with the one /if there is/ but also you don't have to. You may to choose to do not. And this is not a game or game environment. This is the way of story happening with "to do decisions at the moment" element. Films and games has predetermined outcomes. What happens in the film is: sometime, somewhere but in the VR it is here and now. Wherever "here" is and whenever "now" is. And VR needs to find own ways to tell stories. The main task for creators is to find the way to keep viewer/users/s attention. Another global task is and to allows them to "create" the story for themselves. We talk about the story "happening". We don't talk about protagonists anymore. The audience/viewer, user/ is now the characters in the VR story. Viewers are no longer going to be watching characters, they are going to do interact with them. In a 360-degree live-action video, /witch, again is kind of portal to the full 3d SC environment media and experience/, the viewer's interaction is limited to making choices concerning their field of view. And to attract and direct viewer's attention is not that easy. 360-degree video environment allows the viewer a great amount of freedom and there is a risk, as Stephan Grambart notes, of "the audience to ignore your story" [13]. He also suggests different methods like "implement sound and action clue to direct attention towards the primary story area, place points of interests, that add context to active narrative", etc. The VR media already did firsts steps – there is a long way forward, a lots a questions waiting for the answers but…

 

 

Conclusion

 

 

…as we can see the idea for virtual reality is not new. From Plato`s cave to Lumier`s Brothers, from panoramic pictures to 360-degree video, the way is long but the distance is not so big. With the modern technology and with the knowledge and the technics invented and used by VFX artists, virtual reality is another, a NEW method for telling stories.It is a UNIC method – the viewer now is the director but also a story maker and protagonist. And this is the first hudge difference. There is viewer or user but not “audience” anymore. It is more personal and private contact. Virtual reality sets are similar to theatre stages, they are “kind of physical” and the viewer /user, creator/ is surrounded by this reality.  There is no longer distance to the “white screen”. There is no more disturbing factors. Magic is not thin anymore, magic is everywhere. 

Thinking about VR it always reminds me the myth for Pygmalion and Galatea.[14] The story for creator and his work came to life. 

VR experience nowadays is still interaction with some really basic environments. And the some of the main problems like ways to keeping viewer attention still remains. But already have a lot of films in VR. And new and more solutions. New Disney`s “Lion King” – 2019 is a successful project. Still interaction abilities in 360 degree movies are nothing more but watching and observing around. But this is the beginning of building a language of the new VR media. We can accept it as an entry level to more complex form of full 3D CG environment, which is still more on level of interaction than storytelling. 

360 movies are the most logical first step for VR experience. Just like the cinema starts with simple observation “Horse in motion” and "Arriving of the train" and then became a new art, with own stories and own language. VR must grow up and will grow up in a similar way. Get in to the industry and entertainment trough “white screen window”. Viewer to be involved in the well-known story and environment. With more and more 360 films. May be even with remakes of some of the classical movies in 360 format. And as a next step - a full 3 d virtual reality environment. And then following the logic of "stage, actor, and viewer", we will have a stage – let's say Star Wars environment, actors – a CG generated characters, or another viewer or users, sharing the same VR, and may be the first interaction and dialogue, the first VR Thespis, and we will have the basic kind of story, and also a viewer who is an actor also, and also the one who`s telling the story based on the choices he made. This future is not so far. A lot of questions still needs an answer but one is sure, as Andy Serkis said: 

 

“This is it! You can play anything. Anyone can play anything.”[15]

 

 

 

 

 

 

 

--

 

P.S. Possible scenario for VR movie /with all respect to Jean Baudrillard/ - Many years from now, when humanity lives in the entire virtual reality – sharing a common virtual space, interacting, creating different creatures, animals, etc. Do/these creatures/ will ask themselves questions about the world of shadows and the real-world?!?! ….

 

-----------------------------------

 

Reference list:

 

1 – "Forrest Gump" – 1994, director Robert Zemeckis, lead character Forrest Gump (played by Tom Hanks).

 

2 - Aristotle - "Poetry", transl. with notes by Th. Twining, I-II, London 1812

 

3 - Oxford Dictionary Online. 2017. https://en.oxforddictionaries.com/definition/virtual_reality.

 

4 – Jerald, Jason - The VR Book: Human-Centered Design for Virtual Reality,  books.acm.org www.morganclaypool.com ISBN: 978-1-97000-113-6,  A publication in the ACM Books series, #8, First Edition.

 

5 - Weinbaum, Stanley G.  - "Pygmalion's Spectacles", A Project Gutenberg of Australia eBook, eBook No.: 0607251h.html, language: English, September 2006./This eBook was produced by Malcolm Farmer/

 

6 - https://en.wikipedia.org/wiki/Ivan_Sutherland

 

7 – Sutherland, Ivan E.  - Information Processing Techniques Office, ARPA, OSD, Proceedings of IFIP Congress, pp. 506-508, 1965. 

 

8 – "The matrix" – 1999, directed by Wachowski`s, lead character Neo, played by Keanu Reeves.

 

9 -https://www.youtube.com/watch?time_continue=6&v=IEqccPhsqgA&feature=emb_logo

 

10 - Eisenstein, Sergei, Selected works, Volume 1, Writinghs_1922-34, "Montage of attractions", page 25/.

 

11 - Failes Ian, "The reality of virtual filmmaking", VFX Voice magazine, December 13, 2018.

 

12 – Cinema Blend – “James Cameron Hilariously Insults Virtual Reality And Oculus Rift Technology”, OCT. 30. 2014 11:12 AM,

https://www.cinemablend.com/new/James-Cameron-Hilariously-Insults-Virtual-Reality-Oculus-Rift-Technology-67959.html

 

13 - Grambart, Stefan -  "Sleepy Hollow & Narrative in VR (Slide Share)." Published on December 8. 2015. https://www.youtube.com/watch?v=avB57nHI8Ps

 

14 - https://en.wikipedia.org/wiki/Galatea_(mythology)

 

15 - Holmes, Adam – “After playing Gollum, Andy Serkis has continued to rule motion-capture acting — here's why he loves it”, Cinemablend, Jul 13, 2017, 2:51 PM