Bringing 360-degree video and virtual reality (VR) to journalism

 

Imagine your readers having this reaction to a story.

 

This video was a roller coaster ride simulated by a computer. But what if the same thing could happen in journalism?

With cameras and rigs that allow us to film in all directions, and a create 360-degree view.

With platforms like Youtube and Facebook that allow 360-degree video to be uploaded and shared on the Internet.

With mobile apps that bring virtual reality content directly to your phone to be watched on more and more accessible and cheaper headsets.

Virtual Reality technology is the next great wave in media. We are beginning to be at the same point as when cellphones started becoming popular.

For my project I partnered with Fusion. Fusion is a joint venture between Disney and Univision.

One of the goals of Fusion is to experiment with new technologies to prepare their parent companies for the future.

My project was to research how to use 360-degree cameras for Investigative Unit at Fusion. This is the first 360 video we made.

As you see, today we have two types of content in virtual reality, one created by a computer — the other by a camera.

The first one has more history. Because it’s made entirely by a computer it allows for much more interaction.

The second — 360-degree video — is really new, only pioneers have been trying it lately. Because it is created by a camera, we can explore, but we still cannot really interact.

What’s happening now is that these two fields are CONVERGING.

That is why filmmakers are beginning to talk about the possibilities of volumetric filmmaking. The capacity to film and 3D model a real scene, capturing pixels with depth. So we can film people like this woman with depth and place her in a virtual environment.

For my project, I was one of the first three beta testers of the new VR version of Klynt, an interactive editing software.

Klynt VR, is one of the first software programs that will allow us to create interactions inside 360-degree videos. Like this short sample:

And more importantly, it allows editors start placing spatial sound in video.

In this demo you can see how inside this video I’m able to position the sound of the TV or the window so the user will be able to feel the changes as she explores the video.

By now, we can all agree that VR is a new medium. The workflow to produce it is different, the way users interact with the content is new.

And little by little we begin to acknowledge how big of a shift this is. When Molly Swenson from Ryot said the following, during the StoryNext Summit.

Or the creators of The Displaced realized:

Also myself I have encountered many shifts and challenges when trying to do journalism with this technology. For example:

In journalism we used to think: the crew cannot appear in the shot. In 360 video that is impossible. So how do we solve that?

Maybe we are going from asking questions to agreeing to scenes.

And what about using light in 360? See this test:

So maybe we are going from:

The audiovisual language for 360-degree video is completely new.

One thing we know for sure. We will soon be able to put the viewer inside the story.

Meanwhile 360-degree videos and virtual environments are converging.

I would like to encourage you to begin testing low cost cameras and edit short experiments. The tech is changing quickly. No one has mastered it yet just jump in and start using these tools!

The workflows are becoming more and more complex. We have to understand other backgrounds and fields that we can now apply to journalism.

Share your findings! We are in the infancy of this new medium and we have to learn all together the possibilities and complexities of it.

I leave you with quote of the godmother of VR in Journalism. See how bizarre but inspiring this quote becomes in the context of journalism.

This project wouldn’t have been possible with the help and collaboration of many people at Fusion, specially Ketih Summa and Adrian Saravia. Also I wanna thank my classmates at NYU and Jay Rosen, the people at Klynt VR and many conversations I had in these past months.

Also I want to thank my friends and family. Thank you very much.

***Slides and notes of my final presentation at Studio 20 NYU

Depth your pixels!

Depthkit is a new technology to film 3D video. The tool allows you to record pixels with depth, relating two sources: the depth data from a Kinect XBOX and the image from a DSLR camera.

And it’s all open source!

Maybe you are familiar with this documentary project that appeared two years ago in a Kickstarter campaign:

The DepthKit, which many call “the future of filmmaking” is being developed by James George , Alexander Porter, Jonathan Minard and Mike Heavers.

DepthKit has been used in many projects, for example Katerina Cizek, used it for “The Universe Within”, the last part of her multi-year interactive piece Highrise.

Depth Kit is a really complicated process (don’t mistake it with the format that Radiohead used for her videoclip House of Cards in 2008).

Luckily for me this semester I met Sisa Bueno a graduate student at ITP-NYU that had taken an elective class with two of the developers of the Depth Kit and she was looking for a partner in crime to shoot with this format. Sisa and I stayed two nights at the Tisch Film School at NYU, filming for a story she is preparing.

Me, trying to understand that I’m shooting for a depth sensor.

To create a 3D image of a video you are filming first you need to communicate to the DepthKit software the physical distance between both cameras. You mount the Kinect and the DSLR together and you move around the room 180 degree — , so the software can calibrate the distance between the cameras and a black and white check board.

This process took us around 6 hours. Normally it’s not as long as that but the whole process was long (first to understand how the studio worked, the lights, get the passwords for the computer, understand what we were doing…).

And after all that, we were ready to start.

In this picture you can see Sisa Bueno, the DSLR, the kinect and even myself.
Sisa ready to be filmed with the Depth Kit

Sisa managed to direct the whole operation very well, reading her notes from the class and the explanations in the DepthKit website. She is also a documentary filmmaker and it was a pleasure for me to enter into the world of computational video by her hand.

A few days after recording, Sisa showed me the results of our work.

And here a screenshot of our first 3D image!!

Using this technology made me realize how the future of filmmaking will be be made by cameras that can position pixels in the space and by software that allow computational images to be edited together, in a time sequence.

Right now, computer generated content and 360-degree video is processed in different softwares (Unity or Cinema 4D for the first one, Autopano and Adobe Premiere for the second one). But this technologies will probably merge soon and the possibilities of virtual reality and 360-documentary will explode!!!

Last but not least, thanks to Nicholas Hubbard, also a grad student ITP I met at the Storyscapes Tribeca, that connected me with Sisa Bueno!!

New York Times se lanza a la realidad virtual

El periódico estrena su primer documental en 360 grados y demuestra que existen posibilidades narrativas en el video esférico

En mayo del 2014 se filtró al mundo la estrategia interna que el New York Times (NYT) preparaba para adaptarse a la era digital. El llamado Innovation Report era un documento interno que reveló el miedo que el periódico empezaba a tener a sus competidores digitales.

En pocas palabras, el Innovation Report demostraba cómo una institución como el New York Times apostaba por una fuerte estrategia digital. Renovarse, antes que morir.

Desde el Innovation Report hasta hoy, muchas cosas han cambiado en la redacción del diario y este fin de semana más de un millón de subscriptores han recibido en sus casas, junto con el periódico del domingo, unas Google Cardboard: las gafas de cartón que permiten tener una experiemencia de realidad virtual (VR).

Unos días antes de enviar las Cardboard a sus subscriptores, el New York Times organizó una fiesta en Manhattan para celebrar que se unía a la revolución del video en 360 grados y presentaba The Displaced, su primera pieza documental en realidad virtual (de hecho, esta es la segunda que han realizado, la primera fue un reportaje corto que salió en abril).

“119 años atrás la revista del New York Times publicaba la primera fotografía” dijo el editor The New York Times Magazine, Jake Silverstein, en la presentación del NY TIMES VR: “Hoy presentamos nuestra contribución al periodismo en realidad virtual”.

El público en el evento de estrena de la APP NYT VR y el documental The Displaced. Foto encontrada en Twitter

Mientras nosotras, las cobayas y público del evento, nos desplazábamos virtualmente al sur de Sudán, Ucrania y Líbano con The Displaced, Jake Silverstein y Chris Milk (director de la productora VRSE con la que el NYT ha co-creado la pieza) aprovechaban para hacerse fotos para la posteridad de tan simbólico momento.

Jake Silverstein y Chris Milk haciendose una foto con el público de fondo.

Aunque la realidad virtual aún no será consumida ni producida masivamente, poco a poco se irá implementando en nuestras vidas. La tecnología para crear y experimentar VR es cada vez más accesible y son muchos los medios de comunicación que han empezado a probar este nuevo medio. De hecho, si tienes un Iphone 6 o un Android mayor de 4.3 puedes descargar la aplicación del NYT VR y otras, como LittleStar o VRSE, y ver muchos de estos trabajos pioneros. También puedes comprar unas Google Cardboard en Internet por unos 5 y 20 dólares.

Parece que podemos confirmar que, como auguraba hace meses Eva Domínguez: “el 2015 será el año en que el video esférico se hizo popular.” Sí, el mismo año en que los ejecutivos de Sillicon Valley se pasean con sus inventos y softwares 3D por las redacciones de los periódicos de la costa este y la primera vez que la agencia de noticias Associated Press revisa sus estándares periodistícos y éticos para adaptarlos a la grabación en 360.

Afortunadamente para todxs, no hace falta ser el NYT para hacer realidad virtual. Pero si eres el New York Times y consideras que la tecnología es intrínseca a tu trabajo, si combinas tus elevados presupuestos, tu saber hacer y la pionera experiencia de artistas de VR como Chirs Milk, puedes lograr hacer una magnífica pieza como The Displaced.

No voy usar este post para desvelar mucho del documental que han lanzado en VR, solo diré que en estos últimos meses he visto muchos videos en 360 grados y asistido a muchos debates sobre cómo será el nuevo lenguaje audiovisual o si se podrá usar en periodismo.

Hoy el New York Times ha dado un empujón al documental de realidad virtual, primero enviando gafas de VR a tanta gente y promoviendo el uso de esta tecnología, pero muchísimo más importante, demostrando que en realidad virtual existen posibilidades narrativas.

El video en 360 grados no sólo ofrece la posibilidad de sentirse en el lugar donde situaron la cámara, sinó también de percibir como habitamos el mundo. Nos hemos acostumbrado a representar la realidad a través de la fotografía y el video tradicional, a través de encuadres rectangulares y frontales. Pero nuestra sistema perceptivo es mucho más amplio. La direccionalidad de los sonidos, los ruidos de fondo, las zonas donde enfocamos o no la mirada, mover la cabeza para buscar lo que queremos mirar, quedarnos quietas a escuchar…

Hemos estado muchos años creyendo en la ilusión de la imagen cinematógrafica, pero las posibilidades representativas pueden ir mucho más allá. Véanlo por ustedes mismxs.

What is 360-degree video, and how can we apply this to journalism?

This post is a summary of frequently asked questions that I get everyday from friends, family, colleagues and classmates. Right now I’m working on my final thesis about the possibilities of 360-degree video in journalism and I’m partnering with Fusion Network for my hands-on research.

Panoramic frame of a 360-degree video inside a juvenile kids in Louisiana by Fusion Network

360 degree video:

360 video is a new form of shooting and experiencing video that allows the user to look in all the directions of a video-recorded scene. In 360 video or spherical filmmaking, the user, or the audience, no longer looks straight to a flat framed image, but explores the video by moving the fingers (by swiping the image around) or by moving a device like a smartphone or a tablet. 360 video can also be enjoyed in a VR headset although it doesn’t mean all 360 video is virtual reality.

VR Festival Kaleidoscope, NYC October 2015

360 video is not exactly the same as Virtual Reality (VR)

360 video is the simplest form of immersive video, but although one can experience 360 video with goggles or a headset (like Google Cardboard, Oculus Rift or Samsung Gear), it doesn’t mean it’s Virtual Reality or cinematic VR.

Cinematic VR requires 3D vision and ambisonic sound (which means you see different images for each eye like in “real” life and you also hear the direction of the sound as you move your ears). In many cases not only you feel immersed in the VR but you u can actually interact with your body and move around.

Freedom 360 mount. This rig shoots in all directions but still in 2D

Right now, the majority of 360-degree videos are shot in 2 dimensions (so like a “traditional video”) and the sound is often stereo — not ambisonic — and often not even binaural (different sound depending on each ear).

Keep in mind, the majority of 360-degree videos you will see online are just an expanded spherical 2D video. And the majority of VR content out there (today in 2015) is mainly computer generated (CG.).

Immersive is a term we borrowed from videogames

The word “immersive” is borrowed from the videogame industry. The concept of “immersive” was applied to designate videogames that blurred the line between physical and simulated world. So for example, games where the user interacts with a musical instrument, or a toy-gun, or goggles, or a headset. Basically it means that you are immersed in this virtual world created (often) by a computer.

Pioneering project of Nonny de la Peña Hunger in L.A — Photo by emblematic.net

In this instance, Nonny de la Peña was the first journalist to start using videogame technologies and computer generated environments for factual reporting. To describe her work she used the term “immersive journalism”. The pioneering work of Nonny de la Peña Hunger in L.A in 2012 or the most recent “Project Syria” have acquired a great following.

How to experience a 360-degree video

First of all, consider that this technology is really new and sometimes there are still some technicalities that make it difficult to enjoy. For example, if your wifi is not fast enough you will see the images in less quality and it can make the whole experience quite bad.

Youtube has its 360 Channel — you need to use it with Firefox or Google Chrome or install the Youtube APP on your phone.

Google Cardboard — you can put a smartphone on the front side and u are all set!

You should also download apps like VRSE , LittleStar , VRVideo, but you will need a smartphone higher than I-phone 4 or 5 or at least higher than an Android 4.

If you are really into it you should get a Google Cardboard (it can be really cheap, 5$/12$) and you will be able to enjoy some of this content on the Cardboad, you can also check this post with some reviews of more expensive headsets.

Some 360 video cameras have also their websites with videos 360 videos like Bubl Experiencies, 360 Fly, Theta.

Is this the next media revolution?

Nobody can see the future, but what we know is that the biggest tech companies are investing billions of dollars, and that the technology is becoming more and more accessible. Youtube has a 360 video platform, Facebook allows 360 video in its Newsfeed, the New York Times just announced it is going to distribute more than a million Google cardboards to its subscribers… These are big moves and signs of a change.

Other recommendations?

I think the best is to go to the nearest festival you find and enjoy the headsets and some of the VR works you will find there. If you download the APPS mentioned before, there are many interesting projects. Personally I think it is worth to check out the work of Nonny de la Peña and Chris Milk.  Also the work of Be Another Lab (The Machine to be another) or the story of the Nepal earthquake by RYOT.

Why I am interested in 360 video as a journalist?
Learning web and filmmaking changed my mindset as journalist. Learning new tools and working with professionals from different backgrounds helped me conceive stories differently. Because 360-degree video and immersive journalism seems to be the next incoming media revolution, for me, it’s key to understand how this technology will reshape my craft and how it will change society.