This research will allow me to expand my knowledge in regards to Immersive Sound, specifically on Binaural 3D Audio.
Helpful industry-relevant resources will provide answers to several key questions, mapping a complete path towards the subject and project objective.
Analysing the essence of the format, the required technology and the ways to create and consume such products, will allow me to be competent and understand the possible purposes and limits of this technology.
What is Immersive Sound?
Immersive Sound can be defined with two of its main features: space and technology.
Space because it "provides the listener with a natural ("life-like") three-dimensional sound experience unlike anything heard before in traditional 2D surround solutions. It creates the sensation of height all around the audience, transporting them into a more thrilling and deeper audio experience". (Stormaudio, n.d.)
Technology because it involves new formats, codecs and development of devices in order to create multimedia texts available to consumers.
Among the formats, there are Dolby Atmos, Auro 3D, 360 Reality Audio.
I have decided to focus on Binaural 3D because it is the format that requires fewer resources nowadays, both in terms of software and hardware; it intrigues me how it can revolutionise the perception of recorded audio.
What is Binaural 3D Audio?
As described by Dear Reality (2019) in the dearVR Pro manual, "Binaural 3D Audio is a technology that simulates the human spatial hearing via headphones. If you listen to common stereo audio with headphones, the perception of all sound sources is located inside your head - between your left and your right ear. With 3D Audio you get the sound outside of your head where it belongs. A sound appears to emanate from a specific point - anywhere within a full 360° three-dimensional sphere."
As stated, to consume a Binaural 3D Audio product, the listener only requires headphones; a less expensive requirement compared to Dolby Atmos for example, whose systems involve a full set of several speakers. (Dolby, n.d) The same requirement applies to the producers, although in the production stage there may be a limit due to possible headphone restrictions (frequency response, hearing health), as discussed by Tim Gedemer in VR LA (2017) DTS's Immersive Audio panel.
(Dear Reality, 2019)
Which tools are required to process audio in 3D?
Luckily, nowadays with the use of a single plugin per channel in a DAW session, it is possible to convert any sort of audio file to 3D. The choice among the tools to use is quite vast. During my research and investigation, I came across several ones:
dearVR Pro by Dear Reality
360pan suite by Audio Ease
Ambeo Orbit by Sennheiser.
I have opted for dearVR Pro because it is part of a brand that is totally focused on 3D audio for immersive content (Dear Reality, 2020), which has developed dearVR Spatial Connect (Dear Reality, 2020) (a platform that enables mixing using a VR Headset) and dear VR Unity (Dear Reality, 2020) (a plugin to spatialize sounds inside game engine Unity), which I intend to acquire, study and use in future. Also, the plugin seemed fairly straightforward in terms of interface, features and general use.
How does the plugin and processing work?
To place a recording in tridimensional space, it is required to put the plugin in the track's inserts. It works both for mono and stereo tracks, however, it is advised to use mono sources to avoid phase issues and loss of quality (Lane, 2019). Studying the plugin, the ideal way to use stereo sources is to duplicate the track and set the plugin on both tracks with the input channel respectively on L and on R - using the dedicated control on the right master section. (Dear Reality, 2019) This allowed me to solve part of the main issues with stereo sources, although they were still not totally perfect, preferring mono sources for future similar productions.
Some tests provided by Dear Reality in regards of stereo and mono sources, comparing the outcomes of their spatialisation and how the split of stereo in two channels (assigning L & R in the input channel section of each dearVR Pro instance) or their sum in a single mono may differ.
The sound can be visually dragged around a graph (that can switch from cartesian to polar), with the possibility to "position the object all around the listener - behind, in front of, to the right or to the left of and even below or above". (Dear Reality, 2019)
The content will be characterised by the virtual acoustics that the plugin applies, adding reverb to it. This can be tweaked, changed (choosing among a list of different ambiences and locations provided by the developers) or turned off. As the concept is to represent the sound around the listener, depending on its position, the perception may change, with the use of algorithms that change the frequency response and the features of the sound simulating how human ears would behave; this process allows to perceive the illusion of space as it would happen in the real world. The "Occlusion" control next to the graph enables cutting the high frequencies to create a custom presence of the sound. The main settings of the plugins are focused on shaping the space in which the sound is placed, as it is its acoustic character that will define the final result and perception at a given position. Size of the room, reflections, and dry/wet controls of the reverberations are the keys of this simulation.
Other than position and acoustic elements, there is a master section that allows you to:
boost the bass of the source
increase the performance of the process for a static object (this option is also ideal when using very large rooms, the longer the reverb the higher the performance and resources required by the process)
use head tracking with dearVR Spatial Connect
choose the final output depending on the desired format
Several resources that helped me to understand how to use the plugin and complete a Binaural 3D Audio post-production.
The main point of the mixing stage for Binaural 3D Audio floats around the fact that, compared to stereo, the point of view of the listener has changed. Hence it is important to consider where to place the sound elements, visualising them as objects in space.
As Robinson suggests in Australian Film Television and Radio School (2018), less is more and trying to avoid layering sound to mask or cover elements, will result in a clearer and more enjoyable result. Also, it will help to keep the Binaural effect strong, as if the mix is too busy, the effect tends to get weaker.
Distance and height may be great variables to create priorities among the sound sources, sending secondary elements further than main ones. Depending on the genre, decisions may change, shifting what to keep in the primary close area.
Balancing sources around the imaginery 360 sphere is the key to the 3D mix, although reflections, type of reverbs, dry/wet mixing will be required to obtain an acceptable result.
While working on the projects, I have understood that various music genres may respond differently to this format; it is all focused on the objective of the production and how the music is desired to be consumed. Indeed, the sense of "distance" of the sound around the listener creates an initial weird perception of it, especially in genres where the core elements perform perfectly in the "standard" way (Drum Beat in Hip Hop and Bass in Reggae for example). But it is a feeling that may change with more consumption; it is a new experience and the ear must get used to it before being able to completely appreciate it.
Genres like Jazz may benefit from this, as space and sense of ambience is already a core feature. Ambient/Electronic music as well, where the pan-automation of synthesizers and patterns can create engaging and interesting journeys for the listeners. Still, it is important to highlight that it is not a substitute of standard post-production, at least in the refining (equalisation, dynamics) phase of each individual instrument; it is more of an additional step that may take the experience to a different route. (Sullivan, 2020)
From a technical point of view, live recorded music and events may benefit from this format, as it is then possible to recreate a sense of space and environment that is key of the experience of live concerts (ibid). The stage position of the artists on the X, the crowd on the Z and mixed elements on the Y would give a fully immersive perception, that is not possible to convey through the 2D of the stereo.
(Dear Reality, 2019)
Dear Reality GmbH (2017)
Plugin Alliance (2018)
Australian Film Television and Radio School (2018)
ZDF Digital Medienproduktions GmbH (2019)
Dear Reality GmbH (2017)
Dear Reality GmbH (2019)
Dear Reality GmbH (2019)
Dear Reality GmbH (2019)
Auro-3D (2020) Auro-3D. [online] Available at: https://www.auro-3d.com (Accessed 4 April 2020).
Australian Film Television and Radio School (2018) The Right Way to Mix Binaural Audio – Catherine Robinson's Top Tips. Available at: https://www.youtube.com/watch?v=ILcQX_GMd5w (Accessed: 05 April 2020).
Dear Reality (2019) dearVR Pro. Available at https://files.plugin-alliance.com/products/dearvr_pro/dearvr_pro_manual.pdf (Accessed 27 March 2020).
Dear Reality (2020) THE EVOLUTION OF AUDIO MIXING - Dear Reality. [online]. Available at: https://www.dearvr.com/ (Accessed 3 April 2020).
Dear Reality GmbH (2017) dearVR PRO | Best of Binaural mixes. Available at: https://www.youtube.com/watch?v=YyhdEeui_PY (Accessed: 04 April 2020).
Dear Reality GmbH (2017) dearVR PRO | Immersive 3D audio VST, AAX, AU plugin. Available at: https://www.youtube.com/watch?v=zr-5RSiz-8Y (Accessed: 04 April 2020).
Dear Reality GmbH (2019) Dear Reality - Rain Test (Binaural). Available at: https://youtu.be/i_7IzzFVK-c (Accessed: 04 April 2020).
Dear Reality GmbH (2019) Dear Reality - Truck Engine Test (Binaural). Available at: https://youtu.be/dU0i90DvRDE (Accessed: 04 April 2020).
Dear Reality GmbH (2019) Dear Reality - Window Washer Test (Binaural). Available at: https://youtu.be/NyNMvvQih-c (Accessed: 04 April 2020).
Dolby (n.d.) Dolby Atmos For Home. [online] Available at: https://www.dolby.com/us/en/technologies/home/dolby-atmos.html#2 (Accessed 3 April 2020).
Dolby (2020) Dolby Atmos. [online] Available at: https://www.dolby.com/us/en/brands/dolby-atmos.html (Accessed 4 April 2020).
KlangKönner (2018) dearVR music tutorial - mixing jazz music in 3D - (HEADPHONES REQUIRED). Available at: https://www.youtube.com/watch?v=4-dGcjqNzO4 (Accessed: 04 April 2020).
Lane, C. (2019). WHY STEREO RECORDINGS AND SPATIAL AUDIO DON’T (USUALLY) MIX. [Blog] DEAR REALITY BLOG, Available at https://blog.dearvr.com/overview/why-stereo-recordings-and-spatial-audio-dont-usually-mix (Accessed 3 April 2020).
Plugin Alliance (2018) dearVR Pro 3 Dimensional Mixing Techniques. Available at: https://www.youtube.com/watch?v=ZU-wglqm0JE (Accessed: 04 April 2020)
Sony (2020). 360 Reality Audio. [online] Available at: https://www.sony.co.uk/electronics/360-reality-audio (Accessed 4 April 2020).
Stormaudio (n.d.). Immersive Sound. [online] Available at https://www.stormaudio.com/en/immersive-sound/ (Accessed 3 April. 2020).
Sullivan, M. (2020) How Sony’S 360 Reality Audio Makes Recorded Music Sound Like An Immersive Concert. [online] Fast Company. Available at: https://www.fastcompany.com/90413353/sonys-360-reality-audio-makes-recorded-music-sound-like-an-immersive-concert (Accessed 4 April 2020).
VR LA (2017) Immersive Audio for VR Workflow, Presented by DTS. Available at: https://www.youtube.com/watch?v=EQHkmysYnes (Accessed: 05 April 2020).
ZDF Digital Medienproduktions GmbH (2019) Mixing Music in VR with Spatial Connect - The Future is NOW!. Available at: https://www.youtube.com/watch?v=-_Z3UtfqK6s Accessed: 05 April 2020).