Mixed Reality and the Future of the Video Call

Interview responses for LBB Online, full article published here.

How do you think video conferencing / video meetings will evolve in coming years?

You mean to say you don’t enjoy marathons of staring down the barrel of a camera, your face shrouded in bad lighting, the icecream truck wailing all the way from the street through your microphone into a mosaic of rectangles of similarly dubious visions of your friends and colleagues? 

The evolution of video chat will be motivated by three key drivers: innovation (obviously), bespoke functionality, and physical comfort. I omit perhaps two factors - 1. vanity, and 2. the political state of affairs that may influence progress. I’ll intentionally leave them aside for now, as they’re articles unto themselves.

In the near future, there are some simple design changes VC platforms could implement into their existing interface and structure. Better design thinking around user behaviour, channelling the various needs of a video conference into tailored formats to best suit their purpose, reducing the emphasis on face-to-screen, the application of collaborative game mechanics, and the list continues.

A little further down the track, there are functional changes that will affect the way we use VC. Real time audio filters (Google Hangouts), using facial recognition or sophisticated AI to follow a user in their environment (Facebook Portal), utilising user data to provide a personalised experience (which of course opens the door to advertising opportunities, greater funding, and the development of highly commercial VC experiences). 

How can augmented reality and virtual reality aid in this evolution?

Somewhat parallel to the progression of video conferencing, the XR industry continues to make tantalizing steps toward virtual togetherness around shared interactive content. When video streaming and extended realities become inseparable, we’ll barely notice that we’ve made a big shift to 3D content. We’re already seeing it. Computing interfaces are shedding their tethers and attaining a greater understanding of their environment. The ability to scan your world, upload it and have a colleague deploy it virtually in their environment - is in your pocket. 

XR is a digital sense that offers presence and shared stimuli. Body language, eye contact, full body motion, physical spaces, a dimensional perspective, spatial audio, haptics… XR opens up communication to a broader range of our innate human senses. 

Can you outline some of the technical elements of how this can eventually become a reality? For example, from a production standpoint, how will things like real-time processing help?

There’s already been significant steps forward into this reality - it’s happening! Access, innovation, better hardware, the advent of 5G, content standards, generations of millennials and their cohorts who seek experiences as opposed to a linear delivery. Big tech players are pushing further and further into XR, thereby reducing the cost for commercial uptake too. 

Real time processing makes production faster! Options are exponentially increased and the cost of having those options is reduced. Groups of contributors can make collaborative, informed decisions based on vastly more informative prototypes. Not only client-side is the experience different: on the creator side, the tools are accessible. Software for game production such as Unreal & Unity are affordable and extremely powerful engines with which to create cinema-quality real-time content.

Once this evolution begins to happen, how will our Zoom meetings of the future look and feel in your eyes?

A short narrative from the not-too-distant-future...

My watch vibrates, reminding me I have a meeting in 5 minutes, would I like to prepare? No AR preening today, but I’ll head into the waiting room and see who else is on. The meeting pops up on my (barely noticeable) projector - I see a few others - a client, I tap her icon to see more details about her role. The meeting commences, those who are speaking the most gently scale up in the unbound arrangement of portrait bubbles. At the office, a colleague steams milk in the background. AI automatically blocks the high pitch squeal from broadcasting to the group. 

My turn to share - I broadcast a WIP model of the character animation to the group, using their smartphones to view it in their respective environments - I scale it, rotate it, cycle through versions of the animation, adjust the lighting, play with the textures - on the fly we are able to review, test and finalise. Now we’ve locked down the character, we revert to the screen. I throw our latest character into it’s WIP environment - we go through the same process again in-engine - switching cameras on the fly, adjusting shots, adding a few more mountains, change the lighting from dusk to evening, add some particles to the points of interaction. 

After the usual review banter, we wrap up the meeting and log off. A transcript of the meeting is sent to my inbox, the updated prototype is already available to the team of artists to continue working.

In your opinion, will these video meetings of the future be enough to replace in-person meetings?

In the case of most of the meetings I attend, yes absolutely. Would I like my all happy hours to be virtual? Not so much. 

No matter how much we lower the tech friction - once you take the in-person social aspect out of an IRL meeting it still seems more difficult to develop personal relationships with clients, colleagues, collaborators. However, a video meeting in production is a hell of a lot more convenient than an in-person meeting. Nobody has to travel, no need for Kind bars & La Croix, nor to don your best black outfit. Whether video meetings are comparable with IRL meetings or not, they do significantly bolster globalisation. Removing the expense of travel, thereby increasing access to clients, collaborators, friends and audiences across the planet. This reduction in cost and greater access to global talent is of benefit to budgets - which assures that video conferencing is commercially backed to continue improving. 

Where is the concept of interactive digital meetings at right now do you think? How far away is it from being a common concept?

Interactive digital meetings already exist: some folks are having meetings in games, anyone with a headset has probably also at some point had a meeting in VR. The hurdle I see here is translating the interactive component to clear or measurable benefits to a company. What shared interactions are valuable to the meeting outcome that can’t be achieved otherwise? I expect interactive components will creep into meetings somewhat in disguise at first, as platforms test out what users & businesses are interested in gaining from increased functionality. 

Do you know of any companies already actively working in this space? One we're aware of is spatial.io

Social VR platforms do this, and you don’t necessarily need a headset to attend - some host a parallel web VR interface such as Mozilla Hubs. Loomie is in the business of providing you with an avatar to do your talking for you. Pragli is one to watch. There are also smaller seeming interactive components such as Snap camera app, Facebook Portal, Zoom backgrounds

The definition of what makes a meeting digital and interactive is a little fluid - there’s personal interaction e.g. facial AR, the sharing of content could be deemed interactive eg. sharing an asset that everyone can view at the same time in their own environment. The holy grail perhaps being the collaborative manipulation of visuals in real time, without a chunky headset. 

Any parting thoughts?

Chunky headset leads me to the aforementioned motivator - physical comfort. We continue to focus our curiosity on the MR space - a real technology battle point, as we see companies struggle to develop a wearable that is desirable and functional. Our hearts ever stuck on the vision of Tom Cruise whipping blue screens of HUD graphics with the swoosh of his hands. If it burns your forehead, strains your neck, restricts your vision to a tiny field of view, exhausts your brain… it’s not going to replace a computing interface that is ergonomic. Where there is an amount of physical discomfort, the wearable will continue to struggle.

Similarly for video conferencing - we are now familiar with “zoom fatigue”. It’s not natural to maintain eye contact with a group for hours on end, likely unblinking into your screen. I look forward to better UX design and camera tech that allows me to be present in a meeting, without needing to express my concentration via an unwavering stare. All in all, the VC, XR and virtual production industries have seen unprecedented uptake and development thanks to quarantine. It’s a very exciting time for companies and designers working in these spaces, there is no doubt we will continue to see rapid development in immersive online meetings.