The Evolution of Mixed Reality Avatars

Update: MagicLeap announced a virtual character demo:

Extended Reality will breath fresh air into the stale experience of using mobile apps. One of the most popular app genres — social apps — will be the first to be changed by this paradigm shift. We can see hints of the upcoming changes today. For instance Instagram already has virtual humans which attract over a million followers: Lil Mequela

In China there are online celebrities who make over $10,000,000 a year broadcasting their lives via handheld cameras to paying subscribers. These streams will soon be popping out of the tiny restrictions of the mobile screen and appearing as holograms, dancing in your living room. The stage is then set to make take these mass broadcasts and make them individually tailored to each paying subscriber — with each performance being customised by A.I.

The use of Extended Reality together with A.I. mixes together to make a brilliant new show, broadcast 24/7 with you as the star, surrounded by your favorite celebrities.

Welcome to the future of TV.

Japan is firmly ahead in this future with the fictional character, Hatsune Miku, already being adored by fans and appearing all around the world in a holographic form:

The target audience of these Mixed Reality broadcasts is bypassing the classical Otaku — the geek who sits alone and only interacts with girls in computer games — to turn the heads of all youtube viewers of all ages. The future of celebrity broadcasting, and hence influencer marketing, will be a customised, ultra relevant, personal stream.

For example we can imagine youtube celebs like a virtual Beyonce coming into your house, sitting on your sofa and looking across to talk to you, asking you details about your day. This scenario has been cleverly explored in the recent film “Her”.

It will create a new form of broadcast entertainment where the stage is your living room and the virtualized actors are communicating with you in your intimate spaces. The first stage of this will be fixed hologram recordings, locked to a spot in your house via Augmented Reality.For example we can imagine youtube celebs like a virtual Beyonce coming into your house, sitting on your sofa and looking across to talk to you.The paradigm of entertainment will shift from being a distraction to being a companion.

Entertainment will shift from being a distraction to being a companion. Virtual Avatars are already capable of fantastic emotional expression ( note the virtual actor below is being driven by a human performance):

When these lifelike avatars are driven by A.I. the show will never really end. And at the end of each day the bell d’jour will be in your bedroom acting our bedroom stuff:

This next stage of streaming — where A.I is being used to manipulate generative characters will be the most interesting. This kind of entertainment will be created in your house and broadcast out of it.

The celebrity will supply the avatar and then each fan will enjoy a unique experience, depending on their environment and individual reactions. Because of this everyone will be constantly streaming experiences with their favourite characters incase something unique happens.

This kind of bi-directional mass media has never been seen before and is perfectly suited to the Mixed Reality platform. It could soon become the new way of influencer marketing — with popular stars being used to sell everything on a very personal level.

Today we already get glimpses of this future in developer demos:

It is not crazy to think in a couple of years people will live alongside these virtual avatars, using them for entertainment, companionship and more.

Combining A.I. Avatars with an algorithmic gaze will allow them to understand the context of beds, chairs, doors and help them live convincingly alongside their loyal fans.

Magic Leap seems to be at the forefront of this idea with hardware and software available to the public to make this happen today:

Amazon also recognised the importance of characters in the MR space and has pushed ahead with its Sumerian Project:

Although still young, Amazon’s Sumerian paves the way for virtual actors that can appear in VR or AR spaces and can be communicated via Alexa’s natural spoken language.

These Mixed Reality Avatars will have a view of the same room you are looking at ( either via your mobile camera, or via another camera you might install in your home for permanent A.I., which Amazon also sells): DEEPLENS

This algorithmic gaze will give virtual avatars total contextual understanding of your home and life. It is likely that these ‘bot-eye’ cameras will be installed in every room in every home for your personal AR avatars to see through. Amazon is ahead here again, as it already sells Alexa units of varying sizes — intended one for each room. It’s not beyond the imagination to assume users will be happy buying Echo Dots with cameras on for the use of home avatar systems.

For example you could put an A.I. camera in your kitchen, to watch the fridge and stove, and another by your front door to watch you come in. This would be enough for the AI driven M.R. avatar to know what you have in left in your fridge, to remind you to do the shopping, to watch what you have eaten, to tease you about your diet and even teach you to cook. And it can watch the stove whilst you are out of the room. It should do all this naturally. When the kettle is boiling the algorithmic gaze will see the steam, and the Avatar will notify me.

An A.I. camera placed on your front door will ensure your friendly avatar is always there to greet you will a big smile when you come back home. Maybe these avatars will take the shape of dogs, cats, or branded cartoon characters.

And lets not forget there are funded companies working on Mixed Reality technology that does not require the user to wear or hold any devices to see this reality, holographic projections are possible: Lightfields

Mixed Reality, with the combination of Computer Vision, A.I. and digital overlays will allow everyday human behaviours to be offloaded from your mind to another system without needing complex infrastructure to connect devices or expensive robotics to interact with humans and systems. In the near future it is very likely we will live and work alongside intelligent robots, but only in the virtual sense.

Next consider that these M.R. avatars could not only be celebrities — they could be whoever you fancy, or they could be long distance friends, deceased relatives or even clones of yourself. This technology is here today, nascent, and possible. It is one of the areas Ninjar is exploring as future use cases for M.R.

We believe the utilisation of M.R. Avatars will be for more than just entertainment, they will be customer service representatives and skilled tradesman too.

For example if you want to see how a new BMW looks in your driveway, you can just place one there via AR. ( We’ve done plenty of work on this, as you can see here: ninjar.com/work )

But A.R. is not the best sales pitch by itself, not compared to the experience of visiting a showroom and having a brand employee pitch emotionally to you about the upgrade.

M.R. avatars will become virtual salespeople.

Part of our vision at ninjar.com is that when you drop an A.R. car on your driveway you should expect a M.R. salesperson to come along for the ride. They will know what you are looking at and respond correctly — if you move your phone for a closeup of the alloys you can expect them to say “Hey, Good choice on those wheels — I can give them to you for a free upgrade…”

One argument we hear to the idea of Mixed Reality avatars is “But they can’t do anything real — they can’t touch anything… They can’t pick up a hammer and knock down a wall!”.

This is true, but then again — when was the last time you picked up a hammer or used any tool to permanently change the real world? For most of us we live in world where the physical manipulation of hardware is outdated or automated. Food is delivered pre-cooked and in disposable trays. Apartments come serviced and need little maintenance. Home based mechanical repair is a thing of the past, replaced with planned obsolescence, appliance trade-ins and annual upgrade plans. It is almost like modern life has left a void in the homes where real world tools, and robots, are no longer essential.

Conclusion

Mixed Reality avatars, combined with ubiquitous A.I. cameras, might be our main access to computation in the near future. They will see what we see, yet they will have the superhuman intelligence, memory and connections from the cloud. They will become so useful we will miss them when they are not there. They will look astonishingly real:

Extra Notes…

But will this be an authentic experience that people will want? We’ve learnt from mobile apps and social networks that desire is easy to manipulate. Social App designers use a technique called F.O.M.O. — Fear of Missing Out — to drive retention. This psychological trick keeps us checking social feeds for fear we might miss a post from a friend. This hijacking of user’s minds to increase retention will be strengthened further by the natural behaviours of the realistic M.R. avatars and the consideration that they will be a channel for all our important messages, friendships and potential sexual opportunities.

For example, in the world of Mixed Reality avatars, if I want to send you a message to ask you something, I can ‘send’ my avatar — my clone — over to your house to ask. And you can answer back and I will see your avatar, your clone sitting next to me. This natural interface might make today’s chat apps look ridiculously basic and is probably part of the reason why Facebook invested a hefty sum to buy Oculus. They realize the future of Social is MR.

This idea of a virtual space filled with believable avatars that hold and mediate the connections between us and the rest of humanity is and should be scary. This new interface will become the truth of our news, our friends, potential dates and all product sales and will become a powerful platform. The combination of an Information Technology, and a Social Network, along with a emotion display via realistic humans is uniquely powerful.

There are other, possibly negative, effects of surrounding yourself with MR avatars. Consider the fact that when your phone is near you, you become stupider. The findings of a study in Social Psychology publication found that the “mere presence” of a mobile — even when it is not being used — influenced simple mental performance. It was noted that when a user’s phone was in view the person did 20% worse at a given task. Imagine how lazy your mind would become when surrounded by A.I. driven avatars.

Did you enjoy our post? You can comment and join the chat on Medium

We’d love to hear from you. Come find us on Twitter , Facebook and LinkedIn

The Evolution of Mixed Reality Avatars was originally published in Hacker Noon on Medium, where people are continuing the conversation by highlighting and responding to this story.