I’m into the idea of augmented reality because it makes sense that computing is anchored to the real world.
Of course I should be able to pinch my fingers on a paragraph in a real printed book to copy-and-paste the text into my notes. Naturally I should be able to look out of my window and double-blink at my car, bringing up a readout of how much fuel I have and how much time is left on the lease. Or look up at the sky to see the weather forecast printed on a cloud. Or at my sourdough starter to see it ghosted against the size it was early today, so I can tell at a glance how much it has risen.
But augmented reality is always about glasses. See: the Magic Leap One A.R. goggles.
See Apple: the continuous rumours about Apple’s secret smart glasses project. Apple’s existing augmented reality platform which is always about looking through.
BUT.
Glasses are anti-social. I want to show people what I’m working on by having them look over my shoulder. I’ve lost count of the number of times I’ve shared a photo by holding my phone up to Zoom. When A. is cooking, she can call across from another room to have me check the stove – how bad it would be if the stove interface was locked away in her personal glasses. Sure, all computer interface problems are solvable with design and work – but A.R. is creating a lot of unnecessary work if it starts with an interface which is so oriented to the individual from day 1.
And then: how do we get there from here? I can imagine a world where augmented reality is the only interface – Keiichi Matsuda’s still-amazing-and-disconcerting 2016 film is such a vision: HYPER-REALITY.
Can I imagine HYPER-REALITY alongside today’s smartphones? Can I imagine taking glasses on and off just to check a document or two? Not really. But that’s the reality of technology adoption. Adoption and sophistication proceed stepwise.
So what’s the first step into mainstream augmented reality? Perhaps: not glasses.
Here’s KOSKI GAME (2016) by Václav Mlynář/Studio Deform:
KOSKI is a mixed-reality building block game. It is a combination of real, wooden blocks and a virtual app that facilitates digital, interactive game play. With the help of a tablet or smartphone, structures assembled from the blocks come alive. Once a player starts to interact with the blocks, the game begins to reveal it’s hidden worlds, characters, and stories.
Watch the KOSKI intro video (1m23s).
It’s a physical sandbox game for kids. You stack real world wooden blocks. Next to the game area, you prop up your iPad.
The iPad is a magic mirror.
As you stack the blocks, they comes alive in the magic mirror. Figures climb the structure; waterfalls appear.
What struck me, using KOSKI, is how natural it felt. When you’re playing with the blocks, and looking at the augmented reflection in the screen facing you, your neuroplasticity takes over in an instance. It’s as if the blocks in the hands actually have trees growing out of them and tiny cartoon people are balanced there.
We shouldn’t be surprised. When I’m selecting text on my screen, I’m not conscious of my fingers being on the trackpad and the pixels being many inches away. The two feel identical.
(The designer created this game while part of Platform 24, Design Products at the Royal College of Art, while I was also lending a hand. It’s stuck in my head since.)
What if the magic mirror is the right way to start with augmented reality?
Not looking through, which presupposes an individual perspective, whether you’re looking through glasses or looking through a phone.
But instead reflecting what’s between you, as if the magic mirror screen is part of your team, or part of a small group of kids playing, just another group member adding their point of view.
For kids, it’s way more social. Kids already play together. It makes sense to have have just another party saying (visually; nonverbally) hey, let’s have a waterwall here, and hey what if that figure needed a bridge – and the other kids are free to play along or to do something else. (A.R. glasses don’t feel nearly as nuanced.)
For me, individually, I can imagine propping up an iPad on my desk, off to one side, and working on a printed document. When I make edits on the doc, I glance to left to see my work reflected in the tablet screen, and translated into Google Docs edits. The mirror is populated with team-mates adding their comments – checking them is just like looking into the sidebar.
And let’s pretend that one day we’re back in offices…
Abstractly, the magic mirror is a screen that looks back when you’re looking at it, and can intelligently add to whatever it displays.
So, building on that: instead of screen sharing to a room projector, why not hold my smartphone up to the magic mirror, which then captures and magnifies whichever doc I had on my display, showing it to everyone in the room? Simple steps.
A couple of speculative form factors:
- this transparent 55” OLED TV but as a desktop monitor – imagine a regular desktop interface but where your desktop wallpaper is see-through. In magic mirror mode, I would work on paper or my tablet at my desk, and only the reflected, augmented documents would appear: it would feel like sitting opposite someone who was co-editing my work.
- a mirror for the kitchen: I would show it recipes to convert units, and look into it to see my calendar schedule for the day (and my partner would see too). I find this idea more ambient, more natural, and less intrusive than a smart speaker… internet-connected camera aside.
The metaphor here is that augmented reality doesn’t have to feel like a cyborg enhancement. It can feel like a companion, or friend, or team member.
I’m not saying never glasses. With magic mirrors I’m saying perhaps – also? Or, first? Or, let’s try it because the technical barriers are lower and the use cases for immediate? That’s all.
For my own curiosity, I’d be interested to know if anybody is pursuing the magic mirror paradigm for augmented reality – do let me know if you’re working on it.
I’m into the idea of augmented reality because it makes sense that computing is anchored to the real world.
Of course I should be able to pinch my fingers on a paragraph in a real printed book to copy-and-paste the text into my notes. Naturally I should be able to look out of my window and double-blink at my car, bringing up a readout of how much fuel I have and how much time is left on the lease. Or look up at the sky to see the weather forecast printed on a cloud. Or at my sourdough starter to see it ghosted against the size it was early today, so I can tell at a glance how much it has risen.
But augmented reality is always about glasses. See: the Magic Leap One A.R. goggles.
See Apple: the continuous rumours about Apple’s secret smart glasses project. Apple’s existing augmented reality platform which is always about looking through.
BUT.
Glasses are anti-social. I want to show people what I’m working on by having them look over my shoulder. I’ve lost count of the number of times I’ve shared a photo by holding my phone up to Zoom. When A. is cooking, she can call across from another room to have me check the stove – how bad it would be if the stove interface was locked away in her personal glasses. Sure, all computer interface problems are solvable with design and work – but A.R. is creating a lot of unnecessary work if it starts with an interface which is so oriented to the individual from day 1.
And then: how do we get there from here? I can imagine a world where augmented reality is the only interface – Keiichi Matsuda’s still-amazing-and-disconcerting 2016 film is such a vision: HYPER-REALITY.
Can I imagine HYPER-REALITY alongside today’s smartphones? Can I imagine taking glasses on and off just to check a document or two? Not really. But that’s the reality of technology adoption. Adoption and sophistication proceed stepwise.
So what’s the first step into mainstream augmented reality? Perhaps: not glasses.
Here’s KOSKI GAME (2016) by Václav Mlynář/Studio Deform:
Watch the KOSKI intro video (1m23s).
It’s a physical sandbox game for kids. You stack real world wooden blocks. Next to the game area, you prop up your iPad.
The iPad is a magic mirror.
As you stack the blocks, they comes alive in the magic mirror. Figures climb the structure; waterfalls appear.
What struck me, using KOSKI, is how natural it felt. When you’re playing with the blocks, and looking at the augmented reflection in the screen facing you, your neuroplasticity takes over in an instance. It’s as if the blocks in the hands actually have trees growing out of them and tiny cartoon people are balanced there.
We shouldn’t be surprised. When I’m selecting text on my screen, I’m not conscious of my fingers being on the trackpad and the pixels being many inches away. The two feel identical.
(The designer created this game while part of Platform 24, Design Products at the Royal College of Art, while I was also lending a hand. It’s stuck in my head since.)
What if the magic mirror is the right way to start with augmented reality?
Not looking through, which presupposes an individual perspective, whether you’re looking through glasses or looking through a phone.
But instead reflecting what’s between you, as if the magic mirror screen is part of your team, or part of a small group of kids playing, just another group member adding their point of view.
For kids, it’s way more social. Kids already play together. It makes sense to have have just another party saying (visually; nonverbally) hey, let’s have a waterwall here, and hey what if that figure needed a bridge – and the other kids are free to play along or to do something else. (A.R. glasses don’t feel nearly as nuanced.)
For me, individually, I can imagine propping up an iPad on my desk, off to one side, and working on a printed document. When I make edits on the doc, I glance to left to see my work reflected in the tablet screen, and translated into Google Docs edits. The mirror is populated with team-mates adding their comments – checking them is just like looking into the sidebar.
And let’s pretend that one day we’re back in offices…
Abstractly, the magic mirror is a screen that looks back when you’re looking at it, and can intelligently add to whatever it displays.
So, building on that: instead of screen sharing to a room projector, why not hold my smartphone up to the magic mirror, which then captures and magnifies whichever doc I had on my display, showing it to everyone in the room? Simple steps.
A couple of speculative form factors:
The metaphor here is that augmented reality doesn’t have to feel like a cyborg enhancement. It can feel like a companion, or friend, or team member.
I’m not saying never glasses. With magic mirrors I’m saying perhaps – also? Or, first? Or, let’s try it because the technical barriers are lower and the use cases for immediate? That’s all.
For my own curiosity, I’d be interested to know if anybody is pursuing the magic mirror paradigm for augmented reality – do let me know if you’re working on it.