Augmented Reality in React Native
As I am sure you are aware, both iOS and Android have released their respective AR SDKs which are very powerful but, being native, would require 2 separate apps. We wanted to see what was possible sharing a codebase and using React Native. Luckily for us, Viromedia (https://viromedia.com/viroreact/) have released a wrapper for AR and VR that provides a single unified API to interact and build for both platforms. For free. Huge win!
What Wikipedia has to say about AR:
Augmented reality (AR) is an interactive experience of a real-world environment whereby the objects that reside in the real-world are "augmented" by computer-generated perceptual information, sometimes across multiple sensory modalities, including visual, auditory, haptic, somatosensory, and olfactory. The overlaid sensory information can be constructive (i.e. additive to the natural environment) or destructive (i.e. masking of the natural environment) and is seamlessly interwoven with the physical world such that it is perceived as an immersive aspect of the real environment.
We both started off running through the "getting started" exercises and a simple tutorial to get a feel for how it all works and then would set about creating a proof of concept app.
ViroMedia provides an app called TestBed. This is supposed to help the process by allowing you to connect your code via a ngrok proxy. I had limited success with this and found it easier to build the app in the usual way. George had the opposite experience. This may be down to platform as George was working on iOS and I was working on Android.
The tutorial introduces the key parts of the SDK and showed how easy it was to get started. In the code snippet below you can see that in just 54 lines of code you can have a working (but very simple) AR scene running on both platforms.
The POC:
The idea behind the app was that a user could point their device at an image maker (a sign etc.) that would prompt the device to show a 3d animated character with some sound or a video. We also had to consider the fact that some people may not have the most up-to-date devices and that we must provide a fall back for these devices.
We quickly found out that the imageMarker API that allowed you to show an animation on showing the image on the screen worked well on iOS but not so well on Android. This was not an issue with ViroMedia's software, but a limitation on the ARCore SDK provided by Android. If this was improved in the future it would permit the image marker as a usable option. But for now, we had to park that and look to other means of dynamically displaying the animation. Another quirk of Android ARCore is that they disable autofocus when in an ARView which is totally understandable, but can make for a fuzzy screen.
I remembered that we had a box of iBeacons from a few years ago stashed in our server room. We wondered if it would be possible to use these in place of the image markers, so George set about investigating this. In the meantime, I started adding a new AR feature to our proof of concept.
The Portal:
This would allow us to display videos in a portal, like a window. The portal SDK itself was relatively simple but trying to scale the video inside the portal proved too time-consuming. We were in the afternoon of the second day and wanted to have something to show, so I stopped tinkering in the knowledge that perfection could be found at a later date.
Back to the beacons. It turns out that the coin cell batteries in the beacons don't last as long - they had been sat in the server room and gone flat. I was able to download an app that would mimic the functionality of the beacon and the code George had written detected it perfectly. There was one problem though, the memory usage was through the roof.
This was causing consistent fatal errors due to memory exhaustion, and likely due to the already memory-taxing processing of an AR environment. Another idea that could be explored in the future with more time on our side. But this needed to be parked for now.
Other options for triggering the animations
GPS:
Having worked with mobile GPS before, we knew that this may be an option for triggering the AR interactions but were also aware that there were limitations that would likely hinder the experience.
QR Codes:
Although QR codes were the least elegant solution to our problem, it would seem they will be the best. They will be the most reliable and will work on all devices
By the end of the 2 days of research, we were confident that ViroMedia would suit our needs and allow us to write a full, complex AR app experience using the same code base across both platforms.
