Building a location app with ARKit, CoreLocation and Pusher

This is story is brought to you by Hacker Noon’s weekly sponsor, Pusher, who provides real-time APIs for developers.

Augmented Reality (AR) has a lot of interesting and practical use cases. One of them is location.

With iOS 11, the ability to use ARKit to create AR apps and combine them with multiple libraries has opened a lot of possibilities.

In this tutorial, we’re going to combine the power of ARKit, CoreLocation, and Pusher to create a geolocation AR app.

Let’s think of a taxi service. Some services allow you to track on a map the car that is going to pick you up, but wouldn’t be great to have an AR view to see the route of the car and how it gets closer to you?

Something like this:

As you can see, the information to position the car in the AR world is not always accurate, both on the CoreLocation side and on the ARKit side, however, for this use case, most of the time it will be enough.

Here’s what you’ll need:

• A device with an A9 or later processor (iPhone 6s or better, iPhone SE, any iPad Pro, or the 2017 iPad)
• iOS 11
• Xcode 9.1 (or newer)
• A 3D model of a car (in DAE format)

You can find free 3D models on sites like Free3D, Turbosquid, or Google’s Poly.

The most common format is OBJ (with the its textures defined in a MTL file), which can be converted to DAE with a program like Blender.

For this project I chose this model, which it’s available in DAE format.

The math for this project is a bit heavy. I’ll dedicate more time to explain the operations related to geolocation than the ones related to rotating and translating a model with ARKit.

If you don’t know about transformation matrices or how to convert your 3D model to the DAE format, take a look at my previous tutorial about ARKit.

For reference, the source code of this project is on GitHub.

Let’s start by setting up a Pusher app.

Setting up Pusher

If you haven’t already, create a free account at Pusher. Then, go to your Dashboard and create an app, choosing a name, the cluster closest to your location, and iOS as your front-end technology:

This will give you some sample code to get started:

Save your app id, key, secret and cluster values. We’ll need them later.

Finally, go to the App Setting tab, check the option Enable client events and click on Update:

Through this app, the drivers will send their locations as latitude/longitude coordinates along with the direction they’re heading (in degrees) as a client event.

But let’s not get ahead of ourselves, let’s set up the Xcode project first.

Setting up the project

Open Xcode 9 and create a new Single View App:

We’re choosing this option because we are going to manually set up an AR view along with other controls.

Enter the project information, choosing Swift as the language:

Create the project and close it. We’re going to use CocoaPods to install the project’s dependencies. Open a terminal window, go to the root directory of your project and, in case you don’t have CocoaPods installed (or if you want to update it), execute:

sudo gem install cocoapods

Once installed, create the file Podfile with the command:

pod init

Edit this file to set the platform to iOS 11 and add the Pusher’s Swift library as a dependency of the project:

# Uncomment the next line to define a global platform for your project

platform :ios, '11.0'

target 'ARKitCarGeolocation' do

# Comment the next line if you're not using Swift and don't want to use dynamic frameworks

use_frameworks!

# Pods for ARKitCarGeolocation

pod 'PusherSwift', '~> 5.1.1'

end

Once you’ve edited the Podfile, execute the following command to install the dependency:

pod install

In case version 5.1.1 (or later) is not installed (the output of the installation will tell you the installed version), you can update your CocoaPod repository and install the latest version of the library with the command:

pod install --repo-update

Now open the Xcode workspace instead of the project file. The workspace has the dependency already configured:

open ARKitCarGeolocation.xcworkspace

If you build your project at this point, a couple of warnings may show up, but the operation should be successful.

Next, select the file Info.plist, add a row of type Privacy — Camera Usage Description (NCameraUsageDescription) and give it a description. This is required for ARKit to access the camera.

We’ll also need a row of type Privacy — Location When In Use Usage Description (NSLocationWhenInUseUsageDescription) This is required to get the location from your device’s GPS (only when the app is being used, not all the time):

Finally, configure a team so you can run the app on your device:

Now let’s build the user interface.

Building the user interface

Go to Main.storyboard and drag an ARKit SceneKit View to the view:

Next, add constraints to all sides of this view so that it fills the entire screen. You do this by pressing the ctrlkey while dragging a line from the ARSCNView to each side of the parent view and choosing leading, top, trailing, and bottom to the superview, with a value of 0:

Next, add a text view and disable its Editable and Selectable behaviors in the Attributes inspector:

Change its background color (I chose a white color with 50%opacity):

Add a height constraint with a value of 90and leading, top, and trailing constraints with the value 0so it remains fixed to the top of the screen:

In ViewController.swift import ARKit:

import ARKit

Then, create two IBOutlets, one to the scene view and another one to the text view:

You’re ready to start coding the app, but before that, let me explain what needs to be done. However, if you’re already familiar with geolocation concepts or if you’re not interested, feel free to skip the next section.

Understanding how the app works

Imagine you are standing at some point in the world. It doesn’t matter where or in what direction you’re looking at.

Your location is given by two numbers, latitude and longitude.

Latitude is the distance between the North or the South Pole and the equator (an imaginary circle around the Earth halfway between the poles). It goes from 0ºto 90ºfor places to the north of the equator, and 0ºto -90ºfor places to the south of the equator.

Longitude is the distance from the prime meridian (an imaginary line running from north to south through Greenwich, England) to a point at the west or east. It goes from 0to 180º for places to the east of the prime meridian, and 0º to -180ºfor places to the west of the prime meridian.

For example, if you’re in Brazil, your latitude and longitude will be negative because you are on the southwest side of the Earth:

And if you’re in Japan, for example, your latitude and longitude will be positive because you are on the northeast side of the Earth:

This app will take into account your position and the driver’s position in a latitude and longitude coordinate system:

But if it’s easier to you, you can think of your position as the origin (0, 0):

You need to calculate two things:

• The distance between you and the driver
• The angle between the north (or south) line of the Earth and the line connecting you and the driver, which is called bearing.

The distance will tell you how far you have to position the 3D model in the AR world.

The bearing will help you create a rotation transformation to position your model in the right direction at the above distance.

If we were talking about a simple x and y coordinate system, we could get those calculations by applying the Pythagorean theorem and some simple trigonometry, with sine and cosine operations.

But we are talking about latitudes and longitudes of the Earth. And as the Earth is not a flat plane, the math gets more complex.

The distance is calculated by calling just a method of the class CLLocation. It uses the Haversine Formula which, from two different latitude/longitude pairs of values, calculates the distance by tracing a line between them that follows the curvature of the Earth.

On the other hand, we have to calculate the bearing between two different latitude/longitude pairs of values manually. This is the formula:

atan2 ( X, Y )

Where X equals:

sin(long2 - long1) * cos(long2)

And Y equals:

cos(lat1) * sin(lat2) - sin(lat1) * cos(lat2) * cos(long2 - long1)π

Another thing to consider is that for the matrix transformation, you’ll have to use radians instead of degrees as angle units. As the length of an entire circumference is equal to 2π radians (360º), one radian is equal to 180/π degrees.

So this is the plan.

Using Pusher, the drivers will publish their location and direction they’re heading in realtime.

Using CoreLocation, the AR app is going to get your location. It will also listen to the driver’s location updates.

When a location update is received, using the formulas explained above, the app will place a 3D model of a car in a position relative to your location inside the AR world, and it will orient the model to the same direction the driver is heading.

The app is only going to get your location once, so it assumes your location is fixed (which is true most of the time).

In addition, an arrow emoji (⬇️) will be shown on top of the model at all times so you can spot it easily, and the text view you added in the last section will show the status of the app and the distance between you and the car.

Now that you know what to do, let’s get into the code.

Building the app with ARKit and CoreLocation

Let’s start by defining two extensions.

One to provide conversion methods to radians and degrees to all floating point types. Create a new Swift file, FloatingPoint+Extension.swift, with the following content:

And another extension to create an image from a string. Create another Swift file, String+Extension.swift, with the following content (taken from this StackOverflow answer):

You’ll use this extension to create an image out of the arrow emoji (a string). It creates a rectangle of width 100 and height 100 , with a transparent background, to draw the string inside of it with a font size of 90.

Next, open the New File dialog and scroll down to choose the Asset Catalog type:

Enter art.scnassets as the file name (confirming the use of the extension scnassets):

Now copy your model to this folder:

Open the Scene Graph View, select the main node of your model and, in the properties tab, give it a name, which you’ll use to reference it in the code:

Back to ViewController.swift, let’s add the import statements we’ll need:

And the delegates the controller will use:

Next, let’s add some instance variables.

First, a CLLocationManager to request the user location and another variable to store it:

Then, a variable to store the direction the drivers are heading, the distance between them and the user, and the status of the app:

Whenever a new value for the distance or the status is set, the text view will be updated. Notice that the distance is calculated in meters.

Next, a variable to store the root node of the car model and the name of this node, which should be the same than the one you set at the SceneKit editor:

You’ll also need the original (first) transformation of that node:

Why?

To calculate the orientation (rotation) of the model in the best possible way.

Ideally, the driver’s device will always give you the correct heading so you can take the first received reading, rotate the model in that direction, and then calculate the next rotations relative to the first one.

However, if the first reading is wrong (which happens sometimes), the next rotations will be wrong even if the rest of the readings are correct.

So you always need to calculate the orientation as if it was the first time you rotate the model, because once you rotate the model a certain angle the following rotations will be done relative to that angle. Resetting the rotation to 0º won’t work either because of the way transformations work (matrix multiplication).

Finally, you’ll need to store the Pusher object and channel to receive the updates:

Notice the value of the authMethod option.

You’ll be receiving the updates through a private channel. They need to be authenticated by a server. However, at development time, you can use the inline option to bypass the need to set up an auth endpoint as part of a server.

You can learn more about the object’s options here. If you need it, you can learn how to create an authentication endpoint on this page.

In the viewDidLoad function, set up the SceneKit scene and the location service:

Next, configure the AR session:

The option gravityAndHeading will set the y-axis to the direction of gravity as detected by the device, and the x- and z-axes to the longitude and latitude directions as measured by Location Services.

For the users position, when they have authorized the use of the location services, you have to request the location (the requestLocation method is used so the location is requested only once):

Once the user’s location is received, take the last element of the array, update the status, and connect to Pusher (it doesn’t make sense to connect to Pusher before having the users location because all the calculations will be wrong):

In the method connectToPusher you subscribe to private-channel and, when a client-new-location event is received, extract the driver’s latitude, longitude, and heading and update the status and location of the 3D model with the method updateLocation:

In updateLocation, create a CLLocation object to calculate the distance between the user and the driver. Remember that the distance is calculated in meters:

If this is the first update received, self.modelNode will be nil, so you have to instantiate the model:

Next, you need to move the pivot of the model to its center in the y-axis, so it can be rotated without changing its position:

Save the model’s transform to calculate future rotations, position it, and add it to the scene:

Notice that there’s no need to create an ARAnchor to add the node as a child of it. An ARAnchor gives you the ability to track positions and orientations of models relative to the camera.

But in this case, it’s better to work with the child directly. Mostly because you cannot delete or change the position of the whole ARAnchor manually -only of its children.

Finally, create the arrow from an emoji, position it on top of the car (using the y-axis, I got the value by trial and error), and add it as a child of the model (so it stays with it at all times):

This is the definition of the makeBillboardNode method (taken from this StackOverflow answer, modifying the width and height of the plane so the arrow can be properly seen):

Now, if this is not the first update, you just need to position the model, animating the movement so it looks nice:

To position the model, you just need to rotate first, then translate it to the correct position and scale it:

The order is important because of how matrix multiplication works (a*b is not the same than b*a).

In ARKit, rotation in the y-axis is counterclockwise (and handled in radians), so we need to subtract 180º and make the angle negative. This is the definition of the method rotateNode:

I scale the node in proportion to the distance. They are inversely proportional -the greater the distance, the less the scale. In my case, I just divide 1000 by the distance and don’t allow the value to be less than 1.5 or great than 3:

I got these values from trial and error. They will vary depending on the model you’re using.

To translate the node, you have to calculate the transformation matrix and get the position values from that matrix (from its fourth column, referenced by a zero-based index):

To calculate the transformation matrix:

• You use an identity matrix (you don’t have to use the matrix of the camera or something like that, the position and orientation of the driver are independent of your position and orientation.
• You have to calculate the bearing using the formula explained in the previous section:

atan2 (

  sin(long2 - long1) * cos(long2),

  cos(lat1) * sin(lat2) - sin(lat1) * cos(lat2) * cos(long2 - long1)

)

• Using an identity matrix, get a rotation matrix in the y-axis using that bearing.
• The distance is given by the z-axis, so create a four element vector with the distance in the z position to get a translation matrix.
• Multiply both matrices (remember, the order is important) to combine them.
• Get the final transformation by multiplying the result of the previous step with the matrix passed as an argument.

All this is done with the following methods:

About how to rotate in the y-axis, the method returns the inverse of the matrix because rotations in ARKit are counterclockwise. Here’s an answer from Mathematics Stack Exchange that explains rotation matrices pretty well.

And that’s it, time to test the app.

Testing the app

The first time you run the app, you’ll have to give permissions to the camera:

And to the location service:

And wait for a few seconds so the app can get the location and connect to Pusher.

To test it, you’ll need someone that publishes location events while driving.

On this GitHub repository, you can find an app for iOS that publishes location events.

It uses CoreLocation, and the code is pretty similar to the one shown in the previous section but it requests the location information every one or two seconds.

As a note, for the heading measurement, it’s important to hold the device in the direction the driver is heading.

For a quick test, you can use the following Node.js script to manually send some location coordinates (that you can get from this site) every two seconds:

Once you have Node.js installed, you just have to copy this script to a file, let’s say publish.js, create a package.json file with the command:

npm init

Install the Pusher Node.js library with:

npm install --save pusher

Enter your Pusher and location info and execute the script with:

node publish.js

Once the app starts receiving location events, the 3D model of the car will appear in the direction where it is in the real world (with a small size if it’s far from you):

Conclusion

You have learned how to combine the power of ARKit, CoreLocation and Pusher to create an AR app.

You can add more features to make it more useful:

• Adding more information to the screen. For example, you can convert the coordinates of the driver to an address.
• Adding a map so, in addition to seeing the 3D model moving in the world, you can see in which street the car is at any given time.
• Add more car models.
• Change the mechanism to get the car orientation. For example, by using deltas of the location.

However, keep in mind that the app depends on the quality of the information received.

In my tests, for a few seconds after starting the driver’s app, the heading information was completely wrong, and overall, the position was off a few meters.

ARKit occasionally gets confused too. Sometimes this can be a problem, and it is another area of improvement. However, we’re just at the beginning. Without a doubt, these frameworks will be improved over time.

Remember that you can find the entire project on this GitHub repository and you can contact me if you have questions.

This is story is brought to you by Hacker Noon’s weekly sponsor, Pusher, who provides real-time APIs for developers.Originally published on Pusher’s blog.

Building a location app with ARKit, CoreLocation and Pusher was originally published in Hacker Noon on Medium, where people are continuing the conversation by highlighting and responding to this story.