Augmented Reality allows developers to create new engaging experiences, projecting digital objects in a real-world environment. Apple has contributed to the development of this amazing technology by releasing powerful frameworks like RealityKit, which has the capability to render 3D graphics in real time on the most recent Apple devices.

In this tutorial, you’ll understand how RealityKit works behind the scenes and how to create a beautiful app in SwiftUI that implements RealityKit.

RealityKit vs ARKit

Diving into the world of augmented reality can be a little confusing for beginners since Apple has different technologies available for this purpose. ARKit and RealityKit are the main frameworks for building amazing AR experiences, and while they work seamlessly together, they have two different roles:

RealityKit provides high-performance 3D simulation and rendering capabilities you can use to create visionOS apps or to create augmented reality (AR) apps for iOS, macOS, and tvOS. RealityKit is an AR-first 3D framework that leverages ARKit to seamlessly integrate virtual objects into the real world.

Apple Documentation

ARKit integrates hardware sensing features to produce augmented reality apps and games combining device motion tracking, world tracking, scene understanding, and display conveniences to simplify building an AR experience.

Apple Documentation

To understand better the workflow behind these two frameworks, let’s consider the process of displaying a 3D asset on a surface, like a table:

1. ARKit is responsible for tracking the device’s position and orientation in the real world and detecting features like surfaces and objects.
2. ARKit provides information about the environment, such as camera images, depth information, and tracking status.
3. RealityKit uses this information to create and update the AR scene. It places virtual entities in the AR space, aligning them with the detected real-world features.

The USDZ format

USDZ is a file format developed by Apple in collaboration with Pixar, designed specifically for 3D augmented reality experiences. It is crucial for AR applications because it allows developers to efficiently store and exchange 3D assets, animations, and scenes in a compact, standardized format that is optimized for mobile AR performance. When you develop your AR app, remember to adopt this format for your 3D assets. Fortunately, Apple provides a powerful tool to easily convert your assets in .usdz: Reality Converter.

USDZ schemas for AR | Apple Developer Documentation

Add augmented reality functionality to your 3D content using USDZ schemas.

Apple Developer Documentation

Building your first AR app

By the end of this tutorial, you'll have an augmented reality application that places a 3D model of an airplane right on your desk. It's a fun way to explore the possibilities of AR development, even if you're new to it.

Before you start

Before starting to develop the app we need to download and import the model into our project and ask the user to allow our application to have access to the camera.

Start creating a new Xcode project selecting SwiftUI.

Download the model

First of all, we need to download our .usdzmodels model. Fortunately, Apple has a list of free models that you can easily download here. For this project, we will use the model called toy_biplane_idle.usdzand don’t forget to also download the image linked to the model and move it into the Assets folder (we will need it for our basic UI).

1. Download the Toy Biplane Idle model from the 3D models section in the augmented reality quick look gallery and save the image of the model from the website (you can right-click and download the image with most browsers)
2. Import the model into Xcode by dragging and dropping it into your project's
3. Once imported, check the option “Copy items if needed”
4. Add the image of the Toy Biplane model to your Assets folder.

Privacy Options

When developing an app that uses APIs requiring camera access, such as ARKit for augmented reality experiences, it is mandatory to add a description for the request for authorization to access the camera into the Info.plist file of your project.

This step is essential because you must ensure that users are informed and have control over how their device's camera is used.

1. Go into your project settings and navigate to the Info tab, as part of your project’s target
2. Add a new Key in the Custom iOS Target Properties: Privacy - Camera Usage Description
3. Add a string value describing why the app needs access to the camera

Creating the Application

In this project we will need three views:

• The ARViewContainer to set all the things to get RealityKit to work properly.
• A SheetView that will display the device’s camera
• The ContentView for displaying a basic UI

ARViewContainer

This view conforms to the UIViewRepresentable protocol. This means that what we are doing is embedding a UIView object from UIKit (in this case, an ARViewconfiguring from RealityKit) into a SwiftUI view.

// 1.
import SwiftUI
import RealityKit

// 2.
struct ARViewContainer : UIViewRepresentable {
    // 3. 
    @Binding var modelName: String

    // 4. 
    func makeUIView(context: Context) -> ARView {
        return ARView()
    }

    // 5.
    func updateUIView(_ uiView: ARView, context: Context) {}
}

Create a new Swift file and name it ARViewContainer.swift. In the file:

1. Import the SwiftUI and the RealityKit frameworks
2. Create a new structure called ARViewContainer that conforms to the UIViewRepresentable protocol
3. Create a binding variable property called modelName to store the name of the 3D model that will be displayed in the ARView.
4. Create the makeUIView(context:) method, responsible for creating a view object and configuring its initial state.
5. Create the updateUIView(_:context:) method, which will update the state of the specified view with new information from SwiftUI.

The next step is creating the view that will provide us the augmented reality capabilities and present the 3D model. We will do it in the makeUIView(context:) method, instantiating a new ARView used to display the rendered 3D model to the user.

import SwiftUI
import RealityKit
// 1.
import ARKit

struct ARViewContainer : UIViewRepresentable {
    @Binding var modelName: String
    
    func makeUIView(context: Context) -> ARView {
        // 2.
        let arView = ARView(frame: .zero)
        
        // 3.
        let config = ARWorldTrackingConfiguration()
				// 3.a
        config.planeDetection = [.horizontal,.vertical]
				// 3.b
        config.environmentTexturing = .automatic
        
        // 4.
        arView.session.run(config)
        return arView
    }
    
    func updateUIView(_ uiView: ARView, context: Context) {}
}

1. Import the ARKit framework, necessary for tracking the real world with the device sensors
2. Create an ARView object to display the rendered 3D model to the user
3. Create a ARWorldTrackingConfiguration configuration object to set up how to track the real world. Then we need to set two properties:
   1. The .planeDetection property in order to detect flat surfaces such as tables or walls in the camera-captured image
   2. The .environmentTexturing property for providing realistic image-based lighting for virtual objects.
4. Configure the arView instance with the config object we just created. This sets up the AR environment with the specified configuration

Now you need to load the model downloaded before and tell RealityKit how to place it on a surface.

import SwiftUI
import RealityKit
import ARKit

struct ARViewContainer : UIViewRepresentable {
    @Binding var modelName: String
    
    func makeUIView(context: Context) -> ARView {
        let arView = ARView(frame: .zero)
        
        let config = ARWorldTrackingConfiguration()
        config.planeDetection = [.horizontal,.vertical]
        config.environmentTexturing = .automatic
        
        arView.session.run(config)
        return arView
    }
    
    
    func updateUIView(_ uiView: ARView, context: Context) {
        // 1.
        let anchorEntity = AnchorEntity(plane: .any)
        
        // 2.
        guard let modelEntity = try? Entity.loadModel(named: modelName) else { return }
        
        // 3.
        anchorEntity.addChild(modelEntity)
        
        // 4.
        uiView.scene.addAnchor(anchorEntity)
    }
}

1. Create an instance of the AnchorEntity object setting up the plane property to .any to detect any type of plane in the environment. Anchors tell RealityKit how to pin virtual content to real-world objects
2. Load the specified model from the app’s assets using the modelName variable
3. Add the loaded model as a child to the anchorEntity instance
4. Add to the scene the anchor which will cause the 3D model to be displayed in the AR environment

Sheet View

The SheetView consists of two main components: the ARViewContainer, where the augmented reality magic happens, and a button for transitioning from the AR experience to the ContentView.

import SwiftUI

struct SheetView: View {
	// 1.
    @Binding var isPresented: Bool
	// 2.
    @State var modelName: String = "toy_biplane_idle"
    
    var body: some View {
				Text("Hello world!")
    }
}

#Preview {
	// 3. 
    SheetView(isPresented: .constant(true))
}

Create a new SwiftUI file and name it SheetView.swift. In the file:

1. Create a binding to a boolean var to trigger the opening of this View
2. Create a modelName variable to pass the name of the 3D model to the ARViewContainer
3. The Preview will expect a boolean value, you can use .costant(true) to make it work on the canvas

The next step is to use the previously created ARViewContainer in our view.

struct SheetView: View {
    @Binding var isPresented : Bool
    @State var modelName : String = "toy_biplane_idle"
    
    var body: some View {
        // 1.
        ZStack{
            // 2.
            ARViewContainer(modelName: $modelName)
                .ignoresSafeArea(edges: .all)						
        }
    }
}

#Preview {
    SheetView(isPresented: .constant(true))
}

1. Create a ZStack container
2. Put the ARViewContainer view into the ZStack with a .ignoresSafeArea(edges: .all) modifier. This will guarantee that the view will take the entire screen

struct SheetView: View {
    @Binding var isPresented : Bool
    @State var modelName : String = "toy_biplane_idle"
    
    var body: some View {
        // 1.
        ZStack(alignment: .topTrailing) {

            ARViewContainer(modelName: $modelName)
                .ignoresSafeArea(edges: .all)

            // 2.
            Button() {
                isPresented.toggle()
            } label: {
                Image(systemName: "xmark.circle")
                    .font(.largeTitle)
                    .foregroundColor(.black)
                    .background(.ultraThinMaterial)
                    .clipShape(Circle())
            }
    		.padding(24)
        }
    }
}

1. Define the alignment of the ZStack to be .topTrailing to position our button on the top right of the view
2. Create a Button to close the SheetView. Notice that the background of the button is .ultraThinMaterial, so it’s not blocking the content behind it

Content View

This SwiftUI view is responsible for displaying a simple user interface consisting of an image and a button.

struct ContentView: View {
	// 1.
	@State var isPresented: Bool = false

    var body: some View {
        VStack {
            Image(systemName: "globe")
                .imageScale(.large)
                .foregroundStyle(.tint)
            Text("Hello, world!")
        }
        .padding()
    }
}

1. In the ContentView declare a variable named isPresented of a boolean type, that will be used to present the SheetView

import SwiftUI

struct ContentView: View {
		
    @State var isPresented: Bool = false

    var body: some View {
        VStack {
            // 1.
            Image("plane")
                .resizable()
                .scaledToFit()
                .foregroundStyle(.tint)
                .clipShape(RoundedRectangle(cornerSize: CGSize(width: 20, height: 20)))
                .padding(24)

            // 2.
            Button {
                isPresented.toggle()
                    } label: {
                        Label("View in AR", systemImage: "arkit")
                    }.buttonStyle(BorderedProminentButtonStyle())
                .padding(24)


        }
				.padding()
				
        // 3.
        .fullScreenCover(isPresented: $isPresented, content: {
           SheetView(isPresented: $isPresented)
        })
        
    }
}

1. Add an Image view to show the image of the 3D model, downloaded at the beginning
2. Add a Button view to present the SheetView when pressed
3. Add the .fullScreenCover(isPresented:onDismiss:content:) modifier that will be triggered by the isPresented variable, presenting the SheetView covering the entire screen.

Conclusion

Now it’s time to see the result! You are ready to run the app on your iPhone or iPad.

We've seen that RealityKit offers a straightforward and intuitive approach to augmented reality development, unlocking a world of exciting possibilities for developers.

Now you are ready to build your amazing AR app. Don’t forget to check the Human Interface Guidelines in the Augmented Reality section, which will provide valuable insights and best practices for designing augmented reality apps.

Augmented reality | Apple Developer Documentation

Augmented reality (or AR) lets you deliver immersive, engaging experiences that seamlessly blend virtual objects with the real world.

Apple Developer Documentation