RealityKit Development Guide

Purpose: Build 3D content, AR experiences, and spatial computing apps using RealityKit's Entity-Component-System architecture iOS Version: iOS 13+ (base), iOS 18+ (RealityView on iOS), visionOS 1.0+ Xcode: Xcode 15+

When to Use This Skill

Use this skill when:

• Building any 3D experience (AR, games, visualization, spatial computing)
• Creating SwiftUI apps with 3D content (RealityView, Model3D)
• Implementing AR with anchors (world, image, face, body tracking)
• Working with Entity-Component-System (ECS) architecture
• Setting up physics, collisions, or spatial interactions
• Building multiplayer or shared AR experiences
• Migrating from SceneKit to RealityKit
• Targeting visionOS

Do NOT use this skill for:

• SceneKit maintenance (use axiom-scenekit)
• 2D games (use axiom-spritekit)
• Metal shader programming (use axiom-metal-migration-ref)
• Pure GPU compute (use Metal directly)

---

1. Mental Model: ECS vs Scene Graph

Scene Graph (SceneKit)

In SceneKit, nodes own their properties. A node IS a renderable, collidable, animated thing.

Entity-Component-System (RealityKit)

In RealityKit, entities are empty containers. Components add data. Systems process that data.

Entity (identity + hierarchy)
  ├── TransformComponent (position, rotation, scale)
  ├── ModelComponent (mesh + materials)
  ├── CollisionComponent (collision shapes)
  ├── PhysicsBodyComponent (mass, mode)
  └── [YourCustomComponent] (game-specific data)

System (processes entities with specific components each frame)

Why ECS matters:

• Composition over inheritance: Combine any components on any entity
• Data-oriented: Systems process arrays of components efficiently
• Decoupled logic: Systems don't know about each other
• Testable: Components are pure data, Systems are pure logic

The ECS Mental Shift

Scene Graph Thinking	ECS Thinking
"The player node moves"	"The movement system processes entities with MovementComponent"
"Add a method to the node subclass"	"Add a component, create a system"
"Override update(_:) in the node"	"Register a System that queries for components"
"The node knows its health"	"HealthComponent holds data, DamageSystem processes it"

---

2. Entity Hierarchy

Creating Entities

// Empty entity
let entity = Entity()
entity.name = "player"

// Entity with components
let entity = Entity()
entity.components[ModelComponent.self] = ModelComponent(
    mesh: .generateBox(size: 0.1),
    materials: [SimpleMaterial(color: .blue, isMetallic: false)]
)

// ModelEntity convenience (has ModelComponent built in)
let box = ModelEntity(
    mesh: .generateBox(size: 0.1),
    materials: [SimpleMaterial(color: .red, isMetallic: true)]
)

Hierarchy Management

// Parent-child
parent.addChild(child)
child.removeFromParent()

// Find entities
let found = root.findEntity(named: "player")

// Enumerate
for child in entity.children {
    // Process children
}

// Clone
let clone = entity.clone(recursive: true)

Transform

// Local transform (relative to parent)
entity.position = SIMD3<Float>(0, 1, 0)
entity.orientation = simd_quatf(angle: .pi / 4, axis: SIMD3(0, 1, 0))
entity.scale = SIMD3<Float>(repeating: 2.0)

// World-space queries
let worldPos = entity.position(relativeTo: nil)
let worldTransform = entity.transform(relativeTo: nil)

// Set world-space transform
entity.setPosition(SIMD3(1, 0, 0), relativeTo: nil)

// Look at a point
entity.look(at: targetPosition, from: entity.position, relativeTo: nil)

---

3. Components

Built-in Components

Component	Purpose
Transform	Position, rotation, scale
ModelComponent	Mesh geometry + materials
CollisionComponent	Collision shapes for physics and interaction
PhysicsBodyComponent	Mass, physics mode (dynamic/static/kinematic)
PhysicsMotionComponent	Linear and angular velocity
AnchoringComponent	AR anchor attachment
SynchronizationComponent	Multiplayer sync
PerspectiveCameraComponent	Camera settings
DirectionalLightComponent	Directional light
PointLightComponent	Point light
SpotLightComponent	Spot light
CharacterControllerComponent	Character physics controller
AudioMixGroupsComponent	Audio mixing
SpatialAudioComponent	3D positional audio
AmbientAudioComponent	Non-positional audio
ChannelAudioComponent	Multi-channel audio
OpacityComponent	Entity transparency
GroundingShadowComponent	Contact shadow
InputTargetComponent	Gesture input (visionOS)
HoverEffectComponent	Hover highlight (visionOS)
AccessibilityComponent	VoiceOver support

Custom Components

struct HealthComponent: Component {
    var current: Int
    var maximum: Int

    var percentage: Float {
        Float(current) / Float(maximum)
    }
}

// Register before use (typically in app init)
HealthComponent.registerComponent()

// Attach to entity
entity.components[HealthComponent.self] = HealthComponent(current: 100, maximum: 100)

// Read
if let health = entity.components[HealthComponent.self] {
    print(health.current)
}

// Modify
entity.components[HealthComponent.self]?.current -= 10

Component Lifecycle

Components are value types (structs). When you read a component, modify it, and write it back, you're replacing the entire component:

// Read-modify-write pattern
var health = entity.components[HealthComponent.self]!
health.current -= damage
entity.components[HealthComponent.self] = health

Anti-pattern: Holding a reference to a component and expecting mutations to propagate. Components are copied on read.

---

4. Systems

System Protocol

struct DamageSystem: System {
    // Define which components this system needs
    static let query = EntityQuery(where: .has(HealthComponent.self))

    init(scene: RealityKit.Scene) {
        // One-time setup
    }

    func update(context: SceneUpdateContext) {
        for entity in context.entities(matching: Self.query,
                                        updatingSystemWhen: .rendering) {
            var health = entity.components[HealthComponent.self]!
            if health.current <= 0 {
                entity.removeFromParent()
            }
        }
    }
}

// Register system
DamageSystem.registerSystem()

System Best Practices

• One responsibility per system: MovementSystem, DamageSystem, RenderingSystem — not GameLogicSystem
• Query filtering: Use precise queries to avoid processing irrelevant entities
• Order matters: Systems run in registration order. Register dependencies first.
• Avoid storing entity references: Query each frame instead. Entity references can become stale.

Event Handling

// Subscribe to collision events
scene.subscribe(to: CollisionEvents.Began.self) { event in
    let entityA = event.entityA
    let entityB = event.entityB
    // Handle collision
}

// Subscribe to scene update
scene.subscribe(to: SceneEvents.Update.self) { event in
    let deltaTime = event.deltaTime
    // Per-frame logic
}

---

5. SwiftUI Integration

RealityView (iOS 18+, visionOS 1.0+)

struct ContentView: View {
    var body: some View {
        RealityView { content in
            // make closure — called once
            let box = ModelEntity(
                mesh: .generateBox(size: 0.1),
                materials: [SimpleMaterial(color: .blue, isMetallic: false)]
            )
            content.add(box)

        } update: { content in
            // update closure — called when SwiftUI state changes
        }
    }
}

RealityView with Camera (iOS)

On iOS, RealityView provides a camera content parameter for configuring the AR or virtual camera:

RealityView { content, attachments in
    // Load 3D content
    if let model = try? await ModelEntity(named: "scene") {
        content.add(model)
    }
}

Loading Content Asynchronously

RealityView { content in
    // Load from bundle
    if let entity = try? await Entity(named: "MyScene", in: .main) {
        content.add(entity)
    }

    // Load from URL
    if let entity = try? await Entity(contentsOf: modelURL) {
        content.add(entity)
    }
}

Model3D (Simple Display)

// Simple 3D model display (no interaction)
Model3D(named: "toy_robot") { model in
    model
        .resizable()
        .scaledToFit()
} placeholder: {
    ProgressView()
}

SwiftUI Attachments (visionOS)

RealityView { content, attachments in
    let entity = ModelEntity(mesh: .generateSphere(radius: 0.1))
    content.add(entity)

    if let label = attachments.entity(for: "priceTag") {
        label.position = SIMD3(0, 0.15, 0)
        entity.addChild(label)
    }
} attachments: {
    Attachment(id: "priceTag") {
        Text("$9.99")
            .padding()
            .glassBackgroundEffect()
    }
}

State Binding Pattern

struct GameView: View {
    @State private var score = 0

    var body: some View {
        VStack {
            Text("Score: \(score)")

            RealityView { content in
                let scene = try! await Entity(named: "GameScene")
                content.add(scene)
            } update: { content in
                // React to state changes
                // Note: update is called when SwiftUI state changes,
                // not every frame. Use Systems for per-frame logic.
            }
        }
    }
}

---

6. AR on iOS

AnchorEntity

// Horizontal plane
let anchor = AnchorEntity(.plane(.horizontal, classification: .table,
                                  minimumBounds: SIMD2(0.2, 0.2)))

// Vertical plane
let anchor = AnchorEntity(.plane(.vertical, classification: .wall,
                                  minimumBounds: SIMD2(0.5, 0.5)))

// World position
let anchor = AnchorEntity(world: SIMD3<Float>(0, 0, -1))

// Image anchor
let anchor = AnchorEntity(.image(group: "AR Resources", name: "poster"))

// Face anchor (front camera)
let anchor = AnchorEntity(.face)

// Body anchor
let anchor = AnchorEntity(.body)

SpatialTrackingSession (iOS 18+)

let session = SpatialTrackingSession()
let configuration = SpatialTrackingSession.Configuration(tracking: [.plane, .object])
let result = await session.run(configuration)

if let notSupported = result {
    // Handle unsupported tracking on this device
    for denied in notSupported.deniedTrackingModes {
        print("Not supported: \(denied)")
    }
}

AR Best Practices

• Anchor entities to detected surfaces rather than world positions for stability
• Use plane classification (.table, .floor, .wall) to place content appropriately
• Start with horizontal plane detection — it's the most reliable
• Test on real devices; simulator AR is limited
• Provide visual feedback during surface detection (coaching overlay)

---

7. Interaction

ManipulationComponent (iOS, visionOS)

// Enable drag, rotate, scale gestures
entity.components[ManipulationComponent.self] = ManipulationComponent(
    allowedModes: .all  // .translate, .rotate, .scale
)

// Also requires CollisionComponent for hit testing
entity.generateCollisionShapes(recursive: true)

InputTargetComponent (visionOS)

// Required for visionOS gesture input
entity.components[InputTargetComponent.self] = InputTargetComponent()
entity.components[CollisionComponent.self] = CollisionComponent(
    shapes: [.generateBox(size: SIMD3(0.1, 0.1, 0.1))]
)

Gesture Integration with SwiftUI

RealityView { content in
    let entity = ModelEntity(mesh: .generateBox(size: 0.1))
    entity.generateCollisionShapes(recursive: true)
    entity.components.set(InputTargetComponent())
    content.add(entity)
}
.gesture(
    TapGesture()
        .targetedToAnyEntity()
        .onEnded { value in
            let tappedEntity = value.entity
            // Handle tap
        }
)
.gesture(
    DragGesture()
        .targetedToAnyEntity()
        .onChanged { value in
            value.entity.position = value.convert(value.location3D,
                from: .local, to: .scene)
        }
)

Hit Testing

// Ray-cast from screen point
if let result = arView.raycast(from: screenPoint,
                                allowing: .estimatedPlane,
                                alignment: .horizontal).first {
    let worldPosition = result.worldTransform.columns.3
    // Place entity at worldPosition
}

---

8. Materials and Rendering

Material Types

Material	Purpose	Customization
SimpleMaterial	Solid color or texture	Color, metallic, roughness
PhysicallyBasedMaterial	Full PBR	All PBR maps (base color, normal, metallic, roughness, AO, emissive)
UnlitMaterial	No lighting response	Color or texture, always fully lit
OcclusionMaterial	Invisible but occludes	AR content hiding behind real objects
VideoMaterial	Video playback on surface	AVPlayer-driven
ShaderGraphMaterial	Custom shader graph	Reality Composer Pro
CustomMaterial	Metal shader functions	Full Metal control

PhysicallyBasedMaterial

var material = PhysicallyBasedMaterial()
material.baseColor = .init(tint: .white,
    texture: .init(try! .load(named: "albedo")))
material.metallic = .init(floatLiteral: 0.0)
material.roughness = .init(floatLiteral: 0.5)
material.normal = .init(texture: .init(try! .load(named: "normal")))
material.ambientOcclusion = .init(texture: .init(try! .load(named: "ao")))
material.emissiveColor = .init(color: .blue)
material.emissiveIntensity = 2.0

let entity = ModelEntity(
    mesh: .generateSphere(radius: 0.1),
    materials: [material]
)

OcclusionMaterial (AR)

// Invisible plane that hides 3D content behind it
let occluder = ModelEntity(
    mesh: .generatePlane(width: 1, depth: 1),
    materials: [OcclusionMaterial()]
)
occluder.position = SIMD3(0, 0, 0)
anchor.addChild(occluder)

Environment Lighting

// Image-based lighting
if let resource = try? await EnvironmentResource(named: "studio_lighting") {
    // Apply via RealityView content
}

---

9. Physics and Collision

Collision Shapes

// Generate from mesh (accurate but expensive)
entity.generateCollisionShapes(recursive: true)

// Manual shapes (prefer for performance)
entity.components[CollisionComponent.self] = CollisionComponent(
    shapes: [
        .generateBox(size: SIMD3(0.1, 0.2, 0.1)),     // Box
        .generateSphere(radius: 0.1),                   // Sphere
        .generateCapsule(height: 0.3, radius: 0.05)     // Capsule
    ]
)

Physics Body

// Dynamic — physics simulation controls movement
entity.components[PhysicsBodyComponent.self] = PhysicsBodyComponent(
    massProperties: .init(mass: 1.0),
    material: .generate(staticFriction: 0.5,
                        dynamicFriction: 0.3,
                        restitution: 0.4),
    mode: .dynamic
)

// Static — immovable collision surface
ground.components[PhysicsBodyComponent.self] = PhysicsBodyComponent(
    mode: .static
)

// Kinematic — code-controlled, participates in collisions
platform.components[PhysicsBodyComponent.self] = PhysicsBodyComponent(
    mode: .kinematic
)

Collision Groups and Filters

// Define groups
let playerGroup = CollisionGroup(rawValue: 1 << 0)
let enemyGroup = CollisionGroup(rawValue: 1 << 1)
let bulletGroup = CollisionGroup(rawValue: 1 << 2)

// Filter: player collides with enemies and bullets
entity.components[CollisionComponent.self] = CollisionComponent(
    shapes: [.generateSphere(radius: 0.1)],
    filter: CollisionFilter(
        group: playerGroup,
        mask: enemyGroup | bulletGroup
    )
)

Collision Events

// Subscribe in RealityView make closure or System
scene.subscribe(to: CollisionEvents.Began.self, on: playerEntity) { event in
    let otherEntity = event.entityA == playerEntity ? event.entityB : event.entityA
    handleCollision(with: otherEntity)
}

Applying Forces

if var motion = entity.components[PhysicsMotionComponent.self] {
    motion.linearVelocity = SIMD3(0, 5, 0)  // Impulse up
    entity.components[PhysicsMotionComponent.self] = motion
}

---

10. Animation

Transform Animation

// Animate to position over duration
entity.move(
    to: Transform(
        scale: SIMD3(repeating: 1.5),
        rotation: simd_quatf(angle: .pi, axis: SIMD3(0, 1, 0)),
        translation: SIMD3(0, 2, 0)
    ),
    relativeTo: entity.parent,
    duration: 2.0,
    timingFunction: .easeInOut
)

Playing USD Animations

if let entity = try? await Entity(named: "character") {
    // Play all available animations
    for animation in entity.availableAnimations {
        entity.playAnimation(animation.repeat())
    }
}

Animation Playback Control

let controller = entity.playAnimation(animation)
controller.pause()
controller.resume()
controller.speed = 2.0      // 2x playback speed
controller.blendFactor = 0.5 // Blend with current state

---

11. Audio

Spatial Audio

// Load audio resource
let resource = try! AudioFileResource.load(named: "engine.wav",
    configuration: .init(shouldLoop: true))

// Create entity with spatial audio
let audioEntity = Entity()
audioEntity.components[SpatialAudioComponent.self] = SpatialAudioComponent()
let controller = audioEntity.playAudio(resource)

// Position the audio source in 3D space
audioEntity.position = SIMD3(2, 0, -1)

Ambient Audio

entity.components[AmbientAudioComponent.self] = AmbientAudioComponent()
entity.playAudio(backgroundMusic)

---

12. Performance

Entity Count

• Under 100 entities: No concerns
• 100-1000 entities: Monitor with RealityKit debugger
• 1000+ entities: Use instancing and LOD strategies

Instancing

// Share mesh and material across many entities
let sharedMesh = MeshResource.generateSphere(radius: 0.01)
let sharedMaterial = SimpleMaterial(color: .white, isMetallic: false)

for i in 0..<1000 {
    let entity = ModelEntity(mesh: sharedMesh, materials: [sharedMaterial])
    entity.position = randomPosition()
    parent.addChild(entity)
}

RealityKit automatically batches entities with identical mesh and material resources.

Component Churn

Anti-pattern: Creating and replacing components every frame.

// BAD — component allocation every frame
func update(context: SceneUpdateContext) {
    for entity in context.entities(matching: query, updatingSystemWhen: .rendering) {
        entity.components[ModelComponent.self] = ModelComponent(
            mesh: .generateBox(size: 0.1),
            materials: [newMaterial]  // New allocation every frame
        )
    }
}

// GOOD — modify existing component
func update(context: SceneUpdateContext) {
    for entity in context.entities(matching: query, updatingSystemWhen: .rendering) {
        // Only update when actually needed
        if needsUpdate {
            var model = entity.components[ModelComponent.self]!
            model.materials = [cachedMaterial]
            entity.components[ModelComponent.self] = model
        }
    }
}

Collision Shape Optimization

• Use simple shapes (box, sphere, capsule) instead of mesh-based collision
• generateCollisionShapes(recursive: true) is convenient but expensive
• For static geometry, generate shapes once during setup

Profiling

Use Xcode's RealityKit debugger:

• Entity Inspector: View entity hierarchy and components
• Statistics Overlay: Entity count, draw calls, triangle count
• Physics Visualization: Show collision shapes

---

13. Multiplayer

Synchronization Basics

// Components sync automatically if they conform to Codable
struct ScoreComponent: Component, Codable {
    var points: Int
}

// SynchronizationComponent controls what syncs
entity.components[SynchronizationComponent.self] = SynchronizationComponent()

MultipeerConnectivityService

let service = try MultipeerConnectivityService(session: mcSession)
// Entities with SynchronizationComponent auto-sync across peers

Ownership

• Only the owner of an entity can modify it
• Request ownership before modifying shared entities
• Non-Codable component data does not sync

---

14. Anti-Patterns

Anti-Pattern 1: UIKit-Style Thinking in ECS

Time cost: Hours of frustration from fighting the architecture

// BAD — subclassing Entity for behavior
class PlayerEntity: Entity {
    func takeDamage(_ amount: Int) { /* logic in entity */ }
}

// GOOD — component holds data, system has logic
struct HealthComponent: Component { var hp: Int }
struct DamageSystem: System {
    static let query = EntityQuery(where: .has(HealthComponent.self))
    func update(context: SceneUpdateContext) {
        // Process damage here
    }
}

Anti-Pattern 2: Monolithic Entities

Time cost: Untestable, inflexible architecture

Don't put all game logic in one entity type. Split into components that can be mixed and matched.

Anti-Pattern 3: Frame-Based Updates Without Systems

Time cost: Missed frame updates, inconsistent behavior

// BAD — timer-based updates
Timer.scheduledTimer(withTimeInterval: 1/60, repeats: true) { _ in
    entity.position.x += 0.01
}

// GOOD — System update
struct MovementSystem: System {
    static let query = EntityQuery(where: .has(VelocityComponent.self))
    func update(context: SceneUpdateContext) {
        for entity in context.entities(matching: Self.query,
                                        updatingSystemWhen: .rendering) {
            let velocity = entity.components[VelocityComponent.self]!
            entity.position += velocity.value * Float(context.deltaTime)
        }
    }
}

Anti-Pattern 4: Not Generating Collision Shapes for Interactive Entities

Time cost: 15-30 min debugging "why taps don't work"

Gestures require CollisionComponent. If an entity has InputTargetComponent (visionOS) or ManipulationComponent but no CollisionComponent, gestures will never fire.

Anti-Pattern 5: Storing Entity References in Systems

Time cost: Crashes from stale references

// BAD — entity might be removed between frames
struct BadSystem: System {
    var playerEntity: Entity?  // Stale reference risk

    func update(context: SceneUpdateContext) {
        playerEntity?.position.x += 0.1  // May crash
    }
}

// GOOD — query each frame
struct GoodSystem: System {
    static let query = EntityQuery(where: .has(PlayerComponent.self))

    func update(context: SceneUpdateContext) {
        for entity in context.entities(matching: Self.query,
                                        updatingSystemWhen: .rendering) {
            entity.position.x += Float(context.deltaTime)
        }
    }
}

---

15. Code Review Checklist

• Custom components registered via registerComponent() before use
• Systems registered via registerSystem() before scene loads
• Components are value types (structs), not classes
• Read-modify-write pattern used for component updates
• Interactive entities have CollisionComponent
• visionOS interactive entities have InputTargetComponent
• Collision shapes are simple (box/sphere/capsule) where possible
• No entity references stored across frames in Systems
• Mesh and material resources shared across identical entities
• Component updates only occur when values actually change
• USD/USDZ format used for 3D assets (not .scn)
• Async loading used for all model/scene loading
• [weak self] in closure-based subscriptions if retaining view/controller

---

16. Pressure Scenarios

Scenario 1: "ECS Is Overkill for Our Simple App"

Pressure: Team wants to avoid learning ECS, just needs one 3D model displayed

Wrong approach: Skip ECS, jam all logic into RealityView closures.

Correct approach: Even simple apps benefit from ECS. A single ModelEntity in a RealityView is already using ECS — you're just not adding custom components yet. Start simple, add components as complexity grows.

Push-back template: "We're already using ECS — Entity and ModelComponent. The pattern scales. Adding a custom component when we need behavior is one struct definition, not an architecture change."

Scenario 2: "Just Use SceneKit, We Know It"

Pressure: Team has SceneKit experience, RealityKit is unfamiliar

Wrong approach: Build new features in SceneKit.

Correct approach: SceneKit is soft-deprecated. New features won't be added. Invest in RealityKit now — the ECS concepts transfer to other game engines (Unity, Unreal, Bevy) if needed.

Push-back template: "SceneKit is in maintenance mode — no new features, only security patches. Every line of SceneKit we write is migration debt. RealityKit's concepts (Entity, Component, System) are industry-standard ECS."

Scenario 3: "Make It Work Without Collision Shapes"

Pressure: Deadline, collision shape setup seems complex

Wrong approach: Skip collision shapes, use position-based proximity detection.

Correct approach: entity.generateCollisionShapes(recursive: true) takes one line. Without it, gestures won't work and physics won't collide. The "shortcut" creates more debugging time than it saves.

Push-back template: "Collision shapes are required for gestures and physics. It's one line: entity.generateCollisionShapes(recursive: true). Skipping it means gestures silently fail — a harder bug to diagnose."

---

Resources

WWDC: 2019-603, 2019-605, 2021-10074, 2022-10074, 2023-10080, 2023-10081, 2024-10103, 2024-10153

Docs: /realitykit, /realitykit/entity, /realitykit/realityview, /realitykit/modelentity, /realitykit/anchorentity, /realitykit/component

Skills: axiom-realitykit-ref, axiom-realitykit-diag, axiom-scenekit, axiom-scenekit-ref