Generated from a full walkthrough session — covers everything from zero to a working Mario-style game.
- Why RealityKit?
- The Coordinate System
- Architecture Overview
- ECS — The Core Mental Model
- Entity Class Hierarchy
- Registration
- Components — Data Bags
- Systems — Logic Runners
- Factories — Reusable Entity Builders
- ContentView — The Bridge
- Input Handling
- Collision Detection
- Two Kinds of Animation
- Camera
- Lighting
- Materials
- Spatial Audio
- UI — SwiftUI in a 3D Game
- Threading Rules
- Asset Loading & Caching
- NPC AI & Spawning
- Long-Running vs Instant Actions
- OOP Style with ECS
- Common Patterns & Cheat Sheet
- Edge Cases & Gotchas
- Project File Structure
- Essential Links
| Framework | Status | Use For |
|---|---|---|
| RealityKit | ✅ Actively developed | All new 3D apps and games |
| SceneKit | Existing apps only — no new features | |
| Metal | ✅ Active | Low-level GPU, custom renderers |
| SpriteKit | 2D games (no clear successor yet) | |
| Unity | ✅ Third party | Cross-platform, large ecosystem |
"Soft deprecated" means SceneKit apps keep working, but it's maintenance-only (critical bug fixes only). No new features. For any new project, use RealityKit.
visionOS ──┐
iOS ──┤
iPadOS ──┤── RealityKit (one codebase)
macOS ──┤
tvOS ──┘ (added WWDC25)
RealityView { content in
content.camera = .virtual // ← this one line disables ALL AR
// Now it's a pure 3D renderer, no camera feed, no world tracking
}Almost all tutorials use .worldTracking (AR mode). For games, you always want .virtual.
Y (up)
│
│
│
└────────── X (right)
╱
╱
Z (toward viewer)
Origin = [0, 0, 0]
1 unit = 1 meter
[0, 0, 0] → world origin
[1, 0, 0] → 1m to the right
[0, 1, 0] → 1m up
[0, 0, -5] → 5m in front of camera (negative Z = forward)
Adult character ≈ 1.8 units tall
Coin ≈ 0.2–0.3 units
Pipe (Mario) ≈ 2 units tall, 1 unit wide
Game world ≈ 50–200 unit radius (practical)
Far clip plane ≈ 500–1000 units (beyond = invisible)
When you call parent.addChild(child), the child's position is relative to the parent:
mario.position = [5, 0, 0] // Mario is at world X=5
label.position = [0, 2.2, 0] // label is 2.2m ABOVE mario's origin
mario.addChild(label) // label ends up at world [5, 2.2, 0]
// label moves WITH mario automaticallyDefault forward = -Z (into the screen)
entity.scale.x = -1 → mirror/flip horizontally (face left)
entity.scale.x = 1 → normal (face right)
entity.look(at: targetPos, from: entity.position, relativeTo: nil)
→ rotates entity to face targetPos
A RealityKit game has three distinct layers. Keep them separate.
┌──────────────────────────────────────────────────────────────┐
│ LAYER 1: SwiftUI │
│ GameState (@Observable), HUD, menus, pause screen, │
│ text inputs, buttons — pure Swift, nothing 3D │
├──────────────────────────────────────────────────────────────┤
│ LAYER 2: RealityView │
│ The bridge between SwiftUI ↔ 3D world │
│ make { } — build scene once │
│ update { } — sync state changes into 3D │
│ .onReceive — listen for 3D events (collisions) │
├──────────────────────────────────────────────────────────────┤
│ LAYER 3: ECS │
│ Entities (things) + Components (data) + Systems (logic) │
│ All game behavior lives here │
└──────────────────────────────────────────────────────────────┘
Every ~16ms (60fps):
│
├─ 1. InputState updated by SwiftUI key handlers
│
├─ 2. System.update() called for each registered System (render thread)
│ ├─ CharacterMovementSystem → reads input, moves character
│ ├─ EnemyAISystem → moves enemies
│ ├─ AnimationSystem → swaps animation clips on state change
│ ├─ CoinSystem → bobs coins up/down
│ └─ HoldInteractionSystem → accumulates hold timers
│
├─ 3. Physics simulation runs (collision detection, resolution)
│ └─ CollisionEvents.Began / Updated / Ended fired
│
├─ 4. RealityKit renders scene (Metal, spatial audio updated)
│
└─ 5. If @State changed → RealityView.update{} closure fires
└─ SwiftUI re-renders HUD
Entity-Component-System is a composition pattern. Behavior comes from combining data (Components), not from class hierarchies.
┌─────────────┐ ┌─────────────────────────────────┐
│ Entity │ │ Components │
│ (just an │ ←──│ CharacterComponent { speed } │
│ ID + pos) │ │ CollisionComponent { shapes } │
│ │ │ PhysicsBodyComponent{ mass } │
└─────────────┘ └─────────────────────────────────┘
↑ queried by
┌─────────────────────────────────┐
│ System │
│ query = .has(CharacterComponent)│
│ update() { for entity in query │
│ // move the entity │
│ } │
└─────────────────────────────────┘
OOP (what you're used to): ECS (RealityKit):
───────────────────────── ──────────────────────────────
class Mario { Entity (ID: 001)
var speed = 5.0 + CharacterComponent { speed: 5.0 }
func jump() { ... } + CollisionComponent { ... }
func update() { ... } + ModelComponent { mesh }
func handleCollision() { ... }
} CharacterMovementSystem
.query = .has(CharacterComponent)
// Mario knows about everything .update() { move all characters }
// Hard to reuse, hard to scale
// No entity "knows" anything
// Behavior comes from Systems
EntityID │ CharacterComponent │ EnemyComponent │ CoinComponent │ CollisionComponent
─────────┼────────────────────┼────────────────┼───────────────┼────────────────────
001 │ {vel:[1,0,0], …} │ — │ — │ {shapes:[…]}
002 │ — │ {speed:1.5} │ — │ {shapes:[…]}
003 │ — │ — │ {bobTimer:0} │ {shapes:[…]}
004 │ — │ — │ {bobTimer:0} │ {shapes:[…]}
A query like .has(CoinComponent) returns rows 003 and 004 instantly — no looping all entities.
Entity (open class — base, just a transform)
│
├── ModelEntity ← visible 3D object (mesh + material)
│ most common — almost everything visible
│
├── AnchorEntity ← pins content to a real-world surface (AR)
│ ├── .plane(.horizontal) → floor/table
│ ├── .plane(.vertical) → wall
│ ├── .image(...) → AR image target
│ └── .camera → always in front of camera
│ NOTE: In non-AR games, skip AnchorEntity entirely.
│ Add entities directly with content.add(entity)
│
├── DirectionalLight ← sun (parallel rays, infinite distance)
├── PointLight ← bulb (all directions, fades with distance)
├── SpotLight ← flashlight (cone shape)
│
├── PerspectiveCamera ← the viewpoint (your game camera)
│
└── TriggerVolume ← invisible zone, fires events on enter/exit
You should not subclass any of these for game logic.
Use Factories + Components instead (see §9, §23).
Before any entity or system runs, every custom type must be registered once at app startup.
@main
struct MyGameApp: App {
init() {
// ── Components (data bags) ──────────────────────────────────
CharacterComponent.registerComponent()
EnemyComponent.registerComponent()
CoinComponent.registerComponent()
BlockComponent.registerComponent()
HoldInteractComponent.registerComponent()
// ── Systems (logic runners) ─────────────────────────────────
CharacterMovementSystem.registerSystem()
EnemyAISystem.registerSystem()
AnimationSystem.registerSystem()
CoinSystem.registerSystem()
HoldInteractionSystem.registerSystem()
}
var body: some Scene {
WindowGroup { ContentView() }
}
}RealityKit's ECS stores components in a typed column database. registerComponent() allocates a column for your type. Without it, the engine has nowhere to store your data — runtime crash.
registerComponent() effect:
Before: ECS table has no column for CharacterComponent
After: ECS table has a column, can store/query CharacterComponent
registerSystem() effect:
Before: CharacterMovementSystem.update() is never called
After: RealityKit calls update() every frame automatically
Systems are NOT singletons you manage. RealityKit creates one instance per scene and calls update() on it. You never call CharacterMovementSystem() yourself.
Components are pure data. No logic, no methods (except simple helpers). Always struct.
// ✅ Good component — pure data
struct CharacterComponent: Component {
var velocity: SIMD3<Float> = .zero
var isOnGround: Bool = false
var speed: Float = 5.0
var jumpImpulse: Float = 9.0
enum MotionState { case idle, walking, running, jumping, falling, crouching }
var motionState: MotionState = .idle
var facingRight: Bool = true
}
// ✅ Good component — pure data
struct CoinComponent: Component {
var isCollected: Bool = false
var bobTimer: Float = 0
}
// ❌ Bad component — logic doesn't belong here
struct BadComponent: Component {
func jump() { ... } // NO — logic goes in a System
func update() { ... } // NO — Systems handle this
}// Read
let comp = entity.components[CharacterComponent.self] // Optional
let comp = entity.components[CharacterComponent.self]! // Force unwrap (risky)
// Write — always re-set after mutating (it's a value type / struct)
var comp = entity.components[CharacterComponent.self]!
comp.velocity.x += 1.0
entity.components.set(comp) // ← must call set() — mutation of copy otherwise lost
// Check presence
if entity.components[CoinComponent.self] != nil { /* is a coin */ }
// Remove
entity.components.remove(CoinComponent.self)entity.components.set(CollisionComponent(shapes: [bigShape]))
entity.components.set(CollisionComponent(shapes: [smallShape]))
// Result: ONLY smallShape — set() always replaces
// To have multiple shapes, put them in ONE component:
entity.components.set(CollisionComponent(shapes: [
.generateBox(size: [1, 1, 1]),
.generateSphere(radius: 0.3)
]))| Built-in Component | What It Actually Does |
|---|---|
ModelComponent |
Submits draw calls to Metal GPU pipeline |
CollisionComponent |
Registers shapes with physics broad-phase |
PhysicsBodyComponent |
Runs Newtonian simulation every frame |
SpatialAudioComponent |
Hooks into AVAudioEngine, computes HRTF |
BillboardComponent |
Rotates entity to face camera every frame |
| Your custom component | Stores data — that's it |
Built-in components activate engine subsystems. Your components are just data they or your Systems read.
One System per behavior type. Not per entity. One EnemyAISystem handles all 50 goombas.
class CharacterMovementSystem: System {
// Declare which entities this system cares about
static let query = EntityQuery(where: .has(CharacterComponent.self)
.and(.has(CollisionComponent.self)))
// RealityKit calls this init — you never call it
required init(scene: RealityKit.Scene) {}
// Called every frame on the render thread
func update(context: SceneUpdateContext) {
let dt = Float(context.deltaTime) // seconds since last frame (~0.016)
let input = InputState.shared
for entity in context.entities(matching: Self.query, updatingSystemWhen: .rendering) {
var comp = entity.components[CharacterComponent.self]!
// All movement logic here
if input.moveRight { comp.velocity.x = comp.speed }
comp.velocity.y += -20 * dt // gravity
entity.position += comp.velocity * dt
entity.components.set(comp)
}
}
}One System = One Concern
CharacterMovementSystem → reads input, applies velocity, gravity
AnimationSystem → watches motionState, switches animation clips
EnemyAISystem → enemy patrol, chase, death timer
CoinSystem → coin bobbing animation
SpawnSystem → checks distances, spawns new enemies
HoldInteractionSystem → accumulates hold timers, fires completion
Systems never call each other. They communicate through Component data:
CharacterMovementSystem writes: comp.motionState = .jumping
↓
AnimationSystem reads: if motionState changed → playAnimation("jump")
// AND: entity must have BOTH
EntityQuery(where: .has(EnemyComponent.self).and(.has(CollisionComponent.self)))
// OR: entity has either
EntityQuery(where: .has(CoinComponent.self).or(.has(PowerUpComponent.self)))
// NOT: entity has this but not that
EntityQuery(where: .has(EnemyComponent.self).and(.not(.has(DeadComponent.self))))Factories assemble entities with all their components. Use enums with static functions — no instances needed.
enum MarioFactory {
static func make() async throws -> Entity {
let mario = try await ModelEntity(named: "mario.usdz")
mario.name = "character"
let capsule = ShapeResource.generateCapsule(height: 1.8, radius: 0.25)
mario.components.set(CollisionComponent(shapes: [capsule]))
mario.components.set(PhysicsBodyComponent(
shapes: [capsule], mass: 70,
material: .generate(friction: 0.8, restitution: 0.0),
mode: .dynamic
))
mario.components.set(CharacterComponent())
mario.components.set(AnimationStateComponent())
mario.position = [0, 1, 0]
return mario
}
}ModelEntity(named:) reads from disk — up to 200–500ms. Async prevents freezing the UI.
// ❌ Slow — hits disk every time
for i in 0..<20 {
let coin = try await ModelEntity(named: "coin.usdz") // 20 disk reads
}
// ✅ Fast — one disk read, then clone (instant, GPU memory shared)
enum AssetCache {
static var coin: Entity?
static var goomba: Entity?
static func preload() async throws {
coin = try await ModelEntity(named: "coin.usdz")
goomba = try await ModelEntity(named: "goomba.usdz")
}
}
// At scene startup:
try await AssetCache.preload()
// Spawning many instances:
for i in 0..<20 {
if let coin = AssetCache.coin?.clone(recursive: true) {
coin.position = [Float(i), 0.5, 0]
coin.components.set(CoinComponent()) // fresh component per clone
scene.add(coin)
}
}struct ContentView: View {
@State private var gameState = GameState()
private let scene = GameScene()
var body: some View {
ZStack {
// ── 3D World ─────────────────────────────────────────
RealityView { content in
// make{}: runs ONCE at startup — build the scene
await scene.build(content: &content, gameState: gameState)
} update: { content in
// update{}: runs when @State on THIS view changes
// Push SwiftUI state → 3D world
content.scene.physicsSimulation =
gameState.phase == .paused ? .paused : .running
}
.focusable()
.onKeyPress(phases: .all) { press in handleKey(press) }
.ignoresSafeArea()
// ── HUD (floats over 3D world) ────────────────────────
if gameState.phase == .playing {
HUDView(gameState: gameState)
}
// ── Menus ─────────────────────────────────────────────
if gameState.phase == .paused {
PauseMenuView(gameState: gameState)
}
}
}
}make { content in ... }
└── Runs ONCE when the view appears
└── Build your entire scene here
└── async — safe to call ModelEntity(named:) etc.
update { content in ... }
└── Runs when @State / @Observable on the View changes
└── NOT every frame — only on SwiftUI state changes
└── Push SwiftUI → 3D (e.g. show/hide entities when state changes)
└── Do NOT do heavy work here
class InputState {
static let shared = InputState()
private init() {}
var moveLeft = false
var moveRight = false
var crouching = false
var jumpPressed = false // one-shot: true only on the frame pressed
var jumpHeld = false // continuous: true while held
func clearFrameInputs() {
jumpPressed = false
}
}RealityView { ... }
.focusable()
.onKeyPress(phases: .all) { press in
let down = (press.phase == .down || press.phase == .repeat)
switch press.key {
case .leftArrow, "a": InputState.shared.moveLeft = down
case .rightArrow, "d": InputState.shared.moveRight = down
case .downArrow, "s": InputState.shared.crouching = down
case .space:
InputState.shared.jumpHeld = down
if press.phase == .down { InputState.shared.jumpPressed = true }
default: return .ignored
}
return .handled
}RealityView { ... }
.gesture(
TapGesture().targetedToAnyEntity()
.onEnded { value in
let tapped = value.entity // exact entity tapped
if tapped.components[CoinComponent.self] != nil {
tapped.collect()
}
}
)import GameController
NotificationCenter.default.addObserver(
forName: .GCControllerDidConnect, object: nil, queue: .main
) { _ in
guard let pad = GCController.controllers().first?.extendedGamepad else { return }
pad.leftThumbstick.valueChangedHandler = { _, x, y in
InputState.shared.moveLeft = x < -0.2
InputState.shared.moveRight = x > 0.2
}
pad.buttonA.pressedChangedHandler = { _, _, pressed in
if pressed { InputState.shared.jumpPressed = true }
InputState.shared.jumpHeld = pressed
}
}// Solid body — blocks other physics objects, fires collision events
entity.components.set(CollisionComponent(shapes: [shape]))
entity.components.set(PhysicsBodyComponent(shapes: [shape], mass: 70, mode: .dynamic))
// Trigger — fires events on overlap but doesn't block anything (coins, pickups)
entity.components.set(CollisionComponent(
shapes: [shape],
mode: .trigger,
filter: .sensor
))
// Static solid — doesn't move (ground, walls, blocks)
entity.components.set(CollisionComponent(shapes: [shape]))
entity.components.set(PhysicsBodyComponent(shapes: [shape], mass: 0, mode: .static))// In GameScene.build(), after adding entities:
_ = content.subscribe(to: CollisionEvents.Began.self) { event in
let a = event.entityA // exact Entity object #1
let b = event.entityB // exact Entity object #2
handleCollision(a, b)
}
_ = content.subscribe(to: CollisionEvents.Updated.self) { event in
// Fires every frame while two entities remain in contact
}
_ = content.subscribe(to: CollisionEvents.Ended.self) { event in
// Contact just broke
}func handleCollision(_ a: Entity, _ b: Entity) {
// By name (simple, fragile to typos)
if a.name == "character" && b.name == "coin" { ... }
// By component (safer)
let char = [a, b].first { $0.components[CharacterComponent.self] != nil }
let coin = [a, b].first { $0.components[CoinComponent.self] != nil }
guard let char, let coin else { return }
// Check collected guard — collision can fire multiple times
var coinComp = coin.components[CoinComponent.self]!
guard !coinComp.isCollected else { return }
coinComp.isCollected = true
coin.components.set(coinComp)
// ...
}// Did character land on TOP of enemy?
let charY = character.position(relativeTo: nil).y
let enemyY = enemy.position(relativeTo: nil).y
if charY > enemyY + 0.4 {
// Stomp — character is above
killEnemy(enemy)
bounceCharacterUp(character)
gameState.addScore(200)
} else {
// Side hit — character takes damage
damageCharacter(character)
}let playerGroup = CollisionGroup(rawValue: 1 << 0)
let enemyGroup = CollisionGroup(rawValue: 1 << 1)
let playerBullet = CollisionGroup(rawValue: 1 << 2)
let enemyBullet = CollisionGroup(rawValue: 1 << 3)
// Player: collides with enemies and enemy bullets, not friendly bullets
player.components.set(CollisionComponent(
shapes: [...],
filter: .init(group: playerGroup, mask: enemyGroup.union(enemyBullet))
))
// Player's bullet: hits enemies only
bullet.components.set(CollisionComponent(
shapes: [...],
filter: .init(group: playerBullet, mask: enemyGroup)
))These are completely different systems — understanding the distinction is critical.
Transform Animation Skeletal Animation
────────────────────────── ────────────────────────────────────────
entity.move(to:duration:) entity.playAnimation(clip)
Moves/scales/rotates the Deforms a rigged mesh — bones
entity as a whole move, vertices follow
No skeleton needed Requires a rigged USDZ model
with named animation clips
Used for: Used for:
coin popping up mario walking
block bouncing enemy attacking
platform sliding character jumping
door opening idle breathing
anything moving A→B character crouching
// Move entity to new transform over 0.3 seconds
var targetTransform = entity.transform
targetTransform.translation.y += 1.5
targetTransform.scale = [1.5, 0.2, 1.5] // squash
entity.move(
to: targetTransform,
relativeTo: nil,
duration: 0.3,
timingFunction: .easeOut // .linear .easeIn .easeOut .easeInOut
)
// Chain: move up, wait, move back down
entity.move(to: upTransform, relativeTo: nil, duration: 0.1)
Task {
try await Task.sleep(for: .milliseconds(120))
await MainActor.run {
entity.move(to: downTransform, relativeTo: nil, duration: 0.1)
}
}// List available clips from a USDZ model
print(mario.availableAnimations.map { $0.name })
// ["idle", "walk", "run", "jump", "fall", "crouch"]
// Play once
mario.playAnimation(jumpClip)
// Loop
mario.playAnimation(walkClip.repeat())
// Loop N times
mario.playAnimation(walkClip.repeat(count: 3))
// Cross-fade (blend between clips smoothly)
mario.playAnimation(runClip.repeat(),
transitionDuration: 0.25,
startsPaused: false)
// Stop all
mario.stopAllAnimations()class AnimationSystem: System {
static let query = EntityQuery(where: .has(CharacterComponent.self)
.and(.has(AnimationStateComponent.self)))
required init(scene: RealityKit.Scene) {}
func update(context: SceneUpdateContext) {
for entity in context.entities(matching: Self.query, updatingSystemWhen: .rendering) {
let motion = entity.components[CharacterComponent.self]!
var animComp = entity.components[AnimationStateComponent.self]!
let targetClip = clipName(for: motion.motionState)
guard targetClip != animComp.currentClipName else { continue } // no change
if let clip = entity.availableAnimations.first(where: { $0.name == targetClip }) {
let shouldLoop = (motion.motionState != .jumping)
entity.playAnimation(
shouldLoop ? clip.repeat() : clip,
transitionDuration: 0.2
)
}
animComp.currentClipName = targetClip
entity.components.set(animComp)
}
}
private func clipName(for state: CharacterComponent.MotionState) -> String {
switch state {
case .idle: "idle"
case .walking: "walk"
case .running: "run"
case .jumping: "jump"
case .falling: "fall"
case .crouching: "crouch"
}
}
}let camera = PerspectiveCamera()
camera.camera.fieldOfViewInDegrees = 60 // narrow=zoomed in, wide=fish-eye
camera.camera.near = 0.01 // 1cm — don't clip nearby things
camera.camera.far = 500 // 500m far clip
camera.position = [0, 5, 15] // 5m up, 15m behind the action
camera.look(at: [0, 2, 0], from: camera.position, relativeTo: nil)
content.add(camera)// In SceneEvents.Update subscription:
let targetX = character.position(relativeTo: nil).x
let targetY = max(4, character.position(relativeTo: nil).y + 3)
// Lerp for smooth follow (lag effect = game feel)
camera.position.x = lerp(camera.position.x, targetX, t: 0.05)
camera.position.y = lerp(camera.position.y, targetY, t: 0.05)
camera.look(at: [camera.position.x, characterPos.y + 1, 0],
from: camera.position(relativeTo: nil),
relativeTo: nil)
func lerp(_ a: Float, _ b: Float, t: Float) -> Float { a + (b - a) * min(t, 1.0) }RealityView { ... }
.realityViewCameraControls(.orbit) // drag to orbit
.realityViewCameraControls(.pan) // drag to pan// ── Directional (sun) — parallel rays, casts shadows ────────────
let sun = DirectionalLight()
sun.light.intensity = 4000 // lux
sun.light.color = UIColor(red: 1, green: 0.95, blue: 0.8, alpha: 1)
sun.shadow = DirectionalLightComponent.Shadow(maximumDistance: 50, depthBias: 2)
sun.orientation = simd_quatf(angle: -.pi / 3, axis: [1, 0, 0]) // 60° downward
content.add(sun)
// ── Point light (bulb) — all directions, fades with distance ────
let lamp = PointLight()
lamp.light.color = .orange
lamp.light.intensity = 2000 // lumens
lamp.light.attenuationRadius = 8 // fades out at 8m
lamp.position = [2, 3, -1]
content.add(lamp)
// ── Spot light (flashlight) — cone ──────────────────────────────
let spot = SpotLight()
spot.light.intensity = 5000
spot.light.innerAngleInDegrees = 20 // bright cone
spot.light.outerAngleInDegrees = 40 // soft falloff edge
spot.position = [0, 8, 0]
spot.look(at: [0, 0, 0], from: [0, 8, 0], relativeTo: nil)
content.add(spot)
// ── Image-based lighting (IBL) — environment mood ───────────────
if let env = try? await EnvironmentResource(named: "outdoor_sunny") {
content.environment.lighting.resource = env
content.environment.lighting.intensityExponent = 1.0
}// SimpleMaterial — flat color, fast, good for prototyping
SimpleMaterial(color: .blue, isMetallic: false)
SimpleMaterial(color: .yellow, roughness: 0.3, isMetallic: true)
// PhysicallyBasedMaterial — realistic, responds to lighting
var pbr = PhysicallyBasedMaterial()
pbr.baseColor = .init(tint: .green)
pbr.baseColor = .init(texture: .init(try! TextureResource.load(named: "hero_albedo")))
pbr.roughness = .init(floatLiteral: 0.4) // 0=mirror, 1=chalk
pbr.metallic = .init(floatLiteral: 0.0) // 0=plastic, 1=chrome
pbr.normal = .init(texture: .init(try! TextureResource.load(named: "hero_normal")))
pbr.emissiveColor = .init(color: .orange) // glows in darkness
// UnlitMaterial — ignores lighting, always same brightness
// Good for: sky, glow effects, UI planes
UnlitMaterial(color: .white)
// VideoMaterial — plays video on surface (TV screens, cutscenes)
let player = AVPlayer(url: videoURL)
VideoMaterial(avPlayer: player)
// Apply to entity
entity.model?.materials = [pbr]Audio that comes from a specific location in the world — gets quieter with distance, changes with direction.
// Load sound — specify spatial mode
let coinSound = try await AudioFileResource(
named: "coin.wav",
in: Bundle.main,
inputMode: .spatial, // ← positioned in world
loadingStrategy: .preload,
shouldLoop: false
)
let bgMusic = try await AudioFileResource(
named: "bgm.mp3",
in: Bundle.main,
inputMode: .nonSpatial, // ← in your head (music, UI sounds)
loadingStrategy: .stream, // ← stream from disk, don't preload
shouldLoop: true
)
// Play FROM a specific entity — sound comes from that position
coin.playAudio(coinSound) // sounds like it comes from the coin
enemy.playAudio(footstepSound) // sounds like it comes from the enemy
// Tune spatial properties
var spatial = SpatialAudioComponent()
spatial.gain = 0 // 0 dB = normal volume, -6 dB = half
spatial.reverbLevel = -6 // mix of reverb (echo by environment)
enemy.components.set(spatial)
// RealityKit automatically handles:
// ✅ Distance falloff (further = quieter)
// ✅ Doppler effect (moving source = pitch shift)
// ✅ HRTF (directional audio — sounds from left/right/behind)// On collision:
_ = content.subscribe(to: CollisionEvents.Began.self) { event in
if event.entityA.components[CoinComponent.self] != nil {
event.entityA.playAudio(coinSound) // plays from coin position
}
}
// On animation event — from inside a System:
class PlayerMovementSystem: System {
func update(context: SceneUpdateContext) {
// ...movement...
if justLanded {
entity.playAudio(landSound)
}
}
}Human player (actual person) In-game character (Mario)
──────────────────────────── ───────────────────────────────────
Interacts with: Interacts with:
SwiftUI buttons, menus In-world 3D entities
Text fields for name entry Coins, enemies, blocks
Pause/resume controls Physics world
Score display Trigger volumes
Lives in: Lives in:
ZStack overlay (Layer 1) ECS / 3D world (Layer 3)
Pure SwiftUI Entity + Components + Systems
ZStack {
RealityView { ... } // 3D world fills screen
.ignoresSafeArea()
// HUD floats on top — pure SwiftUI
VStack {
HStack {
Text("Score: \(gameState.score)").font(.title.bold()).foregroundColor(.white)
Spacer()
ForEach(0..<gameState.lives, id: \.self) { _ in
Image(systemName: "heart.fill").foregroundColor(.red)
}
}
.padding()
.background(.black.opacity(0.4))
Spacer()
}
// Pause menu
if gameState.phase == .paused {
PauseMenuView(gameState: gameState)
}
}// Floating label above enemy's head — moves with the enemy
let label = Entity()
label.components.set(ViewAttachmentComponent(view: AnyView(
Text("Goomba")
.padding(4)
.background(.black.opacity(0.7))
.foregroundColor(.white)
.cornerRadius(4)
)))
label.position = [0, 2.0, 0] // 2m above enemy origin
enemy.addChild(label) // follows enemy automaticallystruct PauseMenuView: View {
var gameState: GameState
@State private var playerName = ""
var body: some View {
ZStack {
Color.black.opacity(0.6).ignoresSafeArea()
VStack(spacing: 24) {
Text("PAUSED").font(.system(size: 48, weight: .black))
Text("Score: \(gameState.score)").foregroundColor(.yellow)
// Text input — human types their name
TextField("Your name", text: $playerName)
.textFieldStyle(.roundedBorder)
.frame(width: 250)
Button("Resume") { gameState.phase = .playing }
.buttonStyle(.borderedProminent)
Button("Restart") { gameState.reset() }
.buttonStyle(.bordered)
}
.padding(48)
.background(.ultraThinMaterial)
.cornerRadius(24)
}
}
}A child entity participates in physics only if it has CollisionComponent + PhysicsBodyComponent. UI labels never have these — they're automatically excluded:
// This label has NO CollisionComponent → completely invisible to physics
let healthBar = ModelEntity(mesh: .generatePlane(width: 1, height: 0.1),
materials: [UnlitMaterial(color: .red)])
// healthBar.components.set(CollisionComponent(...)) ← NOT added
// healthBar.components.set(PhysicsBodyComponent(...)) ← NOT added
enemy.addChild(healthBar) // follows enemy, ignored by physics ✅Main thread (MainActor) RealityKit render thread
──────────────────────── ─────────────────────────────
SwiftUI views & @State System.update(context:)
content.add(entity) Physics simulation
entity.removeFromParent() Collision detection
entity.move(to:duration:) SceneEvents.Update callbacks
CollisionEvents callbacks
Loading completions (await)
// ✅ Systems run on render thread — safe to read/write components
class CoinSystem: System {
func update(context: SceneUpdateContext) {
var comp = entity.components[CoinComponent.self]!
comp.bobTimer += Float(context.deltaTime) // ✅ fine
entity.components.set(comp) // ✅ fine
}
}
// ❌ Never touch SwiftUI state from a System
class BadSystem: System {
func update(context: SceneUpdateContext) {
gameState.score += 100 // ❌ CRASH — @Observable on main thread
}
}
// ✅ To update SwiftUI from 3D world — use component flag + collision event
// Mark in component (render thread safe):
comp.justCollected = true
// Collision event fires on main thread — update state there:
_ = content.subscribe(to: CollisionEvents.Began.self) { event in
gameState.collectCoin() // ✅ collision events fire on main thread
}
// ✅ To update UI from an async Task:
Task { @MainActor in
coin.removeFromParent()
gameState.score += 100
}enum AssetCache {
// ── Models ─────────────────────────────────────────────────────
static private(set) var mario: Entity?
static private(set) var goomba: Entity?
static private(set) var coin: Entity?
// ── Sounds ─────────────────────────────────────────────────────
static private(set) var coinSound: AudioFileResource?
static private(set) var jumpSound: AudioFileResource?
static private(set) var stompSound: AudioFileResource?
// Call this ONCE at scene setup — before building the level
static func preload() async throws {
async let m = ModelEntity(named: "mario.usdz")
async let g = ModelEntity(named: "goomba.usdz")
async let c = ModelEntity(named: "coin.usdz")
async let cs = AudioFileResource(named: "coin.wav", in: Bundle.main,
inputMode: .spatial, loadingStrategy: .preload, shouldLoop: false)
async let js = AudioFileResource(named: "jump.wav", in: Bundle.main,
inputMode: .spatial, loadingStrategy: .preload, shouldLoop: false)
(mario, goomba, coin) = try await (m, g, c)
(coinSound, jumpSound) = try await (cs, js)
}
// Spawn a new enemy (clone = near instant, no I/O)
static func spawnGoomba(at position: SIMD3<Float>) -> Entity? {
guard let template = goomba else { return nil }
let clone = template.clone(recursive: true)
clone.position = position
clone.components.set(EnemyComponent()) // fresh component per clone
return clone
}
}struct PatrolComponent: Component {
var waypoints: [SIMD3<Float>] = []
var currentIndex: Int = 0
var speed: Float = 1.5
var arrivalThreshold: Float = 0.3
}
class PatrolSystem: System {
static let query = EntityQuery(where: .has(PatrolComponent.self))
required init(scene: RealityKit.Scene) {}
func update(context: SceneUpdateContext) {
for entity in context.entities(matching: Self.query, updatingSystemWhen: .rendering) {
var p = entity.components[PatrolComponent.self]!
let target = p.waypoints[p.currentIndex]
let toTarget = target - entity.position(relativeTo: nil)
if length(toTarget) < p.arrivalThreshold {
p.currentIndex = (p.currentIndex + 1) % p.waypoints.count
} else {
entity.position += normalize(toTarget) * p.speed * Float(context.deltaTime)
entity.look(at: target, from: entity.position(relativeTo: nil), relativeTo: nil)
}
entity.components.set(p)
}
}
}struct SpawnTriggerComponent: Component {
var radius: Float = 8.0
var hasSpawned: Bool = false
var enemyCount: Int = 3
}
class SpawnSystem: System {
static let triggerQuery = EntityQuery(where: .has(SpawnTriggerComponent.self))
static let charQuery = EntityQuery(where: .has(CharacterComponent.self))
required init(scene: RealityKit.Scene) {}
func update(context: SceneUpdateContext) {
guard let character = context.entities(matching: Self.charQuery,
updatingSystemWhen: .rendering).first
else { return }
let charPos = character.position(relativeTo: nil)
for trigger in context.entities(matching: Self.triggerQuery, updatingSystemWhen: .rendering) {
var comp = trigger.components[SpawnTriggerComponent.self]!
guard !comp.hasSpawned else { continue }
if length(charPos - trigger.position(relativeTo: nil)) < comp.radius {
for _ in 0..<comp.enemyCount {
let offset = SIMD3<Float>(Float.random(in: -3...3), 0, 0)
if let enemy = AssetCache.spawnGoomba(at: trigger.position(relativeTo: nil) + offset) {
context.scene.anchors.append(AnchorEntity(world: .zero))
// add to scene — need reference to content
}
}
comp.hasSpawned = true
trigger.components.set(comp)
}
}
}
}func spawnAheadOfCamera(_ camera: Entity, distance: Float = 20) -> SIMD3<Float> {
let forward = camera.transform.matrix.columns.2 // -Z column = forward
let fwd = -normalize(SIMD3<Float>(forward.x, 0, forward.z)) // flatten to XZ
let lateral = Float.random(in: -8...8)
let right = normalize(cross(fwd, [0, 1, 0]))
return camera.position(relativeTo: nil) + fwd * distance + right * lateral
}let distance = length(enemy.position(relativeTo: nil) - character.position(relativeTo: nil))
let speed: Float = 3.0
let timeToArrive = distance / speed // seconds
// "Enemy will reach player in \(timeToArrive) seconds"Instant actions: Long-running actions:
──────────────────────────── ──────────────────────────────────────
coin pop, block bounce platform moving back and forth
entity.move(to:duration:) enemy patrol loop
+ Task.sleep to clean up while true { move, sleep, move, sleep }
enemy death squash countdown timer (HoldInteractComponent)
entity.scale change per-frame Component accumulation
sound plays once background music (shouldLoop: true)
entity.playAudio(sound) attached to scene root entity
func animateCoinCollection(_ coin: Entity) {
var up = coin.transform
up.translation.y += 1.5
coin.move(to: up, relativeTo: nil, duration: 0.3, timingFunction: .easeOut)
Task { @MainActor in
try await Task.sleep(for: .milliseconds(350))
coin.removeFromParent()
}
}// Starts when entity is created, loops forever
func startPlatformOscillation(_ platform: Entity, from startX: Float, range: Float = 4) {
Task { @MainActor in
while !Task.isCancelled {
var right = platform.transform
right.translation.x = startX + range
platform.move(to: right, relativeTo: nil, duration: 2.0, timingFunction: .easeInOut)
try await Task.sleep(for: .seconds(2.1))
var left = platform.transform
left.translation.x = startX
platform.move(to: left, relativeTo: nil, duration: 2.0, timingFunction: .easeInOut)
try await Task.sleep(for: .seconds(2.1))
}
}
}struct HoldInteractComponent: Component {
var requiredHoldTime: Float = 2.0
var currentHoldTime: Float = 0
var isBeingHeld: Bool = false // set by collision system, cleared by hold system
var isCompleted: Bool = false
}
class HoldInteractionSystem: System {
static let query = EntityQuery(where: .has(HoldInteractComponent.self))
required init(scene: RealityKit.Scene) {}
func update(context: SceneUpdateContext) {
let dt = Float(context.deltaTime)
for entity in context.entities(matching: Self.query, updatingSystemWhen: .rendering) {
var comp = entity.components[HoldInteractComponent.self]!
guard !comp.isCompleted else { continue }
if comp.isBeingHeld {
comp.currentHoldTime += dt
if comp.currentHoldTime >= comp.requiredHoldTime {
comp.isCompleted = true
entity.playAudio(activateSound) // sound at entity position
}
} else {
comp.currentHoldTime = max(0, comp.currentHoldTime - dt * 2) // decay
}
comp.isBeingHeld = false // reset; collision will re-set next frame if still touching
entity.components.set(comp)
}
}
}// ✅ Extension on Entity — adds OOP-style methods to any entity
extension Entity {
// Works only if entity has CharacterComponent
func jump() {
guard var comp = components[CharacterComponent.self],
comp.isOnGround else { return }
comp.velocity.y = comp.jumpImpulse
comp.isOnGround = false
components.set(comp)
}
func kill() {
removeFromParent()
}
// Computed properties feel natural
var isCharacter: Bool { components[CharacterComponent.self] != nil }
var isCoin: Bool { components[CoinComponent.self] != nil }
var isEnemy: Bool { components[EnemyComponent.self] != nil }
var isOnGround: Bool {
get { components[CharacterComponent.self]?.isOnGround ?? false }
set {
guard var comp = components[CharacterComponent.self] else { return }
comp.isOnGround = newValue
components.set(comp)
}
}
}
// Now reads naturally:
mario.jump()
enemy.kill()
if mario.isOnGround { ... }// Gives you a "Mario type" feel without subclassing Entity
enum Mario {
static let query = EntityQuery(where: .has(CharacterComponent.self))
static func make() async throws -> Entity { ... }
static func respawn(_ entity: Entity, at position: SIMD3<Float>) {
entity.position = position
var comp = entity.components[CharacterComponent.self]!
comp.velocity = .zero
entity.components.set(comp)
}
}
// Usage:
let mario = try await Mario.make()
Mario.respawn(mario, at: [0, 1, 0])struct EnemyComponent: Component {
enum State { case patrolling, chasing, fleeing, dead }
var state: State = .patrolling
// State transitions defined here as simple functions
mutating func transition(to newState: State) {
guard canTransition(to: newState) else { return }
state = newState
}
private func canTransition(to newState: State) -> Bool {
switch (state, newState) {
case (.dead, _): return false // dead stays dead
default: return true
}
}
}// Post an event from anywhere (render thread safe — just setting a component flag)
comp.justDied = true // set in System
// Collect events in GameScene's SceneEvents.Update subscription (main thread)
_ = content.subscribe(to: SceneEvents.Update.self) { [weak self] _ in
self?.processEvents(gameState: gameState)
}
func processEvents(gameState: GameState) {
// Check component flags, update GameState (main thread safe)
}class EntityPool {
private var available: [Entity] = []
private let factory: () -> Entity
init(size: Int, factory: @escaping () -> Entity) {
self.factory = factory
available = (0..<size).map { _ in factory() }
}
func acquire(at position: SIMD3<Float>) -> Entity {
let entity = available.isEmpty ? factory() : available.removeLast()
entity.position = position
entity.isEnabled = true
return entity
}
func release(_ entity: Entity) {
entity.isEnabled = false // hide but don't destroy
available.append(entity)
}
}
// Usage for bullet spam:
let bulletPool = EntityPool(size: 50) { BulletFactory.make() }
let bullet = bulletPool.acquire(at: mario.position)
scene.add(bullet)
// On hit:
bulletPool.release(bullet)struct HealthComponent: Component {
var max: Int
var current: Int
var isInvincible: Bool = false
var invincibilityTimer: Float = 0 // seconds remaining
var isDead: Bool { current <= 0 }
var fraction: Float { Float(current) / Float(max) }
mutating func takeDamage(_ amount: Int) {
guard !isInvincible else { return }
current = max(0, current - amount)
isInvincible = true
invincibilityTimer = 1.5 // 1.5 second grace period
}
}
class HealthSystem: System {
static let query = EntityQuery(where: .has(HealthComponent.self))
required init(scene: RealityKit.Scene) {}
func update(context: SceneUpdateContext) {
let dt = Float(context.deltaTime)
for entity in context.entities(matching: Self.query, updatingSystemWhen: .rendering) {
var health = entity.components[HealthComponent.self]!
if health.isInvincible {
health.invincibilityTimer -= dt
if health.invincibilityTimer <= 0 {
health.isInvincible = false
entity.isEnabled = true // stop blinking
}
// Blink effect during invincibility
entity.isEnabled = Int(health.invincibilityTimer * 10) % 2 == 0
}
entity.components.set(health)
}
}
}// Create vectors
let pos: SIMD3<Float> = [1, 2, 3]
let pos = SIMD3<Float>(x: 1, y: 2, z: 3)
let pos = SIMD3<Float>.zero
// Math
let sum = a + b
let diff = a - b
let scaled = a * 2.0
let distance = length(a - b) // distance between two points
let direction = normalize(b - a) // unit vector from a to b
let dot = dot(a, b) // dot product
let cross = cross(a, b) // perpendicular vector
// Lerp
let mid = a + (b - a) * t // t: 0.0=a, 1.0=b
// Rotation
let q = simd_quatf(angle: .pi / 2, axis: [0, 1, 0]) // 90° around Y
entity.orientation = q
// Face toward a target
entity.look(at: target, from: entity.position(relativeTo: nil), relativeTo: nil)
// World position (ignoring parent transforms)
let worldPos = entity.position(relativeTo: nil)MeshResource.generateBox(size: [1, 1, 1])
MeshResource.generateBox(size: [1, 1, 1], cornerRadius: 0.1) // rounded
MeshResource.generateSphere(radius: 0.5)
MeshResource.generateCylinder(height: 2, radius: 0.3)
MeshResource.generateCone(height: 1, radius: 0.5)
MeshResource.generatePlane(width: 10, depth: 10)
MeshResource.generateText("Hello", extrusionDepth: 0.05,
font: .systemFont(ofSize: 0.3))ShapeResource.generateBox(size: [1, 1, 1])
ShapeResource.generateSphere(radius: 0.5)
ShapeResource.generateCapsule(height: 1.8, radius: 0.25) // best for characters
ShapeResource.generateConvexHull(from: meshResource) // wraps complex mesh// Dynamic — moves, affected by gravity and forces
PhysicsBodyComponent(shapes: [...], mass: 70, mode: .dynamic)
// Static — never moves (ground, walls, fixed platforms)
PhysicsBodyComponent(shapes: [...], mass: 0, mode: .static)
// Kinematic — you move it manually (moving platforms, enemies)
PhysicsBodyComponent(shapes: [...], mass: 0, mode: .kinematic)// ❌ Common mistake — modifying a COPY, not the stored component
entity.components[CharacterComponent.self]!.velocity.x += 1 // does nothing!
// ✅ Correct — read, mutate, write back
var comp = entity.components[CharacterComponent.self]!
comp.velocity.x += 1
entity.components.set(comp) // ← must re-set// ❌ Without guard — coin collected many times
_ = content.subscribe(to: CollisionEvents.Began.self) { event in
gameState.score += 100 // fires 3 frames in a row during overlap
}
// ✅ With guard — collected exactly once
_ = content.subscribe(to: CollisionEvents.Began.self) { event in
guard var comp = coin.components[CoinComponent.self],
!comp.isCollected else { return }
comp.isCollected = true
coin.components.set(comp)
gameState.score += 100
}// ❌ Token immediately deallocated — callback never fires
content.subscribe(to: CollisionEvents.Began.self) { _ in ... }
// ✅ Store the token as a property
class GameScene {
private var subscriptions: [any Cancellable] = []
func build(...) {
let sub = content.subscribe(to: CollisionEvents.Began.self) { _ in ... }
subscriptions.append(sub) // kept alive as long as GameScene exists
}
}// ❌ Removing entities while iterating a query is undefined behavior
for entity in context.entities(matching: query, ...) {
entity.removeFromParent() // dangerous
}
// ✅ Collect first, then remove
var toRemove: [Entity] = []
for entity in context.entities(matching: query, ...) {
if shouldRemove(entity) { toRemove.append(entity) }
}
Task { @MainActor in
toRemove.forEach { $0.removeFromParent() }
}// ❌ entity.move() on a dynamic physics body — physics will fight it
dynamicBody.move(to: target, relativeTo: nil, duration: 1.0)
// ✅ Use kinematic mode for entities you move manually
body.components[PhysicsBodyComponent.self]?.mode = .kinematic
body.move(to: target, relativeTo: nil, duration: 1.0)
// Or apply forces instead
body.addForce([0, 500, 0], relativeTo: nil)
body.addImpulse([0, 10, 0], relativeTo: nil) // instantaneous#Preview("Coin") {
RealityView { content in
content.camera = .virtual
Task { @MainActor in
if let mario = try? await MarioFactory.make() {
content.add(mario)
mario.playAnimation(mario.availableAnimations.first!.repeat())
}
}
let light = DirectionalLight()
light.light.intensity = 3000
content.add(light)
}
.realityViewCameraControls(.orbit)
.frame(width: 400, height: 400)
}MyGame/
│
├── MyGameApp.swift ← @main — registers ALL systems & components
├── GameState.swift ← @Observable game-wide state (score, lives, phase)
├── InputState.swift ← singleton read by Systems each frame
│
├── Components/ ← ONLY data, zero logic, all structs
│ ├── CharacterComponent.swift
│ ├── EnemyComponent.swift
│ ├── CoinComponent.swift
│ ├── BlockComponent.swift
│ ├── HealthComponent.swift
│ ├── AnimationStateComponent.swift
│ └── HoldInteractComponent.swift
│
├── Systems/ ← ONLY logic, one concern each
│ ├── CharacterMovementSystem.swift
│ ├── EnemyAISystem.swift
│ ├── AnimationSystem.swift
│ ├── HealthSystem.swift
│ ├── CoinSystem.swift
│ ├── SpawnSystem.swift
│ └── HoldInteractionSystem.swift
│
├── Factories/ ← entity builders, async, return Entity
│ ├── CharacterFactory.swift
│ ├── EnemyFactory.swift
│ ├── CoinFactory.swift
│ ├── BlockFactory.swift
│ └── PipeFactory.swift
│
├── Cache/
│ └── AssetCache.swift ← preload once, clone many
│
└── Views/
├── ContentView.swift ← RealityView + ZStack + key handling
├── GameScene.swift ← builds world, subscribes to events
├── HUDView.swift ← score, lives, timer
├── PauseMenuView.swift ← menu for human player
└── GameOverView.swift
- RealityKit Documentation
- RealityView
- Bringing your SceneKit projects to RealityKit — Sample game project download
- WWDC25: Bring your SceneKit project to RealityKit — Start here — ports a real game, non-AR focused
- WWDC25: What's new in RealityKit — new components, tvOS, post-processing
- WWDC24: Discover RealityKit APIs for iOS, macOS, and visionOS — cross-platform fundamentals
- Reality Composer Pro — Free, ships with Xcode. Visual scene editor, shader graph, timeline authoring
- Blender — Free, best 3D modeling + animation tool. Export to USDZ
- Poly Haven — Free PBR textures and 3D models (CC0 license)
- Sketchfab — 3D model marketplace, many free
# Convert any model to USDZ (ships with Xcode)
xcrun usdz_converter input.fbx output.usdz
xcrun usdz_converter input.obj output.usdzLast updated: June 2026 • Based on RealityKit available in Xcode 26 / macOS Tahoe