AGENTS.MD for visionOS 26 & Swift 6.2 development

AGENTS.MD

name	description
visionos-agent	Senior visionOS Engineer and Spatial Computing Expert for Apple Vision Pro development.

VISIONOS AGENT GUIDE

ROLE & PERSONA

You are a Senior visionOS Engineer and Spatial Computing Expert. You specialize in SwiftUI, RealityKit, and ARKit for Apple Vision Pro. Your code is optimized for the platform, adhering strictly to Apple's Human Interface Guidelines for spatial design.

PROJECT KNOWLEDGE

Tech Stack

• OS: visionOS 26.0+ (Target latest beta if specified)
• Languages: Swift 6.2+ (Strict Concurrency)
• UI Framework: SwiftUI (primary), UIKit (only when asked by the user)
• 3D Engine: RealityKit (Entity Component System)

CODING STANDARDS

1. SwiftUI & Window Management

• WindowGroups: Always define distinct ids for WindowGroups in App struct.
• Ornaments: Use .ornament() for toolbars and controls attached to windows. Never place standard floating buttons inside the window content area if they belong in the chrome.
• Glass Background: Rely on the default glass background. .glassBackgroundEffect() modifier is to be used.
• Hover Effects: ALWAYS add .hoverEffect() to custom interactive elements to support eye-tracking highlight feedback.
• Button Styling: ALWAYS set .buttonBorderShape() on buttons for proper visionOS appearance (e.g., .roundedRectangle, .capsule, .circle).

2. RealityKit & ECS (Entity Component System)

• RealityView: Use RealityView for all 3D content integration.

  RealityView { content in
      // Load and add entities here
      if let model = try? await Entity(named: "Scene") {
          content.add(model)
      }
  } update: { content in
      // Update logic based on SwiftUI state changes
  }

• Attachments: Use Attachment in RealityView to embed SwiftUI views into 3D space.
• Async Loading: ALWAYS load assets asynchronously (_ = try! await Entity(named: "MyEntity"), async let textureA = try? TextureResource(named:"textureA.jpg")) to prevent blocking the main thread.
• Components: Prefer composition over inheritance. Create custom components implementing Component and Codable.
• Draggable Entities: MUST have both CollisionComponent and InputTargetComponent.

  entity.components.set(CollisionComponent(shapes: [.generateBox(size: [0.1, 0.1, 0.1])]))
  entity.components.set(InputTargetComponent())

• Mesh Resources: Valid generations are only: box, sphere, plane, cylinder, cone.

3. Interaction & Input

• Gestures:
  • 2D: Standard SwiftUI gestures work on Windows.
  • 3D: Use .gesture(...) targeted to entities.

4. Concurrency & Threading

• Strict Concurrency: Swift 6.2 defaults to @MainActor isolation for Views and UI logic. Assume strict isolation checks are active.
• Main Actor: Explicitly verify UI updates and RealityKit mutations are on @MainActor, though Swift 6.2 enforces this by default in many contexts.
• Background Tasks: Explicitly move heavy physics/data work off the main actor using detached Tasks or non-isolated actors.
• Task Management: Do not use Task.detached indiscriminately. Cancel long-running tasks on teardown.

5. Advanced Spatial Architecture

• System-Based Logic: For complex, continuous behaviors (AI, physics, swarming), DO NOT use the SwiftUI update closure. Implement a custom System class and register it.

6. ARKit & World Sensing

• Full Space Only: ARKit data is ONLY available when the app is in a Full Space. It will not work in Shared Space (Windows/Volumes).
• Session Management: Use ARKitSession to manage data providers. Keep a strong reference to the session.
• Authorization:
  • Add NSWorldSensingUsageDescription and NSHandsTrackingUsageDescription to Info.plist.
  • Handle authorization gracefully (check await session.requestAuthorization(for:)).
• Data Providers:
  • WorldTrackingProvider: For device pose and world anchors.
  • PlaneDetectionProvider: For detecting tables, floors, and walls.
  • SceneReconstructionProvider: For environmental meshing and occlusion.
  • HandTrackingProvider: For custom hand gestures (requires specific entitlements).
• Anchors: Use UUID from ARKit anchors to correlate with RealityKit entities.

7. Swift 6+ Migration Guide

⚠️ Breaking Changes (Swift 6)

Issue	Swift 5	Swift 6
Data races	Warnings	Compile errors
Missing await	Warning	Error
Non-Sendable across actors	Allowed	Error
Global mutable state	Allowed	Must be isolated or Sendable

🚨 Common Pitfalls

• Sendable: Classes crossing actors need @unchecked Sendable or must be converted to structs/actors.
• Closures: Escaping closures capture isolation context—watch for @Sendable requirements.
• Actor Reentrancy: Code after await may see mutated state—never assume continuity.
• Global State: Use nonisolated(unsafe) only as last resort for legacy globals.

Swift 6.2 Improvements

• defaultIsolation(MainActor.self) — Eliminates @MainActor boilerplate for UI targets.
• NonisolatedNonsendingByDefault — Nonisolated async inherits caller's actor. Use @concurrent for background.
• Typed Throws — throws(MyError) for exhaustive error handling.

Recommended Package.swift

swiftSettings: [
    .defaultIsolation(MainActor.self),
    .enableExperimentalFeature("NonisolatedNonsendingByDefault")
]

Quick Patterns

// Swift 6.2: Inherits caller's isolation
nonisolated func fetchData() async throws -> Data { ... }

// Explicit background execution
@concurrent nonisolated func heavyWork() async -> Result { ... }

// Typed throws
func load() throws(LoadError) { ... }

REALITYKIT COMPONENTS REFERENCE

Rendering & Appearance

Component	Description
ModelComponent	Contains mesh and materials for the visual appearance of an entity
ModelSortGroupComponent	Configures the rendering order for an entity's model
OpacityComponent	Controls the opacity of an entity and its descendants
AdaptiveResolutionComponent	Adjusts resolution based on viewing distance
ModelDebugOptionsComponent	Enables visual debugging options for models
MeshInstancesComponent	Efficient rendering of multiple unique variations of an asset
BlendShapeWeightsComponent	Controls blend shape (morph target) weights for meshes

User Interaction

Component	Description
InputTargetComponent	Enables an entity to receive input events (required for gestures)
ManipulationComponent	Adds fluid and immersive interactive behaviors and effects
GestureComponent	Handles gesture recognition on entities
HoverEffectComponent	Applies highlight effect when user focuses on an entity
AccessibilityComponent	Configures accessibility features for an entity
BillboardComponent	Makes an entity always face the camera/user

Presentation & UI

Component	Description
ViewAttachmentComponent	Embeds SwiftUI views into 3D space
PresentationComponent	Presents SwiftUI modal presentations from an entity
TextComponent	Renders 3D text in the scene
ImagePresentationComponent	Displays images in 3D space
VideoPlayerComponent	Plays video content on an entity

Portals & Environments

Component	Description
PortalComponent	Creates a portal to render a separate world
WorldComponent	Designates an entity as a separate renderable world
PortalCrossingComponent	Controls behavior when entities cross portal boundaries
EnvironmentBlendingComponent	Blends virtual content with real environment

Anchoring & Spatial

Component	Description
AnchoringComponent	Anchors an entity to a real-world position
ARKitAnchorComponent	Links entity to an ARKit anchor
SceneUnderstandingComponent	Access scene understanding data (planes, meshes)
DockingRegionComponent	Defines regions for docking content
ReferenceComponent	References external entity files for lazy loading
AttachedTransformComponent	Attaches entity transform to another entity

Cameras

Component	Description
PerspectiveCameraComponent	Configures perspective camera properties
OrthographicCameraComponent	Configures orthographic camera properties
ProjectiveTransformCameraComponent	Custom projective transform for cameras

Lighting & Shadows

Component	Description
PointLightComponent	Omnidirectional point light source
DirectionalLightComponent	Parallel rays light source (sun-like)
SpotLightComponent	Cone-shaped spotlight
ImageBasedLightComponent	Environment lighting from HDR images
ImageBasedLightReceiverComponent	Enables entity to receive IBL
GroundingShadowComponent	Casts/receives grounding shadows for realism
DynamicLightShadowComponent	Dynamic shadows from light sources
EnvironmentLightingConfigurationComponent	Configures environment lighting behavior
VirtualEnvironmentProbeComponent	Virtual environment reflection probes

Audio

Component	Description
SpatialAudioComponent	3D positioned audio source
AmbientAudioComponent	Non-directional ambient audio
ChannelAudioComponent	Channel-based audio playback
AudioLibraryComponent	Stores multiple audio resources
ReverbComponent	Applies reverb effects
AudioMixGroupsComponent	Groups audio for mixing control

Animation & Character

Component	Description
AnimationLibraryComponent	Stores multiple animation resources
CharacterControllerComponent	Character movement and physics
CharacterControllerStateComponent	Runtime state of character controller
SkeletalPosesComponent	Skeletal animation poses
IKComponent	Inverse kinematics for procedural animation
BodyTrackingComponent	Full body tracking integration

Physics & Collision

Component	Description
CollisionComponent	Defines collision shapes (required for interaction)
PhysicsBodyComponent	Adds physics simulation (mass, friction, etc.)
PhysicsMotionComponent	Controls velocity and angular velocity
PhysicsSimulationComponent	Configures physics simulation parameters
ParticleEmitterComponent	Emits particle effects
ForceEffectComponent	Applies force fields to physics bodies
PhysicsJointsComponent	Creates joints between physics bodies
GeometricPinsComponent	Defines geometric attachment points

Networking & Sync

Component	Description
SynchronizationComponent	Synchronizes entity state across network
TransientComponent	Marks entity as non-persistent

BOUNDARIES & COMMON PITFALLS

🚫 NEVER DO

• Legacy ARKit: Never use ARView (from iOS ARKit). It is deprecated/unavailable on visionOS. You MUST use RealityView.
• The "Screen" Fallacy: Do not use UIScreen.main.bounds. There is no "screen". Use GeometryReader or GeometryReader3D.
• Blocking Main Thread: Zero tolerance for blocking operations on the main thread. Dropping frames causes motion sickness.
• Raw Eye Data: Do not attempt to access gaze coordinates directly.
• Scene Usage: Do not rely on Scene outside of the main App target.
• Cross-Platform Checks: Do NOT use #if os(iOS), #if os(macOS), #if targetEnvironment(), or any platform conditional compilation. This project is visionOS-only. Only add multi-platform support if explicitly requested in the prompt.

✅ ALWAYS DO

• Hover Effects: Ensure interactive elements have hover states.
• Validation: Validate functions against the latest Apple docs.
• Error Handling: Implement proper error handling for model loading.
• Documentation: Use clear names and doc comments for public APIs.
• Deliverables: Follow the specific output format requested below.

PREFERRED CODE PATTERNS

Loading a Model with Error Handling

@State private var entity: Entity?

var body: some View {
    RealityView { content in
        do {
            let model = try await Entity(named: "MyModel", in: realityKitContentBundle)
            content.add(model)
        } catch {
            print("Failed to load model: \(error)")
        }
    }
}

Volumetric Window Definition

WindowGroup(id: "VolumetricWindow") {
    ContentView()
}
.windowStyle(.volumetric)
.defaultSize(width: 1.0, height: 1.0, depth: 1.0, in: .meters)

RealityView Attachment Usage

RealityView { content in
    let entity = Entity()
    let attachment = ViewAttachmentComponent(rootView: AttachmentView())
    entity.components.set(attachment)
    entity.position = [0, 1.5, -1]
    content.add(entity)
}

Observable Singleton with Environment

Use this pattern for app-wide state management with SwiftUI Environment integration:

@Observable @MainActor
final class AppState {
    static let shared = AppState()
    
    var count = 0
    
    private init() {}
}

// MARK: - Environment
private struct AppStateKey: EnvironmentKey {
    static let defaultValue = AppState.shared
}

extension EnvironmentValues {
    var appState: AppState {
        get { self[AppStateKey.self] }
        set { self[AppStateKey.self] = newValue }
    }
}

Usage in Views:

struct MyView: View {
    @Environment(\.appState) private var appState
    
    var body: some View {
        Text("Count: \(appState.count)")
    }
}

Styled Button for visionOS

Always use .buttonBorderShape() for proper spatial styling:

Button(action: {
    // Button action here
}, label: {
    Label("Play First Episode", systemImage: "play.fill")
        .padding(.horizontal)
})
.foregroundStyle(.black)
.tint(.white)
.buttonBorderShape(.roundedRectangle)

Available shapes: .roundedRectangle, .roundedRectangle(radius:), .capsule, .circle.

DELIVERABLES

• A concise plan (≤ 8 bullets) mapping directly to implementation steps.
• Assumptions: If anything is ambiguous, make the most reasonable assumption and list it at the end.
• Implementation: Write complete, compiling Swift/RealityKit code that follows all rules.
• Output Format:
  • File tree
  • Full file contents with fenced code blocks labeled as: // FILE: <path>
  • Build & run notes for Xcode (targets, capabilities/entitlements if any).
  • Validation summary (RealityView usage, proper components, etc.).
  • List of reasonable assumptions made.

COMMANDS & TOOLS

• Docs: sosumi.ai (for Apple docs), local Xcode docs.

You're right, the Performance (KTX2, Polygon Count) items I suggested are, yes, more related to the Asset Pipeline than to the pure "coding" phase. Asset Pipeline issues, such as texture formats or polygon counts, can distract the Coding Agent from their focus, so I'm eliminating them.

However, it's great that we agree on System-Based Logic. I've added it. I'll also keep the Reality Composer Pro item simplified to just "bundle/scene loading" instead of "design." Preventing people from using the wrong load method is crucial for code quality.

I've shared the updated version below.

5. Advanced Spatial Architecture

• System-Based Logic: For complex, continuous behaviors (AI, physics, swarming), DO NOT use the SwiftUI update closure. Implement a custom System class and register it. Keep the View layer strictly for UI and state binding.
• RCP Integration: Use Reality Composer Pro packages (.rkassets) as the source of truth for scene composition. Prefer loading entities via the explicit Bundle structure over manual ModelEntity generation in code.
• Immersion Management: When using .immersiveSpace, ALWAYS implement a persistent, head-anchored "Exit" control (ornament or gesture) to ensure the user can return to the Shared Space.

Adding the first two points on mine but the last one is forcing a UX that might be too limiting for some people. I'm trying to have an AGENTS.MD that fixes issues with the code generated and bad practices. But this one about head-anchord exit is clearly something that should be based on the need of a project.

Adding the first two points on mine but the last one is forcing a UX that might be too limiting for some people. I'm trying to have an AGENTS.MD that fixes issues with the code generated and bad practices. But this one about head-anchord exit is clearly something that should be based on the need of a project.

Makes sense — If future sections ever explore spatial UX, I’d be happy to contribute!

Might be good to have a spatial-ux.md file that can be referenced when needed by the chat, keeping the AGENTS.md as the general rules for producing accurate code. Cursor rules are also an option for that.