Spatial Intelligence Engine

The Spatial Intelligence Engine, or SIE, is the spatial layer behind WorldAgents.

The useful split is simple:

• WorldAgents builds and runs agents.
• SIE provides spaces and spatial context about the places where those agents operate.

In this repo, that shows up through spaces, places, simulations, splat assets, semantic layers, visual recognition pipelines, localization results, and vendor spatial-runtime integrations.

Current Product Surface

WorldAgents currently uses spatial context in a few practical ways:

• places describe reusable environments inside an organization
• spaces are editable SIE environments for the semantic layer behind a place
• splat assets provide 3D scene geometry for simulator sessions
• simulations connect agents to environments, start poses, movement settings, and triggers
• optional visual pipelines can sample live or simulated video frames
• the Inspector can expose recognition, localization, and debug evidence for supported pipelines

The agent does not need raw spatial files in its prompt. It needs concise, current context such as what space is active, what object was recognized, what semantic node matched, or what the simulator/session state indicates.

Spaces and Semantic Layers

A space is the editable SIE environment for a real or simulated place. It can be created from a video capture, imported from an existing semantic layer JSON file, or started from scratch.

Each space holds a semantic layer: named rooms, zones, objects, landmarks, routes, and relationships that make the environment understandable to agents. Agents use that layer through SIE context and tools instead of guessing from pixels alone.

Teams can open a space in the SIE space editor to inspect the environment, rename the space, and edit the semantic layer after creation. A video-generated space can therefore become a reviewed, structured environment before agents use it.

This lets a team reuse the same scene across multiple agents or simulations. For example, one store interior, museum floor, worksite, or training area can support a guide agent, an audit agent, and a training agent.

Recognition and Localization

Supported visual pipelines can process sampled frames outside the normal model turn. A pipeline may return recognized objects, localization state, confidence, and debug imagery.

WorldAgents stores the latest useful result on the session and can pass bounded context to the agent. Deeper evidence is for the Inspector and logs, not for normal user-facing responses.

Runtime Flow

  camera, glasses, simulator, or uploaded frame
  -> WorldAgents session
  -> optional visual pipeline or spatial runtime
  -> concise spatial context
  -> agent response or tool call

Recognition and localization do not need to block every response. A session can continue with voice, text, and tools while spatial context updates asynchronously.

Direction

The broader SIE direction is to turn captures, maps, semantic layers, scene assets, and review workflows into reliable agent context. Spaces are part of that product surface now, even though the feature is still WIP.

The near-term product goal is narrower: make spatial context available where it improves an agent's behavior, keep evidence inspectable, and avoid forcing users or clients to handle low-level scene files directly.