Skip to content

Quick Start

This guide walks you through running RTSM and making your first semantic query.

1. Start RTSM

Start the main service:

python -m rtsm

This launches:

Service Address
REST API http://localhost:8002
WebSocket (visualization) ws://localhost:8083/ws
MCP (if enabled) http://localhost:8002/mcp/sse

RTSM listens for RGB-D frames via the configured receiver (WebSocket from Calabi Lens, or ZeroMQ from RealSense + RTABMap).

Replay Mode

To replay a recorded session without a live camera:

python -m rtsm --replay recordings/session1

2. Verify It's Running

curl http://localhost:8002/healthz

{"status": "ok"}

Check detailed stats:

curl http://localhost:8002/stats/detailed

3. List Detected Objects

Once frames are streaming, objects will appear in memory:

curl http://localhost:8002/objects

Response:

{
  "count": 62,
  "objects": [
    {
      "id": "a3f2c1d8",
      "xyz_world": [1.2, 0.4, 2.1],
      "stability": 0.82,
      "hits": 15,
      "confirmed": true,
      "label_primary": "backpack",
      "view_bins": 3
    }
  ]
}

Ask natural language queries:

curl "http://localhost:8002/search/semantic?query=red%20mug&top_k=5"

Response:

{
  "query": "red mug",
  "results": [
    {
      "id": "b7d4e2f1",
      "score": 0.82,
      "label_hint": "mug",
      "confirmed": true,
      "xyz_world": [0.8, 0.2, 1.5]
    }
  ]
}

Find objects near a 3D point:

curl "http://localhost:8002/search/spatial?x=1.0&y=0.5&z=2.0&radius_m=0.5"

6. View in 3D (Optional)

Open the visualization frontend in your browser. The 3D viewer connects to the WebSocket at ws://localhost:8083/ws and shows a live point cloud with detected objects overlaid.

7. Record and Replay Sessions

Record a live session for later replay and benchmarking:

# Record while running pipeline
python -m rtsm --record recordings/my_session

# Record without GPU (raw frame capture only)
python -m rtsm --record recordings/my_session --record-only

Replay a recorded session:

python -m rtsm --replay recordings/session1

This feeds the recorded frames through the full pipeline at the original recording rate — no camera hardware needed. See the Record & Replay Guide for details.

8. Check Analytics (Optional)

While a session is running (live or replay), view runtime analytics:

# Per-stage latency breakdown
curl http://localhost:8002/stats/detailed

# Segmentation analytics (mask counts, confirmation rates)
curl http://localhost:8002/analytics/segmentation

The analytics dashboard is also available in the 3D visualization frontend as a separate tab. See the Analytics Dashboard Guide for details.

Next Steps