NVIDIA has released DeepStream 9.1. The update introduces two new capabilities designed to solve long-standing issues in video analytics: Multi-View 3D Tracking (MV3DT) and AutoMagicCalib (AMC). Both function as agentic skills for coding agents, allowing developers to move from concept to a running pipeline more quickly.
What is DeepStream 9.1
DeepStream is NVIDIA’s toolkit for streaming analytics based on AI. It offers a GStreamer framework for multi-stream, multi-model inference on NVIDIA GPUs. Pipelines combine hardware-accelerated decoding and encoding, TensorRT inference, object tracking, and message-broker integration.
Version 9.1 adds five notable items:
- 13 agentic skills for coding agents.
- The MV3DT skill for cross-camera tracking.
- The AMC skill for automatic calibration.
- NVIDIA JetPack 7.2 support for Jetson Orin and Thor edge devices.
- A unified open-source GitHub repository under CC-BY-4.0 and Apache-2.0.
How MV3DT tracks objects across cameras
MV3DT is the primary skill in this update. It projects detections from multiple calibrated cameras into a shared 3D coordinate system. This allows the system to associate observations of the same object across different views and assign one globally consistent object ID.
The data flow runs in four stages. For detection, each camera stream runs an object detector. MV3DT supports three models out of the box:
- PeopleNetTransformer: a transformer-based people detector, used as the default for pedestrian scenes.
- PeopleNet v2.6.3: a high-efficiency detector based on the DetectNet_v2 architecture.
- RT-DETR 2D: a multi-class detector for pedestrians, transporters, and forklifts.
Next, for monocular 3D perception, each camera uses a 3×4 projection matrix stored in a YAML calibration file. This back-projects 2D bounding boxes into 3D world-space coordinates using a ground-plane assumption. Then, for multi-view association, the tracker shares tracklets using Message Queuing Telemetry Transport (MQTT). MQTT is a lightweight pub/sub messaging protocol. When two cameras observe the same person, it matches tracklets by proximity in 3D world space.
After association, results stream out in three forms. The On-Screen Display (OSD) shows a tiled grid with 2D and 3D bounding boxes. The Bird’s-Eye View (BEV) renders a top-down trajectory map. Kafka messaging delivers per-frame protobuf metadata, including sensor ID, object ID, and 3D bounding box.
How AutoMagicCalib removes manual setup
MV3DT depends on calibrated cameras, which traditionally requires checkerboards and downtime. Instead, AMC calibrates a network by analysing tracked objects in existing video files or streams. It estimates each camera’s intrinsic parameters (focal length, principal point, lens distortion). It also estimates extrinsic parameters (rotation, translation, world position).
Under the hood, the pipeline runs five stages. These are per-camera trajectory extraction, single-view rectification, multi-view tracklet matching, bundle adjustment, and optional VGGT refinement. VGGT (Visual Geometry Grounded Transformer) helps when object movement is limited. AMC runs as a microservice with REST APIs and a web interface. Users supply only a layout image and a few alignment points.
The agentic skills workflow
With MV3DT and AMC defined, the delivery mechanism is the skills themselves. Rather than editing configuration files, you describe intent in natural language. The skills work with Claude Code, Codex, Cursor, and similar agents. Setup is short:
git clone https://github.com/NVIDIA/DeepStream.git
cd DeepStream
# Copy skills into your agent's skill directory (Codex shown)
mkdir -p ~/.codex/skills
cp -r skills/* ~/.codex/skills/
After launching the agent, a single prompt runs the reference app:
deploy mv3dt on the 12-camera sample dataset
From there, the MV3DT skill validates prerequisites, pulls the container, and installs Kafka and Mosquitto broker services. It also downloads model weights, generates the pipeline config, and launches tracking. Notably, if calibration files are missing, it triggers the AMC skills automatically.
DeepStream 9.0 vs 9.1
For context, the table below shows what changed between releases.
| Capability | DeepStream 9.0 | DeepStream 9.1 |
|---|---|---|
| Agentic skills | 2 (deepstream-dev, import-vision-model) | 13 agentic skills |
| Multi-camera 3D tracking | Not shipped as a skill | MV3DT skill + reference app |
| Camera calibration | Manual | AutoMagicCalib (AMC) microservice |
| Jetson support | JetPack 7.1 GA | JetPack 7.2 (Orin, Thor) |
| Sample datasets | — | 4-camera and 12-camera MV3DT sets |
| Distribution | NGC packages + GitHub source | Unified GitHub monorepo |
Use cases with examples
Given these capabilities, the features map to concrete deployments:
- Warehouse safety: track a worker near forklifts across aisles with one ID, using RT-DETR 2D.
- Retail analytics: follow a shopper between camera zones to measure dwell time without re-identification errors.
- Smart-building monitoring: count occupancy across floors and feed Kafka metadata to dashboards.
- Robotics and smart cities: share consistent world coordinates for navigation and incident review.
Interactive explainer
To see the mechanism, the embedded demo below animates one person walking between three camera fields of view. Toggle between naive per-camera 2D tracking and MV3DT 3D fusion to watch the object ID stay consistent.
What it means
For developers building vision systems, the workflow shifts from manual configuration to intent-based prompting. The skills handle the heavy lifting of environment setup, containerisation, and service installation. This reduces the time required to deploy complex multi-camera pipelines and lowers the barrier to entry for integrating advanced tracking logic.
Related articles
