Building Shippy taught us how to build agents

A query about Ghana’s Exclusive Economic Zone returned the boundary source, data cutoff, timestamp, and a link to the Skylight map for verification.

Creating an AI agent for high-stakes work like ocean protection is primarily a reliability problem. A wrong answer could send a patrol vessel miles off course, wasting resources and risking personnel safety.

The Skylight team prioritised building a system they could trust over selecting the best model. Shippy must stay within its limits and verify every output against live satellite and vessel signals, not a static snapshot.

Agent anatomy: skills, soul, and config

The system treats Shippy as three components: a soul, skills, and config.

The soul is the system prompt that defines the persona and sets behavioural boundaries. Skills handle specific request types. These elements are baked into a Docker image, creating a versioned artifact. Config covers the rest, including the agent harness, the Large Language Model, and runtime settings. Secrets like API keys are injected at runtime, so swapping models or harnesses does not require a rebuild.

Shippy uses skills following the same specification as coding tools like Claude Code. These are plain markdown files with structured frontmatter, keeping them versioned and easy to update. Current capabilities include:

• Querying the Skylight API for vessel events and data
• Looking up Exclusive Economic Zone and Marine Protected Area boundaries
• Interpreting vessel track data and movement signals
• Generating interactive map links to jump from an answer to a specific location

The Skylight API query skill encodes the full workflow. When an analyst asks for fishing activity in Panama’s EEZ last month, the instructions direct Shippy to resolve the boundary polygon first, then query events within that geometry, and finally format results with deep links back to the map. Vessel metadata is attributed to partners like Global Fishing Watch or TMT.

A single question can trigger multiple skills. Asking about vessels near the Cordillera de Coiba Marine Protected Area draws on data query skills, ProtectedSeas boundary context, and vessel track interpretation. All of this happens in one dialogue turn.

The soul defines what Shippy will and will not do. It does not make legal determinations about vessel violations—that is a human role. It also does not speculate beyond available data. These boundaries are explicit in the system prompt, making them auditable and easy to revise.

Deterministic tools for a nondeterministic agent

Agents are nondeterministic. You cannot control the model’s decisions, but you can make the tools it uses predictable. Shippy communicates with Skylight through a purpose-built CLI that calls the API, rather than issuing raw calls.

The API has dozens of input types, nested filters, pagination cursors, and complex geometry. Early prototypes allowed Shippy to construct API calls from scratch, producing bugs like malformed pagination that dropped results and geometry encoding errors.

The Skylight CLI collapses this complexity into a predictable interface. Shippy issues a single command with typed filter flags, and the CLI handles authentication, pagination, and structured output. The CLI is self-documenting with extensive help text and detailed error messages that allow recovery without guessing. Output writes to a local JSON file instead of piping through the shell. This avoids buffer limits and breaks with downstream tools like jq. It also lets the agent access query results programmatically in subsequent steps.

Underlying the CLI is a standardised API. Multiple resource types, including events, vessels, regions, satellite imagery, and tracks, are accessible through common search and aggregate operations. Inputs and outputs are defined as typed schemas with field-level descriptions.

This layering means each component can be tested independently. The API has its own test suite. The CLI can be exercised by a human or an agent. Agent skills reference CLI commands that handle the plumbing. Each layer narrows the potential for error.

Sandboxed hosting and isolation

Skylight serves hundreds of government agencies and NGOs across over 70 countries. A fisheries officer in the Philippines has specific Areas of Interest, vessel watchlists, and alert configurations scoped to their account. Shippy must return only their data, and their conversation history must remain invisible to others.

Every user talks to Shippy inside an ephemeral, isolated session. Making this work reliably at scale required significant engineering. The team built Mothership, an agent hosting platform that provisions a dedicated Kubernetes deployment for each user session.

When a user opens a conversation, the system spins up pods packaging the agent runtime, skills, and Skylight CLI. The user’s Skylight JWT is injected at provision time to scope API calls. Files written during analysis exist only within that session and are never shared. Inside the sandbox, the agent can write and run code, install dependencies, and pull in datasets. Network access is restricted to only the services needed.

Evaluating an agent, not a model

Most benchmarks rank general-purpose AI on static questions. They do not capture how an agent behaves in a real workflow, selecting tools, querying live data, acting on results, and knowing when to stop. The team built an evaluation system around Shippy’s actual workflow, scoring the whole agent against live data.

In the framework, subject-matter experts write scenarios and rubrics, choosing criteria and setting weights for each task. A fishing-events query weights data accuracy most heavily, followed by boundary resolution and timeframe. Source attribution and response style carry less weight. Experts annotate responses as correct or incorrect to provide ground truth for the judge.

The pipeline is straightforward. A natural-language prompt runs through the sandbox. An LLM judge grades each criterion from 0 to 1 with written reasoning. The weighted aggregate is checked against a fixed pass threshold.

Tasks execute through Harbor, an open evaluation framework. A Harbor plugin spins up a real Shippy session on the exact version being tested against the same real data a user would encounter. The suite runs in parallel against a specific build, producing a timestamped results file and a report of score changes. The team reruns the suite whenever skills, models, or data change. A version that regresses on evaluation criteria does not reach end users.

Shippy scores consistently on data retrieval and guardrail tasks, correctly refusing military intelligence requests and maintaining user data isolation. Recent patterns show overstepping into tactical recommendations on patrol planning, missed events in geometry-sensitive queries due to boundary simplification, and one case where the agent invented a non-existent CLI command. Each issue informs the next round of skill improvements.

Where we’re headed

The team is opening Shippy to early adopters on a rolling basis to stress-test the system and find questions answered poorly or guardrails needing tightening.

• Agent-driven UI control. Shippy currently returns map links. The next step is driving the Skylight map itself to move to a region, apply filters, and adjust time ranges.
• Model routing. Simple lookups will route to smaller, faster models. The full-weight model will handle complex investigations.
• Cross-thread memory. Context currently does not carry across threads. Building memory will allow Shippy to persist facts like an analyst’s jurisdiction and preferred sources. This means “Show me fishing activity this week” will not require re-specifying the analyst’s EEZ.

Work on Shippy is shaping how the team approaches agents elsewhere at Ai2, including EarthRanger and OlmoEarth. Mothership was built to be general and host other agents. While maritime is the first domain, it is not expected to be the last.

Shippy is built by the Skylight team at Ai2. Skylight is a free maritime domain awareness platform used by 300+ partners across 70 countries.