shipyarddocs
shipyarddocs

Guides

Leadline

What your code did in development, as tabs on a worktree: database queries with their N+1s, request traces, logs and endpoints, recorded on the box and kept there.

Leadline is a plugin that shows what a worktree's app did while it ran in development: every database query, each request's trace, and the logs. A small program on the box, berth-leadline, records it beside your dev servers; four tabs on the worktree read it. Nothing it records leaves the box.

It isn't built in. Install it from Settings → Plugins → Browse, then turn it on under Installed. Because it carries a program for your boxes, the dialog that asks before it runs says so, by the program's name.

What it shows

Open any of the four from a worktree's + menu, or from ⌘K ("Show this worktree's queries"). Each one is about the worktree in front, and only that worktree.

TabWhat is in it
QueriesEvery statement, grouped by shape (the statement with a ? where a value was): calls, average, the slow percentile (p95) and total time. One request running the same statement five times or more is flagged N+1. Sort by any column, filter by SQL text, and open a query for the requests that run it and the ones that show its N+1. Control lists BEGIN, COMMIT, SET and their like, which are kept out of the queries.
TracesRequests, newest first. One opens as a waterfall: every span in the order it ran, each database call by its shape, and the logs written while it ran. Failed keeps the requests with an error in them.
LogsWhat the app logged, newest first, with a level filter and a search. A line opens to its attributes, and a line written during a request is a click from that request's trace.
EndpointsPer route: requests, how many failed, latency, queries per request and time in databases. One opens to its numbers over time, the queries a request runs, and its slowest and failing requests.

The time window at the top right (15 minutes unless you change it) is one for all four tabs. A tab reads again every few seconds while it is on screen, and not at all while it is behind another.

Send to the agent

A slow query, an N+1 or a failing endpoint has Send to the agent. It writes the problem up: what is wrong with its numbers, the statement's shape, the requests that show it, and what done looks like. The sheet shows exactly the text that will be sent, and you can change any of it first. Then it goes to the worktree's agent, held on the box until the agent is idle if it is busy; with no agent in the worktree, it starts one on it.

A statement is only ever written by its shape. The values your app sent (an email address, an id) are not shown in the tabs and are not put in what is sent, and an error's words have their values taken out the same way.

Turning it on for a box

Leadline is off on every box until you turn it on there. Open Leadline (under More in the sidebar, or "Show Leadline" in ⌘K): it lists your boxes and what each one has.

A box saysWhat it means
OffNothing is installed or running. Turn on… asks first, then downloads berth-leadline for the box's platform, checks it against the checksum in the plugin, and has the box run it.
Installing…, Starting…On its way. This takes a moment the first time.
RunningIt is recording. The row says how much it holds and which worktrees have sent something.
FailedThe program is on the box and won't start; the row says why, in the box's words. Try again repeats it. A download that fails, or doesn't match the plugin's checksum, installs nothing: the confirm says why and the box stays Off.
Can't run itEither the box's berthd is from before plugins could install a program (update the box in Settings → Boxes), or there is no build of the program for the box's platform, which the row says and which updating the box doesn't change.
OfflineThe box can't be reached, so its helper can't either.

A worktree's tab on a box where Leadline is off says so, with the same Turn on….

Update appears when the plugin carries a newer program than the box runs. Turn off stops and removes the program and keeps what it recorded for next time. Remove data also deletes what it recorded on that box, for every worktree; it can't be undone.

Leadline runs on a box only while the plugin is on. Turning Leadline off in Settings → Plugins stops and removes the program on every box it runs on, and keeps what each recorded; Remove… there removes that too. Shipyard asks first and names the boxes, and a box that is offline is done when it next connects. Turning the plugin on again starts nothing: you turn Leadline on for each box again. An update to the plugin that brings a new berth-leadline does the same once it is installed, so after allowing the update you turn Leadline on for your boxes again; an update that keeps the same program leaves it running while Shipyard asks you about the new version. See a helper does not outlive its plugin.

Restart your services after turning it off

While Leadline is on, a service's Postgres connection goes through it (see below). A service started then was given that address, so after you turn Leadline off, restart the worktree's services: until then their database connections fail. Turning it off tells you which: the box's row and a notification name each service that was started while Leadline was on and is still running. A service you don't restart is named again each time Leadline is turned off, since it goes through Leadline again whenever Leadline is on; once it has been started with Leadline off, it isn't.

What an app needs to send

Leadline shows a worktree once something arrives from it.

  1. Start the app as a service of its worktree. With Leadline on for a box, Shipyard sets the variables that point an app at it (OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_PROTOCOL, OTEL_SERVICE_NAME, OTEL_RESOURCE_ATTRIBUTES) for the services it starts. A service that was already running needs a restart to get them. A server you start by hand in a terminal isn't given them.
  2. Queries need nothing more. When a service's DATABASE_URL or POSTGRES_URL names a Postgres on the box itself, Shipyard routes it through Leadline, which records every statement as it goes by: its text, its time on the wire, and whether it failed. No SDK, in any language. The address may be in Shipyard's environment or in the worktree's .env.
  3. Traces and logs need an OpenTelemetry SDK in the app. Any OpenTelemetry SDK reads those variables and sends its traces and logs. Without one, Queries still fills in, but nothing ties a query to the request that ran it: Endpoints stays empty and an N+1 goes unflagged.

A tab for a worktree that has sent nothing says so, and says this.

Which databases Leadline sees

Leadline records a service's queries when Shipyard starts the service with a DATABASE_URL or POSTGRES_URL that goes through it. Shipyard finds that address in one of two places:

  • In the environment Shipyard gives the service: env in the repository's .berth/config.json, a kit, or your own settings for the project.
  • In the worktree's .env files, when Shipyard's environment has no such variable. Many projects keep DATABASE_URL only there, and let the dev server load it. When it starts a service, the box reads .env, .env.development, .env.local and .env.development.local in the worktree's own folder (a later file winning), and takes the two variables from them. A variable Shipyard's environment already has is never taken from a file: the project's own setting wins.

Either way the service is started with the address of Leadline's proxy in its environment, and the worktree's files are not touched. That is enough for a dev server that loads .env itself: Next.js, dotenv, Prisma and most others leave a variable alone when the process already has it, so the app connects through Leadline.

An app that overrides from .env goes round Leadline

A loader told to override what the process has (override: true in dotenv, dotenv -o, a script that sources .env before starting the server) puts the file's address back, and the app connects straight to the database. Nothing breaks, and Queries stays empty. Set the address in the project's Shipyard env instead, or turn the override off in development.

An address in .env that names a secret (op://…), that points at another machine, or that is written with a ${NAME} neither the files nor the service's environment set, is left alone, as is a .env that is a link to a file outside the worktree. How the files are read is in Database's guide; the same reader serves both.

A worktree's telemetry is filed under <project>/<worktree>, or the project's name alone for its main checkout; a second service in the same worktree adds :<name>, and shows in the worktree's tabs with a small label.

What is kept, and for how long

Everything is in one SQLite file in a folder the box gives the helper, readable only by you on that box.

  • A row is kept for 72 hours.
  • Past 2 GB, the oldest rows go early.
  • Old rows are dropped once a minute.
  • A worktree you remove takes its rows with it. Shipyard tells Leadline when a worktree is removed, however it was removed, and Leadline closes the connection it kept in front of that worktree's Postgres and deletes what it recorded under the worktree's name. Removing a project from the box does the same for each of its worktrees. If Leadline wasn't running at that moment it isn't told later: those rows age out as usual.

The Leadline screen shows how much a box holds. Remove data empties it.

What is stored is a statement's text and up to 32 of its parameters (200 bytes each), spans with their attributes, and log records. The tabs show a statement by its shape, and leave its parameters out.

Nothing leaves the box

berth-leadline listens on the box's loopback address only, has no password because nothing on the network can reach it, and connects to nothing but the Postgres servers on the same box it was asked to stand in front of. The app reads it through the box's own API, the way it reads everything else on a box.

The one way anything goes further is by your hand: Send to the agent puts the text you see in the sheet into your agent's prompt, which the agent's own provider then sees, like any prompt.

Limits

  • Postgres only, on the box only. Statements are recorded off the wire for a Postgres on the same box. A database on another machine, a URL that insists on TLS (sslmode=require and stronger) or a Unix socket is left alone; its queries show only if the app's own OpenTelemetry SDK reports them. Other databases show the same way: through the app's spans.
  • Services Shipyard starts. A dev server started by hand isn't pointed at Leadline, wherever its database's address is written.
  • An app that overrides its environment from .env connects round Leadline. See which databases Leadline sees.
  • An app that names itself. The tabs show what is filed under the worktree's name. An app that sets its own OTEL_SERVICE_NAME may be filed under that name instead, and then doesn't show in the worktree's tabs.
  • Not live to the millisecond. A tab asks again every 3 seconds (less often for a wide window), not as each row arrives.
  • Wide windows are slow. 15 minutes answers in a few milliseconds; a whole day on a full store can take seconds.
  • The N+1 rule is one number. Five of the same statement in one request. It needs traces, so it needs an OpenTelemetry SDK in the app.
  • Metrics are not kept. An SDK may send them; Leadline takes them and drops them.
  • No comparison with main.
  • No table editor and no SQL console. Those are another plugin's: Database.
  • Where it runs. The program is built for Linux and macOS, on amd64 and arm64.

On this page