Guides
Leadline
What your code did in development, as tabs on a worktree: database queries with their N+1s, request traces, logs and endpoints, recorded on the box and kept there.
Leadline is a plugin that shows what a worktree's app did while it ran in
development: every database query, each request's trace, and the logs.
A small program on the box, berth-leadline, records it beside your dev
servers; four tabs on the worktree read it. Nothing it records leaves the
box.
It isn't built in. Install it from Settings → Plugins → Browse, then turn it on under Installed. Because it carries a program for your boxes, the dialog that asks before it runs says so, by the program's name.
What it shows
Open any of the four from a worktree's + menu, or from ⌘K ("Show this worktree's queries"). Each one is about the worktree in front, and only that worktree.
| Tab | What is in it |
|---|---|
| Queries | Every statement, grouped by shape (the statement with a ? where a value was): calls, average, the slow percentile (p95) and total time. One request running the same statement five times or more is flagged N+1. Sort by any column, filter by SQL text, and open a query for the requests that run it and the ones that show its N+1. Control lists BEGIN, COMMIT, SET and their like, which are kept out of the queries. |
| Traces | Requests, newest first. One opens as a waterfall: every span in the order it ran, each database call by its shape, and the logs written while it ran. Failed keeps the requests with an error in them. |
| Logs | What the app logged, newest first, with a level filter and a search. A line opens to its attributes, and a line written during a request is a click from that request's trace. |
| Endpoints | Per route: requests, how many failed, latency, queries per request and time in databases. One opens to its numbers over time, the queries a request runs, and its slowest and failing requests. |
The time window at the top right (15 minutes unless you change it) is one for all four tabs. A tab reads again every few seconds while it is on screen, and not at all while it is behind another.
Send to the agent
A slow query, an N+1 or a failing endpoint has Send to the agent. It writes the problem up: what is wrong with its numbers, the statement's shape, the requests that show it, and what done looks like. The sheet shows exactly the text that will be sent, and you can change any of it first. Then it goes to the worktree's agent, held on the box until the agent is idle if it is busy; with no agent in the worktree, it starts one on it.
A statement is only ever written by its shape. The values your app sent (an email address, an id) are not shown in the tabs and are not put in what is sent, and an error's words have their values taken out the same way.
Turning it on for a box
Leadline is off on every box until you turn it on there. Open Leadline (under More in the sidebar, or "Show Leadline" in ⌘K): it lists your boxes and what each one has.
| A box says | What it means |
|---|---|
| Off | Nothing is installed or running. Turn on… asks first, then downloads berth-leadline for the box's platform, checks it against the checksum in the plugin, and has the box run it. |
| Installing…, Starting… | On its way. This takes a moment the first time. |
| Running | It is recording. The row says how much it holds and which worktrees have sent something. |
| Failed | The program is on the box and won't start; the row says why, in the box's words. Try again repeats it. A download that fails, or doesn't match the plugin's checksum, installs nothing: the confirm says why and the box stays Off. |
| Can't run it | Either the box's berthd is from before plugins could install a program (update the box in Settings → Boxes), or there is no build of the program for the box's platform, which the row says and which updating the box doesn't change. |
| Offline | The box can't be reached, so its helper can't either. |
A worktree's tab on a box where Leadline is off says so, with the same Turn on….
Update appears when the plugin carries a newer program than the box runs. Turn off stops and removes the program and keeps what it recorded for next time. Remove data also deletes what it recorded on that box, for every worktree; it can't be undone.
Leadline runs on a box only while the plugin is on. Turning Leadline off in
Settings → Plugins stops and removes the program on every box it runs
on, and keeps what each recorded; Remove… there removes that too.
Shipyard asks first and names the boxes, and a box that is offline is done
when it next connects. Turning the plugin on again starts nothing: you turn
Leadline on for each box again. An update to the plugin that brings a new
berth-leadline does the same once it is installed, so after allowing the
update you turn Leadline on for your boxes again; an update that keeps the
same program leaves it running while Shipyard asks you about the new
version. See a helper does not outlive its plugin.
Restart your services after turning it off
While Leadline is on, a service's Postgres connection goes through it (see below). A service started then was given that address, so after you turn Leadline off, restart the worktree's services: until then their database connections fail. Turning it off tells you which: the box's row and a notification name each service that was started while Leadline was on and is still running. A service you don't restart is named again each time Leadline is turned off, since it goes through Leadline again whenever Leadline is on; once it has been started with Leadline off, it isn't.
What an app needs to send
Leadline shows a worktree once something arrives from it.
- Start the app as a service of its worktree. With Leadline on for a
box, Shipyard sets the variables that point an app at it
(
OTEL_EXPORTER_OTLP_ENDPOINT,OTEL_EXPORTER_OTLP_PROTOCOL,OTEL_SERVICE_NAME,OTEL_RESOURCE_ATTRIBUTES) for the services it starts. A service that was already running needs a restart to get them. A server you start by hand in a terminal isn't given them. - Queries need nothing more. When a service's
DATABASE_URLorPOSTGRES_URLnames a Postgres on the box itself, Shipyard routes it through Leadline, which records every statement as it goes by: its text, its time on the wire, and whether it failed. No SDK, in any language. The address may be in Shipyard's environment or in the worktree's.env. - Traces and logs need an OpenTelemetry SDK in the app. Any OpenTelemetry SDK reads those variables and sends its traces and logs. Without one, Queries still fills in, but nothing ties a query to the request that ran it: Endpoints stays empty and an N+1 goes unflagged.
A tab for a worktree that has sent nothing says so, and says this.
Which databases Leadline sees
Leadline records a service's queries when Shipyard starts the service
with a DATABASE_URL or POSTGRES_URL that goes through it. Shipyard
finds that address in one of two places:
- In the environment Shipyard gives the service:
envin the repository's.berth/config.json, a kit, or your own settings for the project. - In the worktree's
.envfiles, when Shipyard's environment has no such variable. Many projects keepDATABASE_URLonly there, and let the dev server load it. When it starts a service, the box reads.env,.env.development,.env.localand.env.development.localin the worktree's own folder (a later file winning), and takes the two variables from them. A variable Shipyard's environment already has is never taken from a file: the project's own setting wins.
Either way the service is started with the address of Leadline's proxy
in its environment, and the worktree's files are not touched. That is
enough for a dev server that loads .env itself: Next.js, dotenv,
Prisma and most others leave a variable alone when the process already has
it, so the app connects through Leadline.
An app that overrides from .env goes round Leadline
A loader told to override what the process has (override: true in
dotenv, dotenv -o, a script that sources .env before starting the
server) puts the file's address back, and the app connects straight to
the database. Nothing breaks, and Queries stays empty. Set the
address in the project's Shipyard env instead, or turn the override
off in development.
An address in .env that names a secret (op://…), that points at
another machine, or that is written with a ${NAME} neither the files nor
the service's environment set, is left alone, as is a .env that is a
link to a file outside the worktree. How the files are read is in
Database's guide; the same
reader serves both.
A worktree's telemetry is filed under <project>/<worktree>, or the
project's name alone for its main checkout; a second service in the same
worktree adds :<name>, and shows in the worktree's tabs with a small
label.
What is kept, and for how long
Everything is in one SQLite file in a folder the box gives the helper, readable only by you on that box.
- A row is kept for 72 hours.
- Past 2 GB, the oldest rows go early.
- Old rows are dropped once a minute.
- A worktree you remove takes its rows with it. Shipyard tells Leadline when a worktree is removed, however it was removed, and Leadline closes the connection it kept in front of that worktree's Postgres and deletes what it recorded under the worktree's name. Removing a project from the box does the same for each of its worktrees. If Leadline wasn't running at that moment it isn't told later: those rows age out as usual.
The Leadline screen shows how much a box holds. Remove data empties it.
What is stored is a statement's text and up to 32 of its parameters (200 bytes each), spans with their attributes, and log records. The tabs show a statement by its shape, and leave its parameters out.
Nothing leaves the box
berth-leadline listens on the box's loopback address only, has no
password because nothing on the network can reach it, and connects to
nothing but the Postgres servers on the same box it was asked to stand in
front of. The app reads it through the box's own API, the way it reads
everything else on a box.
The one way anything goes further is by your hand: Send to the agent puts the text you see in the sheet into your agent's prompt, which the agent's own provider then sees, like any prompt.
Limits
- Postgres only, on the box only. Statements are recorded off the wire
for a Postgres on the same box. A database on another machine, a URL
that insists on TLS (
sslmode=requireand stronger) or a Unix socket is left alone; its queries show only if the app's own OpenTelemetry SDK reports them. Other databases show the same way: through the app's spans. - Services Shipyard starts. A dev server started by hand isn't pointed at Leadline, wherever its database's address is written.
- An app that overrides its environment from
.envconnects round Leadline. See which databases Leadline sees. - An app that names itself. The tabs show what is filed under the
worktree's name. An app that sets its own
OTEL_SERVICE_NAMEmay be filed under that name instead, and then doesn't show in the worktree's tabs. - Not live to the millisecond. A tab asks again every 3 seconds (less often for a wide window), not as each row arrives.
- Wide windows are slow. 15 minutes answers in a few milliseconds; a whole day on a full store can take seconds.
- The N+1 rule is one number. Five of the same statement in one request. It needs traces, so it needs an OpenTelemetry SDK in the app.
- Metrics are not kept. An SDK may send them; Leadline takes them and drops them.
- No comparison with
main. - No table editor and no SQL console. Those are another plugin's: Database.
- Where it runs. The program is built for Linux and macOS, on amd64 and arm64.