Changes touching this path
- the shipper gets its writer, its position and the two boundary conditions that would lose data silently, and the position lives in the SAME database as the rows for a reason the code now keeps: a crash between an insert and a position update double counts, silently, because the re-read yields perfectly valid rows - so commitBatch writes both in one transaction rather than the two calls I first wrote, which would have left my own migration comment saying committed alongside them while the code did no such thing. two stores can disagree after a crash; one cannot disagree with itself.
the last line is consumed only if it is COMPLETE. nginx appends, so a run landing mid-write sees half a line, and parsing it would refuse it as malformed AND advance past it, so the finished line would never be read - a permanent silent loss of one request that no error would report. everything after the final newline is left for the next run. the boundary is found in the BUFFER before anything is decoded, because the position is a byte offset and user agents are routinely non-ascii, so measuring in characters would shift every offset after a multibyte line; the test carries its own control asserting the fixture really is multibyte, since otherwise it would pass under either implementation.
the position is keyed by INODE and not by name, because names rotate - todays access log is tomorrows .log.1 and 1641 made rotation daily and unconditional - and a file SHORTER than our recorded offset was replaced under a reused inode, which logrotate create does routinely, so it is re-read from zero rather than resumed past its opening lines. a refused line still advances the position, or one unparseable line would wedge the shipper behind it forever, and that stall looks exactly like a site with no traffic.
and the collision I predicted before starting gets a guard rather than a note: migration numbers are a shared sequential resource and NOTHING was checking them - land refuses a duplicate ADR number under the harbor lock, which is what kept 0074 and 0075 apart when two sessions wrote them the same night, but it knows nothing about site/migrations. two lanes taking 0002 would collide silently and the loser would never run, because the runner records by name and the winner already recorded it. six directory assertions now cover uniqueness, gaplessness, zero padding, and that each file records itself under its OWN name - the copy-paste mistake that makes a table quietly not exist under a green deploy - proved red by planting a duplicate 0002 (#1756)
823ebe57 · dbf3dbe6… - the beacon counts PEOPLE where the log can only count requests, and the notice that describes it was rewritten in the same change because deploying one without the other publishes a falsehood
loot#1757, over loot#1620 and loot#1617. POST /api/beacon takes one short
message per view and derives SHA-256(daily_salt || ip || ua || site) truncated
to 16 bytes, in process, discarding both inputs. It is canonical for uniques and
sessions and for nothing else: the log stays canonical for volume, because it
sees the ~53 prerendered pages, the cache hits and curl, none of which run any
JavaScript. Two sources counting volume would be two numbers to reconcile.
The key is length-prefixed per field rather than concatenated, and that is a
real defect closed rather than a flourish. One of the fields is the User-Agent,
which the CALLER chooses: with plain concatenation ip=1.2.3.4 ua=5 and
ip=1.2.3.45 ua= hash identically, so a visitor could be merged onto a neighbour.
The mutation that drops the framing reddens exactly that pin and nothing else.
Two decisions are recorded rather than defaulted. NO SALT, NO ROW: a beacon row
without a key adds to a count of beacon rows while contributing nothing to the
only two things this source is canonical for, so it would trade what we came for
against a number competing with the log. And the module is EAGER, which is the
answer to the ticket first hazard - ADR 0071 weighs the document plus eager
assets, so a deferred chunk is paid by every visitor and counted by nobody.
Measured at ~450 B on each of 62 surfaces and RECORDED, and loot#1640 is the
precedent that makes this more than taste: it measured a lazy route at 581 bytes
against 270 eager.
The edge half is new and had to be, because this is the site first
unauthenticated row-writing endpoint and the site vhost had no rate limiting at
all. limit_req at the /api/ PREFIX, not at /api/beacon, so ADR 0074 telemetry
arrives protected rather than silently exposed. Idempotence is remove-then-
reinsert between markers, and loot#1647 install-detect arm is closed in the same
patch since it needed the same only-if-absent vhost step. Its ordering is
load-bearing: the blanket header insert runs while only hand-written blocks
exist, which makes a duplicate unrepresentable instead of guarded against.
Found by RUNNING it, which is the whole argument for the fixture: the marker
ended in (generated), which awk read as a GROUP, so the strip silently never
fired - the second run appended a second /api/ block while grep -v tore the
limit_req line out of the first, leaving a vhost that would have failed nginx -t
on the box. Pinned now, with a control proving the predicate can say no.
The notice had to move with the code. Five published statements would have gone
false, the worst being that these records carry no key and nothing links two
requests to the same person. Rewritten to lead with the key, say the secret is
DESTROYED rather than merely rotated, and disclose that a visitor spanning
midnight is counted twice.
FOUR more contradictions were then found only by RENDERING the page, which is
the third time that has been the only thing that worked. Objecting still said
the log is the only thing we collect from, so this control has nothing to
suppress - three paragraphs after the new text says the browser sends nothing
when you object. Erasure and Your rights both still claimed nothing is keyed to
you. And JSX drops a newline beside a tag, so the page shipped visitorkey and
notstored as single words. The font gate then caught a star glyph absent from
the shipped subset, which would have rendered as tofu.
A test that asserted we do NOT collect a visitor key survived all of this green,
because its regex matched one verb phrasing the new prose never uses. It is
inverted now: the page MUST name the key, MUST say destroyed, MUST disclose the
midnight double-count. Leaving it would have let a later edit delete the
disclosure and stay green.
532 site tests, 322 scripts tests, budget green with the notice raise on the
record.
3c22dcae · dbf3dbe6…diff - the private tier counts ACTIONS at the one seam every handler enters through, and the account reference it carries is the notice second identifier rather than a footnote
loot#1758, the third of loot#1619 three collection points. Every api.private.*
request enters through withPrivateSession, so the row is queued there on the
way OUT with the status actually sent - one instrumentation point rather than
one per route, which means a new endpoint is counted by construction and a test
pins that every api.private.* file uses the seam.
Deliberately narrow: the beacon owns views, the log owns volume, this owns
actions. Counting anywhere wider would either double-count with the beacon or
undercount behind the 60-second cache.
The path is the ROUTE SHAPE, which on this tier is not a formality: owner,
repo and file path all travel in the QUERY STRING, and routeShape drops the
query before any rule runs. The pin asserts the repo name is absent from the
serialised row, not merely that the path looks right.
account_ref is a keyed SHA-256 digest of the Clerk id, 32 hex, stable ON
PURPOSE - the opposite choice from the visitor key, because its entire job is
to let an erasure request find every row of one account. A read of the metrics
database yields nothing a third party issued; an erasure can still compute the
value to delete by. A 401 row carries NULL rather than a reference to nobody,
so the one-subject-tier CHECK reads it as no subject.
The notice moved with it, in nine sentences. The page said in so many words that
the records carry NO account reference and that there is ONE identifier; both
are now false and both are now pinned in the other direction. What the page
does NOT claim is as deliberate: loot#1622 says the reference lives with the
account and is deleted on request, and only the second half is built - no
account-deletion hook removes these rows - so the page says on request and
says the automatic link is not built, and a test refuses the flattering version.
Rendering found two more sentences that had become misleading rather than
false: Erasure said deleting the account deletes it, which now read as covering
the counted rows, and the retention section said keyed rows lose their meaning
when the salt dies, which is true of visitor keys and the opposite of true for
account codes. Both now say which rows they mean.
547 site tests pass; budget green with 263 B of slack left on /privacy, so the
next notice edit will need a recorded raise.
3bb326ce · dbf3dbe6…diff - the opt-in telemetry channel is built, and off means no call because a paired test says so rather than a sentence
loot#1658, building what ADR 0074 specified and loot#1626 shipped only the
guard for. loot telemetry on | off | status | show; an append-only accumulator
that writes one line per dispatched verb, the dispatch-table name, with burn and
maroon collapsed to other AT WRITE TIME so no local trace of an incident exists
either; a sender that runs in main after the verb has printed and decided its
exit code, at most once per 24 hours, behind a 3-second hard timeout, that can
never print, never change the code, and never delay a verb; a spool that IS the
outbox, so show reads the same entry the sender transmits; and a 16-byte id
minted on opt-in and deleted on opt-out, never machine-id.
The guarantee is the pairing. One local listener, the real binary, a sandboxed
config home: init, status, doctor and show with telemetry off make ZERO
connections; on, the same verbs make EXACTLY ONE, and the same verbs again in
the same period make no more; off again makes none. A lone assert-zero passes
when the listener is broken, the binary crashed early, or it pointed at the
wrong port - the exact shape of the logrotate -f check that could never fail.
The endpoint override is https only with the loopback LITERAL the sole
exception, enforced twice, and localhost is refused because what it resolves
to is the resolver business; the refusal happens BEFORE an id is minted, so a
broken override never leaves an identifier that can never be used.
Clause 5 of the field rule is a property of the wire bytes, not a promise:
a test parses every string out of the JSON and requires each to be a
compile-time constant, a member of a fixed array, a dispatch-table name, or
the hex id. The platform is a build-script constant rather than a runtime
assembly from std consts, which is what the rule forbids spelled from constants.
The verb emits machine output - show --json IS the report - so it joins the
emitting tier rather than the prose-only debt list the ceiling forbids growing.
Every count that moved was moved in the documents that state it, because the
censuses read them: 129 verb lines in three sites, 82 dispatched and 80 in the
table and 25 emitting in ADR 0066 with telemetry in the named list, all 82
verbs and the block in the README, the CLI page, and ADR 0074 eight-entry
allowlist - because the endpoint is now the binary ONE destination it contacts
rather than prints, and its ALLOWED row says under what condition.
The destination census flagged three lines that dial nothing: a printed hint
naming http://127.0.0.1 and the two scheme-prefix tests in check_endpoint. They
are respelled rather than exempted - the check splits on the separator and
compares the scheme WORD, same rule, no allowlist row - because a row is a
sentence someone has to be able to defend and these had nothing to defend.
The site half: POST /api/telemetry validates a report against the SAME closed
sets the binary declares, pinned cross-repo by reading the Rust source, and
refuses WHOLE on any field outside them. A valid report becomes source=cli
rows, one per verb invocation under /cli/<verb> plus two bucket rows, all
stamped with the arrival time so no ordering exists to reconstruct; the
telemetry_id rides the row and expires with the 30-day partitions. It sits
behind the same /api/ limit_req the beacon needed, which is why the ADR put it
there. doctor prints on/off, the EFFECTIVE endpoint, last send and the spool.
The notice gains the CLI tier, as ADR 0074 §11 requires: sends nothing unless
you run on, the id links reports and off deletes it, what is never in a report,
show prints it before you decide, and the erasure route - quote your id within
30 days, you hold the key. The test that asserted the page does NOT collect
telemetry is inverted, as it was for the visitor key.
1214 lib and 118 bin tests, 51 suites, 634 site tests, budget and font gates
green, rendered and read.
8a919b01 · dbf3dbe6…diff - the metrics shipper stops failing every tick once its backlog passes 4,095 rows: insertEvents sent one multi-row INSERT for every row a tick read at 16 parameters a row, and Postgres counts bind parameters in a 16-bit field, so past 65,535 node-postgres wrapped the count and the server refused the statement (bind message has 19296 parameter formats but 0 parameters, found by the #2396 site deploy post-deploy tick). eventInsertStatements now splits the rows into statements under MAX_BIND_PARAMETERS, each numbered from $1, run in the same one transaction so the log position still commits with every row or none. pinned at 5,000 rows: split under the limit, every row once in order, placeholders never past a statement own values, red under a single-statement mutation (1 failed, 3 passed) and restored. site gate 808 passed; bash ci/local.sh green against Postgres 18 including the metrics database suite. owes a site deploy (#2401)
4f516a7a · dbf3dbe6…diff
Renames are not followed. loot's tree maps a path to an address, so a rename is a delete and an add. This list is the history of the name, not of the bytes.