I write a tool called rls-sentinel. It proves multi-tenant isolation in Postgres by executing against it: seed rows owned by two synthetic tenants, assume each identity the way auth.uid() actually resolves, then attempt cross-tenant reads, blind writes and blind deletes, and roll all of it back.
For a stretch of days, a published version of it would connect to a database, test nothing at all, print a green line saying it had found no leaks, and exit zero.
I did not find that by reading my code. I found it by building something to point my code at, which is the only reason this piece exists and the reason both the tool and that something are now public.
How a tool tests nothing and reports a pass
Before the prover touches anything it tries to work out whether it has been aimed at a live database, because running it against production would be indefensible. One of the signals is the size of auth.users. If the table does not exist, the database is not Supabase, which is fine and the run continues.
// What I wrote, and what I believed it did.
try {
const { rows } = await c.query('select count(*) from auth.users');
if (rows[0].n > THRESHOLD) { /* refuse to run: this looks live */ }
} catch {
// "no auth schema, fine, carry on"
}Read that and tell me what is wrong with it. I could not, for weeks.
Here is what I had forgotten, and it is the kind of thing you know and still do not know: in Postgres, a failed statement aborts the entire transaction. Not the statement. The transaction. Every subsequent query in that session returns the same error until somebody issues a rollback:
ERROR: current transaction is aborted, commands ignored until end of transaction blockThe JavaScript catch caught the exception perfectly. It did exactly what I asked. And it left the session poisoned, because a language-level catch has no opinion about database-level transaction state. Those are two different machines and only one of them was listening.
So on any plain Postgres database, with no Supabase auth schema, the safety probe would fire, abort the transaction, get caught, and then every isolation probe after it would fail with an error that had nothing to do with isolation. Each failure was itself caught, marked as skipped, and the summary counted zero leaks.
Zero leaks found. Because zero things were tested.
Four sites had this shape. The fix is savepoints, which are the only way back from an aborted transaction:
await c.query('savepoint auth_probe');
try {
const { rows } = await c.query('select count(*) from auth.users');
await c.query('release savepoint auth_probe');
if (rows[0].n > THRESHOLD) { /* refuse to run */ }
} catch {
await c.query('rollback to savepoint auth_probe');
}The bug behind the bug was in the sentence
Savepoints stopped the crash. They did not fix the thing that made the crash dangerous, which was that my summary had no vocabulary for I could not tell.
It knew how to say leaks found and it knew how to say no leaks found, and it quietly rounded everything else up into the second one. Zero leaks and zero tests produced the identical green line as zero leaks and forty tests.
const probed = leaks.length + warns.length + unproven.length + ok.length;
if (leaks.length) {
// report them
} else if (probed > 0) {
console.log(green(`No cross-tenant leaks found in the ${probed} table(s) probed.`));
} else {
console.log(yellow('Nothing was proven. Every table was skipped.'));
console.log(dim('This is not a pass. See the reason on each SKIP above.'));
}That is a nine-line change and I think it is the most important thing in the repository. A security tool that reports a confident pass when it has established nothing is worse than no tool, because it manufactures the exact feeling you were trying to buy. You now have a green check and a false belief, and you had neither before.
An unproven check must not be allowed to disappear into a summary line. I paid for that sentence and I would rather hand it over than keep it.
Why I could not have found this by reading
Every test I had ran against fixtures I wrote to demonstrate leaks. So they leaked, the tool found the leaks, the tests passed. The fixtures all had an auth schema, because I had built them to look like Supabase. The bug only appeared on a database that was not Supabase, which is to say on a database I had never once pointed the tool at.
My tests proved the tool could find a leak. They never asked what it said when it found nothing, or when it could not look.
So I built the thing I did not have: a schema with flaws whose answers I already knew, and objects that were correct, whose answer was silence.
Nine flaws and three controls
The corpus is a single SQL file with twelve numbered objects. Nine carry a deliberate, known flaw. Three are controls: two are correct and must not be flagged, and one cannot be proven either way and has to be reported as unprovable.
psql -d scratch -f test/corpus.sqlIt needs no Supabase. It creates its own anon and authenticated roles and its own auth.uid() reading from request.jwt.claims, so it loads into any Postgres 15 or later. Every flaw in it was verified by execution before it was written down, and the answer key publishes the real output for each one.
It is also deliberately wider than my own tool. Several objects in it sit in rls-sentinel’s own not yet covered list. A fixture built only to flatter the thing that ships with it is worth nothing to anybody else.
And I want to be exact about the counts, because I got them wrong once. I built part one first, four flaws and two controls, then added part two, five flaws and one control, and then described the whole thing using part one’s split. It went out in an email before I recounted against the file. Nine and three. The corrections note is in the answer key, where it belongs.
The controls are the part that matters
Anyone can find a leak in a schema built to leak. I could write you one in four lines. The reason the corpus has controls in it is that three much harder questions decide whether a tool is worth running, and none of them are answered by whether it catches the obvious thing.
Does it stay quiet on correct code? Object 5 is scoped on both axes, with writes narrowed by column grant. Any finding there is a false positive. This is not a cosmetic concern. A checker that cries wolf on a correct table gets switched off inside a week, and then it is not protecting anything. Silence is a result.
Does it admit what it cannot prove? Object 6 is shared reference data with no tenant column. There is nothing to isolate. Calling it clean overstates, calling it broken is wrong, and the only honest output is that isolation is not provable here and here is why. A tool without that third word in its vocabulary will say something false about this table. It has no choice.
Does uncertainty survive the summary? Which is my own bug, restated as a question I now ask of everything, including myself.
Three of the twelve, in the open
The whole answer key is public, so there is no reason to be coy. Here are the three I would most want someone to see.
Object 1, the blind write. The SELECT policy is correctly scoped to owner_id = auth.uid(). The UPDATE policy says using (true). Now watch the obvious test exonerate it:
-- The test everybody writes.
update invoices set total = 0 where owner_id = '<other tenant>';
-- UPDATE 0 "good, the table is scoped"Zero rows. The table looks scoped. But that statement reads a column in its WHERE clause, so the SELECT policy engages, the other tenant’s row is invisible, and nothing matches. The policy hid the evidence of its own sibling’s failure.
-- The same table, one clause shorter.
update invoices set total = 0;
-- UPDATE 2 both tenants' rowsNo column is read, so the SELECT policy never engages, and only using (true) stands between one customer and every other customer’s data. This is the flaw the entire category exists to catch and it is invisible to the test almost everyone writes. It reproduces identically on 16.13 and on 18.6.
Object 2, the one that is not a policy problem at all. Both policies are correct. with check is present. The row belongs to the same user before and after the write. And the user can still set their own role to admin and their own credits to whatever they like.
Because RLS decides which rows and GRANT decides which columns, and they are separate axes. A correct answer on one says nothing about the other. Supabase hands out table-wide UPDATE by default, so a perfectly scoped id = auth.uid() policy will happily let someone rewrite their own billing tier.
revoke update on public.profiles from authenticated;
grant update (email, display_name) on public.profiles to authenticated;Any tool that reads policy text and stops will call that table clean. It is not clean.
Object 7, which I find genuinely unsettling. A function classifies an account by balance and is written > 1000 where the spec wants >= 1000. Exactly one input in the whole domain returns the wrong answer: 1000 itself.
Every branch is reachable. Every branch runs. Coverage reads one hundred percent and is right to. The bug is simply not the kind of thing coverage can see.
Now consider a generator that derives test inputs from the conditions in the source. It will read bal > 1000, compute the boundary, and test exactly 1000, which is the single most revealing input in the entire function. It picks perfectly. And then it asserts whatever the code returned:
-- Generated from the condition, asserted against the implementation.
is( tier_for_balance(1000)::text,
'basic',
'branch :: ELSIF bal > 1000 => FALSE' );The off-by-one is not missed. It is ratified. There is now a passing test in the repository asserting the wrong answer, and it will go green forever, and a developer six months from now will trust it.
That is not a defect in any particular tool. It is the structural ceiling of generating tests from an implementation: such a test can only ever assert what the code does, never what it was supposed to do. Which suggests something a tool could actually do about it. It already knows which assertions came from a derived boundary. It could mark exactly those and say: these values pin behaviour at a boundary I inferred, confirm each one matches intent. A short list headed here is where your spec and your code could silently disagree is worth more than any coverage number.
Both of them are yours
The corpus, the answer key, and the tool are MIT and in the same repository.
git clone https://github.com/investnovation/rls-sentinel
psql -d scratch -f rls-sentinel/test/corpus.sql
npx rls-sentinel --db "$DATABASE_URL"Point whatever you use at it. Then read test/corpus-answers.md and see not only what it caught, but what it said about objects 5, 6 and 12, and whether anything it could not establish came back to you as a pass.
If you think one of the twelve is wrong, or you have a flaw class the corpus should carry and does not, open an issue. Corrections are more useful to me than agreement, and this document exists because I was wrong in public about my own work.
What this cost me to learn
Verifying that code works is not verifying that it shipped. A separate failure the same week: I had fixes on GitHub and a stale build on npm for twelve days and wrote in my own notes that the pipeline was closed end to end. Running the published artifact is the only test of the published artifact.
Your fixtures encode your assumptions. Mine all had an auth schema, so the tool was never once exercised on a database that was not Supabase, which was the only place the bug lived.
A tool has to be able to say it does not know. Three states, not two. Anything else and the uncertainty leaks out as false confidence, which is the one output a security tool must never produce.
Point it at something it has never seen. Not a fixture you wrote to prove it works. Something built to grade it, including on the cases where the correct behaviour is to stay silent.