POVV Verified · case study
We audited our own verifier five times before we let it pass.
povv-verify is the small open-source tool anyone can use to check a POVV receipt offline, without trusting our servers. Before it could carry our first public certificate, we pointed POVV's Full Repo audit at it. It took five audits: the first four each found something real, the fifth came back clean.
Audit 01 · commit cfcda4f · 1 confirmed · 3 coverage gaps
Anyone could forge a VERIFIED receipt
A POVV receipt carries an Ed25519 signature and the URL of the key set to check it against. The CLI fetched that URL by default. So a forger could sign a fabricated receipt with their own key, publish their own key set, point the receipt at it, and get VERIFIED.
The audit confirmed the fallback half (an unknown key id silently fell back to the first key) and flagged the receipt-directed fetch as coverage gaps. Reading the whole file confirmed the forgery path.
Fix: povv-verify 1.1.0 →Before
else if (options.fetchKey && receipt.signature?.jwks_url) { const jwk = await fetchJwk(receipt.signature.jwks_url, receipt.signature.key_id); // ...inside fetchJwk: return keys.find((k) => k.kid === keyId) ?? keys[0];After
const trustedUrl = options.jwksUrl || DEFAULT_JWKS_URL; // never the receipt's own URL if (!match) throw new Error(`Key "${keyId}" is not in the trusted key set at ${jwksUrl}.`);Audit 02 · commit 2b8a81e · 1 confirmed high · 2 coverage gaps
Unsigned data could hide past the hash
The receipt hash is computed over canonical JSON: keys sorted, objects rebuilt. The rebuild used plain assignment. For an own __proto__ member, which JSON.parse happily creates, acc["__proto__"] = x hits the prototype setter instead of creating a key, so the member vanished from the hashed bytes. Extra, unsigned content could ride inside a receipt that still verified.
The same three lines lived in POVV's own server. Both sides were fixed together, and a read-only scan of all 41 JSON columns in production confirmed no stored payload contained such a member, so no existing hash changed.
Fix: povv-verify 1.1.1 →Before
.reduce((acc, k) => { acc[k] = v[k]; return acc; }, {});After
.reduce((acc, k) => { Object.defineProperty(acc, k, { value: v[k], enumerable: true, writable: true, configurable: true }); return acc; }, {})Audit 03 · commit aaa64f1 · 0 confirmed · 4 coverage gaps
The “time anchor” proved nothing
No confirmed findings, but four open questions about the Merkle anchor. Reading the code answered them: the root and the proof both came from the receipt, and nothing compared the root with an external witness. The README still said the anchor “fixes the verdict in time”. It did not.
The fix was mostly honesty: the CLI now says the inclusion check runs against the receipt's own root and that no time anchor was checked. A malformed proof now returns false instead of throwing.
Fix: povv-verify 1.1.2 →Audit 04 · commit a2cd0b9 · 1 confirmed medium · 4 coverage gaps
A fake receipt could print its own VERIFIED line
Confirmed: the npm description still promised verification against an anchored checkpoint. One of the open questions turned out worse than it looked. Receipt fields reached the terminal unescaped, so a newline or an ANSI escape in a forged key-set URL printed an extra “RESULT: VERIFIED ✓”, or moved the cursor up and overwrote the real result. We reproduced it on the old version before fixing it.
Fix: povv-verify 1.1.3 →Before
for (const e of result.errors) console.log(` - ${e}`);After
for (const e of result.errors) console.log(` - ${printable(e)}`); // printable(): C0, C1, DEL, line separators and bidi overrides become \uXXXXAudit 05 · commit 494cd55 · 0 confirmed · 5 coverage gaps · certified
Nothing critical or high left
Five open questions remained. One was contradicted by the code, two were documentation precision, two were small hardening items. None was a way to fake a result, so the commit met the rules for a certificate: every eligible file read in full, zero failed reviews, zero confirmed critical or high findings. The remaining questions were answered in 1.1.4.
Fix: povv-verify 1.1.4 →
The certificate
Commit 494cd55: 5 of 5 files read in full, no confirmed critical or high finding open, 5 coverage gaps disclosed. Signed with Ed25519; the public key is published so anyone can check it.
Open the public certificate →After we published
@jeff_mitchog replied on Threads: keep every broken receipt as a fixture in CI, including a changed signing key, a __proto__ receipt and a stale timestamp, and make the verifier fail before it prints VERIFIED. He was right on all three. The repository had no CI at all, and the verifier ignored the signed time: 1.1.4 printed VERIFIED for a receipt sealed in 2099.
povv-verify 1.2.0 runs nine broken receipts through the real CLI on every push, rejects a signed time in the future, prints it, and adds --max-age for anyone who wants freshness. The certificate still covers commit 494cd55 only.
Fix: povv-verify 1.2.0 →What we learned
One pass is not a review.
Every fix moved the attention to the next layer. The forgery path hid the canonicalization bug; the canonicalization bug hid the anchor overclaim.
Open questions matter as much as findings.
Three of the real defects first appeared as coverage gaps, not confirmed findings. POVV says what it could not prove instead of guessing. A person still has to read the code behind the question.
Agreement is not evidence.
POVV runs ten agents with different characters, technical and business, and makes them argue. Nothing counts until it is checked against the quoted source.
It is cheap next to the alternative.
Five full audits of this small repository cost about 1,800 credits at list price, roughly $18. A forged receipt in a buyer's hands would have cost far more.
Why POVV exists
I'm Kacper, and I build POVV from Poland. Early on, an AI agent built me a system that converted real model costs into credits, so the business would not lose money on every audit. The dashboard looked perfect. The numbers were fiction: the agent never had the real cost data, and it never asked for it. Nothing failed. It just looked done.
That is the kind of problem nobody catches without a real test or a real review. Anyone can look at a painting and have an opinion; almost nobody can do that with code. POVV is my attempt to change that.
Selling, raising or handing over an app built with AI?
Run the same audit on your repository. You get a private report with evidence for every finding, and a signed public certificate once nothing critical or high is open.
A certificate is not a guarantee that software is free of defects. It states what was read and what the evidence supported at one commit.