The House That Sorts: A Sovereign Answer to AI Secret Leaks
The tools we built after our AI agent leaked every secret we had
Yesterday, the house watched. Today, the house learns to sort.
The Situation
Your .env file was exposed. You rotated every key. Now what?
Three questions keep you awake:
- Did I rotate everything? — How do you know no stale key remains?
- Are all machines in sync? — Did the VPS get the same new keys as my laptop?
- Will a backup undo the fix? — If I restore from last week, do I restore leaked keys?
And the hardest constraint: you cannot ask your AI agent to check, because showing it the keys is how the leak happened in the first place.
We call this checking the uncheckable. Here is how we solved it.
For Everyone — The Concept
You go to the locksmith and say: "I changed every lock in the house. Can you verify?"
The locksmith says: "Show me the keys."
You say: "No. If I show you the keys, I need to change them again."
The locksmith thinks. Then: "Show me the last four digits stamped on each key. I can verify they are different from the old ones without being able to copy them."
Four characters. Not enough to open any door. Enough to prove every lock was changed.
This is what we built. A script that reads your secrets file, extracts only the last four characters of each key, and stores them in a registry. The registry can be shared, committed, compared across machines — because four characters of a forty-character key are useless to an attacker but sufficient for verification.
Every major platform already uses this pattern. Go to your Anthropic dashboard, your OpenAI settings, your GitHub token page — they show you the last four characters of your key. You can see them right now. We just automated the comparison.
For the Curious — The Three Tools
Tool 1: The Classifier (Reverse PII)
Before you can verify rotation, you need to know what is a secret and what is not.
A .env file mixes secrets with configuration:
ANTHROPIC_API_KEY=sk-ant-api03-abc...xyz # Secret — must rotate
OPENAI_API_KEY=sk-proj-def...uvw # Secret — must rotate
N8N_PORT=5678 # Config — ignore
TZ=Europe/Paris # Config — ignore
DATABASE_URL=postgresql://user:pass@host/db # Credential — must rotate
We built a classifier that uses the same pattern-matching approach as PII detection — but reversed. Instead of finding personal data in text (Social Security numbers, email addresses), it finds secret data in environment variables (API key prefixes, connection strings, high-entropy values).
Four phases, zero cost:
| Phase | Method | Example |
|---|---|---|
| 1. Prefix | Known key patterns | sk-ant-* → Anthropic, ghp_* → GitHub |
| 2. Structure | URL analysis | postgresql://user:pass@ → credential |
| 3. Name | Key name heuristics | *_PASSWORD → secret, *_PORT → config |
| 4. Entropy | Character distribution | Long, mixed-case, with digits → likely secret |
No LLM. No API call. Pure regex. Free forever.
Tool 2: The Registry (Non-Disclosing Verification)
The classifier identifies your secrets. The registry tracks their rotation state:
meta:
last_rotation: "2026-03-25"
reason: "file-watcher breach"
machine:
hostname: "macbook"
id: "F6B154AF"
keys:
ANTHROPIC_API_KEY:
suffix: "Xk4Q"
class: "SECRET"
detected_by: "anthropic_key"
rotated: "2026-03-25"
probe: "PASS"
rotate_url: "https://console.anthropic.com/settings/keys"
Each key entry contains:
- suffix — last 4 characters (the non-disclosing fingerprint)
- class — what type of secret the classifier detected
- detected_by — which pattern matched (auditable)
- rotate_url — where to go to rotate this key
- probe — did a functional test pass? (curl with
$VAR→ 200 or 401)
The registry is safe to commit. Four characters cannot reconstruct a forty-character key. But they can prove it changed.
Tool 3: The Cross-Machine Check
You run the script on your laptop. You run it on your server. You compare:
✓ ANTHROPIC_API_KEY macbook=Xk4Q vps=Xk4Q synced
✓ DISCORD_WEBHOOK macbook=pQ3a vps=pQ3a synced
✗ OPENAI_API_KEY macbook=m9Tz vps=j5Wp MISMATCH
✓ POSTGRES_PASSWORD macbook=w0rd vps=w0rd synced
One mismatch. One key that was rotated on the laptop but not on the server. Caught before the next backup could immortalize the old key.
During our own migration, this check caught a real forgotten key. The crash test found a real crash.
For the Expert — Architecture
The Ordnung: .env by concern, not by environment
The standard convention splits .env files by deployment stage:
.env.development .env.production .env.test
This assumes multiple developers, multiple environments, CI/CD pipelines. For a solo developer or small team with one server, the split that matters is not where you deploy but what you protect:
.env → Config (ports, paths, booleans) — never rotates
.env.claude → AI provider keys — rotate on IDE breach
.env.webhooks → Notification endpoints — rotate on channel compromise
.env.infra → Database, services, mail — rotate on server breach
.env.platforms → GitHub, social media — rotate on platform breach
Each file is one blast radius. If .env.claude leaks, you rotate AI keys only. Discord keeps working. The database is untouched. Blast radius containment through file-level separation of concerns.
The Forseti Audit: is this key in the right file?
The script doesn't just classify keys — it audits their placement:
⚠ MISPLACED key=ANTHROPIC_API_KEY found_in=.env expected=.env.claude
⚠ MISPLACED key=DISCORD_WEBHOOK found_in=.env expected=.env.webhooks
Run it on your current monolithic .env and it tells you exactly what to move where. The Ordnung sorts itself.
The backup safety rule
Every backup carries a timestamp. Every rotation carries a timestamp. If backup_date < last_rotation_date, that backup contains pre-rotation secrets. Restoring it reintroduces compromised keys.
The registry's last_rotation field makes this comparison trivial. Timestamp the rotation. Timestamp the backup. Compare. Block if stale.
Crash testing your own security
We crash-tested the classifier with deliberately wrong inputs:
- Test 1: 14 fake secrets + 10 config values → 14 detected, 10 skipped, 0 false positives
- Test 2: 11 keys in the wrong file → 11 misplacement warnings
- Test 3: Two machine registries with one mismatch → mismatch caught
- Test 4: The mismatch was real — a key we actually forgot to rotate
The crash test methodology is simple: break it on purpose before it breaks in production. Create fake .env files with known contents, run the classifier, verify the output. If it misses a secret, you find out in the sandbox, not in an incident.
The Script
A few hundred lines of bash. No dependencies beyond grep and awk. Runs on any Unix system.
# Generate registry from your .env files
./forseti-key-registry.sh .env.claude .env.webhooks .env.infra
# Compare two machines
./forseti-key-registry.sh --check
The full source is available and free to use. No cloud. No account. No tracking. Sovereign by design.
The Principle
Yesterday we described a gap: the house watches what it should not see. Today we described the response: sort the house so that each room holds only what belongs to it, verify each lock without touching the key, and crash-test the whole thing before trusting it.
Four characters. Five files. A few hundred lines of bash.
The dog is not the problem. The table is not the problem. The house that does not sort its rooms is the problem. And now it sorts.
But why sort at all? Because what you build must stand the pass of time. Keys rotate. Backups age. Machines fail. The only thing that persists is the structure you chose — the discipline of putting each thing where it belongs, so that whoever opens the house next finds it in order. Not for today. For the day you are not there to explain it.
That is sovereignty. Not control — continuity.
Fix the table. Then sort the house. Then teach the house to check itself — so it stands when you are gone.
Previous: The House That Watches | Related: The Keys to the House