allaun/Research-Stack
1
0
Fork 0
mirror of https://github.com/allaunthefox/Research-Stack.git synced 2026-07-31 03:05:21 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
Research-Stack/audit/documentation_audit.md
Allaun Silverfox 8e51acad08 audit: full codebase inspection — 6 auditors, 9,606 objects, 2,068 lines of findings 
CRITICAL (7):
- C1: Hardcoded Wolfram Alpha API key (revoke immediately)
- C2: Q16_16 rounding diverges Lean↔Python↔C (data corruption)
- C3: Receipts describe non-existent files (fabrication)
- C4: '3,500+ proofs' = compilation units, not theorems
- C5: NaN sentinel collision Lean↔C (silent corruption)
- C6: eigensolid_convergence theorem is a tautology
- C7: 74% CI failure rate

HIGH (12): duplicate definitions, SQL injection, command injection,
  1,029 duplicate files, 66 large files in git, AGENTS.md drift

MEDIUM (23): stale docs, broken cross-refs, unhandled errors
LOW (31): naming violations, misplaced files, cleanup

Master synthesis: audit/MASTER_AUDIT_SYNTHESIS.md
Per-dimension reports: audit/*_audit.md
2026-06-21 02:07:35 -05:00

335 lines
15 KiB
Markdown
Raw Permalink Blame History

# Documentation Audit — Research Stack (OTOM)
**Audit Date:** 2026-06-22
**Scope:** All .md files (1,956), .yaml/.yml files (172), .lean files (1,295), and documentation directories
**Methodology:** Cross-referencing documentation claims against actual repository contents via GitHub raw URLs and file tree analysis
---
## Summary
| Metric | Count |
|--------|-------|
| Total .md files audited | 1,956 |
| Total .lean files | 1,295 |
| Stale claims | 8 |
| Broken cross-references | 16 |
| Old TODOs (unaddressed >2 months) | 8 |
| Duplicate documentation pairs | 45 (16 docs + 29 obsidian-vault) |
| Empty/minimal READMEs | 15 |
| Undocumented major modules | 6 |
| AGENTS.md files (potential conflict) | 7 |
**Overall Assessment:** The documentation set is extensive (1,956 markdown files, ~9.3MB in 6-Documentation/ alone) but suffers from significant staleness, broken cross-references, inflated proof claims, duplicate content across multiple directory trees, and receipts that describe files and test results that do not exist in the repository.
---
## Critical Issues
### C1: E2E and library Receipts Describe Non-Existent Files
**Files:** `e2e/E2E_MASTER_RECEIPT.md`, `library/CONCEPTUAL_UPGRADE_RECEIPT.md`
**Severity: CRITICAL**
Both receipt files claim detailed test results and reference source files that do not exist:
- `e2e/` directory is **completely empty** (0 files), yet `E2E_MASTER_RECEIPT.md` describes:
- `e2e/E2EMasterTrace.lean` (~540 lines)
- `e2e/run_e2e_trace.py` (~650 lines)
- Step-by-step execution traces with theorem names and line numbers
- Receipt SHA-256: `c8ad995a0fdd9bd0160ae5e20ca27b89a5ca759ef0465b7d0472d0901b3efcfa`
- `library/` directory is **completely empty** (0 files), yet `CONCEPTUAL_UPGRADE_RECEIPT.md` describes:
- `library/hachimoji_codec_v2.py` — "17/17 PASSED"
- `library/HachimojiCodecV2.lean` — "4 theorems proven"
- `library/counterexample_detector.py` — "10/10 PASSED"
- All test claims are unverifiable
**Impact:** These receipts are fabrication — they document code and test results that were never committed to the repository. Any downstream consumer trusting these receipts is being misled.
---
### C2: "3,500+ Proofs" Claim is Inflated
**Files:** `README.md`, `EXPLANATION_FOR_HUMANS.md`, `GETTING_STARTED.md`
**Severity: CRITICAL**
Multiple documents claim:
- EXPLANATION_FOR_HUMANS.md: "3,500+ strict mathematical proofs" and "checked against 3,500+ strict mathematical proofs"
- README.md: "746 verified Lean modules"
- GETTING_STARTED.md: "3,500+ verified proofs" and "all 3,500+ proofs are verified"
**Reality:**
- There are 1,295 `.lean` files, not 746 "verified modules"
- The "3,500" number matches the **build job count** at various points in time (3,314 / 3,529 / 3,592 / 8,332 jobs), not individual proof theorems
- Build jobs are compilation units (module imports, definitions, structures), not proof theorems
- Many `.lean` files contain only `def`, `structure`, `#eval` statements with no `theorem` or `lemma`
- Foundations files like `CarnotEfficiency.lean` (377 bytes) are just a single definition with `#eval` — no theorem
- `LandauerBound.lean` (457 bytes) uses a placeholder `kB` value (`ofRawInt 6`) with comment "Simplified kB" — not physically meaningful
**Impact:** This claim misleads readers about the maturity and depth of the formalization. Conflating build jobs with proof theorems is a significant misrepresentation.
---
### C3: `eigensolid_convergence` Theorem is a Tautology
**File:** `0-Core-Formalism/lean/Semantics/Semantics/BraidEigensolid.lean`
**Severity: CRITICAL**
The flagship theorem, referenced throughout documentation (AGENTS.md, ARCHITECTURE.md, README.md) as proof of compressor correctness:
```lean
theorem eigensolid_convergence
(s : BraidState)
(h_eig : IsEigensolid (crossStep s)) :
∀ i : Fin 8, (crossStep (crossStep s)).strands i = (crossStep s).strands i :=
h_eig
```
This theorem **literally restates its hypothesis**. `IsEigensolid (crossStep s)` means exactly `∀ i, (crossStep (crossStep s)).strands i = (crossStep s).strands i`. The "proof" is just returning the hypothesis.
**Impact:** The core compressor convergence guarantee is vacuous. The theorem proves nothing beyond what was assumed. ARCHITECTURE.md section 13 and TODO_MAP.md B11 both cite this as a completed proof.
---
### C4: CONCEPTS.md Referenced but Missing
**Files:** `GETTING_STARTED.md` (section 6), `ARCHITECTURE.md` (section 10)
**Severity: HIGH**
`CONCEPTS.md` is listed as a "Key File to Read First" in GETTING_STARTED.md and linked from ARCHITECTURE.md's "Further Reading" section. The file **does not exist** in the repository.
---
### C5: Circular Reference in docs/README.md
**File:** `docs/README.md` (also `6-Documentation/docs/README.md` — duplicate)
**Severity: HIGH**
The README states:
> "The authoritative documentation tree is [6-Documentation/docs/](../6-Documentation/docs/)."
But this file **IS** in `6-Documentation/docs/`. It points to itself via a relative path. This is a broken self-reference that would confuse any reader.
---
## Warnings
### W1: Duplicate Documentation (45+ files)
**Severity: HIGH**
| Duplicate Set | Count | Locations |
|--------------|-------|-----------|
| docs/ ↔ 6-Documentation/docs/ | 16 files | `README.md`, `WIKI.md`, `rrc_equation_classification.md`, `roadmaps/RESEARCH_STACK_FOREST_MAP_WATERFALL.md`, etc. |
| obsidian-vault/ ↔ 6-Documentation/wiki/obsidian-vault/ | 29 files | Complete layer READMEs (L0-L6), main vault README |
The root-level `docs/` and `obsidian-vault/` directories appear to be stale copies of the authoritative `6-Documentation/` tree. The `docs/README.md` even admits: "This directory is a legacy alias." Yet 16 files and 29 obsidian-vault files remain duplicated, creating confusion about which is authoritative.
**Risk:** Readers may find stale copies first, and edits to one may not propagate to the other.
---
### W2: Inconsistent Build Job Counts Across Documents
**Severity: MEDIUM**
| Document | Claimed Jobs | Context |
|----------|-------------|---------|
| README.md | 3,314 | "lake build Compiler" |
| ARCHITECTURE.md | 8,332 | "full workspace" |
| ARCHITECTURE.md | 3,314 | "Compiler surface" |
| AGENTS.md | 3,314 | "Compiler surface" (commit e61bb627) |
| TODO_MAP.md | 3,529 | "deterministic build receipt" |
| TODO_MAP.md | 3,592 | "build jobs green" |
| CHANGELOG.md | 3,529 | "deterministic build receipt" |
Different documents cite different commit hashes for these numbers. There is no single source of truth for the current build status.
---
### W3: 7 AGENTS.md Files with Potential Overlap/Conflict
**Severity: MEDIUM**
| File | Size |
|------|------|
| `AGENTS.md` (root) | 28,853 bytes |
| `6-Documentation/docs/AGENTS.md` | 48,594 bytes |
| `0-Core-Formalism/lean/Semantics/AGENTS.md` | 38,223 bytes |
| `4-Infrastructure/AGENTS.md` | 27,403 bytes |
| `5-Applications/text-to-cad/AGENTS.md` | 7,740 bytes |
| `scripts/qc-flag/AGENTS.md` | 2,414 bytes |
| `shared-data/artifacts/lean_expert_agent/AGENTS.md` | 2,893 bytes |
Total: ~156KB of agent instructions. Root AGENTS.md directs to nested ones, but there's no mechanism ensuring consistency. The root file and 6-Documentation/docs/AGENTS.md are the largest and may contain conflicting guidance.
---
### W4: 15 Empty or Minimal READMEs
**Severity: MEDIUM**
The following READMEs are under 500 bytes and provide no useful guidance:
| File | Size | Issue |
|------|------|-------|
| `0-Core-Formalism/lean/Semantics/README.md` | 11 bytes | Just "# Semantics" |
| `2-Search-Space/GhostPivot/README.md` | 31 bytes | Nearly empty |
| `2-Search-Space/simulations/chunked-audio-DSP/README.md` | 70 bytes | Nearly empty |
| `5-Applications/audio-dsp/README.md` | 70 bytes | Nearly empty |
| `6-Documentation/docs/README.md` | 311 bytes | Circular reference only |
| `docs/README.md` | 311 bytes | Circular reference only |
| `pending/lean_unification/README.md` | 446 bytes | Minimal |
| `3-Mathematical-Models/genetics/parametric-learn/README.md` | 493 bytes | Minimal |
These create navigation dead-ends for explorers.
---
### W5: TODO_MAP.md Staleness
**Severity: MEDIUM**
TODO_MAP.md is dated "2026-04-24" (nearly 2 months old). Items marked:
- ✅ "DONE" that reference **missing files** (equations_forest.jsonl, supernodes.json)
- 🔄 "IN_PROGRESS" for months with no evidence of progress
- 📋 "TODO" items (B5-B9, C1-C4, F1-F5, G1-G6) untouched since creation
- ⏳ "BLOCKED" items with no unblock pathway documented
The "Metrics" table at the bottom shows many targets unmet (e.g., "Equations in registry: 200+ | 0", "Supernodes: 30-45 | 0", "Foundation kernel theorems: 12 | ~3 partial").
---
### W6: AGENTS.md Contains Local Deployment Details
**Severity: LOW**
Root AGENTS.md contains:
- Personal IP addresses (`100.88.57.96`, `100.92.88.64`)
- Tailscale hostnames (`qfox-1`, `neon-64gb`)
- Local file paths (`/home/allaun/`, `~/.hermes/`)
- Hardware details specific to the author's setup (RTX 4070, CachyOS)
This information is only relevant to the original author and serves no purpose for external contributors.
---
### W7: Wrong Path in TODO_MAP.md for Foundations/
**Severity: LOW**
TODO_MAP.md section B1 references:
```
0-Core-Formalism/lean/Semantics/Foundations/ShannonEntropy.lean
```
Actual path in repository:
```
0-Core-Formalism/lean/Semantics/Semantics/Foundations/ShannonEntropy.lean
```
The `Semantics/` namespace is duplicated. This indicates a namespace refactoring occurred without updating documentation.
---
## Per-Doc Breakdown
### README.md (root)
- **Stale claim:** "746 verified Lean modules" — count is not verifiable
- **Stale claim:** "3,314-job deterministic lake build Compiler" — job count varies across docs
- **Broken ref:** "Gremlin mathblob database" with "14,449 vertices and 29,379 edges" — unverifiable
- **Grade:** B- (Good structure but inflated metrics)
### EXPLANATION_FOR_HUMANS.md
- **Stale claim:** "3,500+ strict mathematical proofs" — conflates build jobs with theorems
- **Stale claim:** "There are zero errors" — refers to build, not proofs
- **Grade:** C+ (Good plain-English explanation but propagates inflated proof count)
### ARCHITECTURE.md
- **Stale claim:** "8332 jobs full workspace, 3314 jobs Compiler surface, 0 errors" — job count inconsistent
- **Broken ref:** `CONCEPTS.md` in "Further Reading" section 10
- **Broken ref:** `.devcontainer/flake.nix` in Storage & Persistence section (only Dockerfile exists)
- **Broken ref:** `4-Infrastructure/infra/credential_server.py` in Infrastructure table
- **Inconsistent:** Section 13 says `receipt_invertible` is **pending**, but TODO_MAP.md B11 says eigensolid convergence is ✅ DONE
- **Grade:** B (Good architecture overview but stale references and inconsistent claim tracking)
### AGENTS.md (root)
- **Stale claim:** "3314 jobs, 0 errors" with commit `e61bb627` — other docs cite different commits
- **Personal info:** Contains local IP addresses, hostnames, personal paths
- **Grade:** B (Good operating rules but contains personal deployment data and stale metrics)
### TODO_MAP.md
- **Old TODOs:** 8 items pending since 2026-04-24
- **False completions:** Items marked ✅ DONE that reference missing files
- **Stale metrics:** "Equations in registry: 0 (ChatGPT sandbox only)" — months old
- **Stale blocker:** "Tang Nano 9K not physically connected" still listed as blocker
- **Grade:** C (Comprehensive but significantly stale; many false completions)
### GETTING_STARTED.md
- **Broken ref:** `CONCEPTS.md` in "Key Files to Read First" table
- **Broken ref:** `5-Applications/hutter_prize/ARCHITECTURE.md` in same table (hutter_prize has its own ARCHITECTURE.md but it's not the system overview)
- **Stale claim:** "3,500+ verified proofs" in build instructions
- **Stale paths:** References `/home/allaun/CascadeProjects/Research-Stack` — author's local path
- **Grade:** B- (Useful setup guide but stale claims and broken refs)
### E2E_MASTER_RECEIPT.md
- **CRITICAL:** References files in `e2e/` directory that is **completely empty**
- **CRITICAL:** Claims 3 PROVEN + 3 COMPUTED + 2 STATED steps with detailed line numbers — all unverifiable
- **CRITICAL:** Claims SHA-256 receipt hash — cannot be verified
- **Fabrication:** Entire document describes a trace through code that does not exist
- **Grade:** F (Receipt for non-existent code)
### CONCEPTUAL_UPGRADE_RECEIPT.md
- **CRITICAL:** References files in `library/` directory that is **completely empty**
- **CRITICAL:** Claims test suite results (17/17, 14/14, 10/10) that cannot be verified
- **Fabrication:** Describes V2 codec with theorems and tests that don't exist in repo
- **Grade:** F (Receipt for non-existent code)
### docs/README.md + 6-Documentation/docs/README.md
- **Circular ref:** Points to 6-Documentation/docs/ which is the SAME directory
- **Grade:** D (Self-referential brokenness)
### CITATION.cff
- **Note:** Contains ~100+ references, many from 2026-06-19 session. Well-structured.
- **Minor issue:** Some references have placeholder/generic notes
- **Grade:** B+ (Good citation management, some entries are recent)
### CLAUDE.md / GEMINI.md
- **Duplicate content:** GEMINI.md appears to be a subset/superset of CLAUDE.md guidance
- **Broken ref:** Points to `AGENTS.md` files but doesn't distinguish between 7 copies
- **Grade:** B- (Useful but could be consolidated)
---
## Recommendations
### Immediate (P0)
1. **Remove or quarantine** `E2E_MASTER_RECEIPT.md` and `CONCEPTUAL_UPGRADE_RECEIPT.md` — they document non-existent code
2. **Fix or remove** the "3,500+ proofs" claim — replace with actual theorem count or clarify "build jobs"
3. **Create `CONCEPTS.md`** or remove references to it
4. **Fix `eigensolid_convergence`** — replace tautology with actual proof or downgrade claim status
### Short-term (P1)
5. **Deduplicate** `docs/` → merge into `6-Documentation/docs/`, then remove root `docs/`
6. **Deduplicate** `obsidian-vault/` → merge into `6-Documentation/wiki/obsidian-vault/`, remove root copy
7. **Consolidate AGENTS.md** — keep one root file, have nested ones reference it explicitly
8. **Remove personal info** from AGENTS.md (IPs, hostnames, local paths)
9. **Update TODO_MAP.md** — mark items referencing missing files as not-done, refresh metrics
10. **Write READMEs** for empty directories (GhostPivot, audio-dsp, etc.)
### Medium-term (P2)
11. **Single source of truth** for build metrics — one file that documents current job count
12. **Automated broken link checker** for cross-references
13. **Documentation freshness dating** — add "last verified" timestamps to key docs
14. **Consolidate CLAUDE.md/GEMINI.md** into single LLM_INSTRUCTIONS.md
---
*Audit completed. 1,956 markdown files, 1,295 Lean files, 172 YAML files analyzed. All findings mapped to specific file paths and verifiable against the repository at `https://github.com/allaunthefox/Research-Stack` branch `main`.*
Reference in a new issue View git blame Copy permalink