allaun/Research-Stack
1
0
Fork 0
mirror of https://github.com/allaunthefox/Research-Stack.git synced 2026-07-31 03:05:21 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
Research-Stack/audit/quality_audit.md
Allaun Silverfox 8e51acad08 audit: full codebase inspection — 6 auditors, 9,606 objects, 2,068 lines of findings 
CRITICAL (7):
- C1: Hardcoded Wolfram Alpha API key (revoke immediately)
- C2: Q16_16 rounding diverges Lean↔Python↔C (data corruption)
- C3: Receipts describe non-existent files (fabrication)
- C4: '3,500+ proofs' = compilation units, not theorems
- C5: NaN sentinel collision Lean↔C (silent corruption)
- C6: eigensolid_convergence theorem is a tautology
- C7: 74% CI failure rate

HIGH (12): duplicate definitions, SQL injection, command injection,
  1,029 duplicate files, 66 large files in git, AGENTS.md drift

MEDIUM (23): stale docs, broken cross-refs, unhandled errors
LOW (31): naming violations, misplaced files, cleanup

Master synthesis: audit/MASTER_AUDIT_SYNTHESIS.md
Per-dimension reports: audit/*_audit.md
2026-06-21 02:07:35 -05:00

409 lines
17 KiB
Markdown
Raw Blame History

# Quality Audit: Research-Stack
**Repository:** https://github.com/allaunthefox/Research-Stack
**Branch:** main
**Audit Date:** 2026-06-21
**Auditor:** Automated CI/CD & Code Quality Audit
---
## Summary
| Metric | Count |
|---|---|
| CI workflows (user-defined) | 4 |
| CI workflows (dynamic/dependabot/pages) | 3 |
| CI workflows passing (latest run) | 1/4 (Wolfram Alpha) |
| CI workflows failing (latest run) | 3/4 |
| Overall CI failure rate (last 100 runs) | 74% |
| Test files (name matches) | 1,282 |
| Actual unit test files (pytest/unittest) | 2 |
| Lean proof test files | 61 |
| Rust test files | 35 |
| Code duplications (duplicate basenames) | 627 |
| Experimental files on main | 57 |
| Large files (>1MB) | 66 |
| Binary files in git | 101 |
| .pist / .pist.meta generated artifacts | 192 |
| Receipt files | 104 |
| Pre-commit config | NOT FOUND |
| .gitattributes | Present (has contradictory LFS config) |
| Version pinning (Python) | Partially pinned (mixed == and >=) |
| Version pinning (Rust) | Locked (Cargo.lock present) |
| Version pinning (JS) | Locked (package-lock.json present) |
| Files needing cleanup | ~297+ |
---
## Critical Issues
### 1. CI/CD: 74% Failure Rate Across All Workflows (SEVERITY: CRITICAL)
The last 100 GitHub Actions runs show a **74% failure rate** (74 failures, 26 successes). This is a systemic CI reliability problem.
**Workflow-by-workflow status (latest run):**
| Workflow | Path | Status |
|---|---|---|
| Math-First Checks | `.github/workflows/math-check.yml` | **FAILING** |
| Batch Compute Runner | `.github/workflows/batch_compute.yml` | **FAILING** |
| Copilot Setup Steps | `.github/workflows/copilot-setup-steps.yml` | Unknown (path-triggered) |
| Wolfram Alpha Verification | `.github/workflows/wolfram-verification.yml` | **PASSING** |
| Pages Build & Deploy | `dynamic/pages/pages-build-deployment` | **FAILING** |
| Dependabot (cargo) | `dynamic/dependabot/dependabot-updates` | **FAILING** |
| Dependabot (npm) | `dynamic/dependabot/dependabot-updates` | Mixed (some pass, some fail) |
**Evidence:** API query `https://api.github.com/repos/allaunthefox/Research-Stack/actions/runs?per_page=100` returns `74 failure` vs `26 success` conclusions.
**Root causes likely include:**
- `math-check.yml` runs schema validation, receipt validation, self-tests, and pre-commit hooks on PR/push. Multiple steps can fail independently.
- `batch_compute.yml` runs daily compute jobs with external WASM dependencies.
- `pages-build-deployment` is a GitHub-managed Pages workflow that may fail due to build output issues.
- Dependabot updates fail on individual ecosystem updates (cargo, uv, npm) indicating dependency resolution issues.
**Impact:** Contributors cannot trust CI signals. Failures are so common they likely get ignored, masking real regressions.
**Remediation:**
1. Fix Math-First Checks first (core quality gate) — inspect job logs to identify failing steps.
2. Fix Pages build — likely a Jekyll/build output issue.
3. Remove or fix the daily Batch Compute workflow (harvests runner minutes, likely wasteful).
4. Stabilize Dependabot by pinning dependency versions and resolving ecosystem conflicts.
---
### 2. Missing Pre-commit Configuration File (SEVERITY: CRITICAL)
`math-check.yml` references `.pre-commit-config.yaml` in its `on.push.paths` trigger and runs `pre-commit run --from-ref "$base" --to-ref "$head"` in the `pre-commit` job.
However, **`.pre-commit-config.yaml` does NOT exist** at the repository root.
**Evidence:**
- The file tree API traversal finds no `.pre-commit-config.yaml`.
- Direct file content API returns 404.
- The CI job installs `pre-commit>=3.7` but has no hooks file to read.
**Impact:** The pre-commit CI job will fail every time it runs (no config file found). This explains one of the failure modes in Math-First Checks.
**Remediation:** Create a `.pre-commit-config.yaml` at the repo root, or remove the pre-commit job from CI.
---
### 3. 66 Large Files (>1MB) Committed to Git (SEVERITY: HIGH)
Multiple files over 1MB are stored in the repository, significantly impacting clone times and repository size. These should be tracked via Git LFS or stored externally.
**Largest offenders:**
| File | Size |
|---|---|
| `shared-data/papers/2026-05/2511.21459.pdf` | **44.88 MB** |
| `5-Applications/text-to-cad/skills/cad/assets/text-to-cad-demo.gif` | **25.07 MB** |
| `5-Applications/text-to-cad/skills/urdf/assets/text-to-cad-demo.gif` | **25.07 MB** |
| `3-Mathematical-Models/equations_compressed/unknown_equations_20260504_134248.compressed` | **24.23 MB** |
| `4-Infrastructure/hardware/cff_accelerator.json` | **13.84 MB** |
| `2-Search-Space/simulations/matter-frequencies/wav-files/*.wav` (17 files x ~11MB) | **~187 MB combined** |
| `2-Search-Space/simulations/Newtonian-Superfluid-Simulation/YouCut_20260426_155638157.gif` | **9.50 MB** |
| `2-Search-Space/simulations/Newtonian-Superfluid-Simulation/demo-simulation.mp4` | **9.44 MB** |
| `.github/assets/rainbow_raccoon_compiler.png` | **6.97 MB** |
**Impact:** Repository bloat. Fresh clones take significantly longer. CI jobs spend time downloading these files even when not needed.
**Remediation:**
1. Move all media files (`.wav`, `.gif`, `.mp4`, `.png`, `.pdf`) to Git LFS or external storage.
2. The `.gitattributes` file already has LFS filters configured but many of these large files are NOT tracked by LFS (the LFS config appears to be present but the files themselves may have been committed before `.gitattributes` was added).
3. Use `git lfs migrate` to move existing large files to LFS.
---
### 4. 192 Generated Artifacts (`.pist` / `.pist.meta`) Committed to Main (SEVERITY: HIGH)
The `0-Core-Formalism/lean/Semantics/` directory contains **96 `.lean.pist` files and 96 `.lean.pist.meta` files**. These are clearly build artifacts / generated files from the Lean compiler.
**Evidence:** Files like:
- `AdaptivePrecision.lean.pist`
- `AdaptivePrecision.lean.pist.meta`
- `FPGAExtraction.lean.pist`
- `MinimumNeuralCompression.lean.pist`
**Impact:** Generated files create merge conflicts, bloat diffs, and become stale. They should be `.gitignore`d and generated at build time.
**Remediation:** Add `*.pist` and `*.pist.meta` to `.gitignore` and remove all committed instances.
---
### 5. 104 Receipt Files (Generated Artifacts) in Repository (SEVERITY: MEDIUM)
The `shared-data/artifacts/deepseek_review/` directory contains receipt files (`.receipt.json`). These are CI-generated artifacts tracking automated review outputs.
**Impact:** Generated receipts create noise in the git history. The CI validates them against SHA-256 (which is good) but storing them in the repo is an anti-pattern for generated artifacts.
**Remediation:** Store receipts as CI artifacts (using `actions/upload-artifact`) rather than committing them to the repository.
---
## Warnings
### W1. Contradictory `.gitattributes` LFS Configuration
The `.gitattributes` file has contradictory rules for the same extensions:
```gitattributes
*.bin filter=lfs diff=lfs merge=lfs -text
*.bin !text !filter !merge !diff
```
Each file type has TWO rules — one enabling LFS and one disabling it. This is clearly a merge artifact or misconfiguration. The second rule overrides the first for many extensions, meaning LFS may not actually be applied.
**Files affected:** `.bin`, `.tar.gz`, `.npz`, `.db`, `.zip`, `.zst`, `.gguf`, `.parquet`, `.iso`, `.so`, `.tar.zst`, `.gz`, `.xz`, `.png`, `.jpg`, `.jpeg`, `.gif`, `.json`, `.step`, `.stp`, `.stl`
**Remediation:** Fix `.gitattributes` to have a single consistent rule per extension. Remove the `!filter !merge !diff` override lines.
---
### W2. Python Dependencies Partially Unpinned
Multiple `requirements.txt` files use `>=` (minimum version) instead of `==` (pinned):
**Unpinned (dangerous):**
- `4-Infrastructure/shim/requirements.txt`: Uses `>=` for ALL 20+ dependencies (`reedsolo>=1.7.0`, `cryptography>=41.0.0`, `qiskit>=1.0.0`, etc.)
- `5-Applications/requirements_swarm_api.txt`: Likely unpinned
**Pinned (good):**
- `2-Search-Space/manifold/requirements.txt`: Uses `==` (`fastapi==0.104.1`, `numpy==1.24.3`)
**No Python lock files** (no `poetry.lock`, no `requirements.lock`, no `Pipfile.lock`) at the root or in most subprojects.
**Impact:** A future release of any dependency (e.g., `qiskit>=1.0.0`, `numpy>=1.24.0`) can break builds without any code changes.
**Remediation:** Pin all Python dependencies with `==` and generate lock files. Use `pip-tools` or `poetry` to manage dependencies deterministically.
---
### W3. Duplicate Basenames Suggest Copy-Paste Code (627 instances)
**Most duplicated filenames:**
| Filename | Count | Likely Issue |
|---|---|---|
| `mod.rs` | 126 | Many separate Rust crates (expected for workspace) |
| `__init__.py` | 123 | Many Python packages (normal) |
| `lib.rs` | 44 | Rust library roots (normal for workspace) |
| `+server.ts` | 28 | SvelteKit API routes (Svelte pattern) |
| `main.rs` | 20 | Multiple entrypoints |
| `test_cli.py` | 6 | **Same test file name in 6 locations** |
| `test_coverage_gaps.py` | 6 | **Same test file name in 6 locations** |
The `test_cli.py` and `test_coverage_gaps.py` appearing in 6 locations each is suspicious — these may be copied between projects without modification, indicating the same test logic duplicated.
**Remediation:** Audit the `test_cli.py` and `test_coverage_gaps.py` files for actual duplication. Extract shared test utilities to a common module.
---
### W4. Lean Files with "Test" in Name That Are Not Tests (61 files)
Many Lean files in `0-Core-Formalism/lean/Semantics/**/Testing/` contain proof attempts and benchmarks, not automated unit tests. They won't be discovered or executed by any test runner.
**Examples:**
- `BitcoinRGFlowTest.lean` — a proof benchmark, not a runnable test
- `VirtualGPUBenchmark.lean` — a benchmark script
- `ReceiptReproducibilityTest.lean` — manual verification steps
**Impact:** These create false confidence. They look like tests but aren't executed by CI.
---
### W5. Workflow Trigger Paths Too Narrow
Several workflows only run on specific path changes, meaning they rarely trigger and failures may go undetected for long periods:
- `math-check.yml`: Only triggers on `0-Core-Formalism/lean/Semantics/**`, `shared-data/**`, etc.
- `wolfram-verification.yml`: Only triggers on `.lean` file changes.
- `batch_compute.yml`: Only runs on schedule (daily) or manual dispatch.
**Impact:** CI failures can persist undetected because the workflows don't run on most changes.
---
### W6. Dependabot Workflow Failures
Dependabot has been consistently failing on:
- `cargo in /4-Infrastructure/servo-fetch for time` — cargo ecosystem updates failing
- `uv in /. for h11` — Python dependency resolution failures
- `npm_and_yarn in /3-Mathematical-Models/bezier-kit for js-yaml` — JS dependency issues
**Impact:** Security patches and bug fixes are not being applied automatically.
---
## Per-Component Breakdown
### 0-Core-Formalism/ (Lean + Rust)
| Metric | Value |
|---|---|
| `.lean` files | 1,305 |
| `.rs` files | 942 |
| Lean test files (name match) | 61 |
| Rust test files (name match) | 35 |
| `.pist` generated artifacts | 96 |
| `.pist.meta` generated artifacts | 96 |
| `_output.txt` artifacts | 1 |
| CI workflows touching this | `math-check.yml`, `wolfram-verification.yml` |
**Issues:**
- 192 generated `.pist` files committed to git (CRITICAL)
- `Quarantine/Q064Experimental.lean` — clearly experimental file on main
- `ExperimentTracker.lean` — experimental tracking file
- `test.lean` at root of semantics directory — temporary test file
- `ExtremeParameterTestEval_output.txt` — generated output committed
---
### 1-Distributed-Systems/
| Metric | Value |
|---|---|
| Rust crates with tests | 3+ (claw: `api/tests/`, `runtime/tests/`, `rusty-claude-cli/tests/`) |
| CI coverage | None specific |
**Issues:**
- No dedicated CI workflow for the distributed systems code
- Tests exist but aren't run in CI
---
### 2-Search-Space/
| Metric | Value |
|---|---|
| Large media files | 17 WAV files (~187 MB), 2 GIFs (~12 MB), 1 MP4 (~9 MB) |
| `test_cli.py` duplicates | 6+ locations |
| Binary test data | 3 large HTML entity files (animal.html, aristotle.html, lion.html) |
**Issues:**
- Simulation media files (WAV, GIF, MP4) should be in LFS or external storage (HIGH)
- `testcases/entity/*.html` files (1-1.5 MB each) are test fixtures — acceptable but large
---
### 3-Mathematical-Models/
| Metric | Value |
|---|---|
| Large compressed file | `equations_compressed/unknown_equations_20260504_134248.compressed` (24.23 MB) |
| CI coverage | None specific |
**Issues:**
- Large opaque `.compressed` file — no indication of what's inside or how to regenerate it
- No tests for mathematical models
---
### 4-Infrastructure/
| Metric | Value |
|---|---|
| Rust crates | Multiple (servo-fetch, waveprobe, ene) |
| Python shims | Multiple |
| `requirements.txt` with unpinned deps | `4-Infrastructure/shim/requirements.txt` |
| Large JSON files | `cff_accelerator.json` (13.84 MB), `cern_pde_extraction.json` (6.44 MB) |
| `flake.nix` / `flake.lock` | Present at root, pinned to `nixos-24.11` |
**Issues:**
- `shim/requirements.txt` has 20+ unpinned `>=` dependencies (WARNING)
- Large JSON data files committed directly (should be LFS)
- `k3s-flake/tests/package.json` — no lock file validation in CI
---
### 5-Applications/
| Metric | Value |
|---|---|
| Large GIF duplicates | `text-to-cad-demo.gif` appears in 2 locations (25.07 MB each) |
| Python backend | `cluster-dashboard/backend/requirements.txt` |
| CI coverage | None specific |
**Issues:**
- 50+ MB of duplicate GIF files (same file in two skill directories)
- No application-level CI testing
---
### .github/workflows/
| Workflow | Size | Purpose | Status |
|---|---|---|---|
| `batch_compute.yml` | 2,074 B | Daily compute job | FAILING |
| `copilot-setup-steps.yml` | 432 B | Lean setup helper | Unknown |
| `math-check.yml` | 6,221 B | Schema + receipt validation | FAILING |
| `wolfram-verification.yml` | 5,577 B | Math formula verification | PASSING |
**Issues:**
- `math-check.yml` references `.pre-commit-config.yaml` that doesn't exist
- `actions/checkout@v6` used in some workflows but `actions/checkout@v4` in others — version inconsistency
- `setup-python@v5` vs `setup-python@v6` — version inconsistency
- `upload-artifact@v4` — good, latest version
- `batch_compute.yml` installs `ffmpeg`, `numpy`, `requests`, `cryptography`, `reedsolo`, `highspy` without version pinning
- No `workflow_call` or reusable workflow patterns — everything is duplicated
---
## File Inventory
### Files Needing Cleanup (delete or .gitignore)
| Category | Count | Pattern |
|---|---|---|
| `.lean.pist` files | 96 | `*.lean.pist` — Lean build artifacts |
| `.lean.pist.meta` files | 96 | `*.lean.pist.meta` — Lean build metadata |
| `_output.txt` files | 1 | `*_output.txt` — generated test output |
| `.consolidation-manifests/` | 2 | Large SHA256 manifest files (1-1.5 MB) |
| Receipt JSON files | 104 | `shared-data/artifacts/deepseek_review/*.receipt.json` |
| Test output / temp | Various | `test-results/`, `.pist`, `.meta` |
### Experimental/Temporary Files on Main
| File | Reason |
|---|---|
| `Q064Experimental.lean` | Contains "Experimental" in name |
| `ExperimentTracker.lean` | Experimental tracking |
| `RydbergExperimentalTest.lean` | Experimental physics test |
| `experiment-001.html` | HTML experiment prototype |
| `test.lean` | Root-level temp test file |
| `Experiment.svelte`, `ExperimentProgress.svelte` | Svelte experiment UI |
| `ranking-diff/src/routes/api/experiments/` | Entire experiment API directory |
### Large Files Requiring LFS or External Storage
| File | Size |
|---|---|
| `shared-data/papers/2026-05/2511.21459.pdf` | 44.88 MB |
| `text-to-cad-demo.gif` (x2) | 50.14 MB total |
| `unknown_equations_*.compressed` | 24.23 MB |
| `cff_accelerator.json` | 13.84 MB |
| `*.wav` files (x17) | ~187 MB total |
| `YouCut_*.gif` | 9.50 MB |
| `demo-simulation.mp4` | 9.44 MB |
| `lean_dependency_dag_parallel.svg` | 8.38 MB |
| `rainbow_raccoon_compiler.png` | 6.97 MB |
| `cern_pde_extraction.json` | 6.44 MB |
---
## Recommendations (Priority Order)
1. **Create `.pre-commit-config.yaml`** or remove the pre-commit job from `math-check.yml` (CRITICAL)
2. **Fix Math-First Checks CI** — inspect logs, fix failing steps (CRITICAL)
3. **Remove `.pist` / `.pist.meta` files** from git and add to `.gitignore` (HIGH)
4. **Move large files to Git LFS** using `git lfs migrate` (HIGH)
5. **Pin Python dependencies** — convert all `>=` to `==`, add lock files (HIGH)
6. **Fix `.gitattributes`** — remove contradictory LFS rules (MEDIUM)
7. **Move receipt files** to CI artifacts instead of committed files (MEDIUM)
8. **Add CI workflows** for Rust and Python test suites (MEDIUM)
9. **Consolidate duplicate test files** (`test_cli.py`, `test_coverage_gaps.py`) (MEDIUM)
10. **Stabilize Dependabot** by resolving ecosystem conflicts (LOW)
---
*End of Audit Report*
Reference in a new issue View git blame Copy permalink