Research-Stack/audit/quality_audit.md

Quality Audit: Research-Stack

Repository: https://github.com/allaunthefox/Research-Stack
Branch: main
Audit Date: 2026-06-21
Auditor: Automated CI/CD & Code Quality Audit

---

Summary

Metric	Count
CI workflows (user-defined)	4
CI workflows (dynamic/dependabot/pages)	3
CI workflows passing (latest run)	1/4 (Wolfram Alpha)
CI workflows failing (latest run)	3/4
Overall CI failure rate (last 100 runs)	74%
Test files (name matches)	1,282
Actual unit test files (pytest/unittest)	2
Lean proof test files	61
Rust test files	35
Code duplications (duplicate basenames)	627
Experimental files on main	57
Large files (>1MB)	66
Binary files in git	101
.pist / .pist.meta generated artifacts	192
Receipt files	104
Pre-commit config	NOT FOUND
.gitattributes	Present (has contradictory LFS config)
Version pinning (Python)	Partially pinned (mixed == and >=)
Version pinning (Rust)	Locked (Cargo.lock present)
Version pinning (JS)	Locked (package-lock.json present)
Files needing cleanup	~297+

---

Critical Issues

1. CI/CD: 74% Failure Rate Across All Workflows (SEVERITY: CRITICAL)

The last 100 GitHub Actions runs show a 74% failure rate (74 failures, 26 successes). This is a systemic CI reliability problem.

Workflow-by-workflow status (latest run):

Workflow	Path	Status
Math-First Checks	.github/workflows/math-check.yml	FAILING
Batch Compute Runner	.github/workflows/batch_compute.yml	FAILING
Copilot Setup Steps	.github/workflows/copilot-setup-steps.yml	Unknown (path-triggered)
Wolfram Alpha Verification	.github/workflows/wolfram-verification.yml	PASSING
Pages Build & Deploy	dynamic/pages/pages-build-deployment	FAILING
Dependabot (cargo)	dynamic/dependabot/dependabot-updates	FAILING
Dependabot (npm)	dynamic/dependabot/dependabot-updates	Mixed (some pass, some fail)

Evidence: API query https://api.github.com/repos/allaunthefox/Research-Stack/actions/runs?per_page=100 returns 74 failure vs 26 success conclusions.

Root causes likely include:

• math-check.yml runs schema validation, receipt validation, self-tests, and pre-commit hooks on PR/push. Multiple steps can fail independently.
• batch_compute.yml runs daily compute jobs with external WASM dependencies.
• pages-build-deployment is a GitHub-managed Pages workflow that may fail due to build output issues.
• Dependabot updates fail on individual ecosystem updates (cargo, uv, npm) indicating dependency resolution issues.

Impact: Contributors cannot trust CI signals. Failures are so common they likely get ignored, masking real regressions.

Remediation:

1. Fix Math-First Checks first (core quality gate) — inspect job logs to identify failing steps.
2. Fix Pages build — likely a Jekyll/build output issue.
3. Remove or fix the daily Batch Compute workflow (harvests runner minutes, likely wasteful).
4. Stabilize Dependabot by pinning dependency versions and resolving ecosystem conflicts.

---

2. Missing Pre-commit Configuration File (SEVERITY: CRITICAL)

math-check.yml references .pre-commit-config.yaml in its on.push.paths trigger and runs pre-commit run --from-ref "$base" --to-ref "$head" in the pre-commit job.

However, .pre-commit-config.yaml does NOT exist at the repository root.

Evidence:

• The file tree API traversal finds no .pre-commit-config.yaml.
• Direct file content API returns 404.
• The CI job installs pre-commit>=3.7 but has no hooks file to read.

Impact: The pre-commit CI job will fail every time it runs (no config file found). This explains one of the failure modes in Math-First Checks.

Remediation: Create a .pre-commit-config.yaml at the repo root, or remove the pre-commit job from CI.

---

3. 66 Large Files (>1MB) Committed to Git (SEVERITY: HIGH)

Multiple files over 1MB are stored in the repository, significantly impacting clone times and repository size. These should be tracked via Git LFS or stored externally.

Largest offenders:

File	Size
shared-data/papers/2026-05/2511.21459.pdf	44.88 MB
5-Applications/text-to-cad/skills/cad/assets/text-to-cad-demo.gif	25.07 MB
5-Applications/text-to-cad/skills/urdf/assets/text-to-cad-demo.gif	25.07 MB
3-Mathematical-Models/equations_compressed/unknown_equations_20260504_134248.compressed	24.23 MB
4-Infrastructure/hardware/cff_accelerator.json	13.84 MB
2-Search-Space/simulations/matter-frequencies/wav-files/*.wav (17 files x ~11MB)	~187 MB combined
2-Search-Space/simulations/Newtonian-Superfluid-Simulation/YouCut_20260426_155638157.gif	9.50 MB
2-Search-Space/simulations/Newtonian-Superfluid-Simulation/demo-simulation.mp4	9.44 MB
.github/assets/rainbow_raccoon_compiler.png	6.97 MB

Impact: Repository bloat. Fresh clones take significantly longer. CI jobs spend time downloading these files even when not needed.

Remediation:

1. Move all media files (.wav, .gif, .mp4, .png, .pdf) to Git LFS or external storage.
2. The .gitattributes file already has LFS filters configured but many of these large files are NOT tracked by LFS (the LFS config appears to be present but the files themselves may have been committed before .gitattributes was added).
3. Use git lfs migrate to move existing large files to LFS.

---

4. 192 Generated Artifacts (.pist / .pist.meta) Committed to Main (SEVERITY: HIGH)

The 0-Core-Formalism/lean/Semantics/ directory contains 96 .lean.pist files and 96 .lean.pist.meta files. These are clearly build artifacts / generated files from the Lean compiler.

Evidence: Files like:

• AdaptivePrecision.lean.pist
• AdaptivePrecision.lean.pist.meta
• FPGAExtraction.lean.pist
• MinimumNeuralCompression.lean.pist

Impact: Generated files create merge conflicts, bloat diffs, and become stale. They should be .gitignored and generated at build time.

Remediation: Add *.pist and *.pist.meta to .gitignore and remove all committed instances.

---

5. 104 Receipt Files (Generated Artifacts) in Repository (SEVERITY: MEDIUM)

The shared-data/artifacts/deepseek_review/ directory contains receipt files (.receipt.json). These are CI-generated artifacts tracking automated review outputs.

Impact: Generated receipts create noise in the git history. The CI validates them against SHA-256 (which is good) but storing them in the repo is an anti-pattern for generated artifacts.

Remediation: Store receipts as CI artifacts (using actions/upload-artifact) rather than committing them to the repository.

---

Warnings

W1. Contradictory .gitattributes LFS Configuration

The .gitattributes file has contradictory rules for the same extensions:

*.bin filter=lfs diff=lfs merge=lfs -text
*.bin !text !filter !merge !diff

Each file type has TWO rules — one enabling LFS and one disabling it. This is clearly a merge artifact or misconfiguration. The second rule overrides the first for many extensions, meaning LFS may not actually be applied.

Files affected: .bin, .tar.gz, .npz, .db, .zip, .zst, .gguf, .parquet, .iso, .so, .tar.zst, .gz, .xz, .png, .jpg, .jpeg, .gif, .json, .step, .stp, .stl

Remediation: Fix .gitattributes to have a single consistent rule per extension. Remove the !filter !merge !diff override lines.

---

W2. Python Dependencies Partially Unpinned

Multiple requirements.txt files use >= (minimum version) instead of == (pinned):

Unpinned (dangerous):

• 4-Infrastructure/shim/requirements.txt: Uses >= for ALL 20+ dependencies (reedsolo>=1.7.0, cryptography>=41.0.0, qiskit>=1.0.0, etc.)
• 5-Applications/requirements_swarm_api.txt: Likely unpinned

Pinned (good):

• 2-Search-Space/manifold/requirements.txt: Uses == (fastapi==0.104.1, numpy==1.24.3)

No Python lock files (no poetry.lock, no requirements.lock, no Pipfile.lock) at the root or in most subprojects.

Impact: A future release of any dependency (e.g., qiskit>=1.0.0, numpy>=1.24.0) can break builds without any code changes.

Remediation: Pin all Python dependencies with == and generate lock files. Use pip-tools or poetry to manage dependencies deterministically.

---

W3. Duplicate Basenames Suggest Copy-Paste Code (627 instances)

Most duplicated filenames:

Filename	Count	Likely Issue
mod.rs	126	Many separate Rust crates (expected for workspace)
__init__.py	123	Many Python packages (normal)
lib.rs	44	Rust library roots (normal for workspace)
+server.ts	28	SvelteKit API routes (Svelte pattern)
main.rs	20	Multiple entrypoints
test_cli.py	6	Same test file name in 6 locations
test_coverage_gaps.py	6	Same test file name in 6 locations

The test_cli.py and test_coverage_gaps.py appearing in 6 locations each is suspicious — these may be copied between projects without modification, indicating the same test logic duplicated.

Remediation: Audit the test_cli.py and test_coverage_gaps.py files for actual duplication. Extract shared test utilities to a common module.

---

W4. Lean Files with "Test" in Name That Are Not Tests (61 files)

Many Lean files in 0-Core-Formalism/lean/Semantics/**/Testing/ contain proof attempts and benchmarks, not automated unit tests. They won't be discovered or executed by any test runner.

Examples:

• BitcoinRGFlowTest.lean — a proof benchmark, not a runnable test
• VirtualGPUBenchmark.lean — a benchmark script
• ReceiptReproducibilityTest.lean — manual verification steps

Impact: These create false confidence. They look like tests but aren't executed by CI.

---

W5. Workflow Trigger Paths Too Narrow

Several workflows only run on specific path changes, meaning they rarely trigger and failures may go undetected for long periods:

• math-check.yml: Only triggers on 0-Core-Formalism/lean/Semantics/**, shared-data/**, etc.
• wolfram-verification.yml: Only triggers on .lean file changes.
• batch_compute.yml: Only runs on schedule (daily) or manual dispatch.

Impact: CI failures can persist undetected because the workflows don't run on most changes.

---

W6. Dependabot Workflow Failures

Dependabot has been consistently failing on:

• cargo in /4-Infrastructure/servo-fetch for time — cargo ecosystem updates failing
• uv in /. for h11 — Python dependency resolution failures
• npm_and_yarn in /3-Mathematical-Models/bezier-kit for js-yaml — JS dependency issues

Impact: Security patches and bug fixes are not being applied automatically.

---

Per-Component Breakdown

0-Core-Formalism/ (Lean + Rust)

Metric	Value
.lean files	1,305
.rs files	942
Lean test files (name match)	61
Rust test files (name match)	35
.pist generated artifacts	96
.pist.meta generated artifacts	96
_output.txt artifacts	1
CI workflows touching this	math-check.yml, wolfram-verification.yml

Issues:

• 192 generated .pist files committed to git (CRITICAL)
• Quarantine/Q064Experimental.lean — clearly experimental file on main
• ExperimentTracker.lean — experimental tracking file
• test.lean at root of semantics directory — temporary test file
• ExtremeParameterTestEval_output.txt — generated output committed

---

1-Distributed-Systems/

Metric	Value
Rust crates with tests	3+ (claw: api/tests/, runtime/tests/, rusty-claude-cli/tests/)
CI coverage	None specific

Issues:

• No dedicated CI workflow for the distributed systems code
• Tests exist but aren't run in CI

---

2-Search-Space/

Metric	Value
Large media files	17 WAV files (~187 MB), 2 GIFs (~12 MB), 1 MP4 (~9 MB)
test_cli.py duplicates	6+ locations
Binary test data	3 large HTML entity files (animal.html, aristotle.html, lion.html)

Issues:

• Simulation media files (WAV, GIF, MP4) should be in LFS or external storage (HIGH)
• testcases/entity/*.html files (1-1.5 MB each) are test fixtures — acceptable but large

---

3-Mathematical-Models/

Metric	Value
Large compressed file	equations_compressed/unknown_equations_20260504_134248.compressed (24.23 MB)
CI coverage	None specific

Issues:

• Large opaque .compressed file — no indication of what's inside or how to regenerate it
• No tests for mathematical models

---

4-Infrastructure/

Metric	Value
Rust crates	Multiple (servo-fetch, waveprobe, ene)
Python shims	Multiple
requirements.txt with unpinned deps	4-Infrastructure/shim/requirements.txt
Large JSON files	cff_accelerator.json (13.84 MB), cern_pde_extraction.json (6.44 MB)
flake.nix / flake.lock	Present at root, pinned to nixos-24.11

Issues:

• shim/requirements.txt has 20+ unpinned >= dependencies (WARNING)
• Large JSON data files committed directly (should be LFS)
• k3s-flake/tests/package.json — no lock file validation in CI

---

5-Applications/

Metric	Value
Large GIF duplicates	text-to-cad-demo.gif appears in 2 locations (25.07 MB each)
Python backend	cluster-dashboard/backend/requirements.txt
CI coverage	None specific

Issues:

• 50+ MB of duplicate GIF files (same file in two skill directories)
• No application-level CI testing

---

.github/workflows/

Workflow	Size	Purpose	Status
batch_compute.yml	2,074 B	Daily compute job	FAILING
copilot-setup-steps.yml	432 B	Lean setup helper	Unknown
math-check.yml	6,221 B	Schema + receipt validation	FAILING
wolfram-verification.yml	5,577 B	Math formula verification	PASSING

Issues:

• math-check.yml references .pre-commit-config.yaml that doesn't exist
• actions/checkout@v6 used in some workflows but actions/checkout@v4 in others — version inconsistency
• setup-python@v5 vs setup-python@v6 — version inconsistency
• upload-artifact@v4 — good, latest version
• batch_compute.yml installs ffmpeg, numpy, requests, cryptography, reedsolo, highspy without version pinning
• No workflow_call or reusable workflow patterns — everything is duplicated

---

File Inventory

Files Needing Cleanup (delete or .gitignore)

Category	Count	Pattern
.lean.pist files	96	*.lean.pist — Lean build artifacts
.lean.pist.meta files	96	*.lean.pist.meta — Lean build metadata
_output.txt files	1	*_output.txt — generated test output
.consolidation-manifests/	2	Large SHA256 manifest files (1-1.5 MB)
Receipt JSON files	104	shared-data/artifacts/deepseek_review/*.receipt.json
Test output / temp	Various	test-results/, .pist, .meta

Experimental/Temporary Files on Main

File	Reason
Q064Experimental.lean	Contains "Experimental" in name
ExperimentTracker.lean	Experimental tracking
RydbergExperimentalTest.lean	Experimental physics test
experiment-001.html	HTML experiment prototype
test.lean	Root-level temp test file
Experiment.svelte, ExperimentProgress.svelte	Svelte experiment UI
ranking-diff/src/routes/api/experiments/	Entire experiment API directory

Large Files Requiring LFS or External Storage

File	Size
shared-data/papers/2026-05/2511.21459.pdf	44.88 MB
text-to-cad-demo.gif (x2)	50.14 MB total
unknown_equations_*.compressed	24.23 MB
cff_accelerator.json	13.84 MB
*.wav files (x17)	~187 MB total
YouCut_*.gif	9.50 MB
demo-simulation.mp4	9.44 MB
lean_dependency_dag_parallel.svg	8.38 MB
rainbow_raccoon_compiler.png	6.97 MB
cern_pde_extraction.json	6.44 MB

---

Recommendations (Priority Order)

1. Create .pre-commit-config.yaml or remove the pre-commit job from math-check.yml (CRITICAL)
2. Fix Math-First Checks CI — inspect logs, fix failing steps (CRITICAL)
3. Remove .pist / .pist.meta files from git and add to .gitignore (HIGH)
4. Move large files to Git LFS using git lfs migrate (HIGH)
5. Pin Python dependencies — convert all >= to ==, add lock files (HIGH)
6. Fix .gitattributes — remove contradictory LFS rules (MEDIUM)
7. Move receipt files to CI artifacts instead of committed files (MEDIUM)
8. Add CI workflows for Rust and Python test suites (MEDIUM)
9. Consolidate duplicate test files (test_cli.py, test_coverage_gaps.py) (MEDIUM)
10. Stabilize Dependabot by resolving ecosystem conflicts (LOW)

---

End of Audit Report