mirror of https://github.com/allaunthefox/Research-Stack.git synced 2026-07-31 03:05:21 +00:00

Brandon Schneider 5f88abf618 initial: sovereign research stack (consolidated, weightless, and lfs-optimized)

2026-05-04 18:11:36 -05:00

16 KiB

Raw Blame History

Unified JSON-L Schema Specification — Manifold Ingestion Stream

Version: 1.0
Status: ACTIVE — replaces discrete Notion/Linear/ENE shims
Ground Truth: 0-Core-Formalism/lean/Semantics/ (bind bridge invariant preservation)
Target: data/substrate_index.db — unified ENE substrate

1. Design Principles

Single Source of Truth: Every state change — whether from Notion, Linear, ENE, or the blockchain proxy — reduces to one JSON-L line. The ENE SQLite substrate is the manifold.
Bind Bridge Compliance: Every event carries a bind struct with lawful (Bool), cost (Q16_16 UInt32), and invariant (String). No Python/Rust shim may compute cost; it is extracted from Lean or propagated from upstream.
Temporal Immutability: t is the single time axis. Events are append-only. Corrections are new events with op: "correct" referencing prev_id.
Genome Addressability: Every event MAY carry a genome field. If present, it is a 6D quantized signature deterministically hashable to an 18-bit address (address = genome.hash() & 0x3FFFF). This allows the event to be routed into the same hierarchical multicast buckets as blockchain blocks.
No Open Strings in Decisions: The src field is Fin 5 (notion, linear, ene, rgflow, swarm). The op field is Fin 5 (upsert, delete, snapshot, correct, attest). All other categorical fields use finite enumerations.

2. Mandatory Root Fields

Field	Type	Description
`t`	`f64`	Unix timestamp with millisecond resolution. Monotonic within a single `src`.
`src`	`Fin 5`	Origin: `notion`, `linear`, `ene`, `rgflow`, `swarm`.
`id`	`String`	Unique within `src`. Format: `<src>:<uuid>` or deterministic hash.
`op`	`Fin 5`	`upsert` \| `delete` \| `snapshot` \| `correct` \| `attest`.
`data`	`Dict`	Source-specific payload. Schema varies by `src` (see §4).
`genome`	`Genome` (optional)	6D quantized spectral signature for RGFlow routing.
`bind`	`Bind` (optional)	Cost, lawful check, invariant extractor. Required for CORE-tier events.
`provenance`	`Provenance`	Node, lake seed, and attestation chain.

3. Common Sub-Structures

3.1 `Genome` — RGFlow Addressable Signature

{
  "mu":   0,  // UInt8  — compression ratio bin (0..7)
  "rho":  0,  // UInt8  — information density bin (0..7)
  "c":    0,  // UInt8  — thermodynamic cost bin (0..7)
  "m":    0,  // UInt8  — manifold curvature bin (0..7)
  "ne":   0,  // UInt8  — negentropy flux bin (0..7)
  "sig":  0   // UInt8  — signal-to-noise bin (0..7)
}

Address: address = (mu << 15) | (rho << 12) | (c << 9) | (m << 6) | (ne << 3) | sig → 18-bit value 0..262143.
Bucket: bucket = address >> 15 → top 3 bits, 0..7. Used for multicast routing.
Determinism: Every src MUST define a pure function event → Genome with no randomness. The genome is a lossy projection of data, not a hash.

3.2 `Bind` — Invariant Preservation Witness

{
  "lawful": true,           // Bool — does this event preserve all invariants?
  "cost": 65536,            // UInt32 Q16_16 — computational/thermodynamic cost
  "invariant": "bindPreservesState: t_k+1 > t_k => forwardBlit", // String
  "class": "informational_bind"  // Fin 5: informational | geometric | thermodynamic | physical | control
}

Cost computation: For rgflow events, cost is the byte count of the compressed genome. For ene events, cost is the L2 norm of the concept vector scaled to Q16_16. For notion/linear, cost is fixed at 0x00010000 (1.0) until a Lean cost function is extracted.
Rule: If lawful == false, the event is written to the substrate but flagged in swarm_work_queue for review. No downstream manifold blit occurs.

3.3 `Provenance` — Attestation Chain

{
  "node": "qfox",              // String — hostname of originating node
  "lake_seed": "nnyyP7DoHS11CNTRL", // String — Tailscale stable node ID
  "tailscale_ip": "100.105.111.120", // String — for routing verification
  "attestation_hash": "sha256:...",  // String — cumulative hash of event + prev attestation
  "prev_id": null            // String or null — previous event in causal chain
}

Attestation: Every event is attested by the node that generated it. The hash chain allows back-trace re-sync when out-of-order events are detected.
Swarm propagation: When an event is pushed via /sync/manifest, the provenance.node field is updated to the forwarding node, preserving the original in a nested origin field.

4. Source-Specific `data` Schemas

4.1 `src: "notion"` — Document Hierarchy Event

{
  "page_id": "7a27bd85-9ff0-44fe-b873-7bb90f2a91a0",
  "block_id": "...",
  "parent_id": "...",
  "title": "Quantum Hamiltonian Manifold",
  "content_hash": "b3f5...f6bf",
  "last_edited_time": "2026-04-24T13:42:00.000Z",
  "property_changes": {
    "Status": {"old": "Draft", "new": "Review"}
  }
}

Genome mapping: mu = len(title) % 8, rho = popcount(content_hash) % 8, c = property_changes.count % 8, m = 4 (stable document), ne = 7 (high semantic density), sig = tier_index("RESEARCH").
ENE mapping: Upsert into packages table. pkg = notion/<page_id>, archetype = "document", concept_anchor.resolution = property_changes["Status"].new.

4.2 `src: "linear"` — Causal Dependency Event

{
  "issue_id": "RES-7",
  "linear_id": "d8f2c1b4-...",
  "title": "Implement bind bridge for Q16_16",
  "state": {"old": "Todo", "new": "In Progress"},
  "priority": {"old": 2, "new": 1},
  "assignee": {"old": null, "new": "allaun"},
  "blocked_by": ["RES-3", "RES-5"],
  "identifier": "RES-7"
}

Genome mapping: mu = priority.new % 8, rho = len(blocked_by) % 8, c = state_transition_cost(state.old, state.new), m = 2 (forming), ne = assignee.new ? 4 : 0, sig = 0.
ENE mapping: Upsert into packages. pkg = linear/<issue_id>, archetype = "issue", session_id = linear_id. concept_anchor.resolution maps from Linear state: Todo→SEED, In Progress→FORMING, Done→STABLE, Canceled→RECOVERED.

4.3 `src: "ene"` — Substrate Atomic Ingestion

{
  "pkg": "semantics/Adaptation",
  "version": "2026-04-24T12:00:00Z",
  "tier": "CORE",
  "domain": "formalization",
  "archetype": "lean_module",
  "concept_anchor": {
    "domain": "formalization",
    "concept": "flow_audit_loop",
    "resolution": "CRYSTALLIZED"
  },
  "concept_vector": [1.0, 0.618, 0.0, 0.0, 0.146, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
  "idea_weights": {"bind primitive": 0.95, "Q16_16": 0.88},
  "files": ["0-Core-Formalism/lean/Semantics/Adaptation.lean"],
  "depends": ["semantics/Q16_16/2026-04-20T00:00:00Z"]
}

Genome mapping: Derived from concept_vector via axis quantization: each dimension is binned to 3 bits using floor(dim * 8 / max_dim).
ENE mapping: Direct upsert into packages table. All fields map 1:1.

4.4 `src: "rgflow"` — Blockchain Genome Event

{
  "chain": "btc",
  "height": 548000,
  "block_hash": "0000000000000000000...",
  "timestamp": 1777031220,
  "genome": {"mu": 3, "rho": 7, "c": 2, "m": 5, "ne": 1, "sig": 6},
  "address": 262143,
  "bucket": 7,
  "raw": {"size": 1456234, "tx_count": 2847, "difficulty": 1.23e14}
}

Genome mapping: Already computed by bitcoin_block_to_genome() / ethereum_block_to_genome() in scripts/blockchain_rgflow_proxy.py. Deterministic 6D quantization from block metadata.
ENE mapping: Upsert into swarm_manifest. t = timestamp, node = provenance.node, genome = json.dumps(genome).

4.5 `src: "swarm"` — Topology / Work Queue Event

{
  "event_type": "topology_change",  // topology_change | task_complete | node_failure | sync_manifest
  "hostname": "architect",
  "tailscale_ip": "100.127.111.7",
  "status": "active",
  "latency_ms": 281.0,
  "load": 0.34,
  "manifest_inserted": 42  // only for sync_manifest events
}

Genome mapping: mu = latency_ms < 100 ? 0 : latency_ms < 300 ? 2 : 4, rho = load * 8 % 8, c = status == "active" ? 0 : 7, m = 4, ne = manifest_inserted > 0 ? 7 : 0, sig = 0.
ENE mapping: Upsert into swarm_nodes or swarm_work_queue depending on event_type.

5. Manifold Convergence Rules

5.1 Forward Blit (Normal Case)

For any source, if Upsert(t_k) is followed by Upsert(t_{k+1}) where t_{k+1} > t_k:

The event is lawful by monotonicity.
The manifold state is updated via a forward blit: state' = state ⊕ data.
The bind.cost is additive: cost_total = cost_total + bind.cost.

5.2 Back-Trace Re-Sync (Out-of-Order)

If an event arrives with t_k < t_latest:

The event is written to the substrate with op: "correct" and prev_id pointing to the event it supersedes.
A back-trace is initiated: the SRAM Scaffolding buffer (last 1024 events in swarm_query_cache) is replayed from t_k to t_latest.
Each replayed event is re-evaluated for bind.lawful. If any event becomes unlawful, the entire segment is quarantined in swarm_work_queue with status: "failed".
The manifold is reconstructed from the quarantine boundary forward.

5.3 SRAM Scaffolding Buffer

The buffer is a circular log of the last 1024 JSON-L events, stored in swarm_query_cache:

query_hash = sha256(event.id + event.t)
subjects = event.src
results = json.dumps(event)
semantic_vector = event.genome || event.data.concept_vector || zeros(14)
ttl = 86400 (24 hours)

6. Ingestion Pipeline

┌─────────────┐   ┌─────────────┐   ┌─────────────┐   ┌─────────────┐
│   Notion    │   │   Linear    │   │    ENE      │   │   RGFlow    │
│   (pull)    │   │   (webhook) │   │   ( Lake )  │   │   (ZMQ/WS)  │
└──────┬──────┘   └──────┬──────┘   └──────┬──────┘   └──────┬──────┘
       │                 │                 │                 │
       └─────────────────┴─────────────────┴─────────────────┘
                           │
                    ┌──────▼──────┐
                    │  JSON-L     │
                    │  Canonical  │
                    │  Converter  │
                    └──────┬──────┘
                           │
                    ┌──────▼──────┐
                    │  Bind Bridge│
                    │  (Lean #eval│
                    │   or cached)│
                    └──────┬──────┘
                           │
              ┌────────────┼────────────┐
              │            │            │
       ┌──────▼──────┐ ┌───▼────┐ ┌────▼─────┐
       │  ENE SQLite │ │Swarm   │ │Multicast │
       │  substrate  │ │Surface │ │239.255.x │
       │  (master)   │ │(mesh)  │ │(buckets) │
       └─────────────┘ └────────┘ └──────────┘

6.1 SQLite Insert Paths

`src`	Target Table	Unique Key	Conflict Resolution
`notion`	`packages`	`pkg = notion/<page_id>`	REPLACE (newer `t` wins)
`linear`	`packages`	`pkg = linear/<issue_id>`	REPLACE
`ene`	`packages`	`(pkg, version)`	REPLACE
`rgflow`	`swarm_manifest`	`(chain, height)`	INSERT OR IGNORE
`swarm`	`swarm_nodes` / `swarm_work_queue`	`hostname` / `task_id`	REPLACE

6.2 Swarm Surface Sync

After SQLite insertion on qfox:

The push loop (_push_loop in swarm_surface.py) broadcasts the new manifest entry to all active peers.
Peers receive via /sync/manifest and insert into their local ENE DB.
If a peer is offline, the entry remains in qfox's DB. When the peer comes back online, the pull loop (_pull_loop) catches up by querying /manifest?limit=1000.

7. Verification Requirements

Per AGENTS.md §4, every def that computes a cost or invariant must have a witness. The JSON-L schema enforces this via:

Eval witness: Every src converter script must include a #eval-style test block (in Lean) or a Python assert block with expected output in a comment.
Theorem witness: The Bind struct must satisfy bindPreservesInvariant : ∀ e : Event, e.bind.lawful → invariant(e.data) = invariant(state').
Totality: No partial functions in the conversion pipeline. Every event → Genome mapping is total and deterministic.

8. Example Stream

{"t":1777042800.000,"src":"rgflow","id":"rgflow:btc:548000","op":"upsert","data":{"chain":"btc","height":548000,"block_hash":"0000...dead","timestamp":1777042800,"address":42,"bucket":5,"raw":{"size":1456234,"tx_count":2847}},"genome":{"mu":3,"rho":7,"c":2,"m":5,"ne":1,"sig":6},"bind":{"lawful":true,"cost":65536,"invariant":"blockHashConserved","class":"informational_bind"},"provenance":{"node":"qfox","lake_seed":"nnyyP7DoHS11CNTRL","tailscale_ip":"100.105.111.120","attestation_hash":"sha256:abc...","prev_id":null}}
{"t":1777042801.000,"src":"linear","id":"linear:RES-7","op":"upsert","data":{"issue_id":"RES-7","title":"Implement bind bridge","state":{"old":"Todo","new":"In Progress"},"priority":1,"assignee":"allaun","blocked_by":["RES-3"]},"genome":{"mu":1,"rho":1,"c":3,"m":2,"ne":4,"sig":0},"bind":{"lawful":true,"cost":65536,"invariant":"stateTransitionValid","class":"control_bind"},"provenance":{"node":"qfox","lake_seed":"nnyyP7DoHS11CNTRL","tailscale_ip":"100.105.111.120","attestation_hash":"sha256:def...","prev_id":null}}
{"t":1777042802.000,"src":"ene","id":"ene:semantics/Adaptation:2026-04-24T12:00:00Z","op":"upsert","data":{"pkg":"semantics/Adaptation","version":"2026-04-24T12:00:00Z","tier":"CORE","domain":"formalization","archetype":"lean_module","concept_anchor":{"domain":"formalization","concept":"flow_audit_loop","resolution":"CRYSTALLIZED"},"concept_vector":[1.0,0.618,0.0,0.0,0.146,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0],"files":["0-Core-Formalism/lean/Semantics/Adaptation.lean"]},"genome":{"mu":0,"rho":5,"c":0,"m":1,"ne":7,"sig":0},"bind":{"lawful":true,"cost":98304,"invariant":"theorem finalLawfulEqEventuallyLawful","class":"informational_bind"},"provenance":{"node":"qfox","lake_seed":"nnyyP7DoHS11CNTRL","tailscale_ip":"100.105.111.120","attestation_hash":"sha256:ghi...","prev_id":null}}

9. Migration Path

Phase 1 — Shim Unification (Now)

swarm_surface.py exposes /sync/manifest and /topology
blockchain_rgflow_proxy.py writes to swarm_manifest directly
notion.js → add JSON-L exporter feeding ENE packages
linear.js → add JSON-L exporter feeding ENE packages

Phase 2 — Lean Ground Truth

Port event → Genome mappings to Lean 4 (0-Core-Formalism/lean/Semantics/JsonL/Converter.lean)
Prove bindPreservesInvariant for each src class
Extract cost functions to Q16_16 (no Float in hot path)

Phase 3 — Hardware Extraction

Verilog generator for JSON-L → genome address decoder
FPGA Warden node runs bind bridge in hardware

Document ID: UNIFIED_JSONL_SCHEMA_v1
Authority: Human architect
Date: 2026-04-24

16 KiB Raw Blame History