Research-Stack/6-Documentation/docs/blockchain_gdrive_byte_stream_ingest_2026-05-10.md

Blockchain GDrive Byte-Stream Ingest

Status: BYTE_STREAM_ROW_GROUP_SURFACE

Claim boundary: this is a transport and receipt surface for mirroring Bitcoin and Ethereum corpus bytes into Google Drive. It does not claim that the full Bitcoin chain, full Ethereum chain, logs, traces, or state have already been mirrored.

Shape

The safe corpus shape is:

source byte stream
  -> fixed-size row groups
  -> raw payload shard
  -> JSON row-group index
  -> ordered manifest receipt
  -> optional rclone upload to Google Drive

This is "parquet-style" because it uses partitioned datasets, row groups, schema sidecars, and ordered shard manifests. It is not yet true Parquet, because the repo does not add pyarrow or other new dependencies without approval.

Partition Path

Drive destination:

Gdrive:topological_storage/research-stack/blockchain-corpus/seed-2026-05-10

Partition format:

chain=<bitcoin|ethereum>/stream=<stream_kind>/run=<run_id>/
  payload/*.bin
  index/*.index.json
  receipts/*.json

Supported Streams

Initial source manifest:

shared-data/data/blockchain_corpus/blockchain_gdrive_stream_sources.json

Seed stream kinds:

bitcoin/raw_block_bytes
bitcoin/bitcoin_core_blkdat_storage_bytes
bitcoin/bitcoin_header_jsonl
bitcoin/block_jsonl
bitcoin/aws_public_blockchain_btc_blocks_parquet
ethereum/execution_block_jsonl
ethereum/logs_or_traces_jsonl
ethereum/aws_public_blockchain_eth_parquet

The current machine has a local Bitcoin Core block-storage surface at:

~/.bitcoin/blocks

Because xor.dat is present, blk*.dat should be treated as Bitcoin Core storage bytes unless a decoder/replay receipt proves decoded raw-block coverage.

The pruned-node-compatible start-to-current lane is:

bitcoin/bitcoin_header_jsonl

That stream exports active-chain header metadata from height 0 through the local Bitcoin Core height source. It is useful for timing, difficulty, chainwork, header linkage, and transaction-count pattern studies, but it is not full block-body or transaction-body coverage.

The public structured-data lane is:

AWS public blockchain data
  btc: s3://aws-public-blockchain/v1.0/btc/
  eth: s3://aws-public-blockchain/v1.0/eth/

This lane is preferable to random bootstrap snapshots for analytics because it is already transformed into compressed Parquet files partitioned by date. The current repo can inventory and byte-stream those objects without adding cloud or Parquet dependencies; decoding Parquet schemas is a later tool decision.

Command

Dry run from a local export:

python3 4-Infrastructure/shim/blockchain_gdrive_stream_ingest.py \
  --chain bitcoin \
  --stream-kind raw_block_bytes \
  --source /path/to/blk00000.dat \
  --shard-bytes 67108864

Execute upload through rclone:

python3 4-Infrastructure/shim/blockchain_gdrive_stream_ingest.py \
  --chain ethereum \
  --stream-kind execution_block_jsonl \
  --source /path/to/ethereum_blocks.jsonl \
  --execute

By default, payload bytes are streamed to Drive and local receipt/index files are kept, but local payload shard bytes are not retained. Add --keep-local-payloads only when a local shard cache is desired.

Streaming mode:

cat /path/to/export.jsonl | python3 4-Infrastructure/shim/blockchain_gdrive_stream_ingest.py \
  --chain ethereum \
  --stream-kind logs_or_traces_jsonl \
  --source - \
  --execute

Bitcoin header stream from a pruned local node:

python3 4-Infrastructure/shim/bitcoin_header_jsonl_export.py \
  --batch-size 256 \
  --progress-every 50000 \
| python3 4-Infrastructure/shim/blockchain_gdrive_stream_ingest.py \
  --chain bitcoin \
  --stream-kind bitcoin_header_jsonl \
  --source - \
  --run-id bitcoin_headers_20260510 \
  --execute

AWS public dataset inventory:

python3 4-Infrastructure/shim/blockchain_public_dataset_inventory.py \
  --chain bitcoin \
  --table blocks \
  --prefix v1.0/btc/blocks/ \
  --max-objects 100

Receipt Boundary

Each receipt proves:

ordered bytes seen by this run
shard count
total bytes
stream SHA-256
ordered shard hash
payload/index local paths
optional Drive upload status

It does not prove:

complete chain coverage
consensus validity
transaction correctness
state reconstruction
trace completeness
Drive retention forever

Those need separate coverage and replay receipts.

Logogram / Waveprobe / Hutter Feedback

The blockchain header lane can be used as a controlled test surface for the existing logogram, waveprobe, metaprobe, and compression probes because it has a strict order, deterministic provenance, and several numeric fields with obvious candidate predictors.

The allowed feedback loop is:

header field stream
  -> declared predictor
  -> residual stream
  -> residual entropy / run structure probe
  -> route-prior receipt
  -> optional Hutter/logogram fixture
  -> byte-exact replay before any compression claim

The forbidden shortcut is:

pattern seen -> compression claim

For this lane, "prediction" means only local numeric residual prediction for encoding. It does not mean price prediction, consensus prediction, or market forecasting.

Current Pull Evidence

Smoke upload:

chain: ethereum
stream: execution_block_jsonl
run: smoke_gdrive_20260510
shards: 2
bytes: 24
decision: ADMIT_PARTIAL_STREAM_TO_GDRIVE
drive: Gdrive:topological_storage/research-stack/blockchain-corpus/seed-2026-05-10/chain=ethereum/stream=execution_block_jsonl/run=smoke_gdrive_20260510

Bitcoin local storage upload:

chain: bitcoin
stream: bitcoin_core_blkdat_storage_bytes
run: bitcoin_core_blkdat_20260510
source: ~/.bitcoin/blocks/blk*.dat
source files: 5
shards: 9
bytes: 549,710,589
stream_sha256: 2651802ad7cd32b2381ddb819eadb784b15277454ec49b4d6d1d7784d407eaad
ordered_shard_hash: c5ca09fab53d24736e46a430bfd744336dc28f3e9e30fd59882a95e93a02a535
decision: ADMIT_STREAM_TO_GDRIVE
drive: Gdrive:topological_storage/research-stack/blockchain-corpus/seed-2026-05-10/chain=bitcoin/stream=bitcoin_core_blkdat_storage_bytes/run=bitcoin_core_blkdat_20260510

Bitcoin header export smoke:

script: 4-Infrastructure/shim/bitcoin_header_jsonl_export.py
heights: 0..2
records: 3
first_hash: 000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f
last_hash: 000000006a625f06636b8bb6ac7b960a8d03705d1ace08b1a19da3fdcc99ddbd
decision: ADMIT_BITCOIN_HEADER_INTERVAL_EXPORT
boundary: header metadata only, not full historical block bodies

Bitcoin full header stream upload:

chain: bitcoin
stream: bitcoin_header_jsonl
run: bitcoin_headers_20260510
height interval: 0..948178
records: 948,179
shards: 10
bytes: 622,419,124
stream_sha256: 902533a906a80ac32c62f3ef34a8a3ffa6bfcd05b3920e18858bad8381b0b6ed
ordered_shard_hash: b0661cc40337dcf29c183fee1281aaa29f72df8c877ed622b9bd0c3f0d5b93ac
decision: ADMIT_STREAM_TO_GDRIVE
drive: Gdrive:topological_storage/research-stack/blockchain-corpus/seed-2026-05-10/chain=bitcoin/stream=bitcoin_header_jsonl/run=bitcoin_headers_20260510

Bitcoin header pattern probe, first 100k:

script: 4-Infrastructure/shim/blockchain_header_pattern_probe.py
source interval: 0..99,999
records: 100,000
source_jsonl_sha256: dc2b683fbcf3ab57e7989e804da6596cfd639ed1c43e38e0c4398e5cf7a764bb
probe_payload_hash: ce5d9c6c33cb49b17cb126603fea04f34d55be458d7fc5ce1ac5ec25caf70f9d
height continuity breaks: 0
bits unique values: 35
longest bits run: 32,256
version unique values: 1
top route candidates:
  height / previous: 16.609640 entropy delta bits per symbol
  height / linear_delta: 16.609640 entropy delta bits per symbol
  chainwork / linear_delta: 16.603503 entropy delta bits per symbol
  chainwork / previous: 12.258624 entropy delta bits per symbol
  mediantime / previous: 5.969642 entropy delta bits per symbol
  time / previous: 5.864786 entropy delta bits per symbol
decision: ADMIT_HEADER_PATTERN_ROUTE_PRIORS
boundary: route-prior diagnostic only; compression gain remains HOLD

AWS public Bitcoin blocks inventory smoke:

script: 4-Infrastructure/shim/blockchain_public_dataset_inventory.py
prefix: v1.0/btc/blocks/
objects listed: 20
listed bytes: 600,921
first object: v1.0/btc/blocks/date=2009-01-03/part-00000-de61ce74-5454-4cdb-9fe6-c262412ac70d-c000.snappy.parquet
first object size: 7,541
inventory_hash: ac34af58f69f62e90efcd34e9f8f6c81d6b9f4730e572f6d3914f251b521b19b
decision: ADMIT_PUBLIC_DATASET_INVENTORY
boundary: object metadata inventory only, not full mirror

AWS public Bitcoin genesis-day Parquet smoke upload:

chain: bitcoin
stream: aws_public_blockchain_btc_blocks_parquet
run: aws_btc_blocks_genesis_day_smoke_20260510
source object: v1.0/btc/blocks/date=2009-01-03/part-00000-de61ce74-5454-4cdb-9fe6-c262412ac70d-c000.snappy.parquet
shards: 1
bytes: 7,541
stream_sha256: cfd11aa178d920e9db64c723af48a711c3ed613c64c7459f5138bdc73968407c
ordered_shard_hash: c3622e783db35a25db89b9db3b812de91a2d46f8246f9d666dcf225ed65e3c03
decision: ADMIT_STREAM_TO_GDRIVE
drive: Gdrive:topological_storage/research-stack/blockchain-corpus/seed-2026-05-10/chain=bitcoin/stream=aws_public_blockchain_btc_blocks_parquet/run=aws_btc_blocks_genesis_day_smoke_20260510

AWS public Bitcoin blocks Parquet object-preserving transfer:

script: 4-Infrastructure/shim/blockchain_public_dataset_transfer.py
inventory: shared-data/data/blockchain_corpus/aws_public_blockchain_btc_blocks_inventory_first120.json
source prefix: v1.0/btc/blocks/
listed interval: 2009-01-03 through 2009-05-07
objects listed: 120
listed bytes: 3,917,232
transfer batch 1: objects 0..19
  receipt: shared-data/data/blockchain_corpus/aws_public_blockchain_btc_blocks_transfer_first20_receipt.json
  objects transferred: 20
  observed bytes: 600,921
  size mismatches: 0
  quarantine count: 0
  receipt_hash: c0f2e4175939b735dea3904a25287075377428ab7f89c29a66f023108a3af622
transfer batch 2: objects 20..119
  receipt: shared-data/data/blockchain_corpus/aws_public_blockchain_btc_blocks_transfer_0020_0119_receipt.json
  objects transferred: 100
  observed bytes: 3,316,311
  size mismatches: 0
  quarantine count: 0
  receipt_hash: 4a61e22c44da8ab8ef8e9a8c9a8b9c2d3f5b840ac4aa65e70955b50b27398ffa
drive object count under public dataset mirror: 120 parquet objects
decision: ADMIT_PUBLIC_DATASET_TRANSFER_TO_GDRIVE
boundary: object-preserving public dataset mirror, not full dataset coverage

AWS public Bitcoin blocks Parquet widened transfer:

script: 4-Infrastructure/shim/blockchain_public_dataset_transfer.py
parallelism: 8 workers
inventory: shared-data/data/blockchain_corpus/aws_public_blockchain_btc_blocks_inventory_first500.json
source prefix: v1.0/btc/blocks/
listed interval: 2009-01-03 through 2010-05-22
objects listed: 500
listed bytes: 16,433,985
transfer batch 3: objects 120..499
  receipt: shared-data/data/blockchain_corpus/aws_public_blockchain_btc_blocks_transfer_0120_0499_receipt.json
  objects transferred: 380
  observed bytes: 12,516,753
  size mismatches: 0
  quarantine count: 0
  receipt_hash: 1b6d62d1ea308a5c784e53e4e0e2fb60b8f125e0355bb15a268be023bc1499f2
drive object count under public dataset mirror: 500 parquet objects
decision: ADMIT_PUBLIC_DATASET_TRANSFER_TO_GDRIVE
boundary: object-preserving public dataset mirror through the first 500 listed BTC block Parquet objects, not full dataset coverage

Major commodity cryptocurrency block-spine pull:

summary receipt: shared-data/data/blockchain_corpus/major_commodity_crypto_transfer_summary_receipt.json
receipt_hash: 5c05c64a11e56d90044c2c0ee9fa351b9ade1aad948dbc8b44a5b34f68ee1750
drive summary: Gdrive:topological_storage/research-stack/blockchain-corpus/seed-2026-05-10/major_commodity_crypto_transfer_summary_receipt.json
chain/table surfaces: 14
objects selected: 1,800
successful object transfers: 1,615
quarantined object attempts: 185
total observed bytes: 8,012,411,009
decision: ADMIT_MAJOR_COMMODITY_CRYPTO_PARTIAL_MIRROR

AWS public blockchain blocks table transfers admitted:

bitcoin:     500 objects, 16,433,985 bytes, 0 quarantines
ethereum:    100 objects, 152,920,601 bytes, 0 quarantines
ton:         100 objects, 516,582,277 bytes, 0 quarantines
cronos:      100 objects, 724,723,576 bytes, 0 quarantines
xrp:         100 objects, 114,843,093 bytes, 0 quarantines
arbitrum:    100 objects, 80,384,418 bytes, 0 quarantines
aptos:       100 objects, 2,707,424,717 bytes, 0 quarantines
base:        100 objects, 2,013,202,129 bytes, 0 quarantines
provenance:  100 objects, 1,642,038,523 bytes, 0 quarantines

Blockchair blocks dump transfers were best-effort because the server returned HTTP 402 for some objects during this pass:

bitcoin-cash: 100 / 100 files, 1,216,928 bytes, 0 quarantines
dogecoin:      79 / 100 files, 22,905,328 bytes, 21 quarantines
dash:          92 / 100 files, 8,083,471 bytes, 8 quarantines
litecoin:      35 / 100 files, 2,544,698 bytes, 65 quarantines
zcash:          9 / 100 files, 9,107,265 bytes, 91 quarantines

The Blockchair partials are not local failures; they are receipted source-side admission limits. Re-running too aggressively is likely to increase 402s rather than improve coverage.

Drive listing verification found 22 entries for the Bitcoin run:

9 payload shards
9 index shards
1 receipt
3 directory entries

Ethereum full corpus remains HOLD_EXPORT_MISSING on this machine: no ~/ethereum-exports or ~/eth-exports source directory was present during this pass.