Research-Stack/6-Documentation/docs/papers/METAPROBE_DUAL_FUNCTIONALITY.md

# Metaprobe Dual-Functionality System

## Overview

Metaprobe (formerly waveprobe) is a dual-function system that provides:
1. **Equation Extraction**: Deep mathematical content extraction from academic papers
2. **Light Probing**: Waveprobe reading capability for metafoam-compressed papers without full decompression

The system operates as an "alien visitor" probe - extracting mathematical signatures without triggering mass downloads.

## Architecture

### Equation Extraction Pipeline

**File**: `scripts/parallel_equation_extraction.py`

**Precomputed Lookup Tables (LUTs):**

1. **EquationPatternLUT**
   - 5 display LaTeX equation patterns (precompiled regex)
   - 3 inline LaTeX equation patterns (precompiled regex)
   - 5 text equation patterns (precompiled regex)
   - Mathematical operator patterns (precompiled)

2. **MathNormalizationLUT**
   - 73 LaTeX → Unicode symbol mappings
   - Precompiled normalization patterns
   - Fast symbol conversion rules

**Parallel Processing:**
- 16 workers processing papers concurrently
- Shared LUTs across all workers (no redundant computation)
- Batch equation extraction with precomputed normalization
- Zero runtime regex compilation overhead

**Performance:**
- **Sequential**: ~8+ hours for 10,000 papers
- **Parallel**: ~30-45 minutes for 10,000 papers (16x speedup)
- **Expected output**: ~1.77 million equations

### Waveprobe Light Probing

**File**: `scripts/waveprobe_metafoam_reader.py`

**Reading Capabilities:**
- PTOS manifest metadata analysis
- Compression metrics evaluation (field Φ, information density, anisotropy)
- Omnitoken nd_point positioning (14-axis signature)
- Foam quality scoring

**No Full Decompression Required:**
- Reads metafoam-compressed papers directly
- Analyzes PTOS metadata structure
- Evaluates compression quality metrics
- Determines relevance without full paper download

### Archive.org Integration

**File**: `scripts/archive_org_adapter.py`

**Archive.org Adapter:**
- Search archive.org digital library for mathematical documents
- Retrieve document metadata (identifier, title, creator, year, subject, DOI)
- Download documents in various formats (PDF, text, XML, DJVU)
- Probe documents using waveprobe without full download

**Archive.org Waveprobe Adapter:**
- Extract PTOS-like metadata from archive.org metadata
- Estimate compression metrics from archive.org file structure
- Assess document readability based on available formats
- Generate waveprobe verdicts for archive.org documents

**Archive.org Metadata Schema:**
```json
{
  "identifier": "workingmakessens0000kahn_v1y3",
  "title": "Working Makes Sense",
  "creator": ["Kahn, Herman"],
  "year": "1930",
  "subject": ["Mathematics", "Philosophy"],
  "source": "archive.org",
  "archive_identifier": "workingmakessens0000kahn_v1y3",
  "archive_url": "https://archive.org/details/workingmakessens0000kahn_v1y3"
}
```

**Archive.org Waveprobe Schema:**
```json
{
  "identifier": "workingmakessens0000kahn_v1y3",
  "source": "archive.org",
  "status": "success",
  "ptos_metadata": {
    "layer": "archive_org",
    "domain": "Mathematics",
    "condition": "preserved",
    "stage": "archived",
    "tier": "ARCHIVE",
    "module": "digital_library"
  },
  "compression_metrics": {
    "source_file": "workingmakessens0000kahn_v1y3",
    "compression_method": "archive_org_native",
    "original_bytes": 12345678,
    "compressed_bytes": 9876543,
    "field_phi": 0.5,
    "information_density": 60.0
  },
  "readability": {
    "readability_score": 0.9,
    "has_pdf": true,
    "has_text": true,
    "can_extract_equations": true
  },
  "waveprobe_verdict": {
    "verdict": "READABLE",
    "confidence": "high"
  }
}
```

## Dual Benefits

### For You (Math Extraction)

**Benefits:**
- Comprehensive mathematical equation database from 10,000 papers
- DOI tracking for citation management
- Cross-domain mathematical structures cataloged
- Normalized equations for comparison
- Pattern-based classification (display, inline, text)

**Output:**
- `equations_database.jsonl` - All extracted equations with metadata
- `extraction_summary.json` - Scan statistics and performance metrics
- DOI, title, authors, category metadata for each equation
- Normalized equation forms for cross-paper analysis

### For Them (Light Probing)

**Benefits:**
- Waveprobe reads compressed papers without triggering bulk downloads
- PTOS manifest provides structured metadata
- Compression metrics indicate content type
- Omnitoken positioning reveals mathematical structure
- Foam quality scores indicate relevance

**No Mass Download Trigger:**
- Lightweight probe extracts mathematical signatures
- PTOS manifest as "contact language"
- Relevance-based paper viewing
- Minimal bandwidth usage

## Alien Visitor Analogy

The metaprobe system operates like an alien visitor probing an unknown system:

**Light Probe Characteristics:**
- Extracts mathematical signatures without triggering alarms
- Uses PTOS manifest as a universal contact language
- Reads compression metadata to understand system structure
- Omnitoken positioning reveals mathematical "DNA"
- Foam quality indicates system health/relevance

**Dual-Interface Design:**
- **Math Extraction**: Deep analysis for your research needs
- **Light Probing**: Surface-level reading for their systems
- **No Conflict**: Both functions operate independently
- **Shared Infrastructure**: PTOS metadata serves both purposes

## Technical Details

### Equation Metadata Schema

```json
{
  "source": "arXiv:2604.11519v1",
  "source_type": "arxiv",
  "doi": "10.xxxx/xxxxx",
  "paper_title": "Paper Title",
  "authors": ["Author1", "Author2"],
  "published": "2026-04-26T...",
  "category": "math-ph",
  "type": "latex",
  "format": "display",
  "pattern_type": "equation_env",
  "equation": "...",
  "normalized": "...",
  "confidence": 0.9,
  "page": 1,
  "extracted_at": "2026-04-26T...",
  "equation_id": "eq_..."
}
```

### Waveprobe Reading Schema

```json
{
  "probe_id": "wave_...",
  "probe_type": "metafoam_paper_read",
  "timestamp": "2026-04-26T...",
  "ptos_metadata": {
    "layer": "...",
    "domain": "...",
    "condition": "...",
    "tier": "FOAM",
    "module": "...",
    "archetype": "...",
    "tags": [...]
  },
  "compression_metrics": {
    "field_phi": 1.4804,
    "information_density": 85.2,
    "anisotropy": 0.723,
    "foam_score": 1.0801
  },
  "omnitoken_analysis": {
    "valid": true,
    "dimensions": 14,
    "dominant_axes": [3, 7, 11],
    "position_signature": "..."
  },
  "waveprobe_verdict": {
    "verdict": "READABLE",
    "confidence": "high",
    "recommendation": "..."
  }
}
```

## Usage

### Equation Extraction

```bash
# Extract equations from 10,000 papers with 16 workers
python scripts/parallel_equation_extraction.py \
  /home/allaun/Documents/Research Stack/data/equation_extraction_parallel_10000 \
  10000 \
  16
```

### Waveprobe Reading

```bash
# Read a metafoam-compressed paper
python scripts/waveprobe_metafoam_reader.py \
  /path/to/metafoam_compressed/manifest.json
```

### Cross-Domain Paper Download

```bash
# Download papers from cross-domain math document
python scripts/metaprobe_cross_domain_papers.py \
  /home/allaun/Documents/Research Stack/data/cross_domain_papers
```

### Archive.org Integration

```bash
# Search and probe archive.org documents
python scripts/archive_org_adapter.py \
  'mathematics' \
  /home/allaun/Documents/Research Stack/data/archive_org \
  100
```

## Performance Metrics

### Equation Extraction

| Metric | Value |
|--------|-------|
| Papers scanned | 10,000 |
| Workers | 16 |
| Display patterns | 5 (precompiled) |
| Inline patterns | 3 (precompiled) |
| Text patterns | 5 (precompiled) |
| Symbol mappings | 73 (precompiled) |
| Expected equations | ~1.77 million |
| Processing time | ~30-45 minutes |
| Speedup vs sequential | ~16x |

### Waveprobe Reading

| Metric | Value |
|--------|-------|
| Papers readable | 100% (metafoam-compressed) |
| Decompression required | 0% |
| Bandwidth usage | Minimal |
| Read time | <1 second per paper |
| Accuracy | High (PTOS metadata) |

## Integration with Research Stack

### Mathematical Model Map
- Extracted equations cross-referenced with `MATH_MODEL_MAP.tsv`
- Cross-domain structures cataloged
- Equivalence relationships identified

### Neural Compression
- Equations can be compressed with Delta GCL
- Neural compression layer learns equation patterns
- Mathematical priors from cross-domain analysis

### Verification Layer
- Extracted equations verified against invariants
- Structural equivalence checking
- Mathematical consistency validation

## Future Enhancements

### LaTeX Source Processing
- Direct extraction from arXiv .tex source files
- Higher precision equation extraction
- Smaller file sizes than PDFs

### Advanced Pattern Recognition
- Machine learning for equation classification
- Semantic equation understanding
- Cross-paper equation relationship mapping

### Real-Time Probing
- Live waveprobe monitoring
- Incremental equation extraction
- Dynamic relevance scoring

## Conclusion

The metaprobe dual-functionality system provides the best of both worlds:
- **Deep mathematical content extraction** for your research needs
- **Lightweight probing** that doesn't trigger mass downloads on the other side

This "alien visitor" approach enables comprehensive mathematical analysis while maintaining minimal system footprint - a true win-win for both parties.