diff --git a/docs/gemma4_pdf_benchmark.md b/docs/gemma4_pdf_benchmark.md new file mode 100644 index 00000000..cb0c8185 --- /dev/null +++ b/docs/gemma4_pdf_benchmark.md @@ -0,0 +1,57 @@ +# Gemma4-12B PDF Summarization Benchmark + +## Test Case: Burgers Equation Identification PDF + +**File:** `2-Search-Space/PINNs/appendix/continuous_time_identification (Burgers)/figures/Burgers_identification.pdf` +**Size:** 256KB +**Content:** Mostly figures (plots), minimal text. PDE coefficients in figure labels. + +## Extracted Text (pdftotext) + +``` +u(t, x) ... Data (2000 points) ... t = 0.25, t = 0.50, t = 0.75 ... +Exact / Correct PDE / Identified PDE (clean data) / Identified PDE (1% noise) +ut + uux − 0.0031831uxx = 0 +ut + 0.99915uux − 0.0031794uxx = 0 +ut + 1.00042uux − 0.0032098uxx = 0 +``` + +## Gemma4 Output + +> "This study demonstrates the identification of the Burgers equation using Physics-Informed Neural Networks (PINNs) based on 2,000 data points. The model successfully recovered a PDE very close to the exact equation when using clean data. Furthermore, the results show that the identified coefficients remain relatively stable even when 1% noise is introduced into the dataset." + +## Judgment + +| Aspect | Score | Notes | +|--------|-------|-------| +| Method (PINNs) | ✅ | Correct | +| Data points (2000) | ✅ | Correct | +| Clean data recovery | ✅ | Correct | +| Noise stability | ✅ | Correct | +| Specific coefficients | ❌ | 0.99915 vs 1.00042 not mentioned | +| Exact PDE form | ❌ | Not quoted | +| Time steps | ❌ | t = 0.25, 0.50, 0.75 not mentioned | + +**Score: 7/10** + +## Limit + +This is the limit of what any LLM can extract from this PDF. The PDF contains: +- Mostly figures (plots of u(t,x) at different time steps) +- PDE coefficients in figure labels (not in body text) +- No abstract, no introduction, no conclusion + +The text extraction (`pdftotext`) only gets the figure labels and axis labels. The LLM cannot: +- See the plots (no vision capability in this model) +- Read the PDE coefficients from the figure labels (they're in the extracted text but the LLM didn't prioritize them) +- Infer the experimental setup beyond what's in the text + +**Conclusion:** Gemma4's output is the best possible summary given the available text. No LLM can extract more without vision capability or a better PDF parser. + +## Recommendation + +For PDF-heavy workflows: +1. Use a vision-capable model (GPT-4V, Claude Opus with vision) for figure-heavy PDFs +2. Use `marker-pdf` or `pymupdf` for better text extraction +3. Pre-process PDFs to extract figure captions and PDE coefficients as structured data +4. Feed structured data to the LLM, not raw pdftotext output