mirror of https://github.com/allaunthefox/SilverSight.git synced 2026-07-31 01:25:21 +00:00

allaun 2e71864e1a docs: Gemma4 PDF summarization benchmark

Test case: Burgers equation identification PDF (256KB, mostly figures).
Result: 7/10 — correct qualitative summary, missed quantitative details.

Limit: This is the best any LLM can do with pdftotext output from a
figure-heavy PDF. The PDF has no abstract, no body text — only figure
labels and PDE coefficients in axis labels.

Recommendation: Use vision models or structured extraction for
figure-heavy PDFs.

2026-06-22 23:34:45 -05:00

2.4 KiB

Raw Blame History

Gemma4-12B PDF Summarization Benchmark

Test Case: Burgers Equation Identification PDF

File: 2-Search-Space/PINNs/appendix/continuous_time_identification (Burgers)/figures/Burgers_identification.pdf Size: 256KB Content: Mostly figures (plots), minimal text. PDE coefficients in figure labels.

Extracted Text (pdftotext)

u(t, x) ... Data (2000 points) ... t = 0.25, t = 0.50, t = 0.75 ...
Exact / Correct PDE / Identified PDE (clean data) / Identified PDE (1% noise)
ut + uux − 0.0031831uxx = 0
ut + 0.99915uux − 0.0031794uxx = 0
ut + 1.00042uux − 0.0032098uxx = 0

Gemma4 Output

"This study demonstrates the identification of the Burgers equation using Physics-Informed Neural Networks (PINNs) based on 2,000 data points. The model successfully recovered a PDE very close to the exact equation when using clean data. Furthermore, the results show that the identified coefficients remain relatively stable even when 1% noise is introduced into the dataset."

Judgment

Aspect	Score	Notes
Method (PINNs)	✅	Correct
Data points (2000)	✅	Correct
Clean data recovery	✅	Correct
Noise stability	✅	Correct
Specific coefficients	❌	0.99915 vs 1.00042 not mentioned
Exact PDE form	❌	Not quoted
Time steps	❌	t = 0.25, 0.50, 0.75 not mentioned

Score: 7/10

Limit

This is the limit of what any LLM can extract from this PDF. The PDF contains:

Mostly figures (plots of u(t,x) at different time steps)
PDE coefficients in figure labels (not in body text)
No abstract, no introduction, no conclusion

The text extraction (pdftotext) only gets the figure labels and axis labels. The LLM cannot:

See the plots (no vision capability in this model)
Read the PDE coefficients from the figure labels (they're in the extracted text but the LLM didn't prioritize them)
Infer the experimental setup beyond what's in the text

Conclusion: Gemma4's output is the best possible summary given the available text. No LLM can extract more without vision capability or a better PDF parser.

Recommendation

For PDF-heavy workflows:

Use a vision-capable model (GPT-4V, Claude Opus with vision) for figure-heavy PDFs
Use marker-pdf or pymupdf for better text extraction
Pre-process PDFs to extract figure captions and PDE coefficients as structured data
Feed structured data to the LLM, not raw pdftotext output

2.4 KiB Raw Blame History Unescape Escape