mirror of
https://github.com/allaunthefox/SilverSight.git
synced 2026-08-18 16:40:33 +00:00
Remove gemma4_pdf_benchmark.md
This commit is contained in:
parent
c04d8d03b9
commit
02804f2981
1 changed files with 0 additions and 57 deletions
|
|
@ -1,57 +0,0 @@
|
||||||
# Gemma4-12B PDF Summarization Benchmark
|
|
||||||
|
|
||||||
## Test Case: Burgers Equation Identification PDF
|
|
||||||
|
|
||||||
**File:** `2-Search-Space/PINNs/appendix/continuous_time_identification (Burgers)/figures/Burgers_identification.pdf`
|
|
||||||
**Size:** 256KB
|
|
||||||
**Content:** Mostly figures (plots), minimal text. PDE coefficients in figure labels.
|
|
||||||
|
|
||||||
## Extracted Text (pdftotext)
|
|
||||||
|
|
||||||
```
|
|
||||||
u(t, x) ... Data (2000 points) ... t = 0.25, t = 0.50, t = 0.75 ...
|
|
||||||
Exact / Correct PDE / Identified PDE (clean data) / Identified PDE (1% noise)
|
|
||||||
ut + uux − 0.0031831uxx = 0
|
|
||||||
ut + 0.99915uux − 0.0031794uxx = 0
|
|
||||||
ut + 1.00042uux − 0.0032098uxx = 0
|
|
||||||
```
|
|
||||||
|
|
||||||
## Gemma4 Output
|
|
||||||
|
|
||||||
> "This study demonstrates the identification of the Burgers equation using Physics-Informed Neural Networks (PINNs) based on 2,000 data points. The model successfully recovered a PDE very close to the exact equation when using clean data. Furthermore, the results show that the identified coefficients remain relatively stable even when 1% noise is introduced into the dataset."
|
|
||||||
|
|
||||||
## Judgment
|
|
||||||
|
|
||||||
| Aspect | Score | Notes |
|
|
||||||
|--------|-------|-------|
|
|
||||||
| Method (PINNs) | ✅ | Correct |
|
|
||||||
| Data points (2000) | ✅ | Correct |
|
|
||||||
| Clean data recovery | ✅ | Correct |
|
|
||||||
| Noise stability | ✅ | Correct |
|
|
||||||
| Specific coefficients | ❌ | 0.99915 vs 1.00042 not mentioned |
|
|
||||||
| Exact PDE form | ❌ | Not quoted |
|
|
||||||
| Time steps | ❌ | t = 0.25, 0.50, 0.75 not mentioned |
|
|
||||||
|
|
||||||
**Score: 7/10**
|
|
||||||
|
|
||||||
## Limit
|
|
||||||
|
|
||||||
This is the limit of what any LLM can extract from this PDF. The PDF contains:
|
|
||||||
- Mostly figures (plots of u(t,x) at different time steps)
|
|
||||||
- PDE coefficients in figure labels (not in body text)
|
|
||||||
- No abstract, no introduction, no conclusion
|
|
||||||
|
|
||||||
The text extraction (`pdftotext`) only gets the figure labels and axis labels. The LLM cannot:
|
|
||||||
- See the plots (no vision capability in this model)
|
|
||||||
- Read the PDE coefficients from the figure labels (they're in the extracted text but the LLM didn't prioritize them)
|
|
||||||
- Infer the experimental setup beyond what's in the text
|
|
||||||
|
|
||||||
**Conclusion:** Gemma4's output is the best possible summary given the available text. No LLM can extract more without vision capability or a better PDF parser.
|
|
||||||
|
|
||||||
## Recommendation
|
|
||||||
|
|
||||||
For PDF-heavy workflows:
|
|
||||||
1. Use a vision-capable model (GPT-4V, Claude Opus with vision) for figure-heavy PDFs
|
|
||||||
2. Use `marker-pdf` or `pymupdf` for better text extraction
|
|
||||||
3. Pre-process PDFs to extract figure captions and PDE coefficients as structured data
|
|
||||||
4. Feed structured data to the LLM, not raw pdftotext output
|
|
||||||
Loading…
Add table
Reference in a new issue