• In Session 1 we used the Vandalizer to extract structured data from documents — now we ask: how good was it?
• We'll define accuracy two ways:
  • Ground truth comparison — measure extracted values against known correct answers
  • Replication — run the same extraction multiple times and measure consistency
• These evaluation strategies can be applied to many types of analytics pipelines

• Optical character recognition (OCR) is your ceiling. If the PDF is scanned, extraction quality can never exceed OCR quality. Dedicated OCR engines and multimodal vision models handle layouts that basic text extraction misses.
• Garbage in, garbage out — OCR errors propagate silently through every downstream step
• This is a problem you solve before you choose an LLM

• Building on Session 2 — defining the expected schema before extraction means structural errors are caught automatically
• If the model expects a date and gets a dollar amount, that's a flag, not a guess
• Without a schema, the model invents its own structure — and it will be different every time
• Structured outputs. Some models support a mode that forces output to conform to a JSON schema. This eliminates parsing failures and guarantees the shape of extracted data matches your data model.

• Recall the token-by-token demo from Session 1 — LLMs sample from a probability distribution, so the same prompt can produce different output each time. Replication is how we make that variability visible.
• Multiple-pass consensus. Run the same extraction multiple times and compare results. Agreement across replicates increases confidence; disagreement flags items for human review.
• Borrow from bench science. Replication is how researchers separate signal from noise — and it works the same way here. If three runs converge on the same answer, that's evidence. If they don't, you've identified an item that needs scrutiny.

Question. "Works for me" in one tool is not an argument that OCR is solved. How far does the same scanned document travel depending on which pipeline touches it first?

1. Pick one of the tier 3 PDFs. Run it through the Vandalizer with the extraction prompt and schema. Notice how much dotsocr recovers.
2. Now take the same PDF to a tool you'd actually use day to day: paste or upload it to ChatGPT, Claude, Copilot, a generic "PDF to text" converter, or the text-only extract from your browser.
3. Score each tool's output against the ground truth JSON. Mark missing, garbled, and silently collapsed fields.
4. The budget table and the stamped/highlighted header are the usual breaking points. Predict which tool will cope before you run it.

Question. Same prompt, same document, different AI tools in different browser tabs. Where do they agree, and where does disagreement flag fields that need review?

1. Pick two or three AI tools you actually use: ChatGPT, Claude, Copilot, Gemini, Vandalizer, a local model. Open each in its own tab.
2. Paste the canonical extraction prompt (prompt.md) into each tool. Upload the clean PDF. Clean input isolates the tool variable so OCR is not in the mix.
3. Save every tool's JSON output. Score each against the ground truth field by field.
4. Build a per-tool agreement matrix. Fields where every tool agrees are cheap to trust. Fields where tools diverge are where human review earns its keep.

Question. Does constraining the output to a JSON schema reduce errors, or does it introduce new failure modes like silent null-filling?

1. Run the extraction prompt on the clean PDF in the Vandalizer with schema enforcement on (structured output mode).
2. Run the same prompt and PDF with schema enforcement off. Ask for JSON in the prompt but don't constrain it.
3. Parse both outputs. Record parse successes, field completeness, and any fields the model silently filled or skipped.
4. Score against the ground truth. Separate "parse failure" from "wrong value" in your tally, they are different failure modes.

Question. Same prompt, same document, same model, three runs. Which fields land on the same answer every time, and which shift between runs?

1. In the Vandalizer, lock the model, prompt, and schema. Run the extraction three times on the tier 3 PDF. Clean PDF works too if the room is tight on time; tier 3 just exposes more variance.
2. Collect all three JSON outputs.
3. Compare field by field across the three runs. Mark each field as concordant (all three match) or discordant.
4. Score the concordant set against ground truth. Route the discordant set to human review.

• Accuracy is one half of trust. Provenance is the other — for every extracted value, can we point back to where it came from in the source document?
• Our next Vandalizer workflow extracts structured requirements from a real NSF solicitation and populates a data model — deadlines, eligibility, page limits, special conditions, budget rules
• This solicitation was chosen on purpose: it has a lot of special requirements that are easy to miss and easy to misread
• Before we run the extraction, take a few minutes to skim the solicitation yourself. Notice what would be tedious to catalog by hand.

And yes — to get the PDF, NSF tells you to print the page to PDF in your browser. Yes, really.

• Define the reporting period and metric rules so the model does not invent them.
• Give the model the actual source table or query output, not a vague request for a narrative.
• Use a template that separates directly supported facts from interpretation.
• Keep a human review step for attribution, causality, anomalies, and edge cases.