Bulk-Convert Digital PDFs to Text and Markdown for a RAG Index (and Catch the Scanned Ones)

Most "chat with your documents" projects die on step one: getting text out of the PDFs. A handful of files is easy. A few hundred reports, manuals and specs from different sources is where you find the edge cases: encrypted files, 400-page monsters, and scans that look like documents but contain only images. This post walks through: a DIY extraction script and its traps, the same batch job with a hosted Actor and real output, chunking the result for a search or RAG index, plus cost and limits. The DIY version With pypdf the basic extraction is a few lines: import io, requests from pypdf import PdfReader def pdf_text(url): data = requests.get(url, timeout=60).content reader = PdfReader(io.BytesIO(data)) return "\n\n".join(page.extract_text() or "" for page in reader.pages) What bites you in production: Scanned PDFs return an empty string, silently. Your index now holds a "document" with no text, and nobody notices. Encrypted files raise errors or return nothing. Size and page limits. One huge file can stall a whole batch, so you need byte and page caps. Bookkeeping. You want page counts, title/author metadata and a status per file, not a pile of strings. The hosted version PDF to Text Markdown Extractor (Digital, No OCR) does this batch job with explicit statuses. Disclosure: I build this Actor at SteadyScrape. It downloads each direct PDF URL and extracts text locally. There's no OCR in this version, so scanned files are labelled image_only and incur no extraction charge. Input matching the current schema: { "pdfUrls": [ "https://www.orimi.com/pdf-test.pdf", "https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf", "https://www.rfc-editor.org/rfc/pdfrfc/rfc8259.txt.pdf" ], "format": "both", "maxFiles": 20, "includeMetadata": true } format is text, markdown or both. You can also upload files in the Apify Console instead of passing links. From Python (token from an environment variable): import os from apify_client import ApifyClient client = ApifyClient(os.environ["APIFY_TOKEN"]) run = client.actor("steady-scrape/pdf-text-markdown").call(run_input={ "pdfUrls": [ "https://www.orimi.com/pdf-test.pdf", "https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf", "https://www.rfc-editor.org/rfc/pdfrfc/rfc8259.txt.pdf", ], "format": "both", }) rows = list(client.dataset(run["defaultDatasetId"]).iterate_items()) Real output from a verified run of exactly those three files (text trimmed): [ {"source": "https://www.orimi.com/pdf-test.pdf", "status": "ok", "pageCount": 1, "chars": 367, "title": "PDF Test Page", "text": "PDF Test File\n\nCongratulations, your computer is equipped with a PDF ...", "chargeEvent": "pdf-extracted"}, {"source": "https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf", "status": "image_only", "note": "No extractable text — PDF looks image-only/scanned. OCR is not included in v1. Not charged.", "text": null, "chargeEvent": null}, {"source": "https://www.rfc-editor.org/rfc/pdfrfc/rfc8259.txt.pdf", "status": "ok", "pageCount": 16, "chars": 23656, "text": "Internet Engineering Task Force (IETF) T. Bray, Ed.\nRequest for Comments: 8259 ...", "chargeEvent": "pdf-extracted"} ] That run charged two PDF extractions plus the run fee and the standard start event; the image-only file added no charge, and the row tells you why. Chunking for a search or RAG index Keep the ok rows, route image_only rows to an OCR queue (or a human), and split the Markdown on headings, with a size fallback: import re def chunks(md, max_chars=2000): parts = re.split(r"\n(?=#{1,3} )", md) for part in parts: for i in range(0, len(part), max_chars): yield part[i:i + max_chars] index, needs_ocr = [], [] for r in rows: if r["status"] == "ok": for n, c in enumerate(chunks(r["markdown"] or r["text"])): index.append({"source": r["source"], "chunk": n, "text": c}) elif r["status"] == "image_only": needs_ocr.append(r["source"]) print(len(index), "chunks;", len(needs_ocr), "files need OCR") Embed index with whatever model you use and store source with each chunk so answers can cite the file. The Markdown headings are heuristic: the PDF format doesn't reliably mark headings, so treat them as hints for splitting, not as a guaranteed document outline. Tables and multi-column layouts can come out in reading-order text rather than structured tables. What it costs From the Store pricing: $0.004 per successfully extracted digital PDF $0.003 once per run, only if at least one PDF was extracted Apify's standard $0.00005 Actor start event 200 successfully extracted digital PDFs in one run cost $0.80305 (200 × $0.004 + $0.003 + $0.00005). Scanned/image-only, empty, encrypted, oversized and failed files incur no extraction charge, and a run where no PDF is extracted doesn't pay the $0.003 run fee; Apify's $0.00005 start event still applies to every run. Limits No OCR. Scanned documents are flagged, not read. Direct PDF links only. It doesn't crawl a website looking for PDFs. Per-run file caps and per-file page and size caps apply. Oversized files return too_large / too_many_pages (no extraction charge). Only process documents you're allowed to use. Integrations Run it via the Apify API, on a schedule, or from Make, Zapier or n8n through Apify's integrations. AI agents can call it through the Apify MCP server, which is handy as a "read this PDF" tool. SteadyScrape (steady-scrape) builds small, pay-per-result data tools on Apify.