arXiv Scraper

Why arXiv for LLM Knowledge Ingest?

Wikipedia gives you breadth. SEP gives you peer-reviewed definitions. But when someone asks "what's the latest evidence for predictive processing in visual cortex?" or "which RAG architectures actually improve factual grounding?" — you need primary literature. arXiv is the open-access preprint server where CS/AI, math, physics, and quantitative biology researchers publish before (and sometimes instead of) journal submission.

arXiv has a unique constraint: its Atom API exposes metadata only — no references, no citation graph. The paper's abstract, authors, categories, and PDF link are available, but its bibliography is locked inside the PDF. This means the arXiv scraper is structurally different from the Wikipedia and SEP scrapers: it's a metadata-and-document fetcher, not a graph crawler. Citation-graph reconstruction requires an external bridge (Semantic Scholar) or full PDF reference extraction.

Architecture

Two implementations targeting different entry patterns:

arxiv.py (category-driven)

  ArxivFetcher
    .arxiv_categories   — 35 CS subcategories mapped to arXiv codes
    .fetch_arxiv_articles(category_key, n)
      1. Build Atom API URL:  cat:{code}&sortBy=submittedDate
      2. Parse feed via feedparser
      3. For each entry: extract arxiv_id, title, date, summary
      4. Download PDF via application/pdf link
      5. Dedup against metadata.json
      6. Persist metadata + PDF to arxiv_articles/

merged_arxiv_union.ipynb cell 24 (search-query-driven)

  1. Build URL:  all:{free-text query}&max_results=N
  2. Parse + normalize to pandas DataFrame
  3. Filter by category list (e.g. cs.CL, cs.LG)
  4. Download first PDF + extract first 2 pages via PyPDF2
  5. Save CSV + JSON + text preview to arxiv_out/
            

Key Design Decisions

Two entry patterns: category browse vs. keyword search

arxiv.py answers "what's new in cs.AI?" — sorted by submission date, category-filtered. The notebook answers "find papers about retrieval-augmented generation" — free-text search, post-filtered by category list. Both patterns are needed: browse for monitoring a field, search for answering a specific question.

35 CS subcategory index

The arxiv_categories dictionary maps human-readable names to arXiv codes, covering the full CS taxonomy. This makes category-driven queries accessible without memorizing codes like cs.CL or cs.NE.

PDF download + text preview

The notebook goes one step further than arxiv.py: after downloading the PDF, it extracts the first 1–2 pages as plain text via PyPDF2. This gives an immediate text preview without requiring a full PDF-to-text pipeline. Enough for an LLM to read the abstract + introduction.

Polite identification

All requests use a shared CogSciWikiBot/0.1 user-agent string with explicit timeouts, matching the same politeness standards as the Wikipedia and SEP scrapers.

arXiv CS Category Index (35 subcategories)

Showing 20 of 35 — full index in source code.

Code: Category-Driven Fetch

def fetch_arxiv_articles(self, category_key, num_articles=3):
    category = self.arxiv_categories.get(category_key, None)
    if not category:
        print(f"Invalid category: {category_key}")
        return

    downloaded_ids = self._load_downloaded_articles()
    new_articles = []

    base_url = "http://export.arxiv.org/api"
    query = (
        f"/query?search_query=cat:{category}"
        f"&sortBy=submittedDate&sortOrder=descending"
        f"&start=0&max_results={num_articles}"
    )
    feed = feedparser.parse(base_url + query)

    for entry in feed.entries:
        arxiv_id = entry.id.split('/')[-1]
        if arxiv_id in downloaded_ids:
            continue
        # ... download PDF, extract metadata, persist ...

Live Output: Recent RAG Papers

Search query: all:retrieval-augmented generation, 3 results, sorted by date:

Papers published yesterday (April 8, 2026). The pipeline fetches real-time preprints.

At a Glance

Three Sources, Three Roles