Ultimate Web ScraperTurn Webpages into Clean Text for AI and LLM Projects
Extract boilerplate-free plain text plus title, author, date, and word count from any list of URLs with the Page Text Extractor — one row per page.
Feeding webpages to an LLM means stripping the junk first: navigation, ads, footers, share widgets. The Page Text Extractor does exactly that — give it a list of URLs, get back one clean row per page with the main content as plain text plus useful metadata.
Add your URLs
Open Page Text Extractor from the side panel. Paste URLs manually, upload a CSV, or use Data Source to pull a URL column from a previous extraction. The first URL loads as a preview so you can sanity-check the target.
Adjust configuration if needed
The defaults work for most sites: a short delay between requests and a page-load wait so JavaScript-rendered articles finish rendering before extraction. Slow sites may want a longer load timeout.
Run the extraction
Start it and let it work through the list in the background. Each page becomes one row with these columns:
| Column | Content |
|---|---|
| url | Source URL |
| title | Page title, upgraded from og:title or the h1 when longer |
| description | Meta or social description |
| content | Main content as plain text |
| word_count | Word count of the content |
| author | Author from metadata or common byline selectors |
| publish_date | Publish date from article metadata or date elements |
Export your corpus
Review in the Data Table and export as CSV or JSON — see Exporting data. JSON is usually the friendliest format for ingestion pipelines.
What "clean" means here
Extraction targets the page's main content container (main, article, common content classes) and always strips scripts, styles, navigation, headers, footers, sidebars, ads, social/share widgets, and comment sections. The output is plain, whitespace-normalized text — not HTML, not Markdown — ready to chunk and embed.
Use cases
- RAG corpora — turn a documentation site or blog into embedding-ready text with source URLs attached.
- Content audits — word counts, authors, and publish dates across an entire site in one table.
- Research datasets — clean article text at scale, with metadata for filtering.
Cover a whole site
Pair it with the Sitemap Explorer: discover every URL the site publishes, export the selection as a URL CSV, and upload that CSV here. That's whole-site coverage in two steps, and both discovery and extraction run locally; the Faster Extraction toggle runs several tabs in parallel.
💾 Repeatable corpora
Save the setup as a recipe to re-run the same URL list later — useful for keeping a corpus fresh.