Ultimate Web ScraperDiscover Every URL a Website Publishes
Use the free Sitemap Explorer to map a site's sitemap into a searchable tree, select the branch you need, and feed it to the Page Extractor or a CSV.
Before you can scrape "all the product pages" or "every blog post", you need their URLs. The Sitemap Explorer reads a site's sitemaps — the URL lists sites publish for search engines — and turns them into a selectable tree. It's free, and it runs entirely locally.
Open it on the target site
Navigate to the site, then open Sitemap Explorer from the side panel (it's also available as a URL source inside the Page Extractor). Switching browser tabs to a different site automatically rescans.
Let discovery run
The tool finds sitemaps automatically, checking the common places sites publish them. Sitemap indexes are followed and compressed sitemaps are handled for you. If you already know a sitemap URL on this site, you can paste it directly. You can stop a long scan early and keep what's been found.
Browse the URL tree
Discovered URLs are deduplicated and grouped by path into a hierarchical tree — /products/..., /blogs/..., /pages/... each become their own branch with counts. Use the tri-state checkboxes to select a whole branch, or select a parent and carve out sub-branches you don't want. The multi-word search filters and auto-expands matches, with "Select matches" to grab everything a query hits.
Select the branch you need
For a store, that's typically the products branch; for a content site, the blog branch. The live counter shows exactly how many URLs you've selected.
Hand the URLs off
Two exits:
- Continue sends the selected URLs straight into the Page Extractor — ideal with Automatic Extract for product data.
- Export CSV downloads the selection as a one-column URL CSV, usable in any tool that accepts CSV upload (Page Text Extractor, Email Extractor, or anything outside the extension).
Scale and privacy
The Explorer collects up to 50,000 URLs per scan (results beyond that are marked as capped). Everything is free: discovery, selection, and the URL CSV export.
It also runs locally, inside the active tab, with your own session — nothing leaves your machine, and because requests carry your cookies, sitemaps behind mild bot protection often load where an external crawler would be blocked.
💡 No sitemap?
Not every site publishes one. If discovery comes up empty, scrape a category or listing page with the List Extractor to collect URLs instead — see Extract data from multiple pages.