Русский
PSB Parser · Website Crawl
Website Crawl: the whole website, not a single page
Crawl finds and collects the pages of a website one after another: by links and sitemap, to a set depth and with a page limit. Every page found can be processed with the Web Scraper features — get HTML, text, Markdown, structured data or AI analysis.
One API for websites, search engines and AI. Pay with credit packages.
What you get for each page
- HTML
- Text
- Markdown
- Structured data
- AI analysis
- All Web Scraper features for the pages found
Website Crawl features
Only what the tool can do today.
Page discovery
- Finding URLs through links on pages
- Sitemap processing
- Crawling to a set depth
Crawl limits
- Maximum page limit
- URL include and exclude rules
- Allowed domain restriction
- Subdomain handling
Clean URLs
- URL normalization
- Query parameter handling
- Duplicate URL removal
How crawling works
Instead of a single page, Crawl finds and collects the pages of a website one after another.
Start URL
Specify the website or section where the crawl starts.
Crawl rules
Depth, maximum pages, allowed domains and subdomains, include / exclude rules for URLs.
Page discovery
Crawl follows the links on pages and the sitemap, normalizes URLs, takes query parameters into account and removes duplicates.
Data for every page
Every page found is processed with the Web Scraper features: HTML, text, Markdown, structured data or AI analysis.
What it is used for
Where Website Crawl saves time on data collection.
Data from the whole website
All pages of a section or website in one job
Content for AI systems
A whole website as Markdown or text
Analytics
Structured data from many pages
Site map
Which pages a website has: from links and the sitemap
Only the section you need
Include / exclude filters and domain restriction
Thousands of pages
Page limit and crawl depth to fit the task
How to connect
Three steps.
Sign up on the website
Create an account — the API key will appear in the dashboard.
Pay for a plan
Choose a credit package — from $3 per 1,000 credits.
Set up the request
Specify the start URL and crawl rules: depth, page limit, domains and URL filters. Choose what to get for each page.
Price
PSB Parser tools
Data collection from websites, search engines and AI in one API.
Website Crawl
Crawl a whole website: links and sitemap, depth, limits, URL filters.
You are hereGoogle Search API
Structured Google results: organic, PAA, AI Overview, Knowledge Graph, ads.
Learn more →More about Website Crawl
What Website Crawl is
Website Crawl is a PSB Parser tool for crawling a whole website. It starts from the start URL, finds new pages by links and through the sitemap and collects them one after another — until it reaches the set depth or page limit.
Only the pages you need
Include / exclude rules, domain restriction and subdomain handling keep only the sections you need in the crawl. URL normalization, query parameter handling and duplicate removal get rid of repeats of the same page.
Data, not just a list of addresses
Every page found goes through the Web Scraper: you can get HTML, text, Markdown, structured data or AI analysis right away — from simply loading HTML to thousands of pages turned into data for APIs, analytics and AI systems.
Website Crawl FAQ
How does Website Crawl differ from Web Scraper?
How does Crawl find pages?
Can I limit the crawl?
What about duplicates and parameters in URLs?
What format does the data come in?
How much does it cost?
Get started with Website Crawl
PSB Parser: from simply loading HTML to crawling thousands of pages and turning web content into data for APIs, analytics and AI systems.