WordPress scraper

What is the WordPress scraper?

The WordPress scraper is a powerful content extraction tool that connects to any public WordPress website through its built-in REST API. It automatically retrieves all published content, including posts, pages, categories, and tags, and presents them in an organized, easy-to-browse format inside your WebGPT dashboard.

Because the tool uses the standard WordPress REST API (which is enabled by default on virtually all WordPress sites), you do not need any login credentials or admin access to the target site. Simply enter the URL and let the scraper do the rest.

How to use the scraper

Follow these steps to scrape content from any WordPress website:

1. Navigate to WordPress Scraper
   From the WebGPT dashboard, go to Utility Tools → WordPress Scraper, or click the WordPress Scraper card on your dashboard.
2. Enter a WordPress site URL
   In the URL input field, type or paste the full URL of any WordPress website you want to scrape. For example: https://example.com. The tool accepts both root domains and subdirectory installations (e.g., https://example.com/blog).
3. Click "Get"
   Press the Get button to start the extraction process. Note that each scrape counts toward your daily usage limit (see Usage limits below).
4. Wait for the progress to complete
   The scraper will begin fetching data from the target site. You will see a progress indicator showing which content types are being retrieved. Depending on the size of the site, this may take anywhere from a few seconds to a couple of minutes.
5. View the extracted data
   Once scraping is complete, the results are organized into clearly labeled tabs:
   • Posts — All published blog posts, including title, excerpt, date, author, and full content
   • Pages — All published static pages (About, Contact, etc.)
   • Categories — The site's category taxonomy with post counts
   • Tags — The site's tag taxonomy with post counts
6. Browse content in the modal preview
   Click on any post or page title to open a modal window showing the full content. This lets you read through articles without leaving the scraper page.

Knowledge base integration

One of the most powerful features of the WordPress scraper is its integration with the AI chatbot system. You can save scraped content directly to your AI chatbot's knowledge base with just a few clicks.

Saving content to the knowledge base

After scraping a website, you can select individual posts or pages and add them to your chatbot's knowledge base:

• Browse the scraped posts or pages
• Click the option to add the content to the knowledge base
• The content will be saved with appropriate tags for easy organization
• The saved content becomes available to your AI chatbots for answering questions

This is an excellent way to quickly build a comprehensive knowledge base from existing content. For example, you could scrape your own website's content and feed it into a customer support chatbot, or scrape industry resources to create a research assistant.

History tab

The WordPress scraper keeps a record of all websites you have previously scraped. The History tab lets you:

• View past scrapes — See a list of all websites you have scraped, along with the date and time of each scrape
• Re-access data — Click on any previous scrape to view the extracted content again without needing to re-scrape the site
• Favorites — Mark frequently scraped sites as favorites for quick access — favorited sites always appear first regardless of the active sort
• Sort — Use the sort dropdown to reorder results by:
  • Date (newest or oldest first)
  • Best SERP rank — sites that ranked highest in search results appear first
  • SERP appearances — sites discovered across the most search queries appear first
  • Results count — sites with the most scraped content appear first
• Filter — Narrow results by domain name, language, country, minimum content count, or SERP keyword

This is useful when you want to compare how a site's content has changed over time, quickly find the top-ranked sites from your SERP research, or revisit data from a previous session.

Usage limits

The WordPress scraper is subject to daily usage limits that vary depending on your subscription plan. Each time you scrape a new website, it counts as one use toward your daily limit.

You can see your current usage and remaining quota in the usage indicator displayed on the scraper page. When you reach your daily limit, the scrape button will be disabled until the next day (limits reset daily).

For details on how many daily scrapes are included in each plan, see the Plans & usage limits page.

Use cases

The WordPress scraper is a versatile research tool. Here are some common ways users take advantage of it:

Competitor content analysis

Scrape competitor websites to understand their content strategy. See what topics they cover, how they organize their categories, and what kind of posts get published. Use this intelligence to inform your own content planning.

Content migration research

If you are planning to migrate content from one WordPress site to another, the scraper gives you a quick overview of everything on the source site. While it is not a migration tool itself, it helps you plan and audit content before starting a migration.

Finding article ideas

Browse the posts and categories of popular blogs in your niche to discover topics you have not covered yet. The scraper makes it easy to quickly scan through hundreds of posts and identify content gaps.

Building chatbot knowledge bases

Use the knowledge base integration to feed scraped content into your AI chatbots. This is one of the fastest ways to build a rich knowledge base, whether from your own sites or from public information sources.

Content inspiration and research

When writing about a new topic, scrape a few authoritative WordPress sites in that space to understand how existing content is structured. This can help you write more comprehensive, well-organized articles.