Sitemaps and robots.txt are two of the most misunderstood tools in technical SEO, largely because they sound like they do the same job: controlling what search engines see. They don’t. One tells crawlers what to look at; the other tells crawlers where they’re welcome. Confusing the two, or assuming blocking something in robots.txt removes it from search, causes some of the most persistent indexing problems on WordPress sites. This guide, part of the Technical SEO series, covers how the two actually work together, alongside WordPress’s own built-in sitemap system.
Crawling versus indexing
Crawling is the process of a search engine’s bot requesting and downloading a page. Indexing is the separate decision to store that page and make it eligible to appear in search results. A page can be crawled and still not indexed (due to thin content or a canonical pointing elsewhere), and, less intuitively, a page can be indexed to some degree without ever being successfully crawled, if other pages link to it with enough context for Google to know it exists. Sitemaps and robots.txt each influence only one side of this: sitemaps help with discovery for crawling, while robots.txt controls crawl access, not the indexing decision itself.
What robots.txt can and cannot do
Robots.txt is a plain-text file that tells well-behaved crawlers which parts of a site they should avoid requesting. It can reduce unnecessary crawl activity on low-value paths (internal search results, admin areas) and can specify a sitemap location. It cannot guarantee a URL stays out of search results, cannot enforce compliance from crawlers that ignore it, and cannot remove a URL that’s already indexed.
Why robots.txt is not a reliable method for removing a URL from search
Google’s own documentation is explicit on this point: a URL disallowed in robots.txt can still appear in search results, typically as a bare URL with no title or description, if other pages link to it, because Google learns the URL exists without ever being permitted to crawl its content. If the actual goal is keeping a page out of search results entirely, the correct tool is a noindex meta tag or HTTP header on the page itself, which requires the page to be crawlable so the directive can actually be seen.
WordPress virtual robots.txt file
If no physical robots.txt file exists in the site’s root directory, WordPress generates a basic virtual one dynamically, without an actual file being present on the server. This default virtual file is minimal and generally permissive. If you need custom rules, either a plugin managing robots.txt or a real, physical file uploaded to the root directory will take precedence, and WordPress will stop generating the virtual version automatically once a physical file exists.
WordPress core XML sitemap functionality
Since WordPress 5.5, core WordPress automatically generates a basic XML sitemap, accessible at /wp-sitemap.xml, with no setup required and no dedicated settings screen to configure it. By default, it includes public post types, taxonomies, author archives, and the homepage. This built-in sitemap is functional but limited; it offers no interface for excluding specific content, and most sites running a dedicated SEO plugin will have that plugin’s own, more configurable sitemap system active instead, effectively superseding the core version.
Sitemap indexes and individual sitemap files
Rather than one enormous file, WordPress’s sitemap system (and most SEO plugin implementations) use a sitemap index file that links out to smaller individual sitemap files, split by post type or in batches once a content type grows large. WordPress core’s implementation caps each individual sitemap file at a maximum of 2,000 entries, with the index able to reference up to 50,000 individual sitemaps, which is far more capacity than nearly any WordPress site will need.
Posts, pages, categories, tags, authors, and custom post types
By default, WordPress’s core sitemap includes all public, publicly queryable post types and taxonomies, meaning posts, pages, categories, tags, and author archives are typically all included automatically, alongside any custom post type registered as public. Not every one of these necessarily belongs in a sitemap for every site; author archives on a single-author blog, for instance, often add little value and are commonly excluded via a filter.
Which URLs generally belong in a sitemap
A sitemap should generally contain only canonical, indexable, genuinely valuable URLs, the actual pages you want search engines to prioritize discovering and re-crawling. This typically means published posts and pages, not drafts, not paginated archive pages beyond the first, not internal search result pages, and not any URL that’s blocked, redirected, or marked noindex elsewhere.
Excluding low-value or non-canonical URLs
Thin or low-value archive types (a tag applied to only one post, for example, as covered in more detail in WordPress Technical SEO Audit: A Step-by-Step Checklist) are reasonable candidates for exclusion from the sitemap even if they remain technically accessible, since a sitemap is meant to highlight priority content rather than list every URL that technically exists on the site.
Noindex versus disallow
These are frequently confused but serve different purposes and should rarely be combined on the same URL. A noindex directive requires the page to be crawled so the directive is seen and honored; disallowing that same URL in robots.txt prevents the crawl entirely, which can prevent Google from ever discovering the noindex instruction, potentially leaving the URL indexed anyway from external references. If the goal is genuine removal from search results, use noindex alone and leave the URL crawlable.
Canonical URLs and sitemap consistency
Every URL listed in a sitemap should match that page’s own declared canonical URL exactly; listing a non-canonical URL variant in the sitemap while the page itself declares a different canonical creates a direct, avoidable conflict. As covered in Canonical URLs, Redirects, and Duplicate Content in WordPress, Google’s guidance explicitly warns against specifying different preferred URLs through different mechanisms.
Blocked resources and rendering
Blocking CSS, JavaScript, image directories, or admin-ajax.php in robots.txt without a specific, verified reason can prevent Google from rendering pages accurately, potentially affecting how mobile usability, layout, and even content visibility are assessed. These resources should generally remain crawlable by default; only restrict them if there’s a concrete, confirmed reason tied to a specific problem, not as a general-purpose precaution.
Query parameters and internal search URLs
URL parameters such as tracking codes, sort orders, and filters—and WordPress internal search result URLs such as ?s=searchterm—can create many low-value or near-duplicate URLs. They generally should not appear in an XML sitemap. A robots.txt disallow can reduce crawling of a clearly defined pattern, but it does not guarantee removal from search. If an internal-search URL must not appear in results, keep it crawlable long enough for a noindex directive to be seen, and confirm the rule with URL Inspection before adding broader crawl restrictions.
Staging and development websites
A staging or development copy of a site should never be indexed. WordPress’s built-in “Discourage search engines from indexing this site” setting (Settings → Reading) adds a sitewide noindex instruction, which is a reasonable temporary measure, though it relies on the setting actually being toggled off again after launch, a step that’s easy to forget. Restricting access at the server level (password protection or IP restriction) is a more reliable safeguard for staging environments specifically, since it prevents crawling entirely rather than relying on a directive crawlers must choose to respect.
Search Console sitemap submission
Submit the sitemap index URL (not each individual child sitemap) under Search Console’s Sitemaps report, and monitor it for reported errors or warnings. A successfully processed sitemap doesn’t guarantee every listed URL will be indexed; it simply confirms Google has successfully read the file and is aware of the URLs it contains.
URL Inspection and Page Indexing reports
Search Console’s Page Indexing report (previously called Index Coverage) shows aggregate indexing status across the site, with specific exclusion reasons for URLs that aren’t indexed. The URL Inspection tool provides a per-URL view, including whether Google can access the page, which canonical it selected, and whether the page is currently indexed, useful for confirming a specific fix took effect.
Sitemap errors and warnings
Common sitemap issues include URLs that return errors, redirect elsewhere, or are blocked by robots.txt despite being listed in the sitemap, as well as a sitemap that fails to load entirely due to a server or plugin conflict. Reviewing the specific errors Search Console reports, rather than assuming the sitemap is simply “fine” because it was submitted successfully, is necessary to catch these.
Multiple SEO plugins and conflicting sitemap systems
Running more than one SEO or sitemap-generating plugin simultaneously can result in multiple, conflicting sitemap files being generated and referenced, confusing both Google and anyone trying to maintain the site. Confirm only one plugin (or WordPress core’s own default, if no plugin is active) is responsible for sitemap generation at any given time.
Migration and domain-change considerations
During a domain migration, update the sitemap to reference the new domain’s URLs, submit the new sitemap in Search Console for the new property, and use the Change of Address tool alongside the redirects covered in the canonical and redirects article referenced above. An outdated sitemap still referencing the old domain after a migration sends a conflicting signal during an already sensitive transition period.
Safe troubleshooting process
Start with Search Console’s Page Indexing report to identify the scope of any problem, then use URL Inspection on specific affected URLs to see exactly what Google observed: whether the page was crawled, what canonical was selected, and whether robots.txt or a noindex directive was involved. Make one change at a time and allow adequate time for recrawling before concluding whether it worked.
Common robots.txt and sitemap mistakes
- Assuming a robots.txt disallow removes a URL from search results
- Combining noindex and disallow on the same URL, preventing the noindex from ever being seen
- Blocking CSS, JavaScript, or admin-ajax.php without a specific, verified reason
- Listing noindexed, blocked, or redirected URLs in the sitemap
- Running multiple plugins that each generate a separate, conflicting sitemap
- Forgetting to disable “Discourage search engines from indexing this site” after launching a site built with it enabled
Practical crawl-control checklist
- Confirm robots.txt returns a 200 status and isn’t blocking important content, CSS, or JS
- Verify the sitemap lists only canonical, indexable, genuinely valuable URLs
- Use noindex, not disallow, for pages that must not appear in search results
- Confirm sitemap URLs match each page’s own declared canonical
- Submit the sitemap index in Search Console and monitor for errors
- Confirm only one plugin or system is generating the site’s sitemap
- Restrict staging sites at the server level, not just with a noindex setting
- Use URL Inspection to confirm fixes on specific URLs after changes
Key Takeaways
- Crawling and indexing are separate processes; robots.txt controls the former, not the latter.
- Robots.txt is not a reliable way to remove a URL from search results; use noindex instead, and keep the page crawlable so the directive can be seen.
- WordPress has generated a basic core sitemap at /wp-sitemap.xml since version 5.5, though most sites with an SEO plugin use that plugin’s more configurable system instead.
- Sitemap URLs should always match each page’s declared canonical to avoid conflicting signals.
- Blocking CSS, JavaScript, or admin-ajax.php without a specific, confirmed reason can distort how Google renders and evaluates a page.
FAQs
Does blocking a page in robots.txt remove it from Google search results?
Not reliably. Google’s documentation confirms a disallowed page can still appear in search results, typically without a title or description, if other pages link to it. Use a noindex directive on the page itself for genuine removal.
Do I need an SEO plugin if WordPress already generates a sitemap?
Not strictly, but WordPress’s core sitemap (since version 5.5) offers no configuration interface for excluding specific content types or pages. Most sites benefit from a dedicated SEO plugin’s more configurable sitemap if fine-grained control is needed.
Should I combine noindex and disallow for pages I want out of search entirely?
No. Disallowing a noindexed page can prevent Google from ever crawling it and seeing the noindex instruction, which can leave the URL indexed anyway based on external links. Use noindex alone and keep the page crawlable.
Is it safe to block admin-ajax.php in robots.txt?
Generally, no, unless you have a specific, confirmed reason. Many themes and plugins rely on admin-ajax.php for front-end functionality, and blocking it without verification can interfere with how a page renders or functions when Google evaluates it.
Why does my sitemap show as submitted successfully but some pages still aren’t indexed?
A successful sitemap submission only confirms Google could read the file; it doesn’t guarantee indexing. Check the Page Indexing report for the specific exclusion reason for those URLs, common causes include thin content, duplicate content, or a canonical pointing elsewhere.
