Google and other crawlers use robots.txt to discover the sitemap. When a Sitemap: line is present, they find the map automatically without manual submission. A one-line addition with outsized impact on discovery speed.
Why this matters
Google offers two ways to surface a sitemap: manual submission via Search Console, or declaration in robots.txt. Manual submission only helps Google. A Sitemap: line in robots.txt works for every crawler - Bing, DuckDuckGo, Yandex, Baidu, AhrefsBot - all of which check robots.txt before any other request.
Without that line, crawlers visiting for the first time rely on internal links alone to discover pages, dramatically slowing indexation of new content. For search engines without a webmaster tools workflow you actively use (Bing Webmaster Tools, Yandex Webmaster), the Sitemap: line is the only way to guarantee complete discovery.
Bing impact specifically: Bing accounts for roughly 7-10% of global search and is the underlying engine for AI-driven search (ChatGPT, Bing Chat, Copilot). If Bing has not crawled your site, AI assistants will not surface it in answers.
How to detect
Open https://YOUR-SITE/robots.txt in a browser and look for a line starting with Sitemap:. Missing? Problem. The expected structure:
User-agent: *
Allow: /
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/wp-sitemap.xmlComplementary check: in Google Search Console > Sitemaps - if the sitemap was not submitted manually and is also missing from robots.txt, Google will not find it automatically. Bing Webmaster Tools has a similar report.
Third check: curl https://YOUR-SITE/robots.txt from a terminal - shows the raw file exactly as Google sees it.
How to fix
If you use an SEO plugin (Yoast, Rank Math, RankPlus, SEOPress), the line should be automatic. Verify in plugin settings:
- Yoast SEO: SEO > Tools > File editor > robots.txt - should display the line. If not, the sitemap is managed but not declared; edit the file here.
- Rank Math: General Settings > Edit robots.txt - manually add the line if missing.
- RankPlus: the "Add sitemap to robots.txt" option should be enabled.
Without an SEO plugin and on a virtual robots.txt from WordPress core (5.5+), use the robots_txt filter in your theme's functions.php or via Code Snippets:
add_filter('robots_txt', function($output, $public) {
if ('1' === $public) {
$output .= "\nSitemap: https://example.com/sitemap.xml\n";
}
return $output;
}, 10, 2);Alternatively, create a physical robots.txt file in the site root (public_html/) with the right content. The physical file overrides the WP virtual one.
Confirm the sitemap URL is correct: https not http, with or without www matching your canonical domain.
Common mistakes
First mistake: putting the Sitemap line inside a User-agent: block. The Sitemap declaration is global - it must live outside any User-agent block, typically at the bottom of the file. Inside a User-agent: Googlebot block, only Googlebot would see it.
Second mistake: declaring multiple sub-sitemaps unnecessarily. If you have several maps (sitemap-posts, sitemap-pages, sitemap-products), declare only the sitemap_index that aggregates them, not each one separately. SEO plugins handle this automatically.
Third mistake: URL with parameters. Sitemap: https://example.com/sitemap.xml?ver=2 looks weird to crawlers. Strip unnecessary query strings.
Fourth mistake: pointing at the wrong domain. After migrating from http to https or adding www, verify robots.txt points at the canonical domain, not the legacy one.
Fifth mistake: Sitemap line alongside a global Disallow: /. If the site is blocked, declaring a sitemap is meaningless. Fix the Disallow first.
Verifying the fix
Refresh /robots.txt and confirm the line is present. Submit the sitemap manually in Google Search Console > Sitemaps - you also get error reports there. In Bing Webmaster Tools, register the site and confirm the sitemap was detected. After a week, watch the Coverage report for an uptick in newly discovered pages.