A robots.txt file tells crawlers which parts of a website they’re allowed to visit, and it lives as a plain text file at yourdomain.com/robots.txt. Most home service businesses only need a few lines: a pointer to the sitemap and, in most cases, no blocks at all, since you generally want Google and Bing to find every public page. The part owners get wrong most often is what the file can’t do: it doesn’t keep a page out of search results, it only stops crawling, and it has nothing to do with your Google Business Profile or Google Ads account.
What robots.txt actually controls
Google’s own documentation draws a hard line between crawling and indexing. A Disallow line in robots.txt tells a crawler not to visit a page, full stop. If other pages still link to that URL, Google can sometimes show it in search results anyway, just with the title and URL and no description, because it never read the page content to write one. If you actually want a page kept out of search results, that’s a job for a noindex meta tag or HTTP header on the page itself, not a robots.txt line. For a contractor site, this matters most for internal pages, an employee login page is better handled with noindex on the page than a robots.txt block alone, otherwise the bare URL can still surface somewhere.
The basic syntax in four lines
A working robots.txt for most business websites comes down to a handful of directives:
User-agent: *
Disallow:
Sitemap: https://www.samplecontractor.com/sitemap-index.xmlUser-agent sets which crawler the block below applies to, the asterisk means all of them. Disallow with nothing after it means nothing is blocked, which is the right default for a public business site. A specific line like Disallow: /admin/ only becomes necessary once there’s a real internal area, a CMS login or a staging preview. Per Google’s documentation, the Sitemap directive isn’t tied to any single user-agent, it applies globally and can appear more than once if you submit multiple sitemaps. It needs a full URL including https:// and the domain, a relative path won’t be read correctly.
One detail trips up a lot of small business owners: Google and Bing don’t treat the same file identically. Bing’s own documentation confirms Bingbot supports the Crawl-delay directive, which throttles how often it requests pages, for example Crawl-delay: 5 for a slower pace. Google ignores that same line entirely, Googlebot’s crawl rate is instead shaped by your server’s response times and the crawl settings inside Search Console. A typical home services site with modest traffic rarely needs Crawl-delay at all, but it’s worth knowing that a line which works for Bing does nothing for Google.
Common mistakes on small business sites
The most expensive mistake is a leftover Disallow: / line that blocks the entire site for whatever crawler reads it. It usually comes from a staging or preview environment, where it was correct, and then gets carried over into the live site by accident during launch. After switching providers or a full redesign, similar to checking 301 redirects after a site move, that one line is worth a direct look. A second common issue is a stale or misspelled sitemap path, often left over after a CMS switch when the new platform generates a different filename, like sitemap-index.xml instead of the old sitemap.xml. A third point is file size: per Google’s specification, only the first 500 kibibytes of a robots.txt file get processed, anything past that is ignored. For a normal contractor site with a few dozen pages that’s never a real concern, but it can matter for sites with long, automatically generated block lists.
The fourth mistake is the easiest to miss and the most damaging: blocking your own design and script files. A rule that’s too broad, like Disallow: /assets/ or Disallow: /css/, can accidentally sweep up the CSS and JavaScript that render the page, and once that happens Googlebot can no longer see the site the way a visitor does. Google’s own webmaster guidelines state directly that disallowing crawling of JavaScript or CSS harms how well its systems render and index a page, which can lead to weaker rankings. Without CSS, Google’s rendering can miss a responsive layout entirely and treat a page that’s genuinely mobile-friendly as a bare text document instead. Broad directory blocks like /assets/, /static/, or /wp-includes/ are worth a second look before they go live.
AI crawlers in robots.txt
Since 2023, a second category of bots has shown up alongside the classic search crawlers, ones that gather content either to train AI models or to answer questions directly. Leaving them unblocked is usually the right call if you want visibility in AI answer engines, as covered in AI Answer Engines and GEO for Businesses. What matters is knowing what each bot is actually for:
| Crawler | Operator | What it’s for |
|---|---|---|
| GPTBot | OpenAI | Collects content for training generative models |
| Google-Extended | A separate opt-out for Gemini training, distinct from regular Googlebot | |
| ClaudeBot | Anthropic | Collects content for training Claude models |
| PerplexityBot | Perplexity | Crawls for indexing that feeds directly into Perplexity’s answers |
| CCBot | Common Crawl | Builds an open dataset many AI training pipelines draw from |
One detail is worth flagging directly: blocking Google-Extended has no effect on regular Google Search rankings, according to Google, since it’s a separate token from the standard Googlebot used for indexing. So a business that blocks training crawlers out of caution isn’t risking its search visibility, but PerplexityBot deserves more thought, since blocking it directly affects whether the site shows up in Perplexity’s own answers. For a home services business trying to pick up leads from AI-driven search, allowing all of these by default is usually the more useful setting than blocking them across the board.
Testing your own robots.txt
The fastest check is typing your own domain with /robots.txt appended into a browser, which shows immediately whether the file exists and what it says. For a more thorough check, curl -I https://www.samplecontractor.com/robots.txt confirms the server actually returns a 200 status instead of a redirect or an error page. Google retired the old robots.txt Tester inside Search Console at the end of 2023 and replaced it with a robots.txt report, which shows when Google last fetched the file and flags parsing warnings, but no longer lets you test individual URLs against the rules. To check whether one specific page is currently blocked, the URL Inspection tool’s live test is the right place to look, reachable from the same Search Console setup used to submit the sitemap.
Next step
For most businesses, a robots.txt file with an open default and a correct sitemap line is all that’s needed, the real risk shows up when there are multiple environments or a provider switch involved. Mr.Site checks this file as part of its managed website service during every redesign and major technical change, so an accidental block doesn’t stay live unnoticed.