I once wrote a rule that blocked “bot” as a substring, feeling pretty efficient about covering every scraper in one go. It also blocked Googlebot, since that string is right there in the name. Traffic quietly dropped for about a week before I noticed. If you’re trying to block bad bots using .htaccess, the mechanics are simple — the part that actually takes care is not shooting yourself in the foot with an overly broad rule.
Quick Answer
- Block by user agent using
mod_rewritein.htaccess, matching known bad bot names againstHTTP_USER_AGENT - Block by IP address for bots that don’t send an identifiable user agent, or that keep coming back under new names
- Never use broad patterns like
botorcrawleralone — you’ll catch legitimate search engine crawlers too - Check your access logs first to find out which bots are actually hitting you before writing any rules
.htaccessblocking only stops bots that don’t spoof their user agent — persistent scrapers may need firewall or CDN-level blocking instead
Why Bad Bots Are a Problem in the First Place
Not every bot is malicious, and that’s actually the crux of why this needs care rather than a blanket rule. Search engines, uptime monitors, and legitimate SEO tools all identify themselves as bots too.
Content scrapers. These crawl your site to copy content wholesale, often to republish elsewhere or feed into something else entirely. They tend to hit every page methodically, fast, with no regard for robots.txt.
Aggressive SEO crawlers eating bandwidth. Tools like some third-party backlink checkers crawl at a volume that’s fine for the tool’s purpose but genuinely taxing on a smaller site’s server resources, even though they’re not “malicious” in the traditional sense.
Comment spam and vulnerability-probing bots. These specifically target login pages, comment forms, and known vulnerable endpoints, trying the same exploits across thousands of sites automatically.
Bots that ignore robots.txt entirely. This is the real reason .htaccess blocking exists as a separate layer. robots.txt is a request, not an enforcement mechanism — well-behaved bots respect it, badly-behaved ones just ignore it completely.
Step-by-Step: Blocking Bots by User Agent
Step 1: Find out who’s actually hitting your site
Before writing any rule, check your access logs:
grep -i "bot\|crawl\|spider" /home/username/access-logs/yourdomain.com | awk -F'"' '{print $6}' | sort | uniq -c | sort -rn | head -30This surfaces the most frequent user agents matching bot-like patterns, sorted by how often they show up. You want specific names from this list, not a generic guess.
Step 2: Add the RewriteEngine directive
At the top of your .htaccess file (or below an existing RewriteEngine On line if one’s already there):
apache
RewriteEngine OnStep 3: Add specific user agent blocks
apache
RewriteCond %{HTTP_USER_AGENT} (MJ12bot|SemrushBot|DotBot|AhrefsBot) [NC]
RewriteRule .* - [F,L]The [NC] flag makes the match case-insensitive. [F,L] returns a 403 Forbidden and stops processing further rules for that request. Each name inside the parentheses, separated by |, gets checked against the same condition.
Step 4: Test carefully before assuming it worked
Use a user agent switcher extension in your browser, or curl with a spoofed user agent, to confirm the block actually triggers:
curl -A "MJ12bot" -I https://yoursite.comYou should see a 403 response. If you get a normal 200, the rule isn’t matching — double-check for typos or whether a different .htaccess file (in a subdirectory) is taking precedence.
Step 5: Add IP-based blocking for bots without a clean user agent string
Some bots send a generic or spoofed user agent, so this needs a different approach:
apache
<RequireAll>
Require all granted
Require not ip 203.0.113.55
Require not ip 198.51.100.0/24
</RequireAll>This is the Apache 2.4 syntax. If your server’s still running the older 2.2-style config, it looks like this instead:
apache
Order Allow,Deny
Allow from all
Deny from 203.0.113.55Check which one your server actually uses before copying blindly — mixing the two syntaxes in the same file tends to just break things rather than doing nothing.
Comparison: User Agent Blocking vs IP Blocking vs robots.txt
| Method | Stops well-behaved bots? | Stops malicious bots? | Notes |
|---|---|---|---|
| robots.txt | Yes | No | Purely advisory, ignored by anything malicious |
| User agent block (.htaccess) | Yes | Sometimes | Only works if the bot honestly identifies itself |
| IP block (.htaccess) | Yes | Sometimes | Bots on rotating IPs or cloud providers get around this fast |
| Firewall/CDN-level (Cloudflare, etc.) | Yes | Mostly | Handles rotating IPs and spoofed agents far better |
None of these are airtight on their own, and that’s kind of the honest answer here — layering a couple of them together gets you most of the way, but a genuinely determined scraper with rotating residential proxies is a different fight entirely.
What Actually Worked For Me
The “bot” substring mistake from the intro — I want to be specific about what actually went wrong, because the lesson isn’t just “be careful,” it’s a bit more specific than that. I’d seen a spike in requests from something calling itself AhrefsBot, wanted to block it fast, and instead of adding that one specific name, I typed a pattern matching anything containing “bot” as a shortcut, figuring it’d catch future ones too.
It did catch future ones. It also caught Googlebot, Bingbot, and a legitimate uptime monitor the client relied on for alerting. Organic traffic dipped steadily over about a week before anyone noticed, since there’s no error message visible to a site owner when Googlebot just quietly stops being let in — it just slowly stops showing up in search results with no obvious trigger event to point to.
I found it by accident, honestly, cross-referencing Search Console’s crawl stats (which had flatlined) against the .htaccess file, and the pattern jumped out immediately once I actually looked at what I’d written weeks earlier. Swapped the broad match for the specific bot names I actually needed to block, and crawl activity picked back up within a couple of days. Lesson stuck: never match a generic word that legitimate infrastructure also uses in its own name.
Advanced Fixes and Edge Cases
Blocking by request pattern instead of user agent. Some bots specifically target certain URL patterns — wp-login.php, xmlrpc.php, admin paths that don’t exist on your setup. You can block based on the requested path itself rather than relying on user agent honesty:
apache
RewriteCond %{REQUEST_URI} ^/xmlrpc\.php$
RewriteRule .* - [F,L]Rate limiting instead of outright blocking. If a bot is legitimate-ish (an SEO tool, say) but hitting too aggressively, mod_evasive or your host’s rate-limiting tools throttle rather than fully block — useful when you don’t want to lose the traffic entirely, just slow it down.
Referrer-based spam blocking. Comment and referrer spam bots sometimes fake a referrer header rather than a user agent. A similar RewriteCond against HTTP_REFERER handles that pattern:
apache
RewriteCond %{HTTP_REFERER} (semalt\.com|buttons-for-website\.com) [NC]
RewriteRule .* - [F,L]Rotating IP scrapers. If a bot keeps coming back under new IPs after you block the old ones, .htaccess-level blocking is genuinely losing proposition long-term. This is where moving to a CDN or WAF layer (Cloudflare, Sucuri) that fingerprints behavior rather than just IP or user agent becomes the more durable fix.
Prevention Tips
- Always check logs before writing a block rule — don’t guess at bot names from memory
- Never use single common words (bot, crawler, spider) alone as your match pattern
- Keep a running list of legitimate bots you rely on (search engines, monitoring tools, any integrations) so you don’t accidentally catch them later
- Re-check Search Console or equivalent crawl stats periodically after adding new rules, to catch accidental over-blocking early rather than weeks later
- Consider a CDN-level bot management layer if .htaccess rules are becoming a constant game of whack-a-mole
FAQ
Will blocking bots in .htaccess slow down my site? Not meaningfully for a reasonable number of rules. It runs before the request reaches PHP, so blocked requests actually consume less server resource than letting them through.
Can I block bots without knowing mod_rewrite syntax? Some security plugins (Wordfence, and similar) offer bot-blocking through a UI without touching .htaccess directly, though the underlying mechanism is often similar.
Why did my rule work in testing but not in production? Check whether a subdirectory has its own .htaccess file overriding the root one, or whether a caching layer is serving cached responses before the rule ever gets evaluated.
Is IP blocking pointless against cloud-hosted bots? Not entirely pointless, but yes, less durable — cloud providers give bots access to huge IP ranges, so a persistent scraper can rotate around a block fairly easily. It slows them down more than it stops them outright.
Should I just block all bots and only allow known good ones? That’s technically possible with an allowlist approach, but it’s higher maintenance and risks blocking legitimate tools you didn’t think to allowlist. Most people are better served by blocking known bad actors specifically.
Editor’s Opinion
the mechanics of blocking bots in htaccess are genuinely simple, it’s the discipline of being specific that trips people up, myself included apparently. check your logs, name the actual bot, don’t get clever with shortcuts that also match the good guys. if a bot keeps coming back after you block it, that’s your sign to stop fighting it at this layer and look at something more robust instead.
