AhrefsBot can be blocked through robots.txt, .htaccess rules, Nginx server configs, Cloudflare WAF rules, or direct IP-range blocking. Each method offers a different level of enforcement — robots.txt asks politely, while IP-based firewall rules shut the door completely. The right choice depends on why you want to block it and how strict you need to be.
This guide walks through every method step by step, explains the trade-offs of each, and covers a scenario most articles skip: how to verify you’re actually dealing with the real AhrefsBot before you start blocking anything.
What AhrefsBot Actually Does (And What AhrefsSiteAudit Is)
Ahrefs runs two separate crawlers, and the distinction matters when you’re deciding what to block.
AhrefsBot is the main crawler. It follows links across the web, records backlink relationships, anchor texts, on-page content, and internal link structures. All of that data feeds into Ahrefs’ core tools — Site Explorer, Keywords Explorer, Content Explorer — and also powers Yep.com, the search engine Ahrefs launched. According to Cloudflare Radar data, AhrefsBot is the single most active SEO crawler on the internet, second only to Googlebot among all web crawlers. It visits over 8 billion pages every 24 hours and updates its index roughly every 15 to 30 minutes.
AhrefsSiteAudit is a separate crawler with its own user-agent string. It powers the Site Audit feature inside Ahrefs, which site owners use to find technical SEO issues on their own domains. If you run audits through Ahrefs Webmaster Tools (which is free), this is the bot doing the crawling.
Why does this matter? Because you might want to block the global crawler (AhrefsBot) to keep competitors out of your backlink data, while still allowing AhrefsSiteAudit so you can run your own technical audits. Or vice versa. Blocking both with a blanket rule when you only meant to target one is a common mistake.
Their user-agent strings look like this:
Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)
Mozilla/5.0 (compatible; AhrefsSiteAudit/6.1; +http://ahrefs.com/robot/site-audit)
Keep these in mind — you’ll need them for the blocking methods below.
Verify Before You Block: Is It Really AhrefsBot?
Before setting up any blocking rules, confirm that the traffic claiming to be AhrefsBot is actually coming from Ahrefs. User-agent strings can be spoofed. Scrapers, bad bots, and even competitors sometimes disguise themselves as known crawlers to slip past security rules.
Ahrefs makes verification straightforward because they publish their full IP ranges and support reverse DNS lookups.
Step 1: Check the IP Against Ahrefs’ Published Ranges
Ahrefs maintains a public list of all IP ranges used by both AhrefsBot and AhrefsSiteAudit. You can pull these as JSON or plain text from their help center at ahrefs.com/robot. The crawlers operate from data centers in Singapore, the United Kingdom, France, Canada, and Germany.
Some of the key CIDR ranges include:
5.39.1.224/27
51.89.129.0/24
51.161.37.0/24
51.195.183.0/24
15.235.27.0/24
176.31.139.0/27
If the IP hitting your server isn’t in any of these ranges, it’s not AhrefsBot — regardless of what the user-agent says.
Step 2: Run a Reverse DNS Lookup
For additional confirmation, do a reverse DNS lookup on the source IP. Legitimate AhrefsBot traffic will always resolve to a hostname ending in ahrefs.com or ahrefs.net.
dig -x 51.89.129.45 +short
If the result comes back with an Ahrefs hostname, run a forward lookup to confirm the match:
dig crawl-51-89-129-45.ahrefs.com +short
The forward result should return the same IP. If either check fails, the traffic is spoofed.
Step 3: Use Cloudflare’s Verified Bot List
If your site sits behind Cloudflare, you get an extra shortcut. Both AhrefsBot and AhrefsSiteAudit are recognized as verified “good” bots in Cloudflare’s bot ecosystem. This means Cloudflare can automatically distinguish real Ahrefs traffic from spoofed traffic in your WAF logs and analytics — no manual DNS lookups required.
Why You Might Want to Block AhrefsBot
Most website owners considering a block fall into one of three camps.
Server Performance
AhrefsBot is aggressive. On shared hosting or resource-constrained servers, its crawl bursts — sometimes hundreds of URLs in quick succession — can compete with real visitors for bandwidth and processing power. If your Time to First Byte spikes during bot activity, that slowdown affects both user experience and Core Web Vitals, which Google does use as a ranking factor.
For sites on higher-end hosting, this is rarely an issue. But budget WordPress installs, small Shopify stores on basic plans, and sites without CDN caching feel the impact.
Competitive Intelligence Protection
Every page AhrefsBot crawls becomes part of Ahrefs’ database. Any paying Ahrefs subscriber can then pull up your backlink profile, see which keywords you rank for, analyze your content strategy, identify your linking patterns, and reverse-engineer what’s working. For businesses in competitive niches — affiliate SEO, SaaS, fintech, e-commerce — that level of transparency can be a strategic liability.
Ahrefs’ own research across approximately 140 million websites found that about 6.31% of all sites block AhrefsBot, making it the third most blocked SEO crawler behind MJ12bot (Majestic) at 6.49% and SemrushBot at 6.34%.
Policy or Compliance Requirements
Some organizations — particularly in finance, healthcare, and government sectors — have blanket policies against non-essential automated crawlers. If your security team classifies all third-party data collectors as unnecessary risk, blocking AhrefsBot becomes a compliance checkbox rather than a strategic decision.
Method 1: Block AhrefsBot via Robots.txt
This is the easiest method and where most site owners should start. AhrefsBot respects robots.txt directives — Ahrefs explicitly documents this on their official bot page.
Full Site Block
Add these lines to your robots.txt file (located in your site’s root directory):
User-agent: AhrefsBot
Disallow: /
To block both crawlers:
User-agent: AhrefsBot
Disallow: /
User-agent: AhrefsSiteAudit
Disallow: /
Block Specific Directories Only
If you want AhrefsBot to crawl your public blog and main pages but stay away from sensitive areas:
User-agent: AhrefsBot
Disallow: /admin/
Disallow: /staging/
Disallow: /internal-reports/
Disallow: /client-portal/
This approach keeps your backlink data visible in Ahrefs (useful for your own analysis and for agencies reviewing your site) while protecting directories that shouldn’t appear in any third-party tool.
Throttle with Crawl-Delay
Rather than blocking entirely, you can slow AhrefsBot down. The crawl-delay directive tells the bot to wait a set number of seconds between consecutive requests:
User-agent: AhrefsBot
Crawl-delay: 10
A value of 10 means AhrefsBot waits at least 10 seconds before requesting the next page. Most site owners set this between 5 and 20 seconds depending on server capacity.
One nuance worth knowing: Ahrefs documents that crawl-delay is honored for HTML page requests, but when the bot renders pages and fetches associated assets — CSS, JavaScript, images — you may still see multiple requests clustered together in your logs. That doesn’t mean the bot is ignoring your directive. It’s the difference between page requests and asset requests.
Recommended Default Config
For most websites, a combined approach works well:
User-agent: AhrefsBot
Crawl-delay: 10
Disallow: /admin/
Disallow: /staging/
Disallow: /private/
User-agent: AhrefsSiteAudit
Allow: /
This setup slows down the main crawler, protects sensitive directories, and still allows you to run your own Site Audit through Ahrefs Webmaster Tools.
Important: If your robots.txt has syntax errors — missing colons, incorrect spacing, wrong capitalization in the user-agent name — AhrefsBot may not recognize your directives and will continue crawling as normal. Validate your robots.txt file before assuming it’s working.
Method 2: Block via .htaccess (Apache Servers)
Robots.txt is a polite request. The bot has to choose to obey it. If you need server-level enforcement — where requests get rejected before they reach your content — .htaccess rules on Apache servers are the next step up.
Block by User-Agent
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} AhrefsBot [NC]
RewriteRule .* - [F,L]
This returns a 403 Forbidden response to any request with “AhrefsBot” in the user-agent string. The [NC] flag makes the match case-insensitive.
To block both Ahrefs crawlers in one rule:
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (AhrefsBot|AhrefsSiteAudit) [NC]
RewriteRule .* - [F,L]
Block by IP Range
For stricter enforcement, block the actual IP ranges Ahrefs crawls from. This works even if someone spoofs the user-agent:
<RequireAll>
Require all granted
Require not ip 5.39.1.224/27
Require not ip 51.89.129.0/24
Require not ip 51.161.37.0/24
Require not ip 51.195.183.0/24
Require not ip 15.235.27.0/24
Require not ip 15.235.96.0/24
Require not ip 15.235.98.0/24
Require not ip 176.31.139.0/27
</RequireAll>
Pull the complete, current list from Ahrefs’ help center before deploying — they occasionally add or adjust ranges.
Method 3: Block via Nginx Server Config
If your server runs Nginx instead of Apache, you won’t have an .htaccess file. Here’s how to achieve the same result.
Block by User-Agent
if ($http_user_agent ~* "AhrefsBot") {
return 403;
}
Place this inside your server block in the Nginx config file.
To block multiple SEO crawlers at once:
if ($http_user_agent ~* "(AhrefsBot|AhrefsSiteAudit|SemrushBot|MJ12bot)") {
return 403;
}
Block by IP Range (More Reliable)
Use Nginx’s geo module for IP-based blocking:
geo $block_ahrefs {
default 0;
5.39.1.224/27 1;
51.89.129.0/24 1;
51.161.37.0/24 1;
51.195.183.0/24 1;
15.235.27.0/24 1;
176.31.139.0/27 1;
}
server {
if ($block_ahrefs) {
return 403;
}
}
After editing, reload Nginx for changes to take effect:
sudo nginx -t
sudo systemctl reload nginx
Method 4: Block via Cloudflare or CDN-Level WAF
If your site uses Cloudflare (or a similar CDN with WAF capabilities), you can block AhrefsBot before traffic even reaches your origin server. This is the most resource-efficient approach because your server never has to process the request at all.
Create a Custom Firewall Rule
In Cloudflare:
- Go to Security > WAF > Custom rules
- Click Create rule
- Name it something clear — “Block AhrefsBot”
- In the Expression Editor, enter:
(http.user_agent contains "AhrefsBot") - Set the action to Block
- Deploy
To block by IP ranges instead (stronger enforcement against spoofed user-agents), use the expression:
(ip.src in {5.39.1.224/27 51.89.129.0/24 51.161.37.0/24 51.195.183.0/24})
A Note on Bot Fight Mode
Cloudflare’s Bot Fight Mode may automatically challenge or block AhrefsBot depending on your settings. However, since Ahrefs is classified as a verified bot in Cloudflare’s system, Bot Fight Mode typically won’t interfere with it unless you’ve set up additional custom rules. Check your Cloudflare analytics under Security > Bots to see how AhrefsBot traffic is being handled.
Method 5: Use Ahrefs Webmaster Tools to Control Crawl Rate
This one doesn’t involve blocking at all — and many site owners don’t know it exists.
If you verify your site in Ahrefs Webmaster Tools (free for any site owner), you get access to a crawl rate adjustment dashboard. This lets you set a custom crawl frequency that matches your server capacity, giving you more precise control than a robots.txt crawl-delay directive.
Ahrefs Webmaster Tools also participates in the IndexNow protocol through Yep.com. When you update content, you can notify Ahrefs directly rather than waiting for the bot to discover changes on its own schedule. This means you can run a slower overall crawl rate without sacrificing data freshness on the pages that matter most.
To set this up:
- Go to ahrefs.com/webmaster-tools
- Verify your site ownership (DNS, HTML file, or meta tag)
- Navigate to the crawl settings for your verified project
- Adjust the crawl speed to your preference
This is the ideal solution for site owners who want accurate data in Ahrefs but need to manage server load.
WordPress-Specific Options
If you’re running WordPress and not comfortable editing server config files directly, a few approaches work well.
Edit robots.txt through Yoast SEO or Rank Math: Both plugins let you edit your robots.txt file from the WordPress dashboard. Go to Yoast > Tools > File Editor, or Rank Math > General Settings > Edit robots.txt, and add your AhrefsBot directives there.
Use a bot-blocking plugin: The Htaccess by BestWebSoft plugin includes a User-Agent Blocking field where you can enter bot names (AhrefsBot, SemrushBot, MJ12bot) one per line. The plugin generates the .htaccess rules automatically — no manual file editing required.
Managed WordPress hosts: Some managed hosting providers (Cloudways, Kinsta, WP Engine) include server-level bot management in their dashboards. Check your host’s documentation before adding redundant rules.
Why Your AhrefsBot Block Might Not Be Working
You’ve added the rules, but your server logs still show AhrefsBot requests. Here’s what’s usually going on.
Robots.txt Changes Aren’t Instant
AhrefsBot doesn’t check your robots.txt file with every single request. It fetches and caches the file periodically, then follows those rules until the next check. After updating your robots.txt, it may take anywhere from hours to a few days before AhrefsBot picks up the changes on its next scheduled crawl.
You Blocked AhrefsBot but Not AhrefsSiteAudit
These are two different crawlers with two different user-agent strings. If you only added a rule for User-agent: AhrefsBot, AhrefsSiteAudit will continue crawling normally. If you’re seeing continued Ahrefs traffic after blocking the main bot, check whether the user-agent in your logs says AhrefsSiteAudit instead.
Crawl-Delay Looks Broken in Raw Logs
When AhrefsBot renders a page, it fetches the HTML and then pulls associated assets — stylesheets, scripts, images. The crawl-delay applies to HTML page requests, not asset requests. So your logs might show a cluster of 10-15 requests within a second, but only one of those is an actual page crawl. The rest are resource fetches. This is documented behavior, not a violation of your crawl-delay directive.
Syntax Errors in Robots.txt
A missing colon after User-agent, an extra space before Disallow, or a misspelled bot name will cause the directive to be silently ignored. Ahrefs warns that their bots cannot parse malformed robots.txt files. Use a robots.txt validator (Google Search Console has one, or use an online tool) to check for syntax issues.
Someone Is Spoofing the AhrefsBot User-Agent
If you’ve confirmed there are no syntax errors and the crawl is coming from IPs outside Ahrefs’ published ranges, you’re not dealing with the real AhrefsBot. A scraper or bad bot is using the Ahrefs user-agent as camouflage. In this case, user-agent blocking won’t help — you need IP-based blocking or WAF rules.
The Trade-Offs: What You Lose When You Block AhrefsBot
Blocking works. But it comes with costs that are worth understanding before you commit.
Your Ahrefs Data Goes Stale
The moment AhrefsBot stops crawling your site, your domain’s data in Ahrefs freezes. Backlink counts stop updating, new links won’t appear, keyword ranking data becomes less accurate, and anyone using Ahrefs to evaluate your site — potential link partners, agencies, investors doing due diligence — will see incomplete information.
Your site doesn’t disappear from Ahrefs entirely. The platform still picks up backlinks from the linking side (when other sites that do allow AhrefsBot link to you, those links still get recorded). But the data becomes progressively less reliable over time.
Site Audit Stops Working
If you block AhrefsSiteAudit along with AhrefsBot, you lose the ability to run free technical audits through Ahrefs Webmaster Tools. For teams that rely on Ahrefs for crawl error detection, broken link monitoring, or Core Web Vitals tracking, this is a meaningful loss.
You Only Block One Tool
Blocking AhrefsBot doesn’t block SemrushBot, MJ12bot (Majestic), DotBot (Moz), or any of the other SEO crawlers that collect similar data. If competitive privacy is your goal, you’d need to block multiple crawlers to make a real difference. And even then, tools have other data sources — clickstream data, SERP scraping, Chrome extension data — that don’t depend on crawling your site directly.
Your Site Disappears from Yep.com
AhrefsBot currently powers the index for Yep.com, the search engine Ahrefs built. Blocking the bot means your pages won’t appear in Yep search results. For most sites, Yep traffic is negligible today. But it’s worth knowing the connection exists.
When Blocking Actually Makes Strategic Sense
Full AhrefsBot blocking makes sense in a limited set of scenarios:
Small sites on tight server budgets where even moderate bot traffic causes measurable slowdowns for real users. If you’re on a $5/month shared hosting plan and AhrefsBot crawl bursts are spiking your TTFB, blocking is a reasonable trade-off.
Private or internal projects — staging environments, internal tools, client portals — that should never appear in any third-party database.
PBN operators and affiliate SEO practitioners who actively want to prevent competitors from mapping their link networks. This is the most common deliberate use case in competitive SEO.
Organizations with strict security policies that prohibit all non-essential automated access by default.
For most public-facing websites, the better move is throttling (crawl-delay) combined with path-specific blocking. You keep your Ahrefs data fresh, protect sensitive directories, and reduce server load — without the downsides of a full block.
AhrefsBot in the Context of 2026 Bot Management
Blocking AhrefsBot in 2026 is one piece of a much bigger bot management picture. The landscape has shifted dramatically with the arrival of AI training crawlers.
GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended, CCBot (Common Crawl), and Meta-ExternalAgent now generate substantial traffic to most websites. According to Cloudflare’s Q1 2026 robots.txt analysis, GPTBot is the most blocked AI crawler, and the total volume of AI crawler traffic has grown significantly compared to traditional SEO bots.
The important distinction for site owners: SEO crawlers like AhrefsBot index your content and make it discoverable (in Ahrefs, in Yep.com). AI training crawlers ingest your content to build language models — they take but don’t send traffic back. The crawl-to-referral ratio tells the story: AI training bots consume far more resources per referral visit they generate compared to SEO crawlers.
If you’re updating your robots.txt to block AhrefsBot, it’s worth auditing your bot rules more broadly. Many site owners in 2026 are adopting a split strategy: allow SEO crawlers (or throttle them), allow AI search retrieval bots (like OAI-SearchBot and PerplexityBot that drive referral traffic), but block AI training crawlers that only harvest content without returning visitors.
Your robots.txt is no longer a file you set once and forget. It’s a living access policy that needs periodic review.
Frequently Asked Questions
Does blocking AhrefsBot hurt my Google rankings?
No. AhrefsBot is completely separate from Googlebot. Blocking it has zero direct impact on how Google crawls, indexes, or ranks your pages. The indirect risk is that you lose access to Ahrefs’ SEO data for your own site, which could make it harder to monitor and improve your SEO performance over time.
Can I block AhrefsBot but still use Ahrefs to analyze my own site?
Partially. If you block AhrefsBot but allow AhrefsSiteAudit, you can still run Site Audit on your verified domains through Ahrefs Webmaster Tools. However, your Site Explorer data (backlinks, organic keywords, traffic estimates) will become stale because that data comes from the main AhrefsBot crawler.
How long does it take for AhrefsBot to stop crawling after I update robots.txt?
There’s no fixed timeline. AhrefsBot fetches and caches your robots.txt file periodically. Changes typically take effect within a few hours to a few days, depending on how frequently the bot is scheduled to revisit your domain. For immediate enforcement, use server-level or firewall-level blocking instead.
Should I block SemrushBot and MJ12bot at the same time?
That depends on your goals. If competitive privacy is the reason, blocking only AhrefsBot while leaving SemrushBot and MJ12bot open doesn’t accomplish much — your competitors can just use those tools instead. If server load is the concern, prioritize blocking the bots that hit your site hardest, which you can identify by reviewing your server access logs.
Is blocking AhrefsBot by user-agent enough, or do I need IP blocking too?
User-agent blocking works for the real AhrefsBot because it honestly identifies itself. But user-agent strings can be spoofed by scrapers and bad bots. If you need enforcement against spoofed requests — or if you’re in an environment where you can’t rely on voluntary compliance — IP-range blocking is the stronger option. Ahrefs publishes their full IP list specifically for this purpose.
What’s the difference between blocking via robots.txt and blocking via .htaccess or firewall?
Robots.txt is a voluntary protocol. It asks bots to stay away, and well-behaved bots (including AhrefsBot) comply. But it consumes no server resources only if the bot actually reads and obeys the file. Server-level blocking (.htaccess, Nginx config) and firewall blocking (Cloudflare WAF) reject the connection at a lower level — the bot’s request is denied before your site processes it, saving server resources. The trade-off is that server and firewall rules require more technical setup and maintenance, especially if Ahrefs updates their IP ranges.
