If you run a website, you have probably seen visits from SEO crawlers in your server logs. Tools like Ahrefs, Moz, and Majestic scan the web to build backlink databases, estimate authority metrics, and power competitive research platforms. While these crawlers can be useful for SEO analysis, some site owners prefer to block them to reduce server load, limit third-party data collection, or keep backlink information less visible to competitors.
TLDR: You can block Ahrefs, Moz, and Majestic crawlers using robots.txt, web server rules, or firewall/CDN settings. The easiest method is adding disallow rules for known bots such as AhrefsBot, DotBot, and MJ12bot. For stronger blocking, combine robots.txt with Apache, Nginx, or Cloudflare rules. Always test the setup afterward to make sure legitimate search engines like Googlebot are not blocked by mistake.
Why Block SEO Crawlers?
Ahrefs, Moz, and Majestic are not malicious by default. They are commercial SEO platforms that crawl websites to collect data about links, pages, anchor text, redirects, and other signals. However, not every website owner wants that activity.
Common reasons for blocking these crawlers include:
- Reducing server load: Large crawlers can request many pages, especially on big websites with archives, filters, or internal search pages.
- Limiting competitor research: SEO tools may reveal your backlink profile, content structure, and site changes to competitors.
- Protecting private or low-value areas: Even if pages are public, you may not want analytics tools crawling staging-style sections, tag pages, or parameter-heavy URLs.
- Cleaner log files: Blocking unnecessary bot traffic can make server logs and analytics easier to interpret.
Before blocking them, remember that these tools may also help you monitor your own SEO performance. If you rely on Ahrefs, Moz, or Majestic for audits, backlink tracking, or reporting, blocking their crawlers may reduce the accuracy of your data.
Method 1: Block Crawlers with robots.txt
The simplest and most common approach is to use the robots.txt file. This file sits at the root of your domain, usually at:
https://example.com/robots.txtIt tells well-behaved crawlers which areas of your website they are allowed or not allowed to access. Ahrefs, Moz, and Majestic generally respect robots.txt instructions, making this a good first step.
Block Ahrefs
Ahrefs uses the user agent AhrefsBot. To block it from your entire website, add:
User-agent: AhrefsBot
Disallow: /Block Moz
Moz has used crawlers such as DotBot. To block Moz’s crawler, add:
User-agent: DotBot
Disallow: /Block Majestic
Majestic’s crawler is commonly identified as MJ12bot. To block it, add:
User-agent: MJ12bot
Disallow: /Combined robots.txt Example
You can combine all three into one file:
User-agent: AhrefsBot
Disallow: /
User-agent: DotBot
Disallow: /
User-agent: MJ12bot
Disallow: /This tells each named crawler not to access any part of your site. It does not block Google, Bing, or other search engines unless you create rules for them too.
Method 2: Block Crawlers at the Server Level
Robots.txt is polite; server-level blocking is stronger. If a crawler ignores robots.txt, or if you want to prevent access before pages are served, you can block requests based on the User-Agent header.
Apache .htaccess Rules
If your site runs on Apache and allows .htaccess rules, you can add:
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} AhrefsBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} DotBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} MJ12bot [NC]
RewriteRule .* - [F,L]This returns a forbidden response to requests from those crawlers. The [NC] flag means the match is case-insensitive, and [F,L] blocks the request immediately.
Nginx Rules
For Nginx, you can add a condition inside your server block:
if ($http_user_agent ~* "(AhrefsBot|DotBot|MJ12bot)") {
return 403;
}After editing your Nginx configuration, test it and reload the service. A typical workflow is:
nginx -t
systemctl reload nginxServer-level rules are particularly useful for high-traffic sites because they stop unwanted bots before your application generates full pages.
Method 3: Use a CDN or Firewall
If your website uses a CDN or web application firewall, such as Cloudflare or another security layer, you can often block crawlers without touching server files. Look for firewall rules, bot controls, or request filtering options.
A typical rule might say:
- If the request user agent contains AhrefsBot, DotBot, or MJ12bot
- Then block, challenge, or rate limit the request
Using a CDN has a major advantage: unwanted requests may be stopped before they ever reach your origin server. This can reduce bandwidth, improve performance, and make bot management easier across multiple domains.
Should You Block by IP Address?
Blocking by IP address is possible, but it is usually less reliable. Large crawlers may use many IP ranges, and those ranges can change over time. You may also accidentally block shared infrastructure or unrelated traffic.
User-agent blocking is easier to maintain, but it has one weakness: user agents can be spoofed. A crawler can pretend to be something else. For reputable SEO tools, user-agent rules are usually enough; for aggressive scraping or abuse, combine user-agent filtering with rate limits, behavioral detection, and firewall rules.
How to Check Whether the Block Is Working
After setting up your rules, verify that they work. You can do this in several ways:
- Check robots.txt: Visit your robots.txt file in a browser and confirm the rules are visible.
- Review access logs: Search your server logs for AhrefsBot, DotBot, and MJ12bot.
- Test with curl: Simulate a bot request using a custom user agent.
For example:
curl -A "AhrefsBot" -I https://example.com/If server-level blocking is active, you may see a 403 Forbidden response. If you are only using robots.txt, the server may still return a normal page, because robots.txt relies on the crawler choosing not to crawl.
Important Mistakes to Avoid
Blocking SEO crawlers is simple, but a small mistake can cause bigger problems. Be careful not to block all bots accidentally. For example, this rule would block every crawler that follows robots.txt:
User-agent: *
Disallow: /That may remove your site from search engine results if Googlebot and Bingbot obey it. Only use broad blocking if you truly want the entire site excluded from search engines and other crawlers.
Also, avoid assuming that blocking Ahrefs, Moz, and Majestic will hide your website completely. Competitors can still use other tools, search engines, cached data, browser extensions, manual research, or their own crawlers. Blocking these bots simply limits data collection by those specific services.
Final Thoughts
Blocking Ahrefs, Moz, and Majestic crawlers can be a sensible move if you want to reduce bot traffic, protect competitive intelligence, or keep your server resources focused on real users and major search engines. Start with robots.txt for a clean and crawler-friendly approach, then add server or firewall rules if you need stricter enforcement.
The best setup is usually layered: use robots.txt to state your preference, server rules to enforce it, and log monitoring to confirm the results. Done carefully, you can block commercial SEO crawlers without harming your visibility in Google, Bing, or other search platforms that matter to your audience.
