How to Fix HTTP 406 Errors for AI Crawlers on HostGator and Wordfence

If you run an automated site audit or check your web server logs and discover an HTTP 406 Not Acceptable status code blocking crawlers like CCBot, GPTBot, or ClaudeBot, your server security layer is rejecting automated requests before they can reach your content.

For WordPress sites hosted on HostGator shared hosting running Wordfence, this issue almost always stems from either server-level security modules (ModSecurity) or WordPress firewall rules.

Here is a step-by-step guide to understanding why this happens and how to resolve it.

What is an HTTP 406 Error?

An HTTP 406 Not Acceptable response means the web server receives a request containing headers (such as User-Agent or Accept) that fail to meet strict server security criteria.

When AI crawlers or audit tools visit your site, they often use simplified headers or non-standard user agents. Security filters interpret these unusual signatures as potentially malicious bot traffic and actively reject the connection with a 406 status.

Step 1: Check Wordfence Firewall & Live Traffic

Wordfence monitors incoming traffic for unknown user agents, missing browser headers, or high request rates.

1. Review Live Traffic Logs:
   • Log into your WordPress Dashboard.
   • Navigate to Wordfence > Live Traffic.
   • Filter by Blocked or search for recent entries involving crawler user agents (e.g., CCBot).
   • If Wordfence is responsible, it will display the specific rule triggered (e.g., False Googlebot, Rate Limit Exceeded, or Header Inspection Failure).
2. Adjust Firewall Settings:
   • Navigate to Wordfence > Firewall > Manage Firewall.
   • Review your Whitelisted URLs and Rules to ensure standard AI training and research crawlers are allowed.
   • Consider setting Wordfence to Learning Mode temporarily under Basic Firewall Options to let the firewall adapt to automated testing requests.

Step 2: Disable or Configure HostGator’s ModSecurity (Most Common Cause)

On HostGator shared hosting accounts, Apache ModSecurity runs in front of WordPress. ModSecurity frequently intercepts non-browser traffic and issues a 406 Not Acceptable error before the request ever reaches Wordfence.

1. Access cPanel:
   • Log into your HostGator cPanel account.
2. Locate ModSecurity:
   • Scroll to the Security section and click ModSecurity.
3. Test by Disabling ModSecurity:
   • Turn ModSecurity Off for your specific domain.
   • Re-run your site audit or crawler check.
4. Resolution:
   • If turning ModSecurity off fixes the 406 error, ModSecurity is the root cause.
   • Recommendation: If you prefer to keep ModSecurity enabled for security, contact HostGator Support via Live Chat and ask them to whitelist the specific ModSecurity rules triggering 406 responses on non-browser User-Agent headers.

Step 3: Inspect .htaccess Rules

Security plugins or past custom configurations may have inserted header inspection directives into your .htaccess file.

1. Open your .htaccess file using cPanel File Manager or FTP.
2. Look for rules containing RewriteCond %{HTTP_USER_AGENT} or Header set directives that block empty or non-standard user agents.
3. If necessary, test with the clean default WordPress .htaccess configuration:

# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>
# END WordPress

Summary Checklist

• [ ] Check Wordfence Live Traffic for blocked crawler requests.
• [ ] Temporarily disable ModSecurity in HostGator cPanel to isolate the issue.
• [ ] Contact HostGator support to whitelist ModSecurity rule IDs if necessary.
• [ ] Verify .htaccess does not contain custom user-agent blocks.

Resolving these firewall blocks ensures your site remains accessible to legitimate search engines, AI models, and automated auditing tools.

Background

I had to do this because the AI Visibility audit tool by Stephen Burns brought up a few 406 errors on my sites, which are on Hostgator and run Wordfence. I could not see any issues in Wordfence, so the block must be with the shared hosting platform.