How to stop excessive bot traffic and AI scrapers
Massive volumes of automated scrapers overwhelm sites with tens of thousands of public pages, driving hosting costs up significantly. This overwhelming bot traffic ruins the reliability of web data and forces owners to look for specialized blocking services to regain control.
Core Friction Pattern
Analysis of 22 peer accounts across online developer and community forums reveals that “How to stop excessive bot traffic and AI scrapers” repeatedly surfaces due to edge-case mismatches between standard defaults and real-world environments. Standard troubleshooting frequently falls short, compelling practitioners to develop custom workarounds.
+49 alternative community-tested workarounds cataloged below.
What the numbers show
Counted from the threads quoted below. How we count
- People have been saying this in most months since September 2026.
- This comes up 98.2× more often in r/Wordpress than across the site as a whole.
What people tried
Every workaround mentioned in the threads below. We haven’t tested any of them — and nobody here is claiming they worked.
Found another way around this?
Share your steps and trade-offs so fellow engineers don't have to reinvent the wheel. No account required.
How to stop excessive bot traffic and AI scrapers
- 1using Vercel and Cloudflare to block trafficDid this help you?
- 2putting the app behind an auth gate with cloudflare turnstileDid this help you?
- 3switching to hcaptchaDid this help you?
- 4using truesign.ai to block crawlersDid this help you?
- 5Introducing a WAF rule in Cloudflare to block visits via wildcard patternsDid this help you?
- 6Adding nginx rules to return a 404 status on pagination URLsDid this help you?
- 7Disabling archives and feeds from being indexed or crawlable using SEO plugins like Yoast or Rank MathDid this help you?
- 8Checking access logs by user agent to analyze bot behaviorDid this help you?
- 9Locking down IT access and trying to secure company dataDid this help you?
- 10Pre-anonymizing data before letting AI systems process itDid this help you?
- 11Manually applying the human or bot session filter in the dashboardDid this help you?
- 12Using Cloudflare WAF and custom WAF rules to block trafficDid this help you?
- 13Setting up rules and cache to manage server loadDid this help you?
- 14blocking entire countries or geographical areas using CloudflareDid this help you?
- 15asking the web host to block traffic from certain regionsDid this help you?
- 16setting up Cloudflare security rules to block common threatsDid this help you?
- 17Setting up Cloudflare with Bot Fight Mode / Block AI Bots enabledDid this help you?
- 18Upgrading hosting plans or increasing PHP workersDid this help you?
- 19Migrating to better hosting companies (e.g., Kinsta, GreenGeeks)Did this help you?
- 20Setting rate limiting or Crawl-delay in robots.txt / server configDid this help you?
- 21Blocking specific IP ranges or countries (Geo-blocking)Did this help you?
- 22Asking support to temporarily filter abusive traffic or apply rate limiting at the server levelDid this help you?
- 23Outright dropping the single IP address instead of trying to overcomplicate rate limiting.Did this help you?
- 24Using fail2ban to scan logs and automate drops.Did this help you?
- 25Using ipset to block known bogus IPs.Did this help you?
- 26Checking the IP's geolocation and blocking it if it's outside the target audience country.Did this help you?
- 27Leaving 'Under Attack' mode on permanently, despite it showing a loading screen to real users.Did this help you?
- 28Creating custom Web Application Firewall (WAF) rules to block or challenge specific IPs, countries, or paths.Did this help you?
- 29Caching HTML at the CDN level so bot requests never reach the origin server running PHP.Did this help you?
- 30Adding specific search engine bot blocks directly to the .htaccess file.Did this help you?
- 31Disabling XML-RPC to prevent brute force attempts.Did this help you?
- 32Using bot throttling plugins to temporarily block scraping behavior.Did this help you?
- 33Setting up Web Application Firewall (WAF) rules to block specific crawler user agents.Did this help you?
- 34Disabling plugins that inject sitewide JavaScript to reduce main thread blocking.Did this help you?
- 35Checking server access logs to identify the exact behavior of the bots.Did this help you?
- 36Looking into adding a txt file (robots.txt) to block crawler access.Did this help you?
- 37Migrating DNS and protection services to specialized platforms like Cloudflare.Did this help you?
- 38Implementing geo-blocking, IP bans, and strict access controls via tools like Netbird.Did this help you?
- 39Reissuing compromised email accounts to affected users.Did this help you?
- 40Setting up Cloudflare WAF or edge caching to block bad traffic before it reaches the origin serverDid this help you?
- 41Installing security plugins like Wordfence, Bottle, or Fail2ban to block or throttle malicious IPsDid this help you?
- 42Implementing object caching via Redis or Memcached to reduce database loadDid this help you?
- 43Blocking scraper IP addresses in CloudflareDid this help you?
- 44Adding Cloudflare WAF or security rules to restrict access to REST API endpointsDid this help you?
- 45Filing DMCA copyright removal requests with Google and the copier's hosting providerDid this help you?
- 46Setting RSS feeds to show excerpts only rather than full postsDid this help you?
- 47Using rate limiting and bot mitigation toolsDid this help you?
- 48adding robots.txt rules blocking bots from specific language foldersDid this help you?
- 49implementing rate limits for user agents that ignore robots.txtDid this help you?
- 50disabling the translation pluginDid this help you?
In their words
Unedited, grouped by where they were said, most upvoted first within each place, each linked to the thread it came from.
“The last few months we have seen a huge uptick in ai attacks combined with what looks like human intervention.. they automated creating accounts, we added a rate limit. They altered their code to stay under our rate limit. We increased the difficulty in our captcha they seemed to crack it anyway.”source ↗
“I looked in my logs and noticed one bot (with the same IP and same user agent) has been making requests to my server at the rate of one request per second for about 5 minutes.”source ↗
“I thought about using the recent module of iptables but I'm not sure if that is the best because if I pick too low of numbers then I block legit users and if I pick too high of numbers then my system might go to a crawl.”source ↗
“I'm doing what I can to secuure my company's data, but I feel at this point it is going to be pointless because it feels like legislation is going to change to make trademark/copyright not a real enforceable thing.”source ↗
“Banning an ip may be an option but what if a legitimate user also uses that IP as well at a later time?”source ↗
“I just spent the weekend updating CloudFlare rules to stop huge volumes of site scraper bots, including going full Bot fight mode. Best I achieved was a 2 hour lull before they moved to a new host.”source ↗
“these last months, maybe the last year, the sessions on my store went crazy : 100-200 simultaneous sessions non-stop, day and night.”source ↗
“My conversion rate in August : 0.66% with the bots, 1.57% without.”source ↗
“Since it's my business I of course need to link my shop, but f*** AI slop and them making use of my stuff.”source ↗
“Is there a way I can block them off Shopify?”source ↗
“I'm not a coder or anything, saw a short Info of a txt that can be added to block the "friendly" bots?”source ↗
“My concern is the contact information.”source ↗
“If someone can simply scrape the pages and collect both the listings and all of their contact details, they could essentially recreate a large part of the directory elsewhere.”source ↗
“What I’m trying to understand is what practical measures experienced WordPress developers use to make large-scale scraping of contact information significantly harder without making the site frustrating for legitimate users or hurting SEO.”source ↗
“I’m currently quite lost on the best solution for this, so I’d appreciate practical advice from people who have dealt with this problem.”source ↗
“I noticed Ive been getting a high load of Traffic from China, its my top traffic source and im located in the USA. I read that I should block China traffic, but how do I do that on Wordpress? Also is it safe for SEO to block traffic from China?”source ↗
“My website crashes every other hour for an hour so down like 50% of the time UNLESS I have under attack mode on in Cloudflare which blocks bots. If I turn it off and the website crashes, it immediately comes back online a minute after turning it on.”source ↗
“Are these number normal? Is there a way to fix this without having under attack mode on 24/7? I feel like it annoys the viewers with that 5 second loading screen and also potentially effect google analytics.”source ↗
“When I try to see the stats for my site I often get the "database connection error" message which would seem to indicate that the db engine is overwhelmed by the requests, forcing me to restart the mysql and apache daemons.”source ↗
“i have a wordpress blog that contains some content and fields, i noticed that there is a competitor website that is running a script on my website and its copying its content 100% and the issue is it is ranking above me”source ↗
“Woocommerce site on godaddy managed wp throwing constant 503s even with plugins disabled”source ↗
“In access log it shows dozens of different IPs from all over the world per second. Possible bot problem?”source ↗
“suddenly started seeing a lot of hits to the translated version of our site that Weglot produces. Like over 7,000 in under 30 minutes. I couldn't find any documentation on the Weglot site for settings I could change to fix the issue so I'm hoping to crowdsource some other ideas or things to look into.”source ↗
“We're experiencing a lot of bot traffic going to those addresses and are not sure why.”source ↗
“I put the Cloudflare rule in place this morning (2-3 hours before writing this) and according to Cloudflare the rule has been triggered 8,430 times since I created it!”source ↗
“My WP site for my self-published books has very low traffic counts, but these last two days, it's seeing hits from all over China.”source ↗
“probably bots scraping content, happens to my art portfolio all the time”source ↗
“Even the dmca won't help because he is creating new websites and domains all run the same script.”source ↗
“Really? I’ve been trying to sort my server with rules and cache.”source ↗
“I had this problem. It was low-rent AI bots hammering my shop. When bot-hammering uses up your server quotas GoDaddy’s throws 503s to your audience.”source ↗
“Had similar resource usage on my server with a very low traffic site. Moved it to a new server. Monitored requests with cloudflare and Google. Noticed a metric ton of search requests for non-existent locations under the domain.”source ↗
“I used the block anthropic function and set up all the waf rules so the site isn't going offline anymore. However after doing that everything slowed down and it says 10 duplicate queries and 10 slow queries and 40 second wait time with query monitor.”source ↗
“Blocking anthropic suddenly triggered a huge mess and I don't know why. Also the bot hits were 700k before blocking now they are 2 million. Though in the events pane it says blocked but they are still served data of 20gb.”source ↗
“I'm battling this problem a content site I run and despite spending over $5k on attempting to solve the problem, its still cat and mouse.”source ↗
“I have sent multiple DMCA reports to the hosting provider, cloudflare and the domain registrar waiting for their response.”source ↗
“tbh this is first time dealing with this stuff and im afraid that google dmca need a lawyer or something should i be hiring?”source ↗
“We host a web app which has around 50,000 statically generated public pages, and the amount of bot traffic is insane.”source ↗
“I’ve never seen anything like it before, our hosting costs are up like crazy, and besides all of this, I can’t trust anything I read or see on this web anymore.”source ↗
“I occasionally suffer this same problem when happy crawlers find my site and use truesign.ai to block them, so far successfully.”source ↗
Where this came up
People with this problem also raised
Related community dilemmas clustered by discussion consensus
Why are automated bot attacks and hacking attempts suddenly surging?
Sites on platforms like WordPress and Shopify are being overwhelmed by relentless daily bot attacks and scripts like wp2shell that exploit unmaintained pages and checkout links. This constant barrage prevents small business owners from operating normally, forcing them to spend money on services like Cloudflare while leaving them vulnerable to site cloning and identity theft.
How to stop an email spam bomb attacking a user inbox
An ongoing email spam bomb floods a user's inbox with hundreds of random newsletter and phishing messages a day in multiple languages. This relentless wave buries legitimate communications and overwhelms standard inbox management, leaving administrators searching for effective filtering and mitigation strategies.
Why does a WordPress site need a plugin for everything?
Too many plugins bog down site performance and turn basic features—like cloning posts—into bloated dependencies. This makes routine site maintenance exhausting and prevents pages from loading quickly.
How to stop cache warming from crashing the server
Running scheduled curl scripts across hundreds of URLs every thirty minutes creates too much parallel traffic as a site grows. This heavy request load hammers the origin server and can easily kill site performance when the sitemap expands.
Why is my website traffic suddenly spiking with no sales?
Sudden traffic surges with flat conversion rates and zero browsing activity usually indicate automated bot traffic. This phantom traffic distorts analytics and wastes server resources without generating actual revenue, leaving store owners unable to tell if real customers are visiting or how to filter the noise.
Why is Intelligo audio causing high CPU on domain controllers?
An Intelligo audio component installed on client workstations floods domain controllers with excessive lookup requests, driving lsass.exe CPU usage above 90% during business hours. Standard troubleshooting like adjusting heartbeat settings or server configurations fails to stop the traffic, leaving servers continuously overloaded.