Rethinking Your Traffic Metrics

5 reasons why higher traffic isn’t always good news for your WordPress site

The 300% AI Bot Surge is Eating Your Resources

  • The Reality: AI bot traffic has skyrocketed by 300% in a single year. Kinsta’s recent analysis of 10 billion requests shows that a staggering 1 in 31 web visits is now an AI bot.
  • The Traffic: Most of this automated scraping isn’t human. These bots are consuming your server’s power without driving a single real visitor, inquiry, or conversion to your site.
  • Not Just Noise: From an infrastructure perspective, every request is real work. At scale, a massive spike in traffic stops being a vanity marketing metric and becomes a highly expensive resource problem.

Is your site optimised, but still sluggish?

It might be time to look beyond human visitors. Since all of our WordPress sites are hosted with our partner, Kinsta, we have a front-row seat to how these infrastructure demands impact live performance. Let’s look at where the system actually breaks down.

A simple robots.txt file just doesn’t cut it anymore

Times have changed: Managing a website has shifted. Before the AI boom, handling crawler traffic was straightforward, you just threw a few rules into a robots.txt file to tell bots which pages to avoid.

  • Completely unmanageable: With the birth of modern AI, relying on a few basic rules is no longer enough to protect a site.
  • A new industry standard: The sheer volume and aggressive nature of these new scrapers mean that manual rules can’t keep up. It’s a change in website management that will likely be adopted across the board by all major hosting providers as time goes on.

This infographic from Kinsta highlights the drastic change in AI bot traffic.

Crawlers are getting stuck in expensive loops

Fewer requests, faster loads: AI bots aren’t just reading your content; they are faster and significantly less disciplined than traditional search engines. Because they crawl aggressively to scrape as much data as possible, they frequently get trapped in infinite query-string loops on dynamic pages (learn more about query-strings here).

  1. The URL Trap: To a human, a product or landing page with a few search filters or sorting parameters looks like the same page. To an AI bot following every single link, each filtered variation looks like a completely new URL.
  2. Dynamic Endpoints: Instead of just indexing static text, these bots hammer uncacheable endpoints like search queries, filtered categories, and checkout links.
  3. Exhausting PHP Workers: Because these dynamic requests bypass standard caching layers, they force the server to execute complex database queries over and over again, blocking legitimate human users from accessing the site.

Taking control with smart bot protection

  • Targeted control, not blind blocking:
    • Block resource-draining AI: You shouldn’t blindly block everything, as you risk hurting your search visibility. The goal is to specifically block or challenge resource draining AI crawlers while ensuring verified search bots (like Google) get through automatically.
    • Environment isolation: Protect different environments separately so your staging and live sites are guarded based on their unique traffic needs.

Our collaborative edge via Kinsta

  • Free built-In defense: Because Kinsta is our hosting partner, all of our WordPress sites now have access to a brand-new, free Bot Protection feature natively in MyKinsta.
  • Formed by real experience: We’ve been proactive about this shift. After noticing severe crawler-driven performance issues on a client’s complex site right after launch, our team worked closely with Kinsta’s support to report the behaviour. Not long after, this dedicated feature emerged.
  • A unique safeguard: This feature is a massive advantage in hosting systems right now, allowing us to proactively manage and challenge automated traffic before it ever slows down your site.
Latest Posts:
Subscribe to Newsletter
Share:

Post Categories

More Articles…