How to Properly Restrict Bingbot from Crawling Unnecessary Links

Posted by Adem on June 12, 2025

When it comes to optimizing website performance and crawl budget, search engine bots play a key role. Bingbot, one of the largest search engine crawlers, should not be overlooked; especially if you're noticing it crawling low-value URLs on your site or causing issues with page speed!

In this article, we'll walk through how to restrict Bingbot effectively using tools like the robots.txt file and crawl delay directives.

Why Restrict Bingbot?

While search engines aim to crawl and index useful content, they can sometimes get stuck in loops of irrelevant or redundant URLs - especially on dynamic or eCommerce sites. This can:

  • Waste your crawl budget
  • Strain your server resources - Causing slow page load times
  • Slow down the indexing of the pages you deem to be important

Identify Unnecessary URLs

Start by auditing your server access logs or using tools like Bing Webmaster Tools or Screaming Frog to identify:

  • URLs with query parameters (e.g. "?color=blue&size=xl")
  • Login and/or irrelevant pages (/admin, /cart, etc.)
  • Staging, development or testing environments
  • Campaign URLs

Once you've listed the URLs or patterns, you can restrict them.

Use robots.txt to Disallow Specific Patterns

The most straightforward way to restrict Bingbot is via the robots.txt file.

User-agent: Bingbot
Disallow: /search
Disallow: /cart
Disallow: /checkout
Disallow: /admin

This tells Bingbot not to crawl any URLs matching these patterns.

Add Crawl-Delay to Throttle Bingbot Requests

Bingbot respects the Crawl-delay directive, which tells it how many seconds to wait between requests. This can help reduce server load without completely blocking the bot.

User-agent: Bingbot
Crawl-delay: 10

This instructs Bingbot to wait 10 seconds between each request.

Note: Googlebot does not respect Crawl-delay, but Bingbot and other (legitimate) crawlers do. You can limit the number of requests that Googlebot makes in your Google Search Console.

Consider Using robots Meta Tags for Page-Level Control

For finer control over indexation, use the robots tag on individual pages:

<meta name="robots" content="noindex,nofollow">

This is useful when you can’t block the URL path entirely, but still want to prevent it from appearing in search results.

Tip: We developed an OpenCart extension to make adding these to all page types a breeze. Take a look here: https://www.opencart.com/index.php?route=marketplace/extension/info&extension_id=45574

Final Thoughts

Properly managing how Bingbot interacts with your site can improve crawl efficiency, reduce server load, and ensure only your most valuable content is indexed. With a correctly written robots.txt and use of the Crawl-delay directive, you can take control of how Bing (and other crawlers) see your site.

Have you noticed aggressive Bingbot activity on your site? Try these tips and let us know how it impacted performance in the comments below!

blog comments powered by Disqus