Bot block with no agent path

The site returns 403, 503, or a challenge page to the agent. The agent stops, and the user goes to a competitor.

29 posts in the last 30 days

What people report

Delta seems to do a great job of blocking every agent I’ve tried.

@owendesign on X

I just tried Best Buy today with no luck. Apparently Best Buy has a bot detection which Muse cannot bypass or click around.

@cj_streetpoet on X

Thumbtack and Yelp block instinct. Would love to find yellow page services that welcome agents.

@suvirjain on X

Do this instead

  • Separate training crawlers from live user agents. Permit signed agents (Web Bot Auth) and known user agents such as ChatGPT-User and Claude-User.
  • If you block the browser path, give an agent path: an API, an MCP server, or a connector. OpenTable does this.
  • Test your own site with a real agent each month. A default block is easy to miss.