How to block crawlers and others user agents

Last Updated -


Question

We are getting hammered by crawlers, how can we block requests with specific user agents? 


Answer

The two options for installing bot blocker that are:

1) Server Wide

We could place the block-bots.conf file above in the universal server-wide configs for you, in which that would block the User-Agent bots for every site on the server.

Unfortunately, this would not be customisable by you and could only enabled/disabled by putting in a support ticket request.

2) Nginx-only site specific .config configs (best)

Our help documents are @ https://support.pagely.com/hc/en-us/articles/115000013112-Changing-your-site-to-NGINX-Only for this.

If a website is switched to be in Nginx-only mode, then we could place the .conf file above to be applied on just this site.

Note: The Nginx-only mode already has all the default WordPress routing configurations built into it, so unless there's any customizations to .htaccess files, it should work. (Although, we always recommend testing after it's switched to make sure the sites plugins still work as intended.)

With this method, you would also be able to modify the user/nginx-server/block-bots.conf file and add or remove new User-Agents by modifying the block list. Say if you saw one in the access logs that wasn't in the most common ones we see, the list could have items appended to it & nginx reloaded to implement the new User-Agent blocks. These would quickly be blocked and never be cached. 

Related Topics

Pagely is the Managed WordPress Hosting Platform designed to exceed the needs of media, business, and Enterprise customers alike. We help the world's biggest brands scale WordPress.

Copyright © 2006-2017 Pagely, Inc. All rights reserved.
Pagely® and WordPress® are registered trademarks.

Pagely
Powered by Zendesk