How to block crawlers and others user agents
Last Updated -
We are getting hammered by crawlers, how can we block requests with specific user agents?
The two options for installing bot blocker that are:
1) Server Wide
We could place the
block-bots.conf file above in the universal server-wide configs for you, in which that would block the User-Agent bots for every site on the server.
Unfortunately, this would not be customisable by you and could only enabled/disabled by putting in a support ticket request.
2) Nginx-only site specific .config configs (best)
Our help documents are @ https://support.pagely.com/hc/en-us/articles/115000013112-Changing-your-site-to-NGINX-Only for this.
If a website is switched to be in Nginx-only mode, then we could place the .conf file above to be applied on just this site.
Note: The Nginx-only mode already has all the default WordPress routing configurations built into it, so unless there's any customizations to .htaccess files, it should work. (Although, we always recommend testing after it's switched to make sure the sites plugins still work as intended.)
With this method, you would also be able to modify the
user/nginx-server/block-bots.conf file and add or remove new User-Agents by modifying the block list. Say if you saw one in the access logs that wasn't in the most common ones we see, the list could have items appended to it & nginx reloaded to implement the new User-Agent blocks. These would quickly be blocked and never be cached.