As you may be aware, there has been periodic disrupted service to our hosting service over the weekend and going into early this afternoon which may have affected your website. We want to take this opportunity to explain the issue and what we’ve done to mitigate this disruption.
We have identified a number of scrapers and bots that have hit a number of our higher profile sites during this time. This is not necessarily malicious, some bots appear to be genuine (but lesser known) search engines looking to obtain information for ranking purposes. However, they have been requesting data from us faster than we can respond, causing a vicious cycle.
Firstly, we have improved our processes to be able to better analyse when something like this happens; it was a challenge to identify the source of the issue as it was spread across a number of sites and a number of culprits. Our new processes will help us narrow down the field much faster.
Secondly, we are working through all our sites now to add a directive which will request that legitimate bots crawl them more slowly to avoid impacting performance. We have also added an additional tool to monitor requests and block any source requesting what we deem too many resources in a timeframe. We already have tools in place that scan and monitor for suspicious activity, this one is a little more robust and less discriminatory; even if traffic looks legitimate but is requesting an unreasonable number of pages it will get temporarily blocked.
We hope this offers some reassurance that we are treating this very seriously and have put suitable measures in place.
If you have any further questions, please do not hesitate to give us a call on 0121 295 5352 or email us at [email protected].
