Have you run into any issues from running all of your scrapers off of AWS, or just from sites detecting that you're accessing large numbers of pages in some sort of obvious pattern? I guess I was hoping there would be sites with more interesting ways to screw with web scrapers (rearranging certain page elements or something) than just throwing up a CAPTCHA.
Most really don't. A lot of big sites don't seem to care, at least in my experience.
The few that I've seen just 'ban' your IP for a few minutes. If you hit Wikipedia too much too quickly, they will essentially refuse to serve you for a while. It was a number of years ago I was doing it, but basically you would be scraping then you would just stop getting info (Maybe I wasn't reading response codes and could've realized quicker what was happening)
Wikipedia provides you with an API and guidelines on how to use it, so you really shouldn't be scraping it directly or so much you hit enforced limits.
A surprisingly small number of sites care. There are some really fun things one can do with random class/id/order variations. It's also fun to feed garbage data to scrapers when you can identify them with very high probability.
But there seems to be little demand for these kinds of systems and just throttling/blocking/CAPTCHA solutions are much simpler.
There are definitely some sites that block entire ip blocks (ex: all of aws). The only real way around this is to use proxies, but if a site's trying to block you, it's probably best to comply, and just stop.
