| unofficial-fish | Posted: 18 Sep 2026, 01:04 AM |
|---|---|
 Member Posts: 8 Joined: 18-June 24 |
Can anybody here maintaining a website tell me what that's like at this time? I am planning to get hosting for my own personal site soon, but I've seen many, many people have difficulties with keeping them stable under these... circumstances. Modern AI scrapers are nearly indistinguishable from a DDoS attack: they come from a thousand residential addresses* with falsified useragents, they do not take no for an answer, and they generate huge amounts of traffic because they aren't smart enough to figure out when they're doing redundant work. I am under no illusions that I can keep anything public-facing out of the data slurry that the crawlers are cooking up, but I do need to make sure that I can have a functioning site without keeping a window open to the access logs 24/7 (yes, I've seen people have to do this.) It's a small, static site, and quite lightweight. It doesn't have the kinds of deep, dynamically generated links that induce the worst stupidity in scrapers (wikis and cgit instances have been particularly affected by this), but the storm keeps getting worse. git.kernel.org used to be able to get by with a fairly light proof-of-work captcha, but at this point the scrapers seem to have more compute to spare than the actual users! *'Residential proxies' (read: botnets. Technically above-board, otherwise identical.) ------------- |
| [top] | |
| mit | Posted: 18 Sep 2026, 10:51 AM |
 yeah whatever Admin Posts: 691 Joined: 4-May 16 |
For a static site it's really not that bad. You'll be fine, just dump some HTML files on a webserver and forget about it. This site is hosted on NearlyFreeSpeech, and I can heartily recommend them. While the scrapers are relentless it's still a small fraction of the traffic I get when someone links here on Hacker News, and it's always managed to survive that, despite being a PHP monstrosity. I absolutely hate the proof-of-work protection, that just can't be the way forward, there's no way the little guy can win a race like that. I'm not sure what the solution is though. ------------- |
| [top] | |
Sign in to post a reply.