| unofficial-fish | Posted: 18 Sep 2026, 01:04 AM |
|---|---|
 Member Posts: 10 Joined: 18-June 24 |
Can anybody here maintaining a website tell me what that's like at this time? I am planning to get hosting for my own personal site soon, but I've seen many, many people have difficulties with keeping them stable under these... circumstances. Modern AI scrapers are nearly indistinguishable from a DDoS attack: they come from a thousand residential addresses* with falsified useragents, they do not take no for an answer, and they generate huge amounts of traffic because they aren't smart enough to figure out when they're doing redundant work. I am under no illusions that I can keep anything public-facing out of the data slurry that the crawlers are cooking up, but I do need to make sure that I can have a functioning site without keeping a window open to the access logs 24/7 (yes, I've seen people have to do this.) It's a small, static site, and quite lightweight. It doesn't have the kinds of deep, dynamically generated links that induce the worst stupidity in scrapers (wikis and cgit instances have been particularly affected by this), but the storm keeps getting worse. git.kernel.org used to be able to get by with a fairly light proof-of-work captcha, but at this point the scrapers seem to have more compute to spare than the actual users! *'Residential proxies' (read: botnets. Technically above-board, otherwise identical.) ------------- |
| [top] | |
| mit | Posted: 18 Sep 2026, 10:51 AM |
 yeah whatever Admin Posts: 699 Joined: 4-May 16 |
For a static site it's really not that bad. You'll be fine, just dump some HTML files on a webserver and forget about it. This site is hosted on NearlyFreeSpeech, and I can heartily recommend them. While the scrapers are relentless it's still a small fraction of the traffic I get when someone links here on Hacker News, and it's always managed to survive that, despite being a PHP monstrosity. I absolutely hate the proof-of-work protection, that just can't be the way forward, there's no way the little guy can win a race like that. I'm not sure what the solution is though. ------------- |
| [top] | |
| unofficial-fish | Posted: 25 Sep 2026, 03:07 AM |
|
Member Posts: 10 Joined: 18-June 24 |
I've had the site up for about a week now, and it's gotten some 4600 bogus requests (it's easy to estimate when you get a negligible amount of legitimate traffic.) This level of background radiation indeed seems habitable. I am using NearlyFreeSpeech, I'm pleased to see from their forum that they're actively working on mitigating this problem. I've set up my site as "no PHP or CGI scripts", so I have to do all the processing at 'build time' (if that's the word to use.) They don't bill for bandwidth at this time so this ensures a constant cost per month. Perhaps I should learn some PHP, but I'm stubborn and have chosen to use pandoc and Makefiles instead... About the proof of work, I hear that it doesn't even work for certain use cases. If the bandwidth of requests, not the CPU time to process them, is enough to cause problems, it can't help you - your server's connection is saturated whether you block the bots or not. ------------- |
| [top] | |
| mit | Posted: 25 Sep 2026, 10:32 AM |
|
yeah whatever Admin Posts: 699 Joined: 4-May 16 |
Back in the day, NearlyFreeSpeech was a purely pay-as-you-go service. You got billed for exactly how much you'd used in resources. Unlike any other hosting, they made more money when your site was successful, compared to hosting with fixed fees, which made more profit if your site went unvisited. Sadly it ended up that 99% of their operating costs were dealing with DDoS attacks, so they had to change their business model. That was before the boom in scrapers. I would definitely prefer to have a statically generated site, I keep meaning to rewrite a lot of the way this site works. I'd have to sacrifice some of the easter eggs though. Back when I wrote most of this, PHP was the recommended way to do it, now it's just a liability. ------------- |
| [top] | |
Sign in to post a reply.