| unofficial-fish | Posted: 18 Sep 2026, 01:04 AM |
|---|---|
 Member Posts: 11 Joined: 18-June 24 |
Can anybody here maintaining a website tell me what that's like at this time? I am planning to get hosting for my own personal site soon, but I've seen many, many people have difficulties with keeping them stable under these... circumstances. Modern AI scrapers are nearly indistinguishable from a DDoS attack: they come from a thousand residential addresses* with falsified useragents, they do not take no for an answer, and they generate huge amounts of traffic because they aren't smart enough to figure out when they're doing redundant work. I am under no illusions that I can keep anything public-facing out of the data slurry that the crawlers are cooking up, but I do need to make sure that I can have a functioning site without keeping a window open to the access logs 24/7 (yes, I've seen people have to do this.) It's a small, static site, and quite lightweight. It doesn't have the kinds of deep, dynamically generated links that induce the worst stupidity in scrapers (wikis and cgit instances have been particularly affected by this), but the storm keeps getting worse. git.kernel.org used to be able to get by with a fairly light proof-of-work captcha, but at this point the scrapers seem to have more compute to spare than the actual users! *'Residential proxies' (read: botnets. Technically above-board, otherwise identical.) ------------- |
| [top] | |
| mit | Posted: 18 Sep 2026, 10:51 AM |
 yeah whatever Admin Posts: 704 Joined: 4-May 16 |
For a static site it's really not that bad. You'll be fine, just dump some HTML files on a webserver and forget about it. This site is hosted on NearlyFreeSpeech, and I can heartily recommend them. While the scrapers are relentless it's still a small fraction of the traffic I get when someone links here on Hacker News, and it's always managed to survive that, despite being a PHP monstrosity. I absolutely hate the proof-of-work protection, that just can't be the way forward, there's no way the little guy can win a race like that. I'm not sure what the solution is though. ------------- |
| [top] | |
| unofficial-fish | Posted: 25 Sep 2026, 03:07 AM |
|
Member Posts: 11 Joined: 18-June 24 |
I've had the site up for about a week now, and it's gotten some 4600 bogus requests (it's easy to estimate when you get a negligible amount of legitimate traffic.) This level of background radiation indeed seems habitable. I am using NearlyFreeSpeech, I'm pleased to see from their forum that they're actively working on mitigating this problem. I've set up my site as "no PHP or CGI scripts", so I have to do all the processing at 'build time' (if that's the word to use.) They don't bill for bandwidth at this time so this ensures a constant cost per month. Perhaps I should learn some PHP, but I'm stubborn and have chosen to use pandoc and Makefiles instead... About the proof of work, I hear that it doesn't even work for certain use cases. If the bandwidth of requests, not the CPU time to process them, is enough to cause problems, it can't help you - your server's connection is saturated whether you block the bots or not. ------------- |
| [top] | |
| mit | Posted: 25 Sep 2026, 10:32 AM |
|
yeah whatever Admin Posts: 704 Joined: 4-May 16 |
Back in the day, NearlyFreeSpeech was a purely pay-as-you-go service. You got billed for exactly how much you'd used in resources. Unlike any other hosting, they made more money when your site was successful, compared to hosting with fixed fees, which made more profit if your site went unvisited. Sadly it ended up that 99% of their operating costs were dealing with DDoS attacks, so they had to change their business model. That was before the boom in scrapers. I would definitely prefer to have a statically generated site, I keep meaning to rewrite a lot of the way this site works. I'd have to sacrifice some of the easter eggs though. Back when I wrote most of this, PHP was the recommended way to do it, now it's just a liability. ------------- |
| [top] | |
| unofficial-fish | Posted: 24 Oct 2026, 06:29 PM |
|
Member Posts: 11 Joined: 18-June 24 |
> Back when I wrote most of this, PHP was the recommended way to do it, now it's just a liability. *wry chuckling* I'm starting to understand why the Gemini protocol people chose to limit it so heavily. It has almost no attack surface: - no PHP, so no PHP2Shell (I suppose in principle you could make a Gemini-PHP) - ... or any of the other PHP security holes, which together make up 30% of the CVE database - trust on first use, no single-point-of-failure root CAs - no Javascript, no tracking - ... or cross-site scripting - ... or sandbox escapes - ... or speculative execution exploits - ... or supply-chain attacks - ... or heavy 'web applications' - ... or WebGL, WebAssembly, LocalStorage, WebGPU ... - no external media, no tracking pixels - ... or page bloat either - deliberately impossible to extend into another monstrous Web Essentially the Web as originally envisioned, with better execution. As a site owner you can get many of these benefits just by not using certain features, but as a user it sure would be nice to have these guarantees about arbitrary sites! I have set up my site such that the 'master' copy lives on my local network, on my own server, and nothing 'private' ever gets uploaded to NFS. Even if all hell broke loose and my site got ransacked it would just be a denial-of-service. ------------- |
| [top] | |
| mit | Posted: 24 Oct 2026, 09:24 PM |
|
yeah whatever Admin Posts: 704 Joined: 4-May 16 |
I don't know much about Gemini protocol, but yeah as a server-side language there's not much in the protocol that could stop you using something like PHP. I have wondered about whether one could make this forum statically generated. Posts are sufficiently infrequent, and scraping by bots so relentless, that it would make sense to have every page rendered out, and regenerate the relevant ones when a post is made. That would still need the generation script and database to be live on the server so it would still have a reasonable attack surface. ------------- |
| [top] | |
Sign in to post a reply.