Post by @jorijn
Another interesting adventure from the life of a #mastoadmin: "Where do all these requests come from?"
These spikes come and go on a fairly steady rhythm (each ~60 minutes). I found this to be interesting, so I investigated which URLs are being requested the most during these spikes.
Turns out: It's the emojis. I don't mind as they're being returned early by Varnish, but why so much? 2.5 million times per 24 hours seems like _a lot_.
Two red lines mark the Mastodon 4.5 upgrade.
Here's a view from seven days earlier. The emojis don't even rank in the top ten most requested URLs.
New data suggests it's a crawler by Tencent, and there is no relation to the latest 4.5 release. Will research a bit more later.
Update on the Tencent crawler situation:
Identified the fingerprint (Chrome/126 on Windows NT 6.1, always from AS132203/132591/45090) and deployed a #Crowdsec scenario to auto-ban them for 90 days.
Already blocked 100+ IPs within the first hour. They rotate constantly, but the fingerprint stays the same.
The attached chart shows traffic from the last 24 hours. All yellow spikes are recurring Tencent crawls.
The Tencent crawler pretends to be "Chrome 126 on Windows 7."
Chrome 126: released June 2024, Windows 7: lost Chrome support in February 2023.
This browser is impossible. Someone copied and pasted a User-Agent string and forgot Windows 7 has been dead for years.
Over 5,100 unique #tencent IPs triggered the ban within roughly 2-3 hours of this scenario going live. These guys don't mess around...