Local timeline
This is one of the harderst parts on dynamically scaling the Mastodon software. Incoming traffic casually spiking 530% above average.
Making some great progress on the #devops side of #tootcommunity
All applications are now managed the GitOps way with Argo CD. Looking to make our repository public, soon.
About the disruption ±12 hours ago: DigitalOcean identified an issue with their private networking connectivity that caused our problems.
The cluster appears to have stabilized and is again, serving traffic very happily. I've gathered pointers and will be reaching out to DigitalOcean to inquire what might have happened. Sorry for any inconvenience.
For the tech guys: Pods weren't able to communicate across the internal K8S network despite restarting kube-proxy/cilium a few times. Intermittent connection timeouts, primarily revealing itself through not being able to resolve vital components such as DBs.
We're working through some infrastructure issues right now. I'm still looking for the cause, but it appears network traffic is is somehow dropping. Sorry about this :)
Started listening to Christmas songs this weekend. That is all. 🎄
It's time for a new update about the #tootcommunity finances. 💰
Spoiler alert: We're no longer operating at a loss. 🙏
https://ko-fi.com/post/Server-costs-per-November-25th-2022-N4N1GLN9H
Watching the local timeline with lots of pleasure today as it's slowly filling with pictures of #turkey for #thanksgiving
As a Dutch guy I'm envious. One can never have enough food-focussed holidays.
I had some surgery to get my knee fixed on 11th of November. Very cool to see this data of my temperature trending above its baseline, indicating my body is working hard to repair itself. Measuring points are taken each morning when I wake up with an Apple Watch.
Did a first flight with the DJI Mini 3 Pro. Kept it short since it was very windy and cold. Didn’t get the chance to play with all the features. Here’s Dutch rush hour. Looking forward to playing around more when I have more time. #dji #drone #photography
On a more personal note: I'm limiting my instance, #tootcommunity, to 25k active users for the foreseeable future to balance all the facets outlined in this thread; so my team and I can figure out where we stand and form a solid basis as we advance. 5/5
New #MastoAdmins are running on adrenaline, and I suspect they'll start getting tired very soon. We'll possibly begin seeing new and fastly grown instances announcing their early retirement soon, and I hope we'll grow despite the pain and level up our commitment. Still, I worry that not all will and that the #fediverse will have to endure short-term pain to be better off long-term. 4/5
Are semi-tech-savvy admins going to practice backup restore procedures or hire #devops people with the required skill set to solve complex (scaling) issues? Will they wake up at night when their servers break down to offer continued 24/7 service to accommodate their user base that lives around the globe? 3/5
What if a network interface breaks down or a small percentage of network traffic turns wrong? New and recently born #MastoAdmins with significant user counts converge at cheap places since they're paying it out of their pocket. Technical support will be slow, and I worry that from a user perspective, this could damage the image of decentralized social media if user-facing issues take too long to resolve. 2/5
There's been some talk about how big a Mastodon instance should grow. Ultimately it's up to the #mastoadmin to push the buttons, but from where I stand, one should not forget topics such as accountability, liability, and (financial) stability. We're helping people grow their social network and supporting businesses in reaching their new potential clients, and I feel that we're shrugging it off. 1/5
I sometimes wonder if mr. Elon is unintentionally stuck in his own echo chamber of likeminded people, served by his own algorithm; being completely oblivious of how most really feel about recent events regarding the bird site. #showerthought
I've been looking at the DJI Mini 3 Pro #drone for a couple of days now, hoping it'll drop a bit in price for Black Friday.
#photography was a huge hobby of mine, years back. Really looking forward to picking it up again.
