Post by @jorijn
We're working through some infrastructure issues right now. I'm still looking for the cause, but it appears network traffic is is somehow dropping. Sorry about this :)
The cluster appears to have stabilized and is again, serving traffic very happily. I've gathered pointers and will be reaching out to DigitalOcean to inquire what might have happened. Sorry for any inconvenience.
For the tech guys: Pods weren't able to communicate across the internal K8S network despite restarting kube-proxy/cilium a few times. Intermittent connection timeouts, primarily revealing itself through not being able to resolve vital components such as DBs.
About the disruption ±12 hours ago: DigitalOcean identified an issue with their private networking connectivity that caused our problems.