#RukiiNet was down for 9 hours 45 minutes today because we installed a new #ElectricCar charger, which cut the power momentarily which apparently killed my UPS. It took me a while to notice.

#SelfHosting

#RukiiNet was down for 26 hours because five cities lost the internet due to cable damage:
https://www.granadahoy.com/granada/averia-caida-internet-provocada-obras-metro-area-metropolitana-granada-solucionada_0_2004026651.html

#SelfHosting

Yesterday and today, the whole of Spain, Portugal and parts of France, and possibly as per some reports, parts of Italy and Belgium lost their power inexplicably. Italy and Belgium reports might have been spurious. This happened about 12:30 local time here in Spain.

The power was out for about 14 hours here, a bit less for some areas as they brought up the grid piece by piece.

The mobile base stations were the first to go.

We have solar panels and batteries which carry over the night, and it was a sunny day, so we maintained power here in the house over the event.

This Mastodon instance was inaccessible for 11 hours, because apparently the fiber optic substations lost their emergency battery power after 6 hours or so, and the final internet connection we had was cut.

There is still no conclusive determination for what happened or why. It seems the most likely explanation currently is some sort of an atmospheric event causing resonance in the electric grid.

https://elpais.com/economia/2025-04-28/apagon-electrico-masivo-en-espana.html

At night, our house was pretty much the only one with power. The other lights visible in the photo are largely cars and flashlights. We still have our Christmas lights in the yard...

The stars were well visible in the sky.

So, lessons learned:
1. Apocalypses (or pseudo-apocalypses) don't necessarily come with a bang. They come with an eery silence and darkness, and no one knows what is happening.
2. I am still unable to play my local Plex series with the good TV, because everything seems to require internet connectivity for no good reasons. I can play them with my phone and on laptop (but with proprietary codecs and without hardware accelerations it's in practice impossible). Playstation Plex simply refuses to function without internet.
3. Don't run multiple high power cooking appliances at once because it overloads the inverter.
4. The house water pump isn't in the emergency power circuit, which is inconvenient and needs fixing.
5. Need to have physical keys for things easily available, and not hidden in places you'd look in the last.
6. Even if the local internet networks would still work, if the global networks aren't available the ISPs happily just cut off all the networks and don't even try to maintain just local networks for convenient local emergency communications purposes. Ditto for electricity.

#SelfHosting #RukiiNet #apocalypse #resilience #blackout #Spain

#RukiiNet relatively uneventfully upgraded to v4.3.4.

Needed setting up some new cryptokeys and mastodon-streaming is apparently a separate Docker image nowadays.

#SelfHosting

Started investigating why I was still getting HTTP 413 Entity Too Large errors for uploading larger image files on this instance. Caddy reverse proxy doesn't impose limits and Mastodon default limits are way higher. What was the cause then?

The Kubernetes nginx-ingress has a default limit of 1 MB requests, and that was giving the error. It was easy to fix by adding an annotation to the ingress.

#SelfHosting #RukiiNet

Upgraded #RukiiNet to the newest Mastodon, 4.2.10.
Everything went painlessly.
#SelfHosting

Rukii.net was down for 7 hours 28 minutes because NAS got a new IP address even though it was statically bound. Changed it to a static IP.
#RukiiNet #SelfHosting

Rukii.net was down for 7 hours 28 minutes today because NAS decided to get a new IP address even though it was bound statically. Switched it to a static IP address.
#RukiiNet #SelfHosting

Rukii.net was down for 12 hours as I was installing a new NAS. This is now done, gives more space and stops all the old issues we had with OpenEBS Jiva.
#RukiiNet #SelfHosting

Rukii.net will go down shortly tomorrow, Saturday morning Spanish time due to solar panel installation. Need to cut the power to rewire it in a more sensible manner.
#RukiiNet #SelfHosting

#RukiiNet #SelfHosting: I had an outage of one and a half days because there was a small fire in the house. The pellet boiler caught fire. Lots of losses of electricity in the event itself and the related fixes.

#RukiiNet #SelfHosting update:
Rukii.net has moved from #Benalmádena to #LasGabias! Everything works again, after an outage of 5 days and 13 hours due to the move.

#RukiiNet #SelfHosting: rukii.net will have an outage 9.11 as I will be moving to Granada. I will try to set things up as fast as possible there, but some downtime will be unavoidable.

#RukiiNet #SelfHosting, rukii.net will go down for a day or two 24.11 or so when we're moving to Granada. The plan is to get the servers up again as soon as possible, but it will necessarily cause some downtime.

#SelfHosting #RukiiNet update:
Updated Mastodon to version v4.2.0. Lots of new features! Be sure to set your settings to allow your public posts to be indexed and searched if you desire so!
Relating to that, there was a jam-tangle with the database migrations, which at first corrupted the database and then with backups it got stuck in an infinite iteration in the duplicate removal of some preview images. I had to manually remove the duplicate relations, which allowed the migration to proceed. The site was down for some 2 hours, and I lost two posts of mine, one of which I rewrote and the other shall remain as if deleted.

#RukiiNet #SelfHosting update:
Migrated away from NFS and Jiva for Postgresql. Now using high-availability Postgres with two replicas instead. Works much faster and is more robust to failures.

#RukiiNet: Backups are now tested, added a cronjob which checks that everything is fine in the cluster and emails me immediately if not. I am also getting alarms from StatusCake if the site is unreachable. Will add an automatic scheduled test for backups being good as well to make sure everything is good in the future as well.

#RukiiNet update:
Sad news: A large amount of data was lost. I was doing maintenance and deleted a jiva pod to make it remount stuff. This was a mistake as it was actually the controller pod, not a replica pod. It caused the whole Postgres volume to be nuked.

Anyhow, I have backups for just this thing, the latest from the day before. In two different instances to be sure. And from the last four days to be very sure. No worries? Murphy's law strikes again.

Turns out I had two separate backup scripts running on different times on two different nodes, to be extra sure that I had backups. The thing is, these processes took a long time and it seems they actually co-occurred, and using the same sqlc dump file name on the pod, they both wrote to it at the same time.

All backups were corrupted. On all the nodes. On all the days. The last backup which wasn't was from the end of January... I restored that one, and everything I could recover from the latter backups, which sadly didn't include all the posts. Luckily pretty much no one on my instance had posted much, except me. Sadly the follow lists were gone as well.

#RukiiNet update: Still keeps happening. 1 hour downtime today. This seems to have been because of a Linux kernel upgrade, and some problem with the Realtek driver. It fails due to a netdev watchdog.

Trying to use the official Realtek DKMS drivers instead, conveniently in Ubuntu distribution but not installed by default, r8168-dkms.

Edit:
This caused problems, had to add a kernel parameter pci=nommconf. Should work now...

#RukiiNet: The nodes kept going down all the time. Upgraded Ubuntus, Kuberneteses and all those things on everything. Caused a downtime of some 8 hours.