If you use CNPG to run postgres and your applications employ a connection pool like HikariCP, make sure to set:
smartShutdownTimeout: 0
Otherwise, everytime you make a change to your cluster instances you get 2-3 minutes of instance downtime. And since regular changes will usually not trigger a database failover, this also means 2-3 minutes of application downtime.
Le 3 février prochain, notre équipe tech et commerciale sera aux Cloud Native Days France.
Elle pourra parler, entre autres, de notre offre de support de #CloudNativePG.
Si vous êtes dans le coin… 😊
Toutes les infos utiles ici : https://www.cloudnativedays.fr/
#PostgreSQL #SGBD
#CNPG #kubernetes
#opensource
Turns out, I wasn't used to have proper GitOps...
But reverting the ansible playbook to a previous commit actually fixed all my issues (of uptime) in literal seconds...
Now back to bringing those cnpg clusters back up and running
#gitops #kubernetes #k3s #homelab #selfhosted #selfhosting #cnpg #postgres #cluster #uptime
@nixCraft I was working hard on getting #nextcloud to work in my #kubernetes cluster. It was a massive pain in the backside because the container does a bunch of horrendous rsync shenanigans when it initialise. This so bad that it chockes up the NFS storage provider AND the server. It basically shafted my whole cluster everytime I tried deploying Nextcloufq Only way out was using object storage and I decided on Garage on top of ZFS dataset. Now my #cnpg DBs do back-ups to that as well, two birds with one stone!
Today, trying to get the k3s cluster back to a working and smooth sailing state...
It has been somewhat unstable since the moving, and with little time to take care of it, outages were around every corner
Turns out, CNPG wasn't clearing out WAL on local replicas, inflating local disk usage, making them unavailable for scheduling...
Now that I understand why everything goes down, it is time to setup ntfy and link with with alertmanager+prometheus to get proper insights on the cluster in realtime
Now, back to rebuilding all Longhorn volumes because of those random outages *sight*
But underlining the importance of having good backups ! Because no data was lost, despite random downtimes
#k3s #devops #kubernetes #prometheus #alertmanager #cluster #cnpg #wal #postgres #psql #ntfy #longhorn #volume #backups #homelab #selfhosted
#DOAG2025 Conference was a blast!
Some pictures from my talk about Proxmox (for my company laptop: nested virtualized under Hyper-V), Oracle26ai, PostgreSQL, Patroni and CNPG on virtualized K8s, virtualized using Prometheus and Grafana.
Slides in English, presented in German (to download and read or view online - esp. if you weren't able to attend, because it was the last slot).
#proxmox #oracle #26ai #postgresql #postgres #enterprisedb #cnpg #k8s #LearnK8s #prometheus #grafana #observability
Sad to miss the spontaneous #CNPG dinner but already heading home from KubeCon (I hope - so far we are 6 minutes delayed). Had a blast meeting the #AKS team, the #AWS #OpenSource team, and many more. Glad being able to show the #documentdb #kubernetes operator here at #kubecon - until next time!!
Our latest blog shows how to set up multi-master replication using pgEdge Enterprise Postgres, Spock, and CloudNativePG — creating a resilient, fully distributed PostgreSQL architecture that runs seamlessly on Kubernetes.
💡 High availability. Low latency. 100% open source.
👉 Read the full blog: https://www.pgedge.com/blog/multi-master-replication-using-pgedge-enterprise-postgres-with-spock-and-cloudnativepg
#programming #kubernetes #k8s #devops #cloudops #cnpg #cloudnativepg #postgresql #postgres #developers
👉 The only #PostgreSQL operator governed by a vendor-neutral community
👉 The only Postgres operator submitted to the #CNCF Sandbox
👉 The most popular Postgres operator on @github
For these reasons, and others (like it's Kubernetes-native approach and cloud-neutrality) we've chosen to support #CNPG to facilitate highly available, scalable #Postgres deployments on #Kubernetes with a developer-friendly deployment process.
Miss the announcement? 📖 https://www.pgedge.com/blog/pgedge-cloudnativepg-simplifying-distributed-postgres-on-kubernetes
🎉 Exciting news! As of today, we now support new container images (and an updated #Helm chart) that are built for compatibility with @CloudNativePG.
#CNPG is an open source K8s operator that automates the lifecycle of #PostgreSQL clusters using native K8s resources, recently accepted as a #CNCF Sandbox project and now the community standard for running #Postgres natively on #Kubernetes.
🐘 Read the blog: https://www.pgedge.com/blog/pgedge-cloudnativepg-simplifying-distributed-postgres-on-kubernetes
#coding #opensource #foss #oss #developers #programming #devops
Having recently experienced a rather horrible #Kubernetes crash, I'm looking for #backup solutions. We're good with PostgreSQL since we're using #CNPG with remote transaction logs to an offsite #S3 bucket. I need something for volumes and maybe Kubernetes resources. #Longhorn offers S3 backups for it's own volumes, but for other #CSI like local #OpenEBS, maybe #Velero? Thoughts?
@pgDayParis is currently accepting proposals in an official Call for Papers! Submit topics like PostgreSQL internals hacking, DevOps related topics, migration from other database systems, or whatever else you may have found interesting in your journey with Postgres. https://2026.pgday.paris/call-for-papers/
#postgres #postgresql #database #devops #sqlserver #oracle #mongodb #redis #debezium #rlanguage #cnpg #clouenativepg #kubernetes #docker #aws #gcp #azure #linux #opensource #foss #oss #paris #france
Next up was the update of the Grafana CNPG cluster from 16.2 to 17.6. Yet again, like for the harbor cluster, the minor update from 16.2 to 16.10 has worked, but the update from 16.10 to 17.6 has failed. The primary came up again without issue, but the secondary is failing with timeline divergence again.
I don't think I will keep CNPG in the long run. It doesn't seem to actually give me anything positive.
Ok I fudged up
Trying to recover the cluster as it stands, I deleted the forgejo db to recover it from the backups...
Thing is, I forgot to configure the backups when migrating from gitea, woops
Result, I managed to salvage some about-to-be-deleted filesystem from a node, and now I'm trying to restart the node momentarily to export a dump... FML
#homelab #selfhosted #cluster #k3s #cnpg #postgres #database #backup #recovery #filesystem #mistake
That's why I debug in prod /s
Found out a new issue with the k3s cluster, some of my pods started getting `Evicted` because of `DiskPressure`
But why ? There is plenty of disk space available ? All of my data is stored on secondary drives, so they shouldn't even clog the main disk !
Turns out, yes, all the big data is stored on secondary disks, but not my databases... Including their backups !
Some of my machines have ~30Gb of root space and an additional ~500Gb for big storage, so those little roots are getting filled up quite quickly (6 months) with those dbs...
Oh well, down to writing a new StorageClass for that effect I guess
#selfhosted #homelab #production #debug #kubernetes #k3s #cnpg #postgres #db #database #storage #linux #container
Adventures in #selfhosting!
I just finished failing to do a "correct" and "proper" upgrade of cloud-native postgres, #cnpg, from using standard #longhorn volume backups to the barman cloud plugin.
I got the plugin loaded accord to the migration docs, but couldn't get it to write to #s3, nor could the pods become ready. I worked at it for hours, but I saw lots of other people online and recently having the same issues and log messages that I was.
The reason I did this in the first place was that I noticed that I had some duplicate backup jobs causing issues with #fluxcd reconciliation.
In the end, I gave up and went back to the original longhorn backups, which have worked and I've already done disaster recovery with (don't ask), and deleted the duplicates.
Currently I'm waiting for the previous primary/write node to fully restart and clear out the barman side car. Then I'll turn flux back on and hopefully things will be good.
Nice, I finally powered back on the NAS, it everything came back up
Longhorn backups, CloudNativePG backups, MinIO, OpenmediaVault, Wireguard
The full stack came back without a hiccup
(Small sike on me tho', I had to SSH into the machine to restart the containers and wireguard because I had forgotten to enable them at boot...)
#selfhosted #homelab #kubernetes #k3s #longhorn #wireguard #cnpg #postgres #minio #openmediavault #backups #sike #nas #ssh
Today's #adventuresInSelfHosting, I was having trouble with #cloudnativePG where I'd always have n-1 pods stable and they'd constantly restart between the replicas in a massive loop, then eventually switch with the primary and continue the process. So, I set the instance count to 3 so there'd at least be one stable read replica at a time.
I finally found out what was wrong. I have my #cnpg cluster in the postgresql-system namespace and I happened to see that I had an operator running in the default cnpg-system namespace. I don't know how long it had been there, so both it and the one in my namespace were both competing for the state. Deleting and cleaning up that old cluster brought immediate stability.
I also realized that I wasn't overriding the default #php configuration for #pixelfed, so when I uploaded an image taking on my phone, the web server would restart. I bumped the php memory up to 1GB for now. For the expected userbase for the upcoming #keyboardvagabond #fediverse space, this should be fine.
Right now the services are running well, but I need to do more testing and get mastodon into an "interesting" state for new visitors. Pixelfed seems the hardest for me in terms of getting content onto the server so that it doesn't look barren.
The todo list for now is:
It's getting close! The services are essentially ready, just not necessarily turned on for signups until I'm ready for a pre-launch or full launch. I want to make sure things are in a good state.
But with the 2 node #kubernetes #cluster, I think things should be good!
By then end, it should look like:
Great productivity tonight !
Upgraded Immich from 1.132.1 to 1.135.3
Migrated its database from pgvector.rs to VectorChord
Upgraded to VectorChord 0.4.3 from 0.3.0
That was actually relatively easy once I started taking my time and reading the migration documentation
Now everything is healthy and up to date, let's see how those performance improvements translate to realworld applications
#immich #selfhosted #homelab #kubernetes #vectorchord #pgvector #psql #cnpg #rtfm
