This is an incident report, a post-mortem writeup and detailed info how we started supporting multiple RHEL minor versions.
Last month PostgreSQL RPM repos were broken for Rocky Linux and AlmaLinux 9 and 10 users due to an OpenSSL update that Red Hat pushed to versions 10.1 and 9.7, which
broke backward compatibility. Actually I broke the repos.
What actually happened?
Red Hat announces new minor releases every 6 months, and they do it a couple of days before PostgreSQL minor releases are out. They did it again on 10 and 11 Nov 2025. In the past only LLVM/CLANG updates were problematic for the PGDG RPM repository, and we
invented (now obsoleted) sysupdates repository to solve the problem. It affected a very minority of our users (only -llvmjit packages were affected). Breakage was for a couple of days, and it only affected new installations (again, only for -llvmjit users).
We always keep our build instances up2date. It also means that we
used to support rolling versions of the OSes and expect users to keep their OSes up2date as well.
Apparently this is not working anymore.
Red Hat’s update policy
Red Hat does not care about ABI compliance anymore between minor releases. Well they say they do, but it is a “
best effort”. Even though they expect their to upgrade to each minor release as they are released (see the link at the end of this paragraph), they also have Extended Update Support (EUS enables customers to remain with the same minor release of Red Hat Enterprise Linux for 24 months, allowing for stable production environments for mission-critical applications.”) . For details please read this:
https://access.redhat.com/articles/rhel-eus .
Users can also tie a system to a specific update of Red Hat Enterprise Linux:
https://access.redhat.com/solutions/238533
Similar thing can be achieved on Rocky Linux / AlmaLinux by running this:
echo '10.0' > /etc/dnf/vars/releasever
(
this solution breaks EPEL 9 as EPEL does not provide minor version directories for EPEL 9).
Why it happened: Red Hat updated OpenSSL packages
As a part of RHEL 10.1 and 9.7 releases, Red Hat pushed updates to the OpenSSL package: 3.2 -> 3.5. I was not expecting this update.
Per
https://openssl-library.org/policies/releasestrat/ OpenSSL says “no API or ABI breaking changes are allowed in a minor or patch release. The following stability rules apply to all changes made to code targeted for a major release from version 3.0.0 or later” and “...from 3.0.0 and above, the OpenSSL versioning scheme uses the format: MAJOR.MINOR.PATCH”. So this update was not supposed to break anything, Red Hat patched OpenSSL very heavily and I assume one or more of the patches caused breakage (they also patch libpq in a similar way)
Why it happened: Lack of testing
Even though we use Red Hat Enterprise Linux for building the packages, we also test them regularly on Rocky Linux and AlmaLinux, especially for PostgreSQL minor releases. This has always been a part of the release procedure and I always did that in the past one day before the release (Wednesday)– except this time. This happened because of a few reasons.
- Rebuilding LLVM/CLANG took way more time than usual. I had to edit the spec files a bit to fix build issues (their SRPM cannot be rebuilt directly!). x86_64 builds were fast enough as I could control the CPU power of the build VMs on the RPM build server, but arm64 build instances have limited resources. ppc64le instances have slightly more vCPUs, though.
- Even though I built packages on Tuesday, a last-minute-ish patch went into the tree which forced me to rebuild all packages on Thursday morning (release day): https://git.postgresql.org/gitweb/?p=pgrpms.git;a=commit;h=d38c84253ee8e16d54b9294571bd702a6977d073
- Over confidence: Not having such a breakage in the past ended up with a “Can do attitude” and lack of concentration while checking the updates. I am working on improving this situation.
Why it happened: User base
Our user base for RHEL and its clones has a wide range. Some users keep their OS’es up2date, some don’t. Some use our supported distros, some don’t (like Oracle Linux). Some users update PostgreSQL every quarter, some don’t. We need to provide them with a solution that would work for the majority of the users. We cannot make everyone happy – but hopefully we made a lot more people happier.
What did I do? Recovery plan part one: LLVM/CLANG packages
As a part of the usual practice, I rebuilt and pushed LLVM/CLANG 20 packages to the sysupdates repository along with the PostgreSQL minor releases.
What did I do? Recovery plan part two: OpenSSL packages
As a part of the immediate band-aid, I rebuilt OpenSSL 3.5.0 SRPMs on the build instances and pushed them to our sysupdates repo on Saturday.. Even though it “solved the problem”,
some users had objections and I think they had a very valid point.
What can be done? Recovery plan part three: Linking LLVM statically
Rebuilding LLVM/CLANG every 6 months is doable, but a non-trivial amount of work as I described above. To avoid the problem we can link LLVM statically. There was a proposed patch – but the discussion happened one day before the release and I did not want to push it without testing – but do we really need it after part 4? See below:
What did I do? Recovery plan part four (obsoleted first three): Supporting last 2 RHEL minor releases
As stated above the current situation was unmanageable and also RHEL has an option to tie an instance to a specific RHEL minor version. So I proposed to support the last 2 RHEL minor releases. So once RHEL 10.2 is out, we’ll archive 10.0 RPMs and shut down the instance. Same for RHEL 9.8/9.6 as well.
We already do that for SLES 15 (and later for SLES 16). For SLES 15 we currently support SP 6 and SP7.
This will help Rocky Linux / AlmaLinux users as well until both Rocky and Alma catch up. This will also help RHEL EUS users.
Even though it sounds easy, it was not. I did not leave any stone unturned and each of them were “can of worms”!
How to support the last 2 RHEL minor releases?
We (me and the PostgreSQL infra team) had to address several issues that was be completed before we released separate packages for last 2 minor releases:
- Build hosts: We installed 6 more build servers for now, and will install more every 6 months and remove the older ones (alternatively we can update the oldest one to the newest one to avoid rebuilds) (RHEL 8 won’t receive minor updates)
2 for x86_64 (will be hosted in arp), 2 for arm64 and 2 for ppc64le. These new instances involved 3 separate teams (one of them is outside PostgreSQL community, for the PPC64LE hosting)
We made sure that they are tied to a specific RHEL release. They will receive updates only for those specific RHEL versions
Example: subscription-manager release --set=10.0
As a part of this process we replaced our arm64 server that was runningRocky 9 and started using RHEL 9.
Installed corresponding Rocky and / or AlmaLinux test servers. They are also tied to specific OS minor releases.
Updated build scripts: I had to overhaul build scripts They currently sync to the “main” directory (i.e. rhel-10-x86_64). I updated them so that they will sync to the minor version directory (i.e.rhel-10.0-x86_64, rhel-10.1-x86_64). Already deployed this to global.sh : https://github.com/pgdg-packaging/pgdg-rpms/commit/ccfdd766aa9e010699f7e21fb6e8bfef276967e1
and updated SLES scripts so that they utilise this variable:
https://github.com/pgdg-packaging/pgdg-rpms/commit/70cc2d5b7ed30b52905e511757a0eeca9ed25dd6
Deployed these to SLES first as we already support multiple service packs where proper testing can be done easily.
Update build scripts: Currently the build system relies on the $DIST variable in the Makefiles in the package names, like f43, rhel10, etc. As a part of the minor release plan, we will add RHEL minor version number to the package name, like .rhel9.6
- Removed relevant lines from Makefile.global* files, like: --define "dist $(DIST)"
- Add these variables to .rpmmacros. Examples are:
- %dist .rhel9.6
- %dist .rhel10.1
- %dist .f43
- %dist .sles15.6
In the past we had symlinks to the major OS version (like rhel-9.6-x86_64 -> rhel-9-x86_64). We got rid of these and created separate directories for the minor OS versions. In order not to break the majority of the user base, we created symlinks from the latest minor version to the major version (i.e. rhel-10.1-x86_64 -> rhel-10-x86_64). I will review this almost every 6 months along with RHEL minor release updates.
- Support policy: RHEL releases new minor version every 6 month, so a minor release will be supported for a year (or more, depending on the EOL status)
- Disk space usage on PostgreSQL.org servers: Adding new packages means more disk space usage on PostgreSQL.org download infrastructure. I’d like to extend my gratitude to the .org infra team for their help and understanding during this work.
Conclusion
I’m relieved that this is sorted. Thanks everyone for their patience and support.
