I packaged bigquery_fdw over 6 years ago. However if you tried to use bigquery_fdw from the PostgreSQL YUM or zypper repositories recently, there is a good chance it did not work: the Python modules it needs could not be imported by the interpreter it runs in. The packages that are in the repositories as of today fix that on RHEL 8, 9 and 10, Fedora 43, 44 and 45, Amazon Linux 2023, and SLES 15 and 16. This post explains what was wrong, what changed, and what (if anything) you need to do.
The full history is in issue #367.
The full history is in issue #367.
What was broken?
bigquery_fdw is a Python foreign data wrapper. It does not run in a standalone Python process: it runs inside the Python interpreter that multicorn2 embeds into the PostgreSQL backend. So every Python module it imports has to be installed for that interpreter.
That is where things went wrong. Its dependencies, google-cloud-bigquery and google-auth, were built for the distribution's default python3, while multicorn2 embeds a newer one on most platforms:

Even where the interpreters matched, there were more problems:
- python3-google-cloud-bigquery was version 1.24.0, from 2020. Its own dependencies (google-api-core, google-cloud-core, google-resumable-media, protobuf, ...) were not in our repositories at all, and because the dependency generator was disabled for bigquery_fdw, nothing flagged them as missing.
- On RHEL 9, python3-google-auth could not even be installed: it needed a newer cryptography than the OS ships.
- multicorn2 itself was broken on RHEL 8, RHEL 9, Amazon Linux 2023 and SLES 15. multicorn.so was linked against the libpython PostgreSQL was built with, not the one its Python code was installed for, and on RHEL 8 and SLES 15 the Python part of multicorn2 ended up in the python3.6 site-packages directory.
What we did to fix: a private Python stack for bigquery_fdw
Building the latest google-cloud-bigquery for each interpreter means also building recent protobuf, grpcio and google-api-core. Installing those into the system site-packages would replace, or conflict with, the copies the operating system ships, and other OS packages depend on those. So the new packages install into a private directory instead:
/usr/lib/pgdg-python/pythonX.Y/site-packages
Only bigquery_fdw looks there: its __init__.py adds that directory to sys.path when multicorn2 loads it. Nothing else on the system sees these modules, so your OS Python packages are untouched.

On RHEL, Fedora and Amazon Linux the packages keep the usual python3.X- names and also provide pgdg-python3.X-. SUSE already ships packages with the usual names, so there they are called pgdg-python3XX-. bigquery_fdw and the stack packages require each other by the pgdg- name, so an OS package can never satisfy the dependency by accident.
google-auth is the one dependency that is not private. Fedora, RHEL 10 (via EPEL) and SLES ship a version that is new enough, so bigquery_fdw uses the OS package there. On RHEL 8, RHEL 9 and Amazon Linux 2023 the OS has no google-auth for the right interpreter, so we ship python3.12-google-auth and python3.13-google-auth.
Along the way, multicorn2 was fixed to link against the right libpython and to install its Python code for the right interpreter, and the unused google-oauthlib dependency was dropped: bigquery_fdw never imported it.
Bumps along the road
Packaging a modern Python stack for nine distributions and five Python versions (3.11 to 3.15) turned up the usual collection of surprises. A few of them:
- OpenSSL 4 on Fedora 45: grpcio 1.84 does not build with it. We carry the upstream fix until grpcio 1.85 is out.
- Old setuptools: several packages use the new PEP 639 license field, which setuptools older than 77 rejects, so the specs rewrite it on older platforms.
- SLES 16: its trove-classifiers is too old to know the "Python 3.15" classifier, and setuptools refused to build proto-plus and google-cloud-bigquery because of it.
- Fedora's google-auth: our first build shipped our own python3-google-auth on Fedora too. That replaced the OS package and broke updates of python3-google-auth+requests, so on Fedora, RHEL 10 and SLES we now use the OS package, as described above.
Every package was built in mock for each platform, and bigquery_fdw was tested end to end: CREATE EXTENSION, a foreign table, and a query that goes all the way to the token request of a (fake) service account.
What you need to do
Just update:
# RHEL, Fedora, Amazon Linux
dnf upgrade bigquery_fdw 'multicorn2*'
# SLES
zypper update bigquery_fdw 'multicorn2*'
The new bigquery_fdw (2.0-15PGDG) pulls in the new stack automatically. If you installed bigquery_fdw before and it never worked, or you updated to 2.0-14 and got the google.cloud import error, update again and it should now import cleanly. After upgrading multicorn2, restart PostgreSQL, or at least reconnect, so that backends load the new multicorn.so.
Note: python3-google-oauthlib and the old python3-google-cloud-bigquery 1.24.0 are retired from our repositories. If you have them installed only because of bigquery_fdw, you can remove them.
Known limitation: with Python 3.12 and later, multicorn2 and PL/Python cannot be used in the same database. See Known Issues in the multicorn2 README.
As I always say: Enjoy!
bigquery_fdw is a Python foreign data wrapper. It does not run in a standalone Python process: it runs inside the Python interpreter that multicorn2 embeds into the PostgreSQL backend. So every Python module it imports has to be installed for that interpreter.
That is where things went wrong. Its dependencies, google-cloud-bigquery and google-auth, were built for the distribution's default python3, while multicorn2 embeds a newer one on most platforms:

Even where the interpreters matched, there were more problems:
- python3-google-cloud-bigquery was version 1.24.0, from 2020. Its own dependencies (google-api-core, google-cloud-core, google-resumable-media, protobuf, ...) were not in our repositories at all, and because the dependency generator was disabled for bigquery_fdw, nothing flagged them as missing.
- On RHEL 9, python3-google-auth could not even be installed: it needed a newer cryptography than the OS ships.
- multicorn2 itself was broken on RHEL 8, RHEL 9, Amazon Linux 2023 and SLES 15. multicorn.so was linked against the libpython PostgreSQL was built with, not the one its Python code was installed for, and on RHEL 8 and SLES 15 the Python part of multicorn2 ended up in the python3.6 site-packages directory.
What we did to fix: a private Python stack for bigquery_fdw
Building the latest google-cloud-bigquery for each interpreter means also building recent protobuf, grpcio and google-api-core. Installing those into the system site-packages would replace, or conflict with, the copies the operating system ships, and other OS packages depend on those. So the new packages install into a private directory instead:
/usr/lib/pgdg-python/pythonX.Y/site-packages
Only bigquery_fdw looks there: its __init__.py adds that directory to sys.path when multicorn2 loads it. Nothing else on the system sees these modules, so your OS Python packages are untouched.

On RHEL, Fedora and Amazon Linux the packages keep the usual python3.X-
google-auth is the one dependency that is not private. Fedora, RHEL 10 (via EPEL) and SLES ship a version that is new enough, so bigquery_fdw uses the OS package there. On RHEL 8, RHEL 9 and Amazon Linux 2023 the OS has no google-auth for the right interpreter, so we ship python3.12-google-auth and python3.13-google-auth.
Along the way, multicorn2 was fixed to link against the right libpython and to install its Python code for the right interpreter, and the unused google-oauthlib dependency was dropped: bigquery_fdw never imported it.
Bumps along the road
Packaging a modern Python stack for nine distributions and five Python versions (3.11 to 3.15) turned up the usual collection of surprises. A few of them:
- OpenSSL 4 on Fedora 45: grpcio 1.84 does not build with it. We carry the upstream fix until grpcio 1.85 is out.
- Old setuptools: several packages use the new PEP 639 license field, which setuptools older than 77 rejects, so the specs rewrite it on older platforms.
- SLES 16: its trove-classifiers is too old to know the "Python 3.15" classifier, and setuptools refused to build proto-plus and google-cloud-bigquery because of it.
- Fedora's google-auth: our first build shipped our own python3-google-auth on Fedora too. That replaced the OS package and broke updates of python3-google-auth+requests, so on Fedora, RHEL 10 and SLES we now use the OS package, as described above.
Every package was built in mock for each platform, and bigquery_fdw was tested end to end: CREATE EXTENSION, a foreign table, and a query that goes all the way to the token request of a (fake) service account.
What you need to do
Just update:
# RHEL, Fedora, Amazon Linux
dnf upgrade bigquery_fdw 'multicorn2*'
# SLES
zypper update bigquery_fdw 'multicorn2*'
The new bigquery_fdw (2.0-15PGDG) pulls in the new stack automatically. If you installed bigquery_fdw before and it never worked, or you updated to 2.0-14 and got the google.cloud import error, update again and it should now import cleanly. After upgrading multicorn2, restart PostgreSQL, or at least reconnect, so that backends load the new multicorn.so.
Note: python3-google-oauthlib and the old python3-google-cloud-bigquery 1.24.0 are retired from our repositories. If you have them installed only because of bigquery_fdw, you can remove them.
Known limitation: with Python 3.12 and later, multicorn2 and PL/Python cannot be used in the same database. See Known Issues in the multicorn2 README.
As I always say: Enjoy!
The author does not allow comments to this entry
No comments