Skip to content

Using huge pages on RHEL 7 and PostgreSQL 9.4

Posted by Devrim Gündüz on
PostgreSQL 9.4 will finally support huge pages. This article will cover about configuring huge pages on RHEL 7 box.


For some info about huge pages, please refer to this article :

As noted in the article, "Hugepages allows large amounts of memory to be utilized with a reduced overhead." The default page size is 4kB on Red Hat releases (and its derivatives). Given that PostgreSQL uses large chunks of pages in the memory, enabling/using huge pages will help improve the performance, which will help the kernel to look up less pages in total.

By default, huge pages is disabled:

sysctl vm.nr_hugepages
vm.nr_hugepages = 0


In PostgreSQL 9.4, there is a new GUC called huge_pages, that controls the behaviour.

There are 3 values for this GUC: on, off and try. By default, it is set to try, which means PostgreSQL will try to use huge pages, if there are enough huge pages in the kernel, and otherwise will not use it. Off will disable, and on will force using huge pages. When this parameter is set to on, and if kernel does not have enough huge pages, PostgreSQL will fail to start:

After setting huge_pages to on in /var/lib/pgsql/9.4/data/postgresql.conf, let's try starting PostgreSQL:

# systemctl start postgresql-9.4.service
Job for postgresql-9.4.service failed. See 'systemctl status postgresql-9.4.service' and 'journalctl -xn' for details.


Ok, it failed as expected. Let's see the status:

# systemctl status postgresql-9.4.service -l
postgresql-9.4.service - PostgreSQL 9.4 database server
Loaded: loaded (/usr/lib/systemd/system/postgresql-9.4.service; disabled)
Active: failed (Result: exit-code) since Fri 2014-07-04 12:46:01 EEST; 45s ago
Process: 1959 ExecStart=/usr/pgsql-9.4/bin/pg_ctl start -D ${PGDATA} -s -w -t 300 (code=exited, status=1/FAILURE)
Process: 1954 ExecStartPre=/usr/pgsql-9.4/bin/postgresql94-check-db-dir ${PGDATA} (code=exited, status=0/SUCCESS)

Jul 04 12:45:55 rhel-7-x86-64 systemd[1]: Starting PostgreSQL 9.4 database server...
Jul 04 12:45:56 rhel-7-x86-64 pg_ctl[1959]: < 2014-07-04 12:45:56.113 EEST >FATAL: could not map anonymous shared memory: Cannot allocate memory
Jul 04 12:45:56 rhel-7-x86-64 pg_ctl[1959]: < 2014-07-04 12:45:56.113 EEST >HINT: This error usually means that PostgreSQL's request for a shared memory segment exceeded available memory, swap space or huge pages. To reduce the request size (currently 148324352 bytes), reduce PostgreSQL's shared memory usage, perhaps by reducing shared_buffers or max_connections.
Jul 04 12:46:01 rhel-7-x86-64 pg_ctl[1959]: pg_ctl: could not start server
Jul 04 12:46:01 rhel-7-x86-64 pg_ctl[1959]: Examine the log output.
Jul 04 12:46:01 rhel-7-x86-64 systemd[1]: postgresql-9.4.service: control process exited, code=exited status=1
Jul 04 12:46:01 rhel-7-x86-64 systemd[1]: Failed to start PostgreSQL 9.4 database server.
Jul 04 12:46:01 rhel-7-x86-64 systemd[1]: Unit postgresql-9.4.service entered failed state.


Ok, now, please revert this setting to off or try, and start PostgreSQL:

# systemctl start postgresql-9.4.service


Great. PostgreSQL 9.4 started, with huge pages disabled. In order to see how many pages you need, first let's find the pid of postmaster process:

# head -n 1 /var/lib/pgsql/9.4/data/postmaster.pid
2979

Now, let's find VmPeak in the proc status:

# grep ^VmPeak /proc/2979/status
VmPeak: 338096 kB


As I wrote above, the default PAGE_SIZE is 4kB on Red Hat. Let's confirm that:

# getconf PAGE_SIZE
4096


Let's divide VmPeak by PAGE_SIZE. It will be 165.0859375 huge pages, so PostgreSQL will need at least 166 huge pages. On RHEL 7, the good way to enable this is using /etc/sysctl.d directory:

# echo "vm.nr_hugepages=166" >> /etc/sysctl.d/postgresql-9.4.conf


Please note that you will need to increase this value depending on the other kernel parameters, like increasing shared_buffers. This is a default PostgreSQL 9.4 installation, so the values are small.

Let's activate it:

# sysctl -p /etc/sysctl.d/postgresql-9.4.conf


... and verify it:
# sysctl vm.nr_hugepages
vm.nr_hugepages = 166


Ok, let's edit /var/lib/pgsql/9.4/data/postgresql.conf , change huge_pages to on, and start PostgreSQL 9.4:


systemctl start postgresql-9.4.service



PostgreSQL will start on your RHEL 7 box, and will be able to benefit from huge pages.

Enjoy!

Trackbacks

No Trackbacks

Comments

Display comments as Linear | Threaded

Marc Linster on :

What kind of performance improvement did you see when you turned on Huge Pages? In general, when will this make a noticeable difference?

Andres Freund on :

VmPeak isn't a good estimator for the amount of hugepages as only shared memory will use hugepages. You should rather use the number of bytes mentioned in the error message.

Devrim Gündüz on :

Andres: Do you think that we need to fix, too?:

http://www.postgresql.org/docs/9.4/static/kernel-resources.html#LINUX-HUGE-PAGES

Shish on :

VmPeak is a static number based on postgres settings, it won't grow as your data set grows?

Is there any problem with setting nr_hugepages too high?

Andres Freund on :

> VmPeak is a static number based on postgres settings, it won't grow as your data set grows?
Well, VmPeak is the biggest overall size of the virtual memory allocated to a process. Including process private memory and shared memory. Now, it won't grow much for the postmaster because there seldomly will be memory allocations from it.

> Is there any problem with setting nr_hugepages too high?

The set aside memory won't be used for anything else. I.e. it's pretty much wasted.

Douglas Hunley on :

These are explicit hugepages; should we still be disabling transparent hugepages? Are you still seeing a vm stall during hugepage defrag?

Devrim Gündüz on :

@Doug: No real tests yet -- though I have a machine at home with 32GB of memory. Can test it there. I'll let you know.

Daniel Swarbrick on :

A minor correction - it looks like you (rightly so) divided the VmPeak by Hugepagesize (typically 2048 kB, see /proc/meminfo), not PAGE_SIZE, to get the value of 165.

Andrew Yoder on :

I take it from the article's title and lead-in line that any PostgreSQL instance before 9.4 will NEVER use transparent huge pages, and therefore no tuning is necessary (given that only PostgreSQL is running on the server)?

Thanks.

Douglas Hunley on :

Correct, sorta. On a db server, you're going to want to turn off the automatic defrag of THP or you'll see db stalls while the VM subsystem does the page compaction.

Add Comment

Enclosing asterisks marks text as bold (*word*), underscore are made via _word_.
Standard emoticons like :-) and ;-) are converted to images.

To prevent automated Bots from commentspamming, please enter the string you see in the image below in the appropriate input box. Your comment will only be submitted if the strings match. Please ensure that your browser supports and accepts cookies, or your comment cannot be verified correctly.
CAPTCHA

Form options