wal_segment_size引數的理解與調優
In PostgreSQL you configure the size of the wal (write ahead log) segments when you compile from source.
If you use an installer or if you use the packages provided by your OS distribution the size of thewal segments is usually 16MB. Although 16MB seems very low
you don’t need to worry about that in most of the cases, it just works fine.
However there are cases where you might want to adjust this, e.g. when you have an application that generates thousands of transactions in a very short time
and therefore forces PostgreSQL to generate huge amounts of wal segments.
In this post we’ll look at a specific case: Usually you want to archive the wal segments for being able to do point in time recovery in case your severs crashes for some reason.
Does the size of the wal segments matter for archiving?
Archiving of wal segments in PostgreSQL is done by specifying anarchive_command. Whatever you put there will be executed by PostgreSQL once a new wal segment is completed.
Usually you’ll find something like this in archive_command (from the documentation):
| 1 2 |
archive_command = 'test ! -f /mnt/server/archivedir/%f && cp %p /mnt/server/archivedir/%f' # Unix
archive_command = 'copy "%p" "C:\\server\\archivedir\\%f"' # Windows
|
Or something like this:
| 1 |
archive_command = 'rsync -a %p postgres@[SOME_OTHER_HOST]:/path/to/wal_archive/%f'
|
Or:
| 1 |
archive_command ='scp %p postgres@[SOME_OTHER_HOST]:/path/to/wal_archive/%f'
|
Lets test how the size of wal segments impact the three ways of archiving outlined above. To begin with lets create 100 files each 16MB (the same as the default wal segment size in PostgreSQL) and 25 files 64MB each:
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
rm -rf /var/tmp/test16mb
mkdir /var/tmp/test16mb
for i in {1..100}; do
dd if=/dev/zero of=/var/tmp/test16mb/${i} bs=1M count=16
done
ls -la /var/tmp/test16mb
rm -rf /var/tmp/test64mb
mkdir /var/tmp/test64mb
for i in {1..25}; do
dd if=/dev/zero of=/var/tmp/test64mb/${i} bs=1M count=64
done
ls -la /var/tmp/test64mb
du -sh /var/tmp/test16mb
du -sh /var/tmp/test64mb
|
This will give us a total size of 1.6GB for each of the wal sizes (16MB and 64MB). Lets start by testing the “cp” way:
| 1 2 3 4 5 6 |
echo 3 > /proc/sys/vm/drop_caches
mkdir -p /var/tmp/target
rm -rf /var/tmp/target/*
time for i in `ls /var/tmp/test16mb`; do
cp /var/tmp/test16mb/${i} /var/tmp/target/
done
|
My result (on a VM local on my notebook):
| 1 2 3 |
real 0m17.444s
user 0m0.275s
sys 0m8.569s
|
The same test for the 64MB files:
| 1 2 3 4 5 6 |
echo 3 > /proc/sys/vm/drop_caches
mkdir -p /var/tmp/target
rm -rf /var/tmp/target/*
time for i in `ls /var/tmp/test64mb`; do
cp /var/tmp/test16mb/${i} /var/tmp/target/
done
|
It is almost 3 times as fast to copy the large files than to copy the smaller files:
| 1 2 3 |
real 0m5.365s
user 0m0.065s
sys 0m1.835s
|
Of course, for production systems, you would copy the files not locally but rather to e.g. NFS mount and then the numbers will change.
What are the numbers for scp? For the smaller files:
| 1 2 3 4 5 6 |
echo 3 > /proc/sys/vm/drop_caches
mkdir -p /var/tmp/target
rm -rf /var/tmp/target/*
time for i in `ls /var/tmp/test16mb`; do
scp /var/tmp/test16mb/${i} root@localhost:/var/tmp/target/
done
|
The result:
| 1 2 3 |
real 2m51.708s
user 0m14.136s
sys 0m35.292s
|
Quite a huge overhead. What is the result with the 64MB files?:
| 1 2 3 4 5 6 |
echo 3 > /proc/sys/vm/drop_caches
mkdir -p /var/tmp/target
rm -rf /var/tmp/target/*
time for i in `ls /var/tmp/test64mb`; do
scp /var/tmp/test64mb/${i} root@localhost:/var/tmp/target/
done
|
Approximately double as fast:
| 1 2 3 |
real 1m23.326s
user 0m10.353s
sys 0m30.814s
|
And finally rsync, for the smaller files:
| 1 2 3 4 5 6 |
echo 3 > /proc/sys/vm/drop_caches
mkdir -p /var/tmp/target
rm -rf /var/tmp/target/*
time for i in `ls /var/tmp/test16mb`; do
rsync -a /var/tmp/test16mb/${i} root@localhost:/var/tmp/target/${i}
done
|
The result:
| 1 2 3 |
real 0m51.624s
user 0m4.488s
sys 0m10.247s
|
For the larger ones:
| 1 2 3 4 5 6 |
echo 3 > /proc/sys/vm/drop_caches
mkdir -p /var/tmp/target
rm -rf /var/tmp/target/*
time for i in `ls /var/tmp/test64mb`; do
rsync -a /var/tmp/test64mb/${i} root@localhost:/var/tmp/target/${i}
done
|
The result:
| 1 2 3 |
real 0m34.342s
user 0m3.623s
sys 0m9.685s
|