Andrew's PostgreSQL blog

Last July, PostgreSQL moved from minimum version 2.5.4a. of flex to 2.5.31, so that we could use re-entrant scanners. The problem with this was that the newer version of flex runs everything in a chain of filters, which it sets up by forking. That's a problem on WIndows, so the latest version of flex supported by the GnuWin32 people is still 2.5.4a. So I created a build of flex version 2.5.35 using Cygwin, and put it together with the Cygwin DLL used to build it, on our download sites.

However, it turned out last December that this didn't work on 64 bit Windows. I spent some hours wrestling with the problem, without great success, and then got diverted onto other things. Yesterday I returned to the problem. Part of my difficulty is that I don't have a 64 bit Windows environment to test with. Enter GoGrid. With a few clicks I was able to set up an instance of Windows Server 2008 64 bit. After some experimentation, I found that using a more modern version of the Cygwin DLL, the existing flex I created last July would work on 64 bit Windows.

This was complicated by the fact that Cygwin doesn't seem to install nicely on at least some virtualized Windows 64 bit installations, including those on both Amazon EC2 and GoGrid - something to do with Terminal Services and Data Execution Prevention, apparently. So I got around that by not installing at all. I downloaded the package containing the DLL from a Cygwin mirror site, extracted the DLL and put it in the directory along with the flex.exe, installed GnuWin32's m4, set up the M4 environment variable appropriately, and it all worked nicely.

A revised winflex has been uploaded to the PostgreSQL download site and should appear on mirrors soon.

This is a good example of how the cloud can work. Setting up a Windows 64 bit environment would have been quite difficult and time consuming for me, but for the cost of about half a latte, I was able to get one within minutes and verify this fix within hours. And I'm glad to have it fixed, because most of the alternatives to having a working flex were fairly unpalatable.

I have been getting my Windows based PostgreSQL Buildfarm animals working with git. So far I have dawn_bat (mingw) and brown_bat (cygwin) done. I don't expect red_bat (MSVC) to be difficult.

The last thing will be to write up usage notes on the wiki. I think I will migrate the Buildfarm Howto there anyway, so it can be part of that process.

I'm still trying to come up with a nice way to put a commit date marker in files automatically. What git really needs, IMNSHO, is a hook that you can apply to the working copy before it does any other commit actions. Or maybe a filter that could be applied on the way in (like the "clean" filter) but that would also be applied to the working copy on a successful commit. Either of these would suit my purpose well.

Gurjeet was just asking on IRC about running VPath builds of PostgreSQL. It's not terribly complicated, but the sentence in the docs that refers to it could possibly be clearer. Essentially what you do it this:

mkdir build_dir
cd build_dir
/path/to/source/tree/configure
make

This is quite a nice way of keeping your source tree (almost) clean of build products.

The buildfarm has had support for testing VPath builds for quite some time, and at least one member, dungbeetle, always does VPath builds.

In last Friday's session on git, Joe Conway asked how he could build PL/R from a contrib directory, without the danger that it might be wiped out by something like "git clean -dfx".

It seems (from my experimentation) like the answer is to symlink the external module into contrib. Then "git clean -dfx" would remove the symlink but not touch the linked directory nor any of the files inside it. In addition, you would probably add "contrib/yourmodule" to your top level project's .git/info/exclude file, so that git would mostly just ignore it completely.

Tom asked at pgCon if we could more clearly identify Vpath builds. It's actually going to be quite easy. But what I need is an icon to represent VPath builds, just as we have a camel icon for Perl enabled builds, and so on. The icons are all 16x16. If anyone has a good idea for such an icon. please let me know.

For some strange reason I decided to count the number of commits in my git repo this morning. It looks like some time in the very recent past we passed the 30,000 mark, since Marc Fournier first put the files under CVS with the inital commit dated 'Tue Jul 9 06:22:35 1996 +0000'. That's a nice milestone. Bruce Momjian and Tom Lane account for over two thirds of those. Here's the list of credits:

  11959 Bruce Momjian 
   8669 Tom Lane 
   2036 Peter Eisentraut 
   1501 Marc G. Fournier 
   1081 thomas 
    631 Michael Meskes 
    616 Neil Conway 
    530 vadim 
    425 Magnus Hagander 
    357 Tatsuo Ishii 
    296 Alvaro Herrera 
    243 Teodor Sigaev 
    220 Heikki Linnakangas 
    206 Andrew Dunstan 
    176 inoue 
    157 bryanh 
    152 Jan Wieck 
    143 barry 
    107 davec 
     72 Simon Riggs 
     72 dennis 
     53 pjw 
     53 pgsql 
     52 peter 
     49 Joe Conway 
     48 Robert Haas 
     47 D'Arcy J.M. Cain 
     36 Takahiro Itagaki 
     20 byronn 
     17 mergl 
      8 Greg Stark 
      6 julian 
      2 vev 
      1 jurka 

48 hours before getting on the plane to go to pgCon, I was very far from sure I was going to make it. I had a nasty sore throat and was very stuffed up. If I hadn't been been scheduled to give a talk I would probably have cancelled, and it was a bit of a struggle to make it through, especially since my talk was right at the end of the conference. Nevertheless, I did go and I think it was worth it.

In my session, I got useful suggestions from Magnus Hagander (can we save space on the buildfarm members by using shallow clones?) and Robert Haas (can we simplify the assurance of buildfarm repo cleanliness using git clean?) among others. I also want to follow up Joe Conway's point about how to handle his foreign PL/R repository nicely within a Git tree. I hope I've made it a tiny bit easier for people like Joe and Tom Lane to take the git plunge, now that we have made the decision that we will definitely move to git around the middle of August. Magnus tells me he'll start getting a test repo up in June some time. Robert is going to write up some suggested "best git practice for committers" on the wiki.

The two outstanding things I learned at the conference were: the exclusion constraints feature, which I had not paid enough attention to, and the way PostgreSQL is being using inside Caxia, the Brazilian financial giant. The first is really the hidden killer feature from my point of view. I can think of many areas where it will be very useful. The second is a wonderful indication of just how well PostgreSQL can work in massive, enterprise level deployments.

Finally, it was great to be part of the great shirt auction, which raised something like $1500 for charity. It was a terrific indication of the spirit of the PostgreSQL community, as well as being tremendous fun. Well done to all those who took part, and especially Jim Nasby and Gavin Roy. I hope someone publishes some pictures of the auction.

I have just sent out this email, which speaks for itself:

I have just released version 4.0 of the PostgreSQL Buildfarm client.

There are two new features:

• The SCM code is substantially rearranged into a separate OO module, with subclasses supporting CVS and Git. New config options support these changes, while old style config parameters for CVS are still supported. Support for running the buildfarm from Git is a requirement before we can move the core community repo, and this meets that requirement (a little later than planned, but as promised almost exactly one year ago).


• The requirement to specify a port for each branch to build with is gone. If the config parameter 'base_port' is specified the code will pick a unique port for the branch, a short number above that setting. The means that the config file does not need to be changed when a new stable postgres release is made. Again, old-style configs continue to be supported.

The new client can be downloaded from: http://pgfoundry.org/frs/?group_id=1000040

Well, there was a little omission in the recipe for a git mirror or the PostgreSQL CVS repository I posted the other day. What I forgot to do was to add --delete to the rsync command to fetch the CVS copy. That was enough to leave junk lying around when files were deleted that made fromcvs do bad things, and I have been unable to get the repo to recover.

At this stage it it totally broken, and rather than waste more time on it I am going to blow the whole thing away and start again.

What a pity.

My buildfarm member brown_bat, which runs on Cygwin, is getting a bit long in the tooth now, so I recently tried to build PostgreSQL on the latest Cygwin. It builds OK and passes the core regression tests, but fails miserably on PLPerl. I asked about this on the Cygwin mailing list, and received this solitary reply:

This is because cygwin perl is compiled with gcc-4 and -fstack-protector and you are trying with gcc-3 without -fstack-protector.

Installing gcc-4 and doing the configure with CC=gcc-4 should do the trick.

Now, there are several things wrong with this. First, it seems to me that building a standard package with a compiler that is not even installed by default, let alone the default compiler for the system, is just asking for trouble.

And second it is just wrong. I tried building with gcc4, and after overcoming some serious build problems still got the same failure. I tried with a perl built with gcc3, and still got the same failure.

I have asked for more help on the Cygwin mailing list and the silence has been deafening.

Now there are people in the PostgreSQL community who sneer at Cygwin and suggest we should no longer support it. I don't like abandoning any platform, however. And the fact is that by far the best way I know of to run psql on Windows is to use Cygwin's psql, which has none of the buffering problems that the native Windows version has, and has a working readline to boot. Now, we don't need to build the server to build Cygwin. But the buildfarm at least is not set up for anything less than a full PostgreSQL build, server and all. So abandoning the server on Cygwin would mean abandoning automated testing for the client as well. So I'm quite reluctant to do that.

It would be nice if we got some help back from the Cygwin community, though.

I could build without plperl. We had to abandon pltcl on Cygwin some years ago. But why should we have to? There aren't a huge number of users of pltcl, but plperl has quite a large number, and that number is growing.

All in all this is a bit frustrating, and something that should have been extremely simple has chewed up some precious hours. I'm going to stop this now and go on with more urgent work. Maybe someone else can take it up.

I have been asked for the details on how I created and maintain the incremental git repo I am currently testing with the git-ized buildfarm client, and how I test the validity or the repo. Here is how it goes.

First, I have a local frequently updated copy of the base CVS repository on a machine called constanza, where the dungbeetle buildfarm client runs. The git stuff runs on a more modern machine called sophia. First I set up a public git repo on GitHub. Next I installed the fromcvs ruby software in /home/gitmirror. Then the local git repo was initialised like this:

cd /home/gitmirror
mkdir pg-cvsconv.git
cd pg-cvsconv.git
git --bare init
git remote add origin git@github.com:oicu/pg-cvs-mirror.git

Then I made a copy of the CVS repo, removed the CVS tags known to cause problems, and ran the import:

cd /home/gitmirror
rsync constanza:/home/cvsmirror/pg /home/cvsmirror >/dev/null
cvs -d /home/cvsmirror/pg/ rtag -d REL7_1_BETA2 pgsql >/dev/null
cvs -d /home/cvsmirror/pg/ rtag -d REL7_1_BETA3 pgsql>/dev/null
cvs -d /home/cvsmirror/pg/ rtag -d REL7_1_BETA pgsql>/dev/null
ruby togit.rb /home/cvsmirror/pg pgsql /home/gitmirror/pg-cvsconv.git
cd pg-cvsconv.git 
git push --all origin

These turn out to be exactly the steps needed to update the repository incrementally, too, so I now have these in a script called from cron every 30 minutes. It doesn't take long to run, usually a minute or two.

Finally, once a day I repack the repo which saves some on space. I did this after the initial import too, of course:

cd /home/gitmirror/pg-cvsconv.git
git repack -a -f -d

Then I have a mirror cloned from github thus:

git clone --mirror git://github.com/oicu/pg-cvs-mirror.git pgbase.git

This is what is used both to run the buildfarm members from, and to check the validity of the conversion. The validity script looks like this:

#!/usr/bin/perl

use strict;

my @branches = qw(
   REL7_4_STABLE
   REL8_0_STABLE
   REL8_1_STABLE
   REL8_2_STABLE
   REL8_3_STABLE
   REL8_4_STABLE
   HEAD
);

my $ts = localtime;

print "@@@@ $ts: fetching updates\n";
chdir "pgbase.git";
system ("git fetch");
chdir "..";
print "\n";

foreach my $branch (@branches)
{
        system("rm -rf pg-check-cvs-$branch");
        mkdir "pg-check-cvs-$branch";
        system("rm -rf pg-check-git-$branch");
        mkdir "pg-check-git-$branch";
        system ("cvs -d /home/cvsmirror/pg export -r $branch " .
                        "    -d pg-check-cvs-$branch pgsql >/dev/null 2>&1");
        chdir "pgbase.git";
        my $gbranch = $branch eq 'HEAD' ? 'master' : $branch;
        system ("git archive $gbranch | tar -C ../pg-check-git-$branch -xf -");
        chdir "..";
        $ts = localtime;
        print "@@@@ $ts: checking $branch\n";
        system ("diff -r --exclude=my2pg.pl --exclude=my2pg.diff " .
                        "    pg-check-cvs-$branch pg-check-git-$branch");
        print "\n";
}

foreach my $branch (@branches)
{
        system("rm -rf pg-check-cvs-$branch");
        system("rm -rf pg-check-git-$branch");
}

That's it

My recent complaint about fromcvs being broken is apparently no longer true. At the time the web site at http://ww2.fs.ei.tum.de/~corecode/hg was throwing errors, so I used the latest copy I had of the software to test, and reproduced the errors that are affecting the back branches of Postgres on the community repository at http://git.postgresql.org/gitweb?p=postgresql.git;a=summary.

Yesterday, following a clue from Kevin Grittner, I revisited the site and got the very latest versions of the fromcvs and rcsparse packages, and tried again. This time the back branches are working properly. Yay!

Now all we need is for the powers that be to fix the community repository ...