Black Bag Operations Network

Far and away the most painful thing I’ve done with a computer since dropping PDP-11 on my foot.

It was a simple idea. Create a new empty Centos 5.3 image and start installing goodies until it works. Three days later and I’m almost there but wow. First off – yum package manager – love the concept – wonderful when it works. Sadly, many useful things are not yum packages and many yum packages are out of date. yum install apache – fab. yum install postgresql went pretty well. Efforts to install the latest postgis from source – utter failure. It then took me about six hours to successfully eradicate the failed install bits that were messing me up and get the yum install working. Ultimately it required me to create symbolic links in odd places for the GEOS library, but I won in the end. Then there’s the issue of running the sql to create a database. This confounded me for a couple hours until I learned that this copy of postgresql shipped with very strange and restrictive pg_hba.conf. I’ve installed PostgreSQL at least a dozen times and this is the first time I’ve encountered this annoyance. The linked file is the best tutorial on use of this file I have found.

The primary annoyance being that, working as root to install PostGIS, I was not allowed to run the sql files as postgres user (ident failed), but as postgres user, I was not allowed to run sql files owned by root. pg_hba.conf file edited to sane defaults and voila PostGIS installed.

Getting a modern python and mod_wsgi installed was made easy using this tutorial. But getting mod_wsgi configured with virtualenv took forever, largely because I kept running into weird permissions problems with apache. It took the use of a big chmod/chown hammer to finally get it to work and I still don’t understand why setting apache to use a directory in my virtualenv as document root for static content doesn’t work, nor does putting a symlink to the directory in apache’s usual doc root.

Getting git installed and setup was straightforward and mostly works thanks to this handy guide. I haven’t set up the auth bit but I can at least sync my project to my development area on my laptop.

Python’s easy_install has been awesome for getting TurboGears2.1 and friends installed – most trouble free part of the whole thing. Finally, I need php for one little utility that generates labeled map pins like we find on Google maps. It is really cool, but it uses the gd support in php and all of the pre-built packages are compiled using –without_gd. So I had to build php from scratch and download dev versions of a whole lot of graphics processing goodies including libpng and freetype. Getting the build done wasn’t bad. Figuring out why it didn’t work in the first place was a drag though.

Probably one more day to get git auth working, sort out the static file permissions, and configure the git repos to know where origin is and I’ll be ready to actually build some software. That’ll be a nice change.

I need a new web app platform. I have to build a system that is geographically aware which points to using PostGIS as my database. The GIS crowd likes python. There are a bunch of nifty frameworks like MapFish and FeatureServer for building geographically aware applications. They are all built on top of Pylons which is more like a box of parts you can use to build web applications in python than a full on framework like Ruby on Rails.

There is also Django and a GIS extension called GeoDjango. I spent over a month futzing with Django before concluding it was a poor fit for application development. Django comes from the publishing world and is about building web sites. I need an application. There is no ajax integration out of the box, the template system is weak and clumsy to extend. While they claim you can swap it for another one, if you do none of the zillions of extensions that are out there work anymore. I clung to it a little longer than I should have because I thought that zillion extensions would save me a lot of time. Turns out, most of those extensions are half baked, unpolished, and more or less undocuented – thus worthless. I also liked The Admin and thought it would save me a lot of effort. But not enough. So – au revoir Django. You’re no replacement for Rails even if The Admin is a bit better than ActiveScaffold.

Pylons, OTOH, tries to maximize choices and make everything swappable. Choose your template system, database layer, url routing layer….. One gets the idea that the developers have seen Trainspotting one too many times. Anyhow, I don’t want to choose every little piece of the application architecture. I don’t want to design a url scheme. I don’t care that it works with 9 databases – just mine. I just want a cohesive framework that I can build in. Enter TurboGears, a Pylons based framework that tries to choose the best of everything and make it work together. I’m sold. It also has a CRUD based admin layer that is pretty sweet.

Now I just need to figure out how to build a stack for deployment on Centos 5.3.

After updating to Snow Leopard, apache stopped working. The console was full of:
9/25/09 8:53:15 PM org.apache.httpd[38614] Warning: DocumentRoot [/usr/docs/dummy-host.example.com] does not exist
9/25/09 8:53:15 PM org.apache.httpd[38614] Warning: DocumentRoot [/usr/docs/dummy-host2.example.com] does not exist
9/25/09 8:53:15 PM org.apache.httpd[38614] Syntax error on line 99 of /private/etc/apache2/extra/httpd-ssl.conf:
9/25/09 8:53:15 PM org.apache.httpd[38614] SSLCertificateFile: file '/private/etc/apache2/server.crt' does not exist or is empty
9/25/09 8:53:15 PM com.apple.launchd[1] (org.apache.httpd[38614]) Exited with exit code: 1
9/25/09 8:53:15 PM com.apple.launchd[1] (org.apache.httpd) Throttling respawn: Will start in 10 seconds

Apparently, Apple decided to blow away my very custom apache2 config and replace it with a generic one. Not cool. As a developer of many web sites, I have a virtual host for every project of the form localhost.domain.tld. These were all gone.

Also, the php was broken. The fix included first downgrading php which required downloading and recompiling every single open source library I had (dozens) because of the new 64 bit object file format.

Then, finally I got the above error and using Time Machine recovered my old httpd.conf and extras folders from my pre-Snow Leopard backup and its all working again.

Updating to Snow Leopard broke my mysql install. Pro-tip: back up your databases before updating to Snow Leopard.

I didn’t do this. To get the data back – recreate the /usr/local/mysql -> /usr/local/mysql_versionbuilddatanumberstuff symbolic link. This gets broken in the install.

I then clobbered the /Library/LaunchDaemons mysql item, and the /Library/StartupItems one. Now mysqld will not launch by default.

Rebooted.

sudo cp /usr/local/mysql/support-files/my-small.cnf /etc/my.cnf

mysqld_safe &

Now I am able to run

mysqldump -u root -A > saved.sql

to dump the databases. Now I downloaded the Mac OS X 64 bit mysql package and run the installer then reboot.

Now load my data into the new server using

mysql -u root < saved.sql

And I'm off to the races.

Not really a service pak. Just no eye candy.
http://arstechnica.com/apple/reviews/2009/08/mac-os-x-10-6.ars

In other news, people doing audio production have found SL to be much more responsive and a whole bunch of glitchy behavior just vanished. I’d buy that for $30.

Just waiting for the UPS guy with the disk…

Well its taken a long time, but I’ve finally gotten around to figuring out why permalinks were broken (it involves getting apache mod_rewrite and various file permissions sorted out).

So maybe the blog is fixed and I’ll start using it again.

OTOH, I’m not super happy with WP lately. It seems like it is collapsing under its own weight. I need to set up something for Musical Freak Show and I’m thinking about trying out Chyrp.

Meanwhile, I’ve got my YouTube channel up and running and have posted my first video. So I’m gradually figuring it out.