Black Bag Operations Network

Far and away the most painful thing I’ve done with a computer since dropping PDP-11 on my foot.

It was a simple idea. Create a new empty Centos 5.3 image and start installing goodies until it works. Three days later and I’m almost there but wow. First off – yum package manager – love the concept – wonderful when it works. Sadly, many useful things are not yum packages and many yum packages are out of date. yum install apache – fab. yum install postgresql went pretty well. Efforts to install the latest postgis from source – utter failure. It then took me about six hours to successfully eradicate the failed install bits that were messing me up and get the yum install working. Ultimately it required me to create symbolic links in odd places for the GEOS library, but I won in the end. Then there’s the issue of running the sql to create a database. This confounded me for a couple hours until I learned that this copy of postgresql shipped with very strange and restrictive pg_hba.conf. I’ve installed PostgreSQL at least a dozen times and this is the first time I’ve encountered this annoyance. The linked file is the best tutorial on use of this file I have found.

The primary annoyance being that, working as root to install PostGIS, I was not allowed to run the sql files as postgres user (ident failed), but as postgres user, I was not allowed to run sql files owned by root. pg_hba.conf file edited to sane defaults and voila PostGIS installed.

Getting a modern python and mod_wsgi installed was made easy using this tutorial. But getting mod_wsgi configured with virtualenv took forever, largely because I kept running into weird permissions problems with apache. It took the use of a big chmod/chown hammer to finally get it to work and I still don’t understand why setting apache to use a directory in my virtualenv as document root for static content doesn’t work, nor does putting a symlink to the directory in apache’s usual doc root.

Getting git installed and setup was straightforward and mostly works thanks to this handy guide. I haven’t set up the auth bit but I can at least sync my project to my development area on my laptop.

Python’s easy_install has been awesome for getting TurboGears2.1 and friends installed – most trouble free part of the whole thing. Finally, I need php for one little utility that generates labeled map pins like we find on Google maps. It is really cool, but it uses the gd support in php and all of the pre-built packages are compiled using –without_gd. So I had to build php from scratch and download dev versions of a whole lot of graphics processing goodies including libpng and freetype. Getting the build done wasn’t bad. Figuring out why it didn’t work in the first place was a drag though.

Probably one more day to get git auth working, sort out the static file permissions, and configure the git repos to know where origin is and I’ll be ready to actually build some software. That’ll be a nice change.

I need a new web app platform. I have to build a system that is geographically aware which points to using PostGIS as my database. The GIS crowd likes python. There are a bunch of nifty frameworks like MapFish and FeatureServer for building geographically aware applications. They are all built on top of Pylons which is more like a box of parts you can use to build web applications in python than a full on framework like Ruby on Rails.

There is also Django and a GIS extension called GeoDjango. I spent over a month futzing with Django before concluding it was a poor fit for application development. Django comes from the publishing world and is about building web sites. I need an application. There is no ajax integration out of the box, the template system is weak and clumsy to extend. While they claim you can swap it for another one, if you do none of the zillions of extensions that are out there work anymore. I clung to it a little longer than I should have because I thought that zillion extensions would save me a lot of time. Turns out, most of those extensions are half baked, unpolished, and more or less undocuented – thus worthless. I also liked The Admin and thought it would save me a lot of effort. But not enough. So – au revoir Django. You’re no replacement for Rails even if The Admin is a bit better than ActiveScaffold.

Pylons, OTOH, tries to maximize choices and make everything swappable. Choose your template system, database layer, url routing layer….. One gets the idea that the developers have seen Trainspotting one too many times. Anyhow, I don’t want to choose every little piece of the application architecture. I don’t want to design a url scheme. I don’t care that it works with 9 databases – just mine. I just want a cohesive framework that I can build in. Enter TurboGears, a Pylons based framework that tries to choose the best of everything and make it work together. I’m sold. It also has a CRUD based admin layer that is pretty sweet.

Now I just need to figure out how to build a stack for deployment on Centos 5.3.

After updating to Snow Leopard, apache stopped working. The console was full of:
9/25/09 8:53:15 PM org.apache.httpd[38614] Warning: DocumentRoot [/usr/docs/dummy-host.example.com] does not exist
9/25/09 8:53:15 PM org.apache.httpd[38614] Warning: DocumentRoot [/usr/docs/dummy-host2.example.com] does not exist
9/25/09 8:53:15 PM org.apache.httpd[38614] Syntax error on line 99 of /private/etc/apache2/extra/httpd-ssl.conf:
9/25/09 8:53:15 PM org.apache.httpd[38614] SSLCertificateFile: file '/private/etc/apache2/server.crt' does not exist or is empty
9/25/09 8:53:15 PM com.apple.launchd[1] (org.apache.httpd[38614]) Exited with exit code: 1
9/25/09 8:53:15 PM com.apple.launchd[1] (org.apache.httpd) Throttling respawn: Will start in 10 seconds

Apparently, Apple decided to blow away my very custom apache2 config and replace it with a generic one. Not cool. As a developer of many web sites, I have a virtual host for every project of the form localhost.domain.tld. These were all gone.

Also, the php was broken. The fix included first downgrading php which required downloading and recompiling every single open source library I had (dozens) because of the new 64 bit object file format.

Then, finally I got the above error and using Time Machine recovered my old httpd.conf and extras folders from my pre-Snow Leopard backup and its all working again.

Updating to Snow Leopard broke my mysql install. Pro-tip: back up your databases before updating to Snow Leopard.

I didn’t do this. To get the data back – recreate the /usr/local/mysql -> /usr/local/mysql_versionbuilddatanumberstuff symbolic link. This gets broken in the install.

I then clobbered the /Library/LaunchDaemons mysql item, and the /Library/StartupItems one. Now mysqld will not launch by default.

Rebooted.

sudo cp /usr/local/mysql/support-files/my-small.cnf /etc/my.cnf

mysqld_safe &

Now I am able to run

mysqldump -u root -A > saved.sql

to dump the databases. Now I downloaded the Mac OS X 64 bit mysql package and run the installer then reboot.

Now load my data into the new server using

mysql -u root < saved.sql

And I'm off to the races.

Not really a service pak. Just no eye candy.
http://arstechnica.com/apple/reviews/2009/08/mac-os-x-10-6.ars

In other news, people doing audio production have found SL to be much more responsive and a whole bunch of glitchy behavior just vanished. I’d buy that for $30.

Just waiting for the UPS guy with the disk…

Well its taken a long time, but I’ve finally gotten around to figuring out why permalinks were broken (it involves getting apache mod_rewrite and various file permissions sorted out).

So maybe the blog is fixed and I’ll start using it again.

OTOH, I’m not super happy with WP lately. It seems like it is collapsing under its own weight. I need to set up something for Musical Freak Show and I’m thinking about trying out Chyrp.

Meanwhile, I’ve got my YouTube channel up and running and have posted my first video. So I’m gradually figuring it out.

I’m fed up with git. I know its the hot new Linus developed ultra distributed no master repository version control system with extra dietary fiber these days. But overall, the user experience sucks and the tool provides no help at all and I end up in the weeds with it daily working with just one other developer. I would not choose it again.

First off – there’s a zillion tutorials for it – all incomplete and leaving out crucial details – which makes finding information about it worse than having no tutorials. You can waste hours looking at trivial blog posts about git that don’t answer the key question about how to manually resolve conflicts.

Second, git doesn’t seem to deal with XCode project files at all. Trying to tell git to ignore them via the config flags doesn’t seem to work. Our project files are ALWAYS in conflict and while we keep resolving them by doing git add – we never get a chance to try to reach common ground and this is actually worse than having no version control on the project file.

Third, I have yet to figure out how to get git to actually put the differences in a file so I can fix them. It continually complains about merge conflicts without actually putting the conflicts where I can find and fix them.

So, I’ve come to conclude that I just don’t “get” git’s model (despite reading a few dozen papers around the web on it) and I never had nearly so much trouble with plain old CVS. So I don’t see what problem git “solves” but I do see a large number that it creates.

Maybe there’s a book somewhere putting all the pieces together…

Reprinted from my old blog – written in 2002.

I’ve been having a discussion with Amy Wohl on this copyright stuff. I don’t think I’ve quite made the point I’m trying to make. I think I’m a witness to one of the greatest hypocracies ever placed squarely in the public eye.

Perhaps I’m in a unique position. It could be that
I have special powers of insight. I can’t tell.
I know this much. I’ve been a professional
performing and recording musician as well as
songwriter. My one public triump was winning a local
radio sonwriting contest in Albuquerque New Mexico
around 1985. I was proclaimed “The hottest band in
New Mexico”. Technically it was a recording project
performed by myself, computers and friends.

I am presently a professional software sytems architect and developer.

And I am amazed.

The very short thesis is that somedevelopers are simultaneously calling for greater access to music, video and entertainment by while insisting that software is different and must be more protected.

The argument for pushing for online access to music is that the increased sales more than compensate for any losses from pirating. The recording industry is labeled as fearful and unwilling to trust the users. Those music people ought to be ashamed!

The take away is that improved access to creative works increases sales. I think this is true.

These same people bristle at the suggestion that improved access to their creative works might also improve sales. Now we’re holding a gun to their head they say. Releasing my ten year old source code will ruin me!

The software industry is trying to rely on the same mechanisms that the recording industry relies on. That copying a work is technically infeasible and should stay that way. The music industry used to keep control by releasing inferior copies that degraded with successive generations of copying. The software industry keeps control by refusing to release source code.

With the advent of digital music, the music industry has lost its built in copy protection. Yet the software people say “have courage”. “Increased exposure will increase your sales.” The numbers seem to imply that this is true.

Of course, if we turn the tables and require the same of software developers they cry “foul”.

Fueling the fire is the rising indignation regarding the ever growing term of copyright. A couple of legal luminaries have begun to fight this on the grounds that the public good is being subverted in the name of corporate profits. A really good explanation of why copyright terms aren’t forever is described in this paper by legal professor Glenn Reynolds.

For those with short attention spans, the paper basically says providing permanent copyright impedes progress because nobody can afford to build on the work of the past. But no copyright impedes progress because nobody is incented to produce new inventions if they can’t make money from them. So a middle ground – limited term copyright (or patents – these also expire) – allows inventors to both build upon the work of the past while getting paid for their work. Its a good mechanism that has been subverted in the name of corporate profits. Because long term copyright holds our culture hostage.

A great example is the Happy Birthday song. It is said to be among the top 3 best known songs in the english speaking world. But it remains encumbered. Unlike the music of the great composers which is all in the public domain now, the song Happy Birthday, truly a core part of our culture, remains a hostage of Warner Communications. This is the best example of copyright abuse I know of.

Of course the software people all agree. Those greedy corporations! Free Mickey Mouse!

The upshot of this uproar is that people are beginning to realize that long term copyright holds our culture hostage, that creative works ought to eventually pass into the public domain.

Even software works.

But they don’t. When a software company goes bust or simply changes direction, old products die. They don’t get updated. The source code disappears. Files written by the software containing content owned by other people become inaccessable. Imagine if this happend to all of our creative content.

The original music business was so primitive that songwriters made money by selling you the sheet music to the song and you had to perform it yourself. Completely open source. Still, Stephen Foster and Scott Joplin made plenty of money in spite of sharing. Today, their works are in the public domain and they are a much loved part of our history and tradition.

Sadly, this isn’t happening with software. Although if Lessig gets his way, that will change and source code will be escrowed to be released into the public domain at the expiration of copyright. This is the duty of the copyright holder and beneficiary.

I suspect that part of the fear among closed source developers is that their work will be borrowed piecemeal and new products built upon it without their being compensated. But they get a ten year lag before that will happen. Thats better than Van Halen got when Tone Loc sampled the opening drum riff for “Jamie’s Cryin” for his “Wild Thing”, or the deal David Bowie and Queen got when Vanilla Ice borrowed the signature riff from “Pressure” for his “Ice Ice Baby”.

It would be very hard to argue that Van Halen lost any record sales to Tone Loc or that David Bowie and Queen lost sales to Vanilla Ice. The financial loss isn’t real. In much the same way that casual Napster downloaders weren’t going to buy the music they pirated anyway, the loss is impossible to quantify in any financial way.

Its called fair use and in general its good for everybody. But it takes courage to see it. Source code must pass into the public domain on expiration of copyright. Its the right thing to do for everybody.

When I was a kid, kids in my neighborhood took halloween seriously. I mean really seriously. After experiencing a modern halloween through the eyes of my daughter, I thought I’d share some thoughts on how to properly “do” halloween. Yes this is late – file it for next year.

Rule #1: Go as a kind of humanoid or anthropmorphised animal. Do not go as some kind of box. Who wants to wear a box all night? If you go to a party, how much fun will you have trying to dance in a box? Can you get into a car and sit down? If not, your costume needs a rethink.

2: Makeup – not masks. There’s the safety/visibility thing. But there is also eating/drinking (if you go to a party) and general level of comfort. Its loads cooler when you use some spirit gum and latex appliances and makeup and you can make facial expressions – than when you simply hide behind a vacuum formed plastic or rubber facade. Plus, you’ll be able to breathe. Exceptions for characters that inherently wear masks – like Darth Vader.

3: Make your costume – don’t buy it. Buy elements, sure. But put them together to create something unique. Last night I saw half a dozen cookie cutter zombies, Darth Vaders, Bobba Fetts, etc… Most of the fun when we were kids was the challenge of coming up with a costume that would out do all of our friends. That took creativity. Especially since most of the ready made stuff available now wasn’t available when I was a kid. If you wanted your guts hanging out you’d buy a mop head and spray paint it red and hang it out a tear in your shirt or something. That was fine. FWIW, I have a double candy for home made costumes policy at our house.

4: I hesitate to mention this one – if you’re not going to be home on halloween, or you don’t want to play – fine – turn out all the lights. Saves time for the kids and your house is LESS likely to be vandalized. When I was a kid, people who didn’t play and left their lights on got their windows soaped and their trees TP’d. Er – so I heard. If you are gonna play – put a jack-o-lantern on the porch. That’s kind of a nice hobo sign that makes it easy to see who is in.

5: Trick or Treating is to be done in the dark. That’s the point. Kids get flashlights, glow sticks, whatever and go door to door in the dark – when they’re most likely to get scared. I appreciate the retail areas having daytime activities – but this is warmup. The real event is finding a stereotypical suburban neighborhood and running from door to door in the dark where you can scare yourself and your friends silly while stocking up on enough candy to last until Christmas.

6: Final beef. Halloween is supposed to be scary. Efforts by the care bear crowd to make it cutesy are not appreciated. When I lived in the ‘burbs and had a steady stream of victims I worked hard to make the house truly frightening to approach. I considered it a triumph if small kids refused to approach without a parent. Bonus if little ones cried. Being scared is fun – like campfire ghost stories and death defying roller coasters are fun. About my third year of this I had people drive up, park across the street – visit my house – get in their car and drive away. This is the sign of a truly successful house. I know not everyone can go to this level – but do try to make your house a little creepy – put a colored bulb in the porch light and carve a scary pumpkin or two. If you have a scary sound effects record – put it on.

Happy Halloween

This server was moved after disk trouble. Let me know if you spot any problems. I think I have everything back up and running.