technology « Black Bag Operations Network

Archive for the ‘technology’ Category

On Database Initialization Scripts

Friday, November 10th, 2006

When building a new system that has a database, there will be a collection of tables that are reference tables. Lists of possible values that contain human readable versions of numeric codes. An example might be a status code for an order. The statuses might be OPEN, PENDING, CANCELLED, SHIPPED. Well written applications will make use of these reference tables to populate the user interface pick lists, thus allowing a new status, like RETURNED, to be added quickly by just adding a row to the database.

Order Status
ID	Name
1	Open
2	Pending
3	Cancelled
4	Shipped
5	Returned

Of course, that means that those tables must be populated. There are right ways and wrong ways to go about this. The worst thing you can do is to simply sit down at a terminal and start typing INSERT statements right to the database. If you do this, you have no reproducibility. You may have several instances of this database, for instance a test version and a production version. Being able to build an application database from a virgin install is critical. So write scripts, keep them in version controlled files, and keep them up to date. If, somewhere down the line, you do add a status code for RETURNED, you should just update the script for that table and run it.

Which brings me to the part that is frequently overlooked and that messes up a lot of people. Database initialization scripts must be written to be IDEMPOTENT. Idempotency is that property that guarantees that performing an action multiple times has exactly the same result as performing it once. Each table initialization script should be IDEMPOTENT.

This means checking for the table and if it is missing, create it. Check for each record and if it is missing, insert it. Hard code the primary key in your reference script so it will not change. If there is a sequence involved, be sure to reinitialize it appropriately.

The reason for this is that, one day, some idiot will accidentally delete or damage part of your database’s reference data, either through coding error or just mistyping commands. Probably, the idiot will be you. You’ll look like much less of an idiot if you can just whip out your trusty database initialization script and know you can instantly repair the damage because your scripts are all written to be IDEMPOTENT.

Posted in programming, software, technology | No Comments »

At the Dynamic Languages Symposium

Monday, October 23rd, 2006

Which kicked off with a bang with Ian Piumarta’s COLA family/project. It seems he has distilled the machinery required to implement objects with late binding into a half dozen operations and a couple of “things”. Then, starting at this level begins building back up to existing dynamic languages – but since everything is implemented in a tiny kernel of primitives, the results are tiny and profound.

After that talk, many of the other talks were just dull and seemed so last century. For instance, the talk on PyPy, a Python implementation written in a subset of Python with code emitters targeting C, Java, and .NET, felt like a huge regression – Squeak did that 10 years ago – now COLA seems to be the next step while PyPy seems to be going over old ground. In fact, it seems the COLA strategy is likely to yield a smaller, faster, and more portable implementation with more powerful introspection.

Posted in software, technology | 4 Comments »

I Love You – Now Change

Monday, October 2nd, 2006

Which company would you prefer to invest in?

Company A:

“Our company is a dynamic organization with the agility to quickly adapt to new market conditions.”or

Company B:

“Our company is stable, well structured, and organized. What we are doing now is a perfect basis for everything we will do in the future.”(Sounds a little like the Bush administration)

The dynamic organization that can change quickly is going to be more successful than a static organization that is set in its ways.

So how come the software industry pundits continue to try to push static programming systems over dynamic ones when dynamic systems are generally more successful? Its senseless.

In C++ and Java, the assumption is that the superclass designer knows best. Whatever interface the original developer has exposed is expected to be the perfect and complete interface for all time. There is no way to extend an existing interface without owning the source code to it.

The implementation, it is recognized, might not be exactly correct in all cases, so the implementation is left open to extension via the one mechanism made available – subclassing (maybe – Java actually allows the most arrogant developers to forbid subclassing via the ‘final’ keyword).

The problem with leaving only subclassing is that subclassing, by itself only provides for extension of the system, not for modification. Fans of Robert C Martin or Bertrand Meyer might recognize this as The Open Closed Principle. Sadly, The Open Closed Principle is only works if you happen to work for a static company like Company B.

The harsh reality is that organizations are organic – they evolve and grow to adapt to new environment conditions. Failure to evolve is death. How can you modify your organization if the software that runs your organization is closed to modification by design? Worse, the underlying tools and technology on which your software is built actually work to enforce The Open Close Principle.

So how else to evolve your system? Objective C has a construct called a Method Category or more commonly just “category”. A category is a collection of methods for a class that may be loaded dynamically – or not. These are collections of additional methods to be added to existing classes. These additions may be made part of the organizations core software assets, or they can be application specific extensions that are too specialized for general consumption.

For instance, a web services application may find it convenient to add some methods to the string class for parsing up web requests, but the billing system doesn’t need this category of methods and so doesn’t bother to load it.

Categories can also make adapting an existing class to a new protocol easy. Not having a number of separate adaptor classes all over the place keeps the number of classes low and the conceptual size of the application architecture smaller.

Finally, categories can allow the user of a class to replace a buggy or inappropriately implemented method with a new implementation without having the source code.

Another useful tool is the ability to replace one class with another. The Objective C tool for this is known as “posing”. One writes a subclass of the original class and then says

[NewClass poseAs: [OldClass class]];

Now saying [OldClass new] actually constructs an instance of NewClass. This can be handy for sneaking superclasses into the class hierarchy and also for debugging around code you don’t own.

Using these techniques, along with message forwarding and delegation, subclassing takes a back seat to application assembly and drops from the most used tool to the mechanism of last resort. After all, its much better to simply arrange the classes you already have into the right structure than to create entirely new code with entirely new bugs.

Smalltalk has similar mechanisms and results in similar designs. Method categories exist and can be loaded as packages. Posing is done quite easily by replacing a class in the Smalltalk class dictionary with another class and executing a become: on all of the old classes instances. Its easy to insert new classes anywhere in the hierarchy and all of the code is easily accessible and modifiable. Subclassing in Smalltalk application development is a relatively rare event.

Of course, if you’re sure you know what you’re doing – perhaps Java and C++ are the right languages. If you’re sure, that is.

Posted in C++, Objective C, Uncategorized, java, programming, smalltalk, software, technology | No Comments »

Blanchard’s Law

Tuesday, September 5th, 2006

When in the course of developing software, the growing feeling that code generation would be a great idea actually means that the environment or language in which you are working isn’t powerful enough for the task at hand.

Years ago I was a C++ fan. It was the first language I encountered that had user definable classes and objects and we used it to build Motif GUI’s that fronted legacy character based systems. After awhile we found that the amount of boilerplate code we were writing was becoming an impediment to change and someone came up with the idea of specifying a number of aspects of the system as models and generating the code from the model.

As I was the lead of the business domain model, this approach was particularly attractive to me as we had dozens of classes that were all implemented using the same idioms, but whose set of attributes changed in minor ways with some frequency. Furthermore, we weren’t actually implementing the business logic within the classes as they were being mapped into an inference engine that provided enforcement of cross object validation rules. So code generation seemed like a great labor saving solution.

Problems occurred fairly soon after with generated code being occasionally hand modified by some maintenance programmer and changes being lost down the road. The generated code had size impacts as well. It’s easy to generate a program too large to compile and run on your current hardware. Our compile times for the system went from 2 to over 30 hours – a problem we solved by parallelizing the build process across a fleet of machines. The project was ultimately cancelled and we didn’t have a chance to see if the giant could fly. But I was left with some lingering doubts as to the efficacy of generating code.

Later on I encountered CORBA. CORBA seemed pretty cool – like Remote Procedure Calls (RPC) but a little easier to use. Interface Definition Language (IDL) looked just like C++ with a couple minor tweaks. You generated your stubs, filled in your code, and you were off. At least until you wanted to expand or change an interface. Maintaining all the generated source files got cumbersome and once again added considerably to the build times.

Java RMI was the same kind of thing – define Java interfaces (instead of IDL) and a whole bunch of marshalling code gets generated. But you end up with an awful lot of code in the end.

A bit later in my career, I encountered dynamic languages. I got a glimpse of HP’s DST (Distributed Smalltalk) which implemented CORBA using a single proxy object that could stand in for any object. NextStep’s Portable Distributed Objects (PDO) was a similar take on this theme. These smart proxies were made possible by the dynamic typing and message sending nature of these languages. Static languages were stuck with code generation because they had to satisfy the compiler’s type constraints.

Consequently, I view the growing trend towards code generation with some horror. No wonder our systems are insanely resource intensive while delivering very little value. After a decade and a half developing software, I have yet to encounter a situation where code generation is a good idea in the end. The answer isn’t to automate your inefficiency. If this is not possible, then it is time to evaluate different tools and approaches.

Posted in C++, Objective C, Uncategorized, java, smalltalk, software, squeak, technology | 3 Comments »

BadPage.info moved to BadPage.net

Tuesday, August 29th, 2006

Spammers have been hammering BadPage.info (also hosted on this machine) with click fraud traffic in hopes that the proxy will forward it.Â It did for a few days – but I’ve locked it down and it won’t anymore.

That didn’t stop the spammers from continuing to pound the machine so I instituted dynamic countermeasures. Â This helped for awhile but now the machine seems to be spending all of its time figuring out which traffic to drop.Â Bandwidth dropped to about 20k/s.Â Enough is enough.Â I parked the domain and moved the app to BadPage.net.Â Let NetworkSolutions handle the traffic for awhile.

Posted in technology, web | No Comments »

Closures for Java?

Thursday, August 24th, 2006

Gilad Bracha of Strongtalk fame has a post talking about the proposed introduction of closures to Java.

I consider Java to be so broken at this point that I view it as “lipstick for a pig” and agree with Bracha’s position that its unfortunate that they were delayed to the point that all the common idioms that might have been done with closures have been established using less elegant techniques. In other words, I think its too late.

Its a chain of constraints produced by a couple of root decisions. First was the insistence on mandatory manifest typing. The Strongtalk system and Objective C language both support optional manifest typing. Objective C does it to provide compile time warnings for programmers to help them catch mistakes. Strongtalk actually uses the type information to call type optimized versions of operations. This technique is part of the foundation of the HotSpot JVM.

The second design problem is Java’s choice of C style function calling semantics vs Smalltalk’s message sending. Message sending is the more flexible and uniform technique. You have an entity at some distance, you send it a message asking it to do something. If it knows what you are asking, it does it and replies with your result. If it doesn’t know what to do, some default action occurs. This is how network available services perform and is also how Smalltalk objects interact.

Java, C++, C take the position that the compiler has assured that the receiver is of a type that MUST understand the message you are sending. They do this by constraining the messages you are allowed to send at compile time. This is meant to prevent errors but it also needlessly limits program flexibility and forces the developer to deal with a second interaction model – one for network resources and another for local ones. The Smalltalker has only one interaction model to contend with.

This is important and profound. In fact, Alan Kay, the man who coined the term “Object Oriented” laments the choice of words saying: “Object-oriented programming is about messages, not the objects. We worry about the objects, but it’s the messages that matter.”

But Java is function-call oriented. Consequently, the closures end up looking like local function declarations – which means they are awkward and ugly to declare and use inline and they are littered with extraneous type information. And they will most likely not catch on.

Java is what it is. Like a mutant frog it is trying to make it to land, asymptotically approaching Smalltalk, a language first released in 1980, and yet, its clear that it can’t get there and I suspect it will die trying. At least, I hope it will.

Posted in java, smalltalk, software, technology | No Comments »

Function Calling vs Message Sending

Thursday, August 24th, 2006

“What happens if your reference refers to a different type of object than you expect?”

Its probably a programming error.

Or not.

In C++, because of the way vtable dispatching works, the program will either crash (hopefully) or the wrong member function will be called. Either way – the end result is probably fatal.

Java is slightly better. In order to call a member function, the java compiler checks to make sure that the function is defined in one of the interfaces implemented by the type of the object reference. The problem arises when the object reference has been widened to a more general type that does not define the desired member function.

How can this happen?

public class A extends Object { public void anAThing() {} … }

A myA = new A(); // make an A
someList.add(myA); // put it in a list … someList.get(0).anAThing; // error – List.get(int) returns Object.

so we have to downcast the Object reference returned by someList.get() to an A reference

((A)someList.get(0)).anAThing(); // Might be OK if someList has an A

Casting is completely type unsafe. Although in Java the result of incorrect casting is an exception. So we could write:

try { ((A)someList.get(0)).anAThing(); }
catch(ClassCastException ex) true

which allows us to handle the casting error and continue. This is a huge step up from the C++ behavior of crashing in that it allows the programmer some control over what to do if the cast is wrong. If we desire a more conventional means of writing this, it is possible to use the instanceof operator and do the check before the cast.

if(someList.get(0) instanceof A) ((A)someList.get(0)).anAThing();

which is pretty much the same thing. Of course, this forces programmers to clutter their code with all sorts of tests or try/catch blocks. It seems to me that getting away from that sort of thing was precisely the reason everybody wanted to switch to object oriented programming in the first place. Polymorphism replaced an awful lot of if ladders and switch statements and here we are putting them back in to work around a runtime system that throws an exception if we guess incorrectly about the type of an object.

Not to mention the idea that its quite possible to have this situation:

public class A extends Object { public void doAThing(); …}
public class B extends Object { public void doAThing(); …}

Object myB = new B();
((B)myB).doAThing(); // fine
((A)myB).doAThing(); // error – object is not an A!

which seems just silly. We have an object that we are pretty sure implements the operation doAThing – but that’s not good enough. We have to know exactly which interface this particular object implements in order to call that method. Thus, type defines protocol in the statically typed world.

The problem is that such an arrangement assumes the existence of what Bart Kosko calls crisp sets and hierarchies. The world isn’t nearly so neat. Its fuzzy. B might well be capable of performing some of A’s operations and in some (but perhaps not all) circumstances, B might be an excellent stand-in for an A. Its a more accurate model. To quote Kosko again. “fuzz up – precision up”.

OK, so what about the dynamically typed languages? How are they better? Assume we have the same classes A and B derived from Object and each implements doAThing.

| anA |

anA := A new.
anA doAThing.

anA := B new.
anA doAThing.

anA := ‘this is a string’.
anA doAThing.

This all works except for the last line when anA refers to a String rather than an instance of A or B. String definitely doesn’t implement doAThing. So what happens?

First, it helps to know that the runtime systems for Smalltalk and Objective C are “message sending” rather than “function calling”. When the programmer tries to send a message to an object that doesn’t respond to that message, the runtime packages the message up as an object and calls a catch-all message instead. In Smalltalk this is usually called doesNotUnderstand: message.

In Objective C, a special message called forwardInvocation: is called to give the programmer a chance to send the message to some other object such as a delegate. The default implementation of forwardInvocation: doesn’t do any forwarding. Instead it just calls another method doesNotRespondToSelector: which raises an exception.

The programmer may choose to respond to these messages in a class specific way – polymorphically, by overriding forwardInvocation: or doesNotUnderstand:

Doing this moves the error handling to a central location rather than forcing the programmer to scatter it throughout the code at the call locations. The end result is cleaner, smaller code.

Plus there’s a bonus. Being able to forward messages provides a clean mechanism for building chain of command patterns and allows an object to be “decorated” with new behaviors dynamically.

Its also easy to do distributed computing by having the forwardInvocation method perform remote procedure calls over the network without the need to do clumsy code generation of proxies and stubs common in C, CORBA, and Java RMI programs. A single proxy class can stand in for any kind of object.

The doesNotUnderstand: message can also provide a trigger for database fetching and object faulting. When a message is sent to a simple database query object that implements almost no messages, doesNotUnderstand: is invoked, the database query is executed, and the object replaced with the results of the fetch. The message is then delivered to the newly fetched object. Such faulting mechanisms can simplify programming and virtually eliminate the need for application programmers to directly interact with a database API.

These extra capabilities are nearly impossible to implement in the statically typed environments and this is clearly a case when the dynamically typed environment yields simpler application code (we don’t have all those try/catch blocks or instanceof tests). Simpler application code means greater reliability with reduced programmer effort. This all translates to faster development times and lower costs.

Posted in C++, Objective C, java, programming, smalltalk, software, technology | 4 Comments »

This server is under attack

Wednesday, August 23rd, 2006

And has been for the last week. I was using mod_proxy to proxy requests to BadPage.info and, due to the lack of sensible documentation (which I still don’t quite understand – a few examples would be useful apache foundation), my web server was turned into a spam forwarding machine.

I’ve disabled proxies (and thus BadPage.info is off the air just now). I’m still getting hammered from indousa.us sending malformed http requests in hopes of them turning into spam. Since I’ve set up explicit deny rules for them, I’m hopeful they’ll check their logs soon and get the hint.

If you’re in the mood to test a server loading script, indousa.us seems like a dandy test bed. Give them my regards.

Update: As soon as I deny that domain, a dozen others arise to take its place. Anyone with extensive skill on the use of mod_rewrite and mod_proxy that cares to help, please give me a shout.

Update: Solved it I think. BadPage.info is back on the air but all other proxy requests are being denied. I am still being hammered though.

Posted in technology, web | No Comments »

Marooned on Gilligan’s Island

Thursday, August 17th, 2006

Just sit right back and you’ll hear a tale….

Gilligan’s Island ran for 3 years and nearly every episode was roughly the same. Some discovery (read “new technology”) provides the castaways with the seed of a new approach towards getting rescued. The approach is tried, there are inevitable problems with the execution (generally introduced by Gilligan), failure ensues, and the plan is forever abandoned in favor of a completely new approach.

In 3 years of regular television, 96 episodes, the castaways worked their butts off on an amazing variety of novel ideas and still they never managed to get anywhere.

If you develop software for living, this starts to sound familiar.

A more logical approach for the castaways would be to pick a promising approach (building some sort of sea-going vessel perhaps) and refining it until it works. Its not nearly as entertaining, but its much more likely to succeed. Of course, the writers and producers are actually to blame – they introduce the plot devices and plans to keep the audience interested in trying to solve the same problem every week.

Of course, the big name producers in the software industry (Sun, Microsoft, IBM, HP) are doing the same thing to their audience (the sofware developers and IT managers). The new and novel is preferred over the tried and true because…, well because new things are just so much more entertaining! Its much more fun to do something new. Even if the old way is predictable and known to work!

Some of the shiny new things we’ve adopted and abandoned: Computer Aided Software Engineering (CASE), Artificial Intelligence (AI), Object Oriented Programming (OO), Virtual Machines, Garbage Collectors, Electronic Data Interchange (EDI), List Processing (LISP), Client-Server, n-tier, O/R Mapping tools, SGML, HTML, Web Applications, XML, .NET, the list goes on and on.

Some of these are languages, some core technologies, and some are architectural approaches. What they have in common is that the hype accompanying their introduction spurred large numbers of developers to drop whatever they were doing just as they began to master it, and run to embrace the next shiny toy. So much for the benefits of experience.

Interestingly, none of these falling stars have been completely abandoned. In fact, many of them are still around. OO programming was the panacea of choice for awhile. While the component world hasn’t quite emerged (actually – several competing worlds exist – COM, CORBA, RMI, PDO…) OO programming is now nearly ubiquitous.

AI, also a grossly overhyped technology and now considered a ‘failure’ by the purveyors of ‘conventional wisdom’ – is still making contributions towards the development of intelligent applications in the areas of manufacturing control, loan application evaluation, and consumer fraud detection. Certainly it didn’t quite live up to its hyped image, but the techniques developed to enable automated reasoning are extremely valuable where they are applicable.

Programming languages are another frequently used plot device. This used to be less of an issue. Some time before C++ started to proliferate, the developers of most operating systems defined a standard binary object file format that all compilers would generate. It was common to use FORTRAN math libraries from C, Pascal, or COBOL programs. You could mix or match to your heart’s content. The object formats were the same so the linker didn’t care.

The introduction of C++, in a misguided effort to make use of existing linker technology, introduced a technique called name mangling and essentially broke the guarantee that object files were interchangable regardless of which compiler were used. C++ code compiled with different compilers couldn’t even interoperate.

The end result was projects now had to select a single language for the entire project and stay with it. The cost of calling out to code written in other languages went up. Thus began the porting wars.

Companies now spend money to port significant portions of their existing software assets to the current darling in the language world. Even conservative aerospace companies have risked introducing new defects by porting reliable and fast math libraries written in FORTRAN to C++ just because “everything is going towards C++”.

Suckers.

They did it again with Java for the same reasons. I expect they’ll do it yet again with C# and .NET.

The real fact of the matter is that there has been no real progress in the software industry in something like 10 years. In fact we’ve lost some interoperability capabilities among programming languages.

As a side effect of changing the rules every year, experience is marginalized and hiring low cost unskilled programmers labor begins to look attractive. After all, even the old timers are going to have to learn everything all over. Thus, the endless march of new stuff for the sake of being new stuff nullifies the value of experience and hinders the development of masters in the field.

Of course, it also means that we are spending too much for each piece of functionality because everything is being done by brute force. Some of the truly elegant and productivity enhancing programming techniques have fallen into disuse because they are too mind expanding for novice developers to grasp quickly, and the top developers who can grasp them are too busy running in place to impart the advanced knowledge.

The only people benefitting from all this ‘new’ technology every year are the producers. Its all about trying to corral developers into their particular play pen and lock them in. Sun and IBM have been doing most of the rounding up lately. Microsoft is kicking off their own cattle drive with their .NET initiative.

The only way off the island is to identify a technology that works for you that you can control. This probably means open source development tools. Build up your expertise with this technology, continue to refine your approach, and stick with it. You’ll be sailing home before you know it.

Posted in programming, software, technology | No Comments »

Blog Migration

Thursday, August 17th, 2006

Once upon a time I had another blog.Â I hated the software that ran it and eventually stopped using it.Â I have a few good posts on programming there though, so I’ll be bringing them over here as new posts.

Posted in programming, technology | No Comments »