"A foolish consistency is the hobgoblin of little minds."

So I was writing some code for some data access code and I realized that there was a certain amount of what be considered "boring sameness" about the stuff I was writing. As I added additional methods for manipulating the state of a business object, I realized that I pretty much copied-and-pasted the last method, tweaking it for the new method. In this particular case, I would copy a Perl object method, change the name and POD documentation, change which SQL string it would use (two variable names), change the variable name being added to the array of bind values, and then tweak the return (if at all). All in all, I would copy and paste 100 lines of code and change usually 4-10 lines. Over the course of the past 30 minutes, I had done that 6 times.

If you write software right now, you are probably saying "so what?" You'd say this because it is very common. At some point, you factor out all the tough or repetitive code and you get down to a "diminishing returns" point where nothing else can really be factored out and that you are reusing the same method calls, tweaked for different scenarios, over and over again. This kind of work can actually be a bit boring—even mindless.

The important thing to point out here is that this "boring" copy and paste is a sign of good foundation underneath your code. Or at least it is in business software or code dealing directly with business processes. The reason this work is boring is because you are basically copying and pasting the same stuff over and over again with small tweaks. The reason this is good is because you have gotten your underlying code so consistent and straightforward that you can copy and paste the same basic stuff all over again.

I think this is important to point out because I so often don't see this boring sameness in the code I am asked to update or maintain. I'm not sure why the people who wrote the code originally didn't try to refactor their code to make it more regular and consistent, but again and again it's something I find missing in a lot of code these days. Perhaps they were in a hurry? Or perhaps they thought such optimization was unneeded? Perhaps they (mis)quoted to themselves Emerson, "Consistency is the hobgoblin of little minds." But that's not what Emerson said. He said:

A foolish consistency is the hobgoblin of little minds.

He never discussed what "foolish" versus "wise" consistency was, but he did make a qualification for his statement by mentioning "foolish consistency" specifically. While we in software engineering may not be poets (or care about Emerson), I think we need to take this quote (and its converse) to heart when developing software.

I've talked about the need for modular programming in another post. I didn't bother to discuss "going too far" because I rarely see that problem in my work. But it is possible to go too far. For instance, in the copying and pasting I was doing, I wondered briefly if there was some pattern I might extract so that I could eliminate even the copying and pasting. I say briefly because I thought about it and discarded the idea. The project I am on currently does not want to leverage any ORM or database query toolkits; whether this is a good idea or not could be debated, but the fact remains it was a constraint for my current work. So if I couldn't use a DB toolkit, any further optimization would either be creating my own DB toolkit (which would be dumb) or making optimization code that is so specific to the current work that it would either be brittle or not useful beyond the specific work I was implementing.

Here's where my early comment about "diminishing returns" meets Emerson's "foolish consistency". To optimize further in an attempt to eliminate the copying and pasting I was down to at this point would be wasted effort. This is where the proverbial 80/20 rule comes in. How you apply 80/20 depends on the situation, but you can pretty much invent almost an infinite number of ways to find 80 percent of something hiding in another 20 percent of something else. Regardless, the point is that I could have further optimized, but the amount of work required versus how much benefit and reuse I would gain later wasn't worth it—especially since in this case, I would have ended up creating my own limited DB toolkit. To demand additional optimization that has no real end benefit to reuse or clarity would be to insist on an additional level of "foolish consistency."

However, there's a definite balance or gray area about what is foolish and wise consistency in software engineering. I personally feel a lot of the ability to judge what is foolish and what is wise comes from experience and by learning from truly elegant software. Here, working through the classic Design Patterns or perhaps Knuth's Art of Computer Programming would help even the most seasoned programmer. The reality is that the relation between elegant software and discipline and experience is significant. You will only be able to produce (and to some extent, judge) the elegance of a solution through experience and training.

The rule of thumb I would leave you with is that design optimization—striving for common, consistent APIs or elegant, concise paradigms—is never "foolish consistency". Taking an extra hour or two to get an API consistent is never wasted effort. Consistent APIs are self-documenting (that is, they are plain to read and using one aspect of the API builds or directly translates into using another aspect). Most code I have found has erred on the side of "there's not enough time", and has regarded most attempts at consistency as foolish. Only painful experience and some training will ever cure those errors.

And who likes pain? So try to aim for boring cut and paste, the kind that isn't duplicating critical code over and over again. If you find that the code you are copying would require you to make a change in more than one place (possibly a dozen places) if some lower API or data structure changed, stop. Pause for a second and ask yourself what kind of consistency you have. Crack open a good book like Design Patterns and see if anything in there might help you. You might be really glad later on!

Oh, and throw in some documentation while you are at it :-)

Leopard and its non-intuitive volume change

Ok, so I have finally figured out why I hear my email screaming in the night.  Well, screaming deserves a little explanation: I use sample of a Jawa from Star Wars for new email notifications (the little scream of excitement when they zap R2D2—sounds a little like "Ooteedee!").  In my new apartment, my office is right next to my bedroom; I can easily hear the sounds from my computer in my bedroom.  Anyways, the last thing you want when you are drifting off into sleepy land is a screaming Jawa telling you email has arrived (and let's face it: email at that hour is usually spam anyways).

So, I took the obvious approach to solving the problem: I pressed mute on my keyboard.  Problem solved, right?  Not quite.

I have my MacBook hooked into a computer stereo speaker system.  One night, I heard some interference from my iPhone and thought it might be the cheap Radio Shack cable I used to hook it in.  So I unplugged the cable and went back to bed.  Just as I'm drifting back to sleep, I hear a screaming Jawa again—but I had hit mute.  What the heck?

I get up again, and hit mute.  Finally, silence.  I wake up the next morning, and I hit unmute (still not hooked into the stereo), and sounds resume.  Later in the day, I realize it isn't plugged into the speakers, so I plug it in.  No sounds.  Huh?  I try unmute again, and then I have sound.

Now maybe I'm just incredibly slow and stupid around bed time and when I get up in the morning (well, I know I'm slow and stupid in the morning), but it took me a few days to finally figure this out.  And I'm pretty sure this is a Leopard change, as I never remember having this much trouble back with Tiger.  Here's the deal ...

As near as I can tell, Leopard is trying to be helpful like the iPhone.  With the iPhone, if you set the volume while listening to the headphones, you'll notice that the volume setting will change if you unplug the headphones.  This is because the iPhone can tell when you have headphones plugged in and when you don't; so the volume for headphones and the volume for the speaker are kept separate.  This is good, when you think about it, as you would probably have the speaker louder than the headphones.  You wouldn't want your headphones to be blasting at the same volume.

It turns out, Leopard does this too.  Again, maybe this happened in Tiger, but I never remember this happening.  As I remember, mute was mute—period.  No matter what was plugged in or not, mute meant no sound.  Well, Leopard tracks volume—along with mute—based on what is plugged in (or not) into the headphones jack.  So you can set the volume level while you have something plugged into the headphone jack, unplug it, and then the volume level is now different.

Perhaps there is some nice feature here that I am missing.  But there's one flaw with this approach: mute should be mute should be mute.  I can't ever imagine a time where you would like to mute your computer sounds, only to have the sounds resume when you unplug it.  Usually when you choose mute, you want mute, regardless of whether your computer is plugged into headphones or not.  I could see the separate volume levels being useful, but not the separate mute settings.

Maybe I'll go create a radar ticket and see if Apple thinks it's a feature ...

How I Stopped Hating JavaScript

For anyone who has worked with me on a project involving JavaScript, this post is going to come as a bit of a shocker. I no longer hate JavaScript. I still have my gripes (which I am assured by JavaScript fans will be fixed in JavaScript 2), but I no longer regard that language as the native tongue of Hades.

To understand why I don't hate JavaScript anymore, I think you have to understand why I hated it in the first place.

Way, way back when I had a funky domain name, I put together this DHTML based sidebar based on some code from Wired's codemonkey site. It involved some JavaScript, and it was pretty darn cool (if a little unusable in terms of simplicity and accessibility). That was my first real introduction to what you could do with JavaScript (besides form validation). And the syntax was an awful like Java ... so what wasn't there to like?

As it turned out, quite a bit. Because doing something simple was one thing; something complex was another. And if clients want anything, it's complex user interfaces. After a few projects, I came up with a few things that made me hate working in JavaScript:

It had Java syntax, but variable scoping was NOT the same.

At the time, the idea of purposefully designing a language that perpetuated the evils of BASIC seemed absurd—so absurd that it earned it automatic "language of the Devil". Some might argue this is unfair, since I don't get on Perl's case about requiring my. Well, I do cut Perl more slack, but that's because it has been around a while and didn't have a lot of older siblings to teach it right from wrong (it had to find out for itself). JavaScript, however, seemed to have no excuse. I'd argue it still has no excuse. Variables should automatically be tied to their lexical scope. Any justification that argues it is a good thing are the ravings of the deranged. Sure, having to tack on more text to access a variable at a higher scope requires more typing—so what?

And here the Java syntax bites you.  The Java syntax is used by JavaScript so that it is more familiar to Java, C++, and even C programmers.  But in using this familiar syntax, you setup a huge tiger pit trap for these programmers. Suddenly things aren't working the way they would intuit them working.  And of all the things to not work as expected, variable scope is not one to "think different" on.

It had Java syntax, but no classes.

I'm an OO guy. Take it for what it's worth, but I think in objects and I have gotten extremely good at designing systems in OO terms. JavaScript had something that looked like Java OO, but was decidedly not Java OO. And in this "sort of classes" stuff coupled with Java-like syntax was another pit for a Java programmer (me) to fall into. So I had all this experience and all sorts of known, working solutions that wouldn't directly map into JavaScript. Or worse, it would map but come back to bite me in some nasty fashion.

Again, you might hold up Perl as another language with quasi-OO and wonder why it doesn't bother me as much. Well, Perl doesn't have Java syntax. It doesn't look, feel, or act like Java syntax. In that, it doesn't trigger false-positives when I look at the code or try to write code. I know that in Perl I have to bless a hash reference and that I don't have anything that is truly private. I grumble about it (a lot), and sometimes I have been known to whine quite annoyingly about it. But I deal with it—cuz that's just Perl (but not Perl 6, thank God). So you could say I was being unfair, but Perl never pretended to be Java friendly (quite the opposite).

JavaScript was all about the browser and the UI.

Most of my career has been spent untangling complex systems from mini-computers to mainframes, figuring out how to weld new technologies on to these existing business systems. I don't get to do too much UI work, and anything that can only do UI work is largely worthless for what I do.

Add to this the fact that UI work is all about learning to deal with frustration. All software is difficult; as I have said in elsewhere, a client often doesn't know what they want until you give them what they asked for—at which point they find out they want something else. A lot of times, this has to do with being unable to fully imagine what is possible; once you see some of what is possible, it inspires you to think of even more and different things for the new system.

No where is this more seen than in UI work. And on top of this "needing to see in order to refine", UI work suffers from also being a lot about taste and preference. Colors are changed, items are shifted, menus are re-worked, features are dropped. That's the reality of UI work; it can be a lot of nitty-gritty work that can sometimes be very frustrating.

Now take all of that base-level frustration of UI work and add on to it the morass of pain that is trying to get JavaScript to execute reliably across IE and Firefox (although, back in the day, it was IE3 and Netscape—shudder). And to make it even more painful, imagine not having an effective stdout to print debugging messages too. And don't even dream of a debugger.

All that taken together, writing JavaScript was like having a root canal for me. Why sign up for that kind of pain?

So what changed?

So if those were my reasons for hating JavaScript, why do I not hate it anymore?

The biggest thing was that I had a project that I wanted to do in my own time, and it was clear that presenting the UI via a browser was the best way to do it. On top of that, the UI was going to be a very complicated one (it uses a <canvas> to draw a lot of complicated heads-up display stuff). So I had a very specific project that I was very interested in—which helped me suffer through my growing pains and learn more about how to "think in JavaScript." As I worked on my project, I resolved each of my problems as best I could.

First, I always use var—always. In many ways, this is similar to my use of my in Perl. I don't hem and haw about whether to use it: I always use it. In this, I haven't accepted JavaScript's logic, I've just learned how to survive it. I still think this is a horrible flaw in the syntax (just as I think it is a flaw in Perl).

Second, I came to terms with prototype based object oriented programming, and once I had it understood, coded up my own "class syntax". JavaScript is very flexible, in some ways just like Lisp is flexible. Many Lispers will respond to some feature missing with, "Well, just write it—that's what Lisp is for." In this, JavaScript has some of that flexibility, and I am not alone in my desire for classes. The JavaScript Rhino Book even has a little recipe. By adopting some of Douglas Crockford's ideas to the recipe, I got a flexible solution that gave me enough of what I wanted to get by.

A colleague of mine has suggested that inheritance isn't all its cracked up to be. He demonstrates some ways that mixins might be superior to class-based inheritance. I'm not sure I entirely agree with all his points, but I do agree that mixins are great (it was my favorite feature when I learned Objective-C). For my part, I like mixins a lot—I just want classes and mixins. I think getting by without either is a pain, although I will confess that I regard doing without classes as a lot more of a burden (but then I've been thinking in terms of classes for most of my career). For those of us coming from Java, we'd appreciate some class-based training wheels until we can fully embrace the wonder that is prototype-based inheritance and mixins. And before someone says "tough," wouldn't a JavaScript programmer appreciate some way to do mixins or truly deferred binding in Java? Neither is better than the other, and both languages could benefit from supporting each feature in a first class manner.

Perhaps JavaScript 2 will solve some of my grumblings (if MS doesn't succeed in derailing the effort). But until then, I'll make do with my pseudo class inheritance toolkit and using mixins liberally when I can. Because I do like mixins. Any Java programmer knows why Interfaces are good and MI is bad. But that doesn't mean you haven't had a time where you really wished you could have some default code for an interface that didn't have to be stashed in some utility class instead of on the object itself.

Third, and finally, JavaScript has a debugger, and it even has a stdout after a fashion (there are several, but I'm partial to Miller's fvlogger). JavaScript can even run outside the browser now, although I don't see a whole lot of reasons to use it instead of something like Perl, Ruby, or Python. But the ability to trace through my JavaScript code easily makes life a lot better. IE still makes life horrible, but with Firefox gaining more and more share, IE is being forced to play nice and to try and not be so broken (although that doesn't stop it sometimes).

So, I don't hate JavaScript anymore. I actually find it kind of fun, especially when it isn't tossing some stupid thing in my face (like forgetting to mark a variable with var). I'm even finding that I wished Java had a few things JavaScript had (mainly, if someone could make a true Java syntax also be dynamic like a script, I would be in heaven). Hopefully as I become more versed in it, I will find more strengths in the language and learn several tricks that I can apply to another project in a different language. I already used my more thorough understanding of prototype-based inheritance to fix a bug in a prototype pattern in my rsh-dal toolkit. I was able to look at my code, see it clearly (as opposed to dimly) as a prototype, and then the bug kind of jumped out at me.

As long as you can keep learning, you ain't dead yet ;-)

Perl 6, oh when will you end my agony?

Larry Wall has issued his latest State of the Onion (we are up to 11 now). I'm glad he does these SotOs every once and a while—it re-engages me with Perl and makes me hopeful for the future.

But I have one issue: when, oh when, will my agony end? When will I be able to tell a sysadmin to install an RPM of Perl 6.0 so I can run some production Perl 6 code? According to Audrey Tang, it's Christmas ... but everyone refuses to specify which Christmas. And they grin when they do it. Which drives me a little crazy.

Frustration aside, they have made great progress with implementing Perl 6. It's so cool to see it working, and to see the syntax start to live in actual code. But it still isn't ready for production.

Could this be the Christmas we are all waiting for? Sadly I doubt it. I don't see any mention of a production-ready Perl 6 on the horizon. Which means we are looking at next Christmas? Sigh.

At least I have Moose.

What Happens When You've Scratched the Itch?

Open source software has been described as starting off as "scratching a developer's personal itch". In my experience, this definitely bears out. Problems that interest me become software that I want to do in my own time. Problems that aren't especially interesting (perhaps even boring) do not become software I write in my own time; usually someone has to pay me to do it. And sometimes I am truly fortunate in that I am both interested in the problem and I am getting paid to come up with a software solution for the same problem. In all honesty, I am a bit of code monkey—anything remotely interesting or intriquing (read: "bright and shiny") usually draws me in, even when the larger project might sound boring to the average guy on the street. What can I say? I love what I do.

But let's focus on the scenario that Raymond is specifically referring to: software that gets written by a programmer in his or her own time, that they are not getting paid for. A lot of this software now ends up as open source or free software. A significant portion (nay, a majority) of the code that allows you to view this post is based on such open software. It may be governed by any number of open or free software licenses (GPL, BSD, MPL, CPL, Perl Artistic, and more). But in the end, it all boils down to some guy or gal hacking together some software, cleaning it up (or maybe not), slapping a usage/copy license on it, and pushing it out into the hard, cruel world that is the Interweb of connecting tubes.

I am one of those guys or gals. I push out quite a bit of code out into the bazaar, where anyone can download and use it. I like writing software, and either I'm wearing the wrong kind of sweater or I have dry skin, because I seem to have a lot of itches. This means that I'm pushing out all sorts of code from time to time.

But if you look at the list of my software, you'll notice something. Some software gets written, and it seldom gets updated. This could be because I'm that good and my software doesn't need updating. But the grimmer truth is that the software doesn't get updated because either the itch is scratched (and interest wanes) or because the software solution itself has become outmoded or irrelevant.

If you look at SourceForge or CPAN, you'll find the problem writ large across the global developer community. Software is fun to write when you are interested, and it's a bit of a chore when you aren't so interested. So when the itch is gone or the software solution isn't as fun as it used to be, software starts to get neglected. Updates become less frequent. Bit rot starts to set in. Pretty soon, the solution becomes all but a footnote in time (or in this case The Wayback Machine).

Now, don't take any of this as a criticism on open source or free software—because it is not. This is, instead, an observation of the reality of writing open source software. While so many of us programmers love to write code and derive quite a bit of pleasure from just the act of writing it, we get even more pleasure from someone else using our software and also thinking it is cool. So what happens when the project isn't as exciting anymore, not as relevant, and no one is using it?

Well, it doesn't get updated. It collects dust. And if things change too much and it sits too long, it might start to suffer from some code rot.

Now a lot of people might see this as reason not to use some coder's little project, or some project that hasn't seen too many recent updates. But, of course, they'd be wrong ;-) Here's the thing: any code that actually works and actually meets your needs—that's very, very good. If it is open source, who cares that it isn't maintained? You have the code. Even better, it's finished and working! So download load it and integrate it according to the usage license. Drop the creator an email—they'll appreciate it! And who knows? They might be willing to dust it off and add your updates.

So what do you do then, if this is the reality of little software projects and toolkits? What do you do when you have scratched the itch? I'm glad you asked!

You document ever single API, interface, and use case.

Document, document, document! Document for your own sanity, if for no other reason. Your greatest enemy is not code rot, but code complexity and your own addled brain. Perhaps no one else will ever use your code, but the odds are that you will—even if it is just copying and pasting rather clever tricks into newer or unrelated projects. So document your code, because what seems elegant and obvious today is going to seem broken, confusing, or downright inscrutable six months from now.

So document your actual code, especially all the tricky bits or any of the hacks you needed to get around some broken interface. But in addition to documenting all the tricky bits of your code, document your classes, your interfaces, your APIs, your configuration options, you installation steps—everything and anything someone would need to use your code if they weren't as brilliant as you. Because that not-so-brilliant person is going to be you in six months.

You add all this documentation for you, but the result is that your code is cleaner, easier to understand, and easier to use. Which makes your life easier later, but it makes your code more likely to be used by others.

Maybe no one will ever use it. But you can move on to your new itch with a clean conscience now. Anyone could download your code and use it, thanks to all that good documentation. And who knows? Maybe that person will be you in a year or two—and boy, will you be thankful you did all that documentation ;-)

Posting More (Hopefully)

This blog is currently sourced into the front page of my corporate web site. As such, it has made me a little more cautious about posting. For instance, I could post about my usage of OmniFocus and my opinion of GTD and how it implements GTD, but do I want that on the front page of my corporate site? I'm not sure, but based on how I wrote my inclusion stuff, I didn't have a lot of choice.

Well, now I do. I revisited the parsing/caching script and patched it to look for an "rsh tech" tag (which shows up as a <category/> tag in the feed). When that's present, I include it in the front page. For now, that will do. I have a new server on order, so a lot of things may be changing when I get the new server up and running.

So, here's to more blogging!