Modular programming is something all of us software engineers are supposed to know about and do. It doesn't matter if you are a procedural programmer (standard C), an object-oriented programmer (Java, C#), or a functional programmer (Lisp). Whatever approach you use, you always strive to be modular to one degree or another because in the end it is the only thing that is going to save your sanity.
I think the degree of modular programming present in code might be the easiest way to judge whether a programmer has "it". I'm still not sure exactly what "it" is and whether it can be taught. Everytime I talk about programming with colleagues, the existence of "it" and who has "it" is always a part of the discussion. My current understanding of "it" is how well a programmer understands the problem that is "writing software". That may sound a bit circular, and it is, but it gets to the heart of the problem. If you have never thought about the practice of writing software and what makes for good code versus bad software, you might not have "it" yet (some would argue you never will—but I'm not sure I believe that). The reality is that you can write just about anything in any language using any design or approach (really, it's true). So while Java and C programmers love to go at it with hammer and tongs, in the end what technology is used has little bearing on the final quality of the code produced with that technology. The real problem, the real hurdle, is how to write code in such a way that your code is easy to write, easy to debug, easy to maintain, and easy to extend or use with other code. Understanding that the real problem is how to write software, and not what technology is used to write it, seems to be the "it" factor in software engineering. If you grok that the "how" is the real issue, you are on your way to becoming a truly great software engineer.
Which brings us back to modular programming and design. How modular your code is seems to be a good judge of how much of "it" you have grasped. If your code is pretty interconnected and monolithic, you haven't thought about software much or have had very limited experience. It could also be that you have only written scripts or small bits of software (the problems of software design seem to only become apparent in complex or large applications). That is, you might be a wiz and cranking out small scripts and utilities, but your styles and disciplines could betray you when you tackle a 100k LOC project. Betray might not even be strong enough. Try "take you out back, rob you, beat you, and leave you for dead."
Lately, I have been trying to understand why developers are so resistant to writing modular code. When I chat with fellow developers about it, they all seem to agree it's a good thing to do. But when I look at their code, it isn't very modular. When I ask why, I generally get the following answers:
- They didn't have enough time.
That is, they were in a hurry and were just cranking out code, and it would have taken too long to make the code truly modular. This is a favorite answer, and hey, isn't it always true? Software engineers never have enough time. - The code is temporary, throw-away, or not important enough.
In this case, the code is regarded as not being worth the "effort" to make it modular. Which in some ways, ties back to the first reason of not having enough time. Modularity is perceived as taking longer to write than non-modular code. - It was too difficult to package the final product in a modular fashion.
This last one is more rare, because it only comes up when I start showing people how to get really modular in the approach to building software. If you take a source tree and break it into 5 independent (but related) modules, the inevitable complaint is that it is too hard to package and distribute the final product. So why not just leave it as one big tree?
There may be other reasons, but these three seem to encompass most of them. I think you might notice a general theme though: too much time or effort required. In the end, most developers write code that isn't very modular because they feel that time and effort involved with writing modular code far out-weighs the potential benefits. As a matter of fact, there are even some developers who think modular code is right up there with documentation: something that isn't critical for functional code, and something only purists and nit-pickers worry about. Heretics!
Again, I think this ties back into how much thought an individual developer has given to the real problems of writing software and with how many large-scale projects they have worked on. If you understand that the real issue is debugging software or maintaining it, and if you know that the majority of your time will be spent in these two tasks, you never see modular programming as too much effort.
This gets to the heart of the issue: modular programming is an investment of time against future problems. You have to understand and believe that debugging and maintenance will be exteremely difficult down the road if you don't decouple your code and break it down into managable pieces. While it may be more effort to set up a project and maintain it as separate pieces and modules, it is much easier than trying to deploy 100k LOC just to debug one small section. And it is a lot easier to have 20 or more developers working in separate modules and pieces rather than having every little mistake break the entire source tree.
If modular programming is an investment of time against future problems, why do so many developers skimp on it? Is it that they are just too short-sighted to see the benefits of modular programming? I'd argue no. I'd also argue that neither are most managers (even though everyone—including me—loves to beat on managers). The problem in both cases seems to be that most programmers and managers are hopeless optimists. That is, they think the program will not be that large, not take that long, or not be used in any unexpected scenarios. But when it comes to writing software, all three of these hopes/beliefs are the most naive of assumptions you could make!
- Software always grows. ALWAYS.
I could write a whole blog post just on this issue. There are many reasons, but it seems to come down to the problem domain never being fully understood. Either you think it is just X and Y and find out that X is really A, B, C, D, and E while there is also this missing Z component you never thought about, or you know all about X, Y, A, B, C, D, and E (but still not Z) and you assume you can get by with just doing one or two—only to find out you can't.
Another reason software grows can be described in my personal, favorite adage: "The client/user doesn't know what they want until you give them exactly what they asked for, only to have them reply, 'Oh, well, that's not what I really want.'" In the same way that a programmer often doesn't understand the problem domain fully, the user has the same issue. Until they get what they thought they wanted, they are unable to tell you exactly what they want. This usually takes several round trips to get right—and results in a lot of work. - Software always takes longer than you think it will. ALWAYS.
I don't know why more people haven't read The Mythical Man-Month. It's not like it is brand new or something—it was first published in 1975! I can't count the number of times I have watched a new project ignore the advice in this book (even when I am vocally shouting the same advice). In the end, what was true in 1975 is still true today: time estimates are always off and people continue to make the same bad assumptions when they are off. If you miss the first milestone, you are going to miss every milestone from there on out. Work as hard as you want, it's just a fact. - Software always gets drafted to do something it wasn't initially designed to do. ALWAYS.
This doesn't mean your software will not be put to its original use; it just means that won't be the only thing it is used for. This is kind of an extension of "when all you have is a hammer, everything looks like a nail". If you have working software, you are going to try and leverage it as much as possible—which sometimes means the code is working outside its design comfort zone. This makes flexibility key to any software—even scripts. Too many times I have stumbled across a script that was just intended to do something simple and now has been hacked and extended to any number of new and unexpected things.
It's ironic that these three areas of "hopeless optimism" are the reason why modular programming is skimped on. It's ironic because modular programming helps mitigate all three of these areas. If you write modular code, it can grow and extend easier. If you write modular code, you can better estimate time for the tasks at hand because the problem has been broken down into smaller, more manageable pieces. If you write modular code, you code should be self-contained, well-documented, and have a clearly defined interface—thus allowing you to drop it in, unchanged, within new code. It's kind of tragic in a way: the one design priniciple that is skipped because programmers and managers assume a program won't be too large, won't take too long, or won't be used in unexpected ways is the exact principle that would have saved them headaches when all three of these eventually do happen.
Modular design can always be employed despite the language or programming approach used. It helps mitigate the issues of code growth, time estimates for coding, and code being used in ways not considered in the initial design. Despite being a principle that is 40 plus years in the making (it predates MMM by a significant amount), modular code still isn't seen as much as we like it to be. I hope to address some of the reasons why it isn't as wide spread in some follow up articles. Check back soon!
2 comments:
Indeed, there is a certain special something that defines an elegant solution to a problem. Modularity is clearly one of the most helpful tools for attaining an elegant solution. But I'm not convinced that modularity is "it".
A number of things came up for me while reading your post:
1.) I had to write everything, from batch programs to web applications in XSL for 4 or 5 years, and I can tell you that the choice of tool-set is important. Using only a hammer is great for any problem that happens to be a nail, but you also end up with a lot of bent screws and other broken crap!
2.) The mere act of making modules won't help if you don't understand why you are making them. One example of "fine-grained modularity" I've seen took a long-running query necessary for only one screen of an application and put it in a class that all screens were descended from. Because it's hidden in a "module" in another file, in another directory, it's easy to miss the problem when you read through your code.
3.) Though this is the only application for it I've seen, I've come across some very effective monoliths in graphics device drivers filled with cut-and-paste code (unrolled loops, for instance).
4.) What "Always" happens to software: If the software doesn't die, it always has those problems. There really is no middle ground between death and problems.
5.) At the end, you mention snappable Lego-like modules. I think that's one of the holy grails of modern computing. It is only the rare occasion where a significant change is truly local - where it does not require, or would not be made more effective by changing an interface, thus changing more code.
6.) Personally, I've been enamored with the idea that meeting business needs in the simplest way is part of the silver bullet, and being willing to tear things apart is another part. I think listening to business is one very underrated developer skill, testing your own work is another.
This was a very interesting read. Thanks for posting!
Glen,
Your one point:
At the end, you mention snappable Lego-like modules. I think that's one of the holy grails of modern computing. It is only the rare occasion where a significant change is truly local - where it does not require, or would not be made more effective by changing an interface, thus changing more code.
You may be correct that changes are rarely local, but a modular, black-box approach to building the system will allow you to whether more global changes better than a less-modular system. You can wrap non-complaint modules in adapters, etc.
In the end, modularity is not the silver bullet, but it is the critical investment that saves you from future pain.
Post a Comment