The Two Biggest Problems in Software Engineering

I solve a very specific set of problems for my clients. I help them to quote, specify and generate custom web applications that will retail for between $5,000 and $50,000 (maybe up to $100,000 and occasionally down to $2,500 at the low end). That means I need to be able to quote, facilitate the specification process and generate and deploy working code for complex custom web applications for a wholesale price of between $1,200 and maybe $12,000 per project to leave plenty of margin for the web shops that I work for.

If you have $1MM to develop a site, you (like everyone) will still have problems in developing the right solution to meet your business needs on time and on budget, but at least you have enough money to hire the kind of people who are smart enough (and will have enough time) to figure it out. (ThoughtWorks was a random choice, but given the kind of people who work for them I’m figuring they have some serious IQ available to solve your business problems).

With the best will in the world, I doubt you’re going to get much of Mr. Fowlers time on a $1,200 project. So, what is a small web shop to do? Every day, customers come to them wanting to “put their entire supply chain online” or to “become the amazon.com of the xxx industry”. And of course, they have $25,000 (tops) for the whole project including specification, graphic design, programming, data entry, debugging, testing and deployment. (Sometimes it’s $5,000 :-<)

Standard Software
Well, the obvious first thought is to use standard software. You could use Miva for e-commerce (or OS commerce if you’re a masochist with a tame programmer :->), Constant Contact for the email newsletter and so on. That is great except for two glaring problems that started me on a journey back in 1999 that is only now starting to bear fruit. The first problem is customization and the second is integration.

Years ago I was running a small website development company and we spent a lot of time investigating the best off the shelf software. However, time after time we would finish a project only to find that the one “essential feature” that the customer didn’t realize until we were almost completed that they needed was something that the package simply couldn’t support. So the client had to put up with software that didn’t meet their business needs or had to pay for us to start again from scratch.

I also noticed that while you could use various pieces of packaged software, getting them to play together nicely was a real issue. This has gotten better over the years, but try implementing a single sign on for your e-commerce, discussion board, secure extranet and email marketing software packages with a single admin screen for managing permissions across all of the systems bearing in mind that the entire integration project must be under 2 man days of work and see how it goes! Then try to keep that working for no additional charge as the versions of all of the various pieces of software change over time :-> That was the problem we HAD to solve.

It was at this time that I realized that we needed a single piece of software to solve all of our customers problems where every piece of the design was flexible enough to be replaced. This was back in 1999, so I raised $625,000 for The Strategy Factory in Houston, Tx to try to build the business.

The First Attempt
It was quite a ride. It was at the height of the Web 1.0 boom and I still remember trying to fit a team of six attorneys (including two partners) from a top Texas law firm into our very small conference room in Houston as they wooed my two partners and myself trying to get a piece of our dot-com business! We went from 5 to 25 employees in under three months and then back from 25 down to 5 in the next three months as the series B failed to materialize after the NASDAQ crashed in 2000. In retrospect there were many things I would have done differently, but we built a business based on raising additional rounds of capital and those rounds failed to materialize. I don’t believe there are any changes we could have made in the execution that would have affected the final outcome. We sold the business for stock in an acquiring dot-com so we could at least pay the two key programmers and keep the dream alive, but that company also went under so I went back to consulting while I tried to figure out where to go from there.

By 2004/5 I was running a small company providing wholesale programming services to small website development shops and I had written a set of procedural generators that did a decent job of generating content management, e-commerce and email marketing systems that I then tweaked by hand and hosted for a very affordable price. I was making a living, but the model wasn’t profitable enough to scale. It was the classic “struggling small business” that would never be more than handful of people and a modest living. It was fine, but not what I’d planned.

As some of you noticed, I got a little more involved in the community in 2006 (:-> - thanks Jared for giving me a shot!). I had a profitable business and spent at least 30-40 hours a week trying to build a deep technical foundation to get the product to the next level. I knew that I needed a complete solution – from quotation to implementation. There just wasn’t the margin in our problem space to do anything by hand.

Intent Driven Design
I realized a long time ago that the optimal approach to developing most simple to moderately complex web applications was an interface driven approach based on the screens. I developed a methodology that iteratively takes users from their business intent (why bother building it?) through their audience roles, objectives, screens, actions and associated steps to fully specify the functional requirements for the application (see the CFDJ article for an overview of the process and look out for an upcoming book later this year). The beauty of the process is that it feeds directly into an application generator and builds traceability and best practices right into every application.

Software Product Line
The problem is that on the kind of budgets we work with, no matter how simple the specification process is, it is still too much for most projects. Most of the end clients we work with have a “with a twist” kind of problem. As such, a Software Product Line is the obvious approach (I’ll post more about SPL’s shortly). We allow clients to select features and have a decision support tree that helps them to select the appropriate functionality to meet their business requirements. They can then just customize any of their functional requirements to meet any unique needs and we use an automated translation process to translate the functional metadata (in the SPL world – a PIM – Platform Independent Model) into the artifact specific metadata that will be used to generate the documentation, db schemas and ColdFusion scripts (in the SPL world a PSM – Platform Specific Model).

Formalizing the specification process is actually the hardest part. If you have a well-designed set of DSLs for your functional metadata, an automated set of translation scripts for generating your artifact metadata and a set of generators for generating the artifacts, you automatically have an agile solution. Because your DSLs are much closer to the business requirements, you can use them as a ubiquitous language for describing the requirements in a way that business users can understand but that can be also used to automatically generate code.

To Summarize
We often talk about eliciting requirements, but the fact is that most of our clients don’t truly KNOW what they want. We need a system to support our clients in developing requirements that will provide an output that can be used to generate the necessary code. Only by generating the majority of the code (and other artifacts such as db schemas, unit tests and documentation) can we create applications that are agile enough to change as quick as our clients minds.

I will be posting a LOT more about some of these concepts over the next few months to try to provide a good set of resources for improving the efficiency of quotation, specification and design.

Any input appreciated!