I studied software development in college and then spent a number of years building software products in different industries. I generally worked as a developer or project manager and my jobs have caused me to interact with people from just about every department in a normal company, including marketing, management, executives, HR, accounting, UI and QA.

Over the years, I have thought about the process of building better products and why there are so many failures and so few successes. My feeling is that the number one problem with software is not bugs; it is ease of use. Bugs can be quantified and fixed but products that are hard to use are not broken, they are built that way to begin with.

The root cause of this problem is threefold: the nature of design, the process and the people at most development companies.

Problems inherent to the design process

Conceptually, the process of software development is quite simple. Someone conceives an idea of what they want a program to do and someone (generally someone else or a team of someone elses') writes the code, tests it, and delivers it to customers. Like other work, software design is a team sport that involves a number of different skill sets and several stages which normally means a lot of different people and departments.

In the best case, a marketing person gets good information from potential customers about a product that satisfies their needs (ie, something they would pay for) and translate that product vision into descriptions or documents that developers can make into a product.

The fundamental problem here is that customers are very bad at both envisioning what they want or what developers could do. Customers are very good at describing what they don't like but only after you show them something to react to or use. This is a huge problem for creating design requirements or functional specs from initial customer interviews.

While customers are bad at describing what they want, engineers are bad at thinking like customers. Engineering has its own mindset and way of thinking but most customers are non-technical and value different things. The problem here is that developers constantly make critical decisions during the development process and those decisions reflect the tastes of engineers not of customers. At the end of the day, there are a lot of products designed and built for other engineers but sold to non-technical customers.

Another problem with the entire software design process that is related to way people think differently is the telephone game. Understanding in one department often gets lost when it is written down or covered in a meeting and then acted upon by another department. This is a communication problem that is endemic in companies and the result is that even product which start off with very clear direction often end up muddled.

The communication problems with software development are further exacerbated by what I call the "invisible world" issue. Everyone knows that building a nuclear reactor is a complex task but it is very hard for most people to grok software. Things that seem easy to do, can be extremely hard to program (like intelligence) while things that appear very impressive can be trivial to code (many UI features fall here). Customers are often reluctant or unable to ask for what they really want because they think it is too hard. Developers in turn can take advantage of customer (and manager) naiveté when discussing changes. Moreover this lack of understanding can even develop into a disdain for customers who are too dumb to understand how great the product (or the engineers) really is.

Solutions

In my mind, any product sold to a consumer but designed for an engineer is a poorly designed product. When one looks around at the sea of mediocre products out there, from cell phones to personal computer software, the conclusion is that few companies do a good job addressing these design problems. The discoverability of features and ease of use of most technology products are extremely poor and not improving.

The solution (or at least improvements) requires changes to both process and people.

People Needs

As I mentioned, product development is a team sport. Many functional areas are involved in addition to actual customers, and all of these groups have their own world views, their own mindsets, and their own values.

To convert customer needs into products it is critical that companies have people that are good communicators, translators if you will, of engineer-speak and consumer-speak. The world-views of these two groups are every bit as different as those we see in different cultures or countries yet few companies recognize or acknowledge this fact. Like international trade negotiations, development companies need people that can transfer meaning between the two cultures to bring out better products.

When working with customers and other product stake-holders, these translators need to be active listeners. They need to hear what the customer is saying as well as what they aren't saying. Much like running a focus group, they need to be able to ask probing questions that get at the real needs, something that the customer may not be aware of themselves. This process is as much an art as a science.

When working with developers, these translators need to be able to articulate the customer vision as well as the business need to a sometimes hostile audience. Developers do not study business and often have a very idealistic (but eminently logical) view of the business needs. Like college professors in their ivory towers, many devs actively disdain the profit motive while others are motivated by their own design choices rather than customer needs or even customers themselves. (I've heard developers throw their hands up in disgust with comments like: "They dont even know what they want!")

These engineering opinions are understandable, especially given the separation most companies put between developers and customers, but the translators need to deal with it. Unlike working with customers, this is not an easy task since many developers are not the most receptive listeners; Engineering-machismo is pervasive throughout the industry along with the idea that engineers are just plain smarter than everyone else. (A mental trait that is also present when working with doctors.)

The actual title or job role of these "translators" is not always clear nor is it the same in every company. But someone needs to be good at it. They also need the power to enforce decisions or the ability to get others to enforce them. Sadly there are very few of these people out there.

Process Changes

In addition to having good translators, companies need a process that encourages better products. That process needs to support each of the problems above with a particular focus on dialog and enabling quick changes.

Since customers have trouble visualizing, dont make them. Involve customers and other stake-holders as early and as often as possible. Even if there is no code, use photoshop or even paper to describe the intended behavior. As Jakob Nielsen has pointed out, it does not take a complex process or lots of resources; it just takes dialog and common sense.

Unfortunately, many companies keep customers as far away from developers as possible. Development at many companies is a highly segmented example of the telephone game. A project is commissioned. Someone else writes a functional spec. Someone else reads the spec to develop a product as best they can. Months or even years pass. A product comes out. Customers hate it but are told that its too hard or late to make changes. Customers grudgingly buy/use the product but are unhappy or resentful. Developers in turn feel like their work is unappreciated and also go away unhappy and resentful.

In response to this status quo, many people have argued for agile development or extreme programming. Basically these are development methods that involve a lot of quick iterations and increased communications (as well as unit testing). This is totally the right way to go.

Another process change that I have recently been thinking about relates to choice. What product is better to use? A product that can do everything and has 25 options to configure or a product that does one thing well with just a few options? Engineers would pick the first product; consumers would pick the second.

Apple is widely regarded as a company that "gets it". They design cutting edge products that consumers and end-users love. The critical thing that few people talk about is that Apple is able to offer these products because they make decisions for you. They recognize that consumers dont want a million options because they dont understand them nor do they want to; Consumers want a product that works, immediately.

Ironically this idea of simplicity is very foreign to engineer-think, which thrives on details and intricacies. To a lot of engineers, it just seems limiting. In engineer-think, the best product is obviously that Swiss Army knife with 50 gadgets that is so big it wont fit in our pocket, not the Buck knife that is, well, just a knife.

This idea of design simplicity is getting more attention with books like "The Paradox of Choice: Why More Is Less", but if you look at products like cell phones or remote controls or PC's, it is still not common in practice despite constant complaints from consumers. My feeling is that the situation wont change because it is hard for developers to do. It just doesn't happen without a visionary at the top who is willing to make the touch choices and stand by the results. It is much safer to throw all the features in there and hope that at least a few of them stick.

The reality at many companies is that there is a big spreadsheet that iterates all 50 features of the competitor's products. Success is viewed by marketing and development as implementing all 50 features they have plus 1 more they dont have. It is much safer in the CYA-school-of-thought to do this than it is to pick the 3 most important features. If the product doesn't sell, someone will have to explain why they dropped the other 47, which is "obviously" the reason the product tanked.

The miracle of Apple Computer is that they try to make best decision for you so that you dont have to understand 50 features or the intricacies of DRM or MP3 encoding, etc. Their products wont please everyone but their recent success shows that the market values companies which can reduce not increase complexity in people's lives. Companies that have a development process that can make decisions and choose the best 1 to 3 choices for its customers, will have better products in the end because those products will be easier to use, the features easier to discover, and thus will get used more.

Another wrinkle with the development process involves the development organization and the informal hierarchy of different roles. Some companies actually have people that focus on ease of use, usability and picking the best 1 to 3 choices for people. The problem is that these people are usually at the bottom of the totem pole, often looked down upon by other engineers and managers. When budget cuts happen, ease of use is often the first thing cut and when products are launched, usability gets the least credit. As essential as these people are to good products, their contribution is still poorly understood or valued by other engineers or the management at most companies. Again this is a process issue which falls back on the difference between how engineers think and how consumers think.

Closing

Those are a few thoughts on the difficulty of building better products. Over time, I hope there are more companies that get the design accolades that Apple Computer does but I suspect that the people who work on these issues will continue to have opportunities (and struggles) for years to come.