Mike Slinn
Mike Slinn

Cost-Benefit Analysis for Documentation

The author has a bone to pick with the software industry, and is trying to find quantitative data.

I contacted market research firms and also spoke with documentation managers in leading software firms. I asked them the following questions:

  1. Do you have any estimates for the cost of software documentation? I would like to know:
    1. Cost to software manufacturers and internal development groups
    2. Cost to end user companies that struggle with substandard documentation.
  2. I am particularly interested in documentation for APIs, especially Java APIs with attendant Javadocs:
    1. Estimates on the number of Java APIs
    2. Total cost apportioned to writing them
    3. Cost to tech support groups who field calls because of inadequate documentation
    4. Cost to end user companies in salary for programmers' wasted time
    5. Opportunity cost to end user corporations
    6. Cost of lost sales to vendors because programming to their APIs was not cost effective.
  3. I am interested in any information about the percentage of companies that are using Javadocs as opposed to other means of generating documentation.

I was surprised to learn that such information was not readily available. Software companies live and die in part according to the quality of their documentation, and yet no business oriented analysis could be found. Surely this is an indication of the immaturity of the discipline of software engineering.

One documentation manager at a well-known Fortune 500 software company said ‘we just assume that documentation is important, and try to do the best we can with the resources that are made available to us. We have never considered measuring the impact of the quality of work that we do on the success of the product, or the product's profitability.’ To be sure, few senior executives are likely to allocate resource to a project because it is the right thing to do – they want tangible justification, usually in terms of increased revenue, decreased cost, improved market share, etc. in order to grant approval. Possibly this disconnect explains why so much documentation is of low quality.

Online Documentation Survey

Unhappy with the lack of information on this subject, the author undertook an online survey of a cross-section of computer professionals. The results support the material that follows.

Thumbnail Estimates

As a technical business consultant, I once had to explain to a customer how one could use estimate the number of pianos to a fair degree of certainty in the City of New York by just looking at the yellow pages. By counting the number of piano tuners, one could guess the number of pianos from simple arithmetic and a few basic assumptions. By varying the assumptions, one could perform a senstivity analysis of the end result for each of the assumptions. In this way one could predict the number of piano tuners in the City of New York with a known degree of certainty – without ever going to New York.

In the absence of any hard data, I tried a similar exercise by using a few assumptions about software documentation costs for Fortune 500 companies. Feel free to alter the assumptions and plug in different values into the spreadsheet model – the results are most interesting regardless of the actual figures used!

From the point of view of large corporations, the training costs for badly written API documentation might be computed as follows:

Effectiveness of newbie learning API
Weeks to learn new API to 70% efficiency
Person-weeks lost per API learning experience
Loaded labor rate ($usd/week)
Cost per person to learn an API
IT staff programming to APIs
Learning cost per enterprise per API
Number of enterprise APIs utilized by average programmers
Cost for all programmers to learn the enterprise APIs that they use
Annual turnover
Annual growth in IT staff
Annual API learning growth
Annual cost for newbies to learn APIs
Percentage of APIs that become obsolete every year
Annual cost to train for replaced APIs
Number of companies
Total cost in Fortune 500 for API learning
Savings from 10% increased productivity

The 10% increase in productivity cited in the last line above is admittedly arbitrary and not defensible. It is the author's opinion that for many software vendors the actual savings might be closer to 35%.

I spoke with a technical support manager at an enterprise software vendor who must remain nameless. He computed the cost to them of for their own poor API documentation as:

loaded hourly rate for tech support personnel
Average number of hours for tech support staff to resolve each case
Average daily cost for tech support staff to resolve cases
Average quarterly cost for tech support staff to resolve cases
Percentage of API issues
Anticipated reduction in API case load
Quarterly cost to tech support due to poor documentation

Enterprise software vendors frequently sell their product through system integrators. The same software vendor that generated the previous table also estimated the costs to their system integrators for poor API documentation as:

Average number of months for new consultants to become productive
Average productivity for newly training consultants (compared to experienced consultants)
Average lost productivity (person-days) for new consultants to become productive
Loaded labor rate for new consultant
Lost productivity cost for new consultants (average per new consultant per quarter)
Course fees and travel for basic, system administration and programming courses
Total training cost per new consultant
Total number of consultants world-wide with all channel partners
Annual churn rate for channel partner's consultants
Number of new vendor consultants per quarter retained by all partners
Total training cost for all new consultants worldwide by all partners
Reduction in training cost (decrease of lost productivity)
Anticipated savings per new consultant
Total quarterly anticipated savings for all partners

Directions for Further Study

Costs for developing good documentation are difficult to come by. What percentage of development cost is typical for documentation? What impact on the bottom line profitability will this have on the software vendor, channel partners and end user corporations?

Case studies would be instructive.