The author has a bone to pick with the software industry, and is trying to find quantitative data.
I contacted market research firms and also spoke with documentation managers in leading software firms. I asked them the following questions:
- Do you have any estimates for the cost of software documentation? I would like to know:
- Cost to software manufacturers and internal development groups
- Cost to end user companies that struggle with substandard documentation.
- I am particularly interested in documentation for APIs, especially Java APIs with attendant Javadocs:
- Estimates on the number of Java APIs
- Total cost apportioned to writing them
- Cost to tech support groups who field calls because of inadequate documentation
- Cost to end user companies in salary for programmers' wasted time
- Opportunity cost to end user corporations
- Cost of lost sales to vendors because programming to their APIs was not cost effective.
- I am interested in any information about the percentage of companies that are using Javadocs as opposed to other means of generating documentation.
I was surprised to learn that such information was not readily available. Software companies live and die in part according to the quality of their documentation, and yet no business oriented analysis could be found. Surely this is an indication of the immaturity of the discipline of software engineering.
One documentation manager at a well-known Fortune 500 software company said ‘we just assume that documentation is important, and try to do the best we can with the resources that are made available to us. We have never considered measuring the impact of the quality of work that we do on the success of the product, or the product's profitability.’ To be sure, few senior executives are likely to allocate resource to a project because it is the right thing to do – they want tangible justification, usually in terms of increased revenue, decreased cost, improved market share, etc. in order to grant approval. Possibly this disconnect explains why so much documentation is of low quality.
Online Documentation Survey
Unhappy with the lack of information on this subject, the author undertook an online survey of a cross-section of computer professionals. The results support the material that follows.
As a technical business consultant, I once had to explain to a customer how one could use estimate the number of pianos to a fair degree of certainty in the City of New York by just looking at the yellow pages. By counting the number of piano tuners, one could guess the number of pianos from simple arithmetic and a few basic assumptions. By varying the assumptions, one could perform a senstivity analysis of the end result for each of the assumptions. In this way one could predict the number of piano tuners in the City of New York with a known degree of certainty – without ever going to New York.
In the absence of any hard data, I tried a similar exercise by using a few assumptions about software documentation costs for Fortune 500 companies. Feel free to alter the assumptions and plug in different values into the spreadsheet model – the results are most interesting regardless of the actual figures used!
From the point of view of large corporations, the training costs for badly written API documentation might be computed as follows:
|Effectiveness of newbie learning API|
|Weeks to learn new API to 70% efficiency|
|Person-weeks lost per API learning experience|
|Loaded labor rate ($usd/week)|
|Cost per person to learn an API|
|IT staff programming to APIs|
|Learning cost per enterprise per API|
|Number of enterprise APIs utilized by average programmers|
|Cost for all programmers to learn the enterprise APIs that they use|
|Annual growth in IT staff|
|Annual API learning growth|
|Annual cost for newbies to learn APIs|
|Percentage of APIs that become obsolete every year|
|Annual cost to train for replaced APIs|
|Number of companies|
|Total cost in Fortune 500 for API learning|
|Savings from 10% increased productivity|
The 10% increase in productivity cited in the last line above is admittedly arbitrary and not defensible. It is the author's opinion that for many software vendors the actual savings might be closer to 35%.
I spoke with a technical support manager at an enterprise software vendor who must remain nameless. He computed the cost to them of for their own poor API documentation as:
|loaded hourly rate for tech support personnel|
|Average number of hours for tech support staff to resolve each case|
|Average daily cost for tech support staff to resolve cases|
|Average quarterly cost for tech support staff to resolve cases|
|Percentage of API issues|
|Anticipated reduction in API case load|
|Quarterly cost to tech support due to poor documentation|
Enterprise software vendors frequently sell their product through system integrators. The same software vendor that generated the previous table also estimated the costs to their system integrators for poor API documentation as:
|Average number of months for new consultants to become productive|
|Average productivity for newly training consultants (compared to experienced consultants)|
|Average lost productivity (person-days) for new consultants to become productive|
|Loaded labor rate for new consultant|
|Lost productivity cost for new consultants (average per new consultant per quarter)|
|Course fees and travel for basic, system administration and programming courses|
|Total training cost per new consultant|
|Total number of consultants world-wide with all channel partners|
|Annual churn rate for channel partner's consultants|
|Number of new vendor consultants per quarter retained by all partners|
|Total training cost for all new consultants worldwide by all partners|
|Reduction in training cost (decrease of lost productivity)|
|Anticipated savings per new consultant|
|Total quarterly anticipated savings for all partners|
Directions for Further Study
Costs for developing good documentation are difficult to come by. What percentage of development cost is typical for documentation? What impact on the bottom line profitability will this have on the software vendor, channel partners and end user corporations?
Case studies would be instructive.