Hello everyone! At long last, attached is a distillation of the research roadmap, based on discussions and sessions at the Dagstuhl seminar. Please take a look and add any comments and thoughts. This remains to be shaped and fleshed out by you, the research community. As part of that process, we plan to spend part of the MTD workshop in Raleigh, NC, next month, on this subject.
In a recent study (Siebra et al., 2016 we hope), we (co-authors include Fabio Silva who will also be joining us in Dagstuhl) developed a concept map (or a conceptual model) of the elements we found to be important to evaluate during TD decision making. The concept map was derived using grounded theory from an in-depth case study of one TD item, its evolution, and the series of decisions that were made about it. The concept map was then used, retrospectively, to analyze several other TD decision-making cases to see if the map captured all relevant factors, and might be useful as a tool for TD decision-making. We found the model to be complete, at least for the cases we tried. The concept map is shown here in two parts (because the concept of Cost ended up being a bit more complicated and needed to be elaborated further. There’s a good bit of overlap with Philippe’s conceptual model that he posted earlier. Coming to a consensus on a conceptual map (or maybe several, we’ll see) is one goal of the workshop.
Key challenge: Prioritizing types or subsets of TD to explicitly manage. Note this is different than prioritizing TD to pay off. Our recent work has indicated that it may not be cost-effective to apply explicit management (i.e. identification, documentation, measurement, monitoring) to all the TD in a project. So, how does one choose what to manage?
Our work: My recent research on technical debt has focused on the cost-effectiveness of explicitly managing TD. The overall research question has been “is it really worth it to measure and track TD in order to make better decisions about paying it off?” My student, Yuepu Guo, and I have conducted four studies, all very small, but all very focused on parts of this question. The first two studies were retrospective studies, where we used historical project data to identify TD items that could have been discovered, to estimate the principal and interest on those TD items at the time, and reason about what decision would have been made if those items had been analyzed using a cost-benefit analysis based on principal and interest. In many, but not all, cases, such a cost-benefit analysis would have resulted in a different decision than what was actually made about paying off the debt, and in some cases the resulting cost savings would have been considerable. These two studies give some evidence of the potential benefit of explicit TD management, and they show that these benefits can be significant in some instances, but not universally.
The second two studies were aimed at characterizing the costs of explicit TD management. Two case studies were conducted on live projects in which a very basic TD management process was introduced. In both cases, the costs of TD management were dominated by the costs of initially identifying the TD in the code base and building an initial TD list. In one of the studies, the project incurred very low costs due to TD management over several sprints, once the initial TD list was built. The second case study was on a project that was larger than the first, in a more mature organization, with tighter budget and schedule constraints and a remote customer. In this second case study, the initial high costs of getting TD management going discouraged the team from continuing to track TD, and explicit TD management was effectively abandoned.
I’ve left out a lot of details, and there are many take-aways from this set of results, but one message that seems clear to me is that trying to tackle TD on a grand scale (i.e. trying to identify and track “all” TD, or even all of one type of TD) is not effective. On the benefit side, not all decisions on TD payoff were improved by having more TD information. On the cost side, the initial cost of constructing any kind of comprehensive TD list is not only high, but can discourage commitment to further TD management.
So how might a project choose what subset of the TD space to focus on for explicit TD management? Which types of TD, or even which specific TD items, should be explicitly identified, documented, estimated and analyzed? Perhaps it should be the obvious ones, i.e. the ones that are “big”, that appear to have a large potential impact. Perhaps it should be the ones that have caused the project problems in the past, i.e. the project’s “pain points”.
What needs to be done? Future research in this area should start with more field studies, in particular case studies that observe in depth the decision making processes in live projects. Without a formal defined process, how do projects decide what TD to eliminate? What criteria are used? The answers to these questions then provide the basis for a set of experiments in which TD items are chosen to pay off using different criteria. The TD items are paid off (or their payoff is simulated), and the criteria are compared to see which are better at choosing TD items whose payoff is most cost-effective.
My vision for a solution to this problem is a clear guideline (maybe a set of questions or a checklist) for a project manager to develop a plan that focuses on managing specific types or subsets of TD that are most likely to pay off in better decisions that save the project resources.