How to measure architectural technical…

How to measure architectural technical debt?

Context
My company (ABB) conducted a survey among hundreds of developers, architects, and product managers, asking about the source of technical debt in their products. Many participants pointed to “poor architecture choices” as an important source for technical debt claiming that they do not have repeatable process to deal with such issues. The presence of poor architecture choices first needs to be established in an objective manner. Then costs and potential benefits for addressing the item need to be associated with each debt items. Afterwards a prioritization for repaying the architectural debt items can be made.

Research Question
How to measure architectural technical debt reliably, objectively, and in a repeatable manner?

Own Work
As architectural documentation is often not in a good condition to be analyzed, the source code and the architects and developers themselves are usually the best information source for measuring architectural debt. Existing source code analysis tools (e.g., SonarQube, NDepend, etc.) only include a few architectural metrics (Survey of Metrics), and mainly focus on design-level issues. We have extended and applied such tools on large-scale ABB software systems with varying results (IEEE Software 2013).
We have also worked on formal structural architecture models (IEEE Software 2011), as well as on formal architecture decision models, as a prerequisite for automated technical debt analysis (WICSA 2014, WICSA 2015, available as Open Source at CodePlex and GitHub).
To interview architects about technical debt, ATAM workshops have proven to be a useful tool to overcome a lack of communication that often leads to technical debt.

Related work
Architecture metrics for source code which are proposed in literature (e.g., Sakar2007, Bouwers2009) often require some subjective parameters as input therefore the metric values may not be comparable across products. Architecture decisions that are not or only indirectly captured in source code, e.g., the choice of a third-party library or the architectural constraints to assure conceptual integrity, are difficult to capture and analyze for architectural debt. Some case studies on architectural debt have been carried out (Bouwers2013, Martini2015).

Future work
More and better architecture metrics need to be established and validated for source code, design documents and requirements. Community benchmarks based on Open Source Systems could help improving existing metrics.
To analyze the technical debt coming out of poor architecture decisions, there is a need to capture decisions more formally, without overwhelming the users. Technical debt analysis tools for such models incorporating software economic models would be the next step.