Dealing with Technical Debt

The phrase “technical debt” has become a commonly used phrase in software development. Technical debt was introduced by Ward Cunningham to describe the cumulative consequences of corners being cut throughout a software project’s design and development.

Imagine you need to add a feature/functionality to your software. You see two ways of doing it, the short & quick & dirty one that will make further changes harder or the clean way that will take longer to add.

Technical debit is analogous to financial debt; the technical debt incurs interest payments which come in the form of extra effort you have to do in the future because of the quick and dirty design choice. We can choose to continue paying the interest, or we can pay down the principal by refactoring the quick and dirty design into the better design.

“Shipping first time code is like going into debt. A little debt speeds development so long as it is paid back promptly with a rewrite… The danger occurs when the debt is not repaid. Every minute spent on not-quite-right code counts as interest on that debt. Entire engineering organizations can be brought to a stand-still under the debt load of an unconsolidated implementation, object-oriented or otherwise.” (Ward Cunningham, 1992)

technical debt

Technical debt trends to accumulate over time

The time you spend below the line takes away from the time you are able to spend above the line in terms of delivering new value to the customers.

Impacts of Technical Debt

  • Too much time maintaining existing systems, not enough time to add value – fixing defects or n * patching the existing code.
  • Takes more effort (time & money) to add new functionality – you are not able to react to what’s happening on the market in terms of competitive products.
  • User dissatisfaction – usually the functionality you’re adding is not what they expect or they struggle with the products due the amount of defects.
  • Latent defects – a defect that is not known to the customer unless he faces an unforeseen situation but at the same time the developer or the seller is aware of the defect.

Business impact of Technical Debt

  • Higher TCO – total cost of ownership (purchase price of asset + cost of operation)
  • Longer TTM – time to market to deliver new value
  • Reduced agility – less ability to react to changes on the market

Consequences:

  • Innovator’s Dilemma – once dominant in the market, your competition can develop and deliver new functionality faster than you
  • studies reveal that every 1$ of competitive advantage gained by cutting quality / taking shortcuts, it costs 4 x times to restore the quality back to the product
  • The biggest consequence – it slows your ability to deliver future features, thus handing you an opportunity cost for lost revenue

The “source” of Technical Debt

Technical debt is comes from work that is not really Done or steps skipped in order to get the product out of the door (schedule pressure).

A “valuable” source of “undone” work is generated by not knowing what’s required and we add code that performs functionality that’s incorrect. In this scenario we potentially try to tweak the functionality with workarounds by doing some “code manipulation”.

Company culture aka “that’s not my job” can sustain the technical debt (QA vs Dev vs DevOps).

Forms of Technical Debt

  • Defects
  • Lack of automated build
  • High code complexity – hard to modify/test
  • Lack of automatic deployment – reduce human steps involvement
  • Lack of unit tests
  • Business Logic in wrong places
  • Too few acceptance tests
  • High cyclometric complexity
  • Duplicated code or modules
  • Unreadable names / algorithms
  • Highly coupled code

All forms of technical debt are associated with code modify / changes and testing – DONE requires testing:

  • acceptance criteria
  • details of the design/solution
  • automate all tests
  • write unit tests and code
  • review documentation
  • test integrated software
  • regression testing of existing functionality
  • fix broken code

It is important to take sprint capacity and perform the above steps every single sprint.

Paying Off Technical Debt

  1. Stop creating new debt
  2. Make a small payment regularly (eg. each sprint)
  3. Repeat step 2 🙂

1.Stop creating new debt

  • Clearly define what “DONE” work is!
  • Know “what” the functionality is – clear acceptance criteria
  • Automated testing
    • repeatable, fast – do it every time the code changes!
    • regression tests – at least every sprint to be able to run through all your regression tests and know that whatever used to work still works.
    • new functionalities – automatically testing so that can be very clear not only works when it’s new but next sprint when we’re modifying things around functionality, it’s still working as expected.
  • Refactor – not rewriting the application/product but increasing the maintainability of the code without changing the behavior. It can be changes to make code readable/understandable (variable/methods name change to match the true meaning), taking our redundant code and creating a method. Fixing defects is different to refactoring. Refactoring is best done very small steps, a little bit at a time and actually requires automatic testing to make sure that the refactor has not changed the behavior. See Martin Fowler – Refactoring – Improving the design of existing code
  • Automated process – build & release -> eliminate human errors = reduce the source of technical debt. Let computers do repetitive work where human can easily induce errors.

Note: Those are some examples and not an exhaustive list of things that can generate technical debt.

2. Pay off existing debt

  • Fix defects – preferable as soon as you find them. Get them at the top of the list, work them off (zero bug policy), get them out of the code, refactor the code, improve the structure, make the names meaningful, reduce complexity, remove duplicate code, improve the test coverage.
  • Test coverage – it doesn’t mean only lines of code, it means testing though the logical use of the code. Test coverage is expensive so it’s most important to test the product with actually being used – the coverage we care about most because it’s how customers are using our software.

technical-debt-healthy
The goal: to pay the tech debt a little bit at a time every single sprint and avoid “technical bankruptcy”!

3. Prevent tech debt

  • cross-functional teams
  • agreed upon definition of done
  • frequent user feedback
  • discipline, transparency – not taking shortcuts

Additional links:

  1. http://martinfowler.com/bliki/TechnicalDebt.html
  2. http://martinfowler.com/books/refactoring.html
  3. https://www.atlassian.com/agile/technical-debt
  4. http://blogs.ripple-rock.com/SteveGarnett/2013/03/05/TechnicalDebtStrategiesTacticsForAvoidingRemovingIt.aspx

Open / Closed Principle

The Open/Closed Principle (OCP) states that all classes should be open for extension but closed for modification. “Open for extension” means that the design of your class should allow adding new functionality as new requirements are generated. “Closed for modification” means that once you have developed the class, you should never modify it, unless you need to fix bugs.

Even the two parts of the principle appear to be opposite, if you structure your code correctly, you should be able to add functionality without editing existing code. How you achieve this ? Your design should reply on abstractions for dependencies, such as interfaces or abstract classes, rather than using concrete implementation. New functionality can be added by adding new classes than implement the interfaces.

The first benefit of OCP to your code is that you don’t need to change the existing code, once it was written, tested, documented. So you reduce the risk on adding bugs into existing code. Another benefit of using contracts is loose coupling and increased flexibility.

ocpSource: Solid motivational pictures

Let’s consider the GoldPriceAlert class from previous article with the extra feature to allow user to decide the way to be notified: console, email, sms.

The above sample code is a basic module for logging alerts. As you can see if you decide to implement the SMS logging type, you need to modify the GoldPriceAlert method. This violates the OCP.

We can easily refactor the code to achieve the OCP by removing the enum LogType that limits the types of alerts. Then we’ll create a class for each type of logger. Additional log types can be added later without changing any existing code.

The logger class still performs all logging but using one of the classes described earlier. In order that the classes are loose coupled, each message logger type implements ILog interface. The GoldPriceAlert class is never aware of the type of the logger being used as the dependency is provided as an ILog instance using constructor injection.

The refactored code:

Single Responsibility Principle

Let’s start our journey on SOLID principles by describing the Single Responsibility Principle (SRP).

SRP states that each class should have one responsibility only and, therefore, only one reason to change. It means that every class should have one job to do or one single purpose. Of course, we still allow adding multiple methods/properties and members as long they relate to that single responsibility.

Applying SRP will change your existing code and the first impact will be at the class level: the classes in your projects will become smaller and cleaner. As a note, you shouldn’t be worried about changing the classes even you will notice the increased number of new classes. All major programming languages allows you to organize classes using namespaces and project folders. Keeping your code into classes tightly focused into a single purpose leads to code that is simpler to understand and maintain.

Having small and cohesive classes reduces the code fragility by decreasing the chance of a class to contain bugs (reducing the need for changes). As the classes will perform only one duty, multiple classes will work together to achieve larger tasks. It allows you to easy modify the features, either by extending existing classes or introducing interchangeable versions.

Along with the other principles, SRP allows you to achieve loose coupling.

srpSource: Solid motivational pictures

Example:

Let’s consider below code snippet that violates the SRP principle and then refactor it to comply with the principle.

The above class will read gold price from an independent trusted “source” and check if the price is under certain threshold and it will decide to alert me if I need to buy some gold.

There are at least few reasons to change the above class. Let’s consider at some point I need to add a new “trusted” independent source for gold price. Perhaps I want to change the way I’m evaluating the opportunity to buy gold. Or I would like a more sophisticated system to be notified instead of logging out to console.

To refactor the code, we’ll separate the functionality into three classes. The first is GoldMeter.cs that will retain GoldPrice property and ReadGoldPrice method as both are close related. The other methods are removed so that the only reason to change is the replacement of the “trusted” source price.

The second class is GoldPriceChecker.cs that will include a very simple method to compare the price with a certain acceptable value. The only reason to change the class is if the acceptable value is changed.

The final class GoldPriceAlert.cs will display the alert if a very simple way. GoldMeter class is dependency injected. The only reason to change the GoldPriceAlert class is if we decide to enhance the alert system.

The refactored code below breaks some SOLID principles just to ensure the SRP is visible. Further refactoring of the code is required to achieve SOLID compliance.

The Solid principles

The SOLID acronym was introduced by Robert Martin (also known as “Uncle Bob” and represents a list of five guidelines that can enhance the maintainability of the software.

Bad dependency management will make your code rigid, fragile, difficult to reuse. Rigid code is that which is difficult to modify, either to change existing functionality or add new features. Fragile code is susceptible to the introduction of bugs, particularly those that appear in a module when another code is changed.

If you follow SOLID principles, you can produce better and flexible code, more robust and with a higher reusability.

The 5 principles:

1. Single Responsibility Principle (SRP) – each class should have one responsibility and, therefore, only one reason to change.

2. Open / Closed Principle (OCP) – all classes and similar units of source code should be open for extension but close to modification.

3. Liskov Substitution Principle (LSP) – code that uses a base class must be able to substitute a subclass without knowing it.

4. Interface Segregation Principle (ISP) – the client should not be forced to depend upon interfaces that they do not use. Instead, those interfaces should be minimized.

5. Dependency Inversion Principle (DIP) – high level modules should depend upon low level modules and that abstractions should not depend upon details.

Development slang :)

  • YAGNI – “You aren’t gonna need it” – Is a principle of extreme programming (XP) that states a programmer should not add functionality until deemed necessary. XP co-founder Ron Jeffries has written: “Always implement things when you actually need them, never when you just foresee that you need them.” Other forms of the phrase include “You aren’t going to need it” and “You ain’t gonna need it”.
  • DTSTTCPW – Do the simplest thing that could possibly work.
  • KISS – It states that most systems work best if they are kept simple rather than made complicated; therefore simplicity should be a key goal in design and unnecessary complexity should be avoided. Keep it simple, stupid – Variations on the phrase include “keep it short and simple” and “keep it simple and straightforward”.
  • Worse is better – New Jersey style, was conceived by Richard P. Gabriel to describe the dynamics of software acceptance, but it has broader application. The idea is that quality does not necessarily increase with functionality. There is a point where less functionality (“worse”) is a preferable option (“better”) in terms of practicality and usability. Software that is limited, but simple to use, may be more appealing to the user and market than the reverse.
  • DRY – Don’t repeat yourself (DRY) is a principle of software development, aimed at reducing repetition of information of all kinds, especially useful in multi-tier architectures. The DRY principle is stated as “Every piece of knowledge must have a single, unambiguous, authoritative representation within a system.” The principle has been formulated by Andy Hunt and Dave Thomas in their book The Pragmatic Programmer. They apply it quite broadly to include “database schemas, test plans, the build system, even documentation.”
  • WET – commonly taken to stand for either “write everything twice” or “we enjoy typing”
  • Basic software development principles

    1. Solid requirements – clear, complete, detailed, cohesive, attainable, testable requirements that are agreed to by all players. Use prototypes to help nail down requirements. In ‘agile’-type environments, continuous close coordination with customers/end-users is necessary.
    2. Realistic schedules – allow adequate time for planning, design, testing, bug fixing, re-testing, changes, and documentation; personnel should be able to complete the project without burning out.
    3. Adequate testing – start testing early on, re-test after fixes or changes, plan for adequate time for testing and bug-fixing. ‘Early’ testing ideally includes unit testing by developers and built-in testing and diagnostic capabilities.
    4. Stick to initial requirements as much as possible – be prepared to defend against excessive changes and additions once development has begun, and be prepared to explain consequences. If changes are necessary, they should be adequately reflected in related schedule changes. If possible, work closely with customers/end-users to manage expectations. This will provide them a higher comfort level with their requirements decisions and minimize excessive changes later on.
    5. Communication – require walkthroughs and inspections when appropriate; make extensive use of group communication tools – groupware, wiki’s, bug-tracking tools and change management tools, intranet capabilities, etc.; insure that information/documentation is available and up-to-date – preferably electronic, not paper; promote teamwork and cooperation; use prototypes and/or continuous communication with end-users if possible to clarify expectations.