Dealing with Technical Debt

The phrase “technical debt” has become a commonly used phrase in software development. Technical debt was introduced by Ward Cunningham to describe the cumulative consequences of corners being cut throughout a software project’s design and development.

Imagine you need to add a feature/functionality to your software. You see two ways of doing it, the short & quick & dirty one that will make further changes harder or the clean way that will take longer to add.

Technical debit is analogous to financial debt; the technical debt incurs interest payments which come in the form of extra effort you have to do in the future because of the quick and dirty design choice. We can choose to continue paying the interest, or we can pay down the principal by refactoring the quick and dirty design into the better design.

“Shipping first time code is like going into debt. A little debt speeds development so long as it is paid back promptly with a rewrite… The danger occurs when the debt is not repaid. Every minute spent on not-quite-right code counts as interest on that debt. Entire engineering organizations can be brought to a stand-still under the debt load of an unconsolidated implementation, object-oriented or otherwise.” (Ward Cunningham, 1992)

technical debt

Technical debt trends to accumulate over time

The time you spend below the line takes away from the time you are able to spend above the line in terms of delivering new value to the customers.

Impacts of Technical Debt

  • Too much time maintaining existing systems, not enough time to add value – fixing defects or n * patching the existing code.
  • Takes more effort (time & money) to add new functionality – you are not able to react to what’s happening on the market in terms of competitive products.
  • User dissatisfaction – usually the functionality you’re adding is not what they expect or they struggle with the products due the amount of defects.
  • Latent defects – a defect that is not known to the customer unless he faces an unforeseen situation but at the same time the developer or the seller is aware of the defect.

Business impact of Technical Debt

  • Higher TCO – total cost of ownership (purchase price of asset + cost of operation)
  • Longer TTM – time to market to deliver new value
  • Reduced agility – less ability to react to changes on the market

Consequences:

  • Innovator’s Dilemma – once dominant in the market, your competition can develop and deliver new functionality faster than you
  • studies reveal that every 1$ of competitive advantage gained by cutting quality / taking shortcuts, it costs 4 x times to restore the quality back to the product
  • The biggest consequence – it slows your ability to deliver future features, thus handing you an opportunity cost for lost revenue

The “source” of Technical Debt

Technical debt is comes from work that is not really Done or steps skipped in order to get the product out of the door (schedule pressure).

A “valuable” source of “undone” work is generated by not knowing what’s required and we add code that performs functionality that’s incorrect. In this scenario we potentially try to tweak the functionality with workarounds by doing some “code manipulation”.

Company culture aka “that’s not my job” can sustain the technical debt (QA vs Dev vs DevOps).

Forms of Technical Debt

  • Defects
  • Lack of automated build
  • High code complexity – hard to modify/test
  • Lack of automatic deployment – reduce human steps involvement
  • Lack of unit tests
  • Business Logic in wrong places
  • Too few acceptance tests
  • High cyclometric complexity
  • Duplicated code or modules
  • Unreadable names / algorithms
  • Highly coupled code

All forms of technical debt are associated with code modify / changes and testing – DONE requires testing:

  • acceptance criteria
  • details of the design/solution
  • automate all tests
  • write unit tests and code
  • review documentation
  • test integrated software
  • regression testing of existing functionality
  • fix broken code

It is important to take sprint capacity and perform the above steps every single sprint.

Paying Off Technical Debt

  1. Stop creating new debt
  2. Make a small payment regularly (eg. each sprint)
  3. Repeat step 2 🙂

1.Stop creating new debt

  • Clearly define what “DONE” work is!
  • Know “what” the functionality is – clear acceptance criteria
  • Automated testing
    • repeatable, fast – do it every time the code changes!
    • regression tests – at least every sprint to be able to run through all your regression tests and know that whatever used to work still works.
    • new functionalities – automatically testing so that can be very clear not only works when it’s new but next sprint when we’re modifying things around functionality, it’s still working as expected.
  • Refactor – not rewriting the application/product but increasing the maintainability of the code without changing the behavior. It can be changes to make code readable/understandable (variable/methods name change to match the true meaning), taking our redundant code and creating a method. Fixing defects is different to refactoring. Refactoring is best done very small steps, a little bit at a time and actually requires automatic testing to make sure that the refactor has not changed the behavior. See Martin Fowler – Refactoring – Improving the design of existing code
  • Automated process – build & release -> eliminate human errors = reduce the source of technical debt. Let computers do repetitive work where human can easily induce errors.

Note: Those are some examples and not an exhaustive list of things that can generate technical debt.

2. Pay off existing debt

  • Fix defects – preferable as soon as you find them. Get them at the top of the list, work them off (zero bug policy), get them out of the code, refactor the code, improve the structure, make the names meaningful, reduce complexity, remove duplicate code, improve the test coverage.
  • Test coverage – it doesn’t mean only lines of code, it means testing though the logical use of the code. Test coverage is expensive so it’s most important to test the product with actually being used – the coverage we care about most because it’s how customers are using our software.

technical-debt-healthy
The goal: to pay the tech debt a little bit at a time every single sprint and avoid “technical bankruptcy”!

3. Prevent tech debt

  • cross-functional teams
  • agreed upon definition of done
  • frequent user feedback
  • discipline, transparency – not taking shortcuts

Additional links:

  1. http://martinfowler.com/bliki/TechnicalDebt.html
  2. http://martinfowler.com/books/refactoring.html
  3. https://www.atlassian.com/agile/technical-debt
  4. http://blogs.ripple-rock.com/SteveGarnett/2013/03/05/TechnicalDebtStrategiesTacticsForAvoidingRemovingIt.aspx

MM 001

“Your people will watch your feet, not your lips” said Tom Peters.

If you can’t start a meeting on time, they will not be able to finish a project on time!

 

Dual-Track Scrum

If your product team is frustrated by the amount of waste and the slow pace in achieving actual business results, consider trying out Dual-Track Scrum.

Dual Track Scrum is a framework in which a product development team is continuously discovering what their customers needs are, and validating those needs using evidence and prototypes. Once an idea or need is validated, the need is then represented in the form of user stories and added into the product backlog.

Dual-Track Scrum advocates that the product owner/manager, the lead developer, and the UX/UI person constantly spend their time in the discovery space, while the rest of the team concentrates their efforts in delivery. So there are two tracks, each with a different focus but with one aim: to build products customers will love.

The Discovery track is all about quickly generating validated product backlog items, and the Delivery track is all about generating releasable software.
Dual Track ScrumSource: here

Open / Closed Principle

The Open/Closed Principle (OCP) states that all classes should be open for extension but closed for modification. “Open for extension” means that the design of your class should allow adding new functionality as new requirements are generated. “Closed for modification” means that once you have developed the class, you should never modify it, unless you need to fix bugs.

Even the two parts of the principle appear to be opposite, if you structure your code correctly, you should be able to add functionality without editing existing code. How you achieve this ? Your design should reply on abstractions for dependencies, such as interfaces or abstract classes, rather than using concrete implementation. New functionality can be added by adding new classes than implement the interfaces.

The first benefit of OCP to your code is that you don’t need to change the existing code, once it was written, tested, documented. So you reduce the risk on adding bugs into existing code. Another benefit of using contracts is loose coupling and increased flexibility.

ocpSource: Solid motivational pictures

Let’s consider the GoldPriceAlert class from previous article with the extra feature to allow user to decide the way to be notified: console, email, sms.

The above sample code is a basic module for logging alerts. As you can see if you decide to implement the SMS logging type, you need to modify the GoldPriceAlert method. This violates the OCP.

We can easily refactor the code to achieve the OCP by removing the enum LogType that limits the types of alerts. Then we’ll create a class for each type of logger. Additional log types can be added later without changing any existing code.

The logger class still performs all logging but using one of the classes described earlier. In order that the classes are loose coupled, each message logger type implements ILog interface. The GoldPriceAlert class is never aware of the type of the logger being used as the dependency is provided as an ILog instance using constructor injection.

The refactored code:

Single Responsibility Principle

Let’s start our journey on SOLID principles by describing the Single Responsibility Principle (SRP).

SRP states that each class should have one responsibility only and, therefore, only one reason to change. It means that every class should have one job to do or one single purpose. Of course, we still allow adding multiple methods/properties and members as long they relate to that single responsibility.

Applying SRP will change your existing code and the first impact will be at the class level: the classes in your projects will become smaller and cleaner. As a note, you shouldn’t be worried about changing the classes even you will notice the increased number of new classes. All major programming languages allows you to organize classes using namespaces and project folders. Keeping your code into classes tightly focused into a single purpose leads to code that is simpler to understand and maintain.

Having small and cohesive classes reduces the code fragility by decreasing the chance of a class to contain bugs (reducing the need for changes). As the classes will perform only one duty, multiple classes will work together to achieve larger tasks. It allows you to easy modify the features, either by extending existing classes or introducing interchangeable versions.

Along with the other principles, SRP allows you to achieve loose coupling.

srpSource: Solid motivational pictures

Example:

Let’s consider below code snippet that violates the SRP principle and then refactor it to comply with the principle.

The above class will read gold price from an independent trusted “source” and check if the price is under certain threshold and it will decide to alert me if I need to buy some gold.

There are at least few reasons to change the above class. Let’s consider at some point I need to add a new “trusted” independent source for gold price. Perhaps I want to change the way I’m evaluating the opportunity to buy gold. Or I would like a more sophisticated system to be notified instead of logging out to console.

To refactor the code, we’ll separate the functionality into three classes. The first is GoldMeter.cs that will retain GoldPrice property and ReadGoldPrice method as both are close related. The other methods are removed so that the only reason to change is the replacement of the “trusted” source price.

The second class is GoldPriceChecker.cs that will include a very simple method to compare the price with a certain acceptable value. The only reason to change the class is if the acceptable value is changed.

The final class GoldPriceAlert.cs will display the alert if a very simple way. GoldMeter class is dependency injected. The only reason to change the GoldPriceAlert class is if we decide to enhance the alert system.

The refactored code below breaks some SOLID principles just to ensure the SRP is visible. Further refactoring of the code is required to achieve SOLID compliance.

The Solid principles

The SOLID acronym was introduced by Robert Martin (also known as “Uncle Bob” and represents a list of five guidelines that can enhance the maintainability of the software.

Bad dependency management will make your code rigid, fragile, difficult to reuse. Rigid code is that which is difficult to modify, either to change existing functionality or add new features. Fragile code is susceptible to the introduction of bugs, particularly those that appear in a module when another code is changed.

If you follow SOLID principles, you can produce better and flexible code, more robust and with a higher reusability.

The 5 principles:

1. Single Responsibility Principle (SRP) – each class should have one responsibility and, therefore, only one reason to change.

2. Open / Closed Principle (OCP) – all classes and similar units of source code should be open for extension but close to modification.

3. Liskov Substitution Principle (LSP) – code that uses a base class must be able to substitute a subclass without knowing it.

4. Interface Segregation Principle (ISP) – the client should not be forced to depend upon interfaces that they do not use. Instead, those interfaces should be minimized.

5. Dependency Inversion Principle (DIP) – high level modules should depend upon low level modules and that abstractions should not depend upon details.

Development slang :)

  • YAGNI – “You aren’t gonna need it” – Is a principle of extreme programming (XP) that states a programmer should not add functionality until deemed necessary. XP co-founder Ron Jeffries has written: “Always implement things when you actually need them, never when you just foresee that you need them.” Other forms of the phrase include “You aren’t going to need it” and “You ain’t gonna need it”.
  • DTSTTCPW – Do the simplest thing that could possibly work.
  • KISS – It states that most systems work best if they are kept simple rather than made complicated; therefore simplicity should be a key goal in design and unnecessary complexity should be avoided. Keep it simple, stupid – Variations on the phrase include “keep it short and simple” and “keep it simple and straightforward”.
  • Worse is better – New Jersey style, was conceived by Richard P. Gabriel to describe the dynamics of software acceptance, but it has broader application. The idea is that quality does not necessarily increase with functionality. There is a point where less functionality (“worse”) is a preferable option (“better”) in terms of practicality and usability. Software that is limited, but simple to use, may be more appealing to the user and market than the reverse.
  • DRY – Don’t repeat yourself (DRY) is a principle of software development, aimed at reducing repetition of information of all kinds, especially useful in multi-tier architectures. The DRY principle is stated as “Every piece of knowledge must have a single, unambiguous, authoritative representation within a system.” The principle has been formulated by Andy Hunt and Dave Thomas in their book The Pragmatic Programmer. They apply it quite broadly to include “database schemas, test plans, the build system, even documentation.”
  • WET – commonly taken to stand for either “write everything twice” or “we enjoy typing”
  • INVEST

    The characteristics of a good user story agreed into SCRUM methodology:

    Letter Meaning Description
    I Independent The user story should be self-contained, in a way that there is no inherent dependency on another user story.
    N Negotiable User stories, up until they are part of an iteration, can always be changed and rewritten.
    V Valuable A user story must deliver value to the end user.
    E Estimable You must always be able to estimate the size of a user story.
    S Scalable (small sized) User stories should not be so big as to become impossible to plan/task/prioritize with a certain level of certainty.
    T Testable The user story or its related description must provide the necessary information to make test development possible.

    Independent – One of the characteristics of Scrum is the ability to move stories around, taking into account their relative priority. If you find user stories that are tightly dependent, a good idea might be to combine them into a single user story.

    Negotiable– The only thing that is fixed and set in stone in an agile project is an iteration backlog (and, even then, it can be broken). While the story lies in the product backlog, it can be rewritten or even discarded, depending on business, market, technical or any other type of requirement by team members.

    Valuable – The focus here is to bring actual project-related value to the end-user. Coming up with technical stories that are really fun to code but bring no value to the end-user violates one of the Agile Principles, which is to continuously deliver valuable software.

    Estimable – If a user story size cannot be estimated, it will never be planned, tasked, and, thus, become part of an iteration. So there’s actually no point in keeping this kind of user story in the Product Backlog at all. Most of the time, estimation cannot be executed due to the lack of supporting information either in the story description itself or directly from the Product Owner.

    Scalable (small sized) – Try to keep your user story sizes to typically a few person-days and at most a few person-weeks. Anything beyond that range should be considered too large to be estimated with a good level of certainty or even “epic” and broken down into smaller user stories. There’s no problem in starting with epic stories, as long as they are broken down when the time to place them in an iteration backlog becomes closer.

    Testable – A story should only be considered DONE, among other things, if it was tested successfully. If one cannot test a story due to lack of information (see “Estimate-able” above), the story should not be considered a good candidate to be part of an iteration Backlog.

    Related material: INVEST in Good Stories, and SMART Tasks by Bill Wake

    Software project management concepts

    Prince2 = Projects in Controlled Environments

    Keys:

    • Is a process driver project management method, contrasts with reactive/adaptive methods (Scrum).
    • divided into manageable stages (enable efficient control of resources)
    • inappropriate for small projects or where requirements are expected to change.
    • each process is specified with it’s key input/outputs and specific goals/activities to be carried out -> automatic control of deviations.

    7 principles:

    • continued business justification
    • learn from experience
    • defined roles and responsibilities
    • manage by stages
    • manage by exception
    • focus on products
    • tailored to suit the project environment

    7 themes:

    • business case
    • organization
    • quality
    • plans
    • risks
    • change
    • progress

    7 processes:

    • Starting up a project (SU)
      • project brief, business justification
      • overall approach, next stage of the project planned
      • asking project board to authorize the next stage (initiating the project)
      • activities:
        • forming project board
        • preparing project brief
        • designing and appointing a project management team
        • defining the project approach
        • planning next stage (initiation)
    • Initiating a project (IP)
      • project brief -> business case
      • ensure quality of the project
      • overall approach to control the project (project controls)
      • creating project files
      • creating plan for the next stage of the project
      • activities:
        • planning quality
        • refining business case and risks
        • setting up project controls
        • setting up project files
        • assembling Project Initiation Document
    • Directing a project (DP)
      • dictates how the project board (roles as executive sponsor/project sponsor) should control the project
      • cover ad hoc direction of the project and how project should be closed down.
      • activities:
        • authorizing initiation
        • authorizing a project
        • authorizing a stage or exception plan
        • giving ad-hoc direction
        • confirm project closure.
    • Controlling a stage (CS)
      • projects should be broken down into stages and these sub-processes dictate of each individual stage should be controlled.
      • include the way in which work packages are authorized and received
      • way of monitoring the progress
      • how we escalate project issues
      • activities:
        • authorizing work package
        • assessing progress
        • capturing and examining project issues
        • reviewing stage status
        • reporting highlights
        • taking corrective action
        • escalating project issues
        • receiving a complete work package
    • Managing stage boundaries (SB)
      • dictates what should be done towards the end of a stage
      • planning next stage, overall project plan
      • risk register
      • business case amendment
      • reporting stage end
      • updating project plan
    • Managing product delivery (MP)
      • control the link between Project Manager and Team Manager(s):
      • objectives:
        • ensure work on products allocated to team is authorized and agreed.
        • team manager(s), team members and suppliers are clear as to what is to be produces and effective effort, cost, timescales.
        • planned products are delivered to expectations and within tolerance
        • activities:
          • accept a work package
          • execute a work package
          • deliver a work package.
    • Closing a project (CP)
      • covers things that should be done at the end of the project.
      • activities:
        • decommissioning a project
        • identify follow-on actions
        • project evaluation review.