Is your team or project moving so fast or under such budget constraints that you are certain you cannot afford unit tests? If so then at the very least, please write acceptance tests.

Unchecked technical debt can quickly become a significant contributing root cause of complete business failure

Acceptance tests - Tests that confirm that the core behavior of a system is behaving as intended, are easier to write than ever before and if used correctly can improve team communication and actually speed up development, even when doing rapid prototyping.

Projects like TaikoJs, and https://github.com/goblinfactory/draki, make it even easier and more affordable than ever to write cross platform, cross Browser tests that test the core functioning of your application in the same way that users use your application. These tests can be read and verified by non developers and become a means to automatically verify that the application is working as intended.

update : since writing this blog post back in 2019 I've since since switched to using Puppetteer, Specflow, Cypress and cypress-cucumber-preprocessorand Jest for automation testing; this blog post still remains 100% relevant today.

If you're a startup, I don't recommend you aim for 100% coverage with unit tests. You quickly reach a point of diminishing returns and money will be better spent delivering new features and running experiments to find product market fit.

And the cost of no acceptance tests?

Your acceptance tests on the other hand become the agreed contract of what is being built. It is the ultimate definition of your 'requirements'.

I therefore highly recommend that you have sufficient tests to prove that your application works exactly as 'required'.

As a challenger business grows, revenues, workload and demand for new features increase, and your code base will rapidly grow in complexity. Having a full suite of acceptance tests is the absolute bare minimum you need as a startup. Everything else can be made more robust when you have the revenue and capital. Even unit tests can be added later, but acceptance tests can almost never be added without people having to guess at what the details of the requirements were at the time the code was written. Without acceptance tests, the knowledge of what was asked for, and what was ultimately agreed during demos and discussions with clients will quickly be lost over time, even if the original team are still in your company.

Technical debt

The result of all the 'good intention assumptions about requirements' is technical debt that manifests itself as a slow exponential increase in bugs causing lengthy delays to releases as manual testing costs sky-rocket. Most startups never fully recover from the resulting strangling costs, eventually paying for it via a crippling penalty in their final exit price (a significant multiple far below investor expectations). Unchecked technical debt can quickly become a significant contributing root cause of complete business failure and non-exit for the founders.

Run-away technical debt if left unchecked, can result in a toxic work environment where employees are punished for speaking the truth, and rewarded for making excuses, causing a cascade of issues as the smartest talent are the first to start leaving.

Acceptance tests on the other hand capture the problem and the solution, and if written correctly include sufficient examples to remove any doubt as to how the requirements should be interpreted. When errors eventually are detected, it's often due to misunderstanding in the requirements, and you have a single source of truth that can be updated to reflect the corrected understanding, either with new examples for specific edge cases, or updates to the requirements themselves.

Unit tests help very little with any of these problems. They only remove a narrow set of bugs caused by code not working as the developer intended, and do not stop bugs caused by team members making faulty assumptions about the problem space and the exact necessary solution.

So can we save money by not writing unit tests?

While the acceptance tests give you evidence that your business is delivering something that the customer will and can use, unit tests are proof for the developers that other developers can use (and extend without breaking) complicated and-or complex components that are used to construct the final solution. The short answer is 'no' you need 'just enough' tests, consisting of unit tests as well as integration tests.

Too many unit tests written in the wrong way can paradoxically create the same business problems as too few tests. Unit tests should test the behavior of the system being tested and not the implementation.

for example:

Lets say your devs write an algorithm for determining the estimated predicted weight of goods being shipped to an address on a given future date by doing something super cool with machine learning and the historical order data.

If the unit tests for that algorithm tested the implementation instead of the 'outcome', then anyone attempting to improve the efficiency of the algorithm would cause tens if not hundreds of unit tests to fail, despite modifications actually improving the results. The business would literally be 'punished' for running an experiment. So these types of unit tests would slow you down, instead of speed you up.

If the tests focused instead on the desired behavior (with the implementation treated as a sealed black box), then the implementation can be slowly modified or experimented with, and the same tests can be used to verify whether the new implementation still delivered the required behavior. Smart tests can even return a measure of the efficiency and you can instantly tell if the code changes made the application better, or worse. Written in this way the unit tests (technically geeks call these 'integration tests') would literally speed up development, and they are not as expensive to write as you think, provided you start writing them correctly early.

The recently published State of DEVOPS 2018 report comparing the elite group (teams with excellent testing and CICD processes) against the low performers concluded that teams with excellence were 2555 times faster (when measuring lead time from committing code, to deploying code), with 7 times lower change failure rate, and 2604 times faster to recover from incident.

2555 times faster to deploying code, with 7 times lower change failure rate, and 2604 times faster to recover from incident

By writing fewer brittle unit tests and focusing effort instead on ensuring you have acceptance tests for all (100%) of your requirements, your business will be taking the first step towards the excellence described in the report, making a significant and immediate measurable impact to the business.

Speak to me, I've helped teams get there

Improving your teams ability to rapidly deliver value, while making trade off decisions about time to market, resourcing and quality, is extremely difficult. This is where Goblinfactory can help. We can give you options, and explain what other companies have done in similar circumstances. We can be your sounding board when working through the tough pragmatic decisions that are in both your business, your team and your customer's best short and long term interests.

Making the un-testable testable</h2>

If you have no tests, and want to talk through what is the best way to start putting tests in place, we can help with that as well. Goblinfactory is expert at working with significantly complex brownfield projects and has extensive experience at working with legacy applications, <br/>making the 'untestable' 'testable'.