What Will It Take To Fully Test Your Software? by Hycel Taylor on Aug 28, 2015

In my last blog I asked, “What is the definition of fully tested software” and further defined that “Fully tested software is code that has been validated and tested to such an extent that all concerned parties have ‘high confidence’ that the code will perform exactly as intended once released to production”.

My statement has provoked a deeper conversation on what it means to fully test your software. Some were frustrated with my definition because they felt it wasn’t definitive enough, some think the idea of fully tested software is ridiculously unattainable, some agreed with the general tenor of the definition and are themselves trying to achieve 100% code test coverage, while others are seeking more opinions as to what fully tested software means. To all of you I say, bravo and thank you for your honest responses; my definition bares much scrutiny because it is relative and subject to one’s own standard of perfection.

What Fully Tested Software Doesn’t Mean

Here’s another approach towards finding your own answer to defining what fully tested software is by considering what fully tested software is not.

Fully tested software does not mean that your software will be 100% bug free.

Is it possible to get your software 100 percent bug free? Yes. Is it likely that you can get your software 100% bug free? Probably not. Unless you can think of and test every possible condition of your code’s business logic, there may always be bugs that could at some point in the distant future become manifest and cause your software to fail in some manner; up until that point, your software is technically 100% bug free.

Fully tested software does not mean your code will stay 100 percent bug free.

Your software is constantly evolving, morphing and growing in complexity from release to release. Without rigorous regression testing, continuous integration testing, vigilant testing of new and modified source code, and the continuous creation and use of really good test data, new bugs will inevitably and unwittingly get introduced into your production environment.

Fully tested software does not mean your code won’t be influenced by third party sources that have not been fully tested and thus contain bugs that may alter the expected behavior of your code.

However, if we set aside the above explanations about what fully tested software is not, there is still an argument for fully testing your software to mitigate any factor that prevents your software from performing as expected in production.

Getting your software fully tested has less to do with an exact definition of what fully tested software is and more about a personal attitude, tenor and approach towards reaching a goal of having your software fully tested. In essence, it is an optimistic and proactive attitude that begs the question, “how do I fully test my software and what do I need to accomplish that goal?”

There are many factors in software testing that make the challenge of fully testing software very difficult. However, in all my years of trying to ensure that the code my team or I had implemented was bug free, the number one challenge in meeting that goal was having an abundance of good test data and good test data design patterns for use in testing.

So, if you could attain all of the test data you desire to fully test your software, what would you do?

The Path To Fully Testing Your Software

The path to fully testing your software is not impossible but it is very challenging and there are ways to meet this goal.

If you have the ability to define, model and generate good test data and employ good test data design patterns (e.g. TestDataLoader Design Pattern & TestDataMap Design Pattern), one path towards getting your software fully tested is by writing good old fashioned functional tests and integration tests in an effort to test 100% of your software.

Here’s the hard truth of the matter; for any ten lines of code that is written within a given method or function, you may need to write three times that amount of code to test it.

A Motivating Example

For our own GenRocket test data generation platform we write one or more integration tests for each service method we implement and are often surprised by the bugs we find and eliminate in our own code when we execute a new test. Here’s a good example of a bug in one of our service methods that may have appeared long after the code was in production.

In our OrganizationService we have a method, deleteProjects, containing business logic to delete a project from an organization. In our production environment the code works flawlessly each time we delete a project. So, what’s wrong with the code? Can you see the bug?

The code within the deleteProjects method ran flawlessly until we created an integration test to test deleting an organization. Lo and behold the test failed in the deleteProjects method on the first line of code:

Do you see the bug now? Neither did we until we realized that our business rules do not always allow an organization to have projects under certain conditions. The business logic to delete an organization calls the deleteProjects method to delete its own projects. In our integration test (shown below) the conditions were not met where an organization should have one or more projects nor did we add at least one project to our test organization prior to attempting its deletion.

First we needed to fix the code in the deleteProjects method and the fix was very simple. In Groovy, the ‘?’ character can be used to avoid a null pointer exception. When an organization has no projects, a null pointer exception was being thrown on projects.toArray(). A null pointer exception would also get thrown on the last line of the method, projects.clear().

After applying the fix we implemented two integration tests to fully test the deletion of an organization. Below is a code snippet of the integration test we implemented, testDeleteOrgNoProjects(), to test deleting an organization that contained no projects.

Why It’s Important To Fully Test Your Code

In this application-specific example I have attempted to show that it is not always the obvious bugs found in code that can bring your software to a screeching halt in production; it is more often the subtle bug that can’t be seen until a combination of events allow it to manifest itself. This is why it is so important to want to fully test your software and strive to that end whether or not you meet the goal of testing 100% of your code.

This is where you step up to the challenge; this is where you draw a line in the sand and ask yourself what will it take to fully test your software and then make it so.