Continuous Software Layer Testing for Better DevOps

  • Using automation can lead to functional and technical stability of multi-tiered, complex software projects.

  • Automating regressions AND functional tests can help implement Test-Driven Development (TDD)which Test-Driven Development defines requirements for developers better than acceptance criteria.

  • Projects should not fear rapid cycles with large numbers of small changes, even when deadlines are tight, as this improves stability and increases quality.

Many Software projects become chaotic despite good project management and carefully considered development processes. The deadline factor, rogue development of prototypes, or the existence of many heterogenous delivery units combine to create chaos. We were able to calm a chaotic project by implementing test automation and test-driven development using our own Test and Automation Framework (TAF-Nuts) – characterized by an emphasis on practicality (just getting the job done), trust (let good testers test and let good developers develop) and simplicity.

This article identifies the characteristics of the chaos and describes how ProArchCon made the project successful. By explaining our approach, we offer suggestions for projects in similar situations. ProArchCon is a member company of the Solutionheads EEIG and the author of this article is an associate member and the senior technologist of the group.

The Situation

At one of Germany’s national mobile, fixed-net and cable network and service providers, our role was to augment the combined on-shore / off-shore, internal delivery unit’s front-end team in software quality for the customer self-service web portal. The legacy system architecture included four backend applications, one front-end application and two separate service layer applications. Each element in the architecture was developed by a different external, off-shore vendor and was tested by further external, off-shore units. Only the acceptance test was carried out by onshore, internal units. The front-end development was carried out by internal delivery units (albeit offshore).

Since software users only ever see the front-end, each error (even those originating in the backend) is perceived as a front-end problem. We saw our role thus in terms of total quality, not confined to front-end quality.

Ensuing Chaos

We entered the project and found it characterized by untested backend and service layer software being rolled into the acceptance test environment. The negative impact was sufficient to completely block development on functional requirements and force a focus on stability issues. This unplanned new focus only exacerbated pressures caused by aggressive deadlines.

The front-end software was developed and functionally tested against stubs. These stubs were based on the official documentation of the backend. Only after successful integration in the test environment (removing the stubs), could the front end declare its software as finished. The front-end development was thus dependent on a stable and functioning backend.

We decided to halt front-end development prior to assessing the quality of the backend and service layer. ProArchCon found evidence, that software was being released uncontrollably into and crashing in the test environment. Indications were:

  • When using tools to test the service, there would be HTTP 400-type errors coming from both the backend and the service layer.

  • The service would return a HTTP 200-type response, but the values and the format of the JSON response did not match the official interface specification.

  • Between any two runs of the software, the errors returned were functionally different, indicating that there were uncontrolled and unplanned changes to the software being constantly implemented.

Implementing Test Driven Development and Automation

ProArchCon attempted to communicate the problems in test to the responsible levels of management in status reports. The individual delivery units claimed that there was no problem, or that the problem was not caused by their unit. We were faced with the situation of having to develop against software that was not ready and of having to prove to others that the software failing in test. To do this and to rapidly improve software quality and stability We introduced test automation and test-driven development into the project. Neither method is new, but our TAF-Nuts framework with its strong focus on practicality, trust and simplicity is rare and was new to the customer and the project.

We find the following explanation of TDD very appropriate:

“Test-Driven development is a software development process that relies on the repetition of a very short development cycle: first the developer writes an (initially failing) automated test case that defines a desired improvement or new function, then produces the minimum amount of code to pass that test, and finally refactors the new code to acceptable standards.” - Farcic, Viktor. “Test Driven Development (TDD): Example Walkthrough.” Technology Conversations, 20 Dec. 2013,

Test Automation of the Regression Test

We used our own Nuts automation tool. Testing the database was accomplished using the DB Nuts for Oracle. Testing the backend processes was accomplished using the BE Nuts for Linux. Testing the service layers was accomplished using the REST and the JSON Nuts and testing the front-end was done using the Web Nut. All five Test Nuts were then combined with a Chain Nut and the chain was scheduled to run every 30 minutes.

A “Nut” is an automation abstraction for a particular technology that allows the automation to be configured rather than “developed” and without knowledge of the technology-specific software implementation.

The same test of the software was run every thirty minutes. Full documentation of each test run was then sent to the developers. Because the first issue was stability, we provided an OK / NOK indication for each service and each layer of the architecture.

Reviewing the results of several days of continuous test, we discovered changes being made to the software and released into the test environment without previous test throughout the day. ProArchCon used the documented test results to communicate to project management and the delivery units and force a focus on stability issues rather than functional development alone.

Our test automation approach further aided backend and service development by indicating which Nut was throwing which error. The development teams could quickly identify which layer and which application was erroring (by comparing the output of the backend, Linux Nuts with the output of the database nuts).

We gave the Nut automation software and configuration to the different delivery units so that they could perform the regression test on their software before delivering into the acceptance test environment.

Test Driven Developement (TDD)

Using test automation for regression test was necessary to improve the stability of the software delivery from the delivery units and to align the efforts of all units. There was still the challenge of meeting aggressive project deadlines fulfilling the functional requirements. To aid in this area, we implemented an end-to-end, test-driven development methodology.

The test-driven development was set up in the following manner to allow for rapid implementation of functional changes to the software:

  • Implement the end-to-end perspective – focusing on logical uses of the software originating with a user request and encompassing all actions at all software tiers to respond to the user request

  • Cycle through large numbers of small changes

  • Use quality gates prior to the release for acceptance test

These three points together help to quickly develop and iteratively expand upon a minimum viable product (MVP) which is essential in defining a functional change for customer-facing software.

End-To-End Perspective

All test cases were first written in the user / customer perspective. The functional, test-driven test case was then automated on the front end. At the beginning of each implementation cycle, the new, automated, functional test was executed. Based on the detailed results, TDD cases were then written for the service and backends layers of the software.

The functional test for the TDD was automated using the ProArchCon Nuts Framework. By automating the test, the developers could expediently execute the test as often as they needed to see if the changes, they developed were successful. Once the automated, TDD functional tests were successful after an implementation cycle, the results were incorporated into the automated regression tests.

Large Number of Small Changes

We convinced the developers to drastically reduce the size of their developed changes. The goal was to be able to have two implementation cycles (including test) per day. This allowed the software to grow slowly in functionality, while being able to pinpoint the causes of errors quickly.

The limited scope of the implementation cycles required more planning at the developer level. While project management concentrated on the requirement and acceptance criteria level, we sat down each morning and each afternoon with the lead developers to plan the detailed changes. This “overhead” focused the efforts of the entire and improved quality and consistency across all tiers of the solution architecture.

Quality Gates Prior to Release for Acceptance

Using a combination of the automated regression and functional TDD tests meant that there could be a strong quality gate at each iteration of software development. An automatic regression test marked the beginning of each implementation cycle across all ties of the solution. Once it was certain that the regression tests were successful, the end-to-end TDD functional automated test cases were performed.

We helped at all layers of the solution architecture with the development of automated TDD test cases. When all development units signaled that they were done with their development, all units would run the automated TDD case. When both the automated regression test and the automated TDD functional test were successful, the next iteration could start.

Reflection and Lessons Learned

The project was a success. With only a three-week delay in an eleven-month project, it went into production including all required functionality. ProArchCon was able to improve stability across all delivery units and set up structures that made subsequent projects easier.

Success Factors

The project benefitted from the existence of a substantial code base prior to the changes and from having developers on board who were familiar with this code base. If the code base and the developers did not exist, it is not likely that such a granular level of control at the developer level would have been possible.

A great deal of the change that we were able to implement was possible because there was project management in place that trusted its developers. If there is a tendency in project management to micro-manage or a lack of responsibility in the senior developers, it would not have been possible to implement such a granular TDD.

Lessons Learned

The conditions may not be the same from project to project, but the following lessons learned are widely applicable:

  1. Stability before Functionality / Software stability first! A good deal of problems in the project were caused by wild development and poor communication between the delivery units. Before new functionality can be implemented in a heterogenous environment, it is important to establish a stable and functioning baseline. By constant/continuous comparison to this baseline, the development and troubleshooting of new functionality becomes easier. Test automation substantially improves stability, reduces test cycle durations, and eliminates manual test errors.

  2. Test-Driven Development Defines Requirements for the Developers In scrum / agile projects there is a lot of effort spent on defining acceptance criteria. Acceptance criteria is useful in defining the use cases of a software or application, but in environments where there is a large proportion of service layer and backend development, a “use case” of the software may be too abstract for development. By defining the results of functional tests of the software at each layer, the requirement to development is much clearer for the developers.

  3. Automate Every Step Implementing test automation across all layers in the software architecture for a project aligns the efforts of the developers and allows for quick assessment of project status and quality. By automating regression test, a project can guarantee stability between and beyond implementation cycles. By automating the functional test in the TDD context, the developers can gain speed in checking their progress and knowing when development is ready for the next steps in the process.


Tom Allen is the general manager and senior technologist at ProArchCon GmbH, an IT service provider based in North-Rhein Westphalia in Germany. ProArchCon has created the Test and Automation Framework – Nuts. The TAF-Nuts includes the Nuts software for coordinating test and automation across all layers of a software architecture as well as the necessary consulting services to improve quality in general. Test automation is the key to software stability and reliability. For more information about TAF-Nuts see:

Tom Allen is also an associate partner with the SolutionHeads EEIG – the group’s senior technologist. More information about the SolutionHeads can be found here:

This Article is available for download here:

Continuous Testing Better DevOps
Download PDF • 108KB

94 Ansichten0 Kommentare

Aktuelle Beiträge

Alle ansehen