Here you will find experience reports, points of view and opinions of our members on a variety of industry, business and social topics.  We have just started publishing, so follow us on

Social Media or visit this page again for more.

All contributions are available for download at the end of the correspondning post.

  • Using automation can lead to functional and technical stability of multi-tiered, complex software projects.

  • Automating regressions AND functional tests can help implement Test-Driven Development (TDD)which Test-Driven Development defines requirements for developers better than acceptance criteria.

  • Projects should not fear rapid cycles with large numbers of small changes, even when deadlines are tight, as this improves stability and increases quality.

Many Software projects become chaotic despite good project management and carefully considered development processes. The deadline factor, rogue development of prototypes, or the existence of many heterogenous delivery units combine to create chaos. We were able to calm a chaotic project by implementing test automation and test-driven development using our own Test and Automation Framework (TAF-Nuts) – characterized by an emphasis on practicality (just getting the job done), trust (let good testers test and let good developers develop) and simplicity.

This article identifies the characteristics of the chaos and describes how ProArchCon made the project successful. By explaining our approach, we offer suggestions for projects in similar situations. ProArchCon is a member company of the Solutionheads EEIG and the author of this article is an associate member and the senior technologist of the group.

The Situation

At one of Germany’s national mobile, fixed-net and cable network and service providers, our role was to augment the combined on-shore / off-shore, internal delivery unit’s front-end team in software quality for the customer self-service web portal. The legacy system architecture included four backend applications, one front-end application and two separate service layer applications. Each element in the architecture was developed by a different external, off-shore vendor and was tested by further external, off-shore units. Only the acceptance test was carried out by onshore, internal units. The front-end development was carried out by internal delivery units (albeit offshore).

Since software users only ever see the front-end, each error (even those originating in the backend) is perceived as a front-end problem. We saw our role thus in terms of total quality, not confined to front-end quality.

Ensuing Chaos

We entered the project and found it characterized by untested backend and service layer software being rolled into the acceptance test environment. The negative impact was sufficient to completely block development on functional requirements and force a focus on stability issues. This unplanned new focus only exacerbated pressures caused by aggressive deadlines.

The front-end software was developed and functionally tested against stubs. These stubs were based on the official documentation of the backend. Only after successful integration in the test environment (removing the stubs), could the front end declare its software as finished. The front-end development was thus dependent on a stable and functioning backend.

We decided to halt front-end development prior to assessing the quality of the backend and service layer. ProArchCon found evidence, that software was being released uncontrollably into and crashing in the test environment. Indications were:

  • When using tools to test the service, there would be HTTP 400-type errors coming from both the backend and the service layer.

  • The service would return a HTTP 200-type response, but the values and the format of the JSON response did not match the official interface specification.

  • Between any two runs of the software, the errors returned were functionally different, indicating that there were uncontrolled and unplanned changes to the software being constantly implemented.

Implementing Test Driven Development and Automation

ProArchCon attempted to communicate the problems in test to the responsible levels of management in status reports. The individual delivery units claimed that there was no problem, or that the problem was not caused by their unit. We were faced with the situation of having to develop against software that was not ready and of having to prove to others that the software failing in test. To do this and to rapidly improve software quality and stability We introduced test automation and test-driven development into the project. Neither method is new, but our TAF-Nuts framework with its strong focus on practicality, trust and simplicity is rare and was new to the customer and the project.

We find the following explanation of TDD very appropriate:

“Test-Driven development is a software development process that relies on the repetition of a very short development cycle: first the developer writes an (initially failing) automated test case that defines a desired improvement or new function, then produces the minimum amount of code to pass that test, and finally refactors the new code to acceptable standards.” - Farcic, Viktor. “Test Driven Development (TDD): Example Walkthrough.” Technology Conversations, 20 Dec. 2013,

Test Automation of the Regression Test

We used our own Nuts automation tool. Testing the database was accomplished using the DB Nuts for Oracle. Testing the backend processes was accomplished using the BE Nuts for Linux. Testing the service layers was accomplished using the REST and the JSON Nuts and testing the front-end was done using the Web Nut. All five Test Nuts were then combined with a Chain Nut and the chain was scheduled to run every 30 minutes.

A “Nut” is an automation abstraction for a particular technology that allows the automation to be configured rather than “developed” and without knowledge of the technology-specific software implementation.

The same test of the software was run every thirty minutes. Full documentation of each test run was then sent to the developers. Because the first issue was stability, we provided an OK / NOK indication for each service and each layer of the architecture.

Reviewing the results of several days of continuous test, we discovered changes being made to the software and released into the test environment without previous test throughout the day. ProArchCon used the documented test results to communicate to project management and the delivery units and force a focus on stability issues rather than functional development alone.

Our test automation approach further aided backend and service development by indicating which Nut was throwing which error. The development teams could quickly identify which layer and which application was erroring (by comparing the output of the backend, Linux Nuts with the output of the database nuts).

We gave the Nut automation software and configuration to the different delivery units so that they could perform the regression test on their software before delivering into the acceptance test environment.

Test Driven Developement (TDD)

Using test automation for regression test was necessary to improve the stability of the software delivery from the delivery units and to align the efforts of all units. There was still the challenge of meeting aggressive project deadlines fulfilling the functional requirements. To aid in this area, we implemented an end-to-end, test-driven development methodology.

The test-driven development was set up in the following manner to allow for rapid implementation of functional changes to the software:

  • Implement the end-to-end perspective – focusing on logical uses of the software originating with a user request and encompassing all actions at all software tiers to respond to the user request

  • Cycle through large numbers of small changes

  • Use quality gates prior to the release for acceptance test

These three points together help to quickly develop and iteratively expand upon a minimum viable product (MVP) which is essential in defining a functional change for customer-facing software.

End-To-End Perspective

All test cases were first written in the user / customer perspective. The functional, test-driven test case was then automated on the front end. At the beginning of each implementation cycle, the new, automated, functional test was executed. Based on the detailed results, TDD cases were then written for the service and backends layers of the software.

The functional test for the TDD was automated using the ProArchCon Nuts Framework. By automating the test, the developers could expediently execute the test as often as they needed to see if the changes, they developed were successful. Once the automated, TDD functional tests were successful after an implementation cycle, the results were incorporated into the automated regression tests.

Large Number of Small Changes

We convinced the developers to drastically reduce the size of their developed changes. The goal was to be able to have two implementation cycles (including test) per day. This allowed the software to grow slowly in functionality, while being able to pinpoint the causes of errors quickly.

The limited scope of the implementation cycles required more planning at the developer level. While project management concentrated on the requirement and acceptance criteria level, we sat down each morning and each afternoon with the lead developers to plan the detailed changes. This “overhead” focused the efforts of the entire and improved quality and consistency across all tiers of the solution architecture.

Quality Gates Prior to Release for Acceptance

Using a combination of the automated regression and functional TDD tests meant that there could be a strong quality gate at each iteration of software development. An automatic regression test marked the beginning of each implementation cycle across all ties of the solution. Once it was certain that the regression tests were successful, the end-to-end TDD functional automated test cases were performed.

We helped at all layers of the solution architecture with the development of automated TDD test cases. When all development units signaled that they were done with their development, all units would run the automated TDD case. When both the automated regression test and the automated TDD functional test were successful, the next iteration could start.

Reflection and Lessons Learned

The project was a success. With only a three-week delay in an eleven-month project, it went into production including all required functionality. ProArchCon was able to improve stability across all delivery units and set up structures that made subsequent projects easier.

Success Factors

The project benefitted from the existence of a substantial code base prior to the changes and from having developers on board who were familiar with this code base. If the code base and the developers did not exist, it is not likely that such a granular level of control at the developer level would have been possible.

A great deal of the change that we were able to implement was possible because there was project management in place that trusted its developers. If there is a tendency in project management to micro-manage or a lack of responsibility in the senior developers, it would not have been possible to implement such a granular TDD.

Lessons Learned

The conditions may not be the same from project to project, but the following lessons learned are widely applicable:

  1. Stability before Functionality / Software stability first! A good deal of problems in the project were caused by wild development and poor communication between the delivery units. Before new functionality can be implemented in a heterogenous environment, it is important to establish a stable and functioning baseline. By constant/continuous comparison to this baseline, the development and troubleshooting of new functionality becomes easier. Test automation substantially improves stability, reduces test cycle durations, and eliminates manual test errors.

  2. Test-Driven Development Defines Requirements for the Developers In scrum / agile projects there is a lot of effort spent on defining acceptance criteria. Acceptance criteria is useful in defining the use cases of a software or application, but in environments where there is a large proportion of service layer and backend development, a “use case” of the software may be too abstract for development. By defining the results of functional tests of the software at each layer, the requirement to development is much clearer for the developers.

  3. Automate Every Step Implementing test automation across all layers in the software architecture for a project aligns the efforts of the developers and allows for quick assessment of project status and quality. By automating regression test, a project can guarantee stability between and beyond implementation cycles. By automating the functional test in the TDD context, the developers can gain speed in checking their progress and knowing when development is ready for the next steps in the process.

Tom Allen is the general manager and senior technologist at ProArchCon GmbH, an IT service provider based in North-Rhein Westphalia in Germany. ProArchCon has created the Test and Automation Framework – Nuts. The TAF-Nuts includes the Nuts software for coordinating test and automation across all layers of a software architecture as well as the necessary consulting services to improve quality in general. Test automation is the key to software stability and reliability. For more information about TAF-Nuts see:

Tom Allen is also an associate partner with the SolutionHeads EEIG – the group’s senior technologist. More information about the SolutionHeads can be found here:

This Article is available for download here:

Continuous Testing Better DevOpspdf

  • Katrin Tillenburg

Der Anfang: Revenue Assurance

Revenue Assurance (RA) ist ein Spezialbegriff der Telekommunikationsbranche, für den ich bisher in keiner Sprache eine Übersetzung gefunden habe. Direkt übersetzt: Einnahmensicherung? Eigentlich ein ganz normaler Prozess, jede Firma sollte sicherstellen, daß sie Ihre Leistungen auch von den Kunden bezahlt bekommt. Oftmals wurde RA als das reine Zählen von CDRs (Call Data Records, die einzelnen Abrechnungssätze für Telefonieleistungen aller Art) angesehen, jedoch hat immer schon wesentlich mehr zu dem Thema gehört. Ein paar simple, aber leicht nachzuvollziehende Beispiele:

  • Endet ein gewährter Rabatt nach der festgelegten Zeitspanne

  • Erhält jeder Kunde eine (korrekte, zeitnahe) Rechnung und zahlt diese auch

  • Ist die Kombination von Tarifen und Diensten gültig

Natürlich auch alles Fragestellungen, die in anderen Industriezweigen eine Rolle spielen.

Viele Kontrollen gab es in diversen Unternehmen selbstverständlich schon vorher, auch wenn man das anders nannte, vielleicht ganz simpel „Controlling“. Unter anderem das Telemanagement Forum – TMF ( hat frühzeitig begonnen, einheitliche Standards wie Metriken und KPIs im RA Umfeld zu definieren. So konnten die Unternehmen von den Erfahrungen anderer profitieren und sich sogar vergleichen. Z.B. bei der Kennzahl „%-Wert der wiederhergestellten Verluste“. (Für weitere Details zu Metriken, KPIs oder dem gesamten RA Framework des TMF kann man sich direkt an das TMF wenden, ich stehe aber auch gerne für Gespräche und Informationen zur Verfügung).

Das TMF führt seit über 8 Jahren regelmäßig Umfragen zum Stand RA weltweit durch, auf die soeben fertiggestellte Umfrage soll später noch eingegangen werden.

Nächste Schritte

Aber nicht nur die Umsätze müssen sichergestellt werden, auch die Kosten die verursacht werden müssen auf ihre Richtigkeit hin überprüft werden. Neudeutsch: Cost Assurance. Bei Telekommunikationsfirmen können diese in einem erheblichen Maße entstehen. Und sogar wenn kein eigenes Netz vorhanden ist sind das sämtliche Vordienstleister, zudem Roaming Gebühren, Sonderrufnummern (z.B.0800) aber heutzutage auch Dienstleistungen aller Art, die über die Telefonrechnung mit bezahlt werden (mobile payments). So hat das Telekommunikationsunternehmen Beziehungen sowie Geldströme mit allen möglichen anderen Firmen, die geprüft werden müssen. Viele Telekommunikationsfirmen betrachten diesen Bereich der Prüfungen als einen festen Bestandteil von RA.

Mit dem Vorhandensein aller Daten zu Kosten und Umsätzen ist dann der Schritt zur Margenanalyse leicht gemacht. Je nach Aufwand und Toolunterstützung kann man dies bis auf Einzelkundenanalyse herunterbrechen und hat somit auch gleich eine Kundenwertanalyse mit implementiert, die als Basis weiterer Kundenkontakte und gezielter Marketingmaßnahmen herangezogen werden kann.

Die Konsequenz: Business Assurance

Im Rahmen ständig neuer Anforderungen an TK-Firmen (Digitalisierung, neue Technologien wie 5G / SD-WAN) werden seit einiger Zeit weitere Sicherungsmaßnahmen sowie Risikodisziplinen unter dem erweiterten Dach der Business Assurance zusammengefasst.

Etwa 90% der Teilnehmer der letzten RA Umfrage des TMFs sehen Business Assurance (BA) als natürliche Erweiterung von RA:

Abbildung 1 / Frage 5

Nach aktueller TMF Definition gehören folgende Disziplinen zu Business Assurance:

  • Revenue / Cost Assurance Management

  • Fraud Management

  • Asset Assurance Management

  • Margin Assurance Management

  • Migration Assurance Management

  • Transformation Assurance Management

  • Ecosystem Assurance Management

  • Regulatory Assurance Management

  • Customer Experience Management

Der bisher aufgezeigte Evolutionspfad zeigt, dass BA Experten deutlich mehr machen, als Datensätze zu zählen. RA war immer schon ein Bereich, der mit nahezu allen Abteilungen des Unternehmens kommunizieren musste. Z.B. der Finanzbereich – hier ist RA nach der letzten Umfrage auch in etwa 60% der befragten Firmen aufgehangen / IT und Billing / Marketing bei der Einführung neuer Produkte/ je nach Unternehmenstruktur mit diversen anderen. Die gezeigte Übersicht der Disziplinen verdeutlicht auch, dass die Anforderungen an eine Business Assurance Abteilung sehr vielfältig sind. Das heißt auch, dass verschiedene Skills und Rollen zu besetzen sind: Business Analysten / Operational Analysten / Daten Analysten / Data Scientisten / …

Auch komplett neue Techniken wie Blockchain, Maschine Learning und AI werden vermutlich zunehmend eine Rolle im Bereich der Business Assurance spielen. Bisher sind die aktiven Überlegungen dazu aber noch gering, hier ist es interessant zu sehen, wie die Entwicklung in der nächsten Umfrage weiter aussehen wird.

Hier die Management Zusammenfassung der diesjährigen Umfrage.

Abbildung 2 / Managementzusammenfassung

Neben des bisher beschriebenen Fakten finden sich hier auch Informationen zur generellen Abdeckung der Prüfungen, den gefundenen Verlusten, den geplanten Erweiterungen sowie einigen anderen Details, alles auch betrachtet in der Entwicklung der vergangenen Jahre.

Die komplette Umfrage ist hier zu finden:

( Weitere Informationen zu Revenue und Business Assurance sind, wie bereits erwähnt auch direkt beim TMF zu erhalten.

Und ich freue mich auf einen regen Austausch zu dem Thema. Danke!

Von RA zu BApdf

This article is also available for

download here >>>>>>>>>

Katrin Tillenburg ist eine unabhängige Beraterin im Telekommunikationsumfeld, spezialisiert auf BSS, Billing und Business Assurance. Seit mehr als 25 Jahren arbeitet sie in internationalen Projekten, ist aktives Mitglied im TMF und Gründungspartner im SolutionHeads Netzwerk.


LinkedIn XING

  • Katrin Tillenburg

The beginning: Revenue Assurance

Revenue Assurance (RA) is a special term in the telecommunications industry for which I have not yet found an immediately understood translation in any language! Actually RA is a ubiquitous process across all service and manufacturing industries at the heart of cash-flow management! Every company needs to ensure that they receive payment from their customers for their goods and services, as scheduled, and for the correct amount.

Often RA was seen as simply “counting” CDRs (Call Data Records - the individual call records for telephony services of all kinds with their associated billing rates). However, there is much more to the subject. Here are a few simple easy to understand examples:

  • Does a granted discount end after the specified time period?

  • Does each customer receive a (correct & prompt) invoice and pay on time?

  • Is the combination of tariffs and services valid?

Of course, these are all questions that many other branches of industry are concerned with and certainly revenue assurance controls already exist in various companies, even if they are termed differently, perhaps quite simply "controlling" or “cash management”.

Among others, the Telemanagement Forum TMF ( started early to define uniform standards such as metrics and KPIs in the RA environment. In this way, companies could benefit from the experiences of others and even compare themselves e.g. with the key figure "% value of recovered losses". (For further details on metrics, KPIs or the entire RA framework of the TMF, you can contact the TMF directly, but I am also available for discussions and information).

The TMF has been conducting regular surveys on the state and development of RA worldwide for more than 8 years and I will discuss some of the findings of the just finished report later.

Next steps

Not only do the revenues have to be assured, but also the costs of delivering that revenue have to be verified and accurately attributed - so called Cost Assurance. For telecommunication companies, these costs can be considerable, especially if they do not operate their own network. Such costs will include all upstream service provider infrastructure charges, roaming charges, special numbers (e.g. 0800) and also increasingly application services of many kinds that are paid for via the telephone bill e.g. mobile payments.

The telecommunication company will have supply and service agreements and of course therefore cash, credit and debit flows with a diverse ecosystem of partners which have to be verified and audited. Many telecommunication companies consider this area of auditing as an integral part of RA.

With the availability of all data on costs and sales across the supply and customer service chain, the step to detailed margin analysis is then made more easily. Depending on the effort required and support of appropriate analytical tools which are readily available, a margin analysis can be broken down to single customer on a per product basis. This enables a detailed customer value analysis as a basis for further customer contacts and targeted marketing measures.

The consequence: Business Assurance

In the context of constantly changing customer requirements for telecommunications companies coupled to rapidly advancing technologies supporting e.g. digitalization, IOT, SDN, 5G / SD-WAN, Edge) additional security measures and risk management disciplines have been combined under the extended umbrella of business assurance for some time now.

About 90% of the participants of the last RA survey of the TM Forum see Business Assurance (BA) as a natural extension of Revenue Assurance (RA):

According to the current TM Forum definition, the following disciplines belong to Business Assurance:

  • Revenue / Cost Assurance Management

  • Fraud Management

  • Asset Assurance Management

  • Margin Assurance Management

  • Migration Assurance Management

  • Transformation Assurance Management

  • Ecosystem Assurance Management

  • Regulatory Assurance Management

  • Customer Experience Management

The evolutionary path so far shows that BA experts do much more than just analyse data sets. While RA has always been an area that has had to communicate with most departments of the company, the RA activity itself sits mainly in Finance in about 60% of the companies surveyed with increasing involvement with IT and Billing and Marketing when introducing new products. However, the requirements for a Business Assurance department are much more diverse with greater impact on company performance, business planning and customer service. This also means that different skills and roles have to be filled e.g. Business Analysts, Operational Analysts, Data Analysts, Data Scientists etc.

Completely new techniques such as block chain, machine learning and AI are also likely to play an increasing role in business assurance. So far, however, there has been little active consideration of these, so it will be interesting to see how developments develop in the next survey.

Here is the management summary of this year's survey.

In addition to the facts described so far, you will also find information on the general coverage of the audits, the losses found, the planned extensions and some other details, all also considered in the development of the past years.

The complete survey can be found here:


Further information about Revenue and Business Assurance can be obtained, as already mentioned, directly from the Telemanagement Forum.

And I am looking forward to a lively exchange on the topic. Thank you.

From RA to BApdf

This article is also available for

download here >>>>>>>>>

More about the entrepreneurs’ network SolutionHeads EEIG can be found at:

Katrin Tillenburg is an independent telco consultant, specialized in the BSS, Billing and Business Assurance. Since more than 25 years working in international projects, she is also an active TMF member and founding partner in the SolutionHeads network.


LinkedIn XING

© 2019 by solutionheads