A few years ago I was asked to Coach and Scrum Master a freshly hired team for a large PLC. The team had been given the task of enhancing a system that formed the core of day to day operations for a business unit turning over several hundred million pounds per year:

• Aside from emergency fixes and legislative changes had no development performed on it for 6 years
• Only a handful of developers that knew the system were still with the company and they were on other critical work.
• The code base had been developed over the last 35+ years and comprised of over 1000 programs or dynamically loadable modules.
• No test cases
• Had a home grown version management system that was little more than source copied into file system folders.
• A terminal (“green screen”) application written in an obscure 3gl programming language which is a variant of PL/B known as DBC.
• Had limited documentation

The client was based in “middle England” and was having difficulty attracting staff, particularly for “legacy systems”. Rather than struggle to find staff, they set the project up in London to tap into a richer talent pool. COBOL was regarded as the closest “main stream” language so a team of COBOL programmers was duly hired. In addition to myself the team comprised of a Project Manager who would act as the Product Owner and interface into the wider business, a BA, two programmers who were available on a consultancy basis (technical advisers in effect) one day a week and a selection of business advisers.

At the start of the project we had to learn everything about the language, the application and environment. This meant we had to read the code, asking lots questions of the consulting developers but mostly learning by doing.

We wanted to get a simple first delivery out from the team, partly to satisfy political issues with in the wider IT department and partly for team morale (nothing beats shipping software to build team confidence). Luckily some fairly simple changes were regarded as highest value deliverables. From the first deliverables we established that:

• We had to rely on manual testing (much of which was performed by business users who had all of the rules stored in their minds!).
• Deployments required each of the 30 plus binaries to be hand cranked into each test system (and live) and needed a lot of checking by the developers (a process that became known as “release bingo” as the sizes and date stamps of each file were called out and cross checked) to make sure the right code had been deployed.
• We had also developers who needed to change the same code with in a sprint, the management of this caused some wastage.

The above was clearly inefficient and prone to human error. We decided on a step by step migration approach to run in parallel with our programme of business based enhancements:
• Move the code into the Corporate GIT Repository
• Devise an automated test framework for system testing, capturing business rules from users in a repeatable form.
• Create CI process that compiled to the code and created a deployment package

We considered introducing “JUnit style” unit tests however this would have meant writing a test frame work and working out how to use this in programs that were riddled with “GOTO” statements and had business logic heavily integrated with screen and keyboard handling activity. Where requirements allowed us to create dynamically loadable modules (DLL) (for example a new pricing engine) we created test harness wrappers that allowed us to test the DLL the code against data with known correct answers (basically very simple but effective unit tests).

As well as allowing the system to be regression tested in a repeatable and predictable way, we also saw the automated test scripts as a part of the missing system documentation. Any future workers on the system will benefit from these and the other pieces of technical documentation the team have created.

When looking for a testing framework (open source, in active development and not requiring a degree in ancient Klingon to make it do anything) we were surprised by how few choices were on offer. This reflects the extent to which terminal based applications have declined. In the end I found an open source Java Swing terminal emulator (JTA25) and was able to link this with JUnit and JEMMY (established Java test frameworks) to create a custom macro language that replicated everything a user could do. By including ASSERT statements in the language we were able to check the screen buffer for expected results. The initial deliverable took 4 days to write. Had the project continued into a second year, we could have extended the tool and macro language to link with Fitnesse.

The basic move to GIT was fairly painless. Alas the team struggled to use GIT in the early days, Partly due to their mainframe background so were unfamiliar with the tools that the ’NIX’ and Windows teams have been using for years and partly because we had to work from the command line with no GUI to mask some of GIT’s finer points. We shared knowledge and produced “cheat sheets” and create a PowerPoint that served as the operating manual.

The development of the CI System presented a further set of problems. Our first task was to create a custom build process. The language variant we were using had a compiler but no “standard” interface with any source management system. The system was running on an old version of AIX and due to several infrastructure resource and operational problems it could not be upgraded beyond forcing us to use JDK1.4.2. We elected to install ANT 1.5, extension tools and Rhino JS to create a custom ANT Build that could be used by developers for their local builds but also for the main Jenkins build process. It took a little over two days to get the basic build working.

The ANT Build also packaged the code into a tar file that could be easily transported between systems. This allowed us to end “release bingo” and actually got deployments down to a few minutes instead of several hours.

The core Dev Ops team were overloaded and we weren’t not a priority so we were advised that it would take months before our build would be added to the central Jenkins server. I invested three days in creating a simple Java program (Java 1.4!) to poll GIT and invoke the ANT Build. By dumping the output into HTML with an auto refresh we were able to create a crude but effective build monitor via an Apache Server.

Once the CI system was running we were able to have good builds automatically deployed to the Development System Test Environment, which is where all of the automated test scripts were manually executed. Had the project continued, we would have extended the build process to automatically run the automated tests.

The code base we inherited contained millions of lines of code. While the system was in active development the practice had been to deprecate code by not calling it which resulted in thousands of lines of dead code, all of which was getting the way of us making changes. Over the course of the project we removed over 100,000 lines of dead code. Automated testing allowed us to verify we hadn’t removed anything critical (large more complex programs used DLL’s) and GIT allowed us to track what we had removed in case of any later query.

The final model proved highly reminiscent of some of the high performance Java and C# projects I have previously worked with:
• Local work and test area for each developer.
• GIT and CI Build to integrate changes (developers synced and compiled locally before checking into GIT)
• A Development System Test Environment to test builds
• An Acceptance Environment into which we deployed good builds with agreement from business representatives (these builds were in effect Release Candidates)
• Final UAT on pre production.
• Release to Production

The key learning’s from this project are:

• We have successfully introduced modern XP engineering practices into a legacy code base.
• The development of automated test scripts allowed:
• Business knowledge to be transferred from the minds of key business users and captured in repeatable test scripts. This also left documentation for any future team when they work on the system.
• The team to improve their understanding of the application and have greater confidence when tackling more invasive/complex changes.
• Helped to ensure that a better quality product was made available for UAT.
• The time taken to develop the ANT build script, CI System and automated deployment:
• Generated massive time savings enabling more time to be spent on feature development.
• Reduced errors and mistakes

The project for ran for a year by the end of which we had completed several major upgrades and numerous small fixes and improvements. The client had circa 20 scrum teams running and we were seen as one of the most productive and efficient teams in the organisation. This mostly due to hard work and commitment of the team; the use of modern software engineering practices greatly contributed to quality and speed of the delivery.