Characteristics of Legacy System Reengineering

Rick Dewar

Division of Informatics, The University of Edinburgh, EH9 3JZ, UK.
Email: Rick.Dewar@ed.ac.uk, Tel: +44 (0)131 650 5139, Fax: +44 (0)131 667 7209

Abstract

This paper presents five patterns which demonstrate some of the characteristic forms and forces associated with the reengineering of legacy systems. A discussion of some of these characteristics is included with the intention that they may ultimately provide a useful means of evolving and navigating the pattern catalogue. The patterns here are all drawn from observation of real industrial case studies.

Introduction

Pattern: Stepping Stone. Live & Let Die. Divide & Modernise. Middleware. Swaying Palms & Shifting Sand.
Problem: How do you move towards the corporate preferred packages? How do you move towards the corporate preferred packages? How can the system be migrated to supportable technology? How can new applications communicate and cooperate with the legacy and access its services? How can the upheaval of the inevitable change in IT systems be minimised?
Solution: Migrate to an interim package that is readily compatible with the legacy and target systems.  Run the target and legacy in parallel for some time, letting the legacy gradually die. Translate the relevant part of the database to a modern format and rewrite/translate the associated code.  Whilst handling any consistency and gateway issues, remove the redundant data and code from the original legacy. Now reengineer the separated component.  Purchase a message oriented middleware product. Provide gateways on the legacy and new applications to allow the distributed systems to connect, interoperate and cooperate. Develop separate applications in lieu of adding complexity to the legacy. Model the domain to capture business objects. Introduce a translation layer between BPs and legacy, communicating through business objects as much as possible. 
Table 1 - Pattern Synopses

The author has recently observed a number of different reengineering efforts involving legacy systems. This paper considers the characteristics of some of these projects and presents, in a pattern form,  the associated problems and solutions discovered in such situations. Five patterns are presented which are likely to be seen a number of times in forthcoming work in some form or other - see Table 1.

Characteristics

It is reasonable to assume that characteristics of the case studies seen so far will recur in a variety of combinations and permutations in other reengineering efforts. With this in mind, this section documents the forces that have provoked the projects, as well as the forms of the legacies that have been encountered. More generally, this section also discusses the often conflicting forces that can influence the direction of reengineering.

Specific Characteristics

The characteristic forces seen so far that have instigated the reengineering of legacy systems have been: The characteristic forms of legacy systems have been: As case study evidence grows, so will these lists of characteristics.

Broader Issues

Generic Forces: The current and future states of legacies are a function of many forces including, but not limited to: what already exists; what resources are and will be available; the culture of the organisation in which they exist; the history of the legacy's development; the organisation's market; and the business processes being supported. These are in addition to the more common project management pressures of time, cost, quality and risk.

Make or Buy: In manufacturing industry there is a recurring dilemma of whether to make a part in-house or to buy it from a third party.  A parallel exists in the development of IT solutions. For a large organisation, there may be economies of scale if off-the-shelf applications are in widespread use. However, such standardisation may stifle creativity and cause disruption to business functions trying to conform. On the other hand, smaller more dynamic companies may find the chaotic evolution of ad hoc solutions allow faster responses to market conditions. They would have to balance this flexibility with not being able to share risks with a vendor and having to retain expensive support personnel. Still another circumstance occurs when an off-the-shelf product does not exist and the company were forced to develop their own systems.

Legacy - Friend or Foe? It is interesting to note that a legacy system need not always constitute a malevolent presence - after all they probably embody a great deal of organisational knowledge and intent. As long as they do not hinder the business and can actively support its activities, they can have a long and successful tenure. However, the company legacy can be malevolent if it is inflexible and dictates business processes which may not be helpful to the company. In addition, the malevolent legacy may consume considerable maintenance resources, process workflow too slowly or be prone to excessive downtime.

Fashion Victims: The author has recently heard of one instance where a legacy's programming language was so obscure that employees were threatening to leave the company since their skill set was not marketable. This meant problems with training, retaining and recruiting staff. In this case, the company is seriously considering major reengineering in order that the applications be written in a more common (fashionable) language.

Roles: The different roles within an organisation can also influence the direction reengineering takes. For instance the IT function may favour a more strategic migration which will improve the maintainability of the legacy and position it better for future developments. However, the marketing function can demand such a quick turn around of new functionality and services to make such proactive measures difficult. In addition, the accounting regime may demand project payback in such a short space of time that more costly future proofing cannot be supported.

Stepping Stone

Context

Your company is part of a group of companies. The group has a strategic IT policy, part of which prescribes the standard packages which business units should adopt. Your business unit has one or more legacy system which do not conform to the group's strategic preferences. It is difficult to migrate to the target systems due the legacies' data and/or functionality being incompatible with the target, however, you have identified a package which is readily compatible with both the legacy and target systems. Group predicts overall savings by moving to specific products, but you and your users may experience inconvenience during the change. Group understands that there are difficulties and expects migrations to take a number of years.

Problem

How do you move towards the corporate preferred packages?

Solution

Migrate to an interim package that is readily compatible with the legacy and target systems.  See Figure 1.


Figure 1 - Moving to Corporate Preferred Packages via an Interim Package

Consequences

By providing an interim package, work can proceed in manageable, stable phases. However, having to purchase it, as well as the target system, will increase overall costs. In an effort to keep down training costs and disruption, minimise changes to the system's external views, for example print-outs and screen formats. You may want to consider Divide & Modernise during the migrations, in order to phase the project. Since the interim solution is transient, and in order to manage expectations, make it clear to users that the change is like for like. However, it is unlikely that system enhancements can cease during the tenure of the interim solution and training will be needed for development personnel.

When no interim solution exists, you may wish to consider Live & Let Die. In addition, you could adopt Divide & Modernise and Middleware to rationalise the responsibilities of the legacy.

Known uses

This pattern has been seen in a small business unit of a large manufacturing group. Their existing main business system was a highly customised database product on an ageing mainframe. Its software and hardware were no longer supported since the original vendor had gone out of business. The target system was an MRP II package specified by group. It would have been difficult and risky to convert the legacy's code and data to the target directly and so an interim package was purchased. This package was supported and would easily accept the functionality and data from the legacy. In addition, its database was ODBC which would make the eventual migration to the target MRP II package more straightforward. The final migration step is still difficult since the MRP II package would necessitate changes to existing business processes and product structures, but the overhead of having to deal with data extraction and consistency has been reduced.

Live & Let Die

Context

Your company is part of a group of companies. The group has a strategic IT policy, part of which prescribes the standard packages which business units should adopt. Your business unit has one or more legacy system which do not conform to the group's strategic preferences. It is difficult to migrate to the target systems due the legacies' data and/or functionality being incompatible with the target. Group predicts overall savings by moving to specific products, but you and your users may experience inconvenience during the change. Group understands that there are difficulties and expects migrations to take a number of years.

Problem

How do you move towards the corporate preferred packages?

Solution

Run the target and legacy in parallel for some time, letting the legacy gradually die. See Figure 2.


Figure 2 - Moving to Corporate Preferred Packages, Letting Legacy Die

Consequences

You can conduct all new business on the target system, introducing training and interfaces as necessary, and handle referencing and existing contracts on the legacy. As old contracts come to an end, the legacy will fall into disuse and the target system will take over. Maintaining two systems which work in parallel, carry out the same function, but do not communicate, will increase running costs. A certain discipline will have to be applied to stop users falling back on the legacy with which they may have more familiarity and confidence - particularly in the beginning. This discipline could be brought about by identifying a specific date at which the legacy will no longer by accessible for editing (see Deprecation [1]).

This solution may not be desirable for situations where legacy data needs to be maintained for a considerable time, for instance life insurance contracts. An alternative is to adopt Stepping Stone. In addition, you may wish to consider rationalising the responsibilities of the legacy in which case Divide & Modernise and Middleware may be of interest.

Known uses

This pattern has been seen in a small business unit of a large manufacturing group. They possessed an ageing Computer-Aided Design (CAD) package which was no longer supported in terms of hardware or software. The data on this system could not be converted into a form which would allow its useful deployment on the group preferred CAD system. In this case, the company chose to run the two systems in parallel. The legacy gradually fell into disuse and now there is only one terminal left to access its data for reference purposes.

Divide & Modernise [1]

Context

You have a legacy system whose technology is obsolete and soon to be unsupported. An area of functionality, such as a payroll component, has been identified which is relatively decomposable from the rest of the legacy. You have decided that significant modifications to the dying legacy system are undesirable. You may also have considered wrapping the system. Although this is sometimes useful, is not a good solution here because it perpetuates the use of the unsupported technology. If a new system is developed to replace part of the old one, the developers will be expected to provide ideal functionality. Consequently, it will be impossible to manage expectations and the project will become large and risky.

Problem

How can the system be migrated to supportable technology?

Solution

Translate the relevant part of the database to a modern format and rewrite/translate the associated code.  Whilst handling any consistency and gateway issues, remove the redundant data and code from the original legacy. Now reengineer the separated component.

Consequences

You should consider whether to migrate the code or the data first depending on your own priorities and resources. Once the code (or data) is migrated you have a stable working system using an interim gateway. When the migration of the data (or code) starts, the gateway will have to be replaced or modified. Whichever you choose to migrate first, constructing the gateways is non-trivial, requiring careful planning.

Nevertheless, by not attempting to change the code you have acquired a new system providing part of the functionality of the original legacy. You are now left with a smaller legacy and a separate, manageably sized component of the original legacy on which you can consider further reengineering. See Figure 3.


Figure 3 - Divide & Modernise

Overall, work proceeds in distinct manageable phases. Even if a requirements explosion does overtake the final restructuring step, the main aim, that of removing the dependency of the functionality on the obsolete technology, will have been achieved.  On the negative side, code and data are migrated before the new requirements of the system are analysed, which means that some of this effort may turn out to have been wasted.

Known uses

This pattern is being used in BT (British Telecommunications plc) as they attempt to migrate data and functionality away from their monolithic customer service system [2]. It is only one of a number of strategies being pursued, such as Middleware. The various forces for change in BT include: a need to provide better customer profiles and market information; and to reduce the amount of training required by operators of the system.

Another example comes from literature where Brodie & Stonebraker [3, pp. 67 & 87] have followed such a strategy. They used a combination of moving the data first (forward migration) and the applications first (reverse migration) to achieve their aims.  Two case studies are provided where the pattern has been applied iteratively until all the legacy has been migrated. The catalyst for both reengineering projects mentioned in the case studies was that maintenance of the legacy was difficult, costly and/or dangerous. One of the projects also wanted to provide customers with up to date information and therefore wished to move away from batch processing. That legacy could not handle on-line transactions.

Middleware

Context

A legacy system exists that is stable and successful. It is in constant use and there are no immediate plans to significantly alter or replace it. However, there are separate systems being developed which could usefully exploit the functionality of and data on the legacy.

Problem

How can new applications communicate and cooperate with the legacy and access its services?

Solution

Purchase a message oriented middleware product. Provide gateways on the legacy and other applications to allow the distributed systems to connect, interoperate and cooperate.

Consequences

The legacy has effectively been wrapped. You will have new applications separate from the legacy which can re-use the existing valuable functionality and data but will not add coupling and complexity to the legacy system (see Figure 4).

Figure 4 - Middleware Architecture

Middleware products can experience capacity problems so the volume of transactions must be anticipated. Instruct the vendors who are tendering that they should supply a basic, working prototype to your own specification and test on your systems to ensure the product can handle the volume of transactions that you have predicted. By having a prototype, you may add cost to the project but will gain a good basis for future development as well as a sound comparative metric for purchasing decisions.

Care should be taken with security and integrity so that the loose coupling between applications and legacy data does not allow data corruption or unnecessary duplication.

Known uses

A major UK financial services organisation has recently followed this strategy after an abortive start where no prototype was procured and capacity issues produced problems for the organisation.

This pattern is also being used in BT (British Telecommunications plc) as they attempt to migrate away from their monolithic customer service system by implementing existing and additional functionality and data in new systems [2]. It is only one of a number of strategies being pursued, such as Divide & Modernise. The main difference here is that BT have developed their own middleware systems. In house development was necessary in 1984 when they began producing middleware due to there not being any commercially available alternatives [4]. However, the principle of prototyping has been followed since their middleware only had a limited application to begin with. The various forces for change in BT include: a need to provide better customer profiles and market information; and to reduce the amount of training required by operators of the system.

Swaying Palms & Shifting Sand

Context

At some time in the future, you will replace your legacy system. The circumstances under which this will happen cannot be predicted and the nature of that change process is unclear, but it will happen and it will cause problems. Your organisation has Business Processes (BPs) which are constantly changing to meet market demands (the swaying palms). The IT legacy (the shifting sand) is based on a widely used generic, domain specific package, but this is being constantly customised to accommodate the changing BPs. When the legacy system is replaced, another domain specific package will take its place. This package will have to have (or be given) similar company specific functionality as the legacy. The organisation would resist developing their own package. Because of it fluid state, the current legacy is poorly documented and knowledge about it resides in the minds and notebooks of the programmers. The customisation is complex and maintenance is expensive.

Problem

How can the upheaval of the inevitable change in IT systems be minimised?

Solution

Develop separate applications in lieu of adding complexity to the legacy. Model the domain to capture business objects. Introduce a translation layer between the BPs and the legacy, communicating through the business objects as much as possible.

Consequences

This pattern offers a proactive strategy to help prepare for the unknown circumstances under which the legacy is to be replaced. By having new and distinct applications separate, the growth in the legacy's size and complexity can be kept to a minimum. The translation layer can protect the BPs from changes in the IT infrastructure to a certain extent and vice versa. Change will still introduce problems but the effects will not be as great.

Figure 5 shows the legacy system supporting the organisation's BPs. The communication between the business and the legacy is conducted through the generic, unchanging business objects - when such objects exist. Since circumstances are constantly changing and the business objects only reflect part of the organisational artefact, the BPs need to communicate directly with the legacy as well. In other words, the legacy is partially wrapped.


Figure 5 - Swaying Palms & Shifting Sand

You may wish to consider rationalising the responsibilities of the legacy in which case Divide & Modernise and Middleware may be of interest.

When modelling the domain, bear in mind that any resulting model will soon be out of date and that only certain entities (for instance invoice or customer) may be all that is of use.

Known uses

A project in a UK bank is employing just such a strategy. Their legacy system is expensive to maintain and will be replaced at some time in the future. Traders in the bank are constantly creating new types of deals as markets and opportunities appear and evolve. These deals are difficult to reconcile with the legacy system until the code has been amended to accommodate them. Their business constants are concepts such as price.

Conclusions

The five patterns presented are: The paper has also tried to categorise some of the case study experiences. At this stage these characteristics are little more than interesting and they provide a higher level of context than any specific pattern. However, it is hoped that such characteristics will help map between related and alternative patterns as the catalogue grows. In addition, they may even be useful to users of the catalogue as they navigate through the alternatives. There are still opportunities to categorise the change process and the target systems in the future as the evidence grows. Again, these may help in cataloguing and accessing the patterns.

As for validating the patterns, there seems to be reason for having some confidence in Divide & Modernise and Middleware by virtue of the evidence seen in their known uses. Furthermore, due to corporations merging, consolidating and globalising (as has been increasingly seen in the oil, automotive and banking sectors), the Stepping Stone and Live & Let Die patterns can only become more common.

Of course, these have been high-level patterns. It is expected that there will be a number of lower level supporting patterns involved in reengineering. For instance the author has already seen examples of problems with vendor management - one of the solutions being to require prototypes before and during the contract. Indeed the organisational patterns of Jim Coplien [5] and the requirements patterns of Bruce Whitenack [6] may prove to have a useful application in supporting reengineering projects.

Acknowledgements

The author would like to acknowledge the support of the UK EPSRC (grant GR/M02491) and his colleagues Perdita Stevens, Rob Pooley and Ashley Lloyd. In particular, thanks must be paid to this paper's EuroPLoP'99 shepherd, Wolfgang Keller.

References

  1. Deprecation and Divide & Modernise have already been published in a similar form: Stevens, P.,  Pooley, R. (1998) Systems Reengineering Patterns,  in proceedings ACM-SIGSOFT 6th International Symposium on the Foundations of Software Engineering, Orlando, Florida, pp.17-23, ISBN 1-58113-108-9.
  2. Harrison, P.F. (1997 ) Customer Service System - past, present and future , "BT Technology Journal", vol.15 , no.1 , pp.29-45.
  3. Brodie, M.L., Stonebraker, M. (1995) "Migrating Legacy Systems : Gateways, Interfaces & the Incremental Approach", Morgan Kaufmann Publishers; ISBN: 1558603301.
  4. Calladine, J. (1997 ) BT Middleware - software as infrastructure , "BT Technology Journal", vol.15 , no.1 , pp.135-146.
  5. Coplien, J.O., A Development Process Generative Pattern Language, http://www.bell-labs.com/people/cope/Patterns/Process/index.html.
  6. Whitenack, B.G., RAPPeL: A Requirements Analysis Process Pattern Language for Object Oriented Development, http://www.bell-labs.com/people/cope/Patterns/Process/RAPPeL/rapel.html.
Further information and other reengineering patterns can be found in [1] or via the group's web page at: http://www.dcs.ed.ac.uk/home/pxs/reengineering-patterns.html.

Copyright (c) 1999 by Rick Dewar. Permission is hereby granted to copy and distribute this paper for the purposes of the EuroPLoP '99 conference.