Change Management

© Paul Christian Nelis, November, 2024 - All Rights Reserved

Experts write books on change management, as it’s a rich topic, and there is lots of detail to explore in order to be thorough in addressing the subject. This article represents a type of cliff-notes approach to broadly address the categories which would be chapters in a full treatment of the topic.  The intent is to ensure the reader is aware of these categories, and can research more completely any areas in which their experience or understanding could be bolstered. 

Successfully managing change, and accounting for that management, are called for in NIST SP 800-53 Rev 5.1.1, specifically the Maintenance Control and System and Service Acquisition families.  Any well run organization will aspire to supporting these guidelines, and regulated organizations will be expected to support them.


In gross measure there are three phases to managing the introduction of change:



In organizations comfortable taking on operational risk, prep is short and the concluding steps are all but ignored.  In organizations which seek to avoid down-time, preparation may be the most involved of the three steps. 



There are several reasons to involve “The Business” in the process of change.  In most of these articles, we will differentiate Informatics from The Business by saying that the Informatics team is that portion of the organization which concerns itself with whether or not the application runs, whether or not users can connect, whether or not the application can connect to other systems it requires, and whether or not the application can be restored if it stops running.  The Business, by contrast, includes the population which use the application to accomplish business objectives.  If it’s an HR application, it’s recruiters, or the HR Helpdesk, or the Benefits Managers who would be “The Business” using the application.  Finance and Accounting represent The Business with respect to the General Ledger’s application(s), etc.


Ensuring that The Business, the consumers of the application, are direct participants in the change process helps cement their buy-in of the change process, and helps protect the overall outcome of the change process.  If Informatics folk are the only ones tapping the keys through the change process, it will typically felt by the business to be done to the business.  When the business participates directly, it’s more likely to feel more collaborative.


While the sociological reasons are important, there are technical reasons as well. There are typically three different types of user accounts that interact with an application: end-user, super-user, and systems administrator.


End-user accounts should be the majority of accounts interacting with a given system.  For good reason, the NIST AC-06 control calls for the limitation of application privileges to the least amount necessary for the execution of tasks given that role within the application.  This means that end-users who are responsible for, say, scheduling encounters between patients and physicians should probably not also be able to assign permissions to other users, or edit the details of the physician’s contact information.  End-user accounts have deliberately limited permissions within the application.


Super Users, by contrast are accounts typically held by senior personnel in the business who need to be able to perform the functions of multiple end-user roles. They often have all the permissions required to do what end-users normally do within the application.


System Administrators ideally do not have the rights of end-users or Super Users. In order to respect the goals of NIST AC-05 there should be limits to what Super Users can do—checks and balances that divide the world of application permissions to limit the ability of one user account to perpetrate fraud within the application.  Systems administrators typically deal with identity management (creating/removing end-user accounts) and the link the application may have with external systems, whether that’s an application in an adjoining server, or a cloud-based application across the country.  Systems Administrators are also typically responsible for keeping the application up and running, and potentially restoring it to a working state if something goes wrong. 


What does this have to do with change management?  Testing the success of a change event should include testing the real-world experiences of these three different types of roles as they interact with the application after the change has occurred.  In order for the organization to have confidence that the change has been successful, end-users should be able to demonstrate that the core functions of their day job still work after the change.  Super Users should still be able to do those things the ordinary end-users cannot, and System Administrators should still be able to accomplish their tasks within the application.


Organizations with tight informatics-business relationships have folk pre-selected within the user community to lead such testing.  This is often seen as a type of perk or acknowledgment of value for the personnel involved. It promotes the notion that they are part of the inner circle, that they’re relied upon more heavily than just any old end-user in the business.  Setting up this type of relationship in advance is incredibly useful for facilitating the change process.  Who knows better than the day-in and day-out user of the system what features must be rock-solid to keep things humming?


Middle management will often believe that they may substitute for actual end-users in such testing, but over the years I have come to believe this is not always true.  What the actual end-users do in practice many managers only know in theory. When it comes to assembling a testing approach it’s usually best to use someone who taps those keys in that way as their day job.


Luckily, the core functions the business performs are relatively static.  They don’t change from one incremental update to the next, so identifying how to test that they work is an upfront investment which can typically be leveraged many times before being revised.  Major application releases may change the specifics of these tests, but unless the business process is being reinvented, the outline of these tests will typically still be intact even from major release to major release.


Where does this leave us?


We’ve identified that we need at least three types of roles to participate in testing after a change has been made—to ensure things still work.  We have suggested that the people performing these tests could be part of a community responsible for such testing, and that there may be some social value in assembling that community.  What else needs to be done in advance?  Specific test cases should be identified and documented, and a Change Control Board should be established.


Successful organizations create and use a Change Control Board to review pending changes, ensuring that the changes are well thought out and well prepared-for.  Very large organizations often stand this group up outside both business and informatics organizations.  Sometimes it’s located within Audit or Compliance departments as it is related to illustrating compliance with NIST CM-03 and NIST SA-10. Because performing the assessment of upcoming changes often benefits from an awareness and familiarity with Informatics topics, the Change Control Board often resides within Informatics, though it may have members from the business, audit, and/or compliance organizations as well.


Regardless of where the group exists, the charter for these groups is straight forward.



Completeness and quality in step one speak directly to the detail around how the change will be effected, and by whom.  By explicitly documenting who will do what, how, and when, a change plan will both give clear insights into the change for the Change Control Board, and serve as a script for those executing the change later on.  “A user gets created” is not nearly as useful a process step as “Jim uses Okta to create test user __Paul __Nelis with email account __Paul__Nelis@healthco.com assigning the billing_manager and billing_user roles.”  A good Change Control Board will insist on instructions anyone could follow, if they had familiarity with the necessary systems.


Similarly, there should be an element of time-scale included in the proposed change. For every production system there will be a time window within which the change must occur.  If the steps of the change take longer than the ordinary down-time window for the application there may need to be some significant debate with the business around delaying the business start, or moving the end-of-day up prior to the change, in order to allow for enough time to implement the change.  One downstream benefits for being specific and accurate with the timeline is that many participants in the change may have only a small part to play.  Knowing when their input will be required can help economize their involvement. Why have Joan sitting around for five hours when she only has 20 minutes of work do to, halfway through?


Key to the value of a Change Control Board is the evaluation of the success criteria. This centers around identifying what will be tested and what outcomes are acceptable. This does not mean that everything must work.  There are situations in which an organization may determine that if core functions work, they can live with some things that don’t yet work.  To Fail Forward is to leave a change in place despite failed acceptance testing. This usually assumes that whatever’s left to fix can be fixed subsequently. This is a fine business decisions as long as it's agreed-upon in advance with the business, and well understood.


Who is running this circus?  Any given change event should have a change manager and a change champion.  The change manager is responsible for ensuring the steps proposed are followed as the change event occurs. The champion, though, is the ideally the person who wants this change to take place at all. In most cases this should be the business leader associated with the application undergoing the change, but it may be their delegate or the product manager associated with the application.  The Change Control Board should ensure these roles are identified and documented.  As sponsors of the change, both the change champion and the change manager should endorse the plan put before the Change Control Board.  Either or both should be welcome at any such meetings held on the topic.


Too many organizations skip or give insufficient attention to step four:  the rollback plan.  It is very very likely that the mechanics of the change will have implications for the rollback plan.  If, for instance, no one created a copy of the configuration file in place before the change occurs, it may be very difficult or impossible to roll back to the prior version of the application.  This is part of the function of the change control board–ensuring that there is a good plan to both identify success of, and protect against failure of, the change event.  The mechanics of implementing the change should include whatever provisions may be necessary to facilitate the rollback plan, should rollback be the required.


Note too that just as the post-change test cases should be outlined in advance, the post roll-back tests cases should be identified in advance.  If the system has to be put back the way it was before the change was attempted, what would we test to ensure it works?  What outcomes should there be for each test?


Testing can be tricky, and there are books written about preparing for and executing testing effectively.  For our purposes, here, I would like to suggest some things to consider.  Testing should be as specific as possible, written out like a recipe for a complicated side dish at holiday time.  The more details put into a recipe the easier it is to follow.  The same is true for a test case.


The act of testing a system modifies the system.  If the test calls for adding a new patient to the application, you’ve now got a new patient cluttering up the database.  Care should be taken to, where possible, reverse out any changes you make to the system for the purposes of testing the system.  In some cases, it may not be possible to remove all of the data input in the act of testing.  In such cases, some guidelines should be created about the test data entered, such that it can be ignored in subsequent reporting which may include the data.  


A great example lies in patient names.  Some organizations use whatever the tester thinks of in the moment, but this can lead to Mickey Mouse and Donald Duck in the patient records.  As awful as it may seem, there can be (there ARE) real world people with these names, so differentiating such test accounts from real patients becomes fraught or impossible.  If the application allows, consider test accounts with unusual characters, such as two underscores at the beginning, to make __Paul __Nelis.  These can be found and redacted from reports even if they cannot be eliminated from the patient database. Should that not be possible, consider scaler pseudo names such as Account Tester001, etc.


For systems which rely upon real data, systems which cannot be tested with fake data, consider a formal agreement with a small set of employees which allows their data to be used in the testing process.  This can help contain any potential fallout should testing not go as planned.


2. Execution


This is often a much smaller portion of the overall change control process, but it benefits from close coordination, especially when there are several different people involved in the change.  Some best practices around the execution stage include scheduling a video conference or audio conference line which participants may join when their portion of the change event comes into focus. Whoever may be managing that particular change event can use such a collaboration technology to ensure direct communications are timely and bi-directional in a way that email alone may not equal.  With video conferencing, the added benefit of screen sharing can greatly reduce confusion and misunderstandings.


Because the change plan approved by the Change Control Board was written with great detail, the steps involved in the change would have been estimated across a timeline.  Doing this will help set people’s expectations around when their part in the process will likely occur. 


As each stage of the change unfolds, the person responsible for managing that change should take notes which capture any hiccups or  especially useful insights revealed.  These could be helpful in the post-mortem process, once the change is complete.


3. Conclusion


It’s important to document the intent to create the change, but also the outcome of the change event.  In order to be compliant with NIST CM-03 and NIST SA-10 you must have, somewhere, a record that you approved the change in advance, and what came of the effort.  NIST and other such guidance frameworks don’t specify how you do such things, just that they be done. The implication, however, is that evidence that such things have occurred should be available on demand for audit cycles.  Whether it’s one long Google Doc containing all the changes contemplated in a year, or a document for each change, or an entry in the Incident Response system is unimportant as long as the planning is clearly illustrated, clearly approved, and the outcome recorded.


Savvy organizations take the slight bit of extra time to engage the participants in the change to learn what may have gone well or to learn where pitfalls may have been uncovered such that the successful approaches can be leveraged, and the pitfalls avoided in the future.