Enhancing decision making for system changes with generative AI

This is a guest post from Booz Allen, an AWS Partner.

It is crucial to manage IT system changes effectively in order to maintain system stability, prevent outages, and mitigate security risks within any organization. Doing so effectively, however, is especially challenging, as organizations must consistently assess change and mitigate risks, particularly in dynamic, fast-paced environments where rapid changes are often necessary. In addressing this challenge, Booz Allen, an HAQM Web Services (AWS) Premier Services Partner, developed an innovative solution using HAQM Bedrock-powered generative artificial intelligence (AI) to enhance the decision-making process for system changes.

The challenge of change management

Change initiators are responsible for submitting clear and well-documented change requests (CRs), but mistakes and oversights can still occur. Issues like inadequate documentation, conflicting changes, and insufficient analysis are common during the planning or submission process. Change authorities and approvers follow a standardized evaluation process to thoroughly assess each request using consistent criteria:

• The reason for the change
• Potential impacts to other systems
• Implementation and recovery plans
• Stakeholders involved or affected
• Compliance with organizational policies or government regulations

As the number of CRs increases into the hundreds or thousands, consistently assessing these factors across numerous change requests is time-consuming, mentally taxing, and prone to human error.

A generative AI solution for change management

To help address these challenges, Booz Allen created a change management AI assistant. This solution leverages HAQM Bedrock to assess each submitted CR ticket against a standardized set of criteria defined in an organization’s system change management policy. The assistant provides a recommendation as to whether a change request should be accepted or rejected, along with detailed reasoning to support its decision. This capability reduces the cognitive load on human reviewers, enabling them to quickly identify areas of concern and accelerate decision making. At the same time, CR submitters receive more rapid and actionable feedback on their change request, allowing them to make the edits necessary to pass the evaluation process.

Key features of the change management assistant

The assistant offers several benefits:

• Comprehensive analysis: The assistant uses the organization’s system change management policy to ask key questions about each ticket, evaluating multiple aspects of the proposed change.
• In-depth compliance checks: Beyond simple heuristics, the assistant verifies if essential elements like rollback and communication plans are included and make sense within the context of the change.
• Strategic alignment: The assistant ensures that the change request aligns with established standards, best practices, and the strategic goals of the organization.
• Easy-to-digest format: Results are presented to operators in a clear, concise manner that contextualizes the information for quick decision making.

Streamlining the review process

One of the primary benefits of this solution is its ability to streamline the review process. For instance, a team with an extensive checklist for reviewing changes may struggle to remember every item as the number of tickets grows. This increasing volume makes it harder to consistently apply the checklist, creating cognitive overload that leads to decisions driven by intuition rather than objective analysis.

Reducing cognitive load through the change management assistant

The Booz Allen change management assistant reduces cognitive load and improves decision-making capabilities. It helps operations teams manage increasing workloads efficiently by automating the initial change review process, allowing team members to focus on broader strategic tasks. In addition, the assistant is effective at highlighting critical areas of change that require closer examination, creating a more focused and productive workflow for both CR ticket submitters and reviewers.

Most importantly, the Booz Allen assistant supplements human expertise with data-driven and evidence-based recommendations, shifting what was once a process relying on subjective interpretations and intuition to one grounded in documented policy and procedures. This shift allows for more consistent outcomes and empowers team members to handle increased workflow volume with greater confidence and objectivity, accelerating an organization’s delivery of outcomes.

How it works

When a user has prepared their CR ticket and is ready to submit for approval, they can initiate a change submission event. This event kicks off the change management process, delivering ticket and implementation plan details to the generative AI assistant. These details are extracted and combined with the organizational change policies to formulate an evaluation prompt sent to the foundation model (FM).

The model uses this prompt to analyze the change request, analyzing key details in the ticket and implementation plan against those policies. When the model has finished analyzing the data, it recommends whether the CR in question should proceed as is, or if more details are needed for the change to be approved. Along with the recommendation, the model provides reasoning as to why the recommendation was made.

The recommendation and reasoning provide change reviewers a starting point when conducting their reviews. They can now focus on the tickets that require their attention while knowing that the approved tickets have been fully evaluated against existing policies.

At the end of the process, the model’s recommendation and rationale are commented back on the CR. This centralizes all change information in a single location and allows for change owners to receive rapid feedback on their submitted change. If a ticket is not recommended to proceed, the change owner can address the feedback provided in the comment and resubmit an updated change for review.

The following figure provides an architectural overview of the change management assistant.

Figure 1. Architectural overview of the change management assistant.

Technical explanation

Change owners provide two essential pieces of information per CR:

1. The CR ticket: Captures high level details of the proposed change, including:

• Type of change
• Change description
• Scheduled time of change
• Other relevant data

2. The implementation plan: Typically provided as an attachment, the implementation plan offers a comprehensive description of the change, including:

• Detailed implementation stages
• Step-by-step procedures
• Points of contact

The ticket evaluation process is a linear one that needs to scale in conjunction with the number of incoming change request tickets. This makes it a great fit for AWS Step Functions and AWS Lambda.

When the change event is begins, the AWS Step Function is initiated, and a Lambda function responsible for the extraction and transformation of the data begins. This occurs in two phases.

First, the most important fields, those that must be evaluated against existing policy, are pulled from the ticket. In selecting only the fields that matter, we are able to minimize the number of input tokens going to the model. Second, we parse the attached implementation plan (i.e., an Excel file) and transform the data to JSON, combining the most important information provided with the ticket details to create a new JSON object.

This object is saved in HAQM DynamoDB for auditing purposes and future analysis, then sent to the next step in the Step Function, which is the judging phase.

During the judging phase, the new JSON object is injected into a pre-defined prompt template based on the FM used. This template combines the extracted CR details and the organizational polices defined in a policy runbook. By maintaining a library of prompt templates, the assistant maintains flexibility in which model is invoked, as different prompt formats work better for the different model families offered by Bedrock. Once the template is constructed, it is sent to the model for analysis.

Once the model responds with its recommendation, the CR record in DynamoDB is updated with the resulting decision, and the output is processed for user readability. The formatted response is then added back to the CR as a comment on the ticket. The entire process typically completes less than five minutes after the implementation plan has been attached to the ticket.

This rapid feedback is beneficial not only for change reviewers but also for change initiators. In receiving feedback early, initiators are alerted to any overlooked aspect of their change and can reassess and make the edits necessary to resubmit for a higher likelihood of approval, without having to wait for the next change review session or change window. Additionally, as initiators become more accustomed to submitting proposed changes and receiving feedback, they gain insight into what constitutes a “good” change request more likely to be recommended for approval.

Best practices and tips for success

At Booz Allen, we understand that a true best practice is something your team can maintain, repeat, and hold themselves accountable to. Each solution should be designed to meet the requirements of your environment and culture. This solution’s broad applicability makes it a great starting point for those beginning their journey with AI and automation.

To date, we have used Bedrock FMs in their original form without any fine-tuning for this specific purpose, as their performance has met our requirements. However, no model is flawless, and we have resolved some unique challenges associated with large language models (LLMs).

One challenge has been the non-deterministic behavior of LLMs. A specific combination of tickets and implementation plan data is sometimes recommended for approval, while other times the same combination is recommended for rejection. To resolve this, each ticket is evaluated under the same prompt some n number of times, where n is an odd number. After the ticket has been evaluated, the prevailing judgement is the final recommendation.

For example, if we evaluate a ticket three times, and the model recommends the ticket be approved two out of the three times, the final recommendation is that the ticket is approved. The change management solution is built to be an assistive tool, so any risk in providing a recommendation is mitigated by the reviewer validating the model’s recommendation

We also focused on improving the prompt itself. Our initial iteration of the prompt proved to be overly rigid in its evaluation. After several iterations, we settled on a balanced approach to enforcing the organization’s best practice.

Real-world impact

Booz Allen is already providing their assistant to government clients. By implementing this generative AI solution, organizations can:

1. Improve consistency in change request evaluations
2. Reduce the risk of overlooking critical factors
3. Accelerate the review process without sacrificing thoroughness
4. Promote strategic alignment to organizational shifts in execution
5. Enhance overall system stability and security

Conclusion

Booz Allen’s innovative use of Bedrock demonstrates the power of generative AI in enhancing decision-making processes for public sector organizations. As the No. 1 provider of AI services to the federal government, Booz Allen is dedicated to empowering the people behind IT operations processes and thereby accelerating outcome delivery. By leveraging this technology to assist in system change management, agencies can make more informed decisions, reduce risks, and ultimately deliver more stable and secure services to the public.

To learn more about how Bedrock and generative AI can transform your organization’s decision-making processes, contact your AWS account team or visit the Bedrock product page.

To learn more about Booz Allen AI, visit this webpage.