Go with the Flow – Process Architecture for Analytics and AI

The best decision-making models also improve the process

Edward H. Vandenberg

March 2023

Summary

1. The effort of an analytics/AI application development project should be to automate and improve a process. A valid model is a relatively ‘smallish’ part of this effort.
2. The measures of success of an analytics application (prediction, business rules, reporting, etc.) are process flow measures.
3. Match the model design to the process steps that will measurably improve the targeted workflows; if necessary, redesign the process to exploit the proposed model or agent.
4. Workflow analysis helps to uncover new opportunities and new data. Start with this analysis in areas of the business that need improvements.
5. Ensure that your key stakeholders and business SMEs are not simply spectators to the project.

The current situation

The quantitative challenge to predicting case-based outcomes with accuracy is what data science focuses on the most. Having used an adequate amount of data, rigorous data prep, and accepted model validation methods, a valid model with reasonable lift is considered a ‘win.’ And, if the project was set up correctly, the business benefits are obvious. Ready, set, go!

But that is not what happens in many situations.  Managers of the targeted business process conduct reviews, consider, ponder and delay starting the implementation (and everyone waits for IT). What once was excitement about improving operating results turns into some concerns. If end users have been brought into the process, they practice silence: not wanting to contradict the ‘smart people’ and the managers who sponsored the project. What’s going on? This can be a difficult gap to bridge for data scientists.  As time progresses, other projects start (and some crises ensue), and the implementation project stalls indefinitely. Ultimately, the project hits the valley of death. The effort is wasted, and opportunity is lost.

Statement of this paper

After 20 years of building and deploying predictive models into core business processes for insurance, the focus of data science on model accuracy is much overrated. And that the team should spend much more time on how the model will improve business process flows. This paper is an introduction to connecting analytics models and AI agents to process flow analysis to achieve better business outcomes.

Getting to the prediction

Of course, accurate predictions are important.  But case-based predictions are based on historical data, which is never exactly like future data. So, an accurate model with a low R-squared or good classification rate comes with a disclaimer.  If the past is very much like the future and the model is implemented strictly, there is a theoretical improvement in the target metric. In the case of insurance pricing, underwriting, and claims, that would ultimately lead to a net underwriting profit. That’s a good thing.

But there are so many real-world influences on the actual results for a line of business that an “accurate” prediction score at the case level is ‘lost in the wind’ and challenging to measure.  

The actual results and timing problems

Insurance product managers know that the results of model validation are theoretical. The predictions look great on paper but won’t necessarily result in a successful year for the book of business. And the measurement of the book of business results is much delayed. Business operating results are reported on an annual basis. A full twelve calendar months of the deployed model are needed to determine the impact of a deployed model. For a model to be ‘fully deployed,’ the team must account for IT integration and testing time, training, and potential regulatory restrictions.  Other things are changing during this time as well. Perhaps things change the foundation data for the model, such as pricing or underwriting practices.  Improvements will not be evenly distributed throughout the inventory because of the varying skill levels of users.  In short, the measurement of the benefits of the model becomes a mess.  A/B testing is a method commonly used in marketing to test the lift of a new offer on a random sample. However, this procedure is often impractical for service environments when treating some cases differently than others may lead to customer complaints and regulatory scrutiny.

The bottom line is that improving the target operating metrics is hypothetical and challenging to measure. Business units are not measured for the accuracy of an individual case (underwriting submission, claim, etc.), even if such a thing exists. In most lines of business, losses are expected, and some underwriting and claim processing misjudgments are unavoidable.

Meanwhile, other impacts of the deployed model are being measured. What is veryreal are the expenses to operate the target business process, primarily labor, and the efficiency and effectiveness of the end-to-end process. Process costs in services industries are the labor effort applied in the steps to handle various transactions. And labor costs are part of the standard reporting for all insurance companies to account for costs not directly related to premiums and claims. They get measured before any line of business results from the model can be detected. This is the ‘valley of death’ for an analytics application. Early in the life of a deployed model, and perhaps for 18 months, the costs of a disruptive process are measured, but financial gains from analytics may not be measurable.

Case-based prediction models are generally integrated into some steps in the workflow of a core process to improve human decisions. End users, such as underwriters, claims adjustors, etc., are presented with prediction scores and various business rules to interpret the score and recommend actions. But, if the implementation of the prediction model adds a single (or multiple) process step, this increases the time it takes to process the case and consumes more resources…. costs go up, and productivity goes down. Secondly, if there is any latency in creating or using the score caused by accessing a new application or retrieving data etc., this adds wait time, perhaps leading to idle resources or frictional costs of switching cases. Again, cost go up, and whether the prediction is better than the existing decision won’t change this.

Another critically important aspect of modeling integration is the people factor.  It is still mostly the situation that models support human judgment rather than replace it. Individual contributors are often measured and compensated based on how many cases they can process over a period of time. If their production decreases, part of their compensation may also decrease. They may look for workarounds to the prediction scores or ignore the model recommendations altogether, further limiting the impact of a model.  If actual users don’t materially benefit, this is a red flag for a successful application. 

Another people factor is the clash between analytics and personal experience, autonomy, and authority. The level of precision with a mathematical model is confounded by the blunt force of ‘rules-of-thumb’ that many users prefer when making decisions over which they have experience and autonomy.  Sometimes these rules are a simple reflection of market conditions. For example, the model says to take a 10% rate modification. The producing broker says the insured is expecting a 10% credit on renewal and has competing quotes. This situation almost always ends in favor of retaining the risk. The potential pain of an underwriting loss sometime in the future is hypothetical versus the real loss of premium (and very often underwriter compensation). This does not necessarily argue for taking away user autonomy but rather a level of humility for data scientists in the part that precision models play in the real world.

Aha!

The effort of an analytics application development project should be to automate and improve a process. A valid model is a relatively ‘smallish’ part of this effort. Accurate models alone are not necessarily going to improve business results.  Aligning analytics to process architecture also aligns the project with the process owners: unit managers and end users.  They are now more drivers rather than spectators, and real operational improvements are now more certain.

And about AI

This discussion applies equally to projects that implement artificial intelligence applications. Artificially intelligent or rational agents complete non-routine cognitive tasks that humans would otherwise do, presumably faster, more consistently, and with at least equal accuracy. Rational agents go well beyond a simplistic prediction model but often incorporate those predictions into the intelligent tasks they can do, such as interpreting photographs, understanding language, generating responses, etc. As a result, rational agents will have an even more significant impact on a process and hold the promise of entirely redesigning processes with new fundamental business, economic, and service models.

“As AI brings new capabilities to a business process, companies need to rethink what tasks are needed, in what frequency, and who does them. When AI is accompanied by partial automation, companies also need to decide what humans will do and what machines will do in their processes. Most AI applications to date seek to improve a given task. But this is missing the larger picture; smart companies are viewing the introduction of AI as the rationale for a new look at end-to-end processes.”

“To truly leverage the power of AI, process design and improvement activities should be incorporated within the AI initiative.”

How AI Is Helping Companies Redesign Processes, Harvard Business Review. March 2, 2023, Thomas Davenport, Matthias Holweg, Dean Jeavons, https://hbr.org/2023/03/how-ai-is-helping-companies-redesign-processes

The inherent automation of algorithms

Analytics is inherently an automation play, even when not explicitly linked to existing activities and processes.

If you break down the process of the deployed model, it is automating several complex and time-consuming steps that would otherwise require labor effort: skilled analysis, and time. 

1. Look up – Models look up reference data, meaning the important predictor variables.  In many cases, this is the same data that a human would look up but might need to search for with some lag time.  Note that models don’t look up data that is not useful, saving time and attention in an otherwise manual process.
2. Cleaning – algorithms are designed to clean data on the fly; this was handled during development time and cleaning raw transaction data part of the code and run-time process of the model.
3. Factoring – singular data is very often factored or combined with other data to create complex variables that are more predictive. On their best days, experienced workers do this naturally.
4. Analysis – the interpretation of case data against historical data, both empirically and statistically, is the heart of the algorithm, and this step is beyond the capabilities of humans, at least at the speed and consistency of an algorithm.
5. Interpreting – the output is often interpreted into human-readable rules that recommend or enforce actions. Again, skilled workers do this naturally. But not every worker is equally skilled. And skill levels impact resource costs.
6. Visualizing – visual graphics help humans interpret what is going on and the context of a case-based prediction in a corpus of cases.

This view may give the development team clues on improving workflow by documenting how the model completes these steps and locating the processes and the steps that can be replaced or augmented.  This will seem evident to data scientists. However, without careful inspection of a process and expert guidance, it is not always obvious where this activity occurs and who does the work. It may be that all of these steps are ‘net new’ simply because a new decision process is needed. (Note that it may be that the legacy business rules are doing some of this work as hard-coded, brittle, and clumsy automation. These rules will confound empirical models and should be replaced. Straight-through processing is an example of business rules on steroids designed more for reducing operating costs than for improving decision-making accuracy).

Process Flow Management

Modern process flow management started after the industrial revolution as artisans and craft guilds began to convert to factories.  Several innovations in technology and management led to the need to organize a shop floor…tools, machinery, material, and people. Today this management activity includes the supply chain logistics of the largest companies in the world. But the analysis is equally valid for service processes. A process is the transformation of inputs to outputs using resources within an information framework. It is simply how organizations work (and how some organizations excel). People, machines, systems, applications, etc., cannot do anything or make anything without following some process.  Rather than a physical organization of steps in a manufacturing environment, service processes involve a logical flow of information gathering, analysis, decisions, and the execution of logical actions. In a transaction environment, this is a flow of many cases. A basic understanding of process flow measurement and management principles will help extract the value of data science applied to work and decision-making.

Process Flow Infographic

A change in the paradigm

The measures of success of an analytics application (prediction, business rules, reporting, etc.) are process flow measures. Specifically: does the application improve critical process flow metrics by eliminating process steps, decreasing wait times, reducing resources, and reducing bottlenecks? If some or all of these metrics are improved, the higher-level target operating metrics are also improved, including net operating profit and customer satisfaction.

It is rational for analytics executives to consider ‘leading with workflow analysis’ to uncover processes that must be ‘fixed’ by analytics.  Workflow metrics and process analysis will show where bottlenecks are, indicating the potential need for a decision-support model. 

Process Flow Measurement

Managers often use financial and customer satisfaction measures as a proxy for process measures.  Unfortunately, these measures are backward-looking and lagging indicators that aggregate many processes and subprocesses.  Critically, these measures do not provide any control information to manage processes as they are being run.

Process flow measurements are leading indicators of financial performance.  They are internal measures that managers can control. They can be measured in real-time, and variations can be tracked statically with dynamic process control diagrams. They can tell managers how efficient and end-to-end flow is and the efficiency of the individual steps. Critically, this analysis is used to improve cycle times and discover and resolve process bottlenecks. In such situations, more resources, better skills, or a different configuration of process activities can be designed (Process Architecture).  Process flow analysis can also determine the capacity of a given process given a set of resources and process capacity. See Managing Business Process Flows, Third Edition, Prentice Hall, 2012.  

Note that these measures do not exist in some organizations and are not captured by existing reporting. It is possible to use system logs to generate process flow data. An area of study called Process Mining may be an overlooked but highly valuable part of workflow analysis (see Addendum).  

Process Flow Metrics

Anatomy of a process – what can be done with the right model(s) and AI agents

1. Reduce Manual handling and rework – models can help to ensure that nearly homogeneous cases are routed to the appropriate process and resources. A high level of variation causes cases to be looped inside, exit a process altogether, or be rerouted to manual handling, sometimes several times.
2. Increase resources – this is analytics’ and AI’s sweet spot.  In a service environment, typical activities include Information gathering, evaluation, analysis, and decision making may be entirely model-driven and automated.  A model is a ‘free’ resource. If it can duplicate process activities usually handled by people and speed up those activities, cycle time is improved, process capacity increases, backlogs are reduced, and the process is more resilient to the variability of arrival rates. (Note that managers often hesitate to let models make decisions without human discretion.  All models have an inherent error rate and built-in bias’. The right balance can ensure that human expertise and the model’s power are additive. But this involves training and transparency).
3. Eliminate a step – models can determine what data is important in a legacy process and, in some cases, eliminate a step completely.  Legacy processes are full of rules, lookups, and fact-checking that do not contribute significantly to a case outcome.  The model development process can test these rules as an input variable to determine if they can be removed.
4. Reduce variability – automating steps helps managers plan resources because an algorithm is more consistent than a human and always shows up for work.
5. Reduce Wait Time – processes involving complex decisions and information gathering inevitably involve waiting. This may result in idle resources but certainly meets an increase in cycle time that impacts the business objectives for the process. 

All about the bottlenecks (hint: don’t make them worse)

No process can run faster than its slowest step when that step is on the critical path. The ideal situation is when an analytics application can specifically target the slowest step, relieving capacity constraints causing bottlenecks.  This may be as simple as an automated data look-up or documentation process that would otherwise require manual steps.  In most cases, rules-based systems and APIs have removed some of these steps long ago. 

Ultimately the target process must achieve its business objectives and meet the customer’s expectations in a competitive environment at a sustainable cost. Thus accuracy, speed, and efficiency are all critical attributes of an analytics application as it affects the target business process in which it lives.

Ideally, but not always, prediction models can improve process flows that use human judgment or actions.  Using the same analysis can also improve legacy process flows already automated by business rules. Practitioners must learn where all the rules are executed, what data they ingest, and what downstream processes they feed. Humans have already been removed from these processes, but this may cause poor business results. Clumsy business rules may cause cases to be re-assignment or exit a process for manual handling, ultimately using more human resources.

Qualified Data Scientists can get the models right, but they may not be as knowledgeable at getting the target process right or may address it only casually. Most project teams include unit managers and end users.  But it is often the case that managers and individual contributors don’t know their process flow metrics with the level of detail to guide the model development team. Unless they apply process flow management, unit managers generally use the same metrics that finance uses.

Finally, the development team typically assumes that greater accuracy of a particular decision is an acceptable trade-off for increasing the processing time when an algorithm and data retrieval are built into process steps. The result is an application that increases cycle time, creates bottlenecks, and ultimately increases hard costs. These assumptions may not be shared by the business unit managers and users. 

This calls for a new workflow metric

None of this good work will speak for itself.  Analytic executives must design and evangelize a new workflow metric that isolates the improvements driven by the model, factoring the improvements made against baseline workflow metrics that can be monitored over time.  This will enable the application to ‘live’ with the workflow and be adjusted or rebuilt as business conditions change. More importantly, it will show how an analytics application materially contributes to process capacity and cost controls.

When everything is new

The case when there is no existing process, or the existing one must be so radically reimagined that there are no baseline process metrics to analyze.  An entirely new process is being constructed more and more, particularly with AI. Yet even in these circumstances, there is a unit of input and a unit of output with some resources consumed and process flow metrics to be considered. In addition, services for consumers are time-bound, and there are dependencies to be considered.  A new process flow architecture will address the critical metrics and create transparency for managers and regulators to inspect AI process flows. Design Thinking is a process to help address situations where a process needs to be created or re-imagined.

Designed in: Process Architecture and Analytics

These prediction model artifacts and the activities of AI rational agents become part of a process architecture. 

Process Architecture is simply the design of a network of resources and activities that perform a specified function. Deep knowledge of the target process flow must be designed into the analytics application from conception through implementation. This redesigns the target process architecture to improve cost, quality, capacity, and customer experience. All data science projects have this as their objective. But the rigor to achieve this objective is often missing, and sometimes so are the results.

Summary

In case-based transaction environments, analytics projects are typically initiated to improve the accuracy and speed of human decisions when a human is still required. When developing the application, data scientists focus on a model’s accuracy (validation results).  While important, accuracy is less tangible than the impact of the model on the workflow where the model will be implemented.  In the real world, accuracy is difficult to measure because there are too many other influences on the outcome of an individual case. However, the labor costs of using the model are real and measurable.  If an application increases resource costs…adds process steps, or adds to case cycle time, it is likely to be resisted and eventually abandoned by the managers of that business process. Therefore, analytics practitioners must aim specifically to improve (or at least not harm) process flow metrics. This means understanding the process flow architecture and the baseline metrics of the target process and designing/testing the application against those metrics. The measures of success of an analytics application (prediction, business rules, reporting, etc.) are process flow measures.  An analysis of process flow measures may point to new analytics projects or to expanding the scope of an application. Either way, incorporating workflow analysis in model development and deployment projects will enable more applications to cross the valley of death.  This outcome feeds a virtuous cycle in all organizations.

Recommendations

1. Include workflow analysis in your project activities and bring on the right team members
2. Time-box model development to a ‘best-effort’ result and shift more project time to workflow analysis and related development activities
3. Measure and document the critical workflow metrics for the as-is processes you are targeting; engage with the business unit manager to understand the meaning and importance of these measures
4. Find any bottlenecks in the process
5. Match the model design to the process steps that will measurably improve the targeted workflows; if necessary, redesign the process to exploit the proposed model or agent
6. Gain agreement with the business process owner on the proposed improvements
7. Monitor workflow measures along with the model results to ensure expected gains

Addendum

Chat GPT weighs in

Question: How can service organizations use workflow measurements?

Service organizations can use workflow measurements in several ways to improve their operations and better serve their customers. Here are a few examples:

Identify bottlenecks: By measuring the time taken at each step of a workflow, service organizations can identify bottlenecks in their processes. This can help them to optimize their workflows and eliminate delays, which can ultimately result in faster and more efficient service delivery.

Improve quality: By measuring the number of errors or rework required at each step of a workflow, service organizations can identify areas where improvements are needed to reduce errors and improve the quality of their service delivery.

Monitor progress: Workflow measurements can help service organizations to monitor progress towards specific goals and targets. By regularly measuring key performance indicators (KPIs), such as response times or customer satisfaction scores, organizations can ensure that they are meeting their targets and make adjustments as necessary.

Identify training needs: Workflow measurements can also help service organizations to identify areas where staff may need additional training or support. For example, if measurements show that a high number of errors occur at a specific step in a workflow, it may indicate that staff requires additional training in that area.

Benchmark performance: By measuring their own performance against industry benchmarks or competitors, service organizations can identify areas where they need to improve to remain competitive. This can help to drive continuous improvement and innovation in service delivery.

Overall, workflow measurements are a valuable tool for service organizations to identify areas for improvement, optimize their processes, and enhance the quality of their service delivery.

My Feedback to CGP:

Consumer satisfaction scores are not workflow metrics.

Process Mining

The goal of process mining is to use event data to extract process information, including automatically discovering process models by observing events recorded by applications and databases. Process data exists in system logs, a nearly “pure” form of data automatically generated by the application, not a user. 

“The idea of process mining is to discover, monitor and improve real processes (i.e., not assumed processes by extracting knowledge from event logs readily available in today’s systems.”

Process Mining, Discovery, Conformance and Enhancement of Business Processes. VilM.P> van der Aalst. Springer, 2011.