Balancing Function and Conservatism – Performance Monitoring on AML/BSA and Fraud Models

Introduction

Deploying software to monitor for potential money laundering or fraud is a simple way that a bank can improve its ability to identify and report illicit activity while at the same time reducing the impact on the business line and analyst resources. A bank can apply the same principles for ongoing monitoring and accuracy testing for money laundering and fraud models that are used for other models, but with the attention capturing false positives while at the same time ensuring that the model is not so conservative that it has an adverse impact on surrounding processes.

In this post, we will provide some guidelines for model monitoring for AML/BSA and fraud models and outline approaches for improving the efficiency of the process while not compromising the main goal of these models: Identifying illicit activity.

Success at identifying illicit activity

The most critical part of models concerned with a financial crime is confirming that they can identify transactions that are consistent with illegal activity. This involves ensuring cases that are flagged as suspicious receive a full review, and that the methodology being used to identify suspicious activity follows good business judgment. If the bank later determines that an investigation uncovered money laundering or fraud that bypassed the model, the bank should investigate the model’s performance to determine whether the model continues to perform effectively or if it needs to be retuned or replaced.

Are surrounding expectations being met?

These are expectations for the process that are expected in addition to the ability of the model to identify illicit activity. An example of this is the expectation of suspicious activity reports being reported within thirty days of their identification. As part of ongoing monitoring, the model owner can evaluate the bank’s ability to continue to meet this expectation. Additionally, the model owner can implement KPIs for the AML/BSA process to ensure that suspicious activity reporting occurs in a timely manner. For more information on creating an ongoing monitoring plan, you can find our earlier post on building a proactive model monitoring plan.  

Reviewing secondary results

While the primary expectations placed on the model by the bank should be prioritized, it’s also important to track secondary impacts of model performance. A common example of this is the effects of excessive conservatism. While this model may flag all or most illicit activity, it may also generate many false positives. These false positives can create other kinds of risk by straining internal processes (the ability of analysts to review all flagged transactions) or reducing the customer’s access to their accounts.

The steps for identifying the secondary impact of the model’s conservatism are as follows:

1. Identify what parameters or settings can be used to change the outcome of the model.
2. Gauge the ability of the model to perform the primary task. For instance, if the model is for fraud, is it able to completely identify all cases of fraud?
3. Identify the impact of model conservatism. For instance, is the main impact of model conservatism the number of analyst resources? Does it directly impact customers?
4. Use current information of secondary impacts to extrapolate how they would change if the model parameters were tuned. Use the results in step three to determine how changes to the model (such as thresholds) would impact the model results.

Consider the following example. A bank is using a model to capture fraud. The use of the model involves one parameter that can be changed: The threshold at which transactions are referred for analyst review. The threshold is bound between 0 and 100 where a high number generates less conservatism. The bank estimates that it has a maximum of 120 analyst work hours per week. These analyst work hours are used reviewing both true and false-positive results from the model. The bank knows that 50 true fraud results were identified out of nearly 10,000 transactions.

To assess the results, the bank varied the risk score threshold to find out the number of true and false positive fraud results that were generated. The number of analyst hours and true positive results is graphed for comparison below. The red horizontal line represents the 120-workhour cutoff described earlier. The left vertical line is the threshold which generates a total workload of 120 hours. The right vertical line is the point at which the model stops identifying true fraud cases.

We can see from the results above that it’s necessary that the risk score falls between 50.6 and 85.9. This gives the model owner a range where the analyst time and model performance can be balanced.

Adding this additional analysis can help the model owner ensure that a model meets its obligation to identify as many illicit transactions as possible without becoming unnecessarily conservative. It also aids in planning at the bank because it shows how the tuning of the model directly affects downstream processes. The impact on downstream processes can then be used as a specific measure to show when a model has become too conservative. For more information on testing a model’s results to determine how its inputs and settings can be used to better understand model behavior, see our earlier post on sensitivity analysis.

Conclusion

The first and most important aim of an AML/BSA or fraud process is to identify illicit activity for action by the financial institution. In the past, this has followed a rule-based process. The past decade has seen the implementation of more sophisticated methodologies that seek to maintain the ability to identify illicit transactions while also seeking to minimize the amount of manual review. To use these models effectively, the bank must use the parameters for tuning each model as part of a larger performance monitoring process where the impact on downstream processes is gauged against the model’s ability to identify illicit transactions. A well-regulated and regularly reviewed process can avoid unnecessary conservatism which harms banking operations while not aiding the performance of the model by the bank. In this post, we introduced some common guidelines for performing this testing as well as a simple numerical example showing how it could be done.

If you have an AML/BSA or fraud model and need to create a performance monitoring plan or assess one that already exists, or have the entire model framework validated, you can contact our team today or send us an e-mail at connect@mountainviewra.com to discuss your model’s performance, your business needs, and risk assessment.

Written by Peter Caya, CAMS