Model risk management for systems used in financial crimes and AML compliance

The governance of model risk management has become a fundamental part of an effective financial crimes program which like all other risks, requires due consideration of the materiality of the risk associated with each model.

Models are defined as “… a quantitative method, system, or approach that applies statistical, economic, financial, or mathematical theories, techniques, and assumptions to process input data into quantitative estimates…” (www.federalreserve.gov/supervisionreg/srletters/sr1107.htm). Commercially available transaction monitoring, fraud and watch list filtering systems utilized to ensure day-to-day compliance as well as similarly designed in house systems are some of the most common examples of “models.” In contrast “methodologies” or those functions, practices, procedures or activities which fail to qualify as “models” lie outside the scope of this article.

Model risk increases with a model’s inherent complexity (e.g., higher uncertainty about inputs and assumptions or greater complexity of quantitative methods) and usage (e.g., wider breadth of the model’s use or greater potential impact of errors caused by inaccurate model results).

Model risk can be mitigated through effective assignment of responsibilities, oversight, controls, documentation, and validation. These mitigation methods should always be applied in a manner commensurate with the underlying model risk.

Model risk is identified, managed, monitored and mitigated via model risk management frameworks that establish model governance structures to ensure appropriate oversight and controls for the following:
Model identification;

• Model development;
• Model usage and implementation;
• Model validation;
• Model monitoring, including annual review and change management;
• Model documentation; and
• Model approval.

As anti-money laundering (AML), sanctions, and fraud models become more complex, examining and testing model validation processes are becoming more central to supervisory examinations of banks’ AML practices. Erroneous or mis-specified models may lead to expensive look-backs or other regulatory fines, in addition to internal inefficiencies. It’s critical that banks evaluate their models on a regular basis to validate whether they are working as intended; confirm conceptual soundness of models; properly document models; and tune models as needed to deliver more meaningful alerts to Bank Secrecy Act (BSA) analysts.

Regulatory Guidance

Regulators expect compliance with the standards established in the model risk management guidance (MRMG) (Federal Reserve SR Letter 11-7, OCC Bulletin 2011-12; FDIC FIL 22-2017) as well as the April 9, 2021, Interagency Statement on Model Risk Management for Bank Systems Supporting Bank Secrecy Act/Anti-Money Laundering Compliance relating to systems or models used by banks to assist in complying with the requirements of Bank Secrecy Act laws and regulations.

In addition, state banking regulators, such as the State of New York’s Department of Financial Services (NYDFS), have established separate standards relative to model certification and validation (govt.westlaw.com/nycrr/Document/If190167b58ac11e6806bc9321b10fb4e?viewType=FullText&origi nationContext=documenttoc&transition-Type=CategoryPageItem&contextData=(sc.Default&bhcp=1).

Compliance Models

Banks use a broad array of automated technology solutions in support of their compliance efforts which offer transaction monitoring, fraud, and sanctions filtering which leverage developed models for each detection program. Systems with developed models may be purchased from third-party vendors or models can be developed in-house. The reliance of banks upon models to ensure appropriate detection scenarios is not without risk. Poorly informed, mis-implemented, improperly tuned, misaligned or misused models can have disastrous consequences to a financial institution to include an erosion of trust by regulators, damaged reputation and financial loss. Proper use of models by banks remains critical to their success to prevent the misuse of the financial system.

Setting Expectations

Regulators expect banks to develop sound risk-focused model risk management practices. First and foremost is the identification of “models” operating within any banking environment. Once identified, models are then rated relative to the impact from the potential adverse consequences based on their incorrect use. Models are typically provided ratings of risk (high, moderate, low) based on their impact of failure or the chance of unintended consequences resulting from model development, inputs or outputs. Models are to be independently validated and “effectively challenged”—on a periodic recurring basis—based on their perceived risk. Higher-risk models should be independently validated every 12 to18 months. Out-of-cycle model validations are to occur when there are significant changes to the bases for the model itself or the environment in which it was created. Lastly there is an expectation that models are “independently” validated by individuals other than the model developers or the model owners themselves. Validators are to possess sufficient technical expertise in which to perform their duties.

Model Validation

Models are distinguished by three distinct components Information input, processing and reporting. Simply said, input, throughput and output. Regulators expect any independent model validation to address five key areas:

1. Model Input: An analysis of the data being ingested by the model.
2. Model Processing: An analysis of how the data ingested by the model is being interpreted(processed) based on a set of pre-determined parameters, settings, parameters or assumptions.
3. Model Output: An analysis of how the model responds to the ingested data based on the pre- determined parameters, settings, parameters or assumptions.
4. Model Governance: A set of activities, policies and procedures which formalize model and model risk management activities for implementation.
5. Conceptual Soundness: An assessment of the quality of the model design and construction, as well as review of documentation and empirical evidence supporting the methods used and variables selected for the model.

(See www.federalreserve.gov/supervisionreg/srletters/sr1107.htm, www.occ.treas.gov/news- issuances/bulletins/2011/bulletin-2011-12a.pdf and www.fdic.gov/news/financial-institution- letters/2017/fil17022a.pdf.)

Regulator Focus

Regulators focus primarily in three areas when reviewing independent model validations: conceptual soundness, data integrity and accuracy, and governance.

When reviewing conceptual soundness regulators assess the quality of the model design based on a review of evidence supporting the methods used and variables selected for the model. In the instance of AML, sanctions, and fraud models, regulators look for evidence of model developers leveraging the institution’s specific fraud and AML risk assessments as well as comparisons against published red flag indicators to assess model effectiveness. Regulators look for algorithmic bias or misalignment with risk profiles, inadequate model documentation or explainability, and the unsubstantiated determination of alert parameters or detection scenarios. Documentation supporting a model’s “conceptual soundness” is oftentimes overlooked or undervalued by model owners and developers.

The second area regulators have emphasized is data. While few regulators are quantitative analysts, they recognize the quality of model outputs depends on the quality of input data and assumptions, and errors in inputs or incorrect assumptions will lead to inaccurate outputs. They seek evidence of an assessment of data quality and relevance, and appropriate documentation.

Evidence of an adequate model risk governance framework is the easiest to assess and is often incomprehensive based on the modeling environment. Regulators seek written Board approved policies addressing model oversight, model control practices, and model validation.

Model Tuning

Nearly every vendor supplied AML surveillance, sanctions and fraud monitoring systems involve the use of modeling. Models should be tuned for optimal performance. Individual rules or scenarios should be calibrated utilizing above and below-the-line threshold testing. AML, sanctions, and fraud models should be tuned to optimize the output results of the monitoring program focused on appropriate detection of suspicious activity. We frequently see financial crimes and AML monitoring systems with overall alert to SAR efficiencies of less than one percent. Suspicious activity monitoring scenarios, rules or agents which produce few or no SARs should be retired. The rationale behind any such retirement should be substantiated in writing by the model owner. When modified the parameters for any monitoring scenario, rule or agent should be established using appropriate periods of data—to account for seasonality—supported by above and below-the-line threshold testing.

Challenges When Validating Machine Learning Models

While machine learning (ML) is at the forefront of increasing efficiencies in AML and fraud models, regulators expect machine learning models to comply with the standards enumerated in the MRMG which guide traditional model development and validation. The validation of ML models poses their own unique set of challenges. As many ML models operate using proprietary algorithms they are oftentimes perceived as “black boxes” beset by complexity, a lack of transparency or “explainability”. The use of both structured and unstructured data by ML models creates their own set of challenges during data validation exercises. Bias in data – caused by flawed human decision making—can prove detrimental to the effectiveness of ML models.

Final Thoughts

Not all models pose the same risk. Model oversight, model control practices, and model validation remain the key components of any model risk management program. Models should be identified, rated for risk, monitored and governed accordingly due to their perceived materiality and criticality to an institution.