The Great Recession spotlighted the impact of financial institutions’ heavier reliance on models to make key strategic decisions, as many models produced incorrect results or failed to perform as intended. Supervisory guidance of 2011 on model risk management’s goal was to direct resources to ensuring active management of model risk. A critical piece of this regulatory mandate was for models to be independently validated to confirm models are performing as expected. One common misunderstanding in this field is if a vendor model validation is sufficient to satisfy regulatory guidance.
Q: If my model vendor has a model validation, why would my institution need a validation?
A: First of all, a vendor “validation” is typically called a model “certification.” The model certification is an independent review and certification that all of the mathematical or statistical calculations in the model are calculated as documented by the model vendor. It is a replication of all calculations confirming accuracy and that calculations are as intended. Typically, the model vendor supplies their clients with a statement of this certification.
What a certification does not do is validate your institution’s use of the model.
Q: So, what does the validation do?
A: In a vendor model validation, we are not replicating the calculations, we know those to be independently certified, we are validating the Institution’s implementation and use of the model. A validation aligns with SR-11-7 Guidance on Model Risk Management and follows the conceptual framework, inputs (data and assumptions), reporting, outcomes testing and the governance (policy, documentation, and controls).
It also establishes any limitations with the model that would need to be understood by the institution. For example, if a model doesn’t have the capability of applying prepayments based on more than one factor, this is a model limitation and disables the institution from forecasting loan prepayments at a more granular level (i.e., based on age and financial incentive).
Q: At my institution, we are outsourcing our modeling, do we still need a validation?
A: Yes, even if your institution outsources, a validation is still required to confirm the model is correctly implemented and utilized for your specific institution. Data quality and assumption inputs, that the framework aligns with your institution, etc. In some ways, it is even more important to have a model validation completed for an outsourced model since you do not run the model yourself. The Institution is relying on a vendor to run the model; however, the Institution is still responsible for the assumptions, data, and inputs in the software as well as the results that are produced.
Q: How often should a validation be completed?
A: This depends on several factors. The first factor is what does your Policy mandate? Some Policies will require a validation every year. The second factor is what is the expectation from your examiners? Also, how complex is your balance sheet? And what is your total assets size? A large Institution with a more complex balance sheet is more likely to have a validation completed every year. Another factor to consider is how “clean” was your last validation? If you had a number of findings and recommendations, you would want to consider having another validation done the following year. Also, have any events occurred like a change in the person responsible for the model (i.e., CFO), data processing changes, or mergers? Changes such as these can trigger the need for a validation.
As you can see, it’s not uncommon to have confusion around some of the more complex details of model risk management. The MVRA Team is here to support your organization and ensure that your models are accurate allowing you to make informed and confident decisions. Visit our solutions page to learn more about how we can help with your model risk management projects today!
Written by Chris Mills, Senior Director
About the Author
Chris has over 25 years experience in financial institution modeling and has been leading MVRA’s model validation services and core deposit / loan analyses teams supporting strategic balance sheet and risk management for over 8 years. She brings a wide range of expertise across treasury, asset/liability management and model risk assessment processes. Experienced with multiple ALM models, she also is skilled in capital modeling, capital markets, liquidity and contingency funding planning, funds transfer pricing, model risk governance practices, and investment banking.
