FSRA - AI-Powered Risk Assessment For Financial Services Regulation in Canada

[#1]

Name: The FSRA team

Country: Canada

Organization: FSRA

The Financial Services Regulatory Authority of Ontario (FSRA) is an independent regulatory agency created to improve consumer and pension plan beneficiary protections in Ontario. The agency is flexible, self-funded and designed to respond rapidly to an evolving commercial and consumer environment.

In this capacity, FSRA:

• Promote high standards of business conduct
• Foster a sustainable, competitive financial services sector
• Respond to market changes quickly
• Promote administration of insurance and pension plans
• Encourage innovation

Awards Categories:

• Best Acceleration Use Case
• Best Data Democratization Program
• Best Approach for Building Trust in AI

Business Challenge:

The Financial Services Regulatory Authority of Ontario (FSRA) had an excellent opportunity to achieve three goals concerning AI and data science:

1. Democratize AI across the enterprise.
2. Implement FSRA's newly established AI Governance Framework.
3. Productionize FSRA's first AI-powered business application.

Achieving these three goals meant overcoming several obstacles.

Firstly, FSRA lacked a data science platform that could bring together professionals from the business, IT, data science and other areas of the organization. Secondly, FSRA lacked the data science governing mechanisms that could operationalize FSRA's newly established AI Governance policies and guidelines concerning AI and ML within FSRA. Thirdly, FSRA lacked the tools to seamlessly tie together disparate data sources and apply advanced AI/ML techniques to analyze these data sources. 

FSRA also needed to build its first AI-powered business application to address several tactical challenges. First and foremost was the automation of internet-based background checks, which were manually conducted for thousands of applications and has become increasingly untenable due to the volume ramp-ups. The existing process was unsustainable in the long run, requiring automation, scalability, and the synthesis of background checks based on specific business criteria. There were numerous disparate and legacy databases, each with different schemas, formatting, and unstructured data inputs.

This diversity made it difficult to integrate and analyze the data efficiently and effectively. The business had to manually search each database separately and then synthesize the search results, leading to a significant loss of productivity.

Moreover, unstructured sources of data, often embedded in comment fields stored inside the databases, presented a known but virtually inaccessible source of valuable information. This limited the business's ability to access and utilize crucial data effectively. The absence of a single view encompassing all the risk signals from external and internal data sources was a considerable obstacle. The business dealt with hundreds of risk signals, yet there was no comprehensive overview of the overall risk posed by an applicant. As a result, the ability to assess the full complement of risks accurately and make informed decisions was severely hampered.

In summary, FSRA aimed to democratize AI, implement an AI Governance Framework, and develop an AI-powered business application but faced obstacles in terms of lacking a data science platform, governing mechanisms, and tools to integrate and analyze disparate data sources, as well as the need to automate internet-based background checks to improve productivity and access crucial data effectively.

Business Solution:

The decision to choose Dataiku was driven by the solution's comprehensive capabilities, flexibility, and user-friendly interface, which made it an ideal solution for addressing the complex requirements of the project. The journey to success involved several key steps.

First, FSRA conducted a thorough assessment of the business challenges faced by the business. The team analyzed the existing manual processes, identified pain points, and determined the scope of automation required. This initial analysis guided the subsequent solution design. FSRA employed various techniques and technologies within the Dataiku platform to address the challenges. The team leveraged pre-trained AI models such as Spacy for NER and Word2Vec for Embeddings to develop custom search capabilities for various watchlists and checks. The team also developed custom keyword dictionaries to train the models. By training the AI models to filter search results based on specific risk criteria, the team automated the process and significantly improved scalability.

Dataiku's plugins played a crucial role in the "Document Analysis" capability. These plugins enabled users to leverage Large Language Models (LLMs) easily to address natural language understanding (NLU) problems that were otherwise unattainable using traditional AI models. This capability empowered FSRA to make informed decisions based on natural language analysis.

Furthermore, Dataiku's native support of Python and R enabled developers to rapidly write code, leverage third-party APIs and build custom recipes and functions to perform numerous tasks that would otherwise have entailed much heavier development and testing. Thanks to the visual interface of Dataiku, the development team was able to build, test and audit models much faster than normal.

To consolidate the risk signals from various sources, FSRA leveraged Dataiku's capabilities for data integration, cleansing, and transformation. The team developed a unified view of risk signals derived from structured and unstructured data, including data from the internet, internal databases, and applicant-submitted media. By consolidating and synthesizing these signals, the "Decision Support" capability provided a clear and prominent risk indicator to support decision-making.

The collaborative nature of the Dataiku platform allowed teams to seamlessly integrate these capabilities, ensuring a cohesive and comprehensive solution. Throughout the project, Dataiku's user-friendly interface facilitated collaboration, enabling users across different roles to interact with and contribute to the development process.

Finally Dataiku's emphasis on AI/ML explain-ability, ease of oversight, and a stand-alone AI Governance offering suited FSRA's needs to operationalize its newly established AI Governance Framework, which describes FSRA's policy and guidelines concerning AI/ML and data science.

Day-to-day Change:

The implementation of the Dataiku solution has significantly transformed our day-to-day operations at FSRA. The solution has had a profound impact on our business processes, leading to increased efficiency, enhanced decision-making, and improved regulatory oversight. Key tactical achievements:

• Reduced the time to manually review applications by 80%
• Automated searches for over 150 risk signals that previously required manual intervention
• Deployed the solution from pilot-to-production in only 12 weeks.

FSRA's first AI-powered application (known as the Decision Support Portal or 'DSP'), made possible by Dataiku, has led to significant improvements in background checks, data integration, and document analysis. By automating internet-based background checks, manual processes have been streamlined, allowing for faster and more comprehensive background checks. Integrating diverse data sources has optimized operations by providing a centralized view, improving data accessibility, and increasing productivity. Dataiku's document analysis capability has revolutionized the analysis of unstructured natural language, allowing for efficient extraction of key regulatory implications from comment fields. Overall, Dataiku has greatly enhanced efficiency and compliance at FSRAO.

Perhaps the more strategic impact is the cultural change brought about by the democratization of data science at FSRA. With the introduction of Dataiku, the tools and resources for AI and data science have become more widely accessible to a larger number of individuals within our organization. Now, our business analysts can directly engage in data science activities, leading to improved efficiency and more effective collaboration with business and IT professionals. This new level of involvement has allowed us to explore the possibilities of AI and data science and has demonstrated to the business the potential impact and benefits of these technologies.

Furthermore, the Data Science Lab at FSRA has greatly benefited from the incorporation of enterprise machine learning (ML) operations capabilities. The ability to operationalize ML models and processes within Dataiku has allowed the Lab to better manage various aspects of our data science workflows. This has not only enhanced the productivity of our lab, but also facilitated the deployment and management of AI systems across the organization. With these capabilities, we can more easily implement and monitor the AI Governance Framework developed for FSRA, ensuring that the application of AI technologies aligns with the organization's goals and regulations.

Business Area Enhanced: Financial Services Specific

Use Case Stage: In Production

Value Generated:

Dataiku brought value to three areas at FSRA:

1. Improved market conduct oversight through our first AI-powered regulatory tool, known as the Decision Support Portal (DSP).
2. AI/ML governance capabilities.
3. Cultural transformation with respect to the use of AI (including operationalization of our new AI Governance Framework policy).

In terms of ROI - accounting for the licensing and implementation costs - the first year ROI is estimated at 35-50%.

Firstly, our first AI-powered application (the DSP) contributed to increased team efficiency, enhanced tech stack efficiency, improved risk management and governance through transparency and explain-ability, as well as opportunities for upskilling and networking. By automating manual processes and offering advanced AI functionalities, Dataiku significantly reduced the time and effort required for complex regulatory tasks, resulting in increased speed and agility. This allowed FSRAO to respond promptly to regulatory changes and industry dynamics, maximizing operational efficiency. Dataiku reduced manual review time by 80% and automated the ingestion and analysis of over 150 operational risk signals.

Secondly, Dataiku delivered value by boosting team efficiency for our Data Science Lab. The platform's user-friendly interface and collaborative features streamlined workflows, enabling teams at FSRAO to work more effectively and make agile decisions. Dataiku's transparency and explain-ability features addressed concerns related to AI models being perceived as black boxes. By providing visibility into the inner workings of models and algorithms, Dataiku ensured transparent deployment. This transparency instilled confidence and trust amongst stakeholders at FSRAO, as they could understand and interpret model results. Consequently, risk management and governance were improved, as the decision-making process became more transparent and explainable.

Finally, Dataiku's value was reflected in its cost-effectiveness. With a reasonable implementation cost, the ROI was exceptionally high. The platform empowered FSRA to achieve greater efficiencies, streamline operations, and improve decision-making without incurring prohibitive financial investments. This cost-efficiency enhanced the overall value proposition of Dataiku for FSRA, enabling the organization to achieve its goals while optimizing resources effectively.

Value Brought by Dataiku:

Through Dataiku, FSRA demonstrated leadership in the use of risk-based AI and data science for the broader financial services regulatory industry in Canada. Our AI and ML Governance Policy, the Data Science Lab, and the AI Decision Support Portal (DSP) are all examples of FSRA's leadership in applying AI and ML to the financial services regulatory domain.

Firstly, Dataiku has brought AI and data science to the forefront of FSRA's executive agenda. Thanks in part to the focus on AI achieved through Dataiku, FSRA devised its first concrete policy guideline for Artificial Intelligence & Machine Learning (AI/ML) Governance.  This policy guideline provides a fundamental underpinning of good management and decision-making at all levels of the organisation regarding the thoughtful and careful use of AI/ML. All AI/ML products and development at FSRA will be required to operate under this governance framework regardless of the source or historic development. This governance document is not only a policy, but it also provides specialized guidance on AI/ML to be employed in conjunction with the FSRA Enterprise Risk Management Policy.

Secondly, Dataiku provided our newly established Data Science Lab with the comprehensive platform that the Lab needs to foster and promote data science and AI throughout the organization. With Dataiku, the Data Science Lab can more effectively collaborate with business analysts using visual and intuitive recipes. Dataiku also provides the operational governance platform required to monitor and maintain complex ML models. In short, Dataiku enabled the Data Science Lab to operate with the right tools and platform.

Finally, our AI Decision Support Portal (FSRA's first AI risk-management application and likely one of the first of its kind in the world amongst regulators) has opened the eyes of our many stakeholders - we now collectively have a much better understanding of what AI can do, its benefit to our stakeholders, and the effort/costs required to leverage AI and data science.

In short, FSRA demonstrated leadership in the use of risk-based AI decision support applications through Dataiku, which brought AI and data science to the forefront of executive decision making, resulted in the development of a policy guideline for AI/ML governance, provided the Data Science Lab with the necessary tools and platform, and increased stakeholders' understanding of the benefits and costs of leveraging AI and data science.

Value Type:

• Improve customer/employee satisfaction
• Reduce cost
• Reduce risk
• Save time
• Increase trust

Value Range: Millions of $