Team members:
- Geetha Ramachandran, Senior Director, Technology
- Paul Schiavone, Senior Director, Market Regulation
- Alexey Egorov, Senior Data scientist, Market Regulation
- Wangfa Zhao, Senior Data scientist, Market Regulation
- Tate Welty, Data Scientist, Market Regulation
- David Ptashny, Director, Member Supervision
- Julian Asano, Senior Principal Specialist, Member Supervision
- Alejandro Colocho, Developer, Member Supervision
- Zihan Shao, Developer, Member Supervision
- Otto Scheel, Lead Developer, Technology
- Huriel Hernandez, Senior Developer, Technology
- Maksim Abadjev, Staff Developer, technology
- Graham Adachi-Kriege, Developer, Technology
- Peter Hopper, Application Engineer, Technology
- Robin Ma, Data Engineer, Technology
- Fumin Yang, Director, Product Management, Technology
- Helen Benton, Product Manager, Technology
Country: United States
Organization: FINRA
FINRA is a not-for-profit organization dedicated to investor protection and market integrity. It regulates one critical part of the securities industry — brokerage firms doing business with the public in the United States. FINRA, overseen by the SEC, writes rules, examines for and enforces compliance with FINRA rules and federal securities laws, registers broker-dealer personnel and offers them education and training, and informs the investing public. In addition, FINRA provides surveillance and other regulatory services for equities and options markets, as well as trade reporting and other industry utilities. FINRA also administers a dispute resolution forum for investors and brokerage firms and their registered employees.
We use innovative AI and machine learning technologies to keep a close eye on the market and provide essential support to investors, regulators, policymakers and other stakeholders.
Awards Categories:
- Best Acceleration Use Case
- Best Positive Impact Use Case
- Best Data Democratization Program
- Best ROI Story
Business Challenge:
The Financial Industry Regulatory Authority (FINRA) is a not-for-profit organization authorized by the U.S. Congress to protect investors and ensure market integrity through effective and efficient regulation of broker-dealers. It writes and enforces rules governing the activities of more than 3,400 broker-dealers representing more than 630,000 brokers, examines firms for compliance, fosters market transparency, and educates investors.
Every day, FINRA oversees up to 600 billion market events, including equities, options and fixed income products in the United States. Petabytes of historical market data needs to be analyzed to uncover insider trading and other strategies used to gain an unfair advantage. This constitutes a significant volume of data that requires considerable computing power for effective analysis.
FINRA’s mission is to protect investors and promote market integrity. To achieve this, it is important to quickly respond to market events, especially in today’s highly volatile market.
FINRA needs to look at the data to answer questions like: "Are there market events/participants that are threatening market integrity?" or "Are there regulatory responses required for these events to fulfill our mission?", and more.
This means efficiently analyzing hundreds of billions of market event data across hundreds of sources. This efficiency is critical when we need to react within days or even hours.
We therefore need to leverage interactive data analytical solutions to respond to critical and potentially harmful market events. Due to highly complex nature of our work, we require close collaboration across examination, surveillance, enforcement, and technology functions.
Our goal is to have an integrated platform to data analytics that avoids duplication, translates data into real results and supports coordinated contributions from all the various groups engaged in this activity.
Business Solution:
We chose Dataiku as the Data Analytics/Data science platform for our users. The aim for this project is to enable and empower our users with platforms and solutions to derive better insights from large volume of data so that they can identify and stop bad actors faster.
Dataiku was leveraged by diverse user personas as listed below and they were all able to come together and collaborate in a single and shared platform. It enabled real-time interactive insights generation across the organization.
- Excel users
- SQL users
- Data scientists
- Data engineers
- Business users and executives
Within a single platform we can analyze data sets of various sizes pulled from multiple different data sources. Between Dataiku’s architecture and Spark’s scalability, we essentially have no data size limits.
Dataiku has enabled a new type of data analysis paradigm, the End User Computing applications (EUCA). The EUCA is essentially an ability for knowledge workers without technical skills to access and analyze data regardless of its size (petabytes!) and the complexity of the analysis (Spark transformations, stats, ML, whatever).
FINRA can build the EUCAs using the following three Dataiku capabilities. First, the no-code environment and visual recipes for defining the transformations. Any non-tech person who mastered Excel can use Dataiku’s visual recipes and start getting insights from the data very quickly. Secondly, Dataiku provides non-coders and Data Scientists with the capability to develop web-based applications using just a few mouse clicks or a short Python program. What could take months to develop can now be done in days. Third, Dataiku’s flexible WebApp features have enabled FINRA to stand up custom Dash-based apps that combine large datasets of pre-processed market data with user-generated inputs to assess risks at member firms. The flexibility of Dash-based WebApps have allowed small teams of data scientists to partner with business SMEs to produce prototype tools in a fraction of the time it would take to develop in traditional development platforms.
We have found that the EUCAs are the fastest and most scalable way to perform ad hoc analyses. Moreover, it is perfect for trying new ideas and innovations.
Day-to-day Change:
Dataiku has not only made data more accessible but also empowered business analysts, non-coders to harness the power of large datasets that were previously out of our reach. Analysts are now autonomous in analyzing data for their own needs, which greatly improves our time-to-insight and enables us to react faster as an organization.
Business Area Enhanced: Analytics
Use Case Stage: In Production
Value Generated:
Leveraging Dataiku for self-service analytics had had a broad impact:
- Faster time-to-market for data solutions: Our analysts are now able to quickly prototype and test out algorithms and logics based on the data. This feature facilitates data-informed decisions, providing an edge to respond faster and with more information to market events. Time to market for these solutions are measured in hours-days compared to months.
- Broader scope of data solutions: Using Dataiku has opened the doors of possibility. It's not just about creating a Spark pipeline, but also about developing applications where a non-technical user can interact with the data. This inclusivity broadens the scope of who can use the platform, making it a valuable tool for a wide range of users. Users have analyzed more than 18 TB of data using Dataiku.
- Productivity and efficiency gains for analysts: The platform is designed to be much quicker and more efficient compared to manual processes. It has been instrumental in the creation of numerous projects and applications, showcasing its versatility and utility in various scenarios.
- Cost savings: Many users have transitioned from other platforms to ours due to its ease of use. This has resulted in substantial cost savings.
In summary, we are now conducting more analyses per year, making it times faster, with the better quality, and spending less money for technology human and compute resources.
Value Brought by Dataiku:
As an end-to-end solution, many capabilities of Dataiku helped FINRA become more data-driven and to shift towards End User Computing. The following Dataiku’s capabilities were most crucial for that:
- Visual recipes: Less technical have been on boarded and now conduct analysis more swiftly and efficiently thanks to built-in data preparation possibilities. .
- Dataiku Applications
- Dataiku webapp hosting
In addition, we recognize a very high value of these Dataiku components:
- Robust big data engine: Dataiku enables us to run analysis on very large data sets. This is especially useful as FINRA operates on varying volumes of data depending on market volatility. The same code should run regardless of the size of the data. Dataiku has enabled the development of solutions and products that work for both small and large data sets. It brings the power of Spark to the analysts who are using the recipes and plugins available in the ecosystem.
- Insights democratization: Many analysts are using Dataiku to build a workflow and create a frontend for their colleagues to use in a more user-friendly manner. This includes applications and web apps, dashboards, and more.
- Automation: Scenarios have been developed to automate the flow, running with user-provided inputs to obtain insights via web apps - almost 200 of these have been created.
Value Type:
- Improve customer/employee satisfaction
- Reduce cost
- Reduce risk
- Save time
Value Range: Millions of $
