Country: United States
FINRA is a not-for-profit organization dedicated to investor protection and market integrity. It regulates one critical part of the securities industry—brokerage firms doing business with the public in the United States. FINRA, overseen by the SEC, writes rules, examines for and enforces compliance with FINRA rules and federal securities laws, registers broker-dealer personnel and offers them education and training, and informs the investing public. In addition, FINRA provides surveillance and other regulatory services for equities and options markets, as well as trade reporting and other industry utilities. FINRA also administers a dispute resolution forum for investors and brokerage firms and their registered employees.
We use innovative AI and machine learning technologies to keep a close eye on the market and provide essential support to investors, regulators, policymakers and other stakeholders.
The Financial Industry Regulatory Authority (FINRA) is a not-for-profit organization authorized by the U.S. Congress to protect investors and ensure market integrity through effective and efficient regulation of broker-dealers. It writes and enforces rules governing the activities of more than 3,400 broker-dealers representing more than 630,000 brokers, examines firms for compliance, fosters market transparency, and educates investors.
Scale of Operations:
Each day, FINRA oversees nearly 600 billion market events across equities, options, and fixed income products within the U.S. This results in petabytes of historical data. The primary challenge is deciphering this vast data pool to identify malicious activities, such as insider trading, and more, which can tilt the market scales unfairly.
Given the unpredictable nature of today’s markets, swift response mechanisms are crucial. At the heart of FINRA’s operations is the objective to ensure investor safety and uphold market integrity. This necessitates rapid data analysis to answer pivotal questions, including the identification of market threats and required regulatory interventions. The ability to analyze vast amounts of data swiftly from diverse sources is fundamental to our mission.
We leveraged Predefined compute cluster approach, as that is one of the recommended approaches for managing compute for the analytics workloads during the pilot period in Dataiku. This addressed some of our scalability needs. However, due to FINRA’s significant data volume, we anticipated capacity bottlenecks using this approach. With these limitations, jobs can become backlogged, causing considerable analysis delays, compromising swift action, and hampering user experience. In addition, users would be dependent on the platform administration team for tailored cluster configurations.
To navigate these challenges, FINRA adopted a more self-sufficient, automated, and team managed strategy for users to adapt computing power to the task at hand, without being limited by the computational power of their laptops or the pre-defined clusters offered by administrators. In this evolved system, each team is empowered with the responsibility of launching, upkeep, and cost management of their respective clusters. This shared responsibility ensures bespoke cluster configurations, fostering rapid and efficient data analysis.
We developed a set of macros called Node Launcher in Dataiku based on Kubernetes. This capability allows users to define the bounds and limits of computational power for each project, thereby eliminating the constraints imposed by computing guidelines.
However, with the increase in user adoption, the need for implementing guardrails became evident. These include:
To fortify this capability, we have Community of Practice sessions, providing a forum for teams to discuss challenges, solutions and learn from one another’s experiences. We also document user stories that are catalogued in a central repository, offering a resource for teams to reference, learn and avoid reinventing the wheel. This approach not only resolve current challenges but also imbibe a culture of collaboration, continuous learning, and shared responsibility.
Business Area Enhanced: Analytics
Use Case Stage: In Production
Dataiku specifically enabled this positive impact by:
This led the entire organization to conducting more efficient data analysis in reaction to market events, making more informed and faster decisions which may have nation-wide impact.
Dataiku has provided a self-service analytics capability where users are not limited by computational power. Leverage elasticity from the cloud to analyze the data based on their needs. Not limited by computational power of their laptop or pre-defined clusters that admins offered.
Value Range: Millions of $