M1 - Building a Mature Analytics and Data Science Practice to Develop ML Models and Become More Pred

nirr3

Name:

Nicklas Ankarstad, Sr. Director, Data Science & Analytics
Nick Hotz, PM, Data Science
David Quach Sr. Data Scientist
Brady Dauzat, Data Scientist
Josh Bender, Data Scientist

Country: United States

Organization: M1

M1 is a financial services company that empowers you to invest, borrow, and spend on one intuitive platform. The Finance Super App® offers tools to customize your investment strategies and automate the big picture—commission-free.

Awards Categories:

• Value at Scale
• Most Impactful Transformation Story
• Most Extraordinary AI Maker(s)

Business Challenge:

M1 was founded in 2015 with the simple idea of building a better financial platform for users to manage their money. We spent the first few years building infrastructure and developing the product. During that time, we relied on manual reporting to understand who our users were and what drove product adoption.

We were far away from any type of predictive analysis that could help us anticipate users’ needs or responses.

At the start of 2021, we had a great engineering practice, including a data engineering team that built a data platform. Despite the data-rich environment, our analysts mainly spent their days querying databases, creating simple charts in a basic reporting tool, or doing their analysis in Excel. They routinely answered the same questions about historical performance and had a limited impact on business outcomes.

As M1 grew, we needed to increase the sophistication and quantity of analysis. To continue to develop, our marketers and leadership needed to better understand:

• Who the users of the platform are.
• What makes a good user good.
• How to improve funding, approval, and renewal rates.
• What drives product activation.
• What our users are saying about us.

To answer these questions and become more predictive, M1 needed to build a mature analytics and data science practice.

Business Solution:

With a growing demand for more predictive insights and the desire to develop machine learning models, M1 hired a data science team and partnered with Dataiku.

Within the first 30 days, our team deployed the first prototype of a model that predicted a user’s expected assets under management at a certain point in their tenure on the M1 platform.

We expanded the models built to include user segmentation using K-means clusters, which helped us develop user personas. Next came a plethora of propensity models that predicted the likelihood that a user would open a specific product. We also built a marketing mix model to help marketers efficiently allocate budgets, along with a renewal model for M1 Plus that accurately predicted whether a user will renew their membership.

The flexibility of Dataiku allowed the data scientists and analysts to quickly build GUI-based AutoML models when the time to market was more important, and code-based models when flexibility was more important.

By taking full advantage of the built-in Jupyter notebooks and python environments, our analysts and data scientists could do more complex write-up analyses beyond just visual analytics. These write-ups influenced product strategies, marketing initiatives, and forecasts.

As our team deployed more models, we struggled to maintain them. Stakeholders often asked if they could trust the output of a model, and the data team had to check the model each time. Without the ability to confirm trust in the model, stakeholder trust deteriorated.

With Dataiku, we automated much of our MLOps checks. We used scenarios to evaluate model performance and set up checks that automatically alerted our team on Slack when model performance drifts below certain thresholds. Today, the data science team can confidently answer stakeholders that our models are performing as expected.

Business Area Enhanced: Accounting/Analytics/Internal Operations/Product & Service Development/Financial Services-Specific

Use Case Stage: In Production

Value Generated:

Moving from basic reporting and excel to deployed ML significantly impacted M1. By implementing Dataiku, M1 has made huge leaps forward in analytical maturity. The ability to build, deploy, and monitor machine learning models, write notebooks, and do diagnostics analysis has reshaped M1’s analytics program. The team has moved from Excel-based reporting to casual impact, exploratory analysis, automated dashboarding, and data pipelines. We can now deploy a model in hours, rather than months. And we’ve saved about 5,000 of work thanks to streamlined processes.

In terms of business impact, the models and analysis produced have changed M1's strategy. This includes which users we talk to about what product, the strength of our product offering, and knowing our users better. For example, the M1 Plus renewal model has helped us set targets for the membership, understand how product adoption influences renewal rates, and served as a kick-off platform for pricing experiments. This has ultimately led to better product experiences and increased revenue.

Many of the marketing-oriented models have helped our marketing team allocate their budgets and resources more effectively. The MMM model, for example, lets our marketing team run what-if scenarios and see the impact of changing channel mix on user growth. Making informed decisions when marketing has saved us approximately $2.5 million and increased our efficiency.

Value Brought by Dataiku:

Using Dataiku, our team quickly prototyped models by writing SQL recipes to ingest data and using the GUI to develop and evaluate several models. The best model was deployed, with the results stored back into a database. The ability to quickly develop and evaluate models accelerated the delivery time from months to hours.

With some model evaluation reporting built in, stakeholder discussions became easier. The team could show how one feature impacted the model outputs through partial dependence plots, what features were most important through feature importance plots, and how changing the cutoff impacts the confusion matrix.

As the data team deployed more models, ensuring performance became a challenge. We no longer reviewed or manually ran the models on a daily basis. So we built an alert system so we would know if a model didn’t run, performance deteriorated, or input data drifted. Luckily with the newer versions of Dataiku, we implemented model performance checks, data drift checks, and alerts if the model fails checks or didn‘t run. The team now monitors a Slack feed to check issues and alerts stakeholders when something is happening.

As the analytics teams built out more dashboards, we realized that analysts were building more and analyzing less. This left data interpretation to business stakeholders. Sometimes that was great, and sometimes it wasn’t.

With Dataiku, analysts have another tool to use notebooks to create repeatable charts and written analyses. Because it’s written in code, we can version-control the analysis through git and leverage the established pre-defined Dataiku connectors to our database. This also reduced the need for analysts to download data to .csv and Excel files to their laptops, avoiding potential security issues. Overall, the analyst and data scientist saved upwards of 5,000 hours by implementing Dataiku.

Value Type:

• Improve customer/employee satisfaction
• Increase revenue
• Reduce risk
• Save time

Value Range: Millions of $

M1-Nick

Very exciting work! Super excited about the fast progress of the team in such short time

KyleB

Great work Nick and team!

tcation

Hats off to Nick and the amazing team!