ALMA: Streamlining Astronomical Operations With Analytics and AI

Dataiku

ALMA leverages Dataiku to automate data monitoring, streamline proposal reviews, and optimize telescope operations, saving hundreds of hours and enhancing astronomical research.

50-120 Hours

Saved of Astronomers on Duty (AoD) and expert staff time by reducing manual assessments

Weekly

Monitoring of observational data (previously unattainable)

1,700

Proposals processed through automation

The Atacama Large Millimeter/submillimeter Array (ALMA) is renowned for its groundbreaking contributions to astronomy, but managing the vast volumes of data, proposals, and observations required to power such achievements poses significant challenges. 

ALMA partnered with Dataiku as part of the AI-for-Good Program — a program that helps NGOs deliver positive impact with data and AI — to implement innovative solutions across various use cases to streamline operations, improve efficiency, and ensure its resources are focused on advancing astronomical discoveries.

 

 

Efficient Monitoring of Observational Data

ALMA’s contact scientists (CS) and science operation specialists oversee thousands of observations (SB) and data processed products to ensure they progress correctly through the observation and data processing pipeline. Manual tracking of these was labor-intensive, leading to delays in identifying and addressing issues, which impacted service delivery.

Using Dataiku, ALMA developed automated dashboards and reports to flag problematic SBs and MOUSs. This solution integrated data from various sources, enabling team members across regions to collaborate and share insights efficiently.

The Impact?

• Automatized regular checks once per week — previously unfeasible due to resource constraints and lack of a data science platform.
• Improved communication among global teams enhanced issue resolution and service reliability.
• Boosted relationships with internal teams (i.e., telescope operations) and external users, enhancing trust and satisfaction.
• Team members were upskilled in data analysis and programming practices.

Optimizing the Proposal Review Process

Each year, ALMA solicits proposals from the astronomy community for ideas on how best to use the telescope. ALMA’s proposal review process involves 1,700 proposals, 1,000 reviewers, and 17,000 reviews, making it the largest in astronomy. Matching proposals to the right reviewers based on their expertise was challenging, with existing algorithms lacking flexibility and accessibility for the broader team.

ALMA used Dataiku to create machine learning (ML) models that parsed the proposal text, inferred the proposal topics, and matched them with reviewers’ expertise. The platform’s integration with Python allowed ALMA to test various algorithms and refine their approach iteratively. As a result, the team developed automated workflows to efficiently process proposals, compute the similarity between proposals and reviewer expertise, and ensure that the right proposals are reviewed by the right people. These automated workflows reduced manual workload and errors, increased productivity through quicker onboarding of new members, and bettered the quality of reviews through the improved proposal-reviewer matching.

Enhancing Quality Assurance for Telescope Operations

Astronomers on Duty (AoDs) at ALMA are responsible for daily telescope operations and, as part of their remit, perform Quality Assurance Level 0 (QA0) to certify observation data quality. Roughly 10% of observations required manual assessment, consuming significant staff time and delaying subsequent processes.

ALMA collaborated with Dataiku to build an ML model capable of classifying QA0 observations. Dataiku’s AutoML and MLOps capabilities accelerated development and deployment, reducing manual intervention. This solution reduced manual QA0 assessments by 82 observations over three months, saving 50 hours initially, with a potential of 120 hours saved when fully deployed. ALMA also benefitted from lowered costs tied to an optimization of computational resource usage and enhanced data quality and reliability.

Transformational Results

Through its collaboration with Dataiku, ALMA has revolutionized the way it works. By automating processes, the organization saves hundreds of hours every year, allowing teams to focus on what truly matters. Dataiku also fosters seamless collaboration across teams with varying expertise, enabling them to work together effortlessly on shared projects. 

These changes have led to smarter decision-making, powered by automated and transparent insights, while optimized resource utilization has significantly cut operational costs.

INDUSTRY

Nonprofit

FOUNDED IN

2013

TECH STACK

Dataiku + K8S + PostgreSQL

KEY DATAIKU CAPABILITIES

Machine Learning

AutoML

Data Insights

XOps

MLOps

ABOUT THE COMPANY

ALMA, the Atacama Large Millimeter/submillimeter Array, is an international astronomy facility — a partnership among Europe, the United States, Canada, Japan, South Korea, Taiwan, and Chile. Located in the Atacama Desert of northern Chile, ALMA is the world's most powerful telescope for observing the cool Universe — molecular gas and dust. It provides unprecedented insight into star formation, galaxy evolution, and the origins of planets and life itself.

www.almaobservatory.com