Metabase is an open-source analytics platform that runs on the JVM. It is available as both a self-hosted JAR and a cloud-hosted service. Customers connect Metabase to databases in a variety of environments, including databases outside Metabase's infrastructure and on-premises databases reached through SSH tunnels.
Metabase transforms let users build scheduled data pipelines. A "query transform" runs a saved query and writes its results to a target table in the same database. Dependencies between transforms form a directed acyclic graph, which Metabase uses to run them in the correct order. Today, each transform executes entirely inside the customer's database.
We are adding a new kind of transform: a "Python transform" that runs user-provided Python code. This will be used for operations that are difficult or impossible in SQL, such as complex numerical analysis or unstructured-text processing.
We want you to lay out how this feature could be implemented from a systems perspective.
- Metabase runs on the JVM and has no existing Python infrastructure.
- Each Metabase Cloud customer deployment has at least two application nodes for high availability.
- These nodes use a shared PostgreSQL database for application state, separate from their connected database.
- Python transforms will initially be available only in Metabase Cloud.
- Support in the open-source edition will follow later and may differ in its implementation.
- PostgreSQL will be the only supported database in v1.
- MongoDB and BigQuery support will follow in later releases.
- Python transforms will fit into the existing transform graph and be coordinated by the existing scheduler.
- Each Python transform produces exactly one target table, like a query transform.
- Assume fewer than 50 Python transform executions per hour in each customer deployment.
- Source tables for Python transforms will typically be moderate in size, < 1m rows
- We must support some common Python data libraries with native extensions, including NumPy and pandas.
We are interested in the major design decisions and their trade-offs, not in reproducing Metabase's implementation. State any additional assumptions you make, and feel free to ask clarifying questions.
We will spend about 40 minutes on the exercise. During that time, please produce a system diagram showing the components involved and the important data flows.
The following may also prove helpful:
- A rough sequence outline for executing one Python transform end to end.
- Pseudocode showing the customer-facing shape of a Python transform.
You may use any diagramming or note-taking tool you prefer.
We do not expect executable code, detailed API or database schemas, or low-level infrastructure specifications.
We are not evaluating your knowledge of Python or its ecosystem. Feel free to ask questions such as "Does it support X?" or "Is there a way to do Y?"