Meltano Data Platform

3 posts · Jul 2018 – May 2020

What this trail covers

Meltano is an open-source ELT tool that allows users to build and manage data integration and transformation pipelines. It supports both CLI and GUI interfaces, with a focus on providing a user-friendly experience for data teams. The tool adheres to Singer specifications for taps and targets, and aims to be the glue between various data components. Future development includes support for PII removal and enhanced CLI user experience. The team is also actively seeking frontend contributors and dogfooding the platform for internal data analytics.

All GitLab trails

May 2020Why GitLab is building Meltano, an open source platform for ELT pipelinesThis post announces the strategic shift of Meltano towards becoming an open-source, self-managed ELT platform. It articulates the 'why' behind this decision, drawing parallels with GitLab's open-source origins. The core problem identified is the 'pay-to-play' nature of current data integration tools, which disadvantages companies unable to afford proprietary solutions or facing data residency constraints. The solution proposed is an open-source community-driven approach to building and maintaining extractors and loaders, leveraging the power of collaboration to overcome the maintenance burden and create a true alternative to proprietary vendors. The post emphasizes that data integration should be a commodity accessible to all.
Apr 20194 Examples of the power of open source analyticsThis post highlights the benefits of open-sourcing GitLab's data analytics codebase, specifically mentioning the contribution of an external user to update a date metadata generation model for Snowflake compatibility. It also emphasizes how the open codebase facilitates community engagement and knowledge sharing within the analytics community, particularly around tools like dbt. The core message is that the code itself is not the competitive advantage, but rather the data and the decisions derived from it.
Oct 2018New Meltano personas, priorities, and updates from the teamThis post introduces two general Meltano user personas: those with engineers on staff and those without. It highlights the need for both CLI and GUI interfaces due to user frustration with existing tools. The post also details the team's iterative development approach, their focus on building more extractors and loaders (taps and targets), and their commitment to dogfooding pipeline failures to improve user experience. An embedded engineer initiative is also mentioned to foster better collaboration and understanding between the Meltano team and data engineers.
Aug 2018Thanks for all the feedback and interest in Meltano!This post details the feedback received on Meltano, an open-source ELT tool. It highlights the team's commitment to building core extractors for common sources like Salesforce, Marketo, and Zendesk. It also mentions plans to apply Meltano to a machine learning project for predicting sales opportunity win probability. The post clarifies that Meltano's integration part is currently GitLab CI-based but explores the idea of a frontend 'production mode' and a 'development mode' within a Python Flask app. It reiterates the vision of a convention-over-configuration framework and the benefits of a monorepo for easier development and contribution. The team confirms that Meltano is not mandatory for all tasks and expresses a desire for a clean API for GitLab CI to allow external plugging. The post emphasizes the ambition of the project and the reliance on an open-source community.
Aug 2018Hey, data teams - We're working on a tool just for youThis post introduces Meltano as a new open-source product for data teams, aiming to provide a complete solution for the data science lifecycle (model, extract, load, transform, analyze, notebook, orchestrate). It addresses the problem of fragmented data sources and tools in business operations, drawing parallels to software development best practices like version control and open-source tools. Contributions are welcomed for Meltano Analysis (UI), Extractors, and Loaders, with an initial focus on Postgres and planned support for Snowflake. Readers are encouraged to contribute by opening issues for feature requests and suggesting improvements to the Data team's work.
Advertisement