Case Studies

How Airbyte Launched A Fully-featured Enterprise API in One Quarter

Nolan Sullivan

June 8, 2023

By partnering with Speakeasy to build its API key management and provide SDKs to users, Airbyte was able to build faster than they thought possible and accelerate the launch of its Airbyte API (opens in a new tab).

Challenge

Airbyte (opens in a new tab) (W’20 YC) is an open-source data movement tool. Airbyte powers data movement by syncing data from applications, APIs & databases to destinations such as data warehouses, data lakes & databases. They are differentiated from other integration tooling by their developer-first approach, which has resulted in a robust and loyal community. Over the last three years, they have grown quickly, to nearly 100 employees and ⭐ 10K on Github (opens in a new tab).

After the initial success of their Open Source tooling, Airbyte users began clamoring for a managed offering. To meet this demand, Airbyte has been working hard to build an enterprise platform that lives up to the high bar set by its open-source project. A major component of building that best-in-class experience was providing access to the Airbyte API to enable users to manage their ELT pipelines programmatically.

Riley Brook is the Product Manager tasked with launching the Airbyte API. When he and his team began scoping the project, the monstrous scale of the task quickly became apparent. To launch the Airbyte API, they not only needed to externalize Airbyte’s internal services, they also needed to build a variety of additional tooling: from authentication and request logging, to SDKs and usage reporting.

After the initial scoping exercise, they set out to answer a vexing question: What is the best way to go from internal services to a fully-featured, externally-available API, when customers are banging down your door?

Initially, they faced some unappealing options:

Dedicate More Resources - Unfortunately, there are no free lunches. The enterprise API already had a team dedicated to it, any additional resources would’ve meant compromising on other, equally important, Airbyte initiatives.
Delay the Launch - This would’ve meant disappointing customers, not something Airbyte is in the habit of doing.
Compromise on Quality - They had already descoped as much as possible. Further cuts to the projects would’ve meant jeopardizing user experience.

Surprisingly, we couldn’t find an easy way to offer secure, self-service API key management to our API users. We were considering building our own solution — which would have taken too long. Speakeasy solved this for us.

Riley Brook, Product @ Airbyte

Solution

Speakeasy’s platform makes it easy to create APIs that are easy for users to consume at a fraction of the cost of building them yourself.

When Speakeasy first met Airbyte there was skepticism that it would be possible for a 3rd party to provide a sufficient and seamless user experience to Airbyte’s Cloud customers.

However, after the first meeting, they came away thinking they’d found a partner they could work with to accelerate the launch of their API. What won them over was Speakeasy’s commitment to:

Code Quality - The attention to detail in the SDKs produced by Speakeasy was easy to see. The libraries were fully typed for a good IDE experience, and each SDK felt native to the language it was written in.
Easy Integration - The Speakeasy product is all integrated via industry-standard methods. The gateway integration could plug into their existing provider (via OIDC), while their OpenAPI spec could be used for creating SDKs.
Extensibility - They could build and modify Speakeasy’s tooling as needed. Speakeasy’s front-end components were all available as React embeds that the Airbyte team could integrate into the existing platform.

Start of the Partnership: API Keys

The collaboration started with fleshing out the first step in the user journey: API access. On the front end, Speakeasy’s react-native embeds slotted seamlessly into the Airbyte platform’s existing look & feel. On the backend, Speakeasy integrated with Airbyte’s API gateway to externalize its internal key management system (see here for the long version (opens in a new tab)). The result is that users can now log into the Airbyte portal and self-manage their API keys: creation, revoking, etc.

“Airbtye API users being able to quickly and easily generate API keys fits perfectly within Airbyte’s user experience, and our focus on frictionless onboarding.”

Riley Brook, Product @ Airbyte

After experiencing the ease of getting the API keys set up, the Airbyte team decided to pull in several of the previously descoped “nice to haves”, using Speakeasy to set up a usage dashboard and request logging to round out their API portal experience.

Results

Three months after the project kicked off, and a quarter ahead of schedule, the Airbyte API (opens in a new tab) is live and being used by clients every day, and they are loving it:

“The Airbyte API enabled us to streamline our entire ELT processing workflow without having to worry about the multiple tools we're using.”

Trish Quintans, Data Engineering Lead, Fillip Fleet

The tooling has enabled Airbyte’s users to manage their keys, understand their usage, and better troubleshoot any integration issues that pop up. But this is just the beginning. The Airbyte team understands that developer experience is an ongoing investment that sets their API apart.

Next Step: SDKs & Terraform

With the API portal solved, attention turned from API access to API integration. To accelerate API integration, Airbyte wanted to support its API with language-idiomatic SDKs.

They also were interested in offering a Terraform Provider (opens in a new tab). Terraform was particularly interesting as Airbyte is a core component of their client’s infrastructure. Offering a provider would allow their users to manage their Airbyte configuration along with the rest of their infrastructure; the perfect user experience.

Speakeasy’s managed SDK pipeline allowed them to create all the interfaces they needed for their API, without burdening the team with additional ongoing maintenance. In mid-April, Airbyte rolled out its first two SDKs: Python (opens in a new tab) & Java (opens in a new tab). The terraform provider is slated for release on June 22nd.

It’s been a dizzying 5 months of work, and we can’t wait to see how the Airbyte API continues to grow, and we look forward to supporting Riley and the whole Airbyte team along the way.

“We looked for the best options in API tooling so we didn’t have to build everything ourselves. We focus on what we do best: ensuring data is accessible everywhere it has value. For our API needs; we have Speakeasy.”

Riley Brook, Product @ Airbyte