Over the last five years, I’ve worked on Debezium, a popular open-source platform for change data capture (CDC). It retrieves change events from the transaction logs of databases such as MySQL and Postgres and emits them in a uniform event format to consumers via data streaming platforms like Apache Kafka, Pulsar or Amazon Kinesis. Reacting to low-latency change events enables all kinds of very interesting use cases, ranging from replication to other databases or (cloud) data warehouses, over updating caches and search indexes, to continuous queries over your operational data, or migrating monolithic architectures to microservices.

Now, CDC is an important part of data pipelines for implementing such use cases, but it’s not the only one. You need to reason about the sink side of your pipelines and how to get your data from the streaming platform into your target system. There’s many critical questions there, such as: How do you propagate type metadata? How do you handle changes to the schema of your data? How to deal with duplicate events? Another concern is processing data, as it flows through your pipelines; you might want to filter records based on specific criteria and patterns, apply format conversions, group and aggregate data, join multiple data streams, and more. Lastly, there’s many other kinds of data sources besides CDC, such as sensor data in IoT scenarios, click streams from websites, APIs, and more.

This all is to say that I am really excited about the chance to take things to the next level and explore the field of stream processing at large, helping people to implement their streaming use cases end-to-end. Very often, once people have implemented their first low-latency data streaming use case and for instance observe data in their DWH within a second or two after a change has occurred in their operational database, there’s no going back, and they want this for everything. Of course it’s impossible to predict the future, but I think stream processing is at this point on the famous "hockey stick" curve right before it’s massively taking off, and it’s the perfect time to join this space.

From a technical perspective, Apache Flink, an open-source "processing engine for stateful computations over unbounded and bounded data streams" is an excellent and proven foundation for this, and it’s a core technology behind Decodable. Getting my feet wet with Flink, learning about it in detail and hopefully contributing to it is one of the first things I’m planning to do. At the same time, I also think there’s lots of potential for further improving the user experience here, for instance by processing CDC events in a transactional way, smoothly handling schema changes, and much more. Exciting times!

Up to this point, I have mostly worked at large, established companies and enterprises during my career. Red Hat, where I’ve been at for the last ten years, grew to more than 20,000 employees during that time. Other places, like the German e-commerce giant Otto Group, had even larger workforce sizes of 50,000 people and more.

As with everything, being with such a large company has its pros and cons. On the upside, it’s a relatively safe bet, there’s brand recognition, you can approach and tackle huge undertakings as part of a big organization. At the same time, there tends to be quite a bit of process overhead, things can take a long time, there can be lots of politics, you need approval and buy-in for many things, etc. Note I am not saying that any of this is necessarily a bad thing (Ok, doing your travel expenses just sucks. Period.), lots of it makes sense and just is a reality in a large organization.

That all being said, I just felt that I want to gather some experience in a small environment, in a start-up company. I want to find out how it is to work in this kind of setting, being part of a small, hyper-focused team, working jointly towards one common goal and shared vision. Coming up with ideas, giving it a try, seeing what flies, and what doesn’t. Putting something minimal yet useful out there and quickly gathering user feedback. Having a good sense for your own impact. Seeing how the company grows and evolves. That’s the kind of sensation I am looking for and which I am hoping to find by working at Decodable.

I could experience a first taste of the agility even before my first day at the company: "Would you feel comfortable to just buy a laptop of your choosing by yourself and expense it?" Sure thing! Some clicks and a few days later I had a very nice MacBook delivered to my doorstep. If you’ve been at bigger organizations, you’ll know how complicated such seemingly simple things like getting a new laptop can be.

At the same time, judging by my impressions during interviews, Decodable is a very mature start-up. Most folks have lots of experience, they are senior, in a very positive sense. Sure, there’s a ton of things on our plates, but there’s no expectation to work crazy hours. Many people here have families, and there’s a very healthy culture where it’s just normal that people have unforeseen situations where they need to pick up their kids on short notice, things like that. People are treated as the grown-ups they are, with lots of autonomy and trust by the leadership. Another key aspect for me is transparency: it’s one of the company’s core values, so everyone has the chance to know what’s going on (technically, business-wise, etc.), which gave me lots of confidence and trust when making the decision to join the team.

One of the clichés in the industry is: "It’s all about the people". And yes, it is a cliché, but I’m also 100% convinced that it is true. You could work on the most amazing piece of technology, but if you don’t get along with the people around you, it won’t be an awful lot of fun. Or rather, it could be really bad.

So getting a vibe for the team and the people at Decodable was one of the most important things to me when I interviewed with them. And all I can say is that I was really impressed. Starting with the founder and CEO Eric Sammer, I had the opportunity to speak with about one third of the company’s employees during the interviewing process (talking to everyone is one of my personal onboarding goals, when do you ever get that chance?). I loved the passion, but also the respectfulness and sincereness of everyone. Needless to say that I’m deeply impressed with what the team has accomplished so far, since Decodable launched last year. I experienced Eric as a very considerate and mindful person, caring deeply about the concerns of the company’s employees. Plus, not only is he a legend in the data space, he’s also super well connected within Silicon Valley, opening up lots of doors for the company. Decodable being his second start-up will surely help us to avoid many mistakes.

In regards to the hiring process itself, it could probably be a topic for a separate blog post. The experience was nothing but excellent, with everyone being very open and transparent, willing to answer any questions I had. It really wasn’t that much of a series of interviews, but rather really good two-way conversations which helped us to get to know each other and find out whether I would be a good fit for Decodable, and whether the company would be a good fit for me. All in all, I very quickly had a feeling that this is a group of people I want to work with. I’m sure the direction of the company and the product can and will be adjusted over time, but this is a team I can’t wait to work with to make this a success.

So those are the three key reasons which made me join Decodable: the exciting field of data streaming, the start-up environment, and a highly competent and friendly team.

In case you’re wondering what exactly I will be doing – that’s something we’re still figuring out. I am a member of the engineering organization, so I will get my fingers onto Apache Flink, but of course also on Decodable’s SaaS product around it. But I’m also planning to continue my fair share of evangelization work and talk about technology and its applications in blog posts or conference sessions. I hope to share my input on the product, be part of customer conversations, and much more. For the beginning, I’ll mostly focus on learning and sharing feedback based on my perspective of being the "new guy" on the team.

Fully adhering to the start-up spirit, I’m sure things will be very much in flux and my responsibilities will shift over time. But that dynamic is exactly what I’m looking for by joining Decodable. Let’s do this!