Get Fully Briefed with Yahoo Finance, delivered straight to your inbox.
Confluent, the company that built a streaming service on top of the open source Apache Kafka project, has always been about helping companies capture streams of data. First it was on prem, then later in the cloud. By moving the streaming service to the cloud, it was able to abstract away a lot of the complexity related to managing the underlying infrastructure.
Today, at the Current conference, the company is introducing a new tool called Stream Designer to make it easier to build a streaming data pipeline in a visual workflow. Users can easily connect a set of data components to build a customized stream of data, and Confluent handles the coding in the background for them, essentially moving the abstraction up the stack from infrastructure to design.
Company co-founder and CEO Jay Kreps says that while it’s still aimed at developers, the goal is to put stream design within reach of more people than Apache Kafka experts.
“We’re still serving developers of all types, but this makes it something where you can just click and build a data pipeline that connects things, transforms data, sends things from place to place. And it inherits and works on the same underlying streaming infrastructure. That means all the data streams are reusable, they’re all real time and they’re all horizontally scalable,” Kreps told TechCrunch.
He says that although the program writes the underlying code for the developer, it is still accessible for developers who need to work with it. “One of the cool things about how we’ve done this, is even though you don’t have to write any code, all the code is there. You can see exactly what it’s doing under the hood. You can see exactly the kind of transformations in SQL that we’d be doing on this data, so it works with the existing developer tool chains if you need to do that,” he said.
Confluent Stream Designer example. Image Credits: Confluent
For Kreps, this is all part of the evolution of Kafka, a project originally conceived at LinkedIn to move massive amounts of data. At first, it was a highly technical undertaking, but over time the company has been trying to make it increasingly accessible to a greater number of people inside an organization.
In addition to Stream Designer, the company is also announcing Stream Governance, which helps ensure the data is being used correctly and only authorized users can see data.
“For a lot of organizations it’s almost like there’s two competing pressures: One is to unlock the data and use it to be effective and serve customers better and be more efficient. And the other is to lock it up and be safe and don’t let anything bad happen. And unless you have tooling that helps with both of these dimensions, you kind of end up a little bit stuck,” Kreps said. Stream Governance helps users make sure they are using the data flowing through these data streams in a safe and compliant way.
While they were at it, Confluent announced a new Confluent for Startups program, designed to get startups using the platform with free credits and access to expertise to help them get started with data streaming technologies.
Stream Designer and Stream Governance are available starting today as part of the Confluent cloud service subscription package, and there will be no additional charge for the new capabilities, he said.
What open source-based startups can learn from Confluent’s success story
Millions of Aussies say they have no savings to fall back on. It comes amid rising interest rates and cost of living pressures.
All of the major banks have passed on the Reserve Bank’s interest rate hike in full. Here’s what customers will be paying.
Aussies on the east coast of the country have been overcharged $10 billion by energy providers taking “supernormal profits”. This is how it happened.
An extremely rare Australian penny has sold for $60,000, breaking records. Here’s what made it so special.
Interest rates are rising as the Reserve Bank tries to get soaring prices under control, but almost 1 million Aussies are set to be really struggling next month. This is what you need to know.
The Reserve Bank of Australia has lifted interest rates for a sixth consecutive month. Here’s the good, the bad and the ugly.
Two law firms are considering class actions against Optus over the data breach. Find out if you are eligible.
The names and email addresses of thousands of current and former Telstra staff have been exposed. Here’s what we know so far.
Millions of Aussies have only a few days left to use up to $300 worth of free vouchers. Here’s what you need to know.
Atlassian kicks off its recruitment drive today. The Aussie company has over 1,000 jobs to fill. This is what’s on offer.
The Big Four banks are offering better deals to new customers than they are to existing mortgage holders. Here’s how to get the best rate.
The local market is expected to lift this morning after US stocks soared overnight. This is your Wednesday morning wrap.
The Reserve Bank board is meeting today and is expected to raise interest rates. Here’s what the big four banks are expecting.