Redpanda

Data streaming platforms (DSP) enable businesses to move and process information in real-time, transferring information from generation at the source to systems with specific use cases, through channels known as data pipelines. Demand for real-time data transfer capabilities is growing, with 72% of IT and engineering leaders making use of the technology as of 2023. Adopters credit real-time data with enabling new opportunities for operational and customer insights as well as potential for more engaging interactions.

Notable use cases of real-time data streaming include Uber's use of GPS tracking, rider requests, and driver status updates to output ETA predictions and handle surge pricing, Netflix tracking user behavior to refine recommendation engines, and retailers like Walmart optimizing inventory management systems through sales predictions and strategically timed personalized discounts. These use cases of so-called data in motion contrast the traditional model of data at rest, which requires the data to be stored, processed, and queried at specific intervals referred to as batch processing, as opposed to the immediate availability of data for system ingestion as facilitated by data streaming.

AI/ML workloads have given rise to increased commercial focus on full utilization of data. Global data center capex was estimated at $434 billion in 2024 and is expected to reach $1.1 trillion by 2029, driven by demand for model training and inference. Businesses are incentivized to track everything from customer and compliance data to property, plant, and equipment (PPE) and cloud configuration monitoring data. In 2025, global data volume is expected to reach 181 zettabytes, up from 15.5 zettabytes ten years ago. Proliferation of devices has raised the average data generation rate to roughly 35.4 MB per person per minute in 2024, up from 9.7 MB in 2019. Modern data at rest solutions can fail to fully capitalize on potential data applications due to the half-life of data: deeper, point-in-time analytics may offer valuable insights for strategic decisions, but operational and tactical decisions require more reflexive action to newly generated events.

The basic requirements for companies to support data-driven insights are storage, compute, and transference. The cloud and advances in hardware have progressed the first two: since 2014, computational power has grown by a factor of 50.1x, and storage costs have fallen by a factor of 15.8x. The limiting factor in growing insights is the ability of companies to circulate data effectively, with 79% of IT leaders in 2024 citing data streaming as a top strategic priority.

As of 2025, the industry-leading data streaming solution is Apache Kafka, which is known for its scalability, performance, and open-source code. Redpanda, a competitor to Apache Kafka, offers much of the same functionality, but in C++, addressing the latency issues associated with Java deployments and focusing on performance, cost, and developer experience. For the majority of use cases, both companies help clients build data pipelines, though Redpanda also offers hardware tuning and tail-end performance gains. Following its most recent funding round, the company is focusing on expanding its customer base and maintaining momentum across a variety of industries.

Redpanda was founded in 2019 by Alex Gallego (CEO). Born in Colombia, Gallego’s obsession with performance and complex systems began in his uncle’s motorcycle repair shop, where he disassembled and reassembled engines as a child. At 14, he moved to the US, where he excelled in school and skipped several grades before attending NYU’s Polytechnic School of Engineering for cybersecurity. Gallego found a mentor at NYU in Professor Nitesh Saxena, with whom he collaborated on research using game theory to orchestrate strategic interactions for advanced security solutions.

After NYU, Gallego worked at data provider FactSet (learning C++) before joining an advertising technology firm, YieldMo, as its first engineer. Frustrated by the limitations of real-time processing application Apache Storm, which he used at YieldMo, Gallego left the company and founded Concord Systems, a high-performance C++ real-time processing application. Concord Systems was acquired by Akamai in 2016, and Gallego joined the company as one of Akamai’s youngest software engineering principals. He focused on optimizing software for hardware, designing a storage engine that improved performance relative to Kafka, laying the foundation for Redpanda.

Redpanda emerged from Gallego’s observation of the fact that the gap between hardware and software created bottlenecks for streaming. The company’s mission since inception has been to minimize overhead configuration and improve developer experience. Since its founding, Redpanda has benefited from the demand for real-time data enablement and grown to roughly 170 employees as of 2025. Gallego’s guiding design principle, “60s to WOW”, has enforced standards to ensure a frictionless developer experience that requires less than 60 seconds to execute processes like setting up a cluster or live processing events.

Core Architecture: The Publisher-Subscriber (Pub/Sub) model

Source: Stéphane Derosiaux

The Pub/Sub model is the foundational architecture for data streaming solutions, and is itself an evolution of traditional message queue systems used by companies like RabbitMQ. These legacy-based messaging systems have historically had limitations, like requiring tight coupling (i.e., the producer and consumer knowledge of one another’s existence, enabling direct communication) and lacking a data retention mechanism, meaning consumers also had to be online to receive messages from the producer.

The Pub/Sub model’s transition to event-driven architecture (EDA) provided messages with a unique identifier (a timestamp) and posted the messages to a message broker, responsible for categorizing them into topics to which consumers can subscribe and receive all incoming data. These brokers play a crucial role in storing the data, mediating communication between producers and consumers, and monitoring the health of these consumers in larger enterprise use cases. This architectural shift allowed each system to have a distributed event log, enabling event replay and reducing complexity, since direct connections to each consumer were no longer needed. These two features, replayability and decoupling, improved both performance and functionality, ultimately allowing users to reconstruct system memory and analyze patterns as a log of messages became available through time.

Apache Kafka has long been the dominant player in distributed data streaming using this model, but Redpanda addresses key inefficiencies in even the improved design. By rethinking Kafka’s core components, Gallego identified three opportunities for refinement: hardware advances hadn’t materialized into reductions in latency, there existed no unified platform to use Kafka, Kafka Connect, and Flink in unison, and the use of storage wasn’t being optimized with respect to price. Through new features and improvements to the underlying infrastructure, Redpanda claims to be 6x more cost-effective and 10x faster than its open-source counterpart.

Source: Quix

Deployment and Architecture Enhancements

Redpanda is designed to be a drop-in replacement for Kafka, offering full API compatibility while improving upon Kafka’s underlying infrastructure. This allows companies to migrate from Kafka without an extensive system overhaul, reducing friction and accelerating transitions. When Redpanda launched as a competitor, Kafka relied on a relatively resource-intensive coordination service called ZooKeeper for managing broker data replication and fault tolerance. Since the founding of Redpanda, Kafka has shifted to coordination service KRaft. This transition was not without friction, and Kafka’s performance divergence with Redpanda persisted. Redpanda comes packaged with a native raft consensus protocol, which is less resource-intensive and provides faster leader reelections. As of March 2025, Kafka 4.0 has been announced with native rat in line with Redpanda, although performance benchmarks have not been released.

Another key improvement versus Kafka is Redpanda’s single binary architecture, which eliminates the need for multiple security dependencies, simplifying deployment and management by having all requisite protocols pre-installed. Because it is based in C++, Redpanda isn’t restricted by a Java Virtual Machine (JVM), which introduces additional program compilers and increased overhead for Kafka. Redpanda also provides Redpanda Console, an intuitive UI for managing clusters, providing visibility into data pipelines and cluster configurations, and eliminating the requirement of debugging via command line.

Performance and Cost

Redpanda is built on the Seastar C++ framework, which includes an intelligent memory use auto-tuner to maximize hardware utilization. Seastar’s thread-per-core architecture ensures that each CPU core has a dedicated thread, reducing context switching (a source of latency defined as the time taken for the CPU to stop executing one task and switch to another) and improving throughput.

Source: Redpanda

Redpanda introduces intelligent tiered storage, automatically transferring older data from SSDs or NVMe drives to more cost-effective cloud object storage. This significantly reduces storage costs while maintaining fast access to frequently used data. Additional features, like the memory auto tuner and the leader rebalancing mechanism, help minimize administrative overhead and manual interventions.

Advanced Features and Scalability

Redpanda hosts several advanced features, including the ability to perform real-time transformations through WebAssembly (WASM), which enables developers to run data transformations in their preferred languages. By pre-processing data streams before they reach applications, Redpanda can help ensure data veracity and privacy compliance. Redpanda has expressed optimism about a future demarcation between the data and control planes for companies operating in the cloud, allowing companies to regain autonomy over their data and service providers only the ability to read the minimum sufficient metadata. The company has contributed to this vision by embracing the BYOC (Bring Your Own Cloud) solution, giving organizations the opportunity to host data streaming within their own cloud configuration.

To ensure horizontal scalability, Redpanda utilizes partitioning and consumer groups, both of which are methods enabling the parallel processing of data streams for improved scalability.

Customer

Redpanda’s customer base spans a range of industries; The company does not focus on a single vertical but instead targets various sectors where real-time analytics provide substantial value.

Gallego has commented in interviews on the product’s particular success with financial firms. In the banking sector, for example, Redpanda has played a critical role in enhancing customer engagement with 360-degree customer insights and feeding real-time fraud detection algorithms. The combination of higher performance and lower cost relative to competitors has made Redpanda particularly successful with algorithmic traders and market makers, in which low latency is requisite.

In retail, sophisticated inventory management systems have been tailored to correlate customer purchase behavior with demand patterns to reduce delays due to depleted stock. In casinos, Redpanda’s customers have used real-time data streaming to improve payment efficiency and detect opportunities for engagement between patrons and operators. Redpanda has also worked with customers in insurance and healthcare for real-time patient monitoring, customers in media and entertainment on recommendation engines, and customers in energy and telecommunications for IoT device and equipment monitoring.

Market Size

The global big data market was valued at $327.3 billion in 2023, at which time the market size was expected to grow to $865.3 billion by 2030, driven by a rise in online transactions and telemetry (the number of devices producing data). Organizations becoming more data-conscious in light of the potential of AI are searching for partners to help them read, query, and incorporate their data into decision-making. As noted earlier, cloud solutions have significantly lowered the barriers to accessing computing resources and storage, making transferring that data the “fourth estate” for data insights. Consequently, the real-time data streaming market was estimated at $10.2 billion in 2023 and is predicted to $35.3 billion by 2032, with an expected CAGR of 18.5% over the period and outpacing the data streaming industry overall.

Source: Kai Waehner

Apache Kafka

Apache Kafka, made open-source by LinkedIn in 2011, remains the industry standard for distributed data streaming platforms. Kafka has set the benchmarks for minimum customer expectations for throughput, data replication fault tolerance, and scalability via brokers and partitioning. These core functionalities are now ubiquitous in data streaming solutions, and competitors frequently either iterate ok Kafka’s API or take heavy inspiration from Kafka’s design.

Kafka is not without limitations. The platform lacks real-time data processing capabilities unless paired with Apache Flink or Kafka Streams, and doesn’t provide built-in governance or security measures. As an open-source platform, it falls short of paid solutions in technical support and maintenance. These shortcomings have led to the rise of several alternatives that seek to address some of these gaps, particularly around real-time processing and enhanced security.

Apache Pulsar

Released by Yahoo! in 2014, Apache Pulsar presents a different architectural approach than Apache Kafka by separating its data broker and storage (BookKeeper) layers. This separation allows Pulsar to scale each layer independently, providing flexibility in resource allocation. Pulsar shares a similar issue with Kafka’s initial architecture, as the added complexity of multiple distributed systems can result in latency divergence, especially compared to optimized alternatives.

While Pulsar’s ability to scale independently is a potential draw, its advantages have been diminished by the advent of Kafka’s tiered storage, which provides similar cost-effective scaling capabilities. Moreover, Pulsar has yet to reach the same level of community engagement as Kafka; There are relatively few tailored tools and debugging resources available for Pulsar.

Confluent

Confluent was founded in 2014 by the co-creators of Apache Kafka, who aimed to commercialize the Kafka framework by offering managed solutions, similar to the origins of Databricks and Apache Spark. In 2021, Confluent went public with a valuation of $828 million, and as of August 2025, it has a market cap of $6.31 billion.

Given that Confluent’s core engine, Kora, is built on Kafka, it shares many of Kafka’s limitations, including the resource intensity associated with the KRaft consensus mechanism. However, Confluent also benefits from a larger open-source community and broad market adoption; both of which provide sources for new features and innovations. A 2024 report commissioned by Redpanda confirmed Redpanda’s claim of a 60% cost-saving potential and a performance boost over Confluent. However, differentials between the two are constantly being matched, e.g., Confluent acquired WarpStream in September 2024 to integrate the BYOC capabilities into their platform.

Cloud-native PaaS and SaaS Solutions from Major Cloud Providers

Major cloud providers such as Google, AWS, and Azure offer their own managed Kafka solutions, branded as Google Cloud Managed Services, AWS MSK, and Azure Events Hub. These cloud-native offerings are based on Kafka’s open-source code but include additional features for integration, control, and architecture.

While cloud-based services can be beneficial for specific workloads, fully managed solutions are not offered by these providers. As of August 2025, Clients are still responsible for deploying and monitoring brokers and configuring connectors. Additionally, these data streamers generally lack tiered storage capabilities and do not provide the same uptime SLAs as dedicated operators. Customers using these services may experience challenges due to vendor lock-in, especially those operating in multi-cloud environments where flexibility and independence are highly valued.

Competitors Without a Core Data Streaming Focus

Aiven: Aiven is a cloud database infrastructure company that caters to AI companies, which also offers a semi-managed cloud-based Kafka product. Aiven, founded in 2016, held a successful Series D round in 2022, raising $210 million at an undisclosed valuation. The most recent valuation disclosure came in late 2021, following their Series C extension, when Aiven was valued at $2 billion and was growing revenue at over 100% per year.

Cloudera: Cloudera is a competitor to Redpanda in the data streaming space, although its focus on data streaming is secondary to its broader data platform business models, which center primarily on data-at-rest. Cloudera provides Kafka and Flink frameworks as part of its data ecosystem. Cloudera was established in 2008 and was taken private by CD&R in 2021 for $5.3 billion. Both companies tend to position their data streaming as a complementary add-on and do not emphasize data streaming as a core offering of their platforms, and generally lack differentiable features and innovations compared to dedicated providers.

Redpanda generates revenue through a range of product tiers, catering to businesses of different sizes and needs. These offerings range from a pay-as-you-go model for smaller-scale or personal project usage to enterprise contract solutions for customers demanding highly performant data streaming. While Redpanda offers a business source license, it comes with strict limitations on commercial deployment.

Source: Redpanda

Redpanda offers four distinct deployment models, allowing users to customize their environment and throughput according to their needs.

1. Serverless deployment leaves Redpanda to manage both the infrastructure and scaling. At this level, throughput options are restricted to 100 MB/s, making it a better fit for small teams or individual projects.

2. The Dedicated deployment model offers the user dedicated clusters, improving performance and reliability compared to the serverless tier. Redpanda still manages the infrastructure, but the additional resources and higher throughput limit make it ideal for small to medium-enterprise use cases.

3. BYOC, previously one of Redpanda’s USPs, allows enterprises to deploy Redpanda’s software within their cloud environment, while Redpanda only handles the software management rather than the infrastructure. It ensures data sovereignty, and the 2GB/s maximum throughput makes it a strong consideration for large-scale sensitive workloads.

4. The Self-managed tier provides companies with the most flexibility and gives customers full control over both software and infrastructure. The platform’s performance is only constrained by the client’s hardware and budget, making it the preferred choice for those requiring more tailored scalability.

The pricing for each tier varies significantly based on the deployment model and any further customizations. The more basic tiers for smaller teams can cost thousands of dollars per year, while the large-scale, fully customized deployments can reach millions of dollars annually.

Following its June 2023 Series C funding round, CEO Alex Gallego revealed that Redpanda had quintupled its revenue over 2023, and went on to say it “[hasn’t] lost [a] deal in seven months against all the competitors,” speaking to the market fit of Redpanda’s product and the company’s ability to effectively compete with the market incumbents Apache Kafka and Confluent.

The company’s enterprise and cloud customer base saw a 160% increase in 2023, with key additions including Alpaca, and Moody’s. This growth trajectory accelerated into 2024, with Redpanda reporting a 300% revenue increase and a 179% expansion in its customer base, driven by the signing of major enterprise clients such as Vodafone, Texas Instruments, and Cisco.

The open-source community has also shown strong engagement with Redpanda, slightly eroding Kafka’s competitive edge, with Redpanda’s Community Edition reaching 10.1K GitHub stars. Employee growth has kept pace with these business expansions; from a small team of 22 employees in 2021, Redpanda has scaled to nearly 200 employees in 2025.

Alongside organic growth, Redpanda has conducted a series of strategic acquisitions to complement its product offering. In 2022, Redpanda acquired CloudHut, a company that specializes in UI for DSPs. The integration enabled Redpanda to develop a more interactive and user-friendly UI to improve visibility over topics and consumer groups, as well as real-time data exploration.

Source: Redpanda

In 2024, continuing its expansion into stream processing, Redpanda acquired Benthos, an open-source stream processing platform known for Benthos Studio, a low-code dashboard to build data pipelines via flowcharts. This feature was rebranded and integrated as Redpanda Connect, and as of August 2025, pre-packages Redpanda with 303 connectors, more than the 244 provided by the Confluent Hub.

Redpanda has also formed key partnerships with complementary companies. For instance, Redpanda integrated with cybersecurity firm Ockam to implement a zero-trust architecture and proper functionality of their BYOC deployment model. Redpanda has also partnered with Mexora, a company with a similar offering to Benthos, a no-code platform for building data pipelines and conducting simple transformations.

Redpanda raised $100 million in its Series D funding round in April 2025 at a valuation of $1 billion. This followed a $100 million Series C in mid-2023 and a $50 million Series B in 2022. With a total $265.5 million raised as of August 2025, Redpanda is backed by funds including Lightspeed Venture Partners, Google Ventures, and Haystack VC. As of April 2025, Confluent, the leading publicly traded data streaming company, trades at a 6.39x revenue multiple. However, given the growth rates and company size, Redpanda’s implied valuation multiple could be significantly higher.

Reports emerged in January 2025 that Redpanda and Snowflake were engaged in discussions regarding a potential acquisition. The deal, if finalized, would have valued Redpanda at approximately $1.5 billion. Such an acquisition would align with attempts to unify the data at rest and data in motion offerings to improve interoperability and reduce any possibility of data silos forming.

AI/ML as a Key Growth Driver in Data Streaming

AI and ML are becoming the primary drivers of growth for data streaming, with 75% of users identifying it as the biggest catalyst as fo 2024. While ML models trained on historical data provide valuable insights, their utility can be limited without real-time updates. The ultimate goal of many AI/ML end-users is to develop models that continuously refine their parameters based on live data, ensuring relevance and accuracy.

AI use cases as of March 2025 can be broadly categorized into predictive AI and generative AI. Predictive AI is instrumental in applications such as dynamic pricing, autonomous vehicle navigation, real-time healthcare analytics, and predictive maintenance. These implementations offer clear benefits in cost efficiency and top-line growth, and require infrastructure that ensures high-quality data ingestion and provides clear audit trails for model inputs to meet traceability requirements.

Generative AI, on the other hand, powers applications like interactive chatbots and targeted advertising. Reinforcement learning, particularly through engagement events, has been able to provide the models with malleability to changes in customer behavior and incorporate these learnings to drive deeper engagement. For chat models, advertising, and recommendations, RAG-enabled models (Retrieval augmented generation) are emerging as the preferred approach, utilizing DSPs to transfer and process event data directly from the sources for immediate ingestion.

Feature Expansion

Given the importance of data veracity and suitability for AI/ML applications, more complex real-time transformations are becoming essential to guarantee accurate, auditable, and regulation-compliant models. A Redpanda survey conducted in late 2023 identified security, data privacy, and system complexity as the primary technical challenges faced by both current and prospective customers.

Legal frameworks like the GDPR and the EU AI Act have enforced stricter model traceability requirements. Redpanda’s tiered storage approach and unlimited data retention may address these challenges better than competitors. Sovereign AI, a new Redpanda feature released as a demo after its annual Streamfest event in early 2025, allows organisations to run open-source model inference locally. Private inference is an attractive option for companies concerned about sensitive information leaving their private network, also tracking data lineage for model input transparency, and the platform itself is secure by design with authentication, access privileges, and audit logs.

The Unification of Batch and Stream Processing

Traditionally, companies have relied on separate technology stacks for data streaming and batch processing. However, the industry is increasingly moving toward unified solutions, as evidenced by partnerships such as Redpanda with Snowflake and Confluent with Databricks.

Where data streaming facilitates real-time analytics, monitoring, live event/transaction processing, and model inference, batch processing remains crucial for historical data analysis, business intelligence, and long-term storage. The growing preference for simplicity suggests an advantage for platforms that can seamlessly integrate data warehouses, lakes, and streaming services into a single ecosystem.

Redpanda One (R1) Engine with the Apache Iceberg Integration

During its 2025 Streamfest event, Redpanda also announced a new engine, Redpanda One (R1), which features integration of Apache Iceberg, an open-source table format used in data lakes. This supports Redpanda’s dynamic schema evolution and provides an opportunity to synchronize how data is stored across streaming and batch processing events, eliminating data silos and improving retrospective analysis by maintaining a single, queryable copy of the data across all platforms.

Source: Redpanda

R1 introduces enhanced flexibility in data storage, giving developers the ability to balance availability, consistency, latency, safety, and networking costs on a per-topic basis. Additionally, the concept of cloud topics eliminates the former intermediary step of storing data on SSDs or NVMe before offloading it to object storage. Instead, only metadata is stored locally, while the actual data resides in cloud storage. This innovation reduces infrastructure costs while maintaining efficient data retrieval capabilities.

The Stickiness of Kafka and Competition from Confluent

Apache Kafka has long been the industry standard for stream processing, creating a significant barrier to entry for new players like Redpanda. The deep roots of the Apache ecosystem mean new offerings must demonstrate tangible improvements to justify the transition. Beyond current distribution, Kadka benefits from an active and large community. Although Redpanda has its own team dedicated to deploying tailored solutions and responding to customer queries, along with several tutorials, Kafka has extensive online learning resources offering customers reassurance in their ability to deploy and optimize the software more autonomously.

Additionally, Redpanda faces strong competition from Confluent. Confluent has demonstrated its ability to rapidly replicate new features, as seen with its BYOC offering in response to Redpanda. Confluent’s combination of brand awareness, association with pioneering streaming, and transparent strategy of enabling Agentic AI workflows makes it difficult for Redpanda to differentiate itself on performance alone, particularly as Confluent continues to close the gap in feature parity.

Cloud-Native Streaming and the Expansion of Data Lake Providers

The increasing adoption of AI/ML for enterprise use cases is driving demand for unified data platforms that integrate data lakes and real-time streaming, eliminating the need for multiple vendors. While Redpanda is optimized for high-performance streaming, it lacks the scale to rapidly expand into adjacent capabilities, meaning it may need to rely on partnerships to compete with larger, full-stack offerings.

Moreover, the open-source nature of Kafka lowers the entry expertise required for cloud providers and data lake incumbents like Snowflake and Databricks, should they decide to prioritize real-time data streaming, particularly to simplify real-time data ingestion. Redpanda’s unique C++ architecture should provide some protection, but it still faces the challenges of compatibility and ease of integration compared to native solutions packaged with these larger integrated platforms.

In the current period, as companies recognize their potential underutilization of data and aim to facilitate more engaging customer interactions and automated event responses, real-time data-streaming is increasingly becoming part of the conventional data stack. As of 2025, Redpanda continues to enable companies to develop data pipelines with a competitive price and performance profile. Trends in agentic workloads will provide strong tailwinds for the industry as data streamers can transfer data immediately upon generation, ensuring model responses and parameters are continuously refined. Within this space, data privacy laws may continue to evolve, warranting Redpanda’s focus on designing software for data sovereignty.

The immediate challenges for Redpanda are its current size and ability to communicate to prospective customers the benefits of switching away from a Kafka-based service. Marketing for customer acquisition, partnerships with enterprise customers, and matching competitor dynamics are Redpanda’s priorities as of August 2025. Redpanda’s ability to respond to customer demands for the erosion of existing data silos will be one key factor in determining its future trajectory.