Apache Flink Gets Major Boosts as Apache Offers New Developer Tools, Enterprise-Ready Security

Confluent, Inc.,the data streaming pioneer, introduced new capabilities to Confluent Cloud to make stream processing and data streaming more accessible and secure. Confluent’s new support of Table API makes Apache Flink® available to Java and Python developers, while its private networking for Apache Flink provides enterprise-level protection for use cases with sensitive data. Additionally, Confluent Extension for Visual Studio Code accelerates the development of real-time use cases while Client-Side Field Level Encryption encrypts sensitive data for stronger security and privacy.

“The true strength of using Apache Flink for stream processing empowers developers to create applications that instantly analyse and respond to real-time data, significantly enhancing responsiveness and user experience,” said Stewart Bond, Research Vice President at IDC. “Managed Apache Flink solutions can eliminate the complexities of infrastructure management while saving time and resources. Businesses must look for a Flink solution that seamlessly integrates with the tools, programming languages, and data formats they’re already using for easy implementation into business workflows.”

More businesses are relying on stream processing to build real-time applications and pipelines for various use cases spanning Machine Learning, predictive maintenance, personalised recommendations, and fraud detection. Stream processing lets organisations blend and enrich their data with information across their business. Apache Flink is the de facto standard for stream processing. However, many teams hit roadblocks with Flink because it is operationally complex, difficult to secure, and has expensive infrastructure and management costs.

“Thousands of teams worldwide use Apache Flink as their trusted steam processing solution to deliver exceptional customer experiences and streamline operations by shifting processing closer to the source, where data is fresh and clean,” said Shaun Clowes, Chief Product Officer at Confluent. “Our latest innovations push the boundaries further, making it easier for developers of all skill levels to harness this powerful technology for even more mission-critical and complex use cases.”

Confluent’s Support for Table API Extends Apache Flink to Teams with Java or Python Experience

Confluent Cloud for Apache Flink offers the SQL API, a powerful and user-friendly tool for processing data streams. While Flink SQL is effective for quickly writing and executing queries, some teams favour programming languages like Java or Python, which allow for more control of their applications and data. This can be especially important when developing complex business logic or custom processing tasks.

Adding support for the Table API to Confluent Cloud for Apache Flink enables Java or Python developers to easily create streaming applications using familiar tools. By supporting both Flink SQL and the Table API, Confluent Cloud for Apache Flink lets developers choose the best language for their use cases.

Support for Table API enables companies to:

• Enhance language flexibility by enabling developers to use their preferred programming languages, taking advantage of language-specific features and custom operations.
• Streamline the coding process by leveraging customers’ integrated development environment (IDE) of choice featuring auto-completion, refactoring tools, and compile-time checks to ensure higher code quality and minimise runtime issues.
• Make debugging easier with an iterative approach to data processing and streamlined CI/CD integration.

“We’re reimagining financial services and it’s imperative that we adopt new technologies that not only protect our customers but are also easily used by our teams,” said Shujahat Bashir, Director of Software Engineering at Thrivent. “Data streaming and stream processing can help power real-time fraud detection, and payments processing for an exceptional customer experience. We are looking forward to using Confluent’s new Apache Flink features including Private Networking, Flexible Schema Management, and Table APIs to extend the power of Flink to the languages and development styles we already use today.”

Support for Table API is available in open preview and is available for testing and experimentation purposes. General availability is coming soon.

Enabling Private Networking for Flink Provides a Secure Environment for Data Streaming Workloads

With more data than ever before and more teams using cloud and hybrid solutions, private networking is essential for protecting against unauthorised access and cyber threats. Confluent Cloud now offers private networking support for Flink, providing a critical layer of security for businesses that need to process data within strict regulatory environments.

By enabling private networking for Flink, Confluent users can:

• Boost data security and privacy between Apache Flink and Kafka by safeguarding in-transit data and ensuring secure connections between clients and Flink within a private network.
• Simplify secure network configuration, making it easier to set up private connections without requiring extensive networking expertise.
• Facilitate flexible and secure stream processing by seamlessly joining and processing data across different Kafka clusters, ensuring data accessibility while adhering to strict security protocols. Private networking support is generally available on AWS for Enterprise and Dedicated clusters. Additional cloud platforms are coming soon.

Confluent Extension for Visual Studio Code Streamlines Workloads and Accelerates Development Cycles

Teams working with real-time data platforms like open source Apache Kafka often struggle with fragmented tools, clunky workflows, and constant switching between environments and other interfaces. This disjointed experience makes it tough to integrate real-time data into applications, slowing down productivity and innovation. Integrated development environments can make this process much easier and 54% of developers globally use Visual Studio Code (VS Code), the most popular choice for code editing. Confluent Extension for Visual Studio Code simplifies the development process by integrating Confluent directly into teams’ preferred integrated development environment.

Confluent Extension for Visual Studio Code enables teams to:

• Streamline topic management to easily create, edit, and browse Kafka topics with intuitive tools that simplify debugging and boost efficiency.
• Code and debug in one place by writing, executing, and debugging Kafka clients, Flink queries, and streaming applications directly in VS Code with enhanced productivity features like code completion.
• Seamlessly manage cloud resources to provision and control Confluent Cloud clusters within VS Code, reducing complexity and streamlining cloud operations.

Confluent Extension for Visual Studio Code supports Kafka clients and is available in early access. New Apache Flink capabilities and general availability will be available later in 2024.

Client-Side Field Level Encryption Safeguards Teams’ Most Sensitive Data While Meeting Regulatory Requirements

Data security and privacy are top priorities for organisations in regulated industries, including financial services, healthcare, and the public sector. These organisations are subject to compliance rules about how sensitive data, like personally identifiable information (PII), can be accessed, moved, and stored. Client-Side Field Level Encryption helps teams protect their most sensitive data by enabling them to encrypt individual fields within their data streams for enhanced security and compliance.

Complementing existing Confluent security features, Client-Side Field Level Encryption further reduces the risk of unwanted access by encrypting data on the client side so that even system admins and users with highly privileged access cannot view messages in plaintext.

With Client-Side Field Level Encryption teams can:

• Improve security of sensitive data and adhere to strict compliance requirements.
• Maintain flexible and granular access control of which specific fields to encrypt.
• Lower total cost of ownership and operational complexity by reducing the need for topic duplication.

Client-Side Field Level Encryption is currently in limited availability and is expected to become generally available soon for all Confluent Cloud users through the company’s Stream Governance suite.