Real-Time Data Processing with Apache Flink and Iceberg

Harness the Power of Stream Processing and Advanced Table Formats for Modern Data Architectures in 2 days

Data doesn't sleep, and the companies that win don't either. Apache Flink has changed the game by letting you process massive streams of data the moment it arrives—whether it's flowing non-stop or coming in batches. It's fast, it's reliable, and it's what the most innovative teams are using to stay ahead.

Enter Apache Iceberg: the missing piece that brings order to data lake chaos. It keeps your massive datasets consistent and manageable, even when you're working at enterprise scale.

Our two-day workshop cuts through the hype and gets your hands dirty with both technologies. You'll walk away knowing how to build actual real-time data systems that solve real problems—the kind that make executives notice and competitors worry.

Learning Outcomes:

By the end of this course, participants will be able to:

Comprehend the core concepts and architecture of Apache Flink and its role in stream processing.
Develop and deploy data processing pipelines using Flink's DataStream API.
Implement stateful stream processing and manage event time semantics effectively.
Integrate Apache Flink with Apache Iceberg to manage large analytic tables efficiently.
Utilize Flink SQL for querying and manipulating data within Iceberg tables.
Apply best practices for building resilient and scalable real-time data applications.

Prerequisites:

Participants should have:

Proficiency in Java or Scala programming languages.
Fundamental understanding of distributed systems and parallel processing concepts.
Basic knowledge of data processing frameworks such as Apache Hadoop or Apache Spark.
Familiarity with SQL and relational database concepts.

Detailed Course Outline:

Introduction to Stream Processing and Apache Flink
1. The Evolution of Data Processing: From Batch to Stream
2. Key Concepts in Stream Processing
3. Overview of Apache Flink
  1. Flink's Position in the Big Data Ecosystem
  2. Core Features and Capabilities
4. Use Cases and Industry Applications
Apache Flink Architecture and Runtime
1. Flink's Distributed Architecture
  1. JobManager and TaskManager Roles
  2. Execution Model and Dataflow
2. Fault Tolerance Mechanisms
  1. Checkpointing and State Snapshots
  2. Exactly-Once Semantics
3. Deployment Modes
  1. Standalone Cluster
  2. YARN and Kubernetes Integrations
Getting Started with Flink's DataStream API
1. Setting Up the Development Environment
2. Building a Simple Flink Application
  1. Defining Data Sources and Sinks
  2. Applying Transformations
3. Understanding Streams and Transformations
  1. Stateless vs. Stateful Transformations
  2. Keyed Streams and Partitioning
Stateful Stream Processing
1. The Importance of State in Stream Processing
2. Managing Operator State and Keyed State
3. Implementing Process Functions
  1. Timers and Event Time Processing
4. State Backends and Their Configurations
  1. Memory State Backend
  2. RocksDB State Backend
Time and Windowing in Flink
1. Understanding Event Time vs. Processing Time
2. Generating and Assigning Timestamps
3. Watermarks and Their Role in Event Time Processing
4. Window Operators and Functions
  1. Tumbling Windows
  2. Sliding Windows
  3. Session Windows
5. Late Data Handling Strategies
Integrating Apache Flink with Apache Iceberg
1. Introduction to Apache Iceberg
  1. Motivation and Key Features
  2. Comparison with Traditional Table Formats
2. Setting Up Iceberg with Flink
  1. Configuring Iceberg Catalogs
  2. Creating and Managing Iceberg Tables
3. Performing Data Operations
  1. Inserting and Deleting Records
  2. Schema Evolution and Partitioning
4. Querying Iceberg Tables with Flink SQL
  1. Writing and Executing SQL Queries
  2. Optimizing Query Performance

Practical, connected learning

My wider training approach brings hands-on implementation and systems thinking together, connecting technology with real operational needs.

Real-Time Data Processing with Apache Flink and Iceberg