Course outline for kafka for beginners

Course outline for kafka for beginners

This is a course that was designed to enable the learner to understand, manage, install, operate and develop on a real life deployable solution of systems using Kafka in a Unix-like operating system.

Course Objective

At the end of this course, the learner shall be able to:

• Have holistic skills for real world system deployment
• Install and operate MS SQL in unix-like operating systems
• Use Kafka in unix-like systems
• Understand Kafka in depth
• Program in Kafka using Java
• Program in Kafka using python
• Troubleshoot common kafka issues
• Understand and configure zookeeper

Prerequisite

This is a very intensive course and is designed for advanced learners. The basic requirements are:

• Understanding of how file systems work
• Programming ability in either Java or python
• Good command of the English language

In order to participate in this course, the learner needs to have access to actual unix-like remote systems, because in real-life scenarios, deployments are done on remorse servers. As such, they will require:

• Access to workstation (of any OS)
• Access to remote server (one per student)

We shall provide individual access to said servers. The students shall remote into them from their workstations during the training process.

The actual course outline shall be as follows:

Day 1:

1. Kafka Theory
   1. Broker
   2. Cluster
   3. Zookeeper
   4. Ensemble
   5. Multiple clusters
   6. Ports
   7. Topics
   8. Message structure
   9. Partitions and distribution
   10. Producer and Consumer architecture
2. Kafka deployment
   1. Operating Ubuntu Server
   2. Setting up IDE to work remotely with ubuntu from Windows OS
   3. Installing required environments
   4. INstalling kafka from scala binaries
3. Using kafka to send and receive messages
   1. Send messages using the console
   2. Consume messages using the console
   3. Running multiple consumers and multiple producers
4. Using Kafka with multiple partitions
   1. Remote brokers
   2. Creating multiple partitions
   3. Accessing specific partitions
   4. Accessing specific offsets
   5. Exploring and understanding the filing system of messages

Day 2:

1. Kafka Cluster with multiple brokers
   1. Configurations and settings
   2. Launching and managing
   3. Using Zookeeper directly for info access on brokers
   4. Multiple partition topics in clusters
   5. Logs
   6. Producing and Consuming inter-cluster
   7. Simulating break-downs
2. Multiple brokers with Replication
   1. Creation and launch
   2. Actual replication
   3. Logs and details exploration
   4. Production and consumption
   5. Storage management
   6. Break-down simulation
   7. Manage and recover from break-downs
3. Consumer Groups
   1. Using the default consumer groups
   2. Multiple consumer groups
   3. Idle consumer groups
4. Assessment

Day 3

1. Performance and Testing
   1. Basic performance test
   2. Tweaking the parameters
   3. Consumer performance
   4. Nonzero LAG values for consumers
2. Programming in Kafka
   1. Java
      1. Setup and configuration
      2. Starting Cluster and Producer
      3. Controlling Producers
      4. Serializers
      5. Auto-committing and refactoring code
      6. Partition assignments
      7. Multiple producers
   2. Python
      1. Producers in python
      2. Producers in python
      3. Hands-on development
3. Assessment