Course outline for kafka, MS SQL and deployments technologies

Course outline for kafka, MS SQL and deployments technologies

This is a course that was designed to enable the learner to understand, manage, install, operate and develop on a real life deployable solution of systems using Kafka, MS SQL 2019, Unix-like operating system, docker, Java, python and other deployment and packaging solutions like kubernetes.

Generally, learners have individual skillsets of each of the adobe mentioned technologies. However, having the power to use them in synergy to deploy real-world solutions is something that is rare.

At the end of this course, the learner shall be able to:

• Have holistic skills for real world system deployment
• Use individual technologies in systematic synchrony
• Use MS SQL 2019 to its fullest
• Understand and use the newest features offered mu MS SQL in 2019
• Install and operate MS SQL in unix-like operating systems
• Use Kafka in unix-like systems
• Understand Kafka in depth
• Program in Kafka using Java
• Program in Kafka using python
• Deploy the technologies using Docker
• Deploy the technologies using Kubernetes

This is a very intensive course and is designed for advanced learners. The basic requirements are:

• Operational understanding of SQL
• Some usage experience in MS SQL
• Either Java or python

In order to participate in this course, the learner needs to have access to actual unix-like remote systems, because in real-life scenarios, deployments are done on remorse servers. As such, they will require:

• Access to workstation (of any OS)
• Access to remorse server (one per student)

We shall provide individual access to said servers. The students shall remote into them from their workstations during the training process.

The actual course outline shall be as follows:

Day 1: Faysal and Talat:

1. Introduction.
2. Overview of Docker, New features in MSSQL-2019 and Azure Data Studio.
3. Docker basics, installation and setup.
   1. Installation on Windows, Mac, Linux.
   2. Images.
   3. Containers.
   4. Processes.
   5. Ports.
   6. Volumes.
   7. Basic commands for creating, pulling images; starting, stopping Containers etc.
4. Docker demonstration.
   1. Pull a php-apache image.
   2. Run the php-apache image in a container.
   3. Port mapping.
   4. Volume mapping.
   5. Data persistence.
   6. Run Nginx via Docker.
5. Exercises.

Day 2: Talat:

1. MSSQL-2019 Deployment with Docker.
   1. Pull linux based mssql image.
   2. Create a container and run the mssql image.
   3. Install Azure Data Studio.
   4. Create tables and make queries.
2. MSSQL basics quick review.
   1. Overview of fundamental commands.
   2. Making databases and tables with Azure Data Studio.
   3. Stored Procedures
   4. T-SQL.
3. New powerful features in MSSQL -2019.
   1. Kubernetes introduction.
   2. Big Data introduction.
   3. Apache Spark introduction.
4. Exercises.

Day 3: Talat:

1. Azure Data Studio.
   1. Extensions.
   2. Notebooks.
   3. Data Visualization.
   4. Python Integration.
   5. Query History.
   6. Machine Learning.
   7. Data portability and migration.
2. Overview of technologies related to MSSQL and way forward 2019, like: Spark, Docker, etc.
3. Exercises and Evaluation.

Day 1: Faysal Aziz

1. Kafka Theory
   1. Broker
   2. Cluster
   3. Zookeeper
   4. Ensemble
   5. Multiple clusters
   6. Ports
   7. Topics
   8. Message structure
   9. Partitions and distribution
   10. Producer and Consumer architecture
2. Kafka deployment
   1. Operating Ubuntu Server
   2. Setting up IDE to work remotely with ubuntu from Windows OS
   3. Installing required environments
   4. INstalling kafka from scala binaries
3. Using kafka to send and receive messages
   1. Send messages using the console
   2. Consume messages using the console
   3. Running multiple consumers and multiple producers
4. Using Kafka with multiple partitions
   1. Remote brokers
   2. Creating multiple partitions
   3. Accessing specific partitions
   4. Accessing specific offsets
   5. Exploring and understanding the filing system of messages

Day 2: Faysal Aziz

1. Kafka Cluster with multiple brokers
   1. Configurations and settings
   2. Launching and managing
   3. Using Zookeeper directly for info access on brokers
   4. Multiple partition topics in clusters
   5. Logs
   6. Producing and Consuming inter-cluster
   7. Simulating break-downs
2. Multiple brokers with Replication
   1. Creation and launch
   2. Actual replication
   3. Logs and details exploration
   4. Production and consumption
   5. Storage management
   6. Break-down simulation
   7. Manage and recover from break-downs
3. Consumer Groups
   1. Using the default consumer groups
   2. Multiple consumer groups
   3. Idle consumer groups
4. Assessment

Day 3: Faysal Aziz

1. Performance and Testing
   1. Basic performance test
   2. Tweaking the parameters
   3. Consumer performance
   4. Nonzero LAG values for consumers
2. Programming in Kafka
   1. Java
      1. Setup and configuration
      2. Starting Cluster and Producer
      3. Controlling Producers
      4. Serializers
      5. Auto-committing and refactoring code
      6. Partition assignments
      7. Multiple producers
   2. Python
      1. Producers in python
      2. Producers in python
      3. Hands-on development
3. Assessment

Resources:

1. Azure DB Stored Procedures etc. >>>