Analytics with Python and Azure Databricks

Module 1 — Programming Refresher

• Introduction to Python
  • Syntax, data types, variables
  • Control structures: loops, conditionals
  • Math operations and functions
• Functions in Python
  • Defining and calling functions
  • Scope and lifetime, arguments, and return values
• Basic I/O Operations
  • Reading from and writing to files
  • Handling different file formats (CSV, JSON)
• Error and Exception Handling
  • Try, except, finally blocks
  • Custom exceptions for robust error management

Module 2 — Reusable Python and NumPy

• Object-Oriented Programming (OOP)
  • Classes, objects, inheritance, polymorphism
  • Encapsulation, abstraction, special methods
• Data Handling with NumPy
  • Array operations, indexing, slicing
  • Basic linear algebra and statistical operations
• Data Manipulation
  • Recursion
  • Data flattening
  • Data Conversion

Module 3 — pandas Data Workflows

• Data Analysis with pandas
  • DataFrame and Series, data wrangling
  • Advanced operations: merging, joining, and concatenating

Module 4 — Visualisation and a Flask Demonstration

• Data Visualization with Matplotlib
  • Create and refine selected plots; compare appropriate chart types.
  • Improve labels, scales and presentation for communicating analytical findings.
• Introduction to Flask for Web Applications
  • Setting up a Flask environment
  • Routing, templates, and form handling

Module 5 — Files, Debugging and Testing

• Working with the OS and File System
  • File system navigation, path manipulation
  • Reading and writing files, directory management

• Debug a selected data-transformation function and write meaningful checks for expected outputs and error handling.
• Apply object-oriented organisation where it makes a small analytics workflow clearer rather than adding unnecessary complexity.

Module 6 — Workspace and Batch Processing

• Data Processing with PySpark
  • Use a prepared Azure Databricks workspace; compare serverless and classic compute, permissions and relevant runtime constraints.
  • DataFrame operations, Spark SQL
• Debugging and Testing
  • Running test scenarios
  • Use supported notebook commands and inspect execution results.

• Data Processing with Spark on Azure Databricks
  • Transformations, joins and an introductory discussion of partitioning and clustering.
  • Advanced data analysis: aggregation, window functions

Module 7 — Streaming Concepts and Advanced Demonstrations

• Fundamentals of Stream Processing
  • Stream vs. batch processing
  • Processing modes, event-time concepts and checkpointing in a selected demonstration.

• Compare joins and data-layout concepts such as bucketing; verify support for the chosen runtime and data format.
• Structured Streaming basics using DataFrame operations.
• Discuss multiple data sources and integration constraints.

Module 8 — Machine-Learning and Integration Overview

• Machine Learning with PySpark
  • Introduction to MLlib
  • Demonstrate a small model example and basic evaluation.
• Overview of Spark ML pipelines.
• Kafka integration concepts and a prepared demonstration where the environment supports it.

• Review a small analytics workflow and explain remaining engineering, data-quality and operational requirements.
• Discuss runtime and compute compatibility before applying demonstrated features to a different workspace.