Managing Big Data with Spark - A Comprehensive Beginners’ Course

Managing Big Data with Spark - A Comprehensive Beginners’ Course

This is a comprehensive course designed for learners without any prior big data or programming knowledge. The course is designed to take the learner through a journey of understanding and comprehending the prerequisites for Apache Spark before delving into Spark itself.

It begins with the concept of Big Data and then slowly moves on to the fundamentals of programming. The language of choice here is python. Along the way, the learner will also be equipped with enough knowledge to use SQL in a Spark and non-Spark environment.

The core strength of Spark is its ability to perform complex in-memory analytics and stream data sizing up to petabytes, making it more efficient and faster than MapReduce. Spark can pull the data from any data store running on Hadoop and perform complex analytics in-memory and in parallel. This capability reduces Disk I/O and network contention, making it ten times or even a hundred times faster. Also, data analytics frameworks in Spark can be built using Java, Scala, Python, R, or even SQLs.

This course will enable the learner to have skillsets in both SQL and python. Since Spark supports both (and other languages), the learner can choose which methodology best suits him / her. One may also use them both in conjunction, making Spark very easy to use and versatile.

Learning Outcome

After the successful participation of this course, the learner may be expected to have skill sets that include:

• Understanding pf RDBMS
• SQL Syntax
• T-SQL Syntax
• Functions
• Procedures
• Data Cleaning
• Understand the basic syntax structures
• Comprehend real world problems into logical steps
• Formulate Algorithms
• Formulate Pseudo codes
• Code in python
• Develop systems in both functional and OOP
• Basic python for analytics
• Apache Spark Foundation and Spark Architecture
• Data Engineering and Data Processing in Spark
• Data Sources and Sinks
• Data Frames and Spark SQL
• Understand how Hadoop clusters are managed by YARN for Spark
• Consume streaming data using Spark

Prerequisites

1. Ability to use a computer
2. Ability to use the internet
3. Manage files and understand basic file systems in the computer
4. Ability to communicate in the English language
5. Dual Screen
6. High Speed Internet
7. Webcam and Microphone
8. PowerShell [must be pre-installed]
9. Filezilla [must be pre-installed]
10. Access to external IP address

Course Outline

The duration of the course is 7 days. Each day will cover 7 hours or training (inclusive of two 15 minute breaks).

1. Introduction
   1. Big Data
   2. Tools
   3. Structure
   4. Pipeline and flow
   5. Data Source
2. Structured Data
   1. RDBMS
   2. SQL
      1. Select
      2. Where
      3. Like
      4. Order
      5. Insert
      6. Update
      7. Delete
      8. IN Operator
      9. Between
      10. Aggregate
      11. Group
      12. Alter
      13. Sub queries
      14. Stored Procedures
      15. Functions
3. Unstructured and Semi Structured Data
   1. Datetime
   2. JSON
   3. XML
4. Python
   1. Basics
      1. IDEs
      2. Environment Variables
      3. Python Documentation
      4. Hello world with python
      5. Modes of Programming
   2. Variables & Collections
      1. Numbers
      2. Python Lists
      3. Python Tuples
      4. Python Dictionaries
      5. Python Sets
      6. Copying
      7. Python Strings
      8. String formatting
      9. Regular Expressions
   3. Operators
      1. Arithmetic Operators
      2. Comparison (Relational) Operators
      3. Assignment Operators
      4. Logical Operators
      5. Membership Operators
      6. Operators Precedence
   4. Decisions and Loops
      1. Decision Making
      2. The if Statement
      3. The if else Statement
      4. For Loop
      5. While Loop
      6. Break And Continue
   5. Functions and Subroutines
      1. Defining Your Own Functions
      2. Parameters
      3. Function Documentation
      4. Passing Collections to a Function
      5. Variable Number of Arguments
      6. Scope
      7. Map
      8. Filter
      9. Lambda
   6. Data Handling
      1. Pandas
      2. Recursion
      3. File Access
5. Spark
   1. Introduction
   2. Execution Model
   3. Dataframes
   4. Transformation
   5. Jobs
   6. Stages
   7. Structured API
   8. RDD
   9. Data Sinks
   10. Reading
   11. Writing
   12. Schema
   13. SQL in Spark
   14. Unstructured Data
   15. Advanced Transformations
   16. User Defined Functions
   17. Aggregations
   18. Grouping
   19. Joins
   20. Shuffles
   21. Optimization
   22. Clustering
   23. Machine Learning with Spark