Course Outline for Spark

Part 1:

Learning outcome:

1. Install and run Apache Spark on a desktop computer or on a cluster
2. Frame big data analysis problems as Spark problems
3. Implement iterative algorithms such as breadth-first-search using Spark
4. Use DataFrames and Structured Streaming in Spark 3
5. Use Spark's Resilient Distributed Datasets to process and analyze large data sets across many CPU's
6. Understand how Spark Streaming lets your process continuous streams of data in real time
7. Share information between nodes on a Spark cluster using broadcast variables and accumulators
8. Analyze non-relational data using HBase, Cassandra, and MongoDB
9. Publish data to your Hadoop cluster using Kafka, Sqoop, and Flume
10. Use Pig and Spark to create scripts to process data on a Hadoop cluster in more complex ways.
11. Analyze relational data using Hive and MySQL
12. Understand how Hadoop clusters are managed by YARN, Tez, Mesos, Zookeeper, Zeppelin, Hue, and Oozie (optional)
13. Consume streaming data using Spark Streaming, Flink, and Storm (optional)

Course Content:

Day 1:

1. Spark basics and RDD interface
2. Hands in installation and setup
3. Using libraries and resources
4. Exercises
5. Assessment

Day 2:

1. SparkSQL, DataFrames and Datasets
2. Advanced Spark
3. Accumulators and BFS in Spark

Day 3

1. Clustering
2. Machine Learning with Spark
3. Streaming
4. GraphUX

Day 4

1. Programming with Pig
2. Spark and RDD revisit
3. Hive
4. RDBMS with Hadoop
5. Flat-DB with Hadoop

Day 5

1. Querying
2. Apache Storm
3. Real World problems exercise