Course Topics / Agenda
Please note that this list of topics is based on our standard course offering, evolved from typical industry uses and trends. We’ll work with you to tune this course and level of coverage to target the skills you need most. Topics, agenda and labs are subject to change, and may adjust during live delivery based on audience skill level, interests and participation.
Getting Started with Scala and Spark
- Introduction to Scala
- Brief history and motivation
- Differences between Scala and Java
- Basic Scala syntax and constructs
- Scala’s functional programming features
- Introduction to Apache Spark
- Overview and history
- Spark components and architecture
- Spark ecosystem
- Comparing Spark with other big data frameworks
- Lab: Practice basic Scala syntax and functional programming concepts using the REPL.
- Setting up the Development Environment
- Basics of Spark Programming SparkContext and SparkSession
- Resilient Distributed Datasets (RDDs)
- Transformations and Actions
- Working with DataFrames
- Spark SQL and Data Sources
- Spark SQL library and its advantages
- Structured and semi-structured data sources
- Reading and writing data in various formats (CSV, JSON, Parquet, Avro, etc.)
- Data manipulation using SQL queries
- Lab: Setting up the Environment and Running a Simple Spark Application.
- Lab: Load and query data from different data sources using Spark SQL.
Data Processing and Spark Programming
- Basic RDD Operations
- Creating and manipulating RDDs
- Common transformations and actions on RDDs
- Working with key-value data
- Basic DataFrame and Dataset Operations
- Creating and manipulating DataFrames and Datasets
- Column operations and functions
- Filtering, sorting, and aggregating data
- Lab: RDD and DataFrame Operations
- Introduction to Spark Streaming
- Overview of Spark Streaming
- Discretized Stream (DStream) operations
- Windowed operations and stateful processing
- Performance Optimization Basics
- Best practices for efficient Spark code
- Broadcast variables and accumulators
- Monitoring Spark applications
- Integrating External Libraries and Tools, Spark Streaming
- Using popular external libraries, such as Hadoop and HBase
- Integrating with cloud platforms: AWS, Azure, GCP
- Connecting to data storage systems: HDFS, S3, Cassandra, etc.
- Lab: Building an End-to-End Spark Application: Create a Spark application to process a large dataset, perform aggregations, and save the results.
- Lab: Implement a simple Spark Streaming application to process real-time data.
Machine Learning Basics with Spark MLlib
- Introduction to Machine Learning Basics
- Overview of machine learning
- Supervised and unsupervised learning
- Common algorithms and use cases
- Introduction to Spark MLlib
- Overview of Spark MLlib
- MLlib’s algorithms and utilities
- Data preparation and feature extraction
- Lab: Data Preparation with Spark MLlib
- Linear Regression and Classification
- Linear regression algorithm
- Logistic regression for classification
- Model evaluation and performance metrics
- Clustering Algorithms
- Overview of clustering algorithms
- K-means clustering
- Model evaluation and performance metrics
- Collaborative Filtering and Recommendation Systems
- Overview of recommendation systems
- Collaborative filtering techniques
- Implementing recommendations with Spark MLlib
- Lab: Basic Machine Learning with Spark MLlib
- Lab: Implementing a Recommendation System
Graph Processing and Generative AI Technologies
- Introduction to Graph Processing
- Overview of graph processing
- Use cases and applications of graph processing
- Graph representations and operations
- Introduction to Spark GraphX
- Overview of GraphX
- Creating and transforming graphs
- Graph algorithms in GraphX
- Lab: Graph Processing with GraphX
- Big Data Innovation! Using GPT and Generative AI Technologies with Spark and Scala
- Overview of generative AI technologies
- Integrating GPT with Spark and Scala
- Practical applications and use cases
Bonus Topics / Time Permitting
- Introduction to Spark NLP
- Overview of Spark NLP
- Preprocessing text data
- Text classification and sentiment analysis
- Lab: Generative AI Technologies and Spark NLP: Integrate GPT for text generation in a Spark application and explore basic NLP tasks using Spark NLP.
- Lab: Text Classification with Spark NLP
- Putting It All Together
- Work on a capstone project that integrates multiple aspects of the course, including data processing, machine learning, graph processing, and generative AI technologies.
Learning Objectives
Working in a hands-on learning environment led by our expert instructor you’ll:
- Develop a basic understanding of Scala and Apache Spark fundamentals, enabling you to confidently create scalable and high-performance applications.
- Learn how to process large datasets efficiently, helping you handle complex data challenges and make data-driven decisions.
- Gain hands-on experience with real-time data streaming, allowing you to manage and analyze data as it flows into your applications.
- Acquire practical knowledge of machine learning algorithms using Spark MLlib, empowering you to create intelligent applications and uncover hidden insights.
- Master graph processing with GraphX, enabling you to analyze and visualize complex relationships in your data.
- Discover generative AI technologies using GPT with Spark and Scala, opening up new possibilities for automating content generation and enhancing data analysis.
If your team requires different topics, additional skills or a custom approach, our team will collaborate with you to adjust the course to focus on your specific learning objectives and goals.
In order to be successful in this course you should possess:
- Basic understanding of Java programming: Familiarity with Java syntax, data structures, and concepts, such as variables, loops, and conditionals.
- Fundamental knowledge of object-oriented programming (OOP): Experience with OOP principles, such as inheritance, encapsulation, and polymorphism, in any programming language.
- Familiarity with data structures and algorithms: A basic grasp of common data structures, such as arrays, lists, and maps, as well as an understanding of simple algorithms, like sorting and searching.
- Experience with distributed systems: Basic awareness of distributed computing concepts, such as data partitioning, parallel processing, and fault tolerance.
- Basic knowledge of databases: Understanding of database concepts, including data storage, querying, and manipulation using SQL or NoSQL databases.
Ten (10) business days’ notice is required to reschedule a class with no additional fees. Notify TOPTALENT LEARNING as soon as possible at 469-721-6100 or by written notification to info@toptalentlearning.com to avoid rescheduling penalties.
Please contact our team at 469-721-6100; we will gladly guide you through the online purchasing process.
You will receive a receipt and an enrollment confirmation sent to the email you submitted at purchase. Your enrollment email will have instructions on how to access the class. Any additional questions our team is here to support you. Please call us at 469-721-6100.
If a student is 15 minutes late, they risk losing their seat to a standby student. If a student is 30 minutes late or more, they will need to reschedule. A no-show fee will apply. Retakes are enrolled on a stand-by basis. The student must supply previously issued courseware. Additional fees may apply.
You will receive a ‘Certificate of Completion’ once you complete the class. If you purchased an exam voucher for the class, a team member from TOPTALENT LEARNING will reach out to discuss your readiness for the voucher and make arrangements to send it.
