Azure Data Engineer Online Training in Hyderabad
Introduction to Azure
·
Introduction to
Azure Cloud
·
What is
difference between Azure Cloud and On-Premises
·
What is
Subscriptions and Resource Groups
·
Different
offerings of Cloud IaaS, PaaS and SaaS
·
Creation of
Virtual Machine
Introduction to Storage
·
Azure Storage
o Azure Blob
o Table
o Message
o Queue
·
Azure Data Lake Store Gen I & Gen II
o What is Data Lake
o Data Lake vs. Hadoop
o Blob Storage vs. Data Lake
o Hierarchical Namespace
o Ingestion through different tools i.e.; Azure Data
Explorer, AzCopy, Azure CLI, Powershell
Introduction to Azure SQL Database
·
Introduction to
Azure SQL Database
·
Why choosing SQL
Server in Azure
·
Azure IaaS vs.
PaaS database offerings
·
IaaS vs. Managed
Instance
·
SQL Server PaaS
deployment options
·
Demo - Azure
Single Database
·
Purchasing models
and Service Tier
·
Azure Database
vs. Azure Data Warehouse
·
Elastic Database
Pool
o Introduction
o Azure Elastic Database
o Demo - Azure Elastic Database
·
Managed Instance
Database
o Introduction
o Azure Managed Instance Database
o Difference between on-premises and managed instance
o Migration options for Managed Instance
o Service tiers for Managed Instance
o Demo - Managed Instance
·
Azure Database
Security
o Introduction
o Azure Database and Managed Instance Security options
o Encrypting Data at Rest and Motion
o High Availability vs. Disaster Recovery
o RTO vs. RPO
o Azure SQL Database High Availability and Disaster
Recovery options
o Azure SQL Database Scaling
·
Installation of SQL Server 2016 and above in
Virtual Machine
·
Creation of External Table or PolyBase in On-Premise
SQL Server
o Creation of Master Key
o Creation of Database Scoped Credential
o Creation of External Data Source
o Creation of
External File Format
o Creation of External Table
·
Creation of External Table or PolyBase in
Azure SQL Data Warehouse
o Creation of Master Key
o Creation of Database Scoped Credential
o Creation of External Data Source
o Creation of
External File Format
o Creation of External Table
·
Different Distribution or Shredding Patterns
o ROUND ROBIN
o HASH
o REPLICATION
·
Cross Query Databases in Azure SQL Database
o Creation of Master Key
o Creation of Database Scoped Credential
o Creation of External Data Source
o Creation of External Table
·
Creation of Elastic Pools in Azure SQL Server
between Databases
Data Warehouse Internals and Architecture
·
Introduction
·
Azure Synapse MPP
Architecture
·
Storage and
Sharding patterns
·
Data Distribution
and Distributing Keys
·
Data Types and
Table Types
·
Partitioning
·
Data Warehouse
Concepts
·
Dimensions and
Facts
·
Types of
Dimensions and Facts
·
Different types
of Schemas in Data Warehouse
·
Relationship
types in Data Warehouse
·
Best Practices
for Fact and Dimension tables
·
Demo - Analyze
Data distribution before migration to Azure Synapse
Azure Data Factory
·
Introduction to
Azure Data Factory
·
Creation of
Linked Services, Datasets, Pipelines
·
Creation of
Integration Runtime and different types
·
Slowly Changing
Dimensions
·
Design and
implement a Type 1 slowly changing dimension with mapping data flows
·
Debug data
factory pipelines
·
Understand the
Azure SSIS Integration Runtime
·
Set-up Azure SSIS
Integration Runtime
·
Run SSIS Package
in Azure Data Factory
·
Migrate SSIS
Packages to Azure Data Factory
·
Integrate SQL
Server Integration Services Packages within Azure Data Factory
·
Activities
o Copy
o Data flow
o Stored Procedure
o Lookup
o ForEach
o Get Metadata
o Filter Activity
o Spark
o U-SQL
o Databricks Notebooks
o Web
o If Condition
o Delete
·
Data Flows
o Derived Column
o Join
o filter
o exists
o conditional
split
o Lookup, Exists
o Select
o Aggregate
o Rank
o Filter
o Sort
o Alter Row
·
Dynamic Queries
in ADF
·
Sending mails
through Logic Apps
·
Few more
Activities ......
·
Dataset and
Pipeline Parameterization
·
Monitor -- Azure
and Visually
·
Setup Alerts from
Azure Data Factory
Realize Integrated Analytical Solutions with Azure
Synapse Analytics
·
Introduction
·
What is Azure
Synapse Analytics
·
How Azure Synapse
Analytics works
·
When to use Azure
Synapse Analytics
·
Create Azure
Synapse Analytics workspace
·
Exercise - Create
and manage Azure Synapse Analytics workspace
·
Describe Azure
Synapse Analytics SQL
·
Explain Apache
Spark in Azure Synapse Analytics
·
Exercise - Create
pools in Azure Synapse Analytics
·
Orchestrate data
integration with Azure Synapse pipelines
·
Exercise-Identifying
Azure Synapse pipeline components
·
Visualize your
analytics with Power BI
·
Understand hybrid
transactional analytical processing with Azure Synapse Link
·
Use Azure Synapse
Studio
·
Understand the
Azure Synapse Analytical processes
·
Explore the Data
hub, Develop hub, Integrate hub
·
Explore the
Monitor hub, Manage hub
·
Describe a modern
data warehouse
·
Define a modern
data warehouse architecture
·
Exercise -
Identify modern data warehouse architecture components
·
Design ingestion
patterns for a modern data warehouse
·
Understand data
storage for a modern data warehouse
·
Understand file
formats and structure for a modern data warehouse
·
Prepare and
transform data with Azure Synapse Analytics
·
Serve data for
analysis with Azure Synapse Analytics
Azure Synapse Analytics
Introduction
·
Why Warehouse in
cloud
·
Traditional vs.
Modern Warehouse architecture
·
What is Synapse
Analytics Service
·
Create Dedicated
SQL Pool and Spark Pool
·
Create Azure
Synapse Analytics Studio Workspace
·
Analyze Data
using Dedicated SQL Pool and Spark Pool
·
Analyze Data
using Apache Spark Notebook
·
Analyze Data
using Serverless SQL Pool
·
Azure Synapse
Benefits
Azure Event Hub,
IoT Hub and Azure Stream Analytics
·
Introduction to
Azure Event Hub, IoT Hub and Stream Analytics
·
Azure Stream
Analytics Job
·
Azure Stream
Analytics Components
·
Azure Stream Analytics Job
·
Batching
Streaming using Azure Event Hub
·
Real Time
Streaming using Azure IoT Hub
·
Types of Window
Functions
o Tumbling Window
o Hoping Window
o Sliding Window
o Session Window
Azure
Databricks
·
Spark Basics
·
Why Spark is
difficult? Why Databricks Evolved?
·
Why Databricks in
Cloud? Introduction to Azure Databricks
·
Demo
·
Provision
Databricks, Clusters and workbook
·
Mount Data Lake
to Databricks DBFS
·
Explore, Analyze,
Clean, Transform and Load Data in Databricks
·
Azure Databricks
Clusters
·
Azure Databricks
other Important Components
·
Databricks -
Monitoring
·
How to create
Cluster
·
How to work with
Databricks File System
·
How to create
notebooks and Integrate with ADF
·
How to
import and export the Notebooks
·
How to connect to
blob, SQL DB from Databricks
·
How to read data
files from Azure Blob and Azure Data Lake Store
§
Using Scala, R,
Python, Spark SQL Language
·
Creating Data
Frames
·
Converting Data
Frames into Temporary Table or Temporary View
·
Incremental and
Full Load with Azure SQL Data Warehouse
·
Understand the architecture
of Azure Databricks spark cluster
·
Understand the
architecture of spark job
·
Read data in CSV
format
·
Read data in JSON
format
·
Read data in
Parquet format
·
Read data stored
in tables and views
·
Write data
·
Describe a
DataFrame
·
Use common
DataFrame methods
·
Use the display
function
·
Exercise:
Distinct articles
·
Describe the
difference between eager and lazy execution
·
Describe the
fundamentals of how the Catalyst Optimizer works
·
Define and
identify actions and transformations
·
Describe the
column class
·
Work with column
expressions
·
Perform date and
time manipulation
·
Use aggregate
functions
·
Exercise:
Deduplication of data
·
Describe the
Azure Databricks platform architecture
·
Perform data
protection
·
Describe Azure
key vault and Databricks security scopes
·
Secure access
with Azure IAM and authentication
·
Describe security
·
Exercise: Access
Azure Storage with key vault-backed secrets
·
Describe the open
source Delta Lake
·
Exercise: Work
with basic Delta Lake functionality
·
Describe how
Azure Databricks manages Delta Lake
·
Exercise: Use the
Delta Lake Time Machine and perform optimization
·
Describe Azure
Databricks structured streaming
·
Perform stream
processing using structured streaming
·
Work with Time
Windows
·
Process data from
Event Hubs with structured streaming
·
Describe bronze,
silver, and gold architecture
·
Perform batch and
stream processing
·
Schedule
Databricks jobs in a data factory pipeline
·
Pass parameters
into and out of Databricks jobs in data factory
·
Integrate with
Azure Synapse Analytics
·
Understand
workspace administration best practices
·
List security
best practices
·
Describe tools
and integration best practices
·
Explain
Databricks runtime best practices
·
Understand
cluster best practices
Azure Cosmos DB
Introduction to NoSQL DB
·
Introduction to
NoSQL
·
SQL vs.
NoSQL
·
Types of NoSQL
·
NoSQL Offerings
by Microsoft
Introduction to Cosmos DB
·
Cosmos DB
Features
·
Cosmos DB -
Multi Model 5 APIs
·
Table Storage vs.
Cosmos DB
·
Provision Cosmos
DB Account
On-Premise Databases Migration
·
DMS -- Database
Migration Service
·
On-Premise SQL
Server to Azure Virtual Machine
·
On-Premise SQL
Server to Azure SQL Server