Overview of Microsoft Azure Data Lake Analytics

154 Pages·2016·3.11 MB·English

TTaabbllee ooff CCoonntteennttss
Overview
What is Data Lake Analytics?
Get started
Azure portal
Visual Studio
Visual Studio Code
Azure PowerShell
Azure CLI
How to
Manage Data Lake Analytics
Azure portal
Azure CLI
Azure PowerShell
.NET SDK
Python SDK
Java SDK
Node.js
Access diagnostic logs
Adjust quota limits
Develop U-SQL programs
U-SQL basics
U-SQL language reference
U-SQL catalog
U-SQL user-defined operators
U-SQL Python extensions
U-SQL R extensions
U-SQL Cognitive extensions
Analyze website logs
U-SQL custom code for Visual Studio Code
U-SQL for Visual Studio Code
Export U-SQL database
Debug U-SQL programs
Monitor and troubleshoot jobs
Troubleshoot an abnormal recurring job
Debug C# code in U-SQL
Use Job Browser and Job View
Use Vertex Execution View
Local run and local debug in VSCode
U-SQL Programmability Guide
Reference
Code samples
Azure PowerShell
.NET
Node.js
Python
REST
CLI
Resources
Azure Data Lake Blog
Azure Roadmap
Give feedback on UserVoice
MSDN Forum
Pricing
Pricing calculator
Service updates
Stack Overflow
Videos
Overview of Microsoft Azure Data Lake Analytics
6/27/2017 • 2 min to read • Edit Online
What is Azure Data Lake Analytics?
Azure Data Lake Analytics is an on-demand analytics job service to simplify big data analytics. You can focus on
writing, running, and managing jobs rather than on operating distributed infrastructure. Instead of deploying,
configuring, and tuning hardware, you write queries to transform your data and extract valuable insights. The
analytics service can handle jobs of any scale instantly by setting the dial for how much power you need. You
only pay for your job when it is running, making it cost-effective. The analytics service supports Azure Active
Directory letting you manage access and roles, integrated with your on-premises identity system. It also includes
U-SQL, a language that unifies the benefits of SQL with the expressive power of user code. U-SQL’s scalable
distributed runtime enables you to efficiently analyze data in the store and across SQL Servers in Azure, Azure
SQL Database, and Azure SQL Data Warehouse.
Key capabilities
Dynamic scaling
Data Lake Analytics is architected for cloud scale and performance. It dynamically provisions resources
and lets you do analytics on terabytes or even exabytes of data. When the job completes, it winds down
resources automatically, and you pay only for the processing power used. As you increase or decrease the
size of data stored or the amount of compute resources used, you don’t have to rewrite code. You can
focus on your business logic only and not on how you process and store large datasets.
Develop faster, debug, and optimize smarter using familiar tools
Data Lake Analytics has deep integration with Visual Studio, so you can use familiar tools to run, debug,
and tune your code. Visualizations of your U-SQL jobs let you see how your code runs at scale, so you can
easily identify performance bottlenecks and optimize costs.
U-SQL: simple and familiar, powerful, and extensible
Data Lake Analytics includes U-SQL, a query language that extends the familiar, simple, declarative nature
of SQL with the expressive power of C#. The U-SQL language is built on the same distributed runtime that
powers the big data systems inside Microsoft. Millions of SQL and .NET developers can now process and
analyze their data with the skills they already have.
Integrates seamlessly with your IT investments
Data Lake Analytics can use your existing IT investments for identity, management, security, and data
warehousing. This approach simplifies data governance and makes it easy to extend your current data
applications. Data Lake Analytics is integrated with Active Directory for user management and permissions
and comes with built-in monitoring and auditing.
Affordable and cost effective
Data Lake Analytics is a cost-effective solution for running big data workloads. You pay on a per-job basis
when data is processed. No hardware, licenses, or service-specific support agreements are required. The
system automatically scales up or down as the job starts and completes, so you never pay for more than
what you need.
Works with all your Azure Data
Data Lake Analytics is optimized to work with Azure Data Lake - providing the highest level of
performance, throughput, and parallelization for your big data workloads. Data Lake Analytics can also
work with Azure Blob storage and Azure SQL Database.
Next steps
Get Started with Data Lake Analytics using Azure portal | Azure PowerShell | CLI
Manage Azure Data Lake Analytics using Azure portal | Azure PowerShell | CLI | Azure .NET SDK | Node.js
Monitor and troubleshoot Azure Data Lake Analytics jobs using Azure portal
Get started with Azure Data Lake Analytics using
Azure portal
8/4/2017 • 1 min to read • Edit Online
Learn how to use the Azure portal to create Azure Data Lake Analytics accounts, define jobs in U-SQL, and submit
jobs to the Data Lake Analytics service. For more information about Data Lake Analytics, see Azure Data Lake
Analytics overview.
Prerequisites
Before you begin this tutorial, you must have an Azure subscription. See Get Azure free trial.
Create a Data Lake Analytics account
Now, you will create a Data Lake Analytics and a Data Lake Store account at the same time. This step is simple
and only takes about 60 seconds to finish.
1. Sign on to the Azure portal.
2. Click New > Data + Analytics > Data Lake Analytics.
3. Select values for the following items:
Name: Name your Data Lake Analytics account (Only lower case letters and numbers allowed).
Subscription: Choose the Azure subscription used for the Analytics account.
Resource Group. Select an existing Azure Resource Group or create a new one.
Location. Select an Azure data center for the Data Lake Analytics account.
Data Lake Store: Follow the instruction to create a new Data Lake Store account, or select an existing
one.
4. Optionally, select a pricing tier for your Data Lake Analytics account.
5. Click Create.
Your first U-SQL script
The following text is a very simple U-SQL script. All it does is define a small dataset within the script and then
write that dataset out to the default Data Lake Store as a file called .
/data.csv
@a =
SELECT * FROM
(VALUES
("Contoso", 1500.0),
("Woodgrove", 2700.0)
) AS
D( customer, amount );
OUTPUT @a
TO "/data.csv"
USING Outputters.Csv();
Submit a U-SQL job
1. From the Data Lake Analytics account, click New Job.
2. Paste in the text of the U-SQL script shown above.
3. Click Submit Job.
4. Wait until the job status changes to Succeeded.
5. If the job failed, see Monitor and troubleshoot Data Lake Analytics jobs.
6. Click the Output tab, and then click .
data.csv
See also
To get started developing U-SQL applications, see Develop U-SQL scripts using Data Lake Tools for Visual
Studio.
To learn U-SQL, see Get started with Azure Data Lake Analytics U-SQL language.
For management tasks, see Manage Azure Data Lake Analytics using Azure portal.
Develop U-SQL scripts by using Data Lake Tools for
Visual Studio
9/25/2017 • 2 min to read • Edit Online
Learn how to use Visual Studio to create Azure Data Lake Analytics accounts, define jobs in U-SQL, and submit
jobs to the Data Lake Analytics service. For more information about Data Lake Analytics, see Azure Data Lake
Analytics overview.
Prerequisites

Visual Studio: All editions except Express are supported.
Visual Studio 2017
Visual Studio 2015
Visual Studio 2013
Microsoft Azure SDK for .NET version 2.7.1 or later. Install it by using the Web platform installer.
A Data Lake Analytics account. To create an account, see Get Started with Azure Data Lake Analytics using
Azure portal.
Install Azure Data Lake Tools for Visual Studio
Download and install Azure Data Lake Tools for Visual Studio from the Download Center. After installation, note
that:
The Server Explorer > Azure node contains a Data Lake Analytics node.
The Tools menu has a Data Lake item.
Connect to an Azure Data Lake Analytics account
1. Open Visual Studio.
2. Open Server Explorer by selecting View > Server Explorer.
3. Right-click Azure. Then select Connect to Microsoft Azure Subscription and follow the instructions.
4. In Server Explorer, select Azure > Data Lake Analytics. You see a list of your Data Lake Analytics accounts.
Write your first U-SQL script
The following text is a simple U-SQL script. It defines a small dataset and writes that dataset to the default Data
Lake Store as a file called .
/data.csv
@a =
SELECT * FROM
(VALUES
("Contoso", 1500.0),
("Woodgrove", 2700.0)
) AS
D( customer, amount );
OUTPUT @a
TO "/data.csv"
USING Outputters.Csv();
SSuubbmmiitt aa DDaattaa LLaakkee AAnnaallyyttiiccss jjoobb
1. Select File > New > Project.
2. Select the U-SQL Project type, and then click OK. Visual Studio creates a solution with a Script.usql file.
3. Paste the previous script into the Script.usql window.
4. In the upper-left corner of the Script.usql window, specify the Data Lake Analytics account.
5. In the upper-left corner of the Script.usql window, select Submit.
6. Verify the Analytics Account, and then select Submit. Submission results are available in the Data Lake
Tools for Visual Studio Results after the submission is complete.
7. To see the latest job status and refresh the screen, click Refresh. When the job succeeds, it shows the Job
Graph, MetaData Operations, State History, and Diagnostics:
Job Summary shows the summary of the job.
Job Details shows more specific information about the job, including the script, resources, and
vertices.
Job Graph visualizes the progress of the job.
MetaData Operations shows all the actions that were taken on the U-SQL catalog.
Data shows all the inputs and outputs.
Diagnostics provides an advanced analysis for job execution and performance optimization.
TToo cchheecckk jjoobb ssttaattee
1. In Server Explorer, select Azure > Data Lake Analytics.
2. Expand the Data Lake Analytics account name.
3. Double-click Jobs.
4. Select the job that you previously submitted.
TToo sseeee tthhee oouuttppuutt ooff aa jjoobb
1. In Server Explorer, browse to the job you submitted.
2. Click the Data tab.
3. In the Job Outputs tab, select the file.
"/data.csv"
Next steps
Run U-SQL scripts on your own workstation for testing and debugging
Debug C# code in U-SQL jobs using Azure Data Lake Tools for Visual Studio Code
Use the Azure Data Lake Tools for Visual Studio Code
Use Azure Data Lake Tools for Visual Studio Code
12/21/2017 • 12 min to read • Edit Online
Learn the Azure Data Lake Tools for Visual Studio Code (VS Code) to create, test, and run U-SQL scripts. The
information is also covered in the following video:
Prerequisites
Azure Data Lake Tools for VSCode supports Windows, Linux, and MacOS.
Visual Studio Code.
For MacOS and Linux:
.NET Core SDK 2.0.
Mono 5.2.x.
Install Data Lake Tools
After you install the prerequisites, you can install Data Lake Tools for VS Code.
To install Data Lake Tools
1. Open Visual Studio Code.
2. Click Extensions in the left pane. Enter Azure Data Lake in the search box.
3. Click Install next to Azure Data Lake tools. After a few seconds, the Install button will be changed to Reload.
4. Click Reload to activate the Azure Data Lake tools extension.
5. Click Reload Window to confirm. You can see Azure Data Lake tools in the Extensions pane.