Azure Data Lake Storage Gen2 Account Configuration in Azure Portal to use in Binary Snap Pack

Overview

You can use this account type to connect the Snaps from Binary Snap Pack with data sources that use the Azure Data Lake Gen2 Storage account. An Azure storage platform includes data services like Blobs, Files, Queues, Tables, and Disks. This article guides you through creating a storage account using the Azure portal.

Prerequisites

An Azure account with a valid subscription to create the Storage Account.

Key steps

The following steps help you to create and register your application in the Azure Data Lake Storage Gen2 account to be used in the Snap.

  • Create the Azure Data Lake Storage Gen2 Storage Account

  • Check Access control (IAM) for the Storage Account

  • Select an app based on the permission

  • Locate the Tenant ID and Client ID for the registered app

  • Locate the Client and secrets ID value for the registered app

Create the Azure Data Lake Storage Gen2 Storage Account

  1. Log in to the Microsoft Azure Portal.

  2. Select Create a resource under Azure services on the Portal Home page or hover over the Storage accounts icon and click + Create.

  3. Select Storage accounts and specify the required details under each tab for the storage accounts. Refer to the storage details table for the configuration settings.

    1. Basic: Project and Instance details.

    2. Advanced: Security and Blob Storage details.

    3. Networking: Network connectivity and routing details.

    4. Data protection: Recovery and Tracking details.

    5. Encryption: Encryption details.

    6. Tags: Tag details.

    7. Review: Complete details of the Storage account.

  4. Specify the Project and Instance details as highlighted under the Basics tab.

     

  5. In the Advanced tab, add details related to Security.

  6. In the Networking tab, specify connectivity, endpoints, and routing details.

     

  7. In the Data protection tab, add details related to account Recovery and Tracking.

     

  8. In the Encryption tab, add details related to the encryption type.

  9. You can specify Resource Manager tags in the Tags tab to help organize your Azure resources.

  10. Click Review+Create to pass the validation.

  11. If the validation passes, click Create to deploy the Resource.

  12. Select Go to Resource, to view the details of the storage account.

The hierarchical namespace needs to be enabled to be a Gen2 account. Currently Binary Snaps support the Gen1 and Gen2 account. Refer to the screenshot provided below to enable the hierarchical namespace option.



Storage account details

Tab Name

Field Name

Type

Description

Tab Name

Field Name

Type

Description

Basics: Project details

Subscription

String

Required. Select the subscription to create the new storage account.

Basics: Instance details

Resource group

String

Required. You can create a new resource group for this storage account, or select an existing one. Learn more about , Resource groups.

Basics: Instance details

Storage account name

String

Specify a unique name for your storage account.

Storage account names must be between 3 and 24 characters in length and may contain numbers and lowercase letters only.

Basics: Instance details

Region

Drop-down

Select the appropriate region for your storage account from the drop-down list. For more information, refer to Regions and Availability Zones in Azure.

All the regions are not supported by all types of storage accounts or redundancy configurations. For more information, refer to Azure Storage redundancy.

Basics: Instance details

Performance

Radio button

Select the appropriate performance as per the purpose of your accounts. The options are:
Standard: Select for general-purpose v2 storage accounts. This is a default option. This type of account is recommended by Microsoft.
Premium: Select this option for scenarios requiring low latency. Under Premium, select the Premium account type to create. The following types of premium storage accounts are available:

  • Block blobs: Recommended for scenarios with high transaction rates or that use smaller objects or require consistently low storage latency.

  • File shares: Recommended for enterprise or high-performance scale applications. Use this account type if you want a storage account that supports both Server Message Block (SMB) and NFS file shares.

  • Page blobs: Recommended for random read and write operations.

Basics: Instance details

Redundancy

Drop-down

Select your desired redundancy configuration. The options available are:

  • LRS: Locally redundant storage - Recommended for non-critical scenarios. Low cost option with basic protection against server rack and drive failures.

  • GRS: Geo-redundant storage - Recommended for backup scenarios. Intermediate option with failover capabilities in a secondary region.

  • ZRS: Zone-redundant storage - Provides protection against data center level failures. Recommended for high availability scenarios.

  • GZRS: Geo-zone-redundant storage - Optimal data protection storage that includes the offerings of both GRS and ZRS. Recommended for critical data scenarios.

If you select a geo-redundant configuration (GRS or GZRS), your data is replicated to a data center in a different region.

Basics: Instance details

Make read access to data available in the event of regional unavailability.

Checkbox

Select this checkbox to get the read access to data in the secondary region.

Advanced Security: Enables you to configure the security settings which will impact the storage account.

Require secure transfer for REST API operations

Checkbox

Optional. Require secure transfer to ensure that incoming requests to this storage account are made only via HTTPS (default). Recommended for optimal Security. For more information, refer to Require secure transfer to ensure secure connections.

 

Allow enabling public access on containers.

Checkbox

Optional. Recommended to selectively enable public access on specific containers.

 

Enable storage account key access

Checkbox

Optional. When enabled, this setting allows clients to authorize requests to the storage account using either the account access keys or an Azure Active Directory (Azure AD) account (default). For more information, refer to Prevent Shared Key authorization for an Azure Storage account.

 

Default to Azure Active Directory authorization in the Azure portal

Checkbox

Optional. When enabled, the Azure portal authorizes data operations with the user's Azure AD credentials by default. For more information, refer to Default to Azure AD authorization in the Azure portal.

 

Minimum TLS version

Drop-down

Required. Select the minimum version of Transport Layer Security (TLS) for incoming requests to the storage account. The default value is TLS version 1.2. For more information, refer to Enforce a minimum required version of Transport Layer Security (TLS) for requests to a storage account.

 

Permitted scope for copy operations

Drop-down

When this property is enabled, the Azure portal authorizes requests to blobs, queues, and tables with Azure Active Directory by default.

 

Enable hierarchical namespace

Checkbox

Optional. For more information, refer to Introduction to Azure Data Lake Storage Gen2.

 

Enable SFTP

Checkbox

Optional. Enable the use of Secure File Transfer Protocol (SFTP) to securely transfer of data over the internet. For more information, refer to Secure File Transfer (SFTP) protocol support in Azure Blob Storage.

 

Enable network file system v3

Checkbox

Optional. Network File System Protocol is enablef for the storage account created. For more information, refer to Network File System (NFS) 3.0 protocol support in Azure Blob storage.

 

Allow cross-tenant replication

Checkbox

Required. By default, users with appropriate permissions can configure object replication across Azure AD tenants. For more information, refer to Prevent replication across Azure AD tenants.

 

Enable large file shares

Checkbox

Optional. Available only for standard file shares with the LRS or ZRS redundancies.

 

Access tier

Radio button

Optional. Hot and Cold based on frequency of usage.

 Networking

Network access

Radio button

Required. By default, incoming network traffic is routed to the public endpoint for your storage account.

Routing preference

Radio button

Required. The network routing preference specifies how network traffic is routed to the public endpoint of your storage account from clients over the internet.

 Data protection

Enable point-in-time restore for containers

Checkbox

Provides protection against accidental deletion or corruption by enabling you to restore block blob data to an earlier state. For more information, refer to Point-in-time restore for block blobs.

Enable soft delete for blobs

 

Checkbox

Optional. Soft delete enables you to recover blobs that were previously marked for deletion, including blobs that were overwritten.

Enable soft delete for blobs

Checkbox

Optional. Blob soft delete protects an individual blob, snapshot, or version from accidental deletes or overwrites by maintaining the deleted data in the system for a specified retention period.

Enable soft delete for containers

Checkbox

Container soft delete protects a container and its contents from accidental deletes by maintaining the deleted data in the system for a specified retention period. For more information, refer to Soft delete for containers (preview).

Enable soft delete for file shares

Checkbox

Optional. Soft delete for file shares protects a file share and its contents from accidental deletes by maintaining the deleted data in the system for a specified retention period. For more information, refer to Prevent accidental deletion of Azure file shares.

Enable versioning for blobs

Checkbox

Optional. Blob versioning automatically saves the state of a blob in a previous version when the blob is overwritten. For more information, refer to Blob versioning.

Enable blob change feed

Checkbox

Optional. The blob change feed provides transaction logs of all changes to all blobs in your storage account, as well as to their metadata. For more information, refer to Change feed support in Azure Blob Storage.

Enable version-level immutability support

Checkbox

Optional. Enable support for immutability policies that are scoped to the blob version. For more information, refer to Enable version-level immutability support on a storage account.

 Encryption

Encryption type

Radio button

By default, data in the storage account is encrypted by using Microsoft-managed keys. You can rely on Microsoft-managed keys for the encryption of your data, or you can manage encryption with your own keys. The options available are:

  • Microsoft-managed keys

  • Customer-managed keys

Encryption

Enable support for customer-managed keys

Radio button

By default, customer managed keys can be used to encrypt only blobs and files. Set this option to All service types (blobs, files, tables, and queues) to enable support for customer-managed keys for all services. You are not required to use customer-managed keys if you choose this option.

Encryption

Enable infrastructure encryption

Checkbox

Required if Encryption type field is set to Customer-managed keys. The available options are:

  • Select a key vault and key: Provides you an option to navigate to the key vault and key that you wish to use.

  • Enter key from URI: Provides you an option with a field to enter the key URI and the subscription.

Encryption

User-assigned identity

Radio button

Required if Encryption type field is set to Customer-managed keys. If you are configuring customer-managed keys at create time for the storage account, you must provide a user-assigned identity to use for authorizing access to the key vault.

Encryption

Enable infrastructure encryption

Checkbox

Optional. By default, infrastructure encryption is not enabled.

Tags

Name

Value

Resource

Dropdown

Optional. Tags are name/value pairs that enable you to categorize resources and helps in billing purpose by the same tag to multiple resources and resource groups. Learn more about Tags.

Review

NA

Labels

Provides details of all the tabs to review and proceed to create the Resource (storage account).

Check Access to the Resource

From the navigation panel, select Access control (IAM) to check the apps that have permission to access the storage account created.

Create an Azure Portal Application for the Storage Account

  1. From the left navigation menu on the home page, select the App registration option.

  2. Under Overview, register a new app or use an existing application. From the search option, select the registered app and make a note of the Application (Client) ID and Tenant ID. 

  3. For a new application, on the Overview page, select Add > App registration. On the Register an application page, specify the Name of the application and click Register. The Application (client) ID is required for the new account.

  4. Under Manage, click Certificates & secrets.

  5. On the Certificates & secrets page, click + New client secret.

  6. On the Add a client secret page, enter the Description and Expires and click Add. Make a note of the Client & secret value that is generated.

Specify Credentials in the Snap Account

  1. Navigate to the Snap of your choice and set up the account with the following details noted in the earlier steps.

    • Tenant ID

    • Access ID

    • Secret key 

  2. Click Save and validate the Snap with the Azure Data Lake Gen2 account.

Related Content