PowerShell is a cross-platform scripting language that allows automation and configuration management tasks. It provides access to a wide range of Azure resources, enabling seamless interaction with services such as Azure Data Factory, Azure SQL Database, Azure Storage, and more. Azure, on the other hand, is a cloud computing platform that offers a broad spectrum of services, including data storage, analytics, virtual machines, and machine learning.

In this blog we are going to see about how to create event hub topic using power shell command. Here we are not only going to see about that alone. We are going to see about how can we create multiple event hubs and also enable capture events for them in power shell script. Hope you are excited to learn about this.

As an Azure Data Engineer, your role revolves around managing, processing, and analyzing vast amounts of data in the cloud. To streamline your workflows and automate repetitive tasks, PowerShell scripts can be a game-changer. Azure provides powerful tools and services that can be leveraged through PowerShell, enabling you to unleash the full potential of your Azure environment.

Benefits of PowerShell in Azure:

  • Automation and Efficiency: PowerShell empowers Azure Data Engineers to automate complex processes, reducing manual effort and minimizing human error. It enables you to create, manage, and scale resources programmatically, thereby increasing operational efficiency.

  • Flexibility and Customization: PowerShell scripts provide the flexibility to tailor your Azure workflows according to specific requirements. You can design custom automation tasks, execute commands across multiple resources, and integrate PowerShell scripts into your existing infrastructure.

  • Integration with DevOps: PowerShell seamlessly integrates with Azure DevOps, enabling continuous integration and deployment (CI/CD) pipelines. This integration facilitates version control, collaboration, and automated testing, ensuring smooth application deployments and updates.


Now we are going to see about the automation and Efficiency part , that can be used by Azure Data engineer. Suppose think in your project , you are using Azure Event hubs . Each event hub will hold the each table data. You already know that we will have more than hundreds of tables required for a project.
So you cannot create event hub for each table that will be very tedious process , as there are many tables, you are going to do the same process again and again.

So we can automate this process using Power shell. Please note that to run the power shell script in Azure portal you must need a storage account. And point to a files hare location in that storage account while opening the power shell terminal in the azure portal. Because the Azure portal use that location for storing the code and for execution purpose.

Now lets see basic things you need to know so that you can quickly start coding the power shell script. This will looks similar to python with curly braces, so it will be easy for you to create the power shell script.

Basically here our concept is we are going to have a file.txt that holds the details of the tables that means the event hub topics that needs to be created in azure portal. Therefore we are going to loop over the file records and based on that we are going to create the event hubs , as simple as that.


Step 1: Passing Parameter

  • The first thing we need to know about is how to pass the parameter to our script. Check the below code,
Param(
    [Parameter (Mandatory=$true)]
    [string]$inputFileName,
    [Parameter(Mandatory=$false)]
    [string]$storageAccount
)

In this code, we can define that the parameter passed is to be a mandatory or not by saying $true or $false. And before the starting of the parameter name we need to mention the datatype of it.


Step 2:  Write-Host 

  • This is nothing but like print statement , which is visible on the console screen. Through which we can display any processing message.

Write-Host "process started"


Step 3:  Defining Variable

  • Next important step is declaring the variable and it is very simple.
$inputFile=$inputFileName
$captureNameFormat="{Namespace}/{EventHub}/{Year}/{Month}/{Day}/{Hour}/{Minute}/{Second}/{PartitionId}"


Step 4: ForEach

  • Now we have came into the main code. Which will loop through the file and do the respective actions. 

Get - Content $inputFile | Select-Object -Skip 1 | ForEach-Object {
    # Trim any leading or trailing whitespace from the line
    $line = $_.Trim()

    #split the line based on comma and store the required values
    $topicName,$partitionCount,$retentionHours, $consumerGroup,$captureInterval,$captureSizeLimit,$captureevents =$line.split(",")

    $result = az eventhubs eventhub create `
                    --name $topicName `
                    --resource-group $resourceGroup `
                    --namespace-name $namespaceName `
                    --partition-count $partitionCount `
                    --retention-time $retentionHours `
                    --enable-capture true `
                    --capture-interval $captureInterval `
                    --capture-size-limit $captureSizeLimit `
                    --destination-name $destinationName `
                    --skip-empty-archives true `
                    --cleanup-policy DELETE `
                    --storage-account $storageAccount `
                    --blob-container $storageContainer `
                    --archive-name-format $captureNameFormat

}


The code given above is self explanatory , with comments. Comments are mentioned with # symbol.
ForEach will loop through each line of the file, and we need to skip the header hence we gave -skip 1.
Then we are using the $_.Trim() to remove the leading and trailing white space. Then we are using Split(",") to split the line where it has comma and assign each value to a variable.
Then we go with executing the create eventhub command using the variables we have.

this ` symbol is use to break the script into next line for increase the readability.


Step 5 : LASTEXITCODE 


  • Using try and catch block is not directly possible in power shell script, for that we need to do some other procedure and make it active. But i can suggest a simple idea. Go with Lastexitcode.

  • If it is 0 then the last executed command is successful, else a failure has been happened.

if ($LASTEXITCODE -eq 0) {
    Write-Host "  create $topicName completed successfully"
elseif ($LASTEXITCODE -ne 0) {
    Write-Host "  create $topicName has been failed"


Step 6:  Run the script

  • Below code will have the run command as well as shows how to get the logs in a file.

  • We can pass the parameter while running the script.
.\create_topics.ps1 -resourceGroup "myresourcegroup" -namespaceName "mynamespace" -storageAccount "storage account name" -storageContainer "Container name" - inputFileName *>&1 | Tee-Object -FilePath "log.txt" 

 Here, please note that the code was saved as create_topic.ps1 file. This is the Power shell script extension. The details showed in the console while running the script will be stored in the log file log.txt

Step 7 : Help 

  • In power shell script we can add the help text in above the code or at bottom of the code, such that while the users want to know about the script or they want to know how to execute the script or they want to know what are the parameters , then they can execute the help command and got to know about the script.

<#
.SYNOPSIS
create multiple event hubs topics based on the configuration in a file topics.txt

.DESCRIPTION 
This script create multiple event hubs topics based on the configuration in a file topics.txt.
Log file will be captures in log.txt

.PARAMETER inputFileName 
It is the input file which holds the configuration details

.NOTES
created by,
timestamp
#>
     

 To see this help statement without opening the script. we need to use the below command


Get-Help .\create_topics.ps1

 

Where create_topics.ps1 is the name of the script. Thus we saw a quick powershell script. Let us also see some of the other use cases.


Some other use cases:

  • Data Ingestion and Transformation: PowerShell scripts can be used to automate data ingestion from various sources into Azure Data Lake Storage or Azure Data Factory pipelines. You can employ PowerShell cmdlets to extract, transform, and load (ETL) data efficiently, ensuring a smooth data pipeline flow.

  • Monitoring and Alerting: With PowerShell, you can monitor the health and performance of your Azure resources. You can create scripts that fetch metrics, check status, and generate alerts based on predefined conditions. This enables proactive monitoring and ensures timely actions are taken to address issues.

  • Security and Access Control: PowerShell scripts facilitate the implementation of robust security measures in your Azure environment. You can automate the configuration of network security groups, access control lists, and role-based access control (RBAC) settings to ensure proper data protection and compliance.

  • Cost Optimization: PowerShell allows you to optimize costs by automating resource scheduling and scaling based on demand. You can create scripts that dynamically scale virtual machine instances, adjust storage capacities, or even shutdown and start resources during non-business hours, ensuring optimal resource utilization and cost savings.


Thus in this blog we saw about the Powershell script its benefits and usecases. And mainly we saw a quick powershell script to automate the process of creating the event hubs. Hope this is helpful

Thank You !!!