Featured image of post How to Define Streaming Units for Microsoft.StreamAnalytics/streamingjobs Resource

How to Define Streaming Units for Microsoft.StreamAnalytics/streamingjobs Resource

You can have some…interesting times when experimenting with Azure Templates. Aimed primarily at those managing complex Azure estates or organisations which have a desire to incorporate their Azure development cycles as part of their DevOps processes, they are no doubt a powerful feature for developers to leverage. Even with my very recent and limited exposure to them, I am already at a place where I can confidently list the range of benefits they can deliver. The only thing I would caveat concerning their usage is that it is imperative that you put aside time to fully understand the precise behaviour that any templates you draft have when targeting your resource groups. This step is primarily to ensure that you do not accidentally remove configuration properties on your resource, overlook an important property/configuration or, ultimately, end up with a large, unexpected credit card charge 🙂

A good example of why I would recommend this careful approach so much can be found when working with Steam Analytics Jobs within Azure, a service which is great for processing high-volume data streams from a variety of sources, such as Application Insights Continuous Export. When considering them for utilisation as part of an Azure template without explicitly setting any specific queries, Azure will automatically provision your resource using 3 Streaming Units, as opposed to 1. We can observe this behaviour when attempting to deploy the following template to the platform:

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "Parameters": {},
    "variables": {},
    "resources": [
        {
            "type": "Microsoft.StreamAnalytics/streamingjobs",
            "apiVersion": "2016-03-01",
            "name": "MyStreamAnalyticsJob",
            "location": "uksouth",
            "properties": {
                "sku": {
                    "name": "Standard"
                }
            }
        }
    ]
}

To give you full control over the exact amount of streaming units deployed alongside the resource, we need to modify the template to include the Query needed when processing input data, as the properties exposed here allow you to define this exact value. Therefore, the above template required modifying as follows:

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "Parameters": {},
    "variables": {},
    "resources": [
        {
            "type": "Microsoft.StreamAnalytics/streamingjobs",
            "apiVersion": "2016-03-01",
            "name": "MyStreamAnalyticsJob",
            "location": "uksouth",
            "properties": {
                "sku": {
                    "name": "Standard"
                },
                "transformation": {
                    "name": "MyQuery",
                    "properties": {
                        "streamingUnits": 1,
                        "query": "SELECT * INTO [YourOutputAlias] FROM [YourInputAlias]"
                    }
                }
            }
        }
    ]
}

So, if you are looking to automate and manage deployments relating to your Stream Analytics resources, it is recommended that all aspects of your jobs are defined out within your template, specifically:

  • Inputs
  • Query
  • Outputs

This recommendation becomes mandatory if you are carrying out frequent template updates to your resource group, as performing a redeployment to an existing resource using the first template illustrated in this post will lead to loss of data; the query defined on the resource will be deleted, and the number of streaming units will be bumped back up to 3. It’s therefore vital, both from billing and also from a resource integrity perspective, that any query development is regularly fed back into your template and factored in as part of any updates. This additional step is a hell of a lot easier to do than you may think, as the portal supports the ability to generate resource templates at the drop of a hat and then moved across quickly to your template file/code repository.

As highlighted previously, Azure Templates are a great feature to get familiar with if you find yourself being weighed down by a particularly cumbersome Azure estate, but they do have their quirks and behaviours that can take some time to understand fully. Thankfully, with proper testing, it is relatively easy to identify and put right issues similar to the one described in this post.

comments powered by Disqus
Built with Hugo
Theme Stack designed by Jimmy