Michael Washam Technical Evangelist – Azure | Server | Web

Windows Azure Storage Analytics allows you to log very detailed information about how a storage account is being used. Each service (Blob/Table/Queues) has independant settings allowing you to have granular control over what data is collected. To enable/disable each setting just add or omit the argument (-LoggingDelete as an example). One of the great things about this service is the ability to set a retention policy so the data can be automatically deleted after a set number of days.

Enabling Storage Analytics per Service

Set-StorageServicePropertiesForAnalytics -ServiceName "Table" `
		-StorageAccountName $storageAccount -StorageAccountKey $storagekey `
		-LoggingDelete -LoggingRead -LoggingWrite -MetricsEnabled -MetricsIncludeApis `
		-MetricsRetentionPolicyDays 5 -LoggingRetentionPolicyEnabled -LoggingRetentionPolicyDays 5 `
		-MetricsRetentionPolicyEnabled 
									
Set-StorageServicePropertiesForAnalytics -ServiceName "Queue" `
		-StorageAccountName $storageAccount -StorageAccountKey $storagekey `
		-LoggingDelete -LoggingRead -LoggingWrite -MetricsEnabled -MetricsIncludeApis `
		-MetricsRetentionPolicyDays 5 -LoggingRetentionPolicyEnabled -LoggingRetentionPolicyDays 5 `
		-MetricsRetentionPolicyEnabled 
									
Set-StorageServicePropertiesForAnalytics -ServiceName "Blob" `
		-StorageAccountName $storageAccount -StorageAccountKey $storagekey `
		-LoggingDelete -LoggingRead -LoggingWrite -MetricsEnabled -MetricsIncludeApis `
		-MetricsRetentionPolicyDays 5 -LoggingRetentionPolicyEnabled -LoggingRetentionPolicyDays 5 `
		-MetricsRetentionPolicyEnabled 		

Retrieving the Current Storage Analytics Settings

	Get-StorageServicePropertiesForAnalytics -ServiceName "Table" `
		-StorageAccountName $storageAccount -StorageAccountKey $storagekey | Format-List
	
	Get-StorageServicePropertiesForAnalytics -ServiceName "Blob" `
		-StorageAccountName $storageAccount -StorageAccountKey $storagekey | Format-List
		
	Get-StorageServicePropertiesForAnalytics -ServiceName "Queue" `
		-StorageAccountName $storageAccount -StorageAccountKey $storagekey | Format-List

Downloading the storage analytics data requires a bit more explanation. Each service has two fundamental types of data (except blob storage which has 3). The two are log data which contains all of the requests for that service (depending on which settings you have enabled) and transactions. Transactions contains numerous metrics that give you a deep understanding of how the storage service is performing. Metrics such as % Success or Avg E2E Latency are extremely useful for understanding your application. Blob storage also has a “Capacity” set of data that will tell you how much storage space blob storage is using broken down by analytics and application data.

Downloading Storage Analytics Data

Get-StorageAnalyticsLogs -ServiceName "Blob" `
	-LocalPath "c:\DiagData\SALogsBlob.log"  `
	-StorageAccountName $storageAccount -StorageAccountKey $storagekey  

Get-StorageAnalyticsMetrics -DataType "Capacity" -ServiceName "Blob" `
    -LocalPath "c:\DiagData\SAMetricsBlob-Capacity.log" `
    -StorageAccountName $storageAccount -StorageAccountKey $storagekey 
						 
Get-StorageAnalyticsMetrics -DataType "Transactions" -ServiceName "Blob" `
	-LocalPath "c:\DiagData\SAMetricsBlob-Transactions.log" `
	-StorageAccountName $storageAccount -StorageAccountKey $storagekey 

Get-StorageAnalyticsLogs -ServiceName "Table" `
	-LocalPath "c:\DiagData\SALogsTable.log"  `
	-StorageAccountName $storageAccount -StorageAccountKey $storagekey  
						 
Get-StorageAnalyticsMetrics -DataType "Transactions" -ServiceName "Table" `
	-LocalPath "c:\DiagData\SAMetricsTable-Transactions.log" `
	-StorageAccountName $storageAccount -StorageAccountKey $storagekey 

Get-StorageAnalyticsLogs -ServiceName "Queue" `
	-LocalPath "c:\DiagData\SALogsQueue.log" `
	-StorageAccountName $storageAccount -StorageAccountKey $storagekey  
						 
Get-StorageAnalyticsMetrics -DataType "Transactions" -ServiceName "Queue" `
	-LocalPath "c:\DiagData\SAMetricsQueue-Transactions.log" `
	-StorageAccountName $storageAccount -StorageAccountKey $storagekey 

For more information on storage analytics and details on understanding the metrics you can use see the following:
http://msdn.microsoft.com/en-us/library/windowsazure/hh343268.aspx

Managing the passwords of your SQL Azure servers is now 100% scriptable. It’s so easy to reset them it’s almost like magic
The code below takes your subscription id and your management certificate and returns all of the SQL Azure servers in your subscription. It takes that collection and passes them to the Set-SqlAzurePassword Cmdlet along with the new password.

Resetting the Passwords of all SQL Azure Servers in your Subscription

$subscriptionId = "Your-ID-Goes-Here"
$cert = Get-Item cert:\CurrentUser\My\YOURCERTTHUMBPRINTGOESHERE
Get-SqlAzureServer -Certificate $cert -SubscriptionId $subscriptionid | `
	Set-SqlAzurePassword -NewPassword "abracadabra0!"

Windows Azure PowerShell Cmdlets 2.0 Update Web Cast Now Available

Creating SQL Azure Servers and Configuring firewall rules is surprisingly easy with the new Windows Azure PowerShell Cmdlets 2.0.

Step 1: Add the WAPPSCmdlets Snapin or Module

Add-PsSnapin WAPPSCmdlets

Step 2: Create a Few Needed Variables

$subscriptionid = "your subscription id"
$cert = Get-Item cert:\CurrentUser\My\yourcertthumbprint
$adminLogin = "sqlAzureLogin"
$adminPassword = "sqlAzurePassword"

Step 3: Call the Cmdlets With Your Variables

$newServer = New-SqlAzureServer -AdministratorLogin $adminLogin -AdministratorLoginPassword $adminPassword `
			       -Location "North Central US" -Certificate $cert `
				   -SubscriptionId $subscriptionid

$newServer | New-SqlAzureFirewallRule -RuleName "EveryBody" `
			-StartIpAddress "0.0.0.0" -EndIpAddress "255.255.255.255"

Obviously, you would want to change the firewall rule if you did not want EVERY IP address to be able to connect to your server but you get the idea.

You’ve deployed a few Sql Azure servers and through no fault of your own the requirement comes up to update all of the firewall rules for each of the Sql Azure Servers.

No Problem!

Adding new SQL Azure Firewall Rules

Get-SqlAzureServer -Certificate $cert -SubscriptionId $subscriptionid | foreach {
  $_ | New-SqlAzureFirewallRule -RuleName "NewRule1" -StartIpAddress "0.0.0.0" -EndIpAddress "1.1.2.2"
  $_ | New-SqlAzureFirewallRule -RuleName "NewRule2" -StartIpAddress "100.1.0.0" -EndIpAddress "100.15.0.0"
}

Removing Rules is Just as Easy

Get-SqlAzureServer -Certificate $cert -SubscriptionId $subscriptionid | foreach {
  $_ | Remove-SqlAzureFirewallRule -RuleName "OldRule1"
  $_ | Remove-SqlAzureFirewallRule -RuleName "OldRule2"
}

This of course requires the Windows Azure PowerShell Cmdlets 2.0

With the new version of Windows Azure PowerShell Cmdlets 2.0 we have made it possible to script the deployment of an entire hosted service directly from PowerShell!

As a proof of concept we have put together a sample script that reads settings from a configuration file and deploys a service package to your Windows Azure Account. What is new about this is if the hosted service or storage account do not exist yet we will create them using the settings you specify!

To get started ensure you have downloaded and installed the Windows Azure PowerShell Cmdlets 2.0.

Next download the sample script from here: https://mlwpublic.blob.core.windows.net/downloads/e2e-ps-deployment.zip

Once the file is unzipped you will need to modify the configuration.xml file with your own settings:

Once configured copy your .cspkg file and .cscfg file to the \deploymentPackage folder beneath the script.

Finally, run the deploy.cmd file which will launch your PowerShell script and deploy the service!

Operations demonstrated in this sample include:

• New-HostedService / Get-HostedService
• New-StorageAccount / Get-StorageAccount
• New-Deployment / Get-Deployment
• Add-Certificate
• Move-Deployment (VIP Swap)

This is part 5 of a series:

• Managing Tracing with Windows Azure PowerShell Cmdlets 2.0
• Managing Performance Counters with Windows Azure PowerShell Cmdlets 2.0
• Viewing your diagnostic settings with Windows Azure PowerShell Cmdlets 2.0
• Managing Windows Event Logs with Windows Azure PowerShell Cmdlets 2.0
• Managing Log Files with Windows Azure PowerShell Cmdlets 2.0

In this example I’m going to show you how you can manage log data from your Windows Azure Application using new functionality in the Windows Azure PowerShell Cmdlets 2.0.

Just as in the other articles you will need to add the PowerShell Snapin (or module):

 
  Add-PsSnapin WAPPSCmdlets 

Ditto, I have this handy initialization block and helper function GetDiagRoles:

Initialization and Helper Function

 

 $storageAccountName = "YourStorageAccountName" 
 $storageAccountKey = "YourStorageAccountKey" 
 $deploymentSlot = "Production" 
 $serviceName = "YourHostedService" 
 $subscriptionId = "YourSubscriptionId" 
 # Thumbprint for your cert goes below
 $mgmtCert = Get-Item cert:\CurrentUser\My\D7BECD4D63EBAF86023BB4F1A5FBF5C2C924902A 
 

 function GetDiagRoles { 
   Get-HostedService -ServiceName $serviceName -SubscriptionId $subscriptionId -Certificate $cert | ` 
   Get-Deployment -Slot $deploymentslot | ` 
   Get-DiagnosticAwareRoles -StorageAccountName $storageAccount -StorageAccountKey $storageKey 
 }
 

What is distinct about managing log files remotely in Windows Azure is determining where on the file system the log files actually are.

I’ve written some samples that show some techniques you can use from PowerShell for determining file locations.

When determining the local path for a named local resource I’m using guidance from MSDN: http://msdn.microsoft.com/en-us/library/ee758708.aspx.

For the rest of the log paths I am essentially just using observations of how the paths are constructed. At this point there is no guarantee this format will change but for the time being it doesn’t look too likely.

Returns the Local Path for IIS Logs

 
function GetIISLogsPath()
{
 $input | foreach { 
  $path = "C:\Resources\Directory\" + $_.DeploymentId + "." + $_.RoleName + ".DiagnosticStore\LogFiles"
  return $path
 }
}

Returns the Local Path for IIS Failed Request Logs

 
function GetIISFailedRequestPath()
{
 $input | foreach {
  $path = "C:\Resources\Directory\" + $_.DeploymentId + "." + $_.RoleName + ".DiagnosticStore\FailedReqLogFiles"
  return $path
 }
}

Returns the Local Path for Crash Dumps

 
function GetCrashDumpsPath($rolename, $deploymentid)
{
 $input | foreach {
  $path = "C:\Resources\Directory\" + $_ + "." + $_ + ".DiagnosticStore\Crashdumps"
  return $path
 }
}

Returns the Local Path for a Named Local Resource

 
function GetLocalResourcePath($localResource)
{
 $input | foreach {
  $path = "C:\Resources\directory\" + $_.DeploymentId + "." + $_.RoleName + "." + $localResource
  return $path
 }
}

Now that we have some helper functions to figure out where are logs are in our Windows Azure Deployment we need a method of modifying the logging data sources in Windows Azure Diagnostics.
The long function below does just that. It takes the local path of the files you want to transfer, what container in Windows Azure Storage to transfer the files to, a quota (can be set to 0) and how often to transfer that directory to storage in minutes.

Configures a local directory in a Windows Azure role to be transferred into Windows Azure Storage.

 
function SetDirectory($path, $container, $quota, $transferAllInMinutes)
{
  $input | foreach { 
    $role = $_ 
    $role | Get-DiagnosticAwareRoleInstances  |
      foreach {
          $roleinstance = $_
          $diagConfig = $role | Get-DiagnosticConfiguration -InstanceId $roleinstance -BufferName Directories
          $existingDirectory = $false
          $newDirectoryConfig = $null
          for($i=0;$i -lt $diagConfig.DataSources.Count; $i++)
          {
             $tmpDiag = $diagConfig.DataSources[$i]
             if($tmpDiag.Path -eq $path)
             {
                $existingDirectory = $true
                $newDirectoryConfig = $diagConfig.DataSources[$i]
                break
             }
          }

          # doesn't already exist in the collection so create a new one
          if($existingDirectory -eq $false) 
          {
            $newDirectoryConfig = New-Object -TypeName Microsoft.WindowsAzure.Diagnostics.DirectoryConfiguration
          }
          $newDirectoryConfig.Container = $container
          $newDirectoryConfig.Path = $path
          $newDirectoryConfig.DirectoryQuotaInMB = $quota
          if($existingDirectory -eq $false)
          {
            $diagConfig.DataSources.Add($newDirectoryConfig)
          }
          
          $role | Set-FileBasedLog -DirectoriesConfiguration $diagConfig.DataSources -BufferQuotaInMB $quota -TransferPeriod $transferAllInMinutes
        }
    }
}

Now the step of actually configuring your role using all of this code.

The snippet below configures IIS logs, IIS failed request logs and a local resource named “customLogging” to be transferred to storage every 15 minutes.

Configuring Logs to be Transferred

 
GetDiagRoles | foreach {
  $CurrentRole = $_
  
  if($CurrentRole.RoleName -eq "MyWebRole")
  {
   # Retrieve the local paths in the role
   $iislogs = $CurrentRole | GetIISLogsPath
   $iisFailedLogs = $CurrentRole  | GetIISFailedRequestPath
   $customLogs = $CurrentRole | GetLocalResourcePath "customLogging"
   
   # Configure diagnostics to transfer log data to storage
   $CurrentRole | SetDirectory $iisFailedLogs "wad-iis-failedreqlogfiles"  0 15
   $CurrentRole | SetDirectory $iislogs "wad-iis-logfiles"  0 15
   $CurrentRole | SetDirectory $customLogs  "wad-customlogging"  0 15 
  }
}

The snippet below downloads the files from each container to the local file system.

Downloading the Data from Storage

 
Save-Container -ContainerName "wad-iis-logfiles" -LocalPath "c:\diagdata" `
			-StorageAccountKey $storagekey -StorageAccountName $storageAccount
			
Save-Container -ContainerName "wad-iis-failedreqlogfiles" -LocalPath "c:\diagdata" `
			-StorageAccountKey $storagekey -StorageAccountName $storageAccount
			
Save-Container -ContainerName "wad-customlogging" -LocalPath "c:\diagdata" `
			-StorageAccountKey $storagekey -StorageAccountName $storageAccount

Finally, to clean up storage from each container we provide you with the Clear-Container cmdlet.

Note: Currently, we do not provide a -From or -To parameter to Clear-Container. To ensure that you do not delete data that may currently be being logged
you will need to ensure your scheduled transfers do not overlap during the same time period of a deletion. Yes, we are looking into improving this scenario.

Cleaning up a Container

 
Clear-Container -ContainerName "wad-iis-logfiles"  `
			-StorageAccountKey $storagekey -StorageAccountName $storageAccount
			
Clear-Container -ContainerName "wad-iis-failedreqlogfiles" `
			-StorageAccountKey $storagekey -StorageAccountName $storageAccount
			
Clear-Container -ContainerName "wad-customlogging" `
			-StorageAccountKey $storagekey -StorageAccountName $storageAccount

This is part 3 of a series.

• Managing Tracing with Windows Azure PowerShell Cmdlets 2.0
• Managing Performance Counters with Windows Azure PowerShell Cmdlets 2.0
• Viewing your diagnostic settings with Windows Azure PowerShell Cmdlets 2.0
• Managing Windows Event Logs with Windows Azure PowerShell Cmdlets 2.0
• Managing Log Files with Windows Azure PowerShell Cmdlets 2.0

Here is a handy helper function to help you view what diagnostics settings are currently configured for your Windows Azure service.

(Tip: I add all of these helper functions to a PowerShell Module (.psm1) script, export functions and variables and include it in my other scripts)

Just as in the other articles you will need to add the PowerShell Snapin (or module):

 
  Add-PsSnapin WAPPSCmdlets 

Ditto, I have this handy initialization block and helper function GetDiagRoles:

Initialization and Helper Function

 
  $storageAccountName = "YourStorageAccountName" 
  $storageAccountKey = "YourStorageAccountKey" 
  $deploymentSlot = "Production" 
  $serviceName = "YourHostedService" 
  $subscriptionId = "YourSubscriptionId" 
  # Thumbprint for your cert goes below 
  $mgmtCert = Get-Item cert:\CurrentUser\My\D7BECD4D63EBAF86023BB4F1A5FBF5C2C924902A 

  function GetDiagRoles { 
    Get-HostedService -ServiceName $serviceName -SubscriptionId $subscriptionId -Certificate $mgmtCert | ` 
        Get-Deployment -Slot $deploymentslot | ` 
        Get-DiagnosticAwareRoles -StorageAccountName $storageAccountName -StorageAccountKey $storageAccountKey 
   } 

PrintDiagnosticSettings Helper Function

 
function PrintDiagnosticSettings
{
      $input | foreach {
      $RoleObject = $_
      $RoleObject | Get-DiagnosticAwareRoleInstances | 
      foreach {
          $roleinstance = $_
          write-host ""
          write-host -BackgroundColor DarkBlue -ForegroundColor White "Retrieving Diagnostic Settings for RoleInstance: " $roleinstance
          
          # Diagnostics Configuration for Logs
          $diagConfig = $RoleObject | Get-DiagnosticConfiguration -BufferName Logs -InstanceId $roleinstance
          write-host -BackgroundColor DarkGreen -ForegroundColor White "Windows Azure Logs - BufferQuota (MB): " $diagConfig.BufferQuotaInMB " LogLevel:" $diagConfig.ScheduledTransferLogLevelFilter " Scheduled Transfer to Storage: " $diagConfig.ScheduledTransferPeriod.ToString()
          write-host ""
          
          # Diagnostics Configuration for Diagnostic Infrastructure Logs
          $diagConfig = $RoleObject | Get-DiagnosticConfiguration -BufferName DiagnosticInfrastructureLogs -InstanceId $roleinstance
          write-host -BackgroundColor DarkGreen -ForegroundColor White "Diagnostic Infrastructure Logs - BufferQuota (MB): " $diagConfig.BufferQuotaInMB " LogLevel:" $diagConfig.ScheduledTransferLogLevelFilter " Scheduled Transfer to Storage: " $diagConfig.ScheduledTransferPeriod.ToString()
          write-host ""
          
          # Diagnostics Configuration for Performance Counters          
          $diagConfig = $RoleObject | Get-DiagnosticConfiguration -BufferName PerformanceCounters -InstanceId $roleinstance
          write-host -BackgroundColor DarkGreen -ForegroundColor White "Performance Counters - BufferQuota (MB): " $diagConfig.BufferQuotaInMB " Scheduled Transfer to Storage: " $diagConfig.ScheduledTransferPeriod.ToString()
          $diagConfig.DataSources | foreach {
            $CounterSpecifier = $_.CounterSpecifier
            $SampleRate = $_.SampleRate
            write-host "  Counter: " $CounterSpecifier " SampleRate: " $SampleRate
          }
          write-host ""
          
          # Diagnostics Configuration for Windows Event Logs          
          $diagConfig = $RoleObject | Get-DiagnosticConfiguration -BufferName WindowsEventLogs -InstanceId $roleinstance
          write-host -BackgroundColor DarkGreen -ForegroundColor White "Windows Event Logs - BufferQuota (MB): " $diagConfig.BufferQuotaInMB " LogLevel:" $diagConfig.ScheduledTransferLogLevelFilter  " Scheduled Transfer to Storage: " $diagConfig.ScheduledTransferPeriod.ToString()
          $diagConfig.DataSources | foreach {
            $logName = $_.ToString()
            write-host " Log Name: " $logName
          }
          write-host ""
          
          # Diagnostics Configuration for Directories          
          $diagConfig = $RoleObject | Get-DiagnosticConfiguration -BufferName Directories -InstanceId $roleinstance
          write-host -BackgroundColor DarkGreen -ForegroundColor White "Directories - BufferQuota (MB): " $diagConfig.BufferQuotaInMB " Scheduled Transfer to Storage: " $diagConfig.ScheduledTransferPeriod.ToString()
          $diagConfig.DataSources | foreach {
            $container = $_.Container
            $path = $_.Path
            write-host "  Path: " $path " Container: " $container " Quota (MB): " $diagConfig.BufferQuotaInMB 
          }
      }
    }
}

Usage

 
  GetDiagRoles | PrintDiagnosticSettings

Output Example