Netatmo weather station data ingestion to Azure Log Analytics

At the time of writing this blog post, the Azure Log Analytics HTTP Data Collector API was released only for a few weeks. I wanted to showcase the possibilities of data and log ingestion into the Log Analytics platform, so I created a demo to prove that we could ingest data other than operating system and IT log data, and do something with it.

Problem

Ingest data from a Netatmo Weather Station to Azure Log Analytics and present data visually appealing.

Solution

The Netatmo Weather Station consists of two sensors, one indoor and one outdoor, to measure environmental metrics like temperature, humidity, CO2, etc.  The general setup of a netatmo is downloading the mobile app to your phone, creating a new Netatmo account and then connecting the weather station to your account.  Once this is done, the weather station uploads sensor readings every X few minutes (Not sure how often, I’m guessing every 5 minutes) to Netatmo.  The app on your phone queries the netatmo service, which displays rich dashboards of the data.

This then also meant that you cannot read data from the sensors directly, however Netatmo exposes an API through their development program Netatmo Connect, to read data from your sensors and systems on your account.

Thus, the high level steps for this solution was:

1. Create a new app on Netatmo Connect
2. Write a Powershell script to read the sensor readings from the Netatmo API and upload into the Log Analytics HTTP collector Api
3. Create some dashboards.

Netatmo Connect

Navigate to at https://dev.netatmo.com/dev/createanapp and create a new app.  This will provide you with a ClientID and a Client Secret. Keep for later.

Powershell Script

I used the powershell script used in the examples section in the official Log Analytics HTTP Data Collector API documentation, and added the netatmo api reads.  This is the code that did all the work.  Note that you need to change all the fields at the top that’s marked ‘xxxxxxxxxxxxxxxxx’with your own config data.  Other than that, you can pretty much copy and paste into something likePowershell ISE and run and it should work.

The short explanation of what the script does is this:

• Auth to Netatmo
• Read sensor data from API
• Deconstruct the json response from Netatmo into a powershell object
• Correlate the different measurements and add a new log entry for each sensor and its values.

Code:

#------------------------------------------------------------------------------  
# THIS CODE AND ANY ASSOCIATED INFORMATION ARE PROVIDED “AS IS” WITHOUT  
# WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING BUT NOT  
# LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY AND/OR FITNESS  
# FOR A PARTICULAR PURPOSE. THE ENTIRE RISK OF USE, INABILITY TO USE, OR   
# RESULTS FROM THE USE OF THIS CODE REMAINS WITH THE USER.  
#------------------------------------------------------------------------------  
$global:access_token = $null
$global:refresh_token = $null
#Netatmo Details. Client ID and secret from creating the App, and a username and password that has access to the api.
$global:client_id='xxxxxxxxxxxxxxxxxxxxxxxxxxx'
$global:client_secret='xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
$global:Netatmo_username='xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
$global:Netatmo_password='xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'

#OMS Workspace Variables
$global:CustomerId = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" #Workspace ID
$global:SharedKey = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
$global:LogType = "Netatmo_WeatherStation" #Create a Log Type name

#Function to send data to OMS workspace
function WriteOMSEvent($SensorID, $SensorName, $MeasurementName, $MeasurementValue){
        #Specify a time in the format YYYY-MM-DDThh:mm:ssZ to specify a created datetime for the records.
        $TimeStampField = [DateTime]::UtcNow.ToString("r")

#Create json record
$json = @"
[{  "Sensor_id": "$SensorID",
    "Sensor_name": "$SensorName",
    "Measurement_name": "$MeasurementName",
    "Value": $MeasurementValue
}]
"@

        #Output the json to console so that we can see the results.
        $json

                # Function to create the authorization signature.
                Function Build-Signature ($customerId, $SharedKey, $date, $contentLength, $method, $contentType, $resource)
                {
                    $xHeaders = "x-ms-date:" + $date
                    $stringToHash = $method + "`n" + $contentLength + "`n" + $contentType + "`n" + $xHeaders + "`n" + $resource

                    $bytesToHash = [Text.Encoding]::UTF8.GetBytes($stringToHash)
                    $keyBytes = [Convert]::FromBase64String($SharedKey)

                    $sha256 = New-Object System.Security.Cryptography.HMACSHA256
                    $sha256.Key = $keyBytes
                    $calculatedHash = $sha256.ComputeHash($bytesToHash)
                    $encodedHash = [Convert]::ToBase64String($calculatedHash)
                    $authorization = 'SharedKey {0}:{1}' -f $customerId,$encodedHash
                    return $authorization
                }


                # Function doing the post to the OMS Workspace.
                Function Post-OMSData($customerId, $SharedKey, $body, $LogType) 
                {
                    $method = "POST"
                    $contentType = "application/json"
                    $resource = "/api/logs"
                    $rfc1123date = [DateTime]::UtcNow.ToString("r")
                    $contentLength = $body.Length
                    $signature = Build-Signature `
                        -customerId $customerId `
                        -sharedKey $SharedKey `
                        -date $rfc1123date `
                        -contentLength $contentLength `
                        -fileName $fileName `
                        -method $method `
                        -contentType $contentType `
                        -resource $resource
                    $uri = "https://" + $customerId + ".ods.opinsights.azure.com" + $resource + "?api-version=2016-04-01"

                    $headers = @{
                        "Authorization" = $signature;
                        "Log-Type" = $LogType;
                        "x-ms-date" = $rfc1123date;
                        "time-generated-field" = $TimeStampField;
                    }

                $response = Invoke-WebRequest -Uri $uri -Method $method -ContentType $contentType -Headers $headers -Body $body -UseBasicParsing
                return $response.StatusCode
            } 

            # The only function after creating the json document that's doing the work.
            Post-OMSData -customerId $customerId -sharedKey $SharedKey -body ([System.Text.Encoding]::UTF8.GetBytes($json)) -logType $LogType  
            }
#End of Function WriteOMSEvent

#Authenticate to the Netatmo API and get an access token.
function Netatmo-DoAuth {
        $postParams = @{
                        grant_type='password';
                        client_id=$client_id;
                        client_secret=$client_secret;
                        username=$Netatmo_username;
                        password=$Netatmo_password
                        }

        $Request = Invoke-WebRequest -Uri https://api.netatmo.com/oauth2/token -Method POST -Body $postParams -UseBasicParsing
        if ($Request.StatusCode -eq 200) {
            $ResponseJson = $Request.Content | ConvertFrom-Json
            $global:access_token = $ResponseJson.access_token
            $global:refresh_token =  $ResponseJson.refresh_token
        }
        ELSE
        { 
         write-error -message $Request.StatusDescription -ErrorId $Request.StatusCode
        }
}

#Read the Netatmo API for Sensor data, convert into an object, and post to OMS.
function ReadSensorData{$SensorpostParams = @{access_token=$access_token}                
        #Request Sensor data from Netatmo
        $SensorRequest = Invoke-WebRequest -Uri https://api.netatmo.com/api/getstationsdata -Method POST -Body $SensorpostParams -UseBasicParsing
        if ($SensorRequest.StatusCode -eq 200) {
            #Convert to object.               
            $SensorContent = $SensorRequest.Content | ConvertFrom-Json

            #Indoor unit sensors
            $MainUnit_id = $SensorContent.body.devices[0]._id
            $MainUnit_module_name = $SensorContent.body.devices[0].module_name
            $MainUnit_wifi_status = $SensorContent.body.devices[0].wifi_status
            $MainUnit_Pressure = $SensorContent.body.devices[0].dashboard_data.Pressure 
            $MainUnit_pressure_trend = $SensorContent.body.devices[0].dashboard_data.pressure_trend
            $MainUnit_Temperature = $SensorContent.body.devices[0].dashboard_data.Temperature
            $MainUnit_temp_trend = $SensorContent.body.devices[0].dashboard_data.temp_trend
            $MainUnit_Humidity = $SensorContent.body.devices[0].dashboard_data.Humidity 
            $MainUnit_CO2 = $SensorContent.body.devices[0].dashboard_data.CO2 
         
            WriteOMSEvent -SensorID $MainUnit_id -SensorName $MainUnit_module_name -MeasurementName "Pressure" -MeasurementValue $MainUnit_Pressure 
            WriteOMSEvent -SensorID $MainUnit_id -SensorName $MainUnit_module_name -MeasurementName "Wifi Signal" -MeasurementValue $MainUnit_wifi_status 
            WriteOMSEvent -SensorID $MainUnit_id -SensorName $MainUnit_module_name -MeasurementName "Temperature" -MeasurementValue $MainUnit_Temperature 
            WriteOMSEvent -SensorID $MainUnit_id -SensorName $MainUnit_module_name -MeasurementName "Humidity" -MeasurementValue $MainUnit_Humidity
            WriteOMSEvent -SensorID $MainUnit_id -SensorName $MainUnit_module_name -MeasurementName "CO2" -MeasurementValue $MainUnit_CO2 

            #Outdoor Unit Sensors
            $OutdoorUnit_id = $SensorContent.body.devices[0].modules[0]._id
            $OutdoorUnit_module_name = $SensorContent.body.devices[0].modules[0].module_name
            $OutdoorUnit_Temperature =  $SensorContent.body.devices[0].modules[0].dashboard_data.Temperature -as [double]
            $OutdoorUnit_temp_trend =  $SensorContent.body.devices[0].modules[0].dashboard_data.temp_trend
            $OutdoorUnit_Humidity =  $SensorContent.body.devices[0].modules[0].dashboard_data.Humidity -as [double]

            WriteOMSEvent -SensorID $OutdoorUnit_id -SensorName $OutdoorUnit_module_name -MeasurementName "Temperature" -MeasurementValue $OutdoorUnit_Temperature 
            WriteOMSEvent -SensorID $OutdoorUnit_id -SensorName $OutdoorUnit_module_name -MeasurementName "Humidity" -MeasurementValue $OutdoorUnit_Humidity 

        }
        ELSE
        {
            if ($SensorRequest.StatusCode -eq 403) {
                write-error -message "Netatmo Unauthorized, refreshing access token"
                Netatmo-DoAuth
                ReadSensorData
                }

         write-error -message $Request.StatusDescription -ErrorId $Request.StatusCode
        }
}

#Call the two main functions to do the work
Netatmo-DoAuth
ReadSensorData

Using Azure automation to run script on a schedule

I wanted to be able to run the script once an hour automatically, which I can do on my workstation, but I decided to do it on an Azure Automation account in the free sku… because why not.

To get this done, I created a new Azure automation account:

 

Once this was done, I created a new runbook, pasted my powershell into it, and setup a schedule running once an hour.

Tested the script, which would show like this if successful:

Over to Log Analytics

After about an hour of sending events over to Log Analytics, you should get the customer fields being in the settings section:

 

Once these fields show, you should see all fields when doing a search:

 

and that pretty much proves the concept.  However I wanted to make it look a bit more awesome, so I created an extra custom solution to make my data look great

and drilling down to the solution looks like this.

and once you click on a metric, it shows details, also in a graph.

If you would like to know how I created the custom solution, let me know and I’ll add a blog post about it.  Also, if you would like to get notified of new posts, please subscribe to my blog top left of the page.