markwragg / powershell-influx Goto Github PK

reading vmware guest disk space and returing free / used space along with tagging vm name and drive path

example drive path is C:\ being returned from vmware, but when calling Write-Influx verbose shows Path=C:\\ now and when invoke-restmethod is called its is now c:\\ and flags invalid tag format\nunamble to parse

i can kind of fudge it by adding a white space after the path, but then they all end up in influx as C:\\ and not C:\

Thanks

Any restrictions on use?

Hi all,

I'm trying to get all Windows server services to write to InfluxDB but am experiencing a slight issue. I initially tried to do the following (I left the basic metrics in to see if it was writing) until I realised that $Metrics does not like nested hash tables.

#Influx Database HTTP Address.
$influxDB = "http://localhost:8086"
    # Get all of your metrics.
    While($true){
        $ValuesList = Get-Counter -Counter "\Memory\Available Bytes","\Processor(*)\% Processor Time"
        $MemoryActive = [math]::Round($ValuesList.CounterSamples[0].CookedValue,3) /1024 /1024
        $MemoryFreeMB = [math]::Round($MemoryActive)
        $ProcessorActive = [math]::Round($ValuesList.CounterSamples[1].CookedValue,2)
        $GetWindowsServices = Get-Service | select -Property Name,DisplayName,Status,StartType
        $Metrics = @{
            "CPU Active" = $ProcessorActive
            "Free RAM MB" = $MemoryFreeMB
            "Services" = $GetWindowsServices
    }
    # Send all Metrics to InfluxDB.
    Write-Influx -Measure Server -Tags @{Prod=$env:COMPUTERNAME} -Metrics $Metrics -Database telegraf -Server http://localhost:8086 -Verbose}

I was wondering if I could do a foreach on the $GetWindowsServices hash table to get this to work. I am learning PowerShell fairly quickly but this has me stumped.

All in all I'm hoping that if this works for the Services then I can use the same format for Windows Event Logs.

Thanks

P.s. I'm very new to github and couldn't find how to add this as a request rather than an issue.

Hi,
Would it be possible to add something to Write-Influx where instead of writing to the database, it outputs in the line protocol format that could be utilised by Telegrafs exec plugin?
I am writing various scripts right now and looking for an easy way to switch them between running under Task Scheduler (ie. direct to DB) and under Telegraf.
Thanks

hello

i saw that the tags are not sorted by key, wher in the doc, they say,

For best performance you should sort tags by key before sending them to the database. The sort should match the results from the Go bytes.Compare function.

https://docs.influxdata.com/influxdb/v1.7/write_protocols/line_protocol_tutorial/

Regards
Tonic8

The previous image is no longer available.

Is there a way to force the format of sent requests to be an Integer or a String? Im having some issues when some of my values are a mixed of INT / String's like this when doing a ForEach loop.

The JobName Var sometimes has values that start like 00250-Name. I I know with influx I can do an "I" or a double quote

$JobName =$_.BackupJobName
$Date =$_.Date
$Name =$_.Name
$Status =$_.Status

Write-Influx -Measure backup_info -Tags @{VMname=$Name} -Metrics @{Date=$Date;Status=$Status;JobName=$JobName} -Database it-stats -Server http://pap-graf01:8086 -Verbose

Hello

on my last usages i got "random" errors in the result formatting. When sending thousands of lines it's hard to see which line got a return HTTP error 400.

It could be useful to get a error debug possibility, like write errors in a file (the data sent and the error code sent back)
like a try and catch and send the data to a variable we can then write in a log file when needed.

Tonic8
(sorry as i'm a newbie on GitHub i don't know how to set this as "enhancement" and not an issue)

Hello

• we cannot use the command line
  write-influx -inputObject $data -Database "Data" -Server "Server"

• the command cannot even be used in splatting

$splat = @{
  InputObject = $Data
  Database = "Data"
  Server = "Server"
}
write-influx @splat

• We can mix the concepts but the InputObject must be passed by the Pipe

$splat = @{
  Database = "Data"
  Server = "Server"
}
$Data |write-influx @splat

or

$Data | write-influx -Database "Data" - Server "Server"

When doing that the error message is:

Cannot convert the "System.Object[]" value of type "System.Object[]" to type "System.Collections.Hashtable".
At c:\test\_modules\Influx\Public\Write-Influx.ps1:101 char:13
+                  $Metrics = $InputObject.Metrics
+                  ~~~~~~~~~~~~~~~~~~~~~
       + CategoryInfo             : Invalid Argument: (:) [], RuntimeException
       + FullyQualifiedErrorId : ConvertToFinalInvalidCastException

Thanks
Tonic8

Any plans or ETA for supporting InfluxDB 2 also? It is in only minor changes to the REST query for posting afaik.

Hello Mark,

Was using Write-Influx successfully to upload data to local InfluxDb 1.7.9. on Windows.

Used the same command to upload to remote server (Unix & cloud based using https and authentication) which did not work. FOr testing I tried -Whatif and saw this unexpected output:
The unexpected thing is that I see multiple lines (one for each Metric value where I was expecting multiple columns and only one line/record. This behaviour was the same for localhost and remote server. Still not sure why remote does not work (Error 400).
Is this an issue with the -Whatif statement or is it really sending out one line per Metric (that would be very ineffecient in regard to DB updates and traffic)

PS C:\Users> Write-Influx -Measure $Measurement -Tags $Tags -Metrics $Metrics -TimeStamp $InfluxTimeStampUTC -Database $Database -Server http://localhost:8086 -whatif
What if: Performing the operation "azo,ChargeID=428569,RunID=51-000022,WorkCtr=0551,Shift_ID=2020-01-27.07,Shift=O,ShopfloorDatum=2020-01-27 ChargeStart="2020-01-27 08:09:03" 1580108943000000000
azo,ChargeID=428569,RunID=51-000022,WorkCtr=0551,Shift_ID=2020-01-27.07,Shift=O,ShopfloorDatum=2020-01-27 Recipe="S41351v1/2" 1580108943000000000
azo,ChargeID=428569,RunID=51-000022,WorkCtr=0551,Shift_ID=2020-01-27.07,Shift=O,ShopfloorDatum=2020-01-27 Weight_pv=0.499 1580108943000000000
azo,ChargeID=428569,RunID=51-000022,WorkCtr=0551,Shift_ID=2020-01-27.07,Shift=O,ShopfloorDatum=2020-01-27 Weight_sv=0.5 1580108943000000000
azo,ChargeID=428569,RunID=51-000022,WorkCtr=0551,Shift_ID=2020-01-27.07,Shift=O,ShopfloorDatum=2020-01-27 ChargeEnd="2020-01-27 08:22:11" 1580108943000000000
azo,ChargeID=428569,RunID=51-000022,WorkCtr=0551,Shift_ID=2020-01-27.07,Shift=O,ShopfloorDatum=2020-01-27 RecipeName="Middenlaag B" 1580108943000000000
azo,ChargeID=428569,RunID=51-000022,WorkCtr=0551,Shift_ID=2020-01-27.07,Shift=O,ShopfloorDatum=2020-01-27 ArticleID="G10236" 1580108943000000000
azo,ChargeID=428569,RunID=51-000022,WorkCtr=0551,Shift_ID=2020-01-27.07,Shift=O,ShopfloorDatum=2020-01-27 ArticleDesc1="101395" 1580108943000000000" on target "http://localhost:8086/write?&db=afp".

PS C:\Users> $Metrics
Name Value

ChargeStart 2020-01-27 08:09:03
Recipe S41351v1/2
Weight_pv 0,499
Weight_sv 0,5
ChargeEnd 2020-01-27 08:22:11
RecipeName Middenlaag B
ArticleID G10236
ArticleDesc1 101395

PS C:\Users> $Tags

Name Value

ChargeID 428569
RunID 51-000022
WorkCtr 0551
Shift_ID 2020-01-27.07
Shift O
ShopfloorDatum 2020-01-27

Hi
I forked your really cool project and noticed following in your private PowerShell-function "Influx/Private/Out-InfluxEscapeString.ps1":

• Escaping of '' is not needed (InfluxLine protocol docs: line protocol does not require users to escape the backslash character \ but will not complai)
• Missing whitespace escaping in 'FieldTextValue'

your code in "Influx/Private/Out-InfluxEscapeString.ps1":

"Measurement" { $String -Replace '(\s|,|\\)', '\$1' }
"FieldTextValue" { $String -Replace '("|\\)', '\$1' }
"Other" { $String -Replace '(\s|=|,|\\|")', '\$1' }
default { $String -Replace '(\s|=|,|\\|")', '\$1' }

my changes in "Influx/Private/Out-InfluxEscapeString.ps1":

"Measurement" { $String -Replace '(\s|,)', '\$1' }
"FieldTextValue" { $String -Replace '(\s|")', '\$1' }
"Other" { $String -Replace '(\s|=|,|")', '\$1' }
default { $String -Replace '(\s|=|,|")', '\$1' }

-> More informations about the changes.

PS: Excuse me for not making an pull request, but nevertheless I hope it helps.

Hello Mark,

Many thanks for your development of the InfluxDb module. It is the best by far and makes working with Influx from scripts a lot easier.

I ran into this behaviour that I would like to verify because it is different from the intention.
Expected -ExcludeEmptymetric to not pass Metric TestValue but in my test it received the value of 38 (Same as Metric 'Mem')

Thanks,
GWS65

*************** Test *******************

$Tags = @{
Server2 = 'ServerTag2'
Location = 'Anywhere'
}

$Metrics = @{
CPU = 44
Mem = 38
TestValue = $null
}

$InfluxLine = ConvertTo-InfluxLineString -Measure WebServer -Tags $Tags -Metrics $Metrics -ExcludeEmptyMetric

Hi, Trying to send zeros using Write-Influx fails as it drops that part of the hash, simple example below.

Many thanks for writing this great module

Write-Influx -Measure WebServer -Tags @{Server='Host01'} -Metrics @{CPU=100; Memory=50} -Database db1 -Server http://127.0.0.1:8086 -Verbose

VERBOSE: Performing the operation "WebServer,Server=Host01 CPU=100
WebServer,Server=Host01 Memory=50 " on target "http://127.0.0.1:8086/write?&db=db1 ".
VERBOSE: POST http://127.0.0.1:8086/write?&db=db1 with -1-byte payload
VERBOSE: received -1-byte response of content type application/json

Write-Influx -Measure WebServer -Tags @{Server='Host01'} -Metrics @{CPU=100; Memory=0} -Database db1 -Server http://127.0.0.1:8086 -Verbose

VERBOSE: Performing the operation "WebServer,Server=Host01 CPU=100 " on target "http://127.0.0.1:
8086/write?&db=db1 ".
VERBOSE: POST http://127.0.0.1:8086/write?&db=db1 with -1-byte payload
VERBOSE: received -1-byte response of content type application/json

Same command just removed the 5 from the memory parameter on 2nd run

Hello

thanks for the work done :). I have now a limitation:

some object property can have null value, the problem is that when this property is exported there will be conflict on the type of data.

Example with Citrix Xendesktop Powershell sdk, what is actually sent:

Xendesktop,computer1 brokeredTime=7, CatalogName="USA1_aa" 12345678898
Xendesktop,computer1 brokeredTime="", CatalogName="USA1_aa" 12345678898

the data is either int or string :)

it could be cool to have a option to set write-influx -emptyValueDoNotsent (please find something better for the name of the option :) ), with a result of:

Xendesktop,computer1 brokeredTime=7, CatalogName="USA1_aa" 12345678898
Xendesktop,computer1 CatalogName="USA1_aa" 12345678898

In this case, zero would not work as the field with no data represent something:

• no data mean no event -> useful to check if a ressource is used or not
• 0 representing user connected in the last hour

Thanks
Tonic8

Hello

$InputObject and Measure got the same position number,

        [Parameter(ParameterSetName = 'MetricObject', ValueFromPipeline = $True, Position = 0)]
        [PSTypeName('Metric')]
        $InputObject,

        [Parameter(ParameterSetName = 'Measure', Mandatory = $true, Position = 0)]
        [string]
        $Measure,

for example we should have for Measure "Position = 1"

Regards
Tonic8

Hello

using an inputobject (generated by ConvertTo-Metric), we can't specify a timestamp (PSDate).
Consequence the timestamp it'sq when the command is executed

so in a script if we execute 3 time convertto-metric and write-influx, because we collect the data from 3 different source, the timestamp will vary between the 3 collect.

Regards
Tonic8

Hello

i finally catch the issue i saw with some data, here the data sent
xdsession,desktopgroupename=spain-prod,PowerState=On,launchedViaHostName=,hypervizor=vcenter1 LogonInProgress="False"

"launchedViaHostname=" had not value, and all the line is not written to the DB.

it should be usefull to not send the empty tag.
REgards
Tonic8

If the command "Write-Influx" is used without tags, an error occurs.

Working Code

Get-Process -Name winlogon | ConvertTo-Metric -Measure test -MetricProperty CPU,PagedmemorySize -TagProperty Handles,Id,ProcessName | Write-Influx -Bucket test -Server https://influxserver:8086 -Token "token" -Verbose

Not working Code

Get-Process -Name winlogon | ConvertTo-Metric -Measure test -MetricProperty CPU,PagedmemorySize  | Write-Influx -Bucket test -Server https://influxserver:8086 -Token "token" -Verbose

Error

Invoke-RestMethod : {"code":"invalid","message":"unable to parse 'test, CPU=\"\" ': missing tag key\nunable to parse 'test, PagedmemorySize=233472 ': missing tag key"}
In C:\Users\ant\Documents\WindowsPowerShell\Modules\Influx\1.0.101\Public\Write-Influx.ps1:202 Zeichen:25
+ ...             Invoke-RestMethod -Uri $URI -Method Post -Body $Body -Hea ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

The error is caused by setting an incorrect comma in the function
According to the following issue, no comma may be present if no tags are transmitted
influxdata/influxdb#3600

In the code, you are using
if ($MetricObject.Tags) { #Build String for Tags }

But $MetricObject.Tags is an emptry hashtable and an empty hastable evaluates "true".
PowerShell/PowerShell#10201

You have to change the code from

  if ($MetricObject.Tags)) {
                $TagData = foreach ($Tag in $MetricObject.Tags.Keys) {
                    if ([string]::IsNullOrEmpty($MetricObject.Tags[$Tag])) {
                        Write-Warning "$Tag skipped as it's value was null or empty, which is not permitted by InfluxDB."
                    }
                    else {
                        "$($Tag | Out-InfluxEscapeString)=$($MetricObject.Tags[$Tag] | Out-InfluxEscapeString)"
                    }
                }
                $TagData = $TagData -Join ','
                $TagData = ",$TagData"
            }

to

  if (($MetricObject.Tags).count -ne 0) {
                $TagData = foreach ($Tag in $MetricObject.Tags.Keys) {
                    if ([string]::IsNullOrEmpty($MetricObject.Tags[$Tag])) {
                        Write-Warning "$Tag skipped as it's value was null or empty, which is not permitted by InfluxDB."
                    }
                    else {
                        "$($Tag | Out-InfluxEscapeString)=$($MetricObject.Tags[$Tag] | Out-InfluxEscapeString)"
                    }
                }
                $TagData = $TagData -Join ','
                $TagData = ",$TagData"
            }

By checking the code gound this if clause.

  if ($MetricObject.Metrics[$Metric] -isnot [ValueType]) { 
                        $MetricValue = '"' + $MetricObject.Metrics[$Metric] + '"'
                    }
                    else {
                        $MetricValue = $MetricObject.Metrics[$Metric] | Out-InfluxEscapeString
                    }
            
                    "$($MetricObject.Measure | Out-InfluxEscapeString)$TagData $($Metric | Out-InfluxEscapeString)=$MetricValue $timeStampNanoSecs"

Can you explain the $MetricObject.Metrics[$Metric] -isnot [ValueType] part?
By testing, I found out that it checks which data types are used. However, I do not quite understand how

some further suggestions from @Tonic8:

• if a field in an object is a DateType, you can convert it in something InfluxDB will understand:

(dir | select -first1).lastWriteTime | gm
(dir | select -first1).lastWriteTime | get-date -f "yyyy-MM-dd HH:mm:ss.fff"
(dir | select -first1).lastWriteTime | convertTo-UnixTimeNanoseconds

where fff is millisecond, (talked here https://stackoverflow.com/questions/28589520/how-to-store-dates-in-influxdb) and here https://docs.influxdata.com/influxdb/v1.5/query_language/data_exploration/#time-syntax

• if a field in an object is something not a "string" but a internal field, like this in Citrix sdk (extract of get-brokerSession | GM):

ConnectionViaIP   NoteProperty string ConnectedViaIP=1.2.3.4
ConnectionMode    NoteProperty Citrix.Broker.Admin.SDK.ConnectionMode ConnectionMode=Brokered

• if you pass it, the conversion is going wrong(missing doublequote), user must prepare the field, to convert it as string, before passing it to convertTo-Metric
• example with a Boolean field (it should have double-quote),
  $test = New-Object PSObject -Property @{testBoolean=$True}
• below the result from the Write-Influx commande in Verbose
  Xendesktop,computer1 testBoolean=true 12345678898

Originally posted by @Tonic8 in #15 (comment)

Thank you for creating this. It helped me a lot in a pinch.

I spent a few days working on collecting my own metrics and reading write-influx several times over to understand how and why everything was set up the way it was, and one thing became really apparent as a missing feature. There isn't an ability to send more than one Measure to influx all in one request (mostly impacting TCP since it's so much slower). If you are storing all of your data in one Measure then there's no issue, but I doubt that's the case for many people. And since Influx supports accepting more than one measure in a single TCP packet, this certainly seems do-able.

I was wondering if this is something you've ever looked at or worked on? I was about to start on a PR for it when someone convinced me that I better check with the author first to see if this is something he would even commit.

Implementation
I was thinking of adding a function like 'Append-InfluxQueue' (or Add-...) which appends data onto a list. Basically all of the params like InputObject, Measure, Metrics, and Tags would be moved from Write-Influx onto Append-InfluxQueue and Write-Influx would become a simple writer (also looping through the list turning it into text). The same is true for UDP as well of course.

Thoughts on this?

P.S. Ever considered building in some windows metrics collectors?

I'm writing a fairly straight forward powershell script for getting the dhcp leases from my pihole instance and displaying them for Grafana.

#!/usr/bin/pwsh
$Leases = (ssh -l user pihole "cat /etc/pihole/dhcp.leases")

$metrics = Foreach ($obj in $Leases) {
    [hashtable]@{
        $obj.Split(" ")[3] = $obj.Split(" ")[2]
    }
}

$InfluxParams = @{
    Measure = 'pihole_dhcp_leases'
    Metrics = $metrics
    Database = 'MyDB'
    Server   = 'http://<influxdb.example.com>:8086'
    Verbose = $true
    Credential = (Get-Credential)
}

Write-Influx @InfluxParams

As an example output, $leases comes in the following format:

1536231722 11:11:11:11:11:11 192.168.x.x host01 11:11:11:11:11:11:11
1536205906 11:11:11:11:11:11 192.168.x.x host02 11:11:11:11:11:11:11

I see where Metrics is properly typed:

$metrics

Name                           Value
----                           -----
host01                     192.168.x.x
host02                     192.168.x.x

$metrics | Get-Member

    TypeName: System.Collections.Hashtable

However, Write-Influx errors at:

Write-Influx : Cannot process argument transformation on parameter 'Metrics'. Cannot convert the "System.Object[]" value of type "System.Object[]" to type "System.Collections.Hashtable".

Is there any explicit typecasting that might interfere here?

It appears that this module only lets you submit a single data point at a time. Is there a method to insert multiple points per request?

Perhaps accept an object in this format : @{ Measure = ''; Tags = @{}; Metrics = @{}; TimeStamp = '' }

in the write-influx function the description of "Timestamp" parameter is missing in the comment/parameters block.

regards
Tonic8

We are using a username and password to connect to our InfluxDB. The function Write-Influx does not include options/parameters to parse username and password to the body part. Are you able to add this functionality?
Thank you
Torben

I have installed your module as well as the TFS Module which you are consuming to make the TFS related calls.

It seems that the underlying TFS Module relies upon instantiating the following global variable before importing the module (either implicitly or explicitly):

$global:tfs=@{
    root_url='https://sometfsserver.com'
    collection='CollectionName'
    project='ProjectName'
}

Per your guide at http://wragg.io/windows-based-grafana-analytics-platform-via-influxdb-and-powershell/ you show calling

Get-TFSBuildMetric -TFSRootURL 'https://mytfsurl.local/tfs' -TFSCollection somecollection -TFSProject someproject

However that call does not return as expected since nowhere in the Get-TFSBuildMetric code do you create/set the $global:tfs variable based upon the parameters passed to Get-TFSBuildMetric. Am I not understanding how this should work? Or is this broken due to changes in the underlying TFS Module? If I have to manually create the global variables, what then is the point of the Parameters being passed to Get-TFSBuildMetric?