markwragg / powershell-influx Goto Github PK
View Code? Open in Web Editor NEWA PowerShell module for interacting with the time-series database platform Influx: https://www.influxdata.com/
License: GNU General Public License v3.0
A PowerShell module for interacting with the time-series database platform Influx: https://www.influxdata.com/
License: GNU General Public License v3.0
reading vmware guest disk space and returing free / used space along with tagging vm name and drive path
example drive path is C:\ being returned from vmware, but when calling Write-Influx verbose shows Path=C:\\
now and when invoke-restmethod is called its is now c:\\ and flags invalid tag format\nunamble to parse
i can kind of fudge it by adding a white space after the path, but then they all end up in influx as C:\\
and not C:\
Thanks
Any restrictions on use?
Hi all,
I'm trying to get all Windows server services to write to InfluxDB but am experiencing a slight issue. I initially tried to do the following (I left the basic metrics in to see if it was writing) until I realised that $Metrics
does not like nested hash tables.
#Influx Database HTTP Address.
$influxDB = "http://localhost:8086"
# Get all of your metrics.
While($true){
$ValuesList = Get-Counter -Counter "\Memory\Available Bytes","\Processor(*)\% Processor Time"
$MemoryActive = [math]::Round($ValuesList.CounterSamples[0].CookedValue,3) /1024 /1024
$MemoryFreeMB = [math]::Round($MemoryActive)
$ProcessorActive = [math]::Round($ValuesList.CounterSamples[1].CookedValue,2)
$GetWindowsServices = Get-Service | select -Property Name,DisplayName,Status,StartType
$Metrics = @{
"CPU Active" = $ProcessorActive
"Free RAM MB" = $MemoryFreeMB
"Services" = $GetWindowsServices
}
# Send all Metrics to InfluxDB.
Write-Influx -Measure Server -Tags @{Prod=$env:COMPUTERNAME} -Metrics $Metrics -Database telegraf -Server http://localhost:8086 -Verbose}
I was wondering if I could do a foreach
on the $GetWindowsServices
hash table to get this to work. I am learning PowerShell fairly quickly but this has me stumped.
All in all I'm hoping that if this works for the Services then I can use the same format for Windows Event Logs.
Thanks
P.s. I'm very new to github and couldn't find how to add this as a request rather than an issue.
Hi,
Would it be possible to add something to Write-Influx where instead of writing to the database, it outputs in the line protocol format that could be utilised by Telegrafs exec plugin?
I am writing various scripts right now and looking for an easy way to switch them between running under Task Scheduler (ie. direct to DB) and under Telegraf.
Thanks
hello
i saw that the tags are not sorted by key, wher in the doc, they say,
For best performance you should sort tags by key before sending them to the database. The sort should match the results from the Go bytes.Compare function.
https://docs.influxdata.com/influxdb/v1.7/write_protocols/line_protocol_tutorial/
Regards
Tonic8
PowerShell-Influx/azure-pipelines.yml
Line 7 in a1ba7b2
The previous image is no longer available.
Is there a way to force the format of sent requests to be an Integer or a String? Im having some issues when some of my values are a mixed of INT / String's like this when doing a ForEach loop.
The JobName Var sometimes has values that start like 00250-Name. I I know with influx I can do an "I" or a double quote
$JobName =$_.BackupJobName
$Date =$_.Date
$Name =$_.Name
$Status =$_.Status
Write-Influx -Measure backup_info -Tags @{VMname=$Name} -Metrics @{Date=$Date;Status=$Status;JobName=$JobName} -Database it-stats -Server http://pap-graf01:8086 -Verbose
Hello
on my last usages i got "random" errors in the result formatting. When sending thousands of lines it's hard to see which line got a return HTTP error 400.
It could be useful to get a error debug possibility, like write errors in a file (the data sent and the error code sent back)
like a try and catch and send the data to a variable we can then write in a log file when needed.
Tonic8
(sorry as i'm a newbie on GitHub i don't know how to set this as "enhancement" and not an issue)
Hello
we cannot use the command line
write-influx -inputObject $data -Database "Data" -Server "Server"
the command cannot even be used in splatting
$splat = @{
InputObject = $Data
Database = "Data"
Server = "Server"
}
write-influx @splat
$splat = @{
Database = "Data"
Server = "Server"
}
$Data |write-influx @splat
or
$Data | write-influx -Database "Data" - Server "Server"
When doing that the error message is:
Cannot convert the "System.Object[]" value of type "System.Object[]" to type "System.Collections.Hashtable".
At c:\test\_modules\Influx\Public\Write-Influx.ps1:101 char:13
+ $Metrics = $InputObject.Metrics
+ ~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : Invalid Argument: (:) [], RuntimeException
+ FullyQualifiedErrorId : ConvertToFinalInvalidCastException
Thanks
Tonic8
Any plans or ETA for supporting InfluxDB 2 also? It is in only minor changes to the REST query for posting afaik.
Hello Mark,
Was using Write-Influx successfully to upload data to local InfluxDb 1.7.9. on Windows.
Used the same command to upload to remote server (Unix & cloud based using https and authentication) which did not work. FOr testing I tried -Whatif and saw this unexpected output:
The unexpected thing is that I see multiple lines (one for each Metric value where I was expecting multiple columns and only one line/record. This behaviour was the same for localhost and remote server. Still not sure why remote does not work (Error 400).
Is this an issue with the -Whatif statement or is it really sending out one line per Metric (that would be very ineffecient in regard to DB updates and traffic)
PS C:\Users> Write-Influx -Measure $Measurement -Tags $Tags -Metrics $Metrics -TimeStamp $InfluxTimeStampUTC -Database $Database -Server http://localhost:8086 -whatif
What if: Performing the operation "azo,ChargeID=428569,RunID=51-000022,WorkCtr=0551,Shift_ID=2020-01-27.07,Shift=O,ShopfloorDatum=2020-01-27 ChargeStart="2020-01-27 08:09:03" 1580108943000000000
azo,ChargeID=428569,RunID=51-000022,WorkCtr=0551,Shift_ID=2020-01-27.07,Shift=O,ShopfloorDatum=2020-01-27 Recipe="S41351v1/2" 1580108943000000000
azo,ChargeID=428569,RunID=51-000022,WorkCtr=0551,Shift_ID=2020-01-27.07,Shift=O,ShopfloorDatum=2020-01-27 Weight_pv=0.499 1580108943000000000
azo,ChargeID=428569,RunID=51-000022,WorkCtr=0551,Shift_ID=2020-01-27.07,Shift=O,ShopfloorDatum=2020-01-27 Weight_sv=0.5 1580108943000000000
azo,ChargeID=428569,RunID=51-000022,WorkCtr=0551,Shift_ID=2020-01-27.07,Shift=O,ShopfloorDatum=2020-01-27 ChargeEnd="2020-01-27 08:22:11" 1580108943000000000
azo,ChargeID=428569,RunID=51-000022,WorkCtr=0551,Shift_ID=2020-01-27.07,Shift=O,ShopfloorDatum=2020-01-27 RecipeName="Middenlaag B" 1580108943000000000
azo,ChargeID=428569,RunID=51-000022,WorkCtr=0551,Shift_ID=2020-01-27.07,Shift=O,ShopfloorDatum=2020-01-27 ArticleID="G10236" 1580108943000000000
azo,ChargeID=428569,RunID=51-000022,WorkCtr=0551,Shift_ID=2020-01-27.07,Shift=O,ShopfloorDatum=2020-01-27 ArticleDesc1="101395" 1580108943000000000" on target "http://localhost:8086/write?&db=afp".
PS C:\Users> $Metrics
Name Value
ChargeStart 2020-01-27 08:09:03
Recipe S41351v1/2
Weight_pv 0,499
Weight_sv 0,5
ChargeEnd 2020-01-27 08:22:11
RecipeName Middenlaag B
ArticleID G10236
ArticleDesc1 101395
PS C:\Users> $Tags
Name Value
ChargeID 428569
RunID 51-000022
WorkCtr 0551
Shift_ID 2020-01-27.07
Shift O
ShopfloorDatum 2020-01-27
Hi
I forked your really cool project and noticed following in your private PowerShell-function "Influx/Private/Out-InfluxEscapeString.ps1":
your code in "Influx/Private/Out-InfluxEscapeString.ps1":
"Measurement" { $String -Replace '(\s|,|\\)', '\$1' }
"FieldTextValue" { $String -Replace '("|\\)', '\$1' }
"Other" { $String -Replace '(\s|=|,|\\|")', '\$1' }
default { $String -Replace '(\s|=|,|\\|")', '\$1' }
my changes in "Influx/Private/Out-InfluxEscapeString.ps1":
"Measurement" { $String -Replace '(\s|,)', '\$1' }
"FieldTextValue" { $String -Replace '(\s|")', '\$1' }
"Other" { $String -Replace '(\s|=|,|")', '\$1' }
default { $String -Replace '(\s|=|,|")', '\$1' }
-> More informations about the changes.
PS: Excuse me for not making an pull request, but nevertheless I hope it helps.
Hello Mark,
Many thanks for your development of the InfluxDb module. It is the best by far and makes working with Influx from scripts a lot easier.
I ran into this behaviour that I would like to verify because it is different from the intention.
Expected -ExcludeEmptymetric to not pass Metric TestValue but in my test it received the value of 38 (Same as Metric 'Mem')
Thanks,
GWS65
*************** Test *******************
$Tags = @{
Server2 = 'ServerTag2'
Location = 'Anywhere'
}
$Metrics = @{
CPU = 44
Mem = 38
TestValue = $null
}
$InfluxLine = ConvertTo-InfluxLineString -Measure WebServer -Tags $Tags -Metrics $Metrics -ExcludeEmptyMetric
Hi, Trying to send zeros using Write-Influx fails as it drops that part of the hash, simple example below.
Many thanks for writing this great module
Write-Influx -Measure WebServer -Tags @{Server='Host01'} -Metrics @{CPU=100; Memory=50} -Database db1 -Server http://127.0.0.1:8086 -Verbose
VERBOSE: Performing the operation "WebServer,Server=Host01 CPU=100
WebServer,Server=Host01 Memory=50 " on target "http://127.0.0.1:8086/write?&db=db1 ".
VERBOSE: POST http://127.0.0.1:8086/write?&db=db1 with -1-byte payload
VERBOSE: received -1-byte response of content type application/json
Write-Influx -Measure WebServer -Tags @{Server='Host01'} -Metrics @{CPU=100; Memory=0} -Database db1 -Server http://127.0.0.1:8086 -Verbose
VERBOSE: Performing the operation "WebServer,Server=Host01 CPU=100 " on target "http://127.0.0.1:
8086/write?&db=db1 ".
VERBOSE: POST http://127.0.0.1:8086/write?&db=db1 with -1-byte payload
VERBOSE: received -1-byte response of content type application/json
Same command just removed the 5 from the memory parameter on 2nd run
Hello
thanks for the work done :). I have now a limitation:
some object property can have null value, the problem is that when this property is exported there will be conflict on the type of data.
Example with Citrix Xendesktop Powershell sdk, what is actually sent:
Xendesktop,computer1 brokeredTime=7, CatalogName="USA1_aa" 12345678898
Xendesktop,computer1 brokeredTime="", CatalogName="USA1_aa" 12345678898
the data is either int or string :)
it could be cool to have a option to set write-influx -emptyValueDoNotsent (please find something better for the name of the option :) ), with a result of:
Xendesktop,computer1 brokeredTime=7, CatalogName="USA1_aa" 12345678898
Xendesktop,computer1 CatalogName="USA1_aa" 12345678898
In this case, zero would not work as the field with no data represent something:
Thanks
Tonic8
Hello
$InputObject and Measure got the same position number,
[Parameter(ParameterSetName = 'MetricObject', ValueFromPipeline = $True, Position = 0)]
[PSTypeName('Metric')]
$InputObject,
[Parameter(ParameterSetName = 'Measure', Mandatory = $true, Position = 0)]
[string]
$Measure,
for example we should have for Measure "Position = 1"
Regards
Tonic8
Hello
using an inputobject (generated by ConvertTo-Metric), we can't specify a timestamp (PSDate).
Consequence the timestamp it'sq when the command is executed
so in a script if we execute 3 time convertto-metric and write-influx, because we collect the data from 3 different source, the timestamp will vary between the 3 collect.
Regards
Tonic8
Hello
i finally catch the issue i saw with some data, here the data sent
xdsession,desktopgroupename=spain-prod,PowerState=On,launchedViaHostName=,hypervizor=vcenter1 LogonInProgress="False"
"launchedViaHostname=" had not value, and all the line is not written to the DB.
it should be usefull to not send the empty tag.
REgards
Tonic8
If the command "Write-Influx" is used without tags, an error occurs.
Working Code
Get-Process -Name winlogon | ConvertTo-Metric -Measure test -MetricProperty CPU,PagedmemorySize -TagProperty Handles,Id,ProcessName | Write-Influx -Bucket test -Server https://influxserver:8086 -Token "token" -Verbose
Not working Code
Get-Process -Name winlogon | ConvertTo-Metric -Measure test -MetricProperty CPU,PagedmemorySize | Write-Influx -Bucket test -Server https://influxserver:8086 -Token "token" -Verbose
Error
Invoke-RestMethod : {"code":"invalid","message":"unable to parse 'test, CPU=\"\" ': missing tag key\nunable to parse 'test, PagedmemorySize=233472 ': missing tag key"}
In C:\Users\ant\Documents\WindowsPowerShell\Modules\Influx\1.0.101\Public\Write-Influx.ps1:202 Zeichen:25
+ ... Invoke-RestMethod -Uri $URI -Method Post -Body $Body -Hea ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand
The error is caused by setting an incorrect comma in the function
According to the following issue, no comma may be present if no tags are transmitted
influxdata/influxdb#3600
In the code, you are using
if ($MetricObject.Tags) { #Build String for Tags }
But $MetricObject.Tags
is an emptry hashtable and an empty hastable evaluates "true".
PowerShell/PowerShell#10201
You have to change the code from
if ($MetricObject.Tags)) {
$TagData = foreach ($Tag in $MetricObject.Tags.Keys) {
if ([string]::IsNullOrEmpty($MetricObject.Tags[$Tag])) {
Write-Warning "$Tag skipped as it's value was null or empty, which is not permitted by InfluxDB."
}
else {
"$($Tag | Out-InfluxEscapeString)=$($MetricObject.Tags[$Tag] | Out-InfluxEscapeString)"
}
}
$TagData = $TagData -Join ','
$TagData = ",$TagData"
}
to
if (($MetricObject.Tags).count -ne 0) {
$TagData = foreach ($Tag in $MetricObject.Tags.Keys) {
if ([string]::IsNullOrEmpty($MetricObject.Tags[$Tag])) {
Write-Warning "$Tag skipped as it's value was null or empty, which is not permitted by InfluxDB."
}
else {
"$($Tag | Out-InfluxEscapeString)=$($MetricObject.Tags[$Tag] | Out-InfluxEscapeString)"
}
}
$TagData = $TagData -Join ','
$TagData = ",$TagData"
}
By checking the code gound this if clause.
if ($MetricObject.Metrics[$Metric] -isnot [ValueType]) {
$MetricValue = '"' + $MetricObject.Metrics[$Metric] + '"'
}
else {
$MetricValue = $MetricObject.Metrics[$Metric] | Out-InfluxEscapeString
}
"$($MetricObject.Measure | Out-InfluxEscapeString)$TagData $($Metric | Out-InfluxEscapeString)=$MetricValue $timeStampNanoSecs"
Can you explain the $MetricObject.Metrics[$Metric] -isnot [ValueType]
part?
By testing, I found out that it checks which data types are used. However, I do not quite understand how
some further suggestions from @Tonic8:
(dir | select -first1).lastWriteTime | gm
(dir | select -first1).lastWriteTime | get-date -f "yyyy-MM-dd HH:mm:ss.fff"
(dir | select -first1).lastWriteTime | convertTo-UnixTimeNanoseconds
where fff is millisecond, (talked here https://stackoverflow.com/questions/28589520/how-to-store-dates-in-influxdb) and here https://docs.influxdata.com/influxdb/v1.5/query_language/data_exploration/#time-syntax
ConnectionViaIP NoteProperty string ConnectedViaIP=1.2.3.4
ConnectionMode NoteProperty Citrix.Broker.Admin.SDK.ConnectionMode ConnectionMode=Brokered
$test = New-Object PSObject -Property @{testBoolean=$True}
Xendesktop,computer1 testBoolean=true 12345678898
Originally posted by @Tonic8 in #15 (comment)
Thank you for creating this. It helped me a lot in a pinch.
I spent a few days working on collecting my own metrics and reading write-influx several times over to understand how and why everything was set up the way it was, and one thing became really apparent as a missing feature. There isn't an ability to send more than one Measure to influx all in one request (mostly impacting TCP since it's so much slower). If you are storing all of your data in one Measure then there's no issue, but I doubt that's the case for many people. And since Influx supports accepting more than one measure in a single TCP packet, this certainly seems do-able.
I was wondering if this is something you've ever looked at or worked on? I was about to start on a PR for it when someone convinced me that I better check with the author first to see if this is something he would even commit.
Implementation
I was thinking of adding a function like 'Append-InfluxQueue' (or Add-...) which appends data onto a list. Basically all of the params like InputObject, Measure, Metrics, and Tags would be moved from Write-Influx onto Append-InfluxQueue and Write-Influx would become a simple writer (also looping through the list turning it into text). The same is true for UDP as well of course.
Thoughts on this?
P.S. Ever considered building in some windows metrics collectors?
I'm writing a fairly straight forward powershell script for getting the dhcp leases from my pihole instance and displaying them for Grafana.
#!/usr/bin/pwsh
$Leases = (ssh -l user pihole "cat /etc/pihole/dhcp.leases")
$metrics = Foreach ($obj in $Leases) {
[hashtable]@{
$obj.Split(" ")[3] = $obj.Split(" ")[2]
}
}
$InfluxParams = @{
Measure = 'pihole_dhcp_leases'
Metrics = $metrics
Database = 'MyDB'
Server = 'http://<influxdb.example.com>:8086'
Verbose = $true
Credential = (Get-Credential)
}
Write-Influx @InfluxParams
As an example output, $leases
comes in the following format:
1536231722 11:11:11:11:11:11 192.168.x.x host01 11:11:11:11:11:11:11
1536205906 11:11:11:11:11:11 192.168.x.x host02 11:11:11:11:11:11:11
I see where Metrics is properly typed:
$metrics
Name Value
---- -----
host01 192.168.x.x
host02 192.168.x.x
$metrics | Get-Member
TypeName: System.Collections.Hashtable
However, Write-Influx
errors at:
Write-Influx : Cannot process argument transformation on parameter 'Metrics'. Cannot convert the "System.Object[]" value of type "System.Object[]" to type "System.Collections.Hashtable".
Is there any explicit typecasting that might interfere here?
It appears that this module only lets you submit a single data point at a time. Is there a method to insert multiple points per request?
Perhaps accept an object in this format : @{ Measure = ''; Tags = @{}; Metrics = @{}; TimeStamp = '' }
in the write-influx function the description of "Timestamp" parameter is missing in the comment/parameters block.
regards
Tonic8
We are using a username and password to connect to our InfluxDB. The function Write-Influx does not include options/parameters to parse username and password to the body part. Are you able to add this functionality?
Thank you
Torben
I have installed your module as well as the TFS Module which you are consuming to make the TFS related calls.
It seems that the underlying TFS Module relies upon instantiating the following global variable before importing the module (either implicitly or explicitly):
$global:tfs=@{
root_url='https://sometfsserver.com'
collection='CollectionName'
project='ProjectName'
}
Per your guide at http://wragg.io/windows-based-grafana-analytics-platform-via-influxdb-and-powershell/ you show calling
Get-TFSBuildMetric -TFSRootURL 'https://mytfsurl.local/tfs' -TFSCollection somecollection -TFSProject someproject
However that call does not return as expected since nowhere in the Get-TFSBuildMetric code do you create/set the $global:tfs variable based upon the parameters passed to Get-TFSBuildMetric. Am I not understanding how this should work? Or is this broken due to changes in the underlying TFS Module? If I have to manually create the global variables, what then is the point of the Parameters being passed to Get-TFSBuildMetric?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.