turbot / steampipe Goto Github PK
View Code? Open in Web Editor NEWZero-ETL, infinite possibilities. Live query APIs, code & more with SQL. No DB required.
Home Page: https://steampipe.io
License: GNU Affero General Public License v3.0
Zero-ETL, infinite possibilities. Live query APIs, code & more with SQL. No DB required.
Home Page: https://steampipe.io
License: GNU Affero General Public License v3.0
I had a query where I tried a unicode domain:
select * from whois_record where domain = 'München.com';
I can Command-Left through the line in history no problem:
Welcome to Steampipe v0.0.14
Type ".inspect" for more information.
> select * from whois_record where domain = 'München.com';
But as soon as I Command-Right over the ü character things break:
~/src/steampipe $ steampipe query
Welcome to Steampipe v0.0.14
Type ".inspect" for more information.
> select * from whois_record where domain =runtime error: slice bounds out of range [:57] with capacity 56
all states should be contained in one file located in internal/steampipe.json
all necessary state information should be read from here
also, consider removing the key from the versions file and make it an array instead of a map
Version: 0.1.0
~ $ steampipe -v
steampipe version 0.1.0
~ $ steampipe plugin install steampipe
Error: plugin install failed for plugin 'steampipe'
running the instal again seems to work...
The zendesk_ticket
table goes pretty nuts now there is no max width for columns. Sideways scroll goes on a long long way.
Should we have a max column width?
Should it wrap or truncate?
What should it be? 80 chars? 120 chars? 1024 chars?
Given that Turbot already acts as a CMDB for resources, could Turbot be added as a backend to query resource metadata?
Running queries against aws.aws_iam_policy
return an error:
Error: pq: rpc error: code = Internal desc = hydrate call getCommonColumns failed with panic AWS_REGION must be set to use the aws extension
it is ignoring the default region set in ~/.aws/config.
While running a query against ACM seems to work fine:
Welcome to Steampipe v0.1.0
Type ".inspect" for more information.
> select * from aws_iam_policy
Error: pq: rpc error: code = Internal desc = hydrate call getCommonColumns failed with panic AWS_REGION must be set to use the aws extension
> select title from aws_acm_certificate
+--------------------------------------+
| title |
+--------------------------------------+
| 82bf4707-0a47-4e55-af8b-ece0dd12fa7d |
| 034b902a-18c2-4686-b4c9-83dc3355e891 |
+--------------------------------------+
Right now, steampipe
operates with a stripped down version of pg12
This needs to be updated to pg14
Tasks:
PG14
image to hub.steampipe.io
FDW
compatible with PG14
to hub.steampipe.io
Log statements at the appropriate level should be written to the console. Currently, you can specify the log level with SP_LOG, and log statements at that level and above will be written to the postgres logs. It would be great for debugging/troubleshooting to also print these to the console (possibly with postgres notice?))
ESC/2A-2D
Steampipe Version: v0.1.0
OS Version: MacOS 10.15.7
Issue
When passing a query as a cli parameter, if the query in question is not correct and returns an error, steampipe does not close the DB session and instead leaves it open, resulting in subsequent requests to fail due to steampipe being "open".
Steps to reproduce:
➜ steampipe query "select foo from bar;"
Error: pq: relation "bar" does not exist
➜ sp_demo steampipe query "select foo from bar;"
Error: You already have a steampipe query session open. To run multiple sessions, first run steampipe service start.
To kill existing sessions run steampipe service stop --force
steampipe query
cannot run if another query
session is open which owns service
steampipe service
cannot start if query
owns service
Query 1:
steampipe query --output json "select * from chaos.all_column_types where id=99 order by id"
Output:
[
{
"array_element": {
"Key": "stringValuesomething-99",
"Value": "value"
},
"boolean_column": false,
"cidr_column": "MTcyLjMxLjAuMC8xNg==",
"date_time_column": "2002-05-05T09:23:00Z",
"double_column": 5.823529411764706,
"empty_hydrate": "I AM A DEFAULT",
"id": 99,
"ipaddress_column": "MTAuMC4wLjc=",
"json_column": {
"Id": 99,
"Name": "stringValuesomething-99",
"Statement": {
"Action": "iam:SimulatePrincipalPolicy",
"Effect": "Deny"
}
},
"string_column": "stringValuesomething-99"
}
]
Query2:
steampipe query "select * from chaos.all_column_types where id=99 order by id"
Output:
Using Ctrl+C in my terminal for other programs allows me to exit quickly, e.g.,
Codys-MacBook-Pro:Desktop cbruno$ node
Welcome to Node.js v14.15.1.
Type ".help" for more information.
>
(To exit, press Ctrl+C again or Ctrl+D or type .exit)
>
For node
, it also gives me the command that I can use to exit from the environment as a helper.
Is this something that can be added while inside steampipe query
as well?
Feature request to enable binary package management on debian and redhat based OSs.
seems like file ~/.steampipe/internal/versions.json is not getting updated when we install any existing plugin again
in below table for plugin hub.steampipe.io/plugins/turbot/aws@latest Version should be 0.0.10 but it is not showing any version
➜ steampipe-plugin-azure git:(main) steampipe plugin list
+--------------------------------------------------+---------+-------------+
| NAME | VERSION | CONNECTIONS |
+--------------------------------------------------+---------+-------------+
| hub.steampipe.io/plugins/turbot/aws@latest | | aws |
| hub.steampipe.io/plugins/turbot/azure@latest | local | azure |
| hub.steampipe.io/plugins/turbot/chaos@latest | | chaos |
| hub.steampipe.io/plugins/turbot/steampipe@latest | 0.0.2 | steampipe |
+--------------------------------------------------+---------+-------------+
when installing plugins, print a link to the documentation
when starting up show more useful message like
Welcome to Steampipe v0.1.0
Type ".help" for more information
Exit with .exit
model on pgcli
currently we wait for all output in order to size the columns
instead, wait for a page of results, and use this for sizing information, wrap any longer columns
NOTE: we may need an indicator when trying to page past the end of the available results
*still fetching*
or something
Update plugins when their tag is moved to a new image
Check for / prompt / install updates to:
Steampipe CLI [done]
Steampipe DB
Steampipe FDW
select * from aws_iam_policy
for pixie eu-west-2 returns 1 row longer than 64k. This breaks the code which detetcs if paging is necessary.
It can be fixed as follows:
func isPagerNeeded(content string) bool {
maxCols, maxRow, _ := gows.GetWinSize()
// let's scan through it instead of iterating over it fully
sc := bufio.NewScanner(strings.NewReader(content))
// explicitly allocate a large bugger for the scanner to use - otherwise we may fail for large rows
buffSize := 256 * 1024
buff := make([]byte, buffSize)
sc.Buffer(buff, buffSize)
lineCount := 0
for sc.Scan() {
line := sc.Text()
lineCount++
if lineCount > maxRow {
return true
}
if len(line) > maxCols {
return true
}
}
return false
}
the shell script requires unzip, which is not installed by default on Ubuntu. We should tar.gz the linux build instead.
This is not currently causing an issue - i only noticed because the version is missing (theres already an open issue for that) and i was trying to match up the versions manually to check what version I had installed. It probably WILL be problematic when implementing steampipe plugin update
....
~/.steampipe/internal/versions.json :
"hub.steampipe.io/plugins/turbot/steampipe@latest": {
"name": "hub.steampipe.io/plugins/turbot/steampipe@latest",
"version": "",
"imageDigest": "sha256:50e6ed50cb04724cb7829ab7a2ba522048162a53b75423f8dc7c46e2fbda7f38",
"installedFrom": "us-docker.pkg.dev/steampipe/plugins/turbot/steampipe:latest",
"lastCheckedDate": "2021-01-13T00:10:23Z",
"installDate": "2021-01-13T00:10:23Z"
}
> .inspect zendesk.zendesk_group
+------------+-----------------------------+--------------------------------+
| Column | Type | Description |
+------------+-----------------------------+--------------------------------+
| created_at | timestamp without time zone | The time the group was created |
| deleted | boolean | True if the group has been |
| | | deleted |
| id | bigint | Unique identifier for the |
| | | group |
| name | text | Name of the group |
| updated_at | timestamp without time zone | The time of the last update of |
| | | the group |
| url | text | API url of the group |
+------------+-----------------------------+--------------------------------+
split large JSON files to reduce noise
select smaller data sets
Running steampipe -v
on a brand new install give an error the first time:
vagrant@vagrant:~$ steampipe -v
Unknown response during version check: 405
steampipe version 0.1.0
vagrant@vagrant:~$ steampipe -v
steampipe version 0.1.0
The first time i ran inspect on a new install after install a plugin, i got the following error:
> .inspect
+------------+--------------------------------------------------+
| Connection | Plugin |
+------------+--------------------------------------------------+
| steampipe | hub.steampipe.io/plugins/turbot/steampipe@latest |
+------------+--------------------------------------------------+
To get information about the tables in a connection, run '.inspect {connection}'
To get information about the columns in a table, run '.inspect {connection}.{table}'
runtime error: slice bounds out of range [1:0]
From turbot/steampipe-plugin-github#6 we see the error message:
Error: pq: rpc error: code = Internal desc = 'List' call requires an '=' qual for column: login
causing confusion & delay.
Can we standardize / catch / improve that error message to be more helpful?
When service
has been shutdown out-of-band, interactive
panics when exiting
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.