versent / unicreds Goto Github PK
View Code? Open in Web Editor NEWunicreds is a CLI which manages secrets in AWS using DynamoDB and KMS.
Home Page: https://github.com/Versent/unicreds
License: MIT License
unicreds is a CLI which manages secrets in AWS using DynamoDB and KMS.
Home Page: https://github.com/Versent/unicreds
License: MIT License
Reproduced with both unicreds 1.5.0 and 1.5.1. Occurring on Mac OS X Sierra using darwin builds from github and within an alpine docker container using linux builds, also running on OS X.
Environment has multiple keys, with only some using the 'env:prod' context.
| => unicreds -r us-east-1 -E 'env:prod' getall -d
• Configure AWS profile= region=us-east-1
• Getting all secrets
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x2496]goroutine 1 [running]:
panic(0x3e5640, 0xc4200140d0)
/usr/local/Cellar/go/1.7/libexec/src/runtime/panic.go:500 +0x1a1
main.main()
/Users/markw/Code/go/src/github.com/versent/unicreds/cmd/unicreds/main.go:171 +0x456
Get returns a proper result:
| => unicreds -r us-east-1 -E 'env:prod' get key
⨯ failed error=InvalidCiphertextException:
status code: 400, request id: 037d2e62-bc06-11e6-9287-8fb5c387f130
Attempted to reproduce with a local build. Instead of reproducing the failure, it simply works as expected with a subset of keys returned. Based on some quick searches, it may be a go bug related to OS X Sierra. The go patch I found was committed on October 17th, so even the 1.5.1 build is likely to be missing it.
Now using a local build to unblock. Filing this issue to bring visibility in case this impacts other users.
Currently the results come back out of order for some weird dynamodb related reason.
With lots of entries it is a pain.
Use case: we have several regions, each has a separate dynamodb storage. It's a bit cumbersome to get current region from instance's metadata and paste it into unicreds arguments. Can do that myself, just wanna discuss.
Hi there, we recently got an email from AWS suggesting that one of our applications was using dynamodb with a slightly older AWS SDK. We just wanted to check that unicreds used an up-to-date SDK? Here is the meat of the email:
We recently analyzed your use of DynamoDB. During that process, we noticed you have a system that makes calls to DynamoDB using either an older AWS SDK or an unknown SDK that may not support SSL/TLS certificates issued by Amazon Trust Services (ATS).
Beginning on May 14, 2018, Amazon DynamoDB will update the SSL/TLS certificates the service uses to secure communications. Once the update is complete, ATS will issue the SSL/TLS certificates used by DynamoDB. ATS is also the certificate authority (CA) used by AWS Certificate Manager.
After DynamoDB migrates to certificates issued by ATS, applications that use an SDK (i.e., one that does not properly handle the new certificates) may stop working. In order for your application to continue to operate normally, you may need to migrate to a newer AWS SDK, or modify your custom SDK to trust ATS.
The following documentation provides more information and guides you through the process: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/ats-certs.html
credstash --region ap-south-1 --table credstash-prod-app get password
<secret>
unicreds --region ap-south-1 --table credstash-prod-app get password --debug
• Configure AWS profile= region=ap-south-1
• Getting highest version secret
⨯ failed error=UnmarshalTypeError: cannot unmarshal binary into Go value of type string
list
seems to work fine:
./unicreds --region ap-south-1 --table=credstash-prod-app list
+----------------------+---------------------+---------------+
| NAME | VERSION | CREATED-AT |
+----------------------+---------------------+---------------+
| password | 0000000000000000000 | Not Available |
I've tried it with secrets having Version>0, and it still does not work and give the same error. Secrets have been stored using credstash.
Hi Team,
I don't believe it's possible to setup a new environment without being on an interactive shell. Please allow the ability to pass the required arguments for non-interactive installation from scripts/userdata.
Side option: build the Table and/or KMS using CFN
Thank you.
using unicreds together with jq could be simpler if uncreds would print the json content to stdout and not error stream. now I have to do uncreds --json get xxx 2>&1 |jq ....
unicreds contains code to assume a different role via STS
https://github.com/Versent/unicreds/blob/master/aws_config.go#L19
But there is no non test code path that executes this code :(
This is a common option for AWS focused tool so it makes sense to add it.
cc @romanrev
Can we use Unicers for the functions that are deployed in AWS Lambda?
Using 1.4.0 on OSX, I get an error every time I supply a version argument to get
:
$ unicreds list -r ap-southeast-2
+---------+---------+--------------------------------+
| NAME | VERSION | CREATED-AT |
+---------+---------+--------------------------------+
| testing | 5 | 2016-09-07 10:27:07 +1000 AEST |
+---------+---------+--------------------------------+
$ unicreds get testing 5 -r ap-southeast-2
unicreds: error: unexpected 5, try --help
Getting the credential without a version argument works as expected.
If Table.scan would return more than 1MB, the result is paginated.
LastEvaluatedKey is set and scan() could be called again, or ScanPages could be used.
This bug appears to exist in credstash too
Running 1.4.0 on OSX, it seems like the --region
flag is required for every command, even if I have the AWS_REGION
environment variable set.
$ unicreds list
⨯ failed error=MissingRegion: could not find region configuration
$ env|grep AWS
AWS_PROFILE=my-profile
AWS_REGION=ap-southeast-2
Issue:
Current workaround:
vagrant@vagrant-ubuntu-trusty-64:~$ unicreds --alias='alias/nonprod-aem-creds-key' put "test-v10-issue" '1'
• stored name=test-v10-issue version=1
vagrant@vagrant-ubuntu-trusty-64:~$ unicreds get test-v10-issue
1
vagrant@vagrant-ubuntu-trusty-64:~$ unicreds --alias='alias/nonprod-aem-creds-key' put "test-v10-issue" '2'
• stored name=test-v10-issue version=2
vagrant@vagrant-ubuntu-trusty-64:~$ unicreds --alias='alias/nonprod-aem-creds-key' put "test-v10-issue" '3'
• stored name=test-v10-issue version=3
vagrant@vagrant-ubuntu-trusty-64:~$ unicreds --alias='alias/nonprod-aem-creds-key' put "test-v10-issue" '4'
• stored name=test-v10-issue version=4
vagrant@vagrant-ubuntu-trusty-64:~$ unicreds --alias='alias/nonprod-aem-creds-key' put "test-v10-issue" '45
> '
• stored name=test-v10-issue version=5
vagrant@vagrant-ubuntu-trusty-64:~$ unicreds --alias='alias/nonprod-aem-creds-key' put "test-v10-issue" '6'
• stored name=test-v10-issue version=6
vagrant@vagrant-ubuntu-trusty-64:~$ unicreds --alias='alias/nonprod-aem-creds-key' put "test-v10-issue" '7'
• stored name=test-v10-issue version=7
vagrant@vagrant-ubuntu-trusty-64:~$ unicreds --alias='alias/nonprod-aem-creds-key' put "test-v10-issue" '8'
• stored name=test-v10-issue version=8
vagrant@vagrant-ubuntu-trusty-64:~$ unicreds --alias='alias/nonprod-aem-creds-key' put "test-v10-issue" '9'
• stored name=test-v10-issue version=9
vagrant@vagrant-ubuntu-trusty-64:~$ unicreds --alias='alias/nonprod-aem-creds-key' put "test-v10-issue" '10'
• stored name=test-v10-issue version=10
vagrant@vagrant-ubuntu-trusty-64:~$ unicreds --alias='alias/nonprod-aem-creds-key' put "test-v10-issue" '11'
⨯ failed error=ConditionalCheckFailedException: The conditional request failed
status code: 400, request id: 724BNHJ30K36ML3H1DEE5704OBVV4KQNSO5AEMVJF66Q9ASUAAJG
You currently cannot install unicreds
if you have a MAC with an M1 processor (arm64) without building from source (which for me was not really straighforward), unless you have Rosetta. Would be nice to have that build and released by default as well.
Given there are some KMS keys a host with the respective IAM profile does not have access to, credstash will return all entities encrypted with the keys which are allowed to be read by the host:
time AWS_DEFAULT_REGION=ap-southeast-2 credstash getall -f csv
ssh_key.accountorch-infra,"-----BEGIN RSA PRIVATE KEY-----
... redacted ...
-----END RSA PRIVATE KEY-----"
ec2.ssh.key.accountorch-common-infra,"-----BEGIN RSA PRIVATE KEY-----
... redacted ...
-----END RSA PRIVATE KEY-----"
real 0m5.469s
user 0m1.634s
sys 0m0.073s
whilst unicreds will bail out with an error:
time unicreds --region ap-southeast-2 getall --csv
⨯ failed error=AccessDeniedException: The ciphertext references a key that either does not exist or you do not have access to.
status code: 400, request id: 58bae15c-10c6-11e6-b3b9-e10a6731ec14
real 0m0.383s
user 0m0.039s
sys 0m0.009s
Currently a password without a linefeed gets printed with an extra linefeed from the unicreds tool.
This has been resolved in credstash with the option -n
(or --noline
). It helps with binary files and within scripts where the output is expected to be the password (and only that).
Credstash added support for KMS Encryption Contexts in this pull request.
Would it be possible to add support for Encryption Contexts in unicreds too?
I'd be keen to have some integration tests. Mocking is great but having integration tests, generating real resources, would be a useful addition.
Is this on the road map? Happy to work on it.
Haven't worked out the pricing yet, but unless you are accessing a ton of secrets all the time, most will provision dynamo with a read and write provisioning of 1 (about $0.59/mo at current pricing). But I've noticed that I get alarms when multiple secrets are read as a server comes up (and it reads say 8 things) in a very short time. Now, I don't want to insert sleeps or something dumb like that, but another option MAY be to put a small DAX accelerator in front of these tiny dynamo tables (which are basically R/O and would benefit from a tiny tiny cache).
So making a placeholder issue here to consider a code change to support specifying a DAX endpoint in lieu of a dynamo table name.
Like I said, not sure if the pricing works out, but might be useful for some use cases...
Hey Team,
I'm considering the option of the creation of a shared/common credstash and as such require cross-account ARNs. Unless there is a smarter way of doing this.
note: I know you can grant aliases to KMS keys cross account - but i'd rather just use the full path for my usecase.
I'm seeing these warnings when using your homebrew tap
==> Tapping versent/taps
Cloning into '/usr/local/Library/Taps/versent/homebrew-taps'...
remote: Counting objects: 5, done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 5 (delta 0), reused 3 (delta 0), pack-reused 0
Unpacking objects: 100% (5/5), done.
Checking connectivity... done.
Warning: Calling Formula.sha1 is deprecated!
Use Formula.sha256 instead.
/usr/local/Library/Taps/versent/homebrew-taps/unicreds.rb:7:in `<class:Unicreds>'
Please report this to the versent/taps tap!
Warning: Calling SoftwareSpec#sha1 is deprecated!
Use SoftwareSpec#sha256 instead.
/usr/local/Library/Taps/versent/homebrew-taps/unicreds.rb:7:in `<class:Unicreds>'
Please report this to the versent/taps tap!
Warning: Calling Resource#sha1 is deprecated!
Use Resource#sha256 instead.
/usr/local/Library/Taps/versent/homebrew-taps/unicreds.rb:7:in `<class:Unicreds>'
Please report this to the versent/taps tap!
Tapped 1 formula (29 files, 20.3K)
Looks like its not you just need to switch this line:
sha1 '70bfdfd9491cec74916d44d29398d0650292fd1e
To use sha256 instead
Set region
set AWS_REGION=ap-southeast-2
Then run unicreds
Set region
[Environment]::SetEnvironmentVariable("AWS_REGION", "ap-southeast-2")
Then run unicreds
Need to test this with JSON data for @daveslutzkin
it would be great if there was a templating feature similar the one implemented here https://github.com/winebarrel/gcredstash . this way go templating could be leveraged to inject secrets into all types of files.
thoughts on implementing this? the above mentioned repo is not very active an currently broken due to the hmac updates made in credstash.
Need to investigate why adding a new secret doesn't create a new record and increment the version.
Hi there,
First: Thanks for saving me to port credstash from Python to Go myself :)
A few questions:
Another thing that I'd like to see is a pre-release being created from master, you could use Travis to do this for you. I have good experience with ghr
to automate this. See this (old) Travis file for an example.
Happy to help out if this is something you'd like too.
When unicreds tries to access a key and access to the key is denied, it returns exit 0, and a message as below:
It would be great if the exitcode in such cases is greater than 0, so Jenkins could fail.
++ unicreds --region ap-southeast-2 --alias=alias/credstash get ca-api-master_database_pass
�[31m ⨯�[0m failed �[31merror�[0m=AccessDeniedException: The ciphertext references a key that either does not exist or you do not have access to.
status code: 400, request id: 30a6b21d-119c-11e6-929f-05d307e0ce89
I noted that 3fbd8ba attempts to resolve the error one gets when region is not passed on the command line:
⨯ failed error=MissingRegion: could not find region configuration
However I still get this error message. I have added the region to both the default
and velo
profiles in ~/.aws/config
.
My command is:
unicreds --profile velo get <mysecret>
Moreover these commands do not work.
AWS_DEFAULT_REGION=ap-southeast-2 unicreds --profile velo get jenkins-ssh-private-key
⨯ failed error=MissingRegion: could not find region configuration
AWS_REGION=ap-southeast-2 unicreds --profile velo get jenkins-ssh-private-key
⨯ failed error=NoCredentialProviders: no valid providers in chain. Deprecated.
But this one does
unicreds --profile velo get <mysecret> -r ap-southeast-2
We have had some issues with automation performing many concurrent requests and overloading DynamoDB. This change will make it more obvious these things need to be tuned.
Also increase the values to a more reasonable while still cheap default being read and write capacity of 4.
Apparent issue relating to versioning of creds (less than 10, and more recent than 2016), see below example, where creds from months ago were updated with put-file, succesfully, but retrieval of creds somehow defaults to a version that seemingly doesn't exist.
This is run off from a windows machine via powershell through CMDer.
λ .\unicreds -r ap-southeast-2 -p saml get license
LICENSE 66666666 2016.03 31-mar-2016 uncounted
hostid=66666666 share=66666666 client_cache=66666666 customer=66666666
akey=66666666 options=nodes=0,links=0,zones=0,mp=1,v=0
_ck=66666666 sig="66666666"
λ .\unicreds -r ap-southeast-2 -p saml get license1 3
⨯ failed error=Secret Not Found
λ .\unicreds -r ap-southeast-2 -p saml get license1 2
LICENSE 66666666 2017.12 31-dec-2017 uncounted
hostid=66666666 share=66666666 client_cache=66666666 customer=66666666
akey=66666666options=nodes=0,links=0,zones=0,mp=1,v=0
_ck=66666666 sig="66666666"
λ .\unicreds -r ap-southeast-2 -p saml get license1 1
⨯ failed error=Secret Not Found
Currently you can only get the latest version of a secret without getting every secret. It would be nice to be able to specify a version in the GetSecret
function: https://github.com/Versent/unicreds/blob/master/ds.go#L148
While unicreds support specifying an alternate profile, it does not have the ability to assume a role using that (or the default) profile as shown in the AWS docs here.
Ideally, there'd be another optional parameter to specify the role to assume.
Even better would be the ability to use pre-defined profiles specified in ~/.aws/config
as in this example:
[profile prod-admin]
role_arn = arn:aws:iam::XXXXXX:role/Admin
source_profile = default
region = us-east-1
which works with the aws cli as in this example (documented here):
$ aws --profile prod-admin ec2 describe-vpcs
[output snip]
Not really sure why ~/.aws/config
seems to be ignored with the SDK but not in the CLI.
I've seen an additional role_arn
parameter in the config as in this terraform example which only just came out in 0.8.7 so you can look how they did it here:
provider "aws" {
region = "${var.aws_region}"
profile = "default"
assume_role {
role_arn = "arn:aws:iam::${var.aws_account_number}:role/Admin"
}
}
Looking over the credstash usage, this would be functionally equivalent to the --arn
parameter
The new go modules feature expects tagged versions to be prefixed with a 'v'. Since the latest release, 1.7.0, is missing the prefix, the go
command finds v1.5.0 as the latest version.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.