As discussed in #38, we can only get access-key-last-used-date currently, which blocks our completion of Milestone 1.

The CI builds always fail. For example, see the failed builds on PR #4.

• Figure out how to expose credentials to Travis CI
• Ask Christoph re: AWS credentials
• Make sure the build is green

Currently, all our integration-test runs are in the same namespace and so can collide. As an example, I hit

Error applying plan:

2 error(s) occurred:

* aws_iam_user.mfa_not_enabled_user: Error creating IAM User mfa_not_enabled_user: EntityAlreadyExists: User with name mfa_not_enabled_user already exists.
        status code: 409, request id: c654263b-196e-11e7-9dbc-6dbf49504827
* aws_iam_user.console_password_enabled_user: Error creating IAM User console_password_enabled_user: EntityAlreadyExists: User with name console_password_enabled_user already exists.
        status code: 409, request id: c6570d4e-196e-11e7-b4b2-712691fb77d1

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

when I tried to run ubuntu@ubuntu-xenial:~/inspec-aws$ bundle exec rake test:setup_integration_tests.

Ideally, we can isolate environments with Terraform, like we can with CFn stacks...

For recommendation 1.16. See

aws iam list-attached-user-policies --user-name <iam_user>

in the CIS benchmark.

Something that came up while @vix633 was working on the policy resource: assertion grammar. (I think this is an important part of InSpec usability, so I want us to get this right/consistent.) For boolean properties, at least, I think we want to lead with a third person verb: exists?, has_console_password?, and has_mfa_enabled?.

This gets interesting/weird with InSpec's assertion grammar: for the ec2 resource, we can say either it { should exist } OR its('exists?') { should be true }. The former is more usable but

• I am not clear how/why this works (Does Ruby/InSpec understand English grammar?),
• I do not know if/how this translates to other methods, like has_console_password?.

@chris-rock , is this worth a live discussion/tutorial, and are you available? I ask because we have more methods coming in now, and the sooner we get this right, the less rework we need to do.

We have user and access-key resources. What else do we need before we can implement a test for Recommendation 1.3?

For recommendation 1.4.

PR #57 .

Instead of generating a random environment identifier during the setup of our integration tests, use a fixed one based on an input. (Consider defaulting the environment identifier to the hostname, or not defaulting it.)

The idea is that it should be possible to imbue these IDs with human-level meaning, so we can link a Terraform environment identifier--which appears in the AWS console--to a person or a task.

We have user and access-key resources. What else do we need before we can implement a test for Recommendation 1.4?

Considering resources we should probably have a look at the NIST requirements and see what is needed.

https://s3.amazonaws.com/quickstart-reference/enterprise-accelerator/nist/latest/assets/NIST-800-53-Security-Controls-Mapping.xlsx

http://docs.aws.amazon.com/quickstart/latest/accelerator-nist/welcome.html

Create a new resource with an access_key_count method. You can get the count from the account summary.

https://aws.amazon.com/blogs/security/an-easier-way-to-determine-the-presence-of-aws-account-access-keys/

If we create a resource with an auto-generated ID (e.g. an access key) in our Terraform plan, then we don't know that ID during our integration tests. Update the integration tests so these IDs (and other values) can be passed from setup to test.

... to look for groups, roles, and users that have a policy. For recommendation 1.22.

Is the resource too abstract? What are the alternatives?

describe PolicyEntities.where(policy_name="AWSSupport") do
  it {should exist}
end

• miles
• adnan
• simon
• jeff
• steff
• alex
• viktor
• chris

Get password expiry in days and see if password expiry is enabled

For recommendation 1.21.

The property should return an array of access-key InSpec resources. This issue is a result of #18 ; and should address #18 's use case.

Please add

https://github.com/Rugbyte
https://github.com/Stiffni
https://github.com/AlexBedley
https://github.com/vix633
https://github.com/JeffreyLyonsD2L

as contributors to the repo.

Now that we have a single aws_iam_user, add the ability to query for users with a new aws_iam_users resource.

Existing query logic for InSpec's core user resource is at

https://github.com/chef/inspec/blob/master/lib/resources/users.rb#L72-L89 and

https://github.com/chef/inspec/blob/master/lib/resources/users.rb#L103-L105

Example usage:

describe aws_iam_users.where(:has_console_password? => true) do
  its('has_mfa_enabled?') { should_not include false }
end

user_provider

• create aws_user_provider.rb - master...Issue9-UserProvider
• make aws_iam_user use aws_user_provider
• fix aws_iam_user unit tests
• add aws_user_provider unit tests
• merge changes above

get_users

• add get_users to aws_user_provider (and unit tests)
• investigate potential paging of users by aws
• run manual integration test
• merge changes above

aws_am_users

• add aws_iam_users.rb which will call get_users to populate the filter table
• aws_iam_users unit tests
• aws_iam_users integration tests

We need to assert that a role allows and denies specific actions, and that a user/group is not assigned both of two groups. See the CIS recommendation for more info.

When running the unit tests, the following warning appears:

/home/ubuntu/inspec-aws/test/unit/resources/aws_iam_password_policy_test.rb:78: warning: circular argument reference - value

Fix the warning.

Examples of Questions of Concerns:

i.e. ensure users didn't open an S3 bucket to the world
i.e. ensure that s3 only open to particular users
i.e. ensure that only a particular service, port, and or IAM
i.e. ensure that I can pass a list of IAMs to the resource and that it can validate that the S3 bucket has or has_not access.
i.e. ensure the S3 bucket is only accessible form https, or a known uri etc.
i.e. ensure that the S3 bucket has logging enabled, set, reporting to the 'known' location
i.e. ensure that the 'POLICY SETTINGS' for S3 is set or not set

How can we implement this recommendation with InSpec resources?

Currently, aws_iam_users loads all of each user's attributes up-front. Refactor so that the loading happens lazily, as needed.

This gives the resource friendly class and instance labels. The format is "". See https://github.com/chef/inspec-aws/blob/master/libraries/aws_iam_access_key.rb#L41 for an example.

• Review PR #10 - Chris and Viktor
• We know how we will write ITs
• We know how we will run ITs
• Add to readme

One option for checking instance roles is to explicitly name the instances to check. In this case, we can just add a 'has_instance_role?' property to our existing EC2 resource.

We'd like to search for instances that "should" have roles, and check those; but we don't see any reliable way to identify such instances--we might need to leave instance identification to the user.

@aduric and @Stiffni , what do you think?

We're leaving AWS resources lying around because our tests don't auto-cleanup. Fix that.

Implement ability to get iam password policy and validate it's properties based on AWS_CIS recommendations 1.5 through 1.11.

• 1.5 Ensure IAM password policy requires at least one uppercase letter - @vix633

• 1.6 Ensure IAM password policy require at least one lowercase letter

• 1.7 Ensure IAM password policy require at least one symbol

• 1.8 Ensure IAM password policy require at least one number

• 1.9 Ensure IAM password policy requires minimum length of 14 or greater

• 1.10 Ensure IAM password policy prevents password reuse

• 1.11 Ensure IAM password policy expires passwords within 90 days or less

Recommendation 1.3 looks for passwords or API keys that

1. are enabled and
2. haven't been used in 90 days.

Design the example usages for a single user. Some questions to answer:

1. Do we want to hang these properties off the user resource, or do we want separate users, console passwords, and access keys?
2. How do we want to address each of the two access keys which each user can have?
3. How do we test the date? (i.e. 'time since last use' < 90 days)

AC:

1. We have a new story, with example usages, on how to implement Recommendation 1.3.

The resource should have the following properties:

• id
• active?
• create_date
• last_used_date

This issue is a result of #18.

Some recommendations don't make sense in a pass/fail environment, while others are not (practically) implementable with the AWS API. Which recommendations won't we implement?

This gives the resource friendly class and instance labels. The format is "". See https://github.com/chef/inspec-aws/blob/master/libraries/aws_iam_access_key.rb#L41 for an example.

We want to create an InSpec profile that clearly demonstrates each of the recommendations. When run against an account, the profile should fail if the account does not follow the recommendations.

Start with the password policy, which should all be in master and ready to go.

If 1.5-1.11 are not yet demoable, then create follow-up issues.

Currently, the configure_test_environment creates a new environment on each run, regardless if one has already been created (and has running resources). This "hides" the old environment, making it more difficult to track/manage.

Before creating a new Terraform env, check to see if the previous one exists and fail if it does. (This might be hard to do until #52 and might have to wait for that fix.)

Viktor and Chris took a look at these: we see that InSpec's unit testing for OS-level resources uses a mock OS setup, see

https://github.com/chef/inspec/blob/master/test/unit/resources/user_test.rb

Do we have some kind of mock environment for AWS, or do we stick with integration tests?

Like it says in the title.

• Create an empty IAM aws_iam_user resource that does nothing
• Add the ability to choose a user by username
• Add "has MFA enabled" member
• Add "has console password" member
• Unit tests (@JeffreyLyonsD2L, @Rugbyte)
• Integration tests that specifically demonstrate (a single-user version of) recommendation 1.2
  • Any dev can run integration tests locally
  • We can use Terraform to create IAM users with MFA

Notes:

• Similar to core InSpec, we will differentiate between a single user (the resource described here) and a set of users (a resource which we have not planned yet).
• As discussed on Slack, let's call the user resource aws_iam_user.

Example usage:

describe aws_iam_user('mfa_test_user') do
  its('is_mfa_enabled?') { should be true }
end

https://github.com/chef/inspec-aws/compare/issue6

Ec2#exists checks for a non-nil id; because the id can be set during construction without any check against AWS, Ec2#exists sometimes reports true when the instance does not actually exist.

See the failing integration test in the issue13 branch.

Profile: inspec-aws-integration-tests
Version: unknown
Target:  local://


  EC2 Instance
     ✔  Example should exist
     ✔  Example image_id should eq "ami-0d729a60"
     ✔  Example instance_type should eq "t2.micro"
  EC2 Instance
     ∅  i-missing should not exist
     expected EC2 Instance i-missing not to exist


Profile: InSpec AWS Resource Pack (inspec-aws)
Version: 1.0.0
Target:  local://

     No tests executed.

Test Summary: 3 successful, 1 failures, 0 skipped
rake aborted!
Command failed with status (1): [bundle exec inspec exec test/integration/v...]
/home/ubuntu/inspec-aws/Rakefile:35:in `block (2 levels) in <top (required)>'
/var/lib/gems/2.3.0/gems/rake-12.0.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => test:integration
(See full trace by running task with --trace)

To fix the issue, consider checking for a non-nil instance (as returned by Ec2#instance) instead.

For recommendation 1.14. See the aws iam list-virtual-mfa-devices command and the text in the CIS benchmark.

In #37, we identify an existing access-key field, its create date, to implement CIS recommendation 1.4. Before we can search for access keys by this field, however, it needs to be added to the user's filter table.

Challenges:

1. How do we query for policies based on their content? See aws iam get-policy-version for an example of querying and AWS's query syntax.
2. How do we express our query in InSpec Policies.where(???) what does FilterTable support here?
3. See more info in the CIS recommendation.

It specifically should support "is null" or a similar query, to find access keys that have never been used. Required by recommendation 1.23.

Similar to #72 .

See #47. It would be good to have a deterministic integration test to cover the following case:

describe aws_iam_password_policy do
  its('prevent_password_reuse?') { should be true }
end


describe aws_iam_password_policy do
  its('number_of_passwords_to_remember') { should eq 1 }
end

Once the test is complete the account should be reverted to its previous state.

Don't allow something like

def f(a, b,
         c,
         d)
do_work
end

I think this should be fairly simple to implement. (Famous last words...)

Fetch the temporary access key and secret key from the metadata API endpoint.
Store those keys and use them when making calls to AWS. This might help

http://docs.aws.amazon.com/sdkforruby/api/Aws/IAM/Client.html

See #49 . It would be good to have a deterministic integration test to cover the following case:

describe aws_iam_root_user do
  its('access_key_count') { should eq 0 }
end

Once the test is complete the account should be reverted to its previous state.

PR #56 .

Currently when you run bundle exec rake test:integration AWS resources are not destroyed if the integration tests fail. There should be an option to ensure the cleanup of AWS resources regardless of the results from the integration tests.

For example you might run bundle exec rake test:integration --force-cleanup