xanzy / chef-guard Goto Github PK

Otherwise internal stuff of Chef-Guard will be shown which may throw people off.

$ knife cookbook upload windows
Uploading windows        [1.34.0]                                         
ERROR: Cookbook windows depends on cookbooks which are not currently      
ERROR: being uploaded and cannot be found on the server.                  
ERROR: The missing cookbook(s) are: 'chef_handler' version '>= 0.0.0'     

$ knife cookbook upload chef_handler                
Uploading chef_handler   [1.1.5]                                          
Uploaded 1 cookbook.                                                      

$ knife cookbook upload windows
Uploading windows        [1.34.0]                                         
Uploaded 1 cookbook. 

The chef_handler is on the chef server but it isn't frozen. Shouldn't I get a 409 Conflict Error error when I try to upload the windows cookbook?

When I tried to upload a cookbook without bumping the version number, Chef Guard correctly gave me an error and did not upload the cookbook. But it did update the config repo. Chef Guard should not update the config repo when the cookbook fails to upload.

d:\chef-repo>knife cookbook upload mycookbook
Uploading mycookbook  [0.2.2]
ERROR: Version 0.2.2 of cookbook mycookbook is frozen. Use --force to override.
WARNING: Not updating version constraints for mycookbook in the environment as the cookbook is frozen.
ERROR: Failed to upload 1 cookbook.

Here is the diff of commit to config.

 {
-  "forcedupload": true,
-  "frozen": true,
+  "forcedupload": false,
+  "frozen": false,
   "name": "mycookbook",
-  "source": "https://github.company.com/_nodeload/chef-cookbooks/mycookbook/legacy.tar.gz/master",
+  "source": "N/A",
   "version": "0.2.2"
 }

And maybe also see if we can always output all files that are missing (one per line), but that one is cosmetic only...

Could be a cookbook specific problem, but needs to be investigated.

I have a client that currently runs a private GitLab server for code repository. Are there any plans to add GitLab or any support besides GitHub?

Possibly with a warning showing the dependencies missing a version which can be bypassed in permissive mode by using --force

I suggest that we use http://goconvey.co/ for the test coverage.

Any thoughts?

Currently when you forked a community cookbook, it will check the community supermarket and the 'forks' Github organisation. If found on Github with the correct tag, it will allow the upload and put the artefact in the private supermarket.

Now every second time that cookbook/version is uploaded, it will do these same checks but it will not check if the cookbook/version is already a artefact in you private supermarket.

Are there reasons to think of why we shouldn't check that one also as a possible source?

When there are 'recipe' lines in the metadata.rb file, the recipe is listed with the cookbook name, which must match the attribute "name".
The error message is not indicative for the error:
ERROR: bad gateway
Response: Failed to upload to the Supermarket: Multipart POST part 'tarball' must contain a README.

Currently it needs a trailing slash when browsing manually, otherwise it will return a bad URL

And maybe more folders are not uploaded. Possibly (likely) only known cookbook subdirs are being uploaded, which means we could/should ignore all missing files that are not in one of those folders in the source cookbook and not in your upload...

When deleting the root databag, the information is not removed on github. This happens when removing the root databag, not when deleting one data bag item.

When you now have additional files locally, compared to Github, the diff may fail on those files. But if these files are in your .gitignore and/or chefignore, then it makes sense that they are missing and shouldn't give you an error.

Maybe an additional way to do this is by excluding all root_files, except for the known once we expect.

When we bootstrap a node, we don't see this back in the audit trail.
Only when we change something to the object it's being recorded.
Looks like we don't catch the node creation calls.

Just like with files that have diffs or are missing

Support caching of result data with a cache (groupcache, memcache with consistent hashing algorithm, etc.) and a JSON language to specify how to invalidate cache (request matches, time, etc.)

Currently when downloading the tar.gz to get the source hashes, that config options is not checked and used. We need to fix that, but that will mean we need to mark the internal and external supermarket differently, so we know when to check this value (as this is only meant for private supermarkets, the public one should always have a valid SSL cert).

In my environment, I can only have services send email from a specific from address. Currently Chef Guard takes the from address from the committers email which is convenient but won't work in my environment. I would like to request that Chef Guard add a configurable from email address in the Chef Guard config file.

Right now you need to run your own Berkshelf API if you also have a private supermarket. If we add the option to use multiple supermarkets, you only need a Berkshelf API when you also have a Chef Server/Organization as cookbook artefact store, or want to index your Github repo's directly (which is not an advised setup by the way).

Double check if there is anything in the code that could be causing this, or if it's just the default limit on some Linux distro's that is too low for cookbooks of this size.

Would be nice to also see who deleted or adjusted the client information.

The metadata.rb file needs to have a name attribute which should match the repo name.
Perhaps also the presence of a metadata.json file messes up chef-guard? I delete the file also, so I cannot test it.

The default for the git installation on Windows is:

Checkout Windows-style, commit Unix-style line endings

This causes Chef Guard to fail with "Cookbook Compare errors" since the local files have \r\n and the repo has \n line endings. It would be nice for Chef Guard to ignore line endings when comparing files.

I've tried to freeze a cookbook but I get an error:

ERROR:   2015/02/10 10:22:56 Failed to get organization ID for : Could not find an organization ID in reply: {"checksums":{"00000000000000000000000000000000":{"needs_upload":true,"url":"http://127.0.0.1:8889/file_store/00000000000000000000000000000000"}},"sandbox_id":"24a1e661d3034f2f5c8149198b5e0d14","uri":"http://127.0.0.1:8889/sandboxes/24a1e661d3034f2f5c8149198b5e0d14"}

Seems like the cookbook module assumes enterprise chef or chef 12 since opensource chef 11 does not have organizations.

Would be nice if you can specify which checks you want to include or exclude (per customer and in general) from the config file...

Currently we always untag the repo, even it the tag was not added by Chef-Guard

Currently it will blacklist all cookbooks when an empty string is used

While doing the file compare, some files are left and checked against the ignore files. These files also end-up in the tar that's uploaded to the supermarket. We should drop these files and make more clean archives, because you could have errors on files that are different and in the ignore if the chefignore is not respected (like when using knife)

1. Create a cookbook named test_1 and upload versions 0.1.0 amd 0.2.0
2. Create and upload a cookbook named test_2 that depends on version 0.1.0 of test_1

depends          'test_1', '0.1.0'

3. Create and upload a cookbook named test_3 that depends on version 0.2.0 of test_1

depends          'test_1', '0.2.0'

4. Upload a role that has test_2 and test_3 cookbooks.

{
  "name": "role_1",
  "description": "Role 1",
  "json_class": "Chef::Role",
  "chef_type": "role",
  "run_list": [
    "recipe[test_2]",
    "recipe[test_3]"
  ]
}

5. View role in Chef Manage
   https://chef.company.com/organizations/myorg/roles/role_1

When the WebUI tried to get the run list of the role, the chef server then goes into an endless loop.

==> /var/log/opscode/nginx/access.log <==
127.0.0.1 - - [12/Mar/2015:10:23:03 -0400]  "POST /organizations/myorg/environments/_default/cookbook_versions HTTP/1.1" 412 "0.069" 394 "-" "Chef Manage/11.16.2 (ruby-2.1.5-p273; ohai-7.4.0; x86_64-linux; +http://opscode.com)" "127.0.0.2:8000" "412" "0.067" "11.16.2" "algorithm=sha1;version=1.0;" "admin" "2015-03-12T14:23:03Z" "asdfgfgSAfscsadf=" 1163

==> /var/log/opscode/opscode-erchef/current <==
2015-03-12_14:23:03.31173 [error] {<<"method=POST; path=/organizations/myorg/environments/_default/cookbook_versions; status=412; ">>,"Precondition Failed"}

==> /var/log/opscode/opscode-erchef/erchef.log <==
2015-03-12 10:23:03.401 [error] {<<"method=POST; path=/organizations/myorg/environments/_default/cookbook_versions; status=412; ">>,"Precondition Failed"}

==> /var/log/opscode/opscode-erchef/crash.log <==
2015-03-12 10:23:03 =ERROR REPORT====
{<<"method=POST; path=/organizations/myorg/environments/_default/cookbook_versions; status=412; ">>,"Precondition Failed"}
==> /var/log/opscode/opscode-erchef/current <==
2015-03-12_14:23:03.40775 [error] {<<"method=POST; path=/organizations/myorg/environments/_default/cookbook_versions; status=412; ">>,"Precondition Failed"}

If I leave the server blank for the supermarket configuration, chef-guard throws an error.

[supermarket]
  server =
  port = 443
  sslnoverify = false
  version = 11.12.0
  user = chef-guard
  key = /opt/chef-guard/supermarket.pem

from /var/log/ghef-guard.log

ERROR:   2014/10/15 16:27:00 Failed to get cookbook list from https:///universe: Get https:///universe: http: no Host in request URL

When trying to upload a cookbook, the server returns a 502 error.

knife cookbook upload test --freeze

Chef-Guard.log

ERROR:   2014/10/15 15:39:31 Failed to dowload recipes/default.rb from the test cookbook: Get
https://chef.company.pvt/bookshelf/organization-00000000000000000000000000000000/checksum-3f4
1d00fdce2d290eee1983e2b85ba16?AWSAccessKeyId=5355300a9f7d0b348d0ff8986f38d5b20cd18895&Expires
=1413401981&Signature=PQ5OVgW9dSge5YqUq2DBHB5Nlw0%3D: x509: certificate signed by unknown authority

With only true/false (on/off) it's kind of hard to migrate from an existing situation to a Chef-Guard managed situation.

With introducing modes, you can choose between silent/permissive/enforced, where silent is like false (off) and enforced is like true (on).

So the new option is permissive which will allow the change, but will give output showing any issues with the change.

Seems this isn't making things easier/better at the moment, so will have a look at an alternative solution.

I have a cookbook with some test fixtures under directory test. I have added the line */test to the file chefignore that is in my cookbook directory.
When I do a knife cookbook upload command chef-guard reports a compare error, as follows:

ERROR: Precondition Failed
Response: 
=== Cookbook Compare errors found ===
The source cookbook contains more files than your upload:
 - test/fixtures/data_bags/sql_server/credentials.json
 - test/fixtures/encrypted_data_bag_secret
 - test/fixtures/environments/test.json

It seems that knife ignores these files accordingly, but chef-guard does not.

Currently you are allowed to upload community cookbooks if they have a new or non-existing version. They will then be treated as private cookbooks.

This is not something you should want. If the cookbook exists in the community you should use an existing version and not create a new private version instead (except for the pending PR use case). As when the community version also bumps the version you now again have two different cookbooks with the same name and version!

I couldn't find any reference to Chef 12.

Could we update the documentation at http://xanzy.io/projects/chef-guard/installation/prerequisites.html to explicit regarding the supported versions of Chef?

Cheers,
-chris

You should never try to upload a forked cookbook, unless you created a PR and am waiting for the PR to be merged... In that case you should be able to move forward and use the altered cookbook.

Updating the normal attributes of a node, Chef Guard commits the default attributes. For example, running knife node edit chf-mynode returns the following:

{
  "name": "chf-mynode",
  "chef_environment": "myenv",
  "normal": {
    "tags": []
  },
  "run_list": [
    "recipe[windows]",
    "recipe[ohai]",
    "recipe[myrecipe]"
 ]
}

If I add attributes at the normal level and save.

{
  "name": "chf-mynode",
  "chef_environment": "myenv",
  "normal": {
    "myattr": {
      "sub1": "data1",
      "sub2": "data2"
    },
    "tags": []
  },
  "run_list": [
    "recipe[windows]",
    "recipe[ohai]",
    "recipe[myrecipe]"
  ]
}

Chef guard commits the default attributes, pulled from the cookbook on the last run, in addition to the normal attributes I just added.

File: nodes/chf-mynode.json
@@ -1,10 +1,42 @@
 {
   "chef_environment": "myenv",
   "chef_type": "node",
-  "default": {},
+  "default": {
+    "chef_handler": {
+      "handler_path": "C:/Chef/handlers",
+      "root_group": "root",
+      "root_user": "root"
+    },
+    "myattr": {
+      "sub1": null,
+      "sub2": null
+    },
+    "ohai": {
+      "hints_path": "C:/chef/ohai/hints",
+      "plugin_path": "C:/chef/ohai_plugins",
+      "plugins": {
+        "ohai": "plugins",
+      }
+    },
+    "windows": {
+      "allow_pending_reboots": true,
+      "reboot_timeout": 60,
+      "rubyzipversion": null
+    }
+  },
   "json_class": "Chef::Node",
   "name": "chf-mynode",
   "normal": {
+    "myattr": {
+      "sub1": "data1",
+      "sub2": "data2"
+    },
     "tags": []
   },
   "override": {},

It seems to me that the default attributes should not be committed since they come from the cookbooks (Which should already be commited) and can change on the next run. Since Chef Guard does not update the default attributes after every run (and should not), the default attributes in the config repo will quickly become out of date and useless.

All replies are expected to contain a JSON formatted body, but when you receive a HTTP error this will be HTML instead. In this case the error you receive is about being unable to parse the JSON as it contains an invalid '<' character.

So we need to make the parsing depended on the HTTP result (or something along those lines)

This causes (over time) hitting the max open file handles...

For the following configuration:

[community]
  supermarket = https://supermarket.getchef.com
  forks = 

Chef Guard will throw a un-descriptive error:

Failed to create custom Git client: No Github config specified for organization: !

Chef Guard is attempting to search forks even when the value is blank. Either the field should be optional, or Chef Guard should fail to load the config when forks is empty.

When updating a role that depends on a cookbook that isn't frozen in permissive mode, Chef Guard says that the changes are saved but when I look at the chef server and the github repo, the updated role is not there.

$ knife role from file roles/test_production.json          
ERROR: Precondition Failed                                 
Response:                                                  
==== Cookbook Constraints errors found ====                
RUNNNING PERMISSIVE MODE: CHANGES ARE SAVED                

 - mycookbook version 0.2.0 needs to be frozen            
===========================================                

Either the chnages should be saved or Chef Guard should not say that they are. I am not sure which one is correct.

After upgrading to Chef 12, I was able to get Chef Guard working. I then installed opscode-manage and made a few changes through the WebUI. I noticed that these changes were not logged to the config repo by Chef Guard.

Initial I configured the chef server like the Chef Guard instructions recommend. This works for the knife commands but not the WebUI.

lb['upstream'] = {
    "opscode-erchef"=>["127.0.0.2"],
}

Assuming opscode-manage was ignoring the opscode-erchef setting and talking directly to 127.0.0.1, I moved opscode_erchef to 127.0.0.2 and had Ghef Guard listen on 127.0.0.1.

# /etc/opscode/chef-server.rb
opscode_erchef['listen'] = "127.0.0.2"

# /opt/chef-guard/chef-guard.conf
[default]
  listen = 127.0.0.1
[chef]
  erchefip = 127.0.0.2

This had the same result as before, knife goes through Chef Guard but the WebUI does not.

I looked into the configuration of opscode-manage, but did not see where to tell it to look for erchef. https://docs.chef.io/config_rb_manage.html

If I update a group of roles (but not all my roles) and upload them all, Chef guard will commit roles that haven't changed.

knife role from file roles/test*.json

Any roles that haven't changed will get a commit with 0 changed files with 0 additions and 0 deletions. A diff email will not be sent for this commit.

I think Chef Guard should not commit changes that don't change anything.

Hey. Sorry can't find anyone using chef-guard on IRC, so asking here.

I've just installed chef-guard and not sure if everything is configured correctly. However there're no errors in logs and it creates commits in config repo for each uploaded cookbook, so I assume it works.

The problem is it looks like it's not running tests for some reason. I'm checking the cookbooks locally with rubocop and foodcritic and they both give lots of errors. However it gets uploaded just fine.
Both rubocop and foodcritic have correct paths in config and exist on the server.

My config:

[default]
  listen = 127.0.0.2
  logfile = /var/log/chef-guard.log
  tempdir = /var/tmp/chef-guard
  mode = silent
  maildomain =
  mailserver =
  mailport = 25
  mailrecipient =
  validatechanges = permissive
  commitchanges = true
  mailchanges = false
  searchgithub = true
  publishcookbook = false
  blacklist =
  gitorganization = 123
  gitcookbookorgs = 123
  includefcs =
  excludefcs =

[chef]
  enterprisechef = false
  server = chef.123
  port = 443
  sslnoverify = false
  erchefip = 127.0.0.1
  erchefport = 8000
  version = 11.12.4
  user = chef-guard
  key = /opt/chef-guard/chef-guard.pem
  s3key = 123
  s3secret = 123

[community]
  supermarket = https://supermarket.getchef.com
  forks =

[supermarket]
  server = supermarket.company.com
  port = 443
  sslnoverify = false
  version = 11.12.0
  user = chef-guard
  key = /opt/chef-guard/supermarket.pem

[graphite]
  server =
  port = 2003

[chefclients]
  path = /opt/chef-guard/clients

[tests]
  foodcritic = /opt/chef/embedded/bin/foodcritic
  rubocop = /opt/chef/embedded/bin/rubocop

[github "123"]
  sslnoverify = false
  token = 123

I'm trying to run chef-guard on my machine, and I'm confused with the listen, erchefport, port parameters. Seems like guard uses the listen parameter and the erchefport to bind to. I would want chef-guard to listen on a different port then the chef server on the same host, and it seems like a config option is missing for it.

Hi,

Quick one, when trying to run the cookbook it complains about trying to get the key from s3.

Is there anyway to ignore this bit, can we just comment out the s3 part?

Regards
Mark

Details:
�[0m
ChefVault::Exceptions::KeysNotFound�[0m
-----------------------------------�[0m
chef-guard/chef.s3_keys could not be found�[0m