cloudfoundry-community / bosh-gen Goto Github PK

View Code? Open in Web Editor NEW

68.0 21.0 21.0 657 KB

Rapid generation of BOSH releases

License: MIT License

Ruby 76.68% Shell 23.32%

cloudfoundry devops ruby bosh

bosh-gen's People

Contributors

Stargazers

Watchers

bosh-gen's Issues

.blobs/ looks like it should be in default .gitignore

When following the example in the README, the process fails when trying to create a release after moving Ruby to the blob store. This looks to be because the .blobs directory that is created as the release starts then appears as an uncommitted change. Adding .blobs to the .gitignore file makes this go away, and given the files in it appear to be large binaries, I assume it is meant to be ignored?

$ rm -rf .blobs
$ bosh create release
Syncing blobs...
ruby/ruby-1.9.3-p194.tar.gz downloaded                                          

# On branch master
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#   .blobs/
nothing added to commit but untracked files present (use "git add" to track)

Your current directory has some local modifications, please discard or commit them first

extract-pkg - expand splats

common/bin/* doesn't example to fetch all files

bosh-gem new error

Error when new, the same of AWS or OpenStack or vSphere:
vagrant@bosh-lite:~$ bosh-gen new test
create

AWS
OpenStack
vSphere
Choose your infrastructure: 3

Using provider vSphere

Username: 1
Password: 1
Server: 1
/var/lib/gems/1.9.1/gems/cyoi-0.11.3/lib/cyoi/cli/providers/provider_cli_vsphere.rb:22:in setup_credentials': undefined local variable or methodh1' for #Cyoi::Cli::Providers::ProviderCliVsphere:0x0000000182ce58 (NameError)
from /var/lib/gems/1.9.1/gems/cyoi-0.11.3/lib/cyoi/cli/providers/provider_cli_vsphere.rb:6:in perform_and_return_attributes' from /var/lib/gems/1.9.1/gems/cyoi-0.11.3/lib/cyoi/cli/provider.rb:18:inexecute!'
from /var/lib/gems/1.9.1/gems/bosh-gen-0.20.1/lib/bosh/gen/generators/new_release_generator.rb:28:in select_provider' from /usr/lib/ruby/vendor_ruby/thor/command.rb:27:inrun'
from /usr/lib/ruby/vendor_ruby/thor/invocation.rb:121:in invoke_command' from /usr/lib/ruby/vendor_ruby/thor/invocation.rb:128:inblock in invoke_all'
from /usr/lib/ruby/vendor_ruby/thor/invocation.rb:128:in each' from /usr/lib/ruby/vendor_ruby/thor/invocation.rb:128:inmap'
from /usr/lib/ruby/vendor_ruby/thor/invocation.rb:128:in invoke_all' from /usr/lib/ruby/vendor_ruby/thor/group.rb:232:indispatch'
from /usr/lib/ruby/vendor_ruby/thor/base.rb:440:in start' from /var/lib/gems/1.9.1/gems/bosh-gen-0.20.1/lib/bosh/gen/cli.rb:18:innew'
from /usr/lib/ruby/vendor_ruby/thor/command.rb:27:in run' from /usr/lib/ruby/vendor_ruby/thor/invocation.rb:121:ininvoke_command'
from /usr/lib/ruby/vendor_ruby/thor.rb:363:in dispatch' from /usr/lib/ruby/vendor_ruby/thor/base.rb:440:instart'
from /var/lib/gems/1.9.1/gems/bosh-gen-0.20.1/bin/bosh-gen:7:in <top (required)>' from /usr/local/bin/bosh-gen:23:inload'
from /usr/local/bin/bosh-gen:23:in `

ruby version: ruby 1.9.3p484 (2013-11-22 revision 43786) [x86_64-linux]

ruby gem list:
gem list

*** LOCAL GEMS ***

activesupport (4.2.1)
aws-sdk (1.60.2)
aws-sdk-v1 (1.60.2)
blobstore_client (1.2983.0, 1.2820.0)
bosh-gen (0.20.1)
bosh-template (1.2983.0, 1.2820.0)
bosh_cli (1.2820.0)
bosh_common (1.2983.0, 1.2820.0)
builder (3.2.2)
bundler (1.3.5)
cf-message-bus (0.3.1)
cf-registrar (1.0.3)
cf-uaa-lib (3.2.1)
CFPropertyList (2.3.1)
cyoi (0.11.3)
daemons (1.1.9)
eventmachine (1.0.3)
excon (0.45.3, 0.43.0)
fission (0.5.0)
fluent-logger (0.4.9)
fog (1.27.0, 1.23.0)
fog-atmos (0.1.0)
fog-aws (0.1.1)
fog-brightbox (0.7.1)
fog-core (1.30.0, 1.27.3)
fog-ecloud (0.1.3)
fog-google (0.0.5)
fog-json (1.0.0)
fog-local (0.2.1)
fog-powerdns (0.1.1)
fog-profitbricks (0.0.3)
fog-radosgw (0.0.4)
fog-riakcs (0.1.0)
fog-sakuracloud (1.0.1)
fog-serverlove (0.1.2)
fog-softlayer (0.3.30)
fog-storm_on_demand (0.1.1)
fog-terremark (0.1.0)
fog-vmfusion (0.1.0)
fog-voxel (0.1.0)
fog-xml (0.1.2)
formatador (0.2.5)
highline (1.6.21)
httpclient (2.4.0)
i18n (0.7.0)
inflecto (0.0.2)
ipaddress (0.8.0)
json (1.8.2)
json_pure (1.8.2, 1.8.1)
little-plugger (1.1.3)
log4r (1.1.10)
logging (1.8.2)
mime-types (2.4.3)
mini_portile (0.6.2)
minitar (0.5.4)
minitest (5.7.0)
msgpack (0.5.9)
multi_json (1.10.1)
nats (0.5.0.beta.16, 0.5.0.beta.14)
net-http-persistent (2.9)
net-scp (1.1.2)
net-ssh (2.9.2)
net-ssh-gateway (1.2.0)
netaddr (1.5.0)
nokogiri (1.6.6.1)
progressbar (0.9.2)
rack (1.5.2)
rdoc (3.9.4)
readwritesettings (3.0.1)
ruby-atmos-pure (1.0.5)
ruby-hmac (0.4.0)
semi_semantic (1.1.0)
steno (1.2.4)
terminal-table (1.4.5)
thin (1.6.3, 1.6.2)
thor (0.18.1.20140116)
thread_safe (0.3.5)
tzinfo (1.2.2)
vcap-concurrency (0.1.0)
yajl-ruby (1.2.1)

make_manifest incorrectly detects CPI on bosh-lite

Hi,

I'm working on a release that requires a CentoOS stemcell. When I set STEMCELL_OS=centos make_manifest fails because it can't find an appropriate stemcell. If I'm not wrong the root cause of this issue is here:

bosh-gen/lib/bosh/gen/generators/new_release_generator/templates/templates/make_manifest.tt

Line 20 in 602fcab

DIRECTOR_CPI=$(echo "$BOSH_STATUS" | grep CPI | awk '{print $2}')

It expects 'bosh status' to return CPI=warden while at least on my environment there is 'cpi' there:
Director
Name Bosh Lite Director
URL https://192.168.50.4:25555
Version 1.3074.0 (00000000)
User admin
UUID 9bca5701-66c0-43ee-a4d0-64813b4b5ece
CPI cpi
dns disabled
compiled_package_cache enabled (provider: local)
snapshots disabled

When I manually set DIRECTOR_CPI=warden everything works well and the right stemcell is uploaded.

Thanks,

[new] Random 10.244.X.0 for warden templates

Currently the generated warden templates are guaranteed to conflict with another deployment - they are all 10.244.2.0/30 upwards. Perhaps pick a random 10.244.X.0 starting IP.

generator to create a golang-based package

Cannot install latest bosh-gen

I tried to install latest bosh-gen but it failed.

gem install bosh-gen -v 0.21.1 --no-ri --no-rdoc
ERROR:  While executing gem ... (Gem::DependencyResolutionError)
    conflicting dependencies fog (~> 1.34.0) and fog (~> 1.11)
  Activated fog-1.11.0
  which does not match conflicting dependency (~> 1.34.0)

  Conflicting dependency chains:
    bosh-gen (= 0.21.1), 0.21.1 activated, depends on
    fog (~> 1.11), 1.11.0 activated

  versus:
    bosh-gen (= 0.21.1), 0.21.1 activated, depends on
    bosh_cli (>= 0), 1.3104.0 activated, depends on
    blobstore_client (~> 1.3104.0), 1.3104.0 activated, depends on
    fog (~> 1.34.0)

It seems current version of blobstore_client requires fog '~~>1.34.0' but bosh-gen requires '~~> 1.11'
https://github.com/cloudfoundry/bosh/blob/master/blobstore_client/blobstore_client.gemspec#L24

As a workaround, install a bit older version of bosh_cli(1.3094.0) at first and it successfully installed.

$ gem install bosh_cli -v 1.3094.0 --no-ri --no-rdoc
$ gem install bosh-gen -v 0.21.1 --no-ri --no-rdoc

Can't create repo

I want to create simple release consisted of apache job + apache package. But see this when I try to init release:

$ bosh-gen new apache                                                            
      create
Auto-detected infrastructure API credentials at ~/.fog (override with $FOG)
/Users/lexsys/.rvm/gems/ruby-2.1.5/gems/cyoi-0.11.2/lib/cyoi/cli/auto_detection/auto_detection_fog.rb:56:in `auto_detection_choices': undefined method `inject' for false:FalseClass (NoMethodError)
    from /Users/lexsys/.rvm/gems/ruby-2.1.5/gems/cyoi-0.11.2/lib/cyoi/cli/auto_detection.rb:41:in `block in aggregated_detector_choices'
    from /Users/lexsys/.rvm/gems/ruby-2.1.5/gems/cyoi-0.11.2/lib/cyoi/cli/auto_detection.rb:39:in `each'
    from /Users/lexsys/.rvm/gems/ruby-2.1.5/gems/cyoi-0.11.2/lib/cyoi/cli/auto_detection.rb:39:in `inject'
    from /Users/lexsys/.rvm/gems/ruby-2.1.5/gems/cyoi-0.11.2/lib/cyoi/cli/auto_detection.rb:39:in `aggregated_detector_choices'
    from /Users/lexsys/.rvm/gems/ruby-2.1.5/gems/cyoi-0.11.2/lib/cyoi/cli/auto_detection.rb:18:in `perform'
    from /Users/lexsys/.rvm/gems/ruby-2.1.5/gems/cyoi-0.11.2/lib/cyoi/cli/provider.rb:57:in `auto_detection'
    from /Users/lexsys/.rvm/gems/ruby-2.1.5/gems/cyoi-0.11.2/lib/cyoi/cli/provider.rb:16:in `execute!'
    from /Users/lexsys/.rvm/gems/ruby-2.1.5/gems/bosh-gen-0.18.4/lib/bosh/gen/generators/new_release_generator.rb:28:in `select_provider'
    from /Users/lexsys/.rvm/gems/ruby-2.1.5/gems/thor-0.19.1/lib/thor/command.rb:27:in `run'
    from /Users/lexsys/.rvm/gems/ruby-2.1.5/gems/thor-0.19.1/lib/thor/invocation.rb:126:in `invoke_command'
    from /Users/lexsys/.rvm/gems/ruby-2.1.5/gems/thor-0.19.1/lib/thor/invocation.rb:133:in `block in invoke_all'
    from /Users/lexsys/.rvm/gems/ruby-2.1.5/gems/thor-0.19.1/lib/thor/invocation.rb:133:in `each'
    from /Users/lexsys/.rvm/gems/ruby-2.1.5/gems/thor-0.19.1/lib/thor/invocation.rb:133:in `map'
    from /Users/lexsys/.rvm/gems/ruby-2.1.5/gems/thor-0.19.1/lib/thor/invocation.rb:133:in `invoke_all'
    from /Users/lexsys/.rvm/gems/ruby-2.1.5/gems/thor-0.19.1/lib/thor/group.rb:232:in `dispatch'
    from /Users/lexsys/.rvm/gems/ruby-2.1.5/gems/thor-0.19.1/lib/thor/base.rb:440:in `start'
    from /Users/lexsys/.rvm/gems/ruby-2.1.5/gems/bosh-gen-0.18.4/lib/bosh/gen/cli.rb:18:in `new'
    from /Users/lexsys/.rvm/gems/ruby-2.1.5/gems/thor-0.19.1/lib/thor/command.rb:27:in `run'
    from /Users/lexsys/.rvm/gems/ruby-2.1.5/gems/thor-0.19.1/lib/thor/invocation.rb:126:in `invoke_command'
    from /Users/lexsys/.rvm/gems/ruby-2.1.5/gems/thor-0.19.1/lib/thor.rb:359:in `dispatch'
    from /Users/lexsys/.rvm/gems/ruby-2.1.5/gems/thor-0.19.1/lib/thor/base.rb:440:in `start'
    from /Users/lexsys/.rvm/gems/ruby-2.1.5/gems/bosh-gen-0.18.4/bin/bosh-gen:7:in `<top (required)>'
    from /Users/lexsys/.rvm/gems/ruby-2.1.5/bin/bosh-gen:23:in `load'
    from /Users/lexsys/.rvm/gems/ruby-2.1.5/bin/bosh-gen:23:in `<main>'
    from /Users/lexsys/.rvm/gems/ruby-2.1.5/bin/ruby_executable_hooks:15:in `eval'
    from /Users/lexsys/.rvm/gems/ruby-2.1.5/bin/ruby_executable_hooks:15:in `<main>'

wait_pid() kills just to root process not its childs as well

Hello,

the wait_pid() function in the ctl_utils.sh is killing just to root process and not the processes started by it. In case the starting script start another script with on its turn starts another and so on, stopping of the root process would not stop all the child processes.

It would be nice if this is extended so the whole tree is killed.

Regards,
Milko

extract-pkg to use `bosh2 add-blob`

Allow users to get read/write s3 keys to community blobstore

A backend API to map public users to community s3 account may be required for this feature. And perhaps each bucket policy should restrict to specific users, etc.

Selecting vSphere as the target results in error

I'm using bosh-gen to create a bosh release and when I select vsphere as the platform I get the following stack strace. It works for AWS.

/Users/jomeara/.rbenv/versions/2.0.0-p645/lib/ruby/gems/2.0.0/gems/cyoi-0.11.3/lib/cyoi/cli/providers/provider_cli_vsphere.rb:22:in `setup_credentials': undefined local variable or method`h1' for #Cyoi::Cli::Providers::ProviderCliVsphere:0x007fa6e2907910 (NameError)
    from /Users/jomeara/.rbenv/versions/2.0.0-p645/lib/ruby/gems/2.0.0/gems/cyoi-0.11.3/lib/cyoi/cli/providers/provider_cli_vsphere.rb:6:in `perform_and_return_attributes'
    from /Users/jomeara/.rbenv/versions/2.0.0-p645/lib/ruby/gems/2.0.0/gems/cyoi-0.11.3/lib/cyoi/cli/provider.rb:18:in`execute!'
    from /Users/jomeara/.rbenv/versions/2.0.0-p645/lib/ruby/gems/2.0.0/gems/bosh-gen-0.22.0/lib/bosh/gen/generators/new_release_generator.rb:28:in `select_provider'
    from /Users/jomeara/.rbenv/versions/2.0.0-p645/lib/ruby/gems/2.0.0/gems/thor-0.19.1/lib/thor/command.rb:27:in`run'
    from /Users/jomeara/.rbenv/versions/2.0.0-p645/lib/ruby/gems/2.0.0/gems/thor-0.19.1/lib/thor/invocation.rb:126:in `invoke_command'
    from /Users/jomeara/.rbenv/versions/2.0.0-p645/lib/ruby/gems/2.0.0/gems/thor-0.19.1/lib/thor/invocation.rb:133:in`block in invoke_all'
    from /Users/jomeara/.rbenv/versions/2.0.0-p645/lib/ruby/gems/2.0.0/gems/thor-0.19.1/lib/thor/invocation.rb:133:in `each'
    from /Users/jomeara/.rbenv/versions/2.0.0-p645/lib/ruby/gems/2.0.0/gems/thor-0.19.1/lib/thor/invocation.rb:133:in`map'
    from /Users/jomeara/.rbenv/versions/2.0.0-p645/lib/ruby/gems/2.0.0/gems/thor-0.19.1/lib/thor/invocation.rb:133:in `invoke_all'
    from /Users/jomeara/.rbenv/versions/2.0.0-p645/lib/ruby/gems/2.0.0/gems/thor-0.19.1/lib/thor/group.rb:232:in`dispatch'
    from /Users/jomeara/.rbenv/versions/2.0.0-p645/lib/ruby/gems/2.0.0/gems/thor-0.19.1/lib/thor/base.rb:440:in `start'
    from /Users/jomeara/.rbenv/versions/2.0.0-p645/lib/ruby/gems/2.0.0/gems/bosh-gen-0.22.0/lib/bosh/gen/cli.rb:18:in`new'
    from /Users/jomeara/.rbenv/versions/2.0.0-p645/lib/ruby/gems/2.0.0/gems/thor-0.19.1/lib/thor/command.rb:27:in `run'
    from /Users/jomeara/.rbenv/versions/2.0.0-p645/lib/ruby/gems/2.0.0/gems/thor-0.19.1/lib/thor/invocation.rb:126:in`invoke_command'
    from /Users/jomeara/.rbenv/versions/2.0.0-p645/lib/ruby/gems/2.0.0/gems/thor-0.19.1/lib/thor.rb:359:in `dispatch'
    from /Users/jomeara/.rbenv/versions/2.0.0-p645/lib/ruby/gems/2.0.0/gems/thor-0.19.1/lib/thor/base.rb:440:in`start'
    from /Users/jomeara/.rbenv/versions/2.0.0-p645/lib/ruby/gems/2.0.0/gems/bosh-gen-0.22.0/bin/bosh-gen:7:in `<top (required)>'
    from /Users/jomeara/.rbenv/versions/2.0.0-p645/bin/bosh-gen:23:in`load'
    from /Users/jomeara/.rbenv/versions/2.0.0-p645/bin/bosh-gen:23:in `<main>'

[new] Add LICENSE.md file

extract-pkg - recreate git submodules

Currently, src/ submodules aren't created.

bosh-gen new does not create bucket in another region; nor setup config/final.yml

I specified eu-central-1 as the region for the bucket, but:

config/final.yml did not specify region: eu-central-1
bucket was actually created in us-east-1 - I think this needs fixing in https://github.com/cloudfoundry-community/cyoi

ctl_setup - auto load from job's packages only

Instead of

bosh-gen/lib/bosh/gen/generators/job_generator/templates/jobs/%job_name%_nginx_rack/templates/bin/ctl_setup.sh.erb

Lines 29 to 32 in e547126

 for package_bin_dir in $(ls -d /var/vcap/packages/*/{,s}bin) 

 do 

 export PATH=${package_bin_dir}:$PATH 

 done

which loads from global set of packages from across all colocated jobs, look up from the job's packages only /var/vcap/jobs/<myjob>/packages/*

Stack trace when attempting `bosh-gen manifest`

Without latest_release_filename specified in config/dev.yml, manifest creation blows up and its not obvious what to do without looking at the source.

mkb@beemo ~/code/counter-boshrelease [master]
± % bosh-gen manifest counter-demo .                                                                  [ruby-2.1.5@bosh]
/Users/mkb/.rvm/gems/ruby-2.1.5@bosh/gems/bosh-gen-0.18.4/lib/bosh/gen/models/release_detection.rb:12:in `expand_path': no implicit conversion of nil into String (TypeError)
        from /Users/mkb/.rvm/gems/ruby-2.1.5@bosh/gems/bosh-gen-0.18.4/lib/bosh/gen/models/release_detection.rb:12:in `initialize'
        from /Users/mkb/.rvm/gems/ruby-2.1.5@bosh/gems/bosh-gen-0.18.4/lib/bosh/gen/generators/deployment_manifest_generator.rb:55:in `new'
        from /Users/mkb/.rvm/gems/ruby-2.1.5@bosh/gems/bosh-gen-0.18.4/lib/bosh/gen/generators/deployment_manifest_generator.rb:55:in `release_detector'
        from /Users/mkb/.rvm/gems/ruby-2.1.5@bosh/gems/bosh-gen-0.18.4/lib/bosh/gen/generators/deployment_manifest_generator.rb:73:in `detect_jobs'
        from /Users/mkb/.rvm/gems/ruby-2.1.5@bosh/gems/bosh-gen-0.18.4/lib/bosh/gen/generators/deployment_manifest_generator.rb:78:in `jobs'
        from /Users/mkb/.rvm/gems/ruby-2.1.5@bosh/gems/bosh-gen-0.18.4/lib/bosh/gen/generators/deployment_manifest_generator.rb:29:in `check_valid_requested_jobs'
        from /Users/mkb/.rvm/gems/ruby-2.1.5@bosh/gems/thor-0.19.1/lib/thor/command.rb:27:in `run'
        from /Users/mkb/.rvm/gems/ruby-2.1.5@bosh/gems/thor-0.19.1/lib/thor/invocation.rb:126:in `invoke_command'
        from /Users/mkb/.rvm/gems/ruby-2.1.5@bosh/gems/thor-0.19.1/lib/thor/invocation.rb:133:in `block in invoke_all'
        from /Users/mkb/.rvm/gems/ruby-2.1.5@bosh/gems/thor-0.19.1/lib/thor/invocation.rb:133:in `each'
        from /Users/mkb/.rvm/gems/ruby-2.1.5@bosh/gems/thor-0.19.1/lib/thor/invocation.rb:133:in `map'
        from /Users/mkb/.rvm/gems/ruby-2.1.5@bosh/gems/thor-0.19.1/lib/thor/invocation.rb:133:in `invoke_all'
        from /Users/mkb/.rvm/gems/ruby-2.1.5@bosh/gems/thor-0.19.1/lib/thor/group.rb:232:in `dispatch'
        from /Users/mkb/.rvm/gems/ruby-2.1.5@bosh/gems/thor-0.19.1/lib/thor/base.rb:440:in `start'
        from /Users/mkb/.rvm/gems/ruby-2.1.5@bosh/gems/bosh-gen-0.18.4/lib/bosh/gen/cli.rb:134:in `manifest'
        from /Users/mkb/.rvm/gems/ruby-2.1.5@bosh/gems/thor-0.19.1/lib/thor/command.rb:27:in `run'
        from /Users/mkb/.rvm/gems/ruby-2.1.5@bosh/gems/thor-0.19.1/lib/thor/invocation.rb:126:in `invoke_command'
        from /Users/mkb/.rvm/gems/ruby-2.1.5@bosh/gems/thor-0.19.1/lib/thor.rb:359:in `dispatch'
        from /Users/mkb/.rvm/gems/ruby-2.1.5@bosh/gems/thor-0.19.1/lib/thor/base.rb:440:in `start'
        from /Users/mkb/.rvm/gems/ruby-2.1.5@bosh/gems/bosh-gen-0.18.4/bin/bosh-gen:7:in `<top (required)>'
        from /Users/mkb/.rvm/gems/ruby-2.1.5@bosh/bin/bosh-gen:23:in `load'
        from /Users/mkb/.rvm/gems/ruby-2.1.5@bosh/bin/bosh-gen:23:in `<main>'
        from /Users/mkb/.rvm/gems/ruby-2.1.5@bosh/bin/ruby_executable_hooks:15:in `eval'
        from /Users/mkb/.rvm/gems/ruby-2.1.5@bosh/bin/ruby_executable_hooks:15:in `<main>'

Trying to install bosh-gen get missing fog-aws 0.1.1, can't gem install it.

fog-aws (0.12.0)

$ gem install bosh-gen
ERROR:  While executing gem ... (Gem::DependencyError)
    Unable to resolve dependencies: bosh-gen requires fog-aws (= 0.1.1)

$ sudo gem install fog-aws 0.1.1
Successfully installed fog-aws-0.12.0
ERROR:  Could not find a valid gem '0.1.1' (>= 0) in any repository

bosh-gen dependencies need to be updated to include mime-types gem

After installing bosh-gen a run through to complete a bosh-gen new <release_name> did not complete because it was missing a required library of mime-types.

Let's update the gem specifications so this gets installed with the bosh-gen gem.

bosh2 config/private.yml schema changed

cloudfoundry/bosh-cli#179

'job -d some-dep' is not putting 'some-dep' in spec file

The template for jobs/name/spec never populates the list of package dependecies

Generate CI pipeline to build and test

single node/no-leader vs cluster (leader/slaves) flag for new projects

`bosh-gen extract-pkg` did not get assets from blobs/x if assets also in src/x

manifest - create deployments/<name>.yml instead of <name>/manifest.yml

This will fit in with how the CLI expects the folder structure to be

[major] upgrade to bosh2 cli, links, cloud-config

fail fast if bosh2 missing
boshlite cloud-config.yml with default name for each concept; plus a public network
config/private.yml schema changed cloudfoundry/bosh-cli#179
use bosh2 add-blob to copy in blobs

See branch https://github.com/cloudfoundry-community/bosh-gen/tree/bosh2

Dependency conflict in 0.18.4

Error message:
$ gem install bosh-gen
ERROR: While executing gem ... (Gem::DependencyResolutionError)
conflicting dependencies net-scp (~~> 1.0.4) and net-scp (~~> 1.1)
Activated net-scp-1.1.0 via:
net-scp-1.1.0 (~~> 1.1), fog-1.11.0 (~~> 1.11), bosh-gen-0.18.4 (= 0.18.4)
instead of (~> 1.0.4) via:
bosh_cli-0.16 (>= 0), bosh-gen-0.18.4 (= 0.18.4)

It seems that version 0.18.4 has a versioning conflict in regard to net-scp.

BTW: Version 0.16.2 is still working.

Cheers,
Julian W

Generated spiff templates to use new `templates:` for jobs

initial job ctl could run hello world script

/bin/sh -c "while true; do echo hello world; sleep 1; done"

Better than non-working "TODO"

Generated README should focus on end-user (final releases)

retain chmod of copied files

If copying over an executable, ensure its still an executable

Add "package <name> -f path/to/blob-1.2.3.tgz -v 1.2.3"

Packages don't need to have explicit versions in files. The above example should produce:

packages/name/spec

---
name: softhsm
dependencies: []
files:
- softhsm/SoftHSMv2-*.tar.gz

packages/name/packaging

...
tar xfv softhsm/SoftHSMv2-*.tar.gz
cd SoftHSMv2-*
./configure --prefix=${BOSH_INSTALL_TARGET}
make -j${CPUS} && make install
# Alternatively, to copy archive contents:
# cp -a SoftHSMv2-*/* $BOSH_INSTALL_TARGET

Copy file globs (e.g. java/*)

Currently its not exploding out globs and copying over files (in both extract-pkg & extract-job)

bosh-gen new - if project has hyphens then job should have underscores

README to reference bosh.io release urls

bosh-lite releases

Is there a way to generate a bosh-lite release? Old blog posts and video demo mentions it but I cannot figure out how to go about it as I am not sure which of the three options (1. AWS 2. OpenStack 3. vSphere) I am supposed to pick when starting new.

package -f dir should assume files 'dir/**/*'

If there is a folder src/gopath/src/extra/... and the command run is:

bosh-gen package myapp -f gopath

Then the generated packages/myapp/spec should be:

name: myapp
files:
- gopath/**/*

make_manifest - use bosh.io to find latest stemcell

Pretty bad bug - it downloads an old warden stemcell that doesn't work with new bosh-lite

Do not copy files into packages - they are immutable

From discussion at https://groups.google.com/a/cloudfoundry.org/forum/?fromgroups#!topic/bosh-users/4WOCMiK6Sjc%5B1-25%5D

bosh upload blobs

The README does not specify how to configure 'bosh upload release', so it crashes.

[openstack] don't include openstack_region if not set

Tutorial outdated?

I am trying to follow the steps on the tutorial provided in README for setting up a cassandra release however I get stuck at the following step:

bosh-gen extract-pkg ../cf-release/packages/dea_jvm7

There is not directory called dea_jvm7. Could it be called something else?

create version branch for CI

https://github.com/starkandwayne/concourse-tutorial/tree/master/20_versions_and_buildnumbers#setup-with-a-git-branch

Set s3 bucket policy on creation

policy:

{
    "Version": "2008-10-17",
    "Statement": [
        {
            "Sid": "AddPerm",
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::NAME-boshrelease/*"
        }
    ]
}

Pickup existing AWS credentials from ~/.aws/credentials

Currently bosh-gen only looks in ~/.fog for known credentials. Also look in ~/.aws/credentials using https://github.com/TwP/inifile parser:

myini = IniFile.load(File.join(ENV['HOME'], '.aws/credentials'))

Perhaps this needs to go in cyoi library

source helper/ctl_setup hangs on redirect_output ${output_label}

it seems that the redirect_output ${output_label} functions hangs
when i manually source this file bash hangs

https://github.com/cloudfoundry-community/bosh-gen/blob/master/lib/bosh/gen/generators/job_generator/templates/jobs/%25job_name%25_simple/templates/helpers/ctl_setup.sh#L28

generate version branch

https://github.com/starkandwayne/concourse-tutorial/tree/master/20_versions_and_buildnumbers

package -f blob/blob.tgz didn't work as expected

$ tree blobs
blobs
└── softhsm
    └── SoftHSMv2-2.3.0.tar.gz

$ bosh-gen package softhsm -f softhsm/SoftHSMv2-2.3.0.tar.gz
      create  packages/softhsm/packaging
tar: Error opening archive: Failed to open 'softhsm/SoftHSMv2-2.3.0.tar.gz'
Skipping unknown file /Users/drnic/Projects/bosh_releases/softhsm-boshrelease/softhsm/SoftHSMv2-2.3.0.tar.gz
      create  packages/softhsm/spec

The way that worked was to include blobs/ - but it would be good to support existing blobs

$ bosh-gen package softhsm -f blobs/softhsm/SoftHSMv2-2.3.0.tar.gz

Feedback: packages created from "apt" are hard to use

I use a "GitHub issue" to share this experience: we were very interested in the "apt" approach for packages but it tends to be very hard to use when the installed package define configuration file under /etc/...

For example, we wanted to use the apt approach to install 'fontconfig' and 'ttf-dejavu' for OpenJDK8 and we switched back to the approach based on source code and compilation.

For example, the fc-cache tool of 'fontconfig' did not want to work under /var/vcap/packages/fontconfig/apt/...:

couldn't find the required '.so', we had to play with LD_LIBRARY_PATH
couldn't find a config file expected under '/etc/...', the param sysroot did not help so I gave up and switched back from apt to src+compilation

As a conclusion, I am convinced that installing native packages with apt is the right way to go but these packages should be installed under their default folder tree and I guess we will have to wait for a docker style approach to do it.

Setting LD_LIBRARY_PATH makes jobs fail on DEA nodes.

Aloha!

Im using bosh-gen to generate a release/job for a check_mk agent for monitoring purposes of our CF installation. The release works like a charm on all the jobs except the DEAs.

In ctl_setup.sh

export LD_LIBRARY_PATH=${LD_LIBRARY_PATH:-''} # default to empty
for package_bin_dir in $(ls -d /var/vcap/packages/*/lib)
do
  export LD_LIBRARY_PATH=${package_bin_dir}:$LD_LIBRARY_PATH
done

On a DEA jobs this will cause issues since the VMs have the rootfs_lucid64 package installed.
On a DEA machine LD_LIBRARY_PATH will end up being /var/vcap/packages/ruby/lib:/var/vcap/packages/rootfs_lucid64/lib:/var/vcap/packages/dea_next/lib

Right after we set the LD_LIBRARY_PATH we continue in the ctl_setup script;

export RUN_DIR=/var/vcap/sys/run/$JOB_NAME
export LOG_DIR=/var/vcap/sys/log/$JOB_NAME
export TMP_DIR=/var/vcap/sys/tmp/$JOB_NAME
export STORE_DIR=/var/vcap/store/$JOB_NAME
for dir in $RUN_DIR $LOG_DIR $TMP_DIR $STORE_DIR
do
  mkdir -p ${dir}
  chown vcap:vcap ${dir}
  chmod 775 ${dir}
done
export TMPDIR=$TMP_DIR

At this point mkdir will fail with a segmentation fault, and thus the bin/JOB_NAME_ctl script used by monit to start the job.

Here are some steps to reproduce the issue (on a DEA node);

$ ldd /bin/mkdir
    linux-vdso.so.1 =>  (0x00007fff4f6df000)
    libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 (0x00007fc64eff0000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc64ec2a000)
    libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007fc64e9eb000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fc64e7e7000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fc64f21c000)

$ export LD_LIBRARY_PATH=/var/vcap/packages/ruby/lib:/var/vcap/packages/rootfs_lucid64/lib:/var/vcap/packages/dea_next/lib

$ mkdir
mkdir: /var/vcap/packages/rootfs_lucid64/lib/libc.so.6: version `GLIBC_2.14' not found (required by mkdir)

# To fix
$ export LD_LIBRARY_PATH=/lib/x86_64-linux-gnu:/var/vcap/packages/ruby/lib:/var/vcap/packages/rootfs_lucid64/lib:/var/vcap/packages/dea_next/lib

$ mkdir
mkdir: missing operand
Try 'mkdir --help' for more information.

Why are we setting the LD_LIBRARY_PATH?

	for package_bin_dir in $(ls -d /var/vcap/packages/*/{,s}bin)
	do
	export PATH=${package_bin_dir}:$PATH
	done

cloudfoundry-community / bosh-gen Goto Github PK

bosh-gen's People

Contributors

Stargazers

Watchers

Forkers

bosh-gen's Issues

Recommend Projects

Recommend Topics

Recommend Org