ENH: Create ceph-formula (potential inspiration) about ceph-formula HOT 24 CLOSED

Good collection of links. Thanks. It might be worthwhile reaching out to the two authors above to see if they're interested in combining efforts here. Otherwise perhaps they'd be interested in having someone else do that for them. Neither project has a license so that would also need to be clarified.

from ceph-formula.

@steveoliver @cholcombe973 do you have any suggestions for developing an Apache 2.0 licensed ceph-formula?

from ceph-formula.

I thought that ceph open sourced calamari that would do cluster turn ups with salt. Turns out it's just for monitoring and management. I think it shouldn't be too hard to get a basic salt formula going for this.

I think we can mostly follow the chef cookbooks. https://github.com/ceph/ceph-cookbook/tree/master/recipes
It should translate

from ceph-formula.

You know now that I look at this again I remember what I got stuck on last time. It's the databags that are missing in salt that made it pretty hard to do this. I think a better approach might be to write a python script and have salt call that script if needed to setup the node. The other option is to write a pure python state file which would give you more power.

from ceph-formula.

Thanks!

• http://docs.saltstack.com/en/latest/topics/development/conventions/formulas.html#abstracting-platform-specific-data (map.jinja)
• https://github.com/saltstack/salt/tree/develop/salt/pillar (pepa)
• https://github.com/saltstack-formulas/salt-formula/blob/master/pillar.example
• https://github.com/saltstack-formulas/postgres-formula/blob/master/pillar.example

I think a better approach might be to write a python script and have salt call that script if needed to setup the node.

• http://docs.saltstack.com/en/latest/ref/modules/ (/srv/salt/_modules -> salt.modules.ceph)

The other option is to write a pure python state file which would give you more power.

• http://docs.saltstack.com/en/latest/ref/renderers/all/salt.renderers.py.html
• http://docs.saltstack.com/en/latest/ref/renderers/all/salt.renderers.pyobjects.html
• http://docs.saltstack.com/en/latest/ref/renderers/all/salt.renderers.pydsl.html
• http://docs.saltstack.com/en/latest/ref/renderers/all/salt.renderers.yaml.html (default)

from ceph-formula.

I gave this some more thought. Salt doesn't have a mechanism to pick a master to execute commands in. I think what salt could do is do the initial setup like getting ceph.conf in the right place, making directories, installing ceph packages. After that it could call a python/java/c++/go/whatever program that forms a quorum on the cluster and then issues the commands to build the cluster.

from ceph-formula.

• http://ceph.com/docs/master/install/manual-deployment/
• http://ceph.com/docs/master/rados/operations/add-or-rm-mons/

Salt doesn't have a mechanism to pick a master to execute commands in.

saltstack/salt#14074

I'm fairly new to ceph. Is there a reason that -- for smaller setups -- this could not be specified in a (hierachical) pillar?

After that it could call a python/java/c++/go/whatever program that forms a quorum on the cluster and then issues the commands to build the cluster.

• http://docs.saltstack.com/en/latest/glossary.html
• http://docs.saltstack.com/en/latest/topics/targeting/grains.html
• http://docs.saltstack.com/en/latest/ref/runners/#examples
• http://docs.saltstack.com/en/latest/topics/mine/index.html#salt-mine
• http://docs.saltstack.com/en/latest/topics/reactor/index.html

from ceph-formula.

What I mean is that pillar and salt install everything on all the ceph
servers at once. From what I know, you can't assemble a ceph cluster like
that. There has to be one node that's a master that initiates the cluster
creation. Maybe someone knows of a better way to do this?

On Wed, Aug 27, 2014 at 2:18 AM, Wes Turner [email protected]
wrote:

• http://ceph.com/docs/master/install/manual-deployment/
• http://ceph.com/docs/master/rados/operations/add-or-rm-mons/

Salt doesn't have a mechanism to pick a master to execute commands in.

saltstack/salt#14074 saltstack/salt#14074

I'm fairly new to ceph. Is there a reason that -- for smaller setups --
this could not be specified in a (hierachical) pillar?

After that it could call a python/java/c++/go/whatever program that forms
a quorum on the cluster and then issues the commands to build the cluster.

• http://docs.saltstack.com/en/latest/glossary.html
• http://docs.saltstack.com/en/latest/topics/targeting/grains.html
• http://docs.saltstack.com/en/latest/ref/runners/#examples
• http://docs.saltstack.com/en/latest/topics/mine/index.html#salt-mine
• http://docs.saltstack.com/en/latest/topics/reactor/index.html

—
Reply to this email directly or view it on GitHub
#1 (comment)
.

from ceph-formula.

I haven't gone through the process of setting up Ceph but Salt can handle setting software up on multiple nodes in parallel or in order.

Use custom events and the Reactor to pass messages between the Master or on multiple Minions, use Salt Mine for sharing information between nodes, use the Orchestrate runner to oversee timing-dependent operations, and use a custom Runner to do a one-time selection of a single node and set a custom grain there.

The points in that issue about automating the selection and switch for a "leader" node are valid. But for a one-time setup it's probably overkill. (The first iteration of this formula doesn't need to address switching the "leader" node to another.) A custom grain could be set on whichever server is selected. E.g.: roles: [ceph, ceph_master]. Perhaps something like this:

1. Manually tag all nodes that are intended for the Ceph cluster with the ceph role.
2. Kick off this Ceph formula by either automatically or manually selecting which server is to be the Ceph Master. Use a custom runner for the former or just state.sls for the latter.
3. That Ceph Master fires a custom event once it is configured and ready.
4. The Salt Master calls an Orchestrate script that starts the other Ceph machines to install and configure themselves as a cluster and then waits for them all to finish.
5. The Salt Master then notifies the Ceph Master to create the quorum.

Did I get any of that right? :)

If someone can volunteer for the Ceph leg-work I can help out with the Salt stuff.

from ceph-formula.

Yeah I think you nailed it. I'd be willing to do the ceph work. What would
you need?
On Aug 27, 2014 5:26 PM, "Seth House" [email protected] wrote:

I haven't gone through the process of setting up Ceph but Salt can handle
setting software up on multiple nodes in parallel or in order.

Use custom events and the Reactor to pass messages between the Master or
on multiple Minions, use Salt Mine for sharing information between nodes,
use the Orchestrate runner to oversee timing-dependent operations, and use
a custom Runner to do a one-time selection of a single node and set a
custom grain there.

The points in that issue about automating the selection and switch for a
"leader" node are valid. But for a one-time setup it's probably overkill.
(The first iteration of this formula doesn't need to address switching the
"leader" node to another.) A custom grain could be set on whichever server
is selected. E.g.: roles: [ceph, ceph_master]. Perhaps something like
this:

1. Manually tag all nodes that are intended for the Ceph cluster with
   the ceph role.
2. Kick off this Ceph formula by either automatically or manually
   selecting which server is to be the Ceph Master. Use a custom runner for
   the former or just state.sls for the latter.
3. That Ceph Master fires a custom event once it is configured and
   ready.
4. The Salt Master calls an Orchestrate script that starts the other
   Ceph machines to install and configure themselves as a cluster and then
   waits for them all to finish.
5. The Salt Master then notifies the Ceph Master to create the quorum.

Did I get any of that right? :)

If someone can volunteer for the Ceph leg-work I can help out with the
Salt stuff.

—
Reply to this email directly or view it on GitHub
#1 (comment)
.

from ceph-formula.

Sorry for the slow reply, I'm a tad swamped so far this week.

I pushed a few placeholder files into the repo just now. Let's start with states to set up the Ceph Master.

First thing is to fill out the lookup.jinja file with platform-specific details. We can start with a single platform to make things easy.

• What are the minimum packages to install the Ceph Master?
• What customizations are required after a default package install?
• Do we need to start thinking about configuration that will be required for the quorum yet?

from ceph-formula.

@whiteinge here's my current ceph state file. This should provide the packages for installing ceph: https://gist.github.com/cholcombe973/994d607858e91724824b
Unfortunately this is a bit old. Emperor is now out of date. Firefly is the newest stable release. I'm pretty sure we can just change the version numbers and it should work

from ceph-formula.

@whiteinge Is lookup.jinja the same as map.jinja? (docs: http://docs.saltstack.com/en/latest/topics/development/conventions/formulas.html#abstracting-platform-specific-data)

from ceph-formula.

Apologies fire the slow reply. I'm on an engagement this week. Yes,
lookup.jinja was meant as the same as the map.jinja. Sorry for the
confusion.

from ceph-formula.

@whiteinge No worries!

Thank you both!

from ceph-formula.

Are we stalled out?

On Thu, Sep 4, 2014 at 7:20 AM, Wes Turner [email protected] wrote:

@whiteinge https://github.com/whiteinge No worries!

Thank you both!

—
Reply to this email directly or view it on GitHub
#1 (comment)
.

from ceph-formula.

@cholcombe973 let's start by getting your existing install states in here and updated for the current version. Mind sending a pull request? After that we can add the custom event. Is that install state the same for both the Ceph master and the other Ceph machines? If so, how do we designate the Ceph master? If not, how are the other Ceph machines installed and configured?

Once those pieces are in place we can add the custom event and write the orchestrate file, then I think we're done minus the documentation.

from ceph-formula.

Sure I'll try to get that done this weekend.
On Sep 12, 2014 5:04 PM, "Seth House" [email protected] wrote:

@cholcombe973 https://github.com/cholcombe973 let's start by getting
your existing install states in here and updated for the current version.
Mind sending a pull request? After that we can add the custom event. Is
that install state the same for both the Ceph master and the other Ceph
machines? If so, how do we designate the Ceph master? If not, how are the
other Ceph machines installed and configured?

Once those pieces are in place we can add the custom event and write the
orchestrate file, then I think we're done minus the documentation.

—
Reply to this email directly or view it on GitHub
#1 (comment)
.

from ceph-formula.

I should be more clear. The ceph master so to speak is only needed to
startup the cluster. Once the cluster is going there is no master
anymore. Ceph has a pretty good walkthrough of how to manually setup a
cluster. http://ceph.com/docs/master/install/manual-deployment/ I'm going
to include some of these steps in my sls file so that it's more legit. My
initial version is pretty crappy

On Sat, Sep 13, 2014 at 8:45 AM, Chris Holcombe [email protected]
wrote:

Sure I'll try to get that done this weekend.
On Sep 12, 2014 5:04 PM, "Seth House" [email protected] wrote:

@cholcombe973 https://github.com/cholcombe973 let's start by getting
your existing install states in here and updated for the current version.
Mind sending a pull request? After that we can add the custom event. Is
that install state the same for both the Ceph master and the other Ceph
machines? If so, how do we designate the Ceph master? If not, how are the
other Ceph machines installed and configured?

Once those pieces are in place we can add the custom event and write the
orchestrate file, then I think we're done minus the documentation.

—
Reply to this email directly or view it on GitHub
#1 (comment)
.

from ceph-formula.

Checkout what I found:
https://github.com/komljen/ceph-salt/tree/master/salt/ceph
If we just boil this down I think it's pretty close to what we need

On Sat, Sep 13, 2014 at 6:30 PM, Chris Holcombe [email protected]
wrote:

I should be more clear. The ceph master so to speak is only needed to
startup the cluster. Once the cluster is going there is no master
anymore. Ceph has a pretty good walkthrough of how to manually setup a
cluster. http://ceph.com/docs/master/install/manual-deployment/ I'm
going to include some of these steps in my sls file so that it's more
legit. My initial version is pretty crappy

On Sat, Sep 13, 2014 at 8:45 AM, Chris Holcombe [email protected]
wrote:

Sure I'll try to get that done this weekend.
On Sep 12, 2014 5:04 PM, "Seth House" [email protected] wrote:

@cholcombe973 https://github.com/cholcombe973 let's start by getting
your existing install states in here and updated for the current version.
Mind sending a pull request? After that we can add the custom event. Is
that install state the same for both the Ceph master and the other Ceph
machines? If so, how do we designate the Ceph master? If not, how are the
other Ceph machines installed and configured?

Once those pieces are in place we can add the custom event and write the
orchestrate file, then I think we're done minus the documentation.

—
Reply to this email directly or view it on GitHub
#1 (comment)
.

from ceph-formula.

@cholcombe973

https://github.com/komljen/ceph-salt/tree/master/salt/ceph

komljen/ceph-salt#2 mentions the https://github.com/ceph/calamari salt states.

This shows how to add roles to nodes: https://github.com/komljen/ceph-salt/blob/master/pillar/environment.sls

https://github.com/ceph/ceph-ansible

• https://github.com/ceph/ceph-ansible#what-does-it-do

from ceph-formula.

Any progress on this? It looks as if ansible is miles ahead unfortunately...

from ceph-formula.

Yeah I haven't had any time. Sorry :-/

On Mon, Aug 10, 2015 at 3:07 AM, Wolodja Wentland [email protected]
wrote:

Any progress on this? It looks as if ansible is miles ahead
unfortunately...

—
Reply to this email directly or view it on GitHub
#1 (comment)
.

from ceph-formula.

@cholcombe973 No worries, I was just interested and stumbled over the ceph-formula and this rather long discussion so I though I'd inquire about the current status. Would be wonderful to have first class ceph support in salt, but we aren't there yet apparently.

from ceph-formula.