These will be there in the comments so if someone changes the default, they'll still be able to see the default without consulting the docs.

There are no logs. file:consult/ on the advanced.config works fine. Not sure what's up.

I tried starting Riak using both riak.conf and an advanced.config. My advanced.config was missing the trailing '.', and this was the error I got:

11:58:45.818 [info] Application lager started on node nonode@nohost
11:58:45.818 [info] Checking /Users/reid/Documents/repos/basho-versioned/riak-2.0/rel/riak/bin/../etc/app.config exists... false
11:58:45.818 [info] Checking /Users/reid/Documents/repos/basho-versioned/riak-2.0/rel/riak/bin/../etc/vm.args exists... false
11:58:45.818 [info] No app.config or vm.args detected in /Users/reid/Documents/repos/basho-versioned/riak-2.0/rel/riak/bin/../etc, activating cuttlefish
11:58:46.030 [info] Adding Defaults
11:58:46.034 [info] Applying Datatypes
11:58:46.051 [info] Validation
11:58:46.054 [info] Applied 1:1 Mappings
11:58:46.057 [info] Applied Translations
11:58:46.058 [info] /Users/reid/Documents/repos/basho-versioned/riak-2.0/rel/riak/bin/../etc/advanced.config detected, overlaying proplists
Error generating config with cuttlefish, there should be logs

It would be nice if cuttlefish could tell me the parse error was in the advanced.config.

This is a biggie. Well, it's big with regard to importance. I'm not sure how difficult it will be. Let's outline how this will need to work.

1. cuttlefish as a riak dependency
2. riak (shell script) needs to start the 🐟, the 🐟's api needs to be fed conf and schema files from the script
3. generated app.config is put somewhere
4. shell script goes on with it's life, using generated.app.config.YYYYMMDD_HHMMSS whatev.filename.conf

I found it to be a commone sep to do something like this:

{mapping, "data_dir", "riak_core.platform_data_dir",
 [{default, "{{platform_data_dir}}"},
  {datatype, string}]}.

{translation,
 "fifo_db.db_path",
 fun(Conf) ->
         cuttlefish_util:conf_get_value("data_dir", Conf)
 end
}.

{translation,
 "leveldb.data_root",
 fun(Conf) ->
         cuttlefish_util:conf_get_value("data_dir", Conf)
 end
}.

{translation,
 "hanoidb.data_root",
 fun(Conf) ->
         cuttlefish_util:conf_get_value("data_dir", Conf)
 end
}.

{translation,
 "bitcask.data_root",
 fun(Conf) ->
         cuttlefish_util:conf_get_value("data_dir", Conf)
 end
}.

aka hide the detailed setting under a simplified name for a system. So what would be great is if there would be something like {alias, "hanoidb.data_root"} which would then act as a 'default' for hanoidb.data_root (as example).

So this would simplify the above code to:

{mapping, "data_dir", "riak_core.platform_data_dir",
 [{default, "{{platform_data_dir}}"},
  {datatype, string},
  {alias, "fifo_db.db_path"},
  {alias, "leveldb.data_root"},
  {alias, "hanoidb.data_root"},
  {alias, "bitcask.data_root"}]}.

which is kind of sweet ;)

Beyond datatype validation.

e.g. ring_size needs to be a power of 2 and greater than one.

I've got something for this in appconf, but will need rework for dat 🐟

it'd be nice if either file existed that it wouldn't be generated each startup. as it stands it just looks like app.config is treated that way.

We won't be able to complete #2 without at least inventing this syntax; however, this ticket also exists for the implementation of the interpreter of that syntax.

Right now, we're leaving it up to the translation fun to tokenize the string and extract the username, but it shouldn't have to. We should provide helpers for this, that's were this issue comes in.

Need to make it configurable. Right now it's always riak.conf

https://github.com/basho/cuttlefish/blob/develop/src/cuttlefish_rebar_plugin.erl#L58

maybe size isn't the right name, but it should be gb, mb, kb, etc...

We need to be able to deprecate settings so they can be improved over time. This goes beyond just failing on settings that have no mapping in that we would like to include information about how to migrate settings from one version to another.

Currently, cuttlefish prints a warning about unknown configuration variables but continues on using the default.

If I were trying to set anti_entropy = passive and accidentally typed anti_entrophy, riak would start with the default of active. You could check the logs and see the warning, but if you just run riak start, there's no information on the console.

app.config would have worked the same way, so it's not crazy behavior, but cuttlefish is better than that.

Running pre1 I noticed this in the log output.

-config /Users/jmeredith/basho/work/riak-security/rel/riak/bin/../etc/generated/app.2013.09.18.17.58.25.config -args_file /Users/jmeredith/basho/work/riak-security/rel/riak/bin/../etc/generated/vm.2013.09.18.17.58.25.args

the etc directory isn't guaranteed to be writable by the node. Generated tmp files need to be written somewhere the application has permissions according to os-specific packaging rules.

https://github.com/basho/cuttlefish/blob/develop/src/cuttlefish_schema.erl#L125

It would be nice if it were possible to have multiple backends with different priorities.

Initial thought:
a backand would be just some kind of module that returns a list of {key, value} tuples

The use of that would be that some state could be stored in the application, perhaps shared over nodes in a distributed system and the 'file' config would be the inital state/backup, but 'in app configuraiton' could overwrite the files, also it would easiely allow to have multiple files that are 'merged'

Here's a plan for how to handle upgrayedds.

If there's an app.config, just use it.

If there's a riak.conf and no app.config, generate an app.config and use it.

If there's bolth a riak.conf and an advanced.config, generate from riak.conf, then overlay advanced options.

app.config handles people who haven't upgraded to dat 🐟 yet.

advanced.config handles people who need to turn extra deep riak knobs.

The default riak.conf file that would ship with riak should be generated from the schema

So I'm trying to follow the instructions on https://github.com/basho/cuttlefish/wiki/Cuttlefish-for-Erlang-Developers#wiki-lists-and-proplists-and-names-oh-my, the only problem is that they don't work.

I'm using cuttlefish at version 0.1.0, and I have also tried using the head of the master branch and I get the same results from both.

so this:

%% HTTP Listeners
%% @doc listener.http.<name> is an IP address and TCP port that the Riak
%% HTTP interface will bind.
{mapping, "listener.http.$name", "riak_api.http", [
  {default, {"127.0.0.1", 8098}},
  {datatype, ip},
  {include_default, "internal"}
]}.

{translation,
 "riak_api.http",
  fun(Conf) ->
      HTTP = cuttlefish_variable:filter_by_prefix("listener.http", Conf),
      [ IP || {_, IP} <- HTTP]
  end
}.

and this:

listener.http.internal = 127.0.0.1:8098
listener.http.external = 10.0.0.1:80

generate this:

{riak_api,[{http,ok}]},

which is not quite what I was hoping that the translation and mapping would do.

Is this a feature that has been removed? or just implemented differently?

Promised @gburd I'd get erlang's +SP in to erlang_vm.schema in the next two weeks.

file and/or directory datatype

Now that lager is started up by the cuttlefish_escript. We'll have to be better about what gets logged. Pretty much everthing is at 'info' currently.

We should move everything to debug. Have one 'info' level message to say "This is enabled" which doesn't output if it's using old style config..

Then we should use "warnings" for any config red flags and "errors" for anything that won't generate a config file.

conf_parse.erl:211: The created fun has no local return
conf_parse.erl:212: Fun application will fail since P :: none() is not a function of arity 2
conf_parse.erl:283: Guard test is_list(S::<<_:8,_:_*8>>) can never succeed
conf_parse.erl:315: The created fun has no local return
cuttlefish_conf.erl:62: Function generate_file/2 has no local return
cuttlefish_conf.erl:65: The call file:open(Filename::any(),'write') breaks the contract (File,Modes) -> {'ok',IoDevice} | {'error',Reason} when is_subtype(File,Filename | iodata()), is_subtype(Filename,name_all()), is_subtype(Modes,[mode() | 'ram']), is_subtype(IoDevice,io_device()), is_subtype(Reason,posix() | 'badarg' | 'system_limit')
cuttlefish_generator.erl:48: The call cuttlefish_validator:func(V::'false' | tuple()) does not have an opaque term of type cuttlefish_validator:validator() as 1st argument
cuttlefish_generator.erl:137: Function set_value/3 has no local return
cuttlefish_generator.erl:138: The call cuttlefish_util:replace_proplist_value(atom(),NewValue::any(),Acc::any()) breaks the contract (string(),any(),[{string(),any()}]) -> [{string(),any()}]
cuttlefish_generator.erl:236: The pattern [{_, Var} | _] can never match the type [string()]
cuttlefish_generator.erl:278: The pattern 'undefined' can never match the type string()
cuttlefish_mapping.erl:94: Invalid type specification for function cuttlefish_mapping:variable/1. The success typing is (cuttlefish_mapping:mapping()) -> 'undefined' | [string()]
cuttlefish_mapping.erl:121: Invalid type specification for function cuttlefish_mapping:validators/2. The success typing is (cuttlefish_mapping:mapping(),_) -> ['false' | tuple()]
cuttlefish_schema.erl:66: The variable _ can never match since previous clauses completely covered the type {'error',[{_,_}]} | {[cuttlefish_translation:translation()],[cuttlefish_mapping:mapping()],[cuttlefish_validator:validator()]}
cuttlefish_schema.erl:125: The attempt to match a term of type cuttlefish_mapping:mapping() | cuttlefish_translation:translation() against the pattern {'error', Desc} breaks the opaqueness of the term
cuttlefish_schema.erl:175: The pattern {'validator', Return} can never match the type {'mapping',tuple()}
Unknown functions:
  getopt:parse/2
  getopt:usage/2
  lager:do_log/9
  lager:md/0
  lager_config:get/2
  lager_msg:new/4
  lager_util:config_to_mask/1
  lager_util:is_loggable/3
  lager_util:level_to_num/1
  rebar_rel_utils:get_target_dir/2
  rebar_rel_utils:load_config/2
 done in 0m2.46s
done (warnings were emitted)

Based on 08915a7

in my tests a 01- file overwrites a 00- file and a 99- one overwrites a 01- one.

https://github.com/basho/cuttlefish/blob/develop/src/cuttlefish_generator.erl#L299-301

Actually, it doesn't log too much, because this message is commented out. I'd like it to not be commented, but only for the initial call.

Also, if you are using multibackend, you'll see the cuttlefish phase messages once for the schema, and then again for every backend you used. I don't want logging stopped entirely in that case, but those messages should be stopped. Probably just introduce an arity 3 version for "RecursiveLogs | boolean()"

Cuttlefish does a great job of explaining errors when they happen while generating app.config and vm.args in memory. The problem here, is that after that phase, if it can't actually write out those files to disk it give the ol' cryptic "failed to generate config with cuttlefish". It could be worse, that message used to end with "there should be logs"

This is yet another attempt to resolve #79, assuming I finally understand the issue. I appreciate the solution presented in #80, but I'm going to suggest that maybe it doesn't go far enough.

The original idea for translations worked like this: {translation, Mapping, Fun}
Mapping: Was the KVCesque location in the app.config that this thing belonged
Fun: The return of this fun() was put in that mapping location.

Then, it was up to the schema writer to write a Fun that returned the value for that location. You could write a fun with case statements and _ -> clauses to return the default, but maybe that's not enough.

Here's the collection of changes I'm considering:

Translation Defaults

Add a proplist to the translation in schema with the ability to add a default value
e.g.

{translation,
 "riak_api.http",
 fun(Conf) ->
        HTTP = cuttlefish_util:filter_by_variable_starts_with("listener.http", Conf),
        [ IP || {_, IP} <- HTTP]
 end,
 [
   {default, []}
 ]
}.

Cuttlefish Schema Writer's API

One thing I've been considering for a while is a schema writer's API. Take the function referenced above: cuttlefish_util:filter_by_variable_starts_with/2. There should be one module with these types of functions in them cfish for brevity on the schema side. The cfish versions of these functions would throw errors on things like notfound or whatever.

This would also make an easy "go to" schema writer's documentation reference.

Better Error Handling at Translation Runtime

Right now we try translations and if they error, return undefined. Instead we could return the specified default. We could also do this based on the error, so maybe return the default on notfound which is desirable behavior. But if there's a different kind of error, we should abort the cuttlefish startup and make the user aware of the error.

For example, if you're trying to take the ceiling of a string, I'm not really satisfied that returning the default and starting up is the right thing to do.

Stricter Translation -spec

@Licenser dropped the {ok, Value} stuff on us in PR #80 but, I don't think it's enough. We can move to the {ok, Value} semantic, but I'd like to add some {error, Message} to that party. This way in the "ceiling of a string" example, we could include try... catch clauses in translations, and then turn the output into something human readable.

How do we log this? Right now it's to stderr, which is grrrrrrrrreat! but what happens when it's not a user starting it?

console.log? but we don't know where is console.log? or do we? ominous tones

Right now riak.conf is a POC, and has one case of each of three levels of complexity for mapping a Riak app.config. We need a version, in which everything tunable by default in riak is tunable by riak.conf

Basically it is possible to have a mapping not take effect when they are not present, right now there isn't a way to have this behaviour for translations.

A simple way to implement this would to change the return value of translation functions to {ok, Value} or undefined in which case undefined would mean the translation has no effect.

basho/riak#413 (comment)

Investigate why zdbbl breaks when datatype is bytesize. reported by @jrwest

2> Conf = [{["a", "b", "c"], 1},{["a", "b", "d"], 2},{["a", "c", "c"], 3}].
[{["a","b","c"],1},{["a","b","d"],2},{["a","c","c"],3}]
3> cuttlefish_variable:fuzzy_matches("a.$name.c", Conf).
[]
4> cuttlefish_variable:fuzzy_matches(["a", "$name", "c"], Conf).
[{"$name","c"},{"$name","b"}]
5> cuttlefish_variable:fuzzy_matches(["a", "$name"], Conf).     
[]
6> cuttlefish_variable:fuzzy_matches(["a", "$name", "d"], Conf).
[{"$name","b"}]

a) we should see "a.$name.c" and tokenize it if it's a string
b) we should allow a search for "a.$name" to find everything that could start with "a.$name"

@gburd let me know what you think is "sensible"

cuttlefish_util:filter_by_variable_starts_with("listener.http", Conf) returns the correct results

cuttlefish_util:filter_by_variable_starts_with("listener.http.", Conf) returns [].

fsm_limit has two valid value types:
an integer, or the atom undefined

undefined has the special meaning of disabling the sidejob subsystem entirely, which we now can't do.

OS: FreeBSD 9.1, Riak 2.0.0pre2

[vagrant@FREEBSD-91 ~]$ sudo cat /var/log/riak/erlang.log.1

=====
===== LOGGING STARTED Wed Sep 25 15:42:01 UTC 2013
=====
15:42:02.294 [info] Application lager started on node nonode@nohost
15:42:02.294 [info] No app.config detected in /usr/local/etc/riak, activating cuttlefish
15:42:02.428 [info] Adding Defaults
15:42:02.435 [info] Applying Datatypes
15:42:02.435 [info] Validation
15:42:02.437 [info] Applied 1:1 Mappings
15:42:02.439 [info] Applied Translations
Error reading config file: no such file or directory
Error reading  -config /usr/local/etc/riak/generated/app.2013.09.25.15.42.02.config -args_file /usr/local/etc/riak/generated/vm.2013.09.25.15.42.02.args
[vagrant@FREEBSD-91 ~]$ ls /usr/local/etc/riak/generated
ls: /usr/local/etc/riak/generated: No such file or directory

There are several cases where durations need to be expressed in magnitudes other than milliseconds. There already exists a duration_secs type but it would be much simpler (requiring fewer translations in the schema) if the type were parameterized by the desired magnitude, e.g.

{datatype, {duration, minutes}}
% or
{datatype, {duration, seconds}}

Hi, I don't know, where is better to address this issue, on riak or on cuttlefish.

I have used riak, branch develop, after compiling a release with make rel.

If I add the configuration to the riak like this:

storage_backend = multi

multi_backend.bitcask_mult.storage_backend = bitcask
multi_backend.leveldb_mult.storage_backend = leveldb

Then I get the after compiling riak with make rel this error:

# bin/riak console
Error generating config with cuttlefish

Without any information whats going wrong.

The stacktrace for the problem is:

error:badarg
[{erlang,list_to_atom,[-1],[]},
 {cuttlefish_datatypes,from_string,2,
                       [{file,"src/cuttlefish_datatypes.erl"},{line,134}]},
 {cuttlefish_generator,transform_supported_type,4,
                       [{file,"src/cuttlefish_generator.erl"},{line,412}]},
 {cuttlefish_generator,transform_extended_type,4,
                       [{file,"src/cuttlefish_generator.erl"},{line,431}]},
 {cuttlefish_generator,'-transform_datatypes/2-fun-3-',3,
                       [{file,"src/cuttlefish_generator.erl"},{line,370}]},
 {lists,foldl,3,[{file,"lists.erl"},{line,1248}]},
 {cuttlefish_generator,map_transform_datatypes,2,
                       [{file,"src/cuttlefish_generator.erl"},{line,59}]},
 {cuttlefish_escript,engage_cuttlefish,1,
                     [{file,"src/cuttlefish_escript.erl"},{line,301}]}]

So this is a bit related to #67, when having some variable shadowing it should be possible in what way it does so. I can see two scenarios:

totally replace the variable

When I define my.setting before (with a higher priority) and use translate or alias to riak_core.something then the riak_core.something from riak_core.schema will never show in the config and even if added be simply ignore (or a warning printed that it's a unknown setting)

defaulting/simplification

Example take the code in #67 for example, there the data_dir in there would be a default for bitcask.data_dir etc. So it would be neat if bitcask.data_dir, would not be present in the generated .config, but if added explictly would overwrite the 'default' provided by data_dir

Add durations as a datatype

reported by @seancribbs

TODO: https://github.com/basho/cuttlefish/blob/develop/src/cuttlefish_generator.erl#L200

I believe there is an edge case, in which weird things will happen if you have a schema define something like "listener.http.$name" and then somebody chooses to add a line

listener.http.$name = 127.0.0.1:8098"

to their .conf file.

The easiest way to fix this, is probably just to have neotoma not allow dollar signs on the left side of the equals when parsing conf files.

After I had a conversation with @jonmeredith, I agree we should re-think the placement of any generated content in the /etc/ directory. In some environments /etc/ is mounted read-only once a known-good config is made. The generated content for cuttlefish should instead go into the platform_data_dir on each platform so we know there is always the ability to write to that data.

/cc @Vagabond @joedevivo @lukebakken @jonmeredith

It would be sweet if it were possilbe to have something like an {include, module} (and/or {include, "top.namespace", module} ) statement that then would pull config definition out of a predefined function (much like the behaviour_info works)

That way libraries could include their config parameters and the autor of a application using (i.e. bitcask, elveldb, lager) would not have to reinvent the wheel (or copy and paste form another app) but could just say {include, lager}

and the lager.erl would export cuttlefish_info(mappings) -> {...} and cuttlefish_info(translations) -> {...}

if your mapping is hey.$name, and your $name is "steve.vinsoki" this thing won't work.

What's worse, if you define key.$name.attribute and you want name to equal "steve.attribute" it'll match on the wrong thing.

dat delimiter.

It would be sweet to have a atom datatype, currently only enums seem to be able to be atoms.

It's easy enough to achiev with a transformation but there really is no reason not to have the type :)

We should validate datatypes on the read of any .conf file

It should have verbose output as to why it didn't work (if it didn't, obvs)

There are many flags for the Erlang emulator that we have not included in erlang_vm.schema.

Setting storage_backend to "eleveldb" instead of "leveldb" is an error, but it's silently ignored. This was seen using cuttlefish commit 09a1aa93, but might also be an issue with previous versions as well.