basho / cuttlefish Goto Github PK
View Code? Open in Web Editor NEWnever lose your childlike sense of wonder baby cuttlefish, promise me?
License: Apache License 2.0
never lose your childlike sense of wonder baby cuttlefish, promise me?
License: Apache License 2.0
The default riak.conf file that would ship with riak should be generated from the schema
Promised @gburd I'd get erlang's +SP in to erlang_vm.schema in the next two weeks.
Here's a plan for how to handle upgrayedds.
If there's an app.config, just use it.
If there's a riak.conf and no app.config, generate an app.config and use it.
If there's bolth a riak.conf and an advanced.config, generate from riak.conf, then overlay advanced options.
app.config handles people who haven't upgraded to dat ๐ yet.
advanced.config handles people who need to turn extra deep riak knobs.
conf_parse.erl:211: The created fun has no local return
conf_parse.erl:212: Fun application will fail since P :: none() is not a function of arity 2
conf_parse.erl:283: Guard test is_list(S::<<_:8,_:_*8>>) can never succeed
conf_parse.erl:315: The created fun has no local return
cuttlefish_conf.erl:62: Function generate_file/2 has no local return
cuttlefish_conf.erl:65: The call file:open(Filename::any(),'write') breaks the contract (File,Modes) -> {'ok',IoDevice} | {'error',Reason} when is_subtype(File,Filename | iodata()), is_subtype(Filename,name_all()), is_subtype(Modes,[mode() | 'ram']), is_subtype(IoDevice,io_device()), is_subtype(Reason,posix() | 'badarg' | 'system_limit')
cuttlefish_generator.erl:48: The call cuttlefish_validator:func(V::'false' | tuple()) does not have an opaque term of type cuttlefish_validator:validator() as 1st argument
cuttlefish_generator.erl:137: Function set_value/3 has no local return
cuttlefish_generator.erl:138: The call cuttlefish_util:replace_proplist_value(atom(),NewValue::any(),Acc::any()) breaks the contract (string(),any(),[{string(),any()}]) -> [{string(),any()}]
cuttlefish_generator.erl:236: The pattern [{_, Var} | _] can never match the type [string()]
cuttlefish_generator.erl:278: The pattern 'undefined' can never match the type string()
cuttlefish_mapping.erl:94: Invalid type specification for function cuttlefish_mapping:variable/1. The success typing is (cuttlefish_mapping:mapping()) -> 'undefined' | [string()]
cuttlefish_mapping.erl:121: Invalid type specification for function cuttlefish_mapping:validators/2. The success typing is (cuttlefish_mapping:mapping(),_) -> ['false' | tuple()]
cuttlefish_schema.erl:66: The variable _ can never match since previous clauses completely covered the type {'error',[{_,_}]} | {[cuttlefish_translation:translation()],[cuttlefish_mapping:mapping()],[cuttlefish_validator:validator()]}
cuttlefish_schema.erl:125: The attempt to match a term of type cuttlefish_mapping:mapping() | cuttlefish_translation:translation() against the pattern {'error', Desc} breaks the opaqueness of the term
cuttlefish_schema.erl:175: The pattern {'validator', Return} can never match the type {'mapping',tuple()}
Unknown functions:
getopt:parse/2
getopt:usage/2
lager:do_log/9
lager:md/0
lager_config:get/2
lager_msg:new/4
lager_util:config_to_mask/1
lager_util:is_loggable/3
lager_util:level_to_num/1
rebar_rel_utils:get_target_dir/2
rebar_rel_utils:load_config/2
done in 0m2.46s
done (warnings were emitted)
Based on 08915a7
maybe size isn't the right name, but it should be gb, mb, kb, etc...
It would be nice if it were possible to have multiple backends with different priorities.
Initial thought:
a backand would be just some kind of module that returns a list of {key, value} tuples
The use of that would be that some state could be stored in the application, perhaps shared over nodes in a distributed system and the 'file' config would be the inital state/backup, but 'in app configuraiton' could overwrite the files, also it would easiely allow to have multiple files that are 'merged'
Currently, cuttlefish prints a warning about unknown configuration variables but continues on using the default.
If I were trying to set anti_entropy = passive
and accidentally typed anti_entrophy
, riak would start with the default of active
. You could check the logs and see the warning, but if you just run riak start
, there's no information on the console.
app.config
would have worked the same way, so it's not crazy behavior, but cuttlefish is better than that.
These will be there in the comments so if someone changes the default, they'll still be able to see the default without consulting the docs.
There are several cases where durations need to be expressed in magnitudes other than milliseconds. There already exists a duration_secs
type but it would be much simpler (requiring fewer translations in the schema) if the type were parameterized by the desired magnitude, e.g.
{datatype, {duration, minutes}}
% or
{datatype, {duration, seconds}}
if your mapping is hey.$name, and your $name is "steve.vinsoki" this thing won't work.
What's worse, if you define key.$name.attribute and you want name to equal "steve.attribute" it'll match on the wrong thing.
dat delimiter.
We need to be able to deprecate settings so they can be improved over time. This goes beyond just failing on settings that have no mapping in that we would like to include information about how to migrate settings from one version to another.
Hi, I don't know, where is better to address this issue, on riak or on cuttlefish.
I have used riak, branch develop, after compiling a release with make rel.
If I add the configuration to the riak like this:
storage_backend = multi
multi_backend.bitcask_mult.storage_backend = bitcask
multi_backend.leveldb_mult.storage_backend = leveldb
Then I get the after compiling riak with make rel this error:
# bin/riak console
Error generating config with cuttlefish
Without any information whats going wrong.
The stacktrace for the problem is:
error:badarg
[{erlang,list_to_atom,[-1],[]},
{cuttlefish_datatypes,from_string,2,
[{file,"src/cuttlefish_datatypes.erl"},{line,134}]},
{cuttlefish_generator,transform_supported_type,4,
[{file,"src/cuttlefish_generator.erl"},{line,412}]},
{cuttlefish_generator,transform_extended_type,4,
[{file,"src/cuttlefish_generator.erl"},{line,431}]},
{cuttlefish_generator,'-transform_datatypes/2-fun-3-',3,
[{file,"src/cuttlefish_generator.erl"},{line,370}]},
{lists,foldl,3,[{file,"lists.erl"},{line,1248}]},
{cuttlefish_generator,map_transform_datatypes,2,
[{file,"src/cuttlefish_generator.erl"},{line,59}]},
{cuttlefish_escript,engage_cuttlefish,1,
[{file,"src/cuttlefish_escript.erl"},{line,301}]}]
We should validate datatypes on the read of any .conf
file
It should have verbose output as to why it didn't work (if it didn't, obvs)
Right now, we're leaving it up to the translation fun to tokenize the string and extract the username, but it shouldn't have to. We should provide helpers for this, that's were this issue comes in.
We won't be able to complete #2 without at least inventing this syntax; however, this ticket also exists for the implementation of the interpreter of that syntax.
cuttlefish_util:filter_by_variable_starts_with("listener.http", Conf) returns the correct results
cuttlefish_util:filter_by_variable_starts_with("listener.http.", Conf) returns [].
How do we log this? Right now it's to stderr, which is grrrrrrrrreat! but what happens when it's not a user starting it?
console.log? but we don't know where is console.log? or do we? ominous tones
Running pre1 I noticed this in the log output.
-config /Users/jmeredith/basho/work/riak-security/rel/riak/bin/../etc/generated/app.2013.09.18.17.58.25.config -args_file /Users/jmeredith/basho/work/riak-security/rel/riak/bin/../etc/generated/vm.2013.09.18.17.58.25.args
the etc directory isn't guaranteed to be writable by the node. Generated tmp files need to be written somewhere the application has permissions according to os-specific packaging rules.
fsm_limit has two valid value types:
an integer, or the atom undefined
undefined has the special meaning of disabling the sidejob subsystem entirely, which we now can't do.
Now that lager is started up by the cuttlefish_escript. We'll have to be better about what gets logged. Pretty much everthing is at 'info' currently.
We should move everything to debug. Have one 'info' level message to say "This is enabled" which doesn't output if it's using old style config..
Then we should use "warnings" for any config red flags and "errors" for anything that won't generate a config file.
Right now riak.conf is a POC, and has one case of each of three levels of complexity for mapping a Riak app.config. We need a version, in which everything tunable by default in riak is tunable by riak.conf
I found it to be a commone sep to do something like this:
{mapping, "data_dir", "riak_core.platform_data_dir",
[{default, "{{platform_data_dir}}"},
{datatype, string}]}.
{translation,
"fifo_db.db_path",
fun(Conf) ->
cuttlefish_util:conf_get_value("data_dir", Conf)
end
}.
{translation,
"leveldb.data_root",
fun(Conf) ->
cuttlefish_util:conf_get_value("data_dir", Conf)
end
}.
{translation,
"hanoidb.data_root",
fun(Conf) ->
cuttlefish_util:conf_get_value("data_dir", Conf)
end
}.
{translation,
"bitcask.data_root",
fun(Conf) ->
cuttlefish_util:conf_get_value("data_dir", Conf)
end
}.
aka hide the detailed setting under a simplified name for a system. So what would be great is if there would be something like {alias, "hanoidb.data_root"}
which would then act as a 'default' for hanoidb.data_root
(as example).
So this would simplify the above code to:
{mapping, "data_dir", "riak_core.platform_data_dir",
[{default, "{{platform_data_dir}}"},
{datatype, string},
{alias, "fifo_db.db_path"},
{alias, "leveldb.data_root"},
{alias, "hanoidb.data_root"},
{alias, "bitcask.data_root"}]}.
which is kind of sweet ;)
This is a biggie. Well, it's big with regard to importance. I'm not sure how difficult it will be. Let's outline how this will need to work.
Add durations as a datatype
So I'm trying to follow the instructions on https://github.com/basho/cuttlefish/wiki/Cuttlefish-for-Erlang-Developers#wiki-lists-and-proplists-and-names-oh-my, the only problem is that they don't work.
I'm using cuttlefish at version 0.1.0, and I have also tried using the head of the master branch and I get the same results from both.
so this:
%% HTTP Listeners
%% @doc listener.http.<name> is an IP address and TCP port that the Riak
%% HTTP interface will bind.
{mapping, "listener.http.$name", "riak_api.http", [
{default, {"127.0.0.1", 8098}},
{datatype, ip},
{include_default, "internal"}
]}.
{translation,
"riak_api.http",
fun(Conf) ->
HTTP = cuttlefish_variable:filter_by_prefix("listener.http", Conf),
[ IP || {_, IP} <- HTTP]
end
}.
and this:
listener.http.internal = 127.0.0.1:8098
listener.http.external = 10.0.0.1:80
generate this:
{riak_api,[{http,ok}]},
which is not quite what I was hoping that the translation and mapping would do.
Is this a feature that has been removed? or just implemented differently?
it'd be nice if either file existed that it wouldn't be generated each startup. as it stands it just looks like app.config is treated that way.
There are many flags for the Erlang emulator that we have not included in erlang_vm.schema
.
@gburd let me know what you think is "sensible"
in my tests a 01- file overwrites a 00- file and a 99- one overwrites a 01- one.
It would be sweet to have a atom datatype, currently only enums seem to be able to be atoms.
It's easy enough to achiev with a transformation but there really is no reason not to have the type :)
Basically it is possible to have a mapping not take effect when they are not present, right now there isn't a way to have this behaviour for translations.
A simple way to implement this would to change the return value of translation functions to {ok, Value}
or undefined
in which case undefined would mean the translation has no effect.
It would be sweet if it were possilbe to have something like an {include, module} (and/or {include, "top.namespace", module} ) statement that then would pull config definition out of a predefined function (much like the behaviour_info works)
That way libraries could include their config parameters and the autor of a application using (i.e. bitcask, elveldb, lager) would not have to reinvent the wheel (or copy and paste form another app) but could just say {include, lager}
and the lager.erl would export cuttlefish_info(mappings) -> {...} and cuttlefish_info(translations) -> {...}
OS: FreeBSD 9.1, Riak 2.0.0pre2
[vagrant@FREEBSD-91 ~]$ sudo cat /var/log/riak/erlang.log.1
=====
===== LOGGING STARTED Wed Sep 25 15:42:01 UTC 2013
=====
15:42:02.294 [info] Application lager started on node nonode@nohost
15:42:02.294 [info] No app.config detected in /usr/local/etc/riak, activating cuttlefish
15:42:02.428 [info] Adding Defaults
15:42:02.435 [info] Applying Datatypes
15:42:02.435 [info] Validation
15:42:02.437 [info] Applied 1:1 Mappings
15:42:02.439 [info] Applied Translations
Error reading config file: no such file or directory
Error reading -config /usr/local/etc/riak/generated/app.2013.09.25.15.42.02.config -args_file /usr/local/etc/riak/generated/vm.2013.09.25.15.42.02.args
[vagrant@FREEBSD-91 ~]$ ls /usr/local/etc/riak/generated
ls: /usr/local/etc/riak/generated: No such file or directory
Cuttlefish does a great job of explaining errors when they happen while generating app.config and vm.args in memory. The problem here, is that after that phase, if it can't actually write out those files to disk it give the ol' cryptic "failed to generate config with cuttlefish". It could be worse, that message used to end with "there should be logs"
After I had a conversation with @jonmeredith, I agree we should re-think the placement of any generated content in the /etc/ directory. In some environments /etc/ is mounted read-only once a known-good config is made. The generated content for cuttlefish should instead go into the platform_data_dir on each platform so we know there is always the ability to write to that data.
file and/or directory datatype
Need to make it configurable. Right now it's always riak.conf
https://github.com/basho/cuttlefish/blob/develop/src/cuttlefish_rebar_plugin.erl#L58
Investigate why zdbbl breaks when datatype is bytesize. reported by @jrwest
Beyond datatype validation.
e.g. ring_size needs to be a power of 2 and greater than one.
I've got something for this in appconf, but will need rework for dat ๐
So this is a bit related to #67, when having some variable shadowing it should be possible in what way it does so. I can see two scenarios:
When I define my.setting before (with a higher priority) and use translate or alias to riak_core.something
then the riak_core.something
from riak_core.schema
will never show in the config and even if added be simply ignore (or a warning printed that it's a unknown setting)
Example take the code in #67 for example, there the data_dir
in there would be a default for bitcask.data_dir
etc. So it would be neat if bitcask.data_dir
, would not be present in the generated .config
, but if added explictly would overwrite the 'default' provided by data_dir
This is yet another attempt to resolve #79, assuming I finally understand the issue. I appreciate the solution presented in #80, but I'm going to suggest that maybe it doesn't go far enough.
The original idea for translations worked like this: {translation, Mapping, Fun}
Mapping: Was the KVCesque location in the app.config that this thing belonged
Fun: The return of this fun() was put in that mapping location.
Then, it was up to the schema writer to write a Fun that returned the value for that location. You could write a fun with case statements and _ ->
clauses to return the default, but maybe that's not enough.
Here's the collection of changes I'm considering:
Add a proplist to the translation in schema with the ability to add a default value
e.g.
{translation,
"riak_api.http",
fun(Conf) ->
HTTP = cuttlefish_util:filter_by_variable_starts_with("listener.http", Conf),
[ IP || {_, IP} <- HTTP]
end,
[
{default, []}
]
}.
One thing I've been considering for a while is a schema writer's API. Take the function referenced above: cuttlefish_util:filter_by_variable_starts_with/2
. There should be one module with these types of functions in them cfish
for brevity on the schema side. The cfish
versions of these functions would throw errors on things like notfound
or whatever.
This would also make an easy "go to" schema writer's documentation reference.
Right now we try translations and if they error, return undefined. Instead we could return the specified default. We could also do this based on the error, so maybe return the default on notfound
which is desirable behavior. But if there's a different kind of error, we should abort the cuttlefish startup and make the user aware of the error.
For example, if you're trying to take the ceiling of a string, I'm not really satisfied that returning the default and starting up is the right thing to do.
@Licenser dropped the {ok, Value}
stuff on us in PR #80 but, I don't think it's enough. We can move to the {ok, Value}
semantic, but I'd like to add some {error, Message}
to that party. This way in the "ceiling of a string" example, we could include try... catch
clauses in translations, and then turn the output into something human readable.
reported by @seancribbs
TODO: https://github.com/basho/cuttlefish/blob/develop/src/cuttlefish_generator.erl#L200
I believe there is an edge case, in which weird things will happen if you have a schema define something like "listener.http.$name" and then somebody chooses to add a line
listener.http.$name = 127.0.0.1:8098"
to their .conf file.
The easiest way to fix this, is probably just to have neotoma not allow dollar signs on the left side of the equals when parsing conf files.
There are no logs. file:consult/
on the advanced.config works fine. Not sure what's up.
I tried starting Riak using both riak.conf and an advanced.config. My advanced.config was missing the trailing '.', and this was the error I got:
11:58:45.818 [info] Application lager started on node nonode@nohost
11:58:45.818 [info] Checking /Users/reid/Documents/repos/basho-versioned/riak-2.0/rel/riak/bin/../etc/app.config exists... false
11:58:45.818 [info] Checking /Users/reid/Documents/repos/basho-versioned/riak-2.0/rel/riak/bin/../etc/vm.args exists... false
11:58:45.818 [info] No app.config or vm.args detected in /Users/reid/Documents/repos/basho-versioned/riak-2.0/rel/riak/bin/../etc, activating cuttlefish
11:58:46.030 [info] Adding Defaults
11:58:46.034 [info] Applying Datatypes
11:58:46.051 [info] Validation
11:58:46.054 [info] Applied 1:1 Mappings
11:58:46.057 [info] Applied Translations
11:58:46.058 [info] /Users/reid/Documents/repos/basho-versioned/riak-2.0/rel/riak/bin/../etc/advanced.config detected, overlaying proplists
Error generating config with cuttlefish, there should be logs
It would be nice if cuttlefish could tell me the parse error was in the advanced.config.
https://github.com/basho/cuttlefish/blob/develop/src/cuttlefish_generator.erl#L299-301
Actually, it doesn't log too much, because this message is commented out. I'd like it to not be commented, but only for the initial call.
Also, if you are using multibackend, you'll see the cuttlefish phase messages once for the schema, and then again for every backend you used. I don't want logging stopped entirely in that case, but those messages should be stopped. Probably just introduce an arity 3 version for "RecursiveLogs | boolean()"
2> Conf = [{["a", "b", "c"], 1},{["a", "b", "d"], 2},{["a", "c", "c"], 3}].
[{["a","b","c"],1},{["a","b","d"],2},{["a","c","c"],3}]
3> cuttlefish_variable:fuzzy_matches("a.$name.c", Conf).
[]
4> cuttlefish_variable:fuzzy_matches(["a", "$name", "c"], Conf).
[{"$name","c"},{"$name","b"}]
5> cuttlefish_variable:fuzzy_matches(["a", "$name"], Conf).
[]
6> cuttlefish_variable:fuzzy_matches(["a", "$name", "d"], Conf).
[{"$name","b"}]
a) we should see "a.$name.c" and tokenize it if it's a string
b) we should allow a search for "a.$name" to find everything that could start with "a.$name"
Setting storage_backend
to "eleveldb" instead of "leveldb" is an error, but it's silently ignored. This was seen using cuttlefish commit 09a1aa93, but might also be an issue with previous versions as well.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.