cuelang / cue Goto Github PK
View Code? Open in Web Editor NEWCUE has moved to https://github.com/cue-lang/cue
Home Page: https://cuelang.org
License: Apache License 2.0
CUE has moved to https://github.com/cue-lang/cue
Home Page: https://cuelang.org
License: Apache License 2.0
cue seems to detect what the spec describes as reference cycles. But for structural cycles, it runs into a stack overflow:
goroutine 1 [running]:
runtime.heapBitsSetType(0xc04ab64070, 0x8, 0x8, 0xc5bcc0)
/usr/lib/google-golang/src/runtime/mbitmap.go:938 +0xa55 fp=0xc028000350 sp=0xc028000348 pc=0x52cea5
runtime.mallocgc(0x8, 0xc5bcc0, 0x1, 0x0)
/usr/lib/google-golang/src/runtime/malloc.go:969 +0x50e fp=0xc0280003f0 sp=0xc028000350 pc=0x522c3e
runtime.newobject(0xc5bcc0, 0x0)
/usr/lib/google-golang/src/runtime/malloc.go:1068 +0x38 fp=0xc028000420 sp=0xc0280003f0 pc=0x523298
cuelang.org/go/cue.(*structLit).expandFields(0xc00025c000, 0xc01907de60, 0xc01907de60)
/usr/local/google/home/freinartz/repos/src/cuelang.org/go/cue/value.go:646 +0x42 fp=0xc028000658 sp=0xc028000420 pc=0x7d88c2
cuelang.org/go/cue.Value.structValOpts(0xc000240b90, 0xc04ab63a00, 0xc01907de60, 0x101000000, 0xc04ab63a00, 0xc00025c000, 0xd98c20, 0xc00025c000, 0xd96780)
/usr/local/google/home/freinartz/repos/src/cuelang.org/go/cue/types.go:813 +0xcd fp=0xc028000788 sp=0xc028000658 pc=0x7d239d
cuelang.org/go/cue.Value.structVal(...)
/usr/local/google/home/freinartz/repos/src/cuelang.org/go/cue/types.go:799
cuelang.org/go/cue.Value.Walk(0xc000240b90, 0xc04ab63a00, 0xc047fff870, 0x0)
/usr/local/google/home/freinartz/repos/src/cuelang.org/go/cue/types.go:1126 +0xe9 fp=0xc028000890 sp=0xc028000788 pc=0x7d4649
cuelang.org/go/cue.Value.Walk(0xc000240b90, 0xc04ab639c0, 0xc047fff870, 0x0)
/usr/local/google/home/freinartz/repos/src/cuelang.org/go/cue/types.go:1129 +0x161 fp=0xc028000998 sp=0xc028000890 pc=0x7d46c1
cuelang.org/go/cue.Value.Walk(0xc000240b90, 0xc04ab63980, 0xc047fff870, 0x0)
/usr/local/google/home/freinartz/repos/src/cuelang.org/go/cue/types.go:1129 +0x161 fp=0xc028000aa0 sp=0xc028000998 pc=0x7d46c1
cuelang.org/go/cue.Value.Walk(0xc000240b90, 0xc04ab63940, 0xc047fff870, 0x0)
...
In #33, @mpvl does IMHO a very good job at highlighting the core difference between cue and jsonnet.
I like cue very much; it really has the potential to tame the complexity of configs.
Its generative aspect works well for templating when the templates have been explicitly designed for extension (and arguably forcing the author to be explicit is instrumental for the success of taming the complexity in the first place)
One of my main annoyances with Helm, is that chart authors are effectively forced to expose every possible field of every k8s object as values in the values.yaml template, because sooner or later some consumer of that chart (usually from outside the administrative domain of the chart author) will have a reasonable use case for the override.
In jsonnet, you can just take an upstream yaml or json object, parse it and apply overrides with jsonnet.
For cue to be useful in this scenario, would we need a function that converts a cue object tree into an object tree that replaces every field value x
with *x | _
.
Then we could do something like (I know that "Defaultify" is a horrible name, cannot currently think of a better name right now):
import (
"encoding/yaml"
"override"
)
res: override.Defaultify(yaml.Unmarshal("""
foo:
bar:
a: 1
b: 2
"""
)) & {
foo bar b: 10
}
once an external value is lifted into cue value model, it can be further refined applying type and value limitations that make sense to the user.
Perhaps I'm missing something. Is there already a way to achieve this today?
(I saw there is an interesting approach for importing external legacy configs based on source level imports, but I think that solves a different use case)
Atm. cue has no builtin's for encoding and decoding strings with base64.
base64 encoded values are used by kubernetes secrets.
I guess a good starting point is the encoding/hex
package. I also would like to implement the missing functionality.
I am not sure if this is pilot error, or a recent commit changed something, but I can't see any objects being listed or dumped.
cd cuelang/cue/doc/tutorial/kubernetes/manual/services
cue cmd dump ./...
cue cmd dump
cue cmd ls
cue cmd ls ./...
None of the above commands produce any output. cmd ls|dump
works if I specify a path to a directory. For example - this works:
cue ls frontend/waiter
Service frontend waiter
Deployment frontend waiter
Is this expected behavior?
The behavior of struct unification specified as
The successful unification of structs
a
andb
is a new structc
which has all fields of botha
andb
Cannot define an upper bound for structs. Specifically, CUE cannot help us verify that the resulting expression is limited to allowed fields.
For example, the unification
base: {
onlyViableField: string | *"myDefault"
}
{
base & {onlyViableFeild: "nonDefaultValue"}
}
yields
{
onlyViableField: "myDefault"
onlyViableFeild: "nonDefaultValue"
}
In kubernetes, the existence of extraneous fields causes failure when kubectl
apply
s the resulting YAML, whereas in other systems the expression above could use silently use the default value erroneously.
When comparing CUE to a typed configuration language (like dhall), this is a qualitative difference in capabilities, since the extraneous field would be found by explicitly typing the result.
Extraneous fields can be hard to find - especially in large, opaque, auto-generated config schemes.
Please take a look at https://github.com/jpillora/opts/tree/subcommands (work in progress). I think cue is a fascinating project and happy to contribute (I have started a similar project using protobufs as starting point).
Thought helping move from Cobra to Opts might be a good first start.
One of this nice things about the work in progress on Opts is basic bash, zsh and fish command completion is free and highly customisable for more complex requirements.
Analogous to the Go's go generate
.
CUE's is designed with code generation in mind and it is to be expected that many CUE files will be generated. A cue:generate directive would allow annotating the methods used to generate the files in a way familiar to Go programmers.
Tracking issue for adding protobuf support to CUE:
I was experimenting with adding a set of validation commands to my cue configuration, in _tools.cue
files. The commands they exec out to will return non-zero exit codes when those commands fail, but running them via the cue <cmd>
runner eats those exit codes and always returns 0. Checking the code this doesn't appear to be something I can add to the print
task.
Lines 299 to 310 in 2b0e7cd
Is there a way of achieving this at present? Supporting passing through the exit codes would make integration of cue commands into pipelines easier.
I have cue file test.cue
:
address: string
And when I call this, I got an error:
$ cue export test.cue
json: error calling MarshalJSON for type cue.Value: json: error calling MarshalJSON for type cue.Value: cannot convert incomplete value "string" to JSON
a <Name>: foo
a b: {}
gives the following on eval:
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x50 pc=0x7dac0e]
goroutine 1 [running]:
cuelang.org/go/cmd/cue/cmd.Execute.func1()
/usr/local/google/home/bde/go/src/cuelang.org/go/cmd/cue/cmd/root.go:86 +0x10a
panic(0xc00980, 0x12eb230)
/usr/lib/google-golang/src/runtime/panic.go:522 +0x1b5
cuelang.org/go/cue.(*structValue).Len(...)
/usr/local/google/home/bde/go/src/cuelang.org/go/cue/types.go:83
cuelang.org/go/cue.Value.Walk(0xc000242d70, 0xc000242e60, 0xc000219870, 0x0)
/usr/local/google/home/bde/go/src/cuelang.org/go/cue/types.go:1327 +0x17e
cuelang.org/go/cue.Value.Walk(0xc000242d70, 0xc000242e10, 0xc000219870, 0x0)
/usr/local/google/home/bde/go/src/cuelang.org/go/cue/types.go:1329 +0x16d
cuelang.org/go/cue.Value.Walk(0xc000242d70, 0xc000242dc0, 0xc000219870, 0x0)
/usr/local/google/home/bde/go/src/cuelang.org/go/cue/types.go:1329 +0x16d
cuelang.org/go/cue.Value.Validate(0xc000242d70, 0xc000242dc0, 0x0, 0x0, 0x0, 0x1, 0x1)
/usr/local/google/home/bde/go/src/cuelang.org/go/cue/types.go:1274 +0x106
cuelang.org/go/cmd/cue/cmd.buildInstances(0x12f5080, 0xc000010c80, 0x1, 0x1, 0xc000010c80, 0x1, 0x1)
/usr/local/google/home/bde/go/src/cuelang.org/go/cmd/cue/cmd/common.go:102 +0x110
cuelang.org/go/cmd/cue/cmd.buildFromArgs(0x12f5080, 0xc00023e950, 0x1, 0x1, 0x8a7560, 0xbee4e0, 0x0)
/usr/local/google/home/bde/go/src/cuelang.org/go/cmd/cue/cmd/common.go:72 +0x7e
cuelang.org/go/cmd/cue/cmd.glob..func2(0x12f5080, 0xc00023e950, 0x1, 0x1, 0x0, 0x0)
/usr/local/google/home/bde/go/src/cuelang.org/go/cmd/cue/cmd/eval.go:50 +0x6a
github.com/spf13/cobra.(*Command).execute(0x12f5080, 0xc00023e920, 0x1, 0x1, 0x12f5080, 0xc00023e920)
/usr/local/google/home/bde/go/src/github.com/spf13/cobra/command.go:762 +0x465
github.com/spf13/cobra.(*Command).ExecuteC(0x12f4960, 0xc000219da0, 0x7ffd2f1f1fba, 0x4)
/usr/local/google/home/bde/go/src/github.com/spf13/cobra/command.go:850 +0x2fc
github.com/spf13/cobra.(*Command).Execute(...)
/usr/local/google/home/bde/go/src/github.com/spf13/cobra/command.go:800
cuelang.org/go/cmd/cue/cmd.Execute()
/usr/local/google/home/bde/go/src/cuelang.org/go/cmd/cue/cmd/root.go:129 +0xd9
main.main()
Thanks for the nice walkthrough, that shows some of the great power of Cue :-)
Parts of it refers to previous versions of Cue:
cue eval ./... > snapshot2
doesn't fail as indicated, since now eval
can output non-atomic string
Under "1 is the default", 1
is not marked as default.
As of current master, the following CUE program doesn't evaluate as expected:
import "encoding/json"
input: "{\"one\": 1, \"two\": 2, \"three\": 3}"
result: json.Unmarshal(input)
Expected output:
{
"input": "{\"one\": 1, \"two\": 2, \"three\": 3}",
"result": {
"one": 1,
"two": 2,
"three": 3
}
}
Actual output (cue export
):
{
"input": "{\"one\": 1, \"two\": 2, \"three\": 3}",
"result": {
"Valid": 1,
"Compact": 2,
"Indent": 3
}
}
Notice the field names in the unmarshaled result have been replaced with the names of methods in pkg/encoding/json
.
This is a tracking issue related to extracting CUE from Go code. The functionality in this issue focusses on use cases where Go is used as the source of truth. Issue #6 focusses on generating Go assuming that CUE is used as the source of truth.
injection:
Allow annotation of Go types with additional CUE constraints and provide a simple API to validate and complete Go values.
extraction:
The primary extraction method looks at struct definitions in Go code, determines how these would be interpreted by the encoding/json package, and then generates CUE definitions for such structs.
In addition, we can extract CUE annotations used in Go code for validation to augment such definitions.
Since CUE is a superset of JSON, I understand that you can paste valid JSON into a .cue
file and it will be guaranteed valid CUE as well.
My question is, is it guaranteed that valid JSON pasted into a .cue
file will never accidentally invoke/trigger any features of CUE that are not in JSON?
For example, I'm guessing that string interpolation can't be triggered accidentally by valid JSON, since (I presume) the escape sequence \(
is invalid JSON. Is CUE designed in such a way that all such features are systematically impossible to trigger from valid JSON?
In other words, is all valid JSON guaranteed to produce exactly the same output when treated as a CUE program? I haven't seen mention of this in the docs.
The tutorial page on Bottom makes it seem like the user can see partial output like this when errors occur:
a: _|_
l: _|_
list: [0, 1, 2]
val: _|_
But instead, the tool outputs messages like this:
cannot unify numbers 2 and 3:
[...]
cannot unify numbers 4 and 5:
[...]
index 3 out of bounds:
[...]
From #13 it sounds like the tool just doesn't support this yet. In the meantime, perhaps the docs should be edited to clarify that the output shown is theoretical, so users know they didn't do something wrong.
This is a tracking issue for improved error support
kubectl allows patching using a "strategic merge". In short, it allows lists to be treated like maps so that the right elements can be merged.
In general, unification of lists is very tedious for automated configuration. A possible solution is for users to define mappings manually. Although CUE can handle reconstructing and merging unknown API, it is still tedious. Ideally users would just use the native API of whatever system they work on.
Also, ideally, mappings should be able to be applied with the same separation of concerns as is possible with normal objects.
One possible approach would be to extend the emit mechanisms for inner objects:
myList: [ ...{ name: string } ]
// before evaluation, the list is converted to a map like this
myList: {
<- "\(x.name)" : x for x in $ // $ means self or current object
// after evaluation, it is converted back to this format.
-> [ x for x in $ ]
}
The top-level emit is then ->
Expr where the ->
can be elided as the top-level value is always a map.
This needs a lot more thought, but having some kind of mechanism like this can be important.
There may also be overlap with a design for field attributes.
Allow constraining string values with regular expressions:
a: re("[a-z][a-z][a-z]")
a: "foo"
The syntax is yet to be determined. But see #1.
Allow diffing between (snapshots of) previous version and current version.
To display the current build and version.
Using types.cue
from https://github.com/cuelang/cue/blob/master/doc/tutorial/basics/types.md errors on running cue eval
.
$ cue eval -i types.cue
--- .
unsupported op &((float)*, int):
./types.cue:2:8
./types.cue:7:10
// /home/ghchinoy/dev/cueexperments
{
point: {
x: float
y: float
}
xaxis: {
x: _|_ // unsupported op &((float)*, int)
y: float
}
yaxis: {
x: float
y: _|_ // unsupported op &((float)*, int)
}
origin: {
x: _|_ // unsupported op &((float)*, int)
y: _|_ // unsupported op &((float)*, int)
}
}```
Line 66 in 433ab7a
The reason for this is that cue evaluates list types too aggressively. This does not change the result of a usual CUE evaluation, but it does disallow introspection of intermediate results as is needed by the openapi package.
There are several ways to handle this, but most ideally and most consistently, cue could treat list item types analoguously to struct templates.
Alternatives are to track more origin information in result nodes.
Excellent effort, would be a great replacement for our inhouse python based system. However we are missing an important feature, documenting fields which are accessible from api/programming which helps auto generating documentation, provides schema comments when autogenerating schemas for third party systems etc.
I can't seem to get import of a local package to work. An example would be great. Maybe that could be folded into kubernetes/manual
?
Or really any programming language, but Go is a good first target.
This input:
outer middle inner: 3
produces this with CUE
{
"outer": {
"middle": {
"inner": 3
}
}
}
and this with YAML:
{
"outer middle inner": 3
}
this seems like too much sugar to be worth it
https://github.com/cuelang/cue/blob/master/doc/tutorial/basics/fold.md
How is CUE different from jsonnet, both coming from Google, and the latter being largely promoted especially in the context of Kubernetes?
Current CUE supports the more old-school 0777 notation. 0o, however, is more legible, has better properties regarding scanning, and seems to be the standard languages are converging on.
The CUE-YAML importer would need to be modified to convert octal numbers accordingly.
The old format should be dropped.
I was a little surprised to see the '.' here:
https://cue.googlesource.com/cue/+/HEAD/doc/tutorial/basics/rangedef.md
int16 >=-32_768 & <=.32_767
When converting an overlay system to use CUE, "parent" template will often hard code defaults, about which CUE will complain if any instances unify a struct overriding such defaults. As this is likely a common case, it makes sense to add refactor support for this.
The relax command can rewrite parent structs to relax constraints. It is a simple variant of anti-unification as proposed in #7.
In a typical workflow, a user would run relax and then manually verify in the vc diffs which relaxations to keep and which reveal bugs.
For instance:
a: >5 // This field must be greater than 5
b: !=null // This field must not be null
c: >3.0 & <10 // (3.0, 10.0) open ended range on both sides
Supported operators:
< > >= <= != ==
These would be defined on the same types as their binary equivalents. Semantically such expressions would define disjunctions of possible values, analogous to ranges.
Introducing these would solve various limitations that are currently present. Such as defining open ended ranges for floats or requiring that a value should not be of a specific type.
https://github.com/cuelang/cue/blob/master/doc/tutorial/basics/stringraw.md
Note the markdown uses "```json" causing the red rendering
ie this...
$ cue eval stringraw.cue
{
msg1: "The sequence \"\\U0001F604\" renders as ๐."
msg2: """
A regular expression can conveniently be written as:
#\"\\d{3}\"#
This construct works for bytes, strings and their
multi-line variants.
"""
}
should be ...
$ cue eval stringraw.cue
{
msg1: "the sequence \"\\U0001F604\" renders as ๐."
msg2: """
A regular expression can conveniently be written as:
#\"\\d{3}\"#
This construct works for bytes, strings and their
multi-line variants.
"""
}
Similarly, https://github.com/cuelang/cue/blob/master/doc/tutorial/basics/bytes.md exhibits similar (missing quotes around cue export
rendered json due to missing quotes in md)
ref also #48
I'm not sure if this is just not implemented yet, but filing an issue to track either way.
At current master, the examples for string interpolation in field names don't seem to work. Any field with an interpolation in the name is silently dropped from the output.
Not sure if this is just not implemented yet, but the examples of breaking reference cycles don't work as of current master:
// CUE knows how to resolve the following:
x: 200
x: y + 100
y: x - 100
constraint violated: _|_(200:delayed constraint (200 & (<0>.y + 100)) violated)
This is a tracking Issue for any OpenAPI changes until most of it is completed. This includes:
This is a tracking issue for the implementation of the commented out sections in the spec for conversions.
For instance:
But also to convert a struct to a specific struct type. The latter is quite handy for various purposes.
This likely means that the cli docs should be generated from the tool code.
The output from eval is a mix of JSON and (invalid) CUE and indents weirdly. It was quickly written before package format was written.
Replace the output with using package format.
Also, it would be useful to see partially evaluated configurations.
Swift raw strings seem to be the nicest design for raw strings out there.
Benefits:
#"This is \ a raw string"#
#"Raw string with interpolation \#(interpolated)."#
##"Raw string without interpolation \#(interpolated)."##
etc.
The current output points folks to http://cuelang.org/pkg/tool for information on available tasks. However that URL doesn't have any content.
$ curl -i http://cuelang.org/pkg/tool
HTTP/1.1 404 Not Found
Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff
X-Cloud-Trace-Context: 06cb78cc27dfe93e8cc9dd3bde03b522;o=1
Date: Mon, 22 Apr 2019 10:32:57 GMT
Server: Google Frontend
Content-Length: 19
404 page not found
Referential cycles are currently implemented. Structural cycles are not, however. So CUE currently allows the definition of infinite structures like:
{ f: { a: f } }
As defined by the spec, infinite structures are not allowed. Infinite structures should be detected and disallowed using an occurs check, as is commonly done in graph unification implementations.
There are some design decisions to be made though. A careful choice of semantics in the occurs check could make CUE not Turing complete and even guarantee termination. This weakens the power of CUE, but may be a desirable property if CUE is to be used for things like smart contracts, where the ability to determine termination would be a major win.
For an example of Turing completeness in CUE see the test called "reentrance" in resolve_test.go, which implements Fibonacci in CUE. The specifics of the semantics of the occurs check will determine whether these shenanigans would be allowed or not.
Depending on the level of restrictions, it should be investigated whether the occurs check can be implemented cheaply using Tomabechi-style cycle detection.
There is currently no way to reference a top-level field with a non-standard character:
"foo-bar": "can't touch this"
a: // cannot reference "foo-bar"
One solution is to introduce $
to refer to the root object. This is possible, but such references often result in poor configuration file design. It is also one other thing to learn for a user.
An alternative approach is to allow a syntax for identifiers with non-standard characters. Swift and GCL allow back quotes for this purpose:
"foo-bar": "can touch this"
a: `foo-bar`
Main drawback: now we have three ways of referring to identifiers. Allowing back quotes also on the left hand side might mitigate this issue. The string approach may be considered a relic of JSON and hardly used unless needed for interpolation.
`foo-bar`: "can touch this"
a: `foo-bar`
"dynamic\(interpolated)": 42
cue fmt could be used to canonicalize LHS identifiers according to this rule.
With this code
x: {a:1}|{a:2}
y: x & {a:3}
I get the following not so helpful message:
empty disjunction: &{{0xc0001fe600} <nil> <nil> [] [{4 0xc00007ee00 <nil> <nil>}] 0xc0001eed20}:
/...<filename>:2:4
terminating because of errors
Implement cue hoist
to extract templates from concrete data definitions. One could see this as cue trim
in the opposite direction.
This can be implemented using the known anti-unification algorithm. It probably makes sense, though, to disregard disjunctions or to limit the number of elements.
While running:
go get -u cuelang.org/go/cmd/cue
I get:
cmd/cue/cmd/vet.go:66:7: undefined: fVerbose
https://github.com/cuelang/cue/blob/master/cmd/cue/cmd/vet.go#L66
CL https://cue-review.googlesource.com/c/cue/+/1660 introduced an alternative semantics for default values that allow also default values to be fully associative, commutative, idempotent, and distributive.
This still needs to be implemented. The implementation should go hand in hand with tidying up and clarifying the semantics of references and the use of defaults in arithmetic and the combination thereof.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.