dart-lang / build Goto Github PK

Non-incremental builds only.

There is an open question here as to how this should be configured. Especially for transitive deps this gets a bit interesting ;).

Probably the best thing to do here is just quit with some sort of error message?

https://www.dartdocs.org/documentation/build/0.3.0%2B6/

drops you on a 404.. the doco there just the link from pub dart packages is not point at index.

thanks

After a build, the next build needs to know which files were generated previously, so that they aren't treated as inputs.

Proposed solution:

Output a build_outputs.json file under a new hidden folder .build, eventually this will be replaced by the dependency graph file, but for now it will just be a list of all output file ids. Just using JSON for now as its the easiest, eventually more formats need to be evaluated.

On startup, check if the file exists and:

• If it does exists, read the file and ignore all listed files as inputs. Probably also delete those files from disk (otherwise we can easily end up with stranded files, we can be smarter about this later on).
• If it doesn't exist, call declareOutputs on all builders in all phases, recording all declared outputs which would conflict with an existing file on disk. If there are conflicts then summarize the results to the user to confirm deletion of these files, and then start over.

If I launch build/watch with "pub run tool/watch.dart" I see an error like:

[INFO] Build: Unrecognized uri scheme http found for library in build script, falling back on full rebuild.

I tweaked it to print the full URI and got:

[INFO] Build: Unrecognized uri scheme http://localhost:53974/watch.dart found for library in build script, falling back on full rebuild.

Just "dart tool/watch.dart" doesn't have this problem.

When packages are nested inside of other packages, duplicate watch events fire (one for each package), and we need to make sure we handle that case well.

Today if any files in any packages are updated it will run a rebuild, which often ends up being a no-op. Instead we should at least verify that the updates are for files which either exist in the asset graph (ie: they are a dependency) or which match one of the input sets for one of the phases.

You can't output files in other packages, but it is common to want to output a file which corresponds to a file in another package (maybe its the compiled js file, or a metadata file). Sometimes these files can just be published with that package, but there are a lot of situations where that might not be the ideal case, and it makes sense for the application package to do all the code generation.

My proposed solution is to add the following as methods on BuildStep (which has a package graph representation):

/// Gives a path corresponding to [original] in the local package.
AssetId rootPackageAssetFor(AssetId original);

// Should this return a Uri or a String?
String assetIdToUri(AssetId id);

// Should this take a Uri or a String for the first arg?
// Relative paths will be relative to [from]. Should this be an optional arg?
AssetId uriToAssetId(String uri, AssetId from); 

The rootPackageAssetFor method would return a path like lib/$someSpecialFolder/${original.package}/${original.path} where someSpecialFolder is a special folder name that we decide to reserve for this purpose.

These can still be cached on disk, but should live in some hidden directory, not wherever the rest of the build outputs are written.

For example the angular transformer has many intermediate assets which it cleans up later on. These assets still can't be deleted by later builders, but that shouldn't be an issue.

Non-incremental, local package support only.

If I use the build function I can run the build as many times as I like and all is well, but when I change it to use watch then it runs once but then on changes blows up with

InvalidOutputException: Asset: jason_schemer|lib/src/m2m/schema_to_vcore.g.forwards.dart
Cannot overwrite inputs.

jason_schemer|lib/src/m2m/schema_to_vcore.g.forwards.dart is an output but this seems to suggest watch is thinking it is an input too

This needs to see if any dependencies change as well.

Now that the resolver is fixed, it appears to be dreadfully slow :(.

Need to make a Build class to wrap up internal state, instead of zone-local values. Its getting a bit out of control and probably hurting performance ;).

Today each rebuild will list the file system for all files. We should instead just keep an updated set of valid inputs, based on updates that are seen.

Breaking, yes, but good style

https://www.dartlang.org/effective-dart/style/#prefer-using-lowercamelcase-for-constant-names

• test/builder/build_step_impl_test.dart fails when ran standalone, on travis.
• test/analyzer/resolver_test.dart fails when ran standalone locally and on travis.

Both are happy when ran via pub run test

Today we only handle deletes (by regenerating the file), but if you modify them we ignore it. This is because we don't have a way of detecting edits currently, the same modified timestamp logic as regular inputs doesn't work since it is always newer than the asset graph.

It is also unclear what should even be done in this case?

Don't need to do anything with it yet

I have a case where

• I'm using a watch.dart that's identical to the build.dart, it just calls 'watch' instead of 'build'
• When I make changes and re-run build.dart, it notices the changes and correctly updates everything
• But if I run watch.dart, and make the same changes
  --> it notices that I saved a file and triggers a build
  --> but, the build does nothing -- says '0 output files'

Am I correct that running the watcher should be equivalent to repeatedly running the build script -- or did I miss something there?

Thanks!

Noticed in build_impl.dart you do string formatting on an exception.

1. These calls in Logger have positional error, stackTrace args that can be used
2. This call doesn't format things nicely with stack_trace – which makes it hard to debug issues.

If you have a graph like foo.dart => foo.dart.summary, things that depend on foo.dart.summary should only get regenerated if that file changes.

Today that is not the case, everything is invalidated up front.

I think this should just be implemented as a general CachedAssetReader which takes an instance of AssetReader, and delegates to that whenever there isn't a cached asset available. There would also be a CachedAssetWriter which takes an instance of AssetWriter and a CachedAssetReader (so it can populate the readers cache with new values).

Another option is to just implement an AssetCache class which gets passed into individual implementations of AssetReader/AssetWriter, but then each of those implementations needs to know about the cache.

Today only one build script per package is supported because all scripts share the same asset graph.

We should at least provide some errors or something around this.

How to use it, phases, etc.

It could be that you simply supply this when you create them, in which case this issue can just be closed, but its a common enough case that we might want something more general.

On

https://github.com/google/built_json.dart/tree/master/example

(add a "watch" script under "tool", that similarly runs all three generators)

I see odd behaviour: changing input files seems to correctly trigger builds, and updated output, but only a subset of the expected changes happen. It looks like the issue is over-caching of input sources -- when I debug, I see old sources as input to some of the generators.

For example change compound_value.dart to rename a field in both CompoundValue and CompoundValueBuilder. The field should be renamed in the generated output, compound_value.g.dart, from BuiltValueGenerator. But no change is made.

Oddly, the input change is detected by BuiltJsonGenerator: again in compound_value.dart, comment out the 'serializer' field. The generated output serializers.g.dart is correctly updated to remove the reference to it. Uncomment, and it's correctly updated again.

I tried to dig into the caching to figure out what's going on here, but got a bit lost. Any pointers appreciated :) ... otherwise, could you take a look, please?

In my project, I need to generate classes from a XML file.

My InputSet is the single xml file. The output assets are determined after passing the xml file (I am trying to generate a file for each class). Due to the declareOutputs function, I have to parse the xml file by using file system directly inside the declareOutputs function and cache the generated code (I don't want to parse it again in the build function). This does not feel right to me. So I ended up jamming all classes into a single file.

Why do we need the declareOutputs function? IMO, Let the user decide which file to write in the build function will be much flexible. (Like the transformer which only require inputs).

Will start with JSON just to get things up and running, but proto etc should be evaluated.

Blocked by dart-lang/sdk#26138

It just...hurts

Doesn't need to block on ongoing builds yet

When embedded inside another build system like bazel we just want to skip this.

One of my potential inputs to build is metadata in the form of a JSON schema or Swagger (OpenAPI). What will be output is not actually known until parsing the file(s).

Not sure if I'm just thinking about things wrong with how you'd use build so any other options are appreciated.

Triggered here

https://github.com/dart-lang/build/blob/master/lib/src/generate/build_impl.dart#L523

causes the error message "Files may only be output in the root (application) package." which is misleading. It should be something like, "Cannot overwrite input."