pistondevelopers / eco Goto Github PK

Dev-dependencies are currently not supported.

Add basic support where the order is not changed, and dev-dependencies are only updated if the library needs update.

• Add dev-dependency support to the Cargo.toml syntax
• Add dependencies::Package::dev_dependencies
• Convert meta data to dev-dependencies in dependencies
• Print dev-dependencies in dependencies
• Add dev-dependency support to the dependency info syntax
• Add dev-dependency support to the update info syntax
• Add update::Package::dev_dependencies
• Print dev-dependencies in update
• Use dev-dependencies to find latest version used by any dependency.
• Find dev-dependencies that needs update.

Eco needs a model of how it knows when a library causes a breaking change, because all automated processes must be correct or have known error cases, which might be hard to figure out from the source. This issue is for tracking the model and proofs based on the planned design.

The definition of "breaking change" used here:

• Does not include dependency version conflicts, when a library has multiple dependency paths to versions of same library that differs in the first non-zero number.
• Includes a breaking change caused by the existence of a newer version of the library that differs in the first non-zero number.

In summary, when a library is listed in the extract info, it is known whether the library will cause a breaking change or not:

might_cause_breaking_change(library) -> knowledge
([is_listed_in_extract_info] [:] true) -> [:] known

If a library is not listed in the extract info, it is not known whether it will cause a breaking change. It might be possible to detect whether it causes a dependency conflict, which is why the definition of "breaking change" does not include dependency conflicts. By design this desirable because:

1. Some libraries are stable and have stable dependencies. We are not interested in extracting information in this case.
2. A project might choose to run on a specific version for a time period to speed up development. We are interested in the breaking change information related to a localized part of the ecosystem.
3. The known error case is when a library listed in the extract info has a dependency that is not listed. In a such case a breaking change is not detected. However, if one of the listed libraries updates its dependencies, a potential dependency version conflict might be detected.

Proof

The extract info tells which part of the ecosystem that we are interested in knowing about breaking changes. For each library, it is either listed in the extract info or not.

is_listed_in_extract_info(library) -> bool

For each library listed in the extract info, the newest version of the library is known because it is listed in the Cargo.toml given by the url:

know_newest_version(library) -> knowledge
([is_listed_in_extract_info] [:] true) -> [:] known

If we know the newest version, we can detect whether a breaking change occurs:

might_cause_breaking_change(library) -> knowledge
([know_newest_version] [:] X) -> [:] X

Since we know the newest version of libraries listed in the extract info, we know whether they might cause breaking changes:

might_cause_breaking_change([is_listed_in_extract_info] [:] true) -> [:] known

Needed by #19.

Should convert from meta data to Vec<Package>.

I believe the same conversion code can be used for Package when extracting from Cargo.toml.

"publish": false tells that the version number of the package should not be updated.

This is a step toward doing automatic PR using update information.

The idea is that Eco could be part of another tool that did some automated tasks to keep an ecosystem healthy. By changing it into a library it can be reused in other projects.

• Change "Eco" into a library
• Move main to an example "piston"

See #1

This should be limited to the part of the ecosystem we are interested in. The document does not necessarily contain all the dependencies the libraries have.

• List of libraries (name, version, dependencies)
• Each dependency has name and version

Example:

{
    "pistoncore-input": {
        "version": "0.4.0",
        "dependencies": {}
    },
    "pistoncore-window": {
        "version": "0.6.0",
        "dependencies": {
            "pistoncore-input": {
                "version": "0.4.0"
            }
        }
    },
    "pistoncore-event_loop": {
        "version": "0.7.0",
        "dependencies": {
            "pistoncore-input": {
                "version": "0.4.0"
            },
            "pistoncore-window": {
                "version": "0.6.0"
            }
        }
    },
    "piston": {
        "version": "0.7.0",
        "dependencies": {
            "pistoncore-input": {
                "version": "0.4.0"
            },
            "pistoncore-window": {
                "version": "0.6.0"
            },
            "pistoncore-event_loop": {
                "version": "0.7.0"
            }
        }
    }
}

When a dependency version starts with "^" or ends with ".*", it means the same as removing this information before parsing the version number.

See #5

Add meta syntax for extracting information from Cargo.toml.

• Package name
• Version
• Dependencies (name + version, support both [dependencies.foo] and [dependencies] list)

https://github.com/PistonDevelopers/editor

Sometimes updating is not possible, or you need to wait for merging. It would be nice to have a way to ignore the updates resulting from a specific dependency of a library, such that one can focus on the other updates.

Add "ignore-version": "0.2.0"` to extract information only if the current version number does not match the ignored version.

This way, if one sends a PR to update a library, one can add ignore-version which will filter out updates until the PR gets merged.

• Update extract syntax, replacing "override-version" with "ignore-version".
• Replace Extract::override_version with ignore_version.
• When extracting from Cargo.toml, check for string equality, don't push to package_data if equal.

Currently it prints out in generate_update_info_from.

Currently the design of Eco solves the update problem for dependencies, but lacks a proper solution for dev-dependencies. In order to handle dev-dependencies we need a model of how they behave.

The ideal thing would be to recommend update actions that keeps a library from breaking the examples/tests/benchmarks (dev stuff).

• A library with dev-dependencies gets sandwiched between the desire to get it updated quickly so other libraries can update, and the desire to keep the dev stuff from breaking.
• Usually, you don't want to break the dev stuff.
• When a dev-dependency depends on the library itself, it is impossible to update the library without breaking the dev-stuff.
• If it is impossible to avoid breaking dev stuff, then this knowledge could be used to give other libraries higher priority to update the ecosystem faster as a whole.
• Unless there is a "hole" in the ecosystem, the depth order of a dev-dependency can be used to detect whether it possibly can depend on the library. The depth order must be greater than the library to depend on it.
• If a dev-dependency is a hole, then it could depend on the library, but usually not.
• Usually you want to update all the dev dependencies in one go. This means either you split the update into dependencies and dev-dependencies or keep them together.
• If an update is split into dependencies and dev-dependencies, you usually want to update dev-dependencies as soon as these libraries are updated.
• If any dev dependency depends on a library with higher depth, you usually want to delay the update such that update of dependencies and dev-dependencies can be done in one step.

A = has dev dependencies
B = has any dev dependendency depending on library
C = is impossible to avoid breaking dev stuff
D = all dev-dependencies have less or equal depth than the library
E = a dev-dependency is a hole
F = split update of dependencies and dev-dependencies

!A, !(B, C), D, !E, !F.
(B, C), A, !D, ?!E, F.
D, ?A, !(B, C), ?!E, !F.
E, A, ?!(B, C), !D, ?!F.

This uses a new version of the piston-meta language that is more readable.

When a breaking change occurs in the libraries listed in the extraction info, all dependency version conflicts can be solved by upgrading to the newest version, see #16.

When a library is not listed in the extraction info, a dependency version conflict might occur. I believe the right way to solve this is to use the newest version used by any dependency, because:

1. Given a localized part of the ecosystem, a downstream library depending on that part should not experience dependency version conflict after the information from Eco is used to upgrade the libraries.
2. Dependency version conflicts are caused by multiple dependency paths to different versions of the same library. This is possible when a subset of the localized ecosystem contain different versions of the same library.
3. It is not possible to prevent a downstream library from using a subset such that a dependency conflict does not occur.

Also, a localized part of the ecosystem might use an older version of a library to speed up development for a time period. In this case it is desirable to have the easiest upgrade possible.

The only solution that satisfy these constraints is by upgrading to the newest version used by any dependency.

• For all libraries not listed in the extraction info, find the newest version
• Add an update action for versions of the same library that uses an older version

Eco has some dependencies that updates from time to time.

For example, the Github version contains breaking changes that are not yet released. Because of a breaking change in one of the dependencies, the library is published to crates.io with a patch update.

• The version on crates.io is newer than the version in the repo
• The dependencies on crates.io are unknown

This library has recently been bumped to 0.1.0, so we need the analysis.

The extract list contains the packages in the ecosystem. It would be nice if you could print out a todo-list.

Sometimes multiple packages belong to the same repo, and in such cases it is desirable to group the updates together.

• Usually, external libraries depended on with higher order than an internal core library does not need update.
• External libraries with lower order does not get in the way for grouping.
• Reordering within same level does not affect grouping, but multiple packages in the same repo usually has incremental order, so naive reordering does not work.

Some Cargo.toml files contains single line comments start with "#".

Can use a similar trick by replacing whitespace rules with a custom rule supporting comments, see https://github.com/bvssvni/math_notation/blob/master/assets/syntax.txt.

See #19

This algorithm is a bit tricky and requires some planning:

• Solve by computing a version per package and its dependencies.
• The version of a dependency might depend on the computed version of another package.
• Even if the dependency is not among the listed packages, one would like the newest version to avoid dependency conflicts.
• When the computed version is the same as the old version, the package/dependency does not need to update.
• When any dependency needs update, the computed version satisfies all constraints, because it only matters that at least one dependency needs update, while multiple can be ignored.
• Can initialize a HashMap storing new versions per library, then walk through ordered by dependency depth and detect whether any dependency needs update.
• The version used as dependency is always same or older than package version, so this is automatically accounted for in listed libraries.
• Before detecting whether a dependency needs update, could add all dependencies, listed or not and store the latest version used as dependency.

Suggestion:

• Create a HashMap named new_versions.
• For each dependency, put the latest version used in new_versions.
• For each package, put the package version in new_versions (should be newer or equal).
• For each package sorted by dependency depth, detect whether any dependency needs update and increment the version in new_versions if it does.

Proof

These are the constraints satisfied by the suggested algorithm. The notation used is PistonDevelopers/piston#736.

Some variables are treated as functions of time X(t), where the final result is t when all constraints are satisfied. This makes it easier to describe properties that hold for some but not all variables. When t is not specified it means it is equal on both sides.

any_dependency_needs_update(library) -> bool

if any_dependency_needs_update([:] X(t)) {
  any_dependency_needs_update([:] X(t + 1))
}

old_version(library) -> version

new_version(library) -> version
// Start with new version set to old version.
([:] X(0)) -> old_version([:] X(0))

new_version([:] X(t + 1)) = max(
  if new_version([:] X(t)) = old_version([:] X(t))
   & any_dependency_needs_update([:] X(t)) {
    increment(new_version([:] X(t)))
  } else {
    new_version([:] X(t))
  },
  max_version_as_dependency([:] X(t))
)

increment(version) -> version

increment([:] X) > [:] X

max_version_as_dependency(library) -> version

max_version_as_dependency([:] X) <= old_version([:] X)

needs_update(library) -> bool
([:] X) -> old_version([:] X) < new_version([:] X)
([any_dependency_needs_update] [:] true) -> [:] true

The algorithm that generates the update info might overflow when dependencies are circular. This will happen in the depth_of function. Currently this is invalid input.

The elmesque library uses piston_window as dev-dependency, which is listed for update.

There are more libraries to add.

• image
• hematite_nbt
• opengex
• dev_menu
• gfx_debug_draw
• skeletal_animation
• hematite_server
• gfx_text
• asset_store
• current
• wavefront_obj
• png
• mush
• gif
• button_controller
• gfx_window_shared
• color_quant
• piston-shaders
• piston-shaders_graphics2d
• piston-float
• dual_quaternion
• texture_packer
• table
• shapes
• array
• select_color
• music

See #1

fn extract_dependency_info_from(extract_info: &str) -> Result<String, String>;

Move from the extract module to dependencies::write.

The dependencies information is generated from the extract module, but will be needed in the update module. Instead of using two different structures it seems to make sense to use one structure shared between the two modules. This could be added to its own dependencies module.

• Create new dependencies module
• Move extract::Package to dependencies::Package
• Move extract::Dependency to dependencies::Dependency

See #1

This should follow JSON syntax like the other custom formats.

• An order starting at 0 that tells which library should be updated before others. Multiple libraries can have same order, and those can be updated in parallel.
• Bump information containing both old and new version. This can be used to see whether it is a minor update or a breaking change, and also to detect whether the update info is outdated.

Example:

{
    "piston2d-gfx_graphics": {
        "order": 0,
        "bump": {
            "old": "0.4.0",
            "new": "0.5.0"
        },
        "dependencies": {
            "piston2d-graphics": {
                "bump": {
                    "old": "0.4.0",
                    "new": "0.5.0"
                }
            }
        }
    }
}

See #1

• Add dependency on semver
• Add update module
• Add Bump struct
• Add Dependency struct
• Add Package struct
• Convert meta data from dependency info to Vec<Package>
• Parse and convert dependency info
• Compute depth of dependencies for each package
• Detect libraries that need to be updated
• Incrementing semver (increase first non-zero number)
• Print out update info

Depth is given by if dependencies.len() == 0 { 0 } else { max_depth(dependencies) + 1 }

Currently it is not possible to simulate breaking changes because override-version property in the extract format is ignored.

An external library that depends on a listed library will be assigned order 0. This leads to wrong update order. It is easy to fix, by adding the library to the extract list. Opened this issue to figure out whether something should be done to prevent this.

• Crater - Testing builds of the Rust compiler against massive number of crates.
• Cargo-Crusader - Testing impact on downstream crates before publishing to crates.io
• MultiRust - Managing multiple Rust installations
• Cargo-Outdated - See when new versions of dependencies are available
• Cargo-Dot - Generate graphs of Cargo's dependencies

In the "freetype" example in the piston-examples repo, the new version was older than the old version:

"find_folder": {
  "bump": {
    "old": "0.2.0",
    "new": "0.1.1"
  }
}

Not sure right place to ask this and here seems as good as any.

Looking at a selection of Cargo.toml files from across piston, they all appear to have declared specific versions of their dependencies. This is weird to me - I want to push out a new version of sdl2_window that adds the new joystick events (so depends on pistoncore-input 0.7), but seems like I'll have to update virtually every project that depends on input since they're tied to 0.6. It seems that using more flexible dependencies (~> 0.7.0) would make updates much easier.

1. Maybe this "just works" with rust and I'm being too paranoid?
2. Perhaps this is deliberate pre-1.0 to ensure everything is always on latest?
3. (not totally related) It's not obvious to me how to use eco's output to sequence a release of the new sdl2_window. Do I have to do them all in order? There are a lot...

These are used by the asset_store library.

See #1

Could use JSON syntax for compability with other tools.

A simple way to start is to point directly to the Cargo.toml:

pistoncore-input {
    "url": "https://raw.githubusercontent.com/PistonDevelopers/piston/master/src/input/Cargo.toml"
}

pistoncore-window {
    "url": "https://raw.githubusercontent.com/PistonDevelopers/piston/master/src/window/Cargo.toml"
}

pistoncore-event_loop {
    "url": "https://raw.githubusercontent.com/PistonDevelopers/piston/master/src/event_loop/Cargo.toml"
}

piston {
    "url": "https://raw.githubusercontent.com/PistonDevelopers/piston/master/Cargo.toml"
}

According to https://docs.npmjs.com/misc/semver when a dependency starts with >=, breaking changes are allowed. For example, >=1.2.7 matches 2.5.3. It means almost the same thing as * except that a minimum version is required.

These dependencies can cause conflicts when the minimum version is newer than the other dependency versions. However, the update info should not be generated for such dependencies.

• Ignore >= when parsing version, such that the version information can be used.
• Use the version information to compute new version.
• Do not generate update information for dependencies starting with >=.

The syntax of all custom formats documents should be a subset of JSON. Adding a JSON meta syntax will make it easy to check this in the unit tests.

Currently it fetches them one by one, which is slow.

Dependencies with a "*" requirement accepts any version. It is not possible to derive any version information from it.

The interpretation of "*" is "breaking changes are allowed for this version". If a library has a such dependency there must be at least one other dependency that needs update in order to increment the version for the library.

When computing new versions for libraries, dependencies with "*" could be ignored, and no update information will be generated for such dependencies.

Currently it uses stderr_unwrap which will print to standard error without returning the error message. The problem is that the message does not contain any information about the url or package name.

Announcement: https://users.rust-lang.org/t/the-libc-crate-is-now-at-v0-2-0/3509

This is for tracking issues with upgrading libc to 0.2.

• rand rust-random/rand#81 rust-random/rand#83
• freetype-sys uses libz-sys which depends on libc 0.1. rust-lang/libz-sys#2
• glutin
• sdl2-sys
• sdl2
• glfw

Second phase:

• gl brendanzab/gl-rs#366 (fixed, but not published)
• glium glium/glium#1348

The major usage of this tool will be to check if libraries are updated and what versions they should have when some of the dependencies have a breaking change.

An idea is to use piston_meta to decouple some steps, such that the data can be generated and tested in various ways:

• A custom format for extract info (libraries to extract information from)
• An algorithm for extracting information and generating data for versions and dependencies
• A custom format for dependency info (describes all we need to know about the ecosystem)
• An algorithm for generating update info
• A custom format for update info (libraries that need to be updated and new version number)

There are 3 different custom formats:

1. extract_info which tells where to get dependency_info
2. dependency_info contains all information required to reason about the ecosystem
3. update_info contains actions that needs to be done

extract_info_from(extract_info) -> dependency_info
update_info_from(dependency_info) -> update_info

Some points:

• What is the motivation?
• What does the tool currently do?
• Link to other tools that helps with the ecosystem

Get rid of the unwraps.

Because of #17, it is necessary to know the dependencies not listed in the extract info to resolve dependency version conflicts.

When a library gets deprecated and replaced with another, it is currently not possible to discover which libraries that needs a replace.

• Requires a replace operation.

Depends on PistonDevelopers/gfx_window_shared#7.