scala-steward-org / scala-steward Goto Github PK

https://twitter.com/phdoerfler/status/1039789583666487296

see typelevel/cats#2499

We could group updates with the same groupId and the same currentVersion and newerVersions as updates that need to be applied in lockstep.

Are there facilities in scala-steward to configure version ranges?

For example lightbend/ssl-config#133 updates joda-time from 2.9.9 to 2.10. Now, leaving aside that right this moment I don't know if that's a safe update for ssl-config, assuming we can't do that upgrade how do I inform scala-steward that for the moment we want only updates in the 2.9.x version series?

(Thanks for the bot, looks like a great addition to the ecosystem!)

I'd like to use this on my company's codebase to help save maintenance time. Is that possible?

We have a custom sbt plugin that is hosted at the companies artifactory. I think the resolver that is defined in plugins.sbt is not being used when checking updates for this plugin.

Our plugins.sbt looks like this:

externalResolvers := Seq("Company Repo" at "https://repo.company.io/artifactory/")

addSbtPlugin("io.company" % "company-plugin" % "0.1.15")

The error we’re getting when running Scala Steward

[info] Loading global plugins from /root/.sbt/1.0/plugins
[info] Loading settings for project updates-build from plugins.sbt ...
[info] Loading project definition from /home/jenkins/workspace/updates/project
[info] Updating ProjectRef(uri("file:/home/jenkins/workspace/updates/project/"), "updates-build")...
[warn]     module not found: io.company#company-plugin;0.1.15
[warn] ==== typesafe-ivy-releases: tried
[warn]   https://repo.typesafe.com/typesafe/ivy-releases/io.company/company-plugin/scala_2.12/sbt_1.0/0.1.15/ivys/ivy.xml
[warn] ==== sbt-plugin-releases: tried
[warn]   https://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/io.company/company-plugin/scala_2.12/sbt_1.0/0.1.15/ivys/ivy.xml
[warn] ==== local: tried
[warn]   /root/.ivy2/local/io.company/company-plugin/scala_2.12/sbt_1.0/0.1.15/ivys/ivy.xml
[warn] ==== public: tried
[warn]   https://repo1.maven.org/maven2/io/company/company-plugin_2.12_1.0/0.1.15/company-plugin-0.1.15.pom
[warn] ==== local-preloaded-ivy: tried
[warn]   /root/.sbt/preloaded/io.company/company-plugin/0.1.15/ivys/ivy.xml
[warn] ==== local-preloaded: tried
[warn]   file:////root/.sbt/preloaded/io/company/company-plugin_2.12_1.0/0.1.15/company-plugin-0.1.15.pom
[warn]     ::::::::::::::::::::::::::::::::::::::::::::::
[warn]     ::          UNRESOLVED DEPENDENCIES         ::
[warn]     ::::::::::::::::::::::::::::::::::::::::::::::
[warn]     :: io.company#company-plugin;0.1.15: not found
[warn]     ::::::::::::::::::::::::::::::::::::::::::::::

And thanks for the awesome project. We're running it successful in production for a few weeks on 10+ repos!

We'd like to leverage this awesome project to update the dependencies on our internal project dependencies. To do that, scala-steward must support GitLab. I've started to work on a native implementation.

I'm opening this MR just to leave a trace of that on the internet. That way, if someone else planned/was planning or is already doing it, we can coordinate better.

Thanks again for the project, it is really a breeze to work it.

This has gone terribly wrong: lightbend/paradox#263

... instead of regular expressions.

I didn't fully understand how caseapp handles Boolean when I named the config as execSandbox and set the default to true. This issue is to rename the config name from execSandbox to disableSandbox.

This PR WellFactored/clovis#15 updates the version number of enumeratum-circe by changing the value of enumeratumVersion from 1.5.13 to 1.5.19. However, that val is also used to define the version of the core enumeratum library, which does not have a 1.5.19 version, so the subsequent build breaks.

Is there a way to integrate scala-steward with scalafix, so that when a library maintainer supplied a scalafix migration from one version to another (such as the one in the works for fs2 0.10 to 1.0), that steward can automatically apply it in the PR along with the build change?

see lightbend/mima#242

https://github.com/erikvanoosten/metrics-scala/pull/108/files

If a repository has more than n (say 30) open pull requests by Scala steward it makes sense to ignore this repository until the open PR count drops below the threshold n. Many open pull requests are a sign that a repository is abandoned and working on that is just a waste of resources.

Scala steward should also propose sbt updates and not only library and plugin upates.

See circe/circe-benchmarks#28

It would be nice for scala-steward to be able to work with mill and maybe other build tools as well. As I understand this requires three components:

1. read dependencies from build
2. rewrite dependencies in build file
3. find newer dependencies

I believe the 3) could be shared between build tools, but I dont know hard would that be.
It would be nice to learn how hard 1) and 2) could be.

I assume 1) is as simple as executing proper mill command and parsing the results.
2) sounds like could be done with scalafix, but I dont know it well enough to say something for sure.

GUI git clients that show gravatars for commits would look even better with a Gravatar for scala-steward :)

... in a perfect world to its release notes.

https://developer.github.com/enterprise/2.15/v3/enterprise-admin/#endpoint-urls

REST API endpoints—except Management Console API endpoints—are prefixed with the following URL:

http(s)://hostname/api/v3/

https://github.com/fthomas/scala-steward/blob/771cf5f9ac4632d1adf4e5fdba478a301944e026/modules/core/src/main/scala/eu/timepit/scalasteward/github/url.scala#L23-L24

Hi @fthomas, I've discovered that firejail doesn't work in Docker:

$ firejail echo hello
Warning: an existing sandbox was detected. echo will run without any additional sandboxing features

Running it with the --force flag, which is supposed to disable the PID namespace checking, results in the following error:

$ firejail --force echo hello
Error clone: main.c:2519 main: Operation not permitted

Additionally, the --force flag was removed in firejail 0.9.54 and apparently it is not possible to run firejail in Docker anymore (I asked about this recently netblue30/firejail#2579).

I think we should remove firejail from scala-steward docker image and force it to run without the sandbox when the command isn't found or make it explicit in the docs that the sandboxing features don't work when running scala-steward in Docker.

So that we don't hit the rate limit, see https://developer.github.com/v3/#rate-limiting

Abstract:
When a ModuleID is written with version detached into another variable, scala-steward seems to be unable update the version:

val cats_effect_version = "1.1.0"
val cats_effect = "org.typelevel" %% "cats-effect" % cats_effect_version

Detail:
See https://github.com/pshirshov/izumi-r2/blob/develop/sbt/sbt-izumi-deps/src/main/scala/com/github/pshirshov/izumi/sbt/deps/IzumiDeps.scala#L21

After importing the Izumi Project here, we've got two PRs from scala-steward: 7mind/izumi#460 7mind/izumi#461 both only concern ModuleID's that were written all at once while the large IzumiDeps object above was not processed. (Note, there's another bad case in IzumiDeps - constructing ModuleIDs with .map - however, this case likely can't be handled sanely, so we'll just flattenize those.)

ProcessAlg.execSandboxed is an alternative to ProcessAlg.exec that runs programs in a sandbox. If one fully trusts the repos scala-steward is working on execSandboxed could be just an alias for ProcessAlg.exec. It would be great to have a config option for this case.

Since #197 is merged, scala-steward can work on any repo regardless of the git workflow an organization chooses to use. So, companies now have the option of using it on their internal private repos. This issue is to see if there is any interest in adding a list of companies that are using scala-steward somewhere in the repo to be well informed of the community that is benefiting from scala-steward.

Something similar to https://github.com/circe/circe#community

@fthomas Thoughts?

See https://github.com/pdalpra/gh4s/pull/15 for an example.

see https://github.com/maximn/TaxCalculator/pull/1/files

The diff is:

- libraryDependencies ++= Seq("org.specs2" %% "specs2-core" % "3.+" % "test")
+ libraryDependencies ++= Seq("org.specs2" %% "specs2-core" % "4.3.4 

see plokhotnyuk/jsoniter-scala#180

As suggested by @fwbrasil in zio/zio-quill#1195 (comment).

Currently scala-steward checks every repository for new dependencies on every run. Checking for updates is a costly operation since it means cloning the repository, starting sbt, and checking for library and plugin updates. Even for small repositories which are up-to-date this process takes roughly a minute on my server. This is not viable if scala-steward should be able to handle 1000+ repositories which I very much like to achieve.

My current idea to reduce the overall workload is the following:

I propose to skip checking repositories where scala-steward would not open new PRs or update existing PRs. This would require to persist some infos about the repos scala-steward has worked on. E.g if we know of each repo

• the current SHA1 of the base branch
• the current list of library and plugin dependencies
• all PRs scala-steward has created

we should be able to decide if we need to process a repository without cloning it and running sbt dependencyUpdates. The SHA1 can be used check if the upstream repository has changed and our existing PRs need to be updated or if the list of dependencies has changed.
Before working on any repository we would need to check if any of the libraries or plugins scala-steward knows about has a newer version and then only process repositories whose dependencies have updates.

See coursier/coursier#1081

see scala/scala-xml#274

steps

lightbend/paradox#268

- addSbtPlugin("com.geirsson"          % "sbt-ci-release"  % "1.2.1")
+ addSbtPlugin("com.geirsson"          % "sbt-ci-release"  % "1.2.5")

problem

Not sure why sbt-ci-release change was made here.

expectation

sbt version update should be done using project/build.properties.

See rchain/rchain#1880 (comment).

FilterAlg currently discards updates where nextVersions.head is a known bad version. Here are some examples. Instead of discarding them, we could also just remove the bad version in nextVersions and update to the new nextVersions.head. If nextVersions only contains one element, we would again discard that update.

It looks like scala-steward doesn't try to update the scala version.

For example in the build.sbt of metrics-scala we have: scalaVersion := "2.12.6" and crossScalaVersions := Seq("2.11.12", "2.12.6"). Version 2.12.7 is already out.

BenFradet/dashing#43

See circe/circe-iteratee#9

I have a project, which only has the SBT project layout at branch sbt, see here: https://github.com/ReactivePlatform/Pragmatic-Scala
, seems like steward needs an update to support this kind of usage.

Sometimes scala-steward updates version numbers of dependencies that are similar to the current update but not part of it. They are similar because these dependencies use the same version and their artifactIds have the same prefix. One example is http4s/http4s#2343.

We could fix such errors by trying to update version numbers in two steps. The first step would try to update only lines that contain the exact groupId, artifactId, and version. If this step changes any lines we're done. If it doesn't yield any changes we would use the current strategy for updating version numbers. In the case of http4s/http4s#2343 this strategy would have updated the version numbers of jawn-json4s and jawn-play but not of jawn-fs2 because it has a different groupId.

If a project uses multiple versions of the same dependency, Scala Steward does a bad job at handling them in a coordinated fashion which results in bad PRs like circe/circe-jackson#36 or scala/scala-collection-compat#432. This is a common situation for projects that support multiple Scala versions or when a project provides integration for multiple versions of another library (e.g. circe-jackson25, circe-jackson26, etc.).

see erikvanoosten/metrics-scala#118

cb372/scalacache#206

It tried to "update" from 3.0.0-RC2-d0feeba to 3.0.0-fbcb270, which is actually older.

To be honest I find this versioning scheme confusing myself, so I'm not surprised it got tripped up by it 😉

see zio/zio-quill#1200 (comment)

see coursier/sbt-coursier#19

Could we do that?

To identify the batch we could label all pull requests made by the bot by the current build number.
If all were successful and merged we could trigger somehow a release?

Challenges

• What if new features were introduced? Should we check the commits as well if they are only dependency update commits?
• A changelog should also be updated, this also ( needs to | could ) be automated

... instead of adding it to repos.md. I guess this requires making some kind of GitHub app for Scala steward. The only thing I'd need from GitHub would be a list of repos that have Scala steward "installed".

Hi @fthomas

Thanks for this great bot. I love it.

I have some interest in making scala-steward work for repositories that don't have the luxury of allowing forks and limits users to work with branches out of one main repository.

I tried to tweak scala-steward to my use-case and got it to work. (Needs some clean up)
ArulselvanMadhavan@18a3b11

If you are interested in adding this as a feature behind a config flag(or some other better option), let me know. I would like to contribute towards that effort.

Like the Play framework, which put the real sbt project in /framework folder.

scala-steward is capable of running in a project that uses maven for project definition by using https://github.com/sbt/sbt-pom-reader to start sbt and get dependencies. If this can't be done reliably, it may need to be opt-in on a project-by-project basis.

Once the dependencies are enumerated, applying a different set of heuristics to the sbt version replacements seems straightforward enough.