apache / incubator-pekko Goto Github PK

The query view implementation we use need access to certain Actor internals and therefore has to reside within the akka/pekko package.
As we want to use our persistence serializer for snapshots, this normally generates a warning which we want to ignore by using a whitelist to mark those serializers as accepted.

We need to produce a Pekko equivalent for samples or start guides and fix up the docs references to https://developer.lightbend.com/start/?group=

With #91 we have managed to replace the sbt-paradox-akka theme with the standard one from paradox however it was just done by manually putting in all of the sbt-paradox-* plugins. Ideally however we would like to create our own sbt-paradox-pekko sbt plugin akin to sbt-paradox-akka. This will become especially important when we actually want to create our own bespoke pekko doc theme which we will share within all of our pekko modules.

The only slightly complicating factor is that currently pekko core sbt build enforces JDK 1.8 (sbt doesn't even load unless SBT is its run with JDK 1.8 on the path). The problem here is due to the design of one of sbt-paradox's core dependencies parboiled, you can only use specific versions of parboiled on JDK 1.8 versus later. In summary, pekko core only only use an outdated set of sbt-paradox-* dependencies where as other pekko modules which are built with newer JDK's will only support the newer sbt-paradox-* dependencies.

Thankfully I have managed to resolve this issue, assuming you have sbt-paradox-pekko with the newer sbt-paradox-* dependencies you can do this in pekko core's build.sbt

// We have to deliberately use older versions of sbt-paradox because current Pekko sbt build
// only loads on JDK 1.8 so we need to bring in older versions of parboiled which support JDK 1.8
addSbtPlugin("org.apache.pekko" % "sbt-paradox-pekko" % "HEAD+20230107-1453" excludeAll(
  "com.lightbend.paradox","sbt-paradox",
  "com.lightbend.paradox" % "sbt-paradox-apidoc",
  "com.lightbend.paradox" % "sbt-paradox-project-info"
))
addSbtPlugin("com.lightbend.paradox" % "sbt-paradox" % "0.9.2" force())
addSbtPlugin("com.lightbend.paradox" % "sbt-paradox-apidoc" % "0.10.1" force())
addSbtPlugin("com.lightbend.paradox" % "sbt-paradox-project-info" % "2.0.0" force())

I already have a prototype sbt-paradox-pekko plugin locally, it doesn't implement any theme but at least it brings in the sbt-paradox-* dependencies and will allow us to enforce a consistent theme.

• Upload sbt-paradox-pekko sbt plugin from local git repo to a temporary org
• Create an INFRA ticket to setup an incubator-sbt-paradox-pekko git repo
• Publish the sbt-paradox-pekko sbt plugin to the apache maven repo so it can be used in pekko repos. Can also use this simple project as a test bed for #85

We need to produce a Pekko equivalent and fix up the docs references to https://developer.lightbend.com/guides/akka-quickstart-scala/

Likewise for akka-quickstart-java

If we don't produce new g8 templates for pekko, we need to ensure that the docs either remove the refs to the g8 templates or we keep the refs to the akka g8 templates but ensure the text about them is correct (including mentioning the need to migrate the code generated for the generated projects).

Currently, lazyFuture and lazySingle is implemented with lazySource which require a sub materialization, which can be avoid by composing or providing a dedicated graphstage.

The podling scan at https://incubator.apache.org/clutch/pekko.html currently reports a long list of errors. This seems to be because the project site at http://pekko.apache.org/ is currently missing.

• Creating the site is already covered by #99
• This issue covers following the guidance at https://infra.apache.org/project-site.html to ensure the necessary data is provided on the landing page (i.e., currently, docs/src/main/paradox/index.md

Currently the README and CONTRIBUTING files contain references to akka and lightbend. While it may be correct and proper to reference them as original sources for the code base, other uses seem inappropriate. This change should removed most of the akka and lightbend phrases from the documentation.

There are many classes that use Akka in their names in the public API—like AkkaException, AkkaClusterTypedSerializer, AkkaJacksonModule, etc.—as well as many tests that start with Akka. The Akka name should either renamed to Pekko or removed entirely. In several cases these names are used in configuration as well.

This will also complicate the code migration for Akka projects, so we should take note of which classes have changed names.

Please document the CLA requirement, and explain how to fulfil it.

CONTRIBUTING.md line 91 mentions "Sign the CLA", but I can't find the CLA. There should be a link to the CLA and details of how to sign it.

Is the required CLA https://www.apache.org/licenses/contributor-agreements.html ? If so, does that process have to be followed (i.e., send a signed PDF to Apache) or is there bot assistance?

Separately - how does the project validate that CLAs have been signed by all contributors?

Thanks.

• when we change the package names, then the mima binary compatibility checks will fail anyway
• we are also likely to start releasing as v1.0.0 meaning that the old Akka version numbers will be out of synch with Pekko's
• we will add back the checks after we get our first release out

We don't yet want to remove deprecated code. Not for v1.0.0 anyway.

The @deprecated annotations have Akka versions in them. We should probably fix the annotations to highlight that the version is an Akka version.

Classic remoting (old, hard to diagnose bugs, another axis in difficult remoting test suite)

Originally posted by @jrudolph in https://github.com/mdedetrich/akka-apache-project/discussions/8#discussioncomment-3904101

It's currently using Netty 3

There were probably be a number of iterations.

In projects/Publish.scala:

  override lazy val projectSettings = Seq(
    publishTo := Some(akkaPublishTo.value),
    publishRsyncHost := "[email protected]",
    credentials ++= akkaCredentials,
    organizationName := "Lightbend Inc.",
    organizationHomepage := Some(url("https://www.lightbend.com")),
    sonatypeProfileName := "com.typesafe",
    startYear := Some(2009),
    developers := List(
        Developer(
          "akka-contributors",
          "Akka Contributors",
          "[email protected]",
          url("https://github.com/akka/akka/graphs/contributors"))),
    publishMavenStyle := true,
    pomIncludeRepository := { x =>
      false
    },
    defaultPublishTo := target.value / "repository")

The copyright header checks will need to be changed to allow Lightbend/Typesafe licenses and Apache licenses. The Lightbend/Typesafe licenses will remain on the old files but new files should have Apache licenses.

Scala 2.12 support (e.g. ByteString has forked sources for 2.12 and 2.13 that lead to various bugs in the past), see also https://github.com/mdedetrich/akka-apache-project/discussions/27

Originally posted by @jrudolph in https://github.com/mdedetrich/akka-apache-project/discussions/8#discussioncomment-3904101

And now JDK 21 is released too.

In #67 we renamed all of the classes that contain Akka in the name to Pekko. We should prune these renamed classes and figure out which ones are relevant for the migration script.

If you look at any ASF jar, you will find these 2 files.

We should change our config loading and reference.conf to use pekko everywhere rather than akka. It looks like the config loading is mostly in the Settings class, but that also exposes the raw Config instance, which is used directly to load akka.* configs in other classes.

It makes sense to have a way to support the akka configuration prefix instead of or in addition to the pekko prefix, but I know there was some question about whether to do it for 1.0. Given that config loading is mostly centralized it should be relatively simple to do.

• sbt has good plugins for doing releases to Nexus - note ASF has its own Nexus instance, we don't use Sonatype. https://repository.apache.org/
• also need to gpg sign and generate SHA256/SHA512 digests for the source and binary distributions (#78)

Update README.md and CONTRIBUTING.md to clearly describe the current state of the project and how best to contribute to it.

We want to encourage people to help get Pekko to v1.0, and ensure it's clear that this is being actively worked on. We don't want to discourage people with inaccurate docs, or by appearing to claim Pekko is ready for use now when it's not.

I'm keen to hear other folks' thoughts, but here are some things I think we could do to improve the situation:

• Near the top of the README, say "Pekko is part of the Apache Incubator program and we are actively working towards a v1.0 release. It's not yet ready for production use, but please get involved with the community and contribute to the effort."
• In CONTRIBUTING (or possibly even README), add a section explaining that we are working on incubator-pekko first and not touching any of the other repos. This is quite important and not immediately obvious.
• In CONTRIBUTING, remove (or demote) the section that lists how Akka used to use tags (labels). We aren't using these yet. If we are using labels etc, explain that instead.
• Agree on label/milestone usage: I can see we're using milestone:1.1.0 for things that don't belong in the initial release; I think we should use blocked for issues that are blocked, e.g., #101. I wonder if there's a way we can label priorities or classes of issues to make the list more manageable?
• In CONTRIBUTING, add some kind of table of contents - otherwise it's not obvious that build instructions are down below.

I also think we should try to get the complete list of things needed to reach v1.0 raised as issues, rather than relying on memory and mailing lists. That way we have an accurate work list, and we can tell folks "check GitHub for the current list of open issues, read the discussion, and dive in".

Feedback welcome - I'm new to this project and am probably missing important context. Thanks!

As discussed in #104 (comment), due to a current limitation in sbt, when doing a cross publish amongst Scala versions using the + prefix it generates a separate timestamp for each Scala version where as ideally we should have a unique timestamp for an entire single publish operation.

A workaround as described in https://stackoverflow.com/a/21062454/1519631 is to pre-save the unique timestamp to a file and then read that timestamp along with a custom command alias.

Create a migration guide explaining to a user of Akka how to migrate their project to Pekko.

We hope to automate as much of this as possible under #22 but first we need to determine the steps and lay them out so early adopters can follow them and gain experience.

This should be documented in docs/src/main/paradox/project/migration-guide-akka-pekko.md, linked from migration-guides.md in the same location, and also linked prominently from the GitHub homepage README.md and the project homepage docs/src/main/paradox/index.md.

Steps include:

• adapting Akka dependencies "com.typesafe.akka" -> "org.apache.pekko"
• adapting JAR names akka-actor_2.13.jar -> pekko-actor_2.13.jar
• adapting imports import akka.actor -> import org.apache.pekko; import pekko.actor - see #58
• adapting class names Akka* to Pekko* - see #67
• adapting packages and corresponding paths akka -> org.apache.pekko, if the codebase inserts code into the Akka namespace
• updating third-party dependencies to versions that depend on Pekko
• adapting configuration to use the pekko key - see #63
• discussion of how to approach Akka remote applications - see #108

Under ASF guidelines, projects deliver source releases. Binaries like jars are a convenience.

See https://logging.apache.org/log4j/2.x/download.html for an idea of what the official downloads are meant to look like.

We will deliver jars to Maven Central too - but we also need to these source and binary distributions - typically in tgz and zip formats.

https://github.com/apache/daffodil appears to use sbt-native-packager plugin to create these. Release are downloaded from https://daffodil.apache.org/releases/3.4.0/

https://github.com/apache/incubator-nlpcraft also has a sbt build (not many ASF projects do) - but it also has a maven build and it isn't immediately clear how they package their distros.

There are other sbt plugins that may be a better fit - https://index.scala-lang.org/xerial/sbt-pack is one worth looking at

The source archive will probably just be a snapshot of the files in the git repo at release time - possibly with some stuff pruned. We should probably have a separate markdown file that describes the sbt commands to build the jars and docs.

The binary archive is a bit more problematic. For an app like Apache Spark, the bin archive provides a 1 stop solution for getting all the jars and the shell scripts to run spark-submit, the Spark REPL, etc.
A Pekko binary archive would include all the Pekko jars built for this Git repo and all the dependency jars. We should probably aim to have one or more sample apps and the shell (or sbt) scripting necessary to run those sample apps. These sample apps can be very simple examples that just prove that the jars all work together.

References

• https://infra.apache.org/release-distribution

In #60 we managed to get nightly builds published to the Apache directory. While this is the official place to publish Apache nightlies, its technically not a maven repository (its just a directory where you rsync the contents).

Apache does have an official maven where you can publish snapshots too (see https://repository.apache.org/content/groups/snapshots/, https://repository.apache.org/ and https://infra.apache.org/publishing-maven-artifacts.html), so in addition to rsyncing our nightlies we should also publish this as an a snapshot to Apache's maven repo.

aim is that all the akka-* jars will become pekko-* jars

As a quick way to remove the akka branding, lets replace akka-paradox plugin with paradox basic theme. We can make better with more Pekko branding later.

As discussed in #28 with @nvollmar

Using this issue to gather some feedback before posting the proposal to the Apache legal JIRA

Current proposal (will be updated, based on feedback)

/*
 * Copyright (C) 2009-2022 Lightbend Inc. <https://www.lightbend.com>
 *
 * Updates since 2022-10-24 are licensed to the Apache Software Foundation (ASF) under
 * one or more contributor license agreements.  See the NOTICE file distributed with this
 * work for additional information regarding copyright ownership.
 *
 * Lightbend and ASF license this file to you under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software distributed under
 * the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
 * ANY KIND, either express or implied.  See the License for the specific language
 * governing permissions and limitations under the License.
 */

Next to including the Lightbend notice, changes form the default header text are

Updates since 2022-10-24 are 

Lightbend and ASF license

So I noticed that when publishing Pekko we are doing so using JDK 11. While there is no problem with this, at the same time the current sbt build will refuse to load with a JDK that is newer than 1.8.

We should either pubilsh using JDK 1.8 (at least for the 1.0.x branch) or just remove the sbt check?

Artery UDP based on Aeron (hard to deploy, little used, one extra dimension in hard to stabilize remoting (and everything above like cluster, persistence etc) tests)

Originally posted by @jrudolph in https://github.com/mdedetrich/akka-apache-project/discussions/8#discussioncomment-3904101

The design blog: https://akka.io/blog/article/2016/12/02/streams-in-artery & https://akka.io/blog/article/2016/12/05/aeron-in-artery

Apache has certain requirements for releasing a public version. We should strive for as much automation as possible. This ticket should be an overview over all steps necessary and the progress on those items. Please add items as needed.

Let's only consider public releases like RCs and GAs here (but not snapshots).

(Please add your name and/or PRs and issues to the items as needed)

References:

• Apache Release Management
• Apache Release Distribution Policy
• Apache Publishing Maven Artifacts
• Apache Release Signing

Organizational steps

• Create release template to keep track progress of release process per release (see the old template at https://github.com/apache/incubator-pekko/blob/main/scripts/release-train-issue-template.md)
• Setup release managers (Who will it be?)

Maven-related steps

• Publish to Apache Nexus into staging
• Initiate voting
• Propagate to Maven Central when vote is positive
• ...

Apache-related steps

• (one-time) setup signing infrastructure (how are keys created, where can a release be signed)
• Create signed release and upload artifacts (where and when does that happen? Can signing keys be safely used from GHA?)
• ...

In this PR #53 (comment) to get CI running we ended up having to use sbtx instead of plain sbt due to having to upgrade a github action. It turns out that the change to sbtx was that in the older versions of the github action, it was aliasing sbt to sbtx (see https://discord.com/channels/632150470000902164/922600050989875282/1042858091387043872). We should investigate into using pure sbt rather than sbtx since sbtx isn't really supported that much anymore

https://github.com/apache/incubator-pekko/actions/runs/3898732569

Will look into it soon, on meeting

Currently its possible to merge a PR even if the status checks fail or even in the small span of time before they get triggered. Now that we have CI running properly, we should update the Require status checks to pass before merging under Branch Protection Rules for the main branch so that people cannot easily merge PR's if one of the status checks fail.

@pjfanning I am not sure if this is to be done directly under github actions or whether it needs to be organised with an INFRA ticket/.asf.yaml?

Remove any remaining references to Lightbend or Akka in the documentation (including logos)

Remove external links to Akka books, blog articles, etc.

While looking at #103 I noticed that the pekko project already happens to have infrastructure setup to publish nightlies (aka snapshots) to an rsync directory in the exact same manner that was implemented in #60.

This is done by using https://github.com/lightbend/sbt-publish-rsync and you can see the details at https://github.com/apache/incubator-pekko/blob/main/project/Publish.scala.

We should either investigate using the sbt-publish-rsync plugin or remove it entirely from pekko since https://github.com/apache/incubator-pekko/blob/main/.github/actions/sync-nightlies/action.yml exists. My own personal preference would be to use sbt-publish-rsync since its far simpler (due to it being done by sbt the command to publish would be quite trivial especially when combined with publishing to Maven snapshots directory, see #103) and I like reducing the direct dependance on shell scripts. From a cursory glance of looking at https://github.com/lightbend/sbt-publish-rsync/blob/master/src/main/scala/com/lightbend/sbt/publishrsync/PublishRsyncPlugin.scala#L45-L49 it shouldn't be problematic however there is a risk that changes might need to be done to sbt-publish-rsync plugin for Apache's use, if needed this does present an opportunity to create an sbt-publish-rsync-apache plugin.

@seglo @pjfanning Thoughts?

There is a performance issue for Java users when using jackson-module-scala v2.13.x. Not too many users have reported issues but it might be best to update any v2.13 dependency to v2.14.1 (or newer version). We can keep the v2.11.4 dependency for Scala 2.x. We can upgrade to jackson v2.14.x for all Scala versions after Pekko's initial release.

FasterXML/jackson-module-scala#576

Upgrade Jackson dependency to 2.13.4 for
1: CVE-2020-36518
2: CVE-2022-42004
3: CVE-2022-42003

We still have the old Akka for templates when issues/pull requests are made (see https://github.com/apache/incubator-pekko/tree/main/.github/ISSUE_TEMPLATE), should remove/replace them as necessary.

The github rules should not allow merging of pull requests when there are

1. open changes requested.
2. ongoing conversations (not resolved conversations).

I am not certain how to configure this on the Apache git system.

As discussed in https://github.com/mdedetrich/akka-apache-project/discussions/28, pekko core will need to handle wire incompatibility issues arising from the change in the akka:// address prefix to pekko://. There may also be other wire incompatibility changes, for example if we decide to change the default 2552 port and anything else that we may have missed.

The solution will also need to play with the release strategy that we decide for pekko. Since the previous discussion I have done some thinking about it and at least to me the most amicable solution would be to make these fields configurable with typesafe config with different default values depend on the pekko release branch, i.e.

For pekko 1.0.x we would have

acceptPrefix = ["akka", "pekko"] // The prefix's to use when checking against incoming requests
sendPrefix = "pekko" // The prefix to use when making send requests

And for pekko 1.1.x (on the assumption that we don't want to accept migrating current Akka clusters to this Pekko branch by default)

acceptPrefix = ["pekko"] // The prefix's to use when checking against incoming requests
sendPrefix = "pekko" // The prefix to use when making send requests

(Note that the same strategy can also be applied to the port 2552`.

This means that when doing a rolling upgrade from an existing akka cluster to pekko 1.0.x you would temporarily change sendPrefix to "akka" (so that currently existing Akka clusters don't refuse connections from new pekko nodes) and once the rolling upgrade is complete then you would change sendPrefix to "pekko". After that point you are then free to upgrade to pekko 1.1.x with acceptPrefix as ["pekko"] without any problems and making this configurable with typesafe config means that if users have bespoke update requirements its relatively easy for them to handle it.

This also leaves the door open to first deprecating the acceptPrefix/sendPrefix settings later down the Pekko 1.1.x release cycle and then at some point in the future hardcoding it as pekko.

We should provide some automated migration that users of akka can use that adapts their codebase to use pekko.

This migration should automatically

• adapt the dependencies
  • in sbt
  • in maven
• adapt the codebase
  • imports in scala
  • imports in java

These docs are Lightbend specific.

https://lists.apache.org/thread/8orjtlx2w42y3j45pswmf9nbtf64gp77

Change to

• Scala 2.3.10 due to CVE - https://www.cvedetails.com/cve/CVE-2022-36944/
• Scala 3.2.0 -- Scala team recommend using latest Scala 3 version - https://www.scala-lang.org/blog/2022/08/17/long-term-compatibility-plans.html
• Scala 3.3.0 - the first LTS version of Scala 3 should be out soon

Now that we have github discussions enabled in incubator pekko, we should try to see if its possible to migrate relevant discussions from https://github.com/mdedetrich/akka-apache-project/discussions to https://github.com/apache/incubator-pekko/discussions.

On the right hand side there is a "Transfer this Discussion" link, I tried doing it myself but since I don't have write permissions to this repo I cannot move it. @pjfanning I just added you as a collaborator to https://github.com/mdedetrich/akka-apache-project, let me know if this ends up working out. If it doesn't I think we will just have to start from scratch again.

I think only Akka is there, but not pekko.

Which will help reviewing.
demo: real-logic/agrona#268
https://codeql.github.com/docs/codeql-overview/supported-languages-and-frameworks/

We need to rename the package from akka to either pekko or org.apache.pekko (this is still being decided). One thing to note is that this will break some tests, i.e. https://github.com/apache/incubator-pekko/blob/6680c47dcc2305906a44d7794081682211d7ee0b/akka-actor-tests/src/test/scala/akka/util/ByteStringSpec.scala#L983-L995 is dependant on the fully qualified class name so it will also need to be updated when the package changes (there may be other examples of this as well).

Building the project on windows using sbt compile fails with the following exception:

[info] welcome to sbt 1.7.1 (Temurin Java 1.8.0_332) [info] loading settings for project incubator-pekko-build from plugins.sbt ... [info] loading project definition from C:\Entwicklung\incubator-pekko\project [info] loading settings for project pekko from build.sbt ... [info] resolving key references (62652 settings) ... java.util.regex.PatternSyntaxException: Unexpected internal error near index 1 \ at java.util.regex.Pattern.error(Pattern.java:1969) at java.util.regex.Pattern.compile(Pattern.java:1708) at java.util.regex.Pattern.<init>(Pattern.java:1352) at java.util.regex.Pattern.compile(Pattern.java:1028) at java.lang.String.split(String.java:2380) at java.lang.String.split(String.java:2422) at org.apache.pekko.PekkoValidatePullRequest$$anonfun$$nestedInanonfun$buildSettings$1$1.applyOrElse(ValidatePullRequest.scala:49) at org.apache.pekko.PekkoValidatePullRequest$$anonfun$$nestedInanonfun$buildSettings$1$1.applyOrElse(ValidatePullRequest.scala:47) at scala.PartialFunction.$anonfun$runWith$1$adapted(PartialFunction.scala:145) at scala.collection.Iterator.foreach(Iterator.scala:943) at scala.collection.Iterator.foreach$(Iterator.scala:943) at scala.collection.AbstractIterator.foreach(Iterator.scala:1431) at scala.collection.IterableLike.foreach(IterableLike.scala:74) at scala.collection.IterableLike.foreach$(IterableLike.scala:73) at scala.collection.AbstractIterable.foreach(Iterable.scala:56) at scala.collection.TraversableLike.collect(TraversableLike.scala:407) at scala.collection.TraversableLike.collect$(TraversableLike.scala:405) at scala.collection.AbstractTraversable.collect(Traversable.scala:108) at org.apache.pekko.PekkoValidatePullRequest$.$anonfun$buildSettings$1(ValidatePullRequest.scala:47)

The problem is caused by using the file.separator system property as regular expression in ValidatePullRequest.scala:49. On Windows this is the backslash which is not a valid regular expression.

The problem can be fixed by using a hardcoded slash as splitting regular expression. This also works on windows.

https://github.com/apache/incubator-pekko/blob/main/cluster/jmx-client/pekko-cluster

• Not just superficial println text
• also JMX and other akka refs in the script

ManifestInfo(this).checkSameVersion("Akka", allModules, logWarning = true)

Need to work out what to replace this with

part of #16