restatedev / e2e Goto Github PK
View Code? Open in Web Editor NEWE2E tests for Restate
License: MIT License
E2E tests for Restate
License: MIT License
JavaPersistenceAcrossRuntimeRestartsTest > startAndStopRuntime(CounterBlockingStub, ContainerHandle)
times out after 10 seconds: https://github.com/restatedev/e2e/actions/runs/5060557094/jobs/9083598421#step:6:658. It might be caused by too tight timeouts.
java.util.concurrent.TimeoutException: startAndStopRuntime(dev.restate.e2e.functions.counter.CounterGrpc$CounterBlockingStub, dev.restate.e2e.utils.ContainerHandle) timed out after 10 seconds
at org.junit.jupiter.engine.extension.TimeoutExceptionFactory.create(TimeoutExceptionFactory.java:29)
at org.junit.jupiter.engine.extension.SameThreadTimeoutInvocation.proceed(SameThreadTimeoutInvocation.java:58)
at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92)
at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:217)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:213)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:138)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:68)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:107)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.processAllTestClasses(JUnitPlatformTestClassProcessor.java:110)
at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.access$000(JUnitPlatformTestClassProcessor.java:90)
at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor.stop(JUnitPlatformTestClassProcessor.java:85)
at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.stop(SuiteTestClassProcessor.java:62)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
at com.sun.proxy.$Proxy2.stop(Unknown Source)
at org.gradle.api.internal.tasks.testing.worker.TestWorker$3.run(TestWorker.java:193)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:113)
at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:65)
at worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
at worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
At the moment for each test we log everything to stout/stderr, including the logs of the containers. We should rather collect in a more organized way the logs, to simplify the debugging. One idea is to push logs of containers to separate files, in the build/test-results
directory, without logging them to stderr
Add the ability to create typescript services to test along with other jdk based services. This task mostly requires setting up the build pipeline for building node services and containerize them. Perhaps this is useful: https://github.com/node-gradle/gradle-node-plugin/blob/master/docs/usage.md
With #32 we introduced that the testing task depends on jibDockerBuild
. In order to avoid pulling the base image/layers every time from Dockerhub, it would be great if jib could cache these layers and reuse them.
Enable JUnit 5 parallel testing, parallelizing on a class level, and allowing opt-in parallelization on a method basis.
Some prior work:
An e2e test run has timed out on main of restate: https://github.com/restatedev/restate/actions/runs/5324833472/jobs/9645302380. Has this run been extremely slow, are we setting the timeout too tight or is the variance of the verification test still too large?
Publish the test utils module, to be used in the factoring-demo repository
InvokeOrderingTest > ordering(boolean[], CoordinatorBlockingStub, ListServiceBlockingStub) > [1] [true, false, true] FAILED
io.grpc.StatusRuntimeException: UNKNOWN: Syscalls MUST be non-null. Make sure you're creating the RestateContext within the same thread/executor where the method handler is executed. Current thread: vert.x-worker-thread-4
at app//io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:271)
at app//io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:252)
at app//io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:165)
at app//dev.restate.e2e.services.coordinator.CoordinatorGrpc$CoordinatorBlockingStub.invokeSequentially(CoordinatorGrpc.java:457)
at app//dev.restate.e2e.runtime.InvokeOrderingTest.ordering(InvokeOrderingTest.kt:45)
InvokeOrderingTest > ordering(boolean[], CoordinatorBlockingStub, ListServiceBlockingStub) > [2] [false, true, false] FAILED
java.lang.AssertionError:
Expecting actual:
["0", "2", "0", "1", "2"]
to contain exactly (and in same order):
["0", "1", "2"]
but some elements were not expected:
["0", "2"]
at dev.restate.e2e.runtime.InvokeOrderingTest.ordering(InvokeOrderingTest.kt:52)
https://github.com/restatedev/restate/actions/runs/5289973587/jobs/9573676036?pr=512
In order to better support M1 architectures, we should automatically detect the host architecture and configure jib's platform accordingly.
We could think about improving the setup for local testing. At the moment we are always pulling images from ghcr.io and the java-sdk.jar
from the Github packages repository. When testing new things out, it would be great if one can build things locally and then the e2e
picks these things up from the local caches.
One solution for the image pull problem is to only enable pulling always if running on CI via introducing an env variable.
A potential solution for sharing java-sdk
artifacts with the e2e
project is to use Gradle's composite build feature. The problem I hit with this approach was that the java-sdk.jar
is produced by :java-sdk:sdk
and not the root project. This required the configuration of a substitution pattern that created a permanent coupling between the two projects.
An alternative solution for the java-sdk
build artifact problem could be the usage of mavenLocal
to share build artifacts. This is, however, discouraged by the Gradle folks: https://docs.gradle.org/current/userguide/declaring_repositories.html#sec:case-for-maven-local.
In order to run the e2e tests as part of PRs to the runtime or the Typescript SDK or as part of the release process, we need to create a GA workflow which can run with snapshot artifacts (e.g. specified via a revision). Once this is possible, we can integrate the e2e tests into the CI of the different repositories.
Currently the node services do not contain any tests for awakeable and completeAwakeable.
We can probably add a bit more context to what each e2e test tests (in the form of a descriptive test method or a comment providing context). This would help others to get more quickly onboarded into the e2e code base and we can more easily keep track which features are tested and which not.
This includes:
The verification test timed out after 1 minute: https://github.com/restatedev/e2e/actions/runs/5183061436/jobs/9340510065#step:9:2101.
java.util.concurrent.TimeoutException: stoppingTheRuntime(dev.restate.e2e.services.verification.verifier.CommandVerifierGrpc$CommandVerifierBlockingStub, dev.restate.e2e.utils.ContainerHandle) timed out after 1 minute
at org.junit.jupiter.engine.extension.TimeoutExceptionFactory.create(TimeoutExceptionFactory.java:29)
at org.junit.jupiter.engine.extension.SameThreadTimeoutInvocation.proceed(SameThreadTimeoutInvocation.java:58)
at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92)
at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:217)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:213)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:138)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:68)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:107)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.processAllTestClasses(JUnitPlatformTestClassProcessor.java:110)
at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.access$000(JUnitPlatformTestClassProcessor.java:90)
at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor.stop(JUnitPlatformTestClassProcessor.java:85)
at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.stop(SuiteTestClassProcessor.java:62)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
at com.sun.proxy.$Proxy2.stop(Unknown Source)
at org.gradle.api.internal.tasks.testing.worker.TestWorker$3.run(TestWorker.java:193)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:113)
at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:65)
at worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
at worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
Add a test using failpoints, where a failure is simulated after sending the proposal, but before committing the offset. See https://github.com/restatedev/runtime/pull/228 for more details
This is an umbrella issue capturing the missing e2e tests for the Typescript SDK.
It should be possible somehow to tie the jibDockerBuild
task to the compile lifecycle, allowing to execute it before tests, hence removing the need to have to execute two commands to first build the container images, and then run the tests. Or maybe, even better, it's possible to let tests.test depend on jibDockerBuild
of the other modules
It would be kick-ass if we could run the e2e test suite with every CI run in restatedev/restate and restatedev/sdk-* (CI runs for pushes and PRs). That way people would get feedback directly on their PRs if a change broke something.
This includes:
Our Node Dockerfile accepts the NPM_TOKEN
as a build arg. That way it will be recorded in the history of the image and others can retrieve the secret. It would be better to use --mount=type=secret,id=NPM_TOKEN
to mount a secret. However, so far I couldn't find a way to provide the secret to the build process. It seems that this is not supported by the Gradle Docker plugin we are using.
This is the same issue as restatedev/java-sdk#55.
Right now the test infra supports only stopping and restarting the runtime container. We should provide a solution to start and stop all the containers we deploy.
Add support for deploying our e2e testing artifacts to JFrog using the Github workflows.
This Github action must:
This action must be configured to trigger periodically the test
With #1, we set up the dependencies of the e2e tests by downloading the runtime and sdk-java repo, and manually build it every time. This works as a first prototype, but we should find out how to:
We need a dashboard with test results. Some ideas:
On an arm architecture (MacBook with M1 processor), it is almost impossible to work with tests that are using the Kafka test container. The problem is that the Kafka test container only supports amd64 platforms atm and due to this the Kafka code needs to go through QEMU which is insanely slow. As a consequence, most often the test fails because QEMU is not fast enough to bring up the Kafka container.
One alternative could be to use Redpanda. Redpanda offers images or amd64 and arm64 platforms which should make it a lot nicer for different development hardware. As of creating this issue, there is no Repdanda testcontainer availabe yet (see redpanda-data/redpanda#4671). Hence, we would have to use a GenericContainer
to create the test container ourselves.
Another alternative could be to bump our Kafka test container to version 7.1.0-1-ubi8
which seems to have arm support (see testcontainers/testcontainers-java#5029).
At the moment it is very difficult to run an individual test using gradle
. One way is to provide a test filter via --tests '*FoobarTest*'
. However, one needs to invoke the right testing task since one cannot run build
with this flag. I think this is an artifact of having a few non-standard testing tasks registered.
When running a single testing task, then it seems that we always run all test
tasks as well (I suspect that this is caused by the TestReport
task).
What is also very hard is to run a test/testing task (other than the test
task) repeatedly. What gradle does is to not run the task if it is up to date. For the test
task, there is a cleanTest
task which deletes the build/test-results/test
directory. I suspect that this one is used to check whether the task is up to date or not. For our custom test tasks, we don't have such a clean task and therefore one always needs to clean in order to re-run the tests. An alternative could be to provide the --rerun
flag.
Instead of only logging the container name and tag (especially when using a non-immutable tag like latest), the e2e tests should also log the hash of the container so that one knows which container image was used.
Setup e2e tracing, for exporting spans when executing e2e tests.
Similar to restatedev/runtime#131 we should add a timeout for the Github workflows.
We need to add the node variant of the test checking side effects error propagation, currently in ErrorsTest
For the scheduled e2e runs, I am currently receiving an email notification. It seems that I am the only one receiving these mails. I think it would be better to make failed runs more widely noticeable (e.g. via posting a failed run to a slack channel or generating a daily digest) so that the relevant people can react to failed e2e runs.
See #77 (comment)
Check if VerificationTest
is matching what executes in verification_test
in the runtime repo. Once that is done, we can remove the verification_test
code and shave off some minutes from CI :)
With restatedev/runtime@8a45abc2092e245b1ae63a8a93980eea2387f4c6 we fixed how the method identifier is parsed from a CloudEvent subject. Surprisingly, this was not covered by our existing e2e test that covered this case. We should investigate why this was the case.
See https://github.com/restatedev/e2e/actions/workflows/e2e.yaml for the effects.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.