hiddeco / cronjobber Goto Github PK

During maintenance purpose I want to disable all CronJobs, For default Kubernetes CronJob we can execute following command

kubectl patch cronjobs <job-name> -p '{"spec" : {"suspend" : true }}'

However I cannot see an option to do this in CronJobber . Is there a way to disable and enable Cronjob without playing with schedules.

Deprecated in v1.16, planned for removal in v1.22. Use apiextensions.k8s.io/v1 CustomResourceDefinition instead.

CRDs v1 allows the TZCronJob to be registered as part of the K8s API and be detected on Helm using

.Capabilities.APIVersions.Has "cronjobber.hidde.co/v1alpha1"

At the moment I'm managing the detection manually

As an admin I can easily deploy the examples you provide.
As another user (e.g. ci user) I get the following error:

tzcronjobs.cronjobber.hidde.co "my-dummy-service" is forbidden: User "ci-user" cannot get resource "tzcronjobs" in API group "cronjobber.hidde.co" in the namespace "mynamespace": RBAC: clusterrole.rbac.authorization.k8s.io "mynamespace-letsencrypt-role" not found

Do you have any ideas where to look? Do I have to change the rbac in some way?

We're using fluxcd and cronjobber, we would like to automatically update the cronjob image.
Simple testing seems to show that currently, fluxcd cannot understand the cronjobber resource.

First of all thanks for this awesome component. While official Kubernetes is expecting to add a Timezone soon but it is getting delayed a lot and we are using CronJobber which works great.

I am using EKS 1.21 and noticed an issue where CronJobber cronjobs will keep spinning up new pods if there are any error, or ImagePullBackoff , CrashLoopBack etc.. it is not honoring the fields concurrencyPolicy: Forbid . This works well with default CronJob API i.e. apiVersion: batch/v1 but not with CronJobber apiVersion: cronjobber.hidde.co/v1alpha1.

Is it possible to add a fix for this I am assuming this is happening because of the updates on CronJob in v 1.20

apiVersion: cronjobber.hidde.co/v1alpha1 kind: TZCronJob metadata: creationTimestamp: "2022-05-18T20:02:02Z" generation: 19 labels: app: cronjob-test-qa chart: test-cronjobs-68 release: cronjob-test-qa name: cronjob-test-qa namespace: qa resourceVersion: "600673922" uid: 667b2f60-02a9-4d19-992a-9b0dba7e8d50 spec: concurrencyPolicy: Forbid failedJobsHistoryLimit: 1 jobTemplate: metadata: labels: app: cronjob-test-qa spec: backoffLimit: 2

I am trying to get cronjobber running on EKS 1.23

The error:

Unable to update status for production/test-cronjob (rv = 8509036): tzcronjobs.cronjobber.hidde.co "test-cronjob" not found

Stacktrace:

github.com/hiddeco/cronjobber/pkg/controller/cronjobber.syncOne
	/go/src/github.com/hiddeco/cronjobber/pkg/controller/cronjobber/controller.go:257
github.com/hiddeco/cronjobber/pkg/controller/cronjobber.(*TZCronJobController).syncAll
	/go/src/github.com/hiddeco/cronjobber/pkg/controller/cronjobber/controller.go:146
github.com/hiddeco/cronjobber/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1
	/go/src/github.com/hiddeco/cronjobber/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
github.com/hiddeco/cronjobber/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil
	/go/src/github.com/hiddeco/cronjobber/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
github.com/hiddeco/cronjobber/vendor/k8s.io/apimachinery/pkg/util/wait.Until
	/go/src/github.com/hiddeco/cronjobber/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88

Any ideas?

Nice job, thanks for releasing this!

Before starting to use this, I would like to know a little more about the behavior of cronjobber during edge cases:

• what happens if it goes down (even for a brief period) and a cronjob was scheduled during that interval?
• what happens if a cronjob is scheduled during DST intervals?
• is the behavior of scheduled jobs the same as if I configured the same values on a cronjob, retry-policy-wise?

I cannot find a Helm chart version of it at https://hub.helm.sh
Do you have plans to create one or should everyone do it on their own manually?

Great Job! Thanks for sharing!.

I have a (probably) dummy question. How can I programmatically create a TZCronJob (in particular, I am using python). Is there and endpoint I need to call?

As reference, for regular CronJob, I am using pip package kubernetes==9.0.0 doing this:

cron_job.spec = V1beta1CronJobSpec(job_template=job_template, schedule=schedule)

client.BatchV1beta1Api(self._client).create_namespaced_cron_job(namespace_name, body=cron_job)

which in turn ends up calling

'/apis/batch/v1beta1/namespaces/{namespace}/cronjobs', 'POST'

Any advise very welcomed!
Alex

There is breaking change in client-go v0.18.0 when is context required for REST client and is not possible to use generated clientset because context is missing (eg. in Get() function).

DoD:

• cronjobber clientset can be used with client-go version >= 0.18.0

In the README it states:

• REFRESH_PERIOD (default: 7d) is how often it looks for updates

However, in docker-entrypoint.sh, and in the init-updatetz initContainer it is set to use REFRESH_INTERVAL:

sleep "${REFRESH_INTERVAL:=7d}"

There's been some changes to the Kubernetes CronJob controller since v1.13.3

It'd be good to track upstream and periodically sync CronJobber to the upstream CronJob version to avoid diverging too much form the defaults.

Maybe it'd be a good idea to periodically (using CronJobs 😉?) run a script to update the timezone db instead of mounting the host one. That way CronJobber would not depend on the host having a timezone db at all.

The error:

Attempted to run a job with an invalid time zone: Europe/Berlin. open /usr/local/go/lib/time/zoneinfo.zip: no such file or directory

Hi, I am trying to run cronjobber in an Openshift v4.6 cluster; Kubernetes v1.19. I was able to deploy cronjobber via Helm chart provided in the repo but when I schedule a TZCronJob, I am getting this error when the job starts:

Warning FailedCreate 78s (x91 over 16m ) cronjobber Error creating job: jobs.batch "hello-1630089060" is forbidden: cannot set blockOwnerDeletion in this case because cannot find RESTMapping for APIVersion batch/v1beta1 Kind TZCronJob: no matches for kind "TZCronJob" in version "batch/v1beta1"

Here is the TZCronJob yaml I am using:

apiVersion: cronjobber.hidde.co/v1alpha1
kind: TZCronJob
metadata:
  name: hello
spec:
  schedule: "30 15 * * *"
  timezone: "America/Sao_Paulo"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: hello
              image: busybox
              args:
                - /bin/sh
                - -c
                - date; echo "Hello, World!"
          restartPolicy: OnFailure

With regular cronjobs you can create jobs from a cronjob when you want to run a task inmediatly:

kubectl create job --from=cronjob/<name of cronjob> <name of job>

With cronjobber this does not work:

$ kubectl create job --from=tzcronjob/update-campaign-metrics update-campaign-metrics-now
error: from must be an existing cronjob: no kind "TZCronJob" is registered for version "cronjobber.hidde.co/v1alpha1" in scheme "k8s.io/kubernetes/pkg/kubectl/scheme/scheme.go:28"

$ kubectl create job --from=cronjob/update-campaign-metrics update-campaign-metrics-now
Error from server (NotFound): cronjobs.batch "update-campaign-metrics" not found

Would it be possible to fix this?

CronJobber is deployed on cronjobber NS and other regular CronJobs are deployed on NS zzzzz

To test my TZCronJob was deployed with schedule: */1 20 * * * @ 20.02 then changed to schedule: 15 20 * * * with the same result as below

There is a TODO just above that log statement. Am I suffering from a race condition of some sort? :(

{"level":"debug","ts":"2020-11-05T20:12:58.892Z","caller":"cronjobber/controller.go:136","msg":"Found 1 groups"}
{"level":"debug","ts":"2020-11-05T20:12:58.901Z","caller":"cronjobber/controller.go:281","msg":"No unmet start times for my-ns/my-job"}
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/controller.go:126","msg":"Found 16 jobs"}
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/controller.go:133","msg":"Found 1 cronjobs"}
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/utils.go:67","msg":"Job with non-TZCronJob parent, name xxxxx-1 namespace zzzzz
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/utils.go:81","msg":"Unable to get parent uid from job xxxxx-1 in namespace zzzzz}
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/utils.go:67","msg":"Job with non-TZCronJob parent, name xxxxx-2 namespace zzzzz
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/utils.go:81","msg":"Unable to get parent uid from job xxxxx-2 in namespace zzzzz}
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/utils.go:67","msg":"Job with non-TZCronJob parent, name xxxxx-3 namespace zzzzz
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/utils.go:81","msg":"Unable to get parent uid from job xxxxx-3 in namespace zzzzz}
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/utils.go:67","msg":"Job with non-TZCronJob parent, name xxxxx-4 namespace zzzzz
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/utils.go:81","msg":"Unable to get parent uid from job xxxxx-4 in namespace zzzzz}
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/utils.go:67","msg":"Job with non-TZCronJob parent, name xxxxx-5 namespace zzzzz
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/utils.go:81","msg":"Unable to get parent uid from job xxxxx-5 in namespace zzzzz}
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/utils.go:67","msg":"Job with non-TZCronJob parent, name xxxxx-6 namespace zzzzz
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/utils.go:81","msg":"Unable to get parent uid from job xxxxx-6 in namespace zzzzz}
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/utils.go:67","msg":"Job with non-TZCronJob parent, name xxxxx-7 namespace zzzzz
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/utils.go:81","msg":"Unable to get parent uid from job xxxxx-7 in namespace zzzzz}
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/utils.go:67","msg":"Job with non-TZCronJob parent, name xxxxx-8 namespace zzzzz
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/utils.go:81","msg":"Unable to get parent uid from job xxxxx-8 in namespace zzzzz}
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/utils.go:67","msg":"Job with non-TZCronJob parent, name xxxxx-9 namespace zzzzz
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/utils.go:81","msg":"Unable to get parent uid from job xxxxx-9 in namespace zzzzz}
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/utils.go:67","msg":"Job with non-TZCronJob parent, name xxxxx-10 namespace zzzzz
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/utils.go:81","msg":"Unable to get parent uid from job xxxxx-10 in namespace zzzzz}
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/utils.go:67","msg":"Job with non-TZCronJob parent, name xxxxx-11 namespace zzzzz
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/utils.go:81","msg":"Unable to get parent uid from job xxxxx-12 in namespace zzzzz}
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/utils.go:67","msg":"Job with non-TZCronJob parent, name xxxxx-12 namespace zzzzz
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/utils.go:81","msg":"Unable to get parent uid from job xxxxx-14 in namespace zzzzz}
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/utils.go:81","msg":"Unable to get parent uid from job xxxxx-15 in namespace zzzzz}
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/utils.go:81","msg":"Unable to get parent uid from job xxxxx-16 in namespace zzzzz}
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/utils.go:81","msg":"Unable to get parent uid from job xxxxx-17 in namespace zzzzz}
{"level":"debug","ts":"2020-11-05T20:13:08.919Z","caller":"cronjobber/controller.go:136","msg":"Found 1 groups"}
{"level":"debug","ts":"2020-11-05T20:13:08.929Z","caller":"cronjobber/controller.go:281","msg":"No unmet start times for my-ns/my-job"}

The cronjob kind has this feature —from and wondering if we can add here? Ie, kubectl create job —from=cronjob/ .

Do you apply the same concurrent policy of k8scronjob?
Even if I apply 'forbid' and 'replace', it seems to be set to 'allow'.

If the cronjobber must be a singleton, I think stateful set is probably more appropriate.

deply/deploy.yaml

ok

vi test.yaml
~
timezone: "Asia/Seoul"
~

kubectl apply -f test.yaml

in 5 sec or 6 sed the pod was starting..

scheldule was in 5 min or 10min from now
ex)
date
Wed Sep 2 10:29:48 KST 2020

test.yaml
Schedule: "35 10 * * *"

Hello,

I have a two CronJob the one set to 0 1 * * * in Europe/Paris which is running at 3 CEST, the other one scheduled at 0 3 * * * in Europe/Paris ran at 5 CEST...

Do you have any idea why it's not running at the time it was scheduled ?

Thanks,

Trying to deploy cronjobber-updatetz gives error about non existing image, it seems url is not working anymore.

https://quay.io/repository/hiddeco/cronjobber-updatetz

  Warning  Failed     8m48s (x4 over 10m)  kubelet, ip-172-20-54-167.ec2.internal  Failed to pull image "quay.io/hiddeco/cronjobber-updatetz:0.1.1": rpc error: code = Unknown desc = Error response from daemon: Get https://quay.io/v2/: EOF

Hi,

Thank you for this controller, great work! Unfortunately, I had this odd issue recently and I was not able to replicate the issue in a test environment therefore I don't know the root cause. I think this is a critical bug, because it's about an intermittently occurring unexpected behaviour: A job for each TZCronjob get scheduled immediately, regardless of their schedules defined in TZCronJob, when cronjobber pod is recreated.

Platform: AWS EKS 1.14

Scenario
1- cronjobber 0.2.0 was installed with updatetz

2- A TZCronJob is created with schedule "45 14 * * 2" and timezone "Europe/London", no startingDeadlineSeconds set

3- A job is scheduled and completed successfully on 12 May 1:45 pm UTC, which is expected

NAME                COMPLETIONS   DURATION   AGE
somejob-1589291100  1/1           7s         2d

4- Cronjobber pod gets recreated at "Wed, 13 May 2020 15:13:52 +0000":

cronjobber container logs

{"level":"info","ts":"2020-05-13T15:14:24.116Z","caller":"cronjobber/main.go:75","msg":"Starting Cronjobber version 0.2.0 revision f860bc912c395c58fa72741d8d34e8bf4b1a2c00"}
{"level":"info","ts":"2020-05-13T15:14:24.117Z","caller":"cronjobber/controller.go:106","msg":"Starting TZCronJob Manager"}

updatetz container logs

2020-05-13T15:14:29+0000 Local Time Zone database updated to version 2020a on /tmp/zoneinfo

5- Although its last schedule was for 12 May 1:45 pm UTC and it shouldn't be scheduled until next week, another job of this TZCronJob is started at "Wed, 13 May 2020 15:14:24 +0000", which is exactly the same to the second with the start time of cronjobber.

NAME                COMPLETIONS   DURATION   AGE
somejob-1589291100  1/1           7s         2d
somejob-1589294700  1/1           17s        23h

Now, this happened for all of our TZCronJobs. They got rescheduled immediately after cronjobber pod is recreated 😰

Observation
1- updatetz updates the local tz database for the first time in the pod 5 seconds later than cronjobber controller's start time.

2- I've noticed that unix timestamps on Job specs can lead to something. 1589291100 (the expected behaviour) is 12 May 1:45 pm UTC, which is good.
It gets interesting for 1589294700: This is 12 May 2:45 pm (UTC) -> so it looks like cronjobber could not get the timezone data from the mounted volume when it started up. I guess this caused cronjobber to misinterpret existing schedules. Cronjobber synced the existing cronjobs with the misinterpreted tzdata, which led kube-scheduler to interpret that the last schedules were missed although they weren't missed and hence kube-scheduler rescheduled all jobs.

Potential workarounds/suggestions
Since I couldn't replicate the scenario, below would be only blind guesses;
1- We could set startingDeadlineSeconds, but that would not solve the issue permanently. There will be some possibility for the issue to reoccur if [misinterpreted t] + startingDeadlineSeconds > x, where t is the tzcronjob's last successful execution timestamp and x is the time when cronjobber pod is recreated.

2- I haven't written a single line of Go (yet) so this may be gibberish, but I think there may be some missing/wrong variable initialisation. For example, if there is an initial value of the current timestamp, that is used when cronjobber is not able to read tzdata, then this magic number should be of a max value, instead of a min value. In other words, it's better to do nothing (don't sync) than doing something wrong (rescheduling).

3- Cronjobber should not start until updatetz does its job at least once. Maybe docker-entrypoint in updatetz can touch a file on another volume and an init container within the same pod will check if this file exists in this volume. Since there will be an init container, the cronjobber container will not start until the init container completes successfully.

in https://kubernetes.io/docs/reference/using-api/deprecation-guide/#v1-22 the CRD apiextensions.k8s.io/v1beta1 is deprecated and only apiextensions.k8s.io/v1 is supported going forward.

also some of the fields need to be updated:

CustomResourceDefinition in version "v1" cannot be handled as a CustomResourceDefinition: strict decoding error: 
unknown field "spec.additionalPrinterColumns", 
unknown field "spec.subresources", 
unknown field "spec.version"

the deprecation guide also provides the alternative fields to use

As you know, in cronjob, you can create jobs through --from.
However, tzcronjob seems impossible. Is there a way?