containrrr / shepherd Goto Github PK

Hello,

Shepherd can't update services. It's works in a different swarm with exacly sames services.
Stacks was deployed with docker stack deploy -c compose-file.yml --with-registry-auth --resolve-image always stack_name
Services can be manually updated with docker service update service_name --force

 failed to configure transport: error pinging v2 registry: Get "https://registry.CENSORED/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Error updating service test_debian Image registry.CENSORED/debian:bullseye-slim_1.0 does not exist or it is not available

Hi,
I see the value of having a --env BLACKLIST_SERVICES.

In my case I would like to be more granular way to select only a few services that will be impacted by shepherd. I run about 60 services at the moment and only 3 of them (QA stuff) should be impacted by shepherd.

Example

--env ONLY_SERVICES="webapp1 webapp2 webappdev"

Thanks for your work!

As new build image nowadays using new OCI manifest, the script that use docker manifest cant extract information of new image. The recommended way is to use docker buildx imagetools inspect --raw. The problem is about insecure registry, i dont really know how to make buildx imagetools use insecure registry.

• docker/buildx#1509
• docker/cli#3990

I had this "issue" also with ourboros, but what are the reasons for a container to do infinite updates? My nextcloud-container is updates every 5 minutes with the same sha...I use nextcloud:18 tag and I get always "Shepered Container has updated nextcloud" notification. The logs show a corresponding result. The same SHA is always updated to the same SHA...

I am running some nightly-build servers in docker swarm which should be updated once during every night. So i first set SLEEP_TIME to 24h and launched the service at night. :)
Since the service updates take some time the next update window (sleep 24h) is slowly shifting into the morning. Additionally, if i change something in the service configuration and shepherd gets redeployed, i would have to do this again at night.
So, I came up with the idea to implement a configurable fixed time for the service update.

The first version is straight forward for my mentioned nightly use case. An environment variable SLEEP_UNTIL is introduced, which uses a 24-hour time format with the option to specify additional days: HH[:MM[:SS]] [add days]. This allows 24h, 48h, 72h, ... schedules that run at a fixed time. Examples: 23 --> every day at 23:00:00; 13:30:10 2 --> every third day at 13:30:10
This is the corresponding pull request: #86

I have also used watchtower in the past (which does not support swarm by the way containrrr/watchtower#43 ) and liked the idea of being able to specify cron schedules. After I finished the SLEEP_UNTIL version, I had the ambition to make it more general by using a cron schedule definition. Parsing a cron schedule in Bash was a little bit to much for me, but during my research I found out that the builtin printf of the Korn Shell ksh93 can calculate the time to the next run for a given crontab. I have not found a package of ksh93 in the alpine repos, so ksh93 is compiled during docker build. Using an environment variable SLEEP_SCHEDULE, a cron schedule can be defined to control the service updates.
The pull request for the SLEEP_SCHEDULE version: #87

I have made pull requests for both versions to have an open discussion, but my favorite is the second one using the crontab. 😀

Hi,
First time i have used shepherd, the container crashed on first run after this failed updated.
On automatic restart it started again, still hit same failure but didn't crash.

First run.

Fri May  6 14:53:38 PDT 2022 Timezone set to America/Los_Angeles
Fri May  6 14:53:38 PDT 2022 Enabling synchronous service updates
Fri May  6 14:53:38 PDT 2022 Excluding services: "shepherd_shepherd wordpress_db wordpress_wordpress"
Fri May  6 14:53:44 PDT 2022 Trying to update service adguard_adguard1 with image adguard/adguardhome:latest
Fri May  6 14:53:55 PDT 2022 No updates to service adguard_adguard1!
Fri May  6 14:54:01 PDT 2022 Trying to update service adguard_adguard2 with image adguard/adguardhome:latest
service update paused: update paused due to failure or early termination of task p9qhqj30zf4g7dfon5vszyy6x
Fri May  6 14:54:07 PDT 2022 Service adguard_adguard2 update failed!
Fri May  6 14:54:07 PDT 2022 Rolling adguard_adguard2 back
service rollback paused: update paused due to failure or early termination of task 0fqovvetycro3400b9z391rm2

Second Run

Fri May  6 14:54:13 PDT 2022 Timezone set to America/Los_Angeles
Fri May  6 14:54:13 PDT 2022 Enabling synchronous service updates
Fri May  6 14:54:13 PDT 2022 Excluding services: "shepherd_shepherd wordpress_db wordpress_wordpress"
Fri May  6 14:54:19 PDT 2022 Trying to update service adguard_adguard1 with image adguard/adguardhome:latest
Fri May  6 14:54:26 PDT 2022 No updates to service adguard_adguard1!
Fri May  6 14:54:32 PDT 2022 Trying to update service adguard_adguard2 with image adguard/adguardhome:latest
service update paused: update paused due to failure or early termination of task 4n1gx6a1pxb4dfvi3bw4ygxp1
Fri May  6 14:54:37 PDT 2022 Service adguard_adguard2 update failed!
Fri May  6 14:54:37 PDT 2022 Rolling adguard_adguard2 back
Fri May  6 14:54:47 PDT 2022 Trying to update service adguard_adguardhome-sync with image ghcr.io/bakito/adguardhome-sync:latest
Fri May  6 14:54:53 PDT 2022 No updates to service adguard_adguardhome-sync!
Fri May  6 14:55:00 PDT 2022 Trying to update service agent_agent with image portainer/agent:latest
Fri May  6 14:55:21 PDT 2022 No updates to service agent_agent!

Is there something specific i should dig into to understand the issue?

Hi djmaze,
I have an weird Problem with images that are hosted on registry.gitlab.com: Sometimes shepherd updates the image from image:tag@hash to image:tag. After sleeping shepherd updates the image back from image:tag to image:tag@hash. Its always the same image-version, no updates happened in the registry. That only happens with gitlab-images, docker-hub works fine.

Here my log:

[Shepherd] Service idm2cn-dev_api-frontend updated
Wed Jan 20 09:01:34 CET 2021 Service idm2cn-dev_api-frontend was updated from registry.gitlab.com/selma-tud/idm2cn-frontend:master@sha256:280d8f29b2ec118abd682a8c44246fd4a741d8ac4fb63a2b477c0c70bae8438f to registry.gitlab.com/selma-tud/idm2cn-frontend:master

[Shepherd] Service idm2cn-dev_api-frontend updated
Wed Jan 20 09:04:30 CET 2021 Service idm2cn-dev_api-frontend was updated from registry.gitlab.com/selma-tud/idm2cn-frontend:master to registry.gitlab.com/selma-tud/idm2cn-frontend:master@sha256:280d8f29b2ec118abd682a8c44246fd4a741d8ac4fb63a2b477c0c70bae8438f

I actually don't think it's a bug in your software, it looks like somethings wrong with PreviousSpec and Spec. Unfortunately, I can't find anything about it online and would be happy if you could point me in the right direction.

After upgrading docker to 17.06, shepherd no longer works, as updating a service requires the flag --with-registry-auth to be passed manually on each service update if the service was created with this flag.

I tried making a fork, and apply this flag for all service updates if engine is higher than 17.05, but though the command executed works if I'm connected directly to the docker node, it doesn't work as expected when executed by shepherd.

Could it be some missing permissions for the shepherd service?

I can observe CPU spikes (it's the container that uses the most CPU over 70 services/containers). I'll experiment with CPU limits and report :_p

I found your project from v2tec/watchtower researching support for swarm mode updates. One of the interesting features I liked was the ability to use labels on the other containers rather than hardcoded white/black lists. Would it be possible to implement something like that here? Thanks.

It appears that the service is getting stuck sometimes when asking docker to docker service update… on one cluster it's been stuck for over 24 hours…

$ docker exec -ti 97926cd69f86 ash
/ # ps -ef
PID   USER     TIME  COMMAND
    1 root      0:00 {shepherd} /bin/bash /usr/local/bin/shepherd
  301 root      6:35 docker service update x --detach=false --with-registry-auth --image=x
  315 root      0:00 ash
  321 root      0:00 ps -ef

If I kill pid=301 then it unblocks and continues on.

Consider calling docker service update via timeout to catch this?

https://github.com/djmaze/shepherd/blob/master/shepherd#L101

if ! timeout ${TIMEOUT_SECONDS:-300} docker "${config_flag[@]}" service update "$name" $detach_option $registry_auth $no_resolve_image_flag ${UPDATE_OPTIONS} --image="$image" > /dev/null

# timeout --help
BusyBox v1.34.1 (2021-11-23 00:57:35 UTC) multi-call binary.

Usage: timeout [-s SIG] SECS PROG ARGS

Run PROG. Send SIG to it if it is not gone in SECS seconds.
Default SIG: TERM.

Given a list of services deployed in a docker swarm using docker stack deploy -c compose.yml zeiterfassung where zeiterfassung is the name of the created stack.
This will result in shepherds script line docker service ls --quiet --filter "${FILTER_SERVICES}" --format '{{.Name}}' to give the following results:

zeiterfassung_api-mock
zeiterfassung_backend
zeiterfassung_frontend
zeiterfassung_shepherd
zeiterfassung_traefik
zeiterfassung_ubw-mock

If I write my blacklisted services like described in the README, i.e. BLACKLIST_SERVICES: "shepherd traefik",
then these won't be blacklisted, since the names in the blacklist don't exactly match the service names.
In my opinion however, the blacklisted names should not need to include the stack name, since that can be different across swarms. Could we perhaps wildcard match the blacklisted services by default?

Probably using shepherd wrong, but Docker is continuously restarting the services that shepherd updates, even when the image has not changed. I am using Docker 17.06.0-ce on Ubuntu 16.04.

What am I doing wrong? Alternatively, anything I can do to help fix or debug the problem further?

Maybe related to moby/moby#34242?

Whould it be possible to make shepherd exclude specific services or only monitor some?

This might be a dumb question

I am about to implement shepherd in my swarm.
Currently it has 28 containers across 25 services.

Will each check every 5 mins trip the rate limit counter, or just pulls on an image change?

The images don't change frequently - often for weeks at a time.

I am receiving this error:

Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.

I am not using docker swarm, should I be concerned? Will shepherd work and update my containers despite this message?

I'm getting an odd issue with Shepherd.

I have a Gitlab private repository and shepherd updates are failing with...
Get https://registry.gitlab.com/v2/wizewerx/trebuchet/manifests/latest: denied: access forbidden

Here is my docker-compose:

version: '3.3'
services:
  shepherd:
    image: mazzolino/shepherd:latest
    environment:
      FILTER_SERVICES: label=com.wizewerx.autodeploy
      SLEEP_TIME: 1m
      WITH_INSECURE_REGISTRY: 'true'
      WITH_REGISTRY_AUTH: 'true'
    volumes:
     - /var/run/docker.sock:/var/run/docker.sock:ro
    networks:
     - default
    logging:
      driver: json-file
    deploy:
      placement:
        constraints:
         - node.role == manager

I understand that the registry credentials need to be somehow acquired/passed to Shepherd and the used to login? Is this the case?

If so - where / how do I pass the credentials and where does the docker login happen - I can't see that it does.

Thanks for your help and for a super utility.

Personally, I haven't been using shepherd for a long time now. (I am pursueing a different, infrastructure-as-code based approach which I might blog about in the near future.)

That's a bad precondition for maintaining a project. So if there is anyone who is still actively using it (and preferably already demonstrated their ability to contribute to this project), please step up and show your interest here.

Hi There,

Great idea for a service but wondered whether it was possible (or you could consider) extending the schedule from a simple frequency of minutes to a more cron-like schedule? For example “0 1 1 * *” would cause the updates to run at 01:00am on the first day of every month, or “0 1 * * *” would cause the update to run at 00:01am every day.

The reason for this is that it would be easier to anticipate when updates would occur rather than every 5mins say?

If users still wanted to updates to be checked every 5 mins could use the cron schedule of “ */5 * * * *”?

Thanks!

I've scrolled through the closed issues and discovered that authentication against Dockerhub or private registries requires /root/.docker/config.json to be mounted. So I did:

• docker login on every node of my swarm
• Mount the aforementioned file into the container in the exact same location and make sure it's in the right place and readable
• Add the WITH_REGISTRY_AUTH=true environment variable to the service

but I still get the toomanyrequests error from Dockerhub. When I docker pull on the cli of my nodes, everything is fine. Shepherd, however, does not seem to get the authentication straight. The log says

Send registry authentication details to swarm agents,

immediately followed by:

toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit,

which only happens when I pull unauthenticated in my company.

Is there anything else I can try?

Thanks in advance!

if no update is required, the running image will be rebuilt anyway and will be offline for a while. this shouldn't be possible!!!

Sheperd unable to fetch image from a private dockerhub repository even with WITH_REGISTRY_AUTH set and login details added

logs:
organization/image:latest does not exist or it is not available

Hi,

I want to have shepherd to update only two services. I don't understand the syntax around label for shepherd. I do use labels as you can see.

I think the docs should be more specific around this.
Thank you!

cmd for shepherd

docker service create \
  --name "$CTN_shepherd" --hostname "$CTN_shepherd" \
  --replicas "1" \
  --constraint "node.role==manager" \
  --restart-condition "on-failure" \
  --limit-cpu "0.1" \
  --limit-memory "16M" \
  --env SLEEP_TIME="1m" \
  --env FILTER_SERVICES="10000013-edge-ghost,10000014-stable-ghost" \
  --mount type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock,ro \
  --mount type=bind,source=/root/.docker/config.json,target=/root/.docker/config.json,ro \
  ${IMG_shepherd}

error for shepherd

root@hulu1:~/deploy-setup# docker service logs -f tool-shepherd
tool-shepherd.1.nbh559jelcqo@hulu2    | Wed Dec 16 10:15:39 EST 2020 Timezone set to US/Eastern
tool-shepherd.1.nbh559jelcqo@hulu2    | Wed Dec 16 10:15:46 EST 2020 Enabling synchronous service updates
tool-shepherd.1.nbh559jelcqo@hulu2    | invalid argument "10000013-edge-ghost,10000014-stable-ghost" for "-f, --filter" flag: bad format of filter (expected name=value)
tool-shepherd.1.nbh559jelcqo@hulu2    | See 'docker service ls --help'.
tool-shepherd.1.nbh559jelcqo@hulu2    | Wed Dec 16 10:15:55 EST 2020 Sleeping 1m before next update

inspect my app

docker service inspect --pretty 10000013-edge-ghost

(...)
ID:		3vgh3inten0myykdlgfkbbkx3
Name:		10000013-edge-ghost
Labels:
 traefik.backend=10000013-edge-ghost
 traefik.backend.loadbalancer.method=drr
 traefik.backend.loadbalancer.swarm=true
 traefik.docker.network=ntw_front
 traefik.enable=true
 traefik.frontend.entryPoints=http,https
 traefik.frontend.priority=100
 traefik.frontend.redirect.entryPoint=https
 traefik.frontend.redirect.permanent=true
 traefik.frontend.rule=Host:trial.firepress.link;PathPrefix:/edge
 traefik.passHostHeader=true
 traefik.port=2368
 traefik.weight=10
Service Mode:	Replicated
 Replicas:	1
 
(...)

Hello!
I'm using shepherd to update all my services and work fine. But I have a problem when some service fail. If the service fail while shepherd try update, not continue with other services. I have the same problem when service has a rollback configuration.

Is there any solution?

Hi,
I have a private Harbor registry which is duplicated.
Both are load balanced and accessible via CLI.

When I attempt a jump from the main to the secondary Harbor, Shepherd fails with the errors:
no such manifest; Error updating service; Image <image_name> does not exist or it is not available.

Manual docker pull is successful.

Jumping back to the main Harbor does not fix the issue.

Restarting the service with update --force does not help.
Running docker stack deploy --prune --with-registry-auth -c docker-compose.yml also not helpful.

compose version: 3.8
service config:
shepherd:
image: mazzolino/shepherd:latest
logging: scalable-logging
deploy:
replicas: 1
placement:
constraints:
- "node.role==manager"
hostname: shepherd-updater
environment:
- SLEEP_TIME=5m
- BLACKLIST_SERVICES=dev_shepherd dev_nginx telemetry_
- WITH_REGISTRY_AUTH=true
- FILTER_SERVICES=label=autoupdate
- IMAGE_AUTOCLEAN_LIMIT=3
- TZ=UTC
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- /home/core/.docker/config.json:/root/.docker/config.json:ro

with the new Docker Hub rate limiting that is coming in, is there a way of using authentication to Docker Hub?

I noticed that there is a WITH_REGISTRY_AUTH environment variable, but the only option appears to be true ... but where is it getting the authentication details from?

I currently have shepherd and apprise deployed via a stack created in portainer, and I have just enabled the authentication in portainer (which seems to be working). I can provide my current stack compose if you need it.

Is there anything else that needs to be done other than WITH_REGISTRY_AUTH="true" and the portainer authentication?

Services with cap-add NET_ADMIN do not restart with cap-add functions after being updated by Sheperd.

Looking at the last commit, I feel this project should release a version.

Cheers!

Hi,

shepherd seams not to be working with swarm anymore, any idea?

house_keeping_service_shepherd.1.b3rmqmqvd2zh@docker-swarm    | Sat Nov 12 19:51:14 CET 2022 Trying to update service apps_service_pdns_dns_admin with image ngoduykhanh/powerdns-admin:latest
house_keeping_service_shepherd.1.b3rmqmqvd2zh@docker-swarm    | "docker service update" requires exactly 1 argument.
house_keeping_service_shepherd.1.b3rmqmqvd2zh@docker-swarm    | See 'docker service update --help'.
house_keeping_service_shepherd.1.b3rmqmqvd2zh@docker-swarm    |
house_keeping_service_shepherd.1.b3rmqmqvd2zh@docker-swarm    | Usage:  docker service update [OPTIONS] SERVICE
house_keeping_service_shepherd.1.b3rmqmqvd2zh@docker-swarm    |
house_keeping_service_shepherd.1.b3rmqmqvd2zh@docker-swarm    | Update a service
house_keeping_service_shepherd.1.b3rmqmqvd2zh@docker-swarm    | Sat Nov 12 19:51:14 CET 2022 Service apps_service_pdns_dns_admin update failed on shepherd!
house_keeping_service_shepherd.1.b3rmqmqvd2zh@docker-swarm    | Sat Nov 12 19:51:14 CET 2022 Rolling apps_service_pdns_dns_admin back

docker version 20.10.12 running in swarm mode

BR Takalele

This project is exactly the tool I'm looking for; thank you for creating it!

I'm using docker-compose to deploy an application I created, I was wondering if you could provide information on the docker-compose usage detailed in the docs.

  shepherd:
    build: .
    image: mazzolino/shepherd

What is the build: . key-value pair specifying?

Hi,

Just started to use this, its working great but it seems to stop on a particular container, can you help ?

I mean I have quite a few services - and all isn't listed here but it seems to have stopped on the last run - there is no output saying "no updates...."

Enabling synchronous service updates


Trying to update service bitcoin-testnet_bcoin with image iangregsondev/bcoin:latest


No updates to service bitcoin-testnet_bcoin!


Trying to update service bitcoin-testnet_bitcoind with image iangregsondev/bitcoind:0.19.0.1


No updates to service bitcoin-testnet_bitcoind!


Trying to update service bitcoin-testnet_lnd1 with image iangregsondev/lnd:latest


No updates to service bitcoin-testnet_lnd1!


Trying to update service bitcoin-testnet_lnd2 with image iangregsondev/lnd:latest


No updates to service bitcoin-testnet_lnd2!


Trying to update service bitcoin-testnet_loop1 with image iangregsondev/loop:latest

Am I doing something wrong?

The only thing I can see with this particular service is that it's not currently deployed. Would this be a problem?

Hi, it would be beautiful if at line 52 of sheperd file (sleep "$sleep_time") there will be a "cron style" check.
Let me explain: on an ENV var you insert a string like "30 22 * * 1". This string means "at 22:30 of every day in every month but only on monday".
Then on script you can check if this conditions are met and if yes you fire the update.
May be a useful idea?

I would really appreciate some sort of way to prevent a container from being upgraded during a specific window or to specifically allow only during a specific window. I understand that this usually relies on docker rolling updates but I suspect I would still see downtime and I'd like to ensure that it happens when no one is using the service.

Here is the place to say thank you to folk(s) who are working hard on this project.

« One of the greatest ways to show your appreciation to open source projects you enjoy is to open an issue that let people say thank you »

Disclaimer: I don't have any kind of connexion or personal interest with maintainers(s) of this project. Pure gratitude here.

Cheers!

Hi

how do I put multiple services in this environment variable? FILTER_SERVICES

what I tried (that does NOT work):
- "FILTER_SERVICES=name=dockerimage"
- "FILTER_SERVICES=name=anotherimage"

this also doesn't work:
- "FILTER_SERVICES=name=dockerimage, name-anotherimage"

this also doesn't work:
- "FILTER_SERVICES=name=dockerimage, anotherimage"

so please advice
thank you

Hi

I have the issue, that Shepherd cannot update a service with multiple replicas (or deployed in global mode).
The Shepherd docker just has this entry in the log:

Fri Jan 27 22:31:16 CET 2023 Trying to update service PROXY_whoami2 with image containous/whoami:latest

and then nothing happens. I just stalls and nothing happens for days, until you either kill the process in the container or kill the container.

We need to make sure this does not:

• break CI
• break any other release automation
• works well with any existing docs links

Is there an issue with the configuration at https://hub.docker.com/r/mazzolino/shepherd/ ?

In the logs I see the successful update of my services, but they won't be re-deployed automatically. I have to deploy the stack again to deploy the new version.

Is this intended?

Using the latest docker version.

It skipped shepherd_shepherd and wordpress_db services as expected.
However it didn't skip wordpress_wordpress like i expected.

Did i do something wrong?

Fri May  6 14:54:13 PDT 2022 Timezone set to America/Los_Angeles
Fri May  6 14:54:13 PDT 2022 Enabling synchronous service updates
Fri May  6 14:54:13 PDT 2022 Excluding services: "shepherd_shepherd wordpress_db wordpress_wordpress"
...
Fri May  6 15:00:42 PDT 2022 Trying to update service wordpress_wordpress with image wordpress:latest
Fri May  6 15:01:03 PDT 2022 Service wordpress_wordpress was updated!
Fri May  6 15:01:03 PDT 2022 Cleaning up old docker images, leaving last 2
Fri May  6 15:01:03 PDT 2022 Sleeping 60m before next update

This is my stack.

version: "3"
services:
  shepherd:
    build: .
    image: mazzolino/shepherd
    environment:
      - SLEEP_TIME=60m
      - IGNORELIST_SERVICES="shepherd_shepherd wordpress_db wordpress_wordpress"
      - ROLLBACK_ON_FAILURE=true
      - TZ=America/Los_Angeles
      - IMAGE_AUTOCLEAN_LIMIT=2
#      - APPRISE_SIDECAR_URL=apprise-microservice:5000      	
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    deploy:
      placement:
        constraints:
        - node.role == manager

When my shepherd service try update the other services in its log appear the next error. About one month was working fine but now I can't update my services.

The images exists on gitlab because if I update the service using its stack work fine.

Get https://registry.gitlab.test.com/v2/web-developer/ms-compras-ventas/manifests/dev-latest: unauthorized: HTTP Basic: Access denied

Error updating service ms-compras-ventas_dev-api! Image registry.gitlab.test.com/web-developer/ms-compras-ventas:dev-latest does not exist or it is not available

Hi,
I use version 0.4.0 for a while now and I see many commits. Any plan to release?
Thanks!

Would it be possible and useful to implement a possibility to notify updated services over Slack or Email.
I think to a process like this:
shepherd recognised a new image > triggers the update > if it was successful, send the notification.

@djmaze What do you think about features like this?

I found no docs or issues about this topic. Was it discussed already?

Thank you.

Checking my shepherd log and see

shepherd_worker.1.jxikrxr1ap9z@homelab | /usr/local/bin/shepherd: line 68: "5": syntax error: operand expected (error token is ""5"") shepherd_worker.1.jxikrxr1ap9z@homelab | xargs: 'echo' terminated by signal 13 shepherd_worker.1.jxikrxr1ap9z@homelab | "docker rmi" requires at least 1 argument. shepherd_worker.1.jxikrxr1ap9z@homelab | See 'docker rmi --help'. shepherd_worker.1.jxikrxr1ap9z@homelab | shepherd_worker.1.jxikrxr1ap9z@homelab | Usage: docker rmi [OPTIONS] IMAGE [IMAGE...]

is it this because i deploy over compose?
my compose
`

version: "3.5"
services:
worker:
image: mazzolino/shepherd
environment:
- SLEEP_TIME="180m"
- BLACKLIST_SERVICES="shepherd"
- WITH_REGISTRY_AUTH="true"
- WITH_INSECURE_REGISTRY="true"
- WITH_NO_RESOLVE_IMAGE="false"
- IMAGE_AUTOCLEAN_LIMIT="5"
- TZ=Asia/Jakarta
networks:
- shepherd
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
deploy:
placement:
constraints:
- node.role == manager
networks:
shepherd:
driver: overlay
name: shepherd
`

Thanks.

The check for DOCKER_CLI_EXPERIMENTAL isn't short circuiting the docker manifest call, which is failing due to my installation not enabling those features (expected) however due to this failure, the images will never update with shepherd since it falls into the if statement saying the images don't exist. Code snippet below causing the issues:

    if ! DOCKER_CLI_EXPERIMENTAL=enabled docker manifest inspect $insecure_registry_flag $image > /dev/null; then
        logger "Error updating service $name! Image $image does not exist or it is not available"
      else

By commenting out the if statement, everything worked as suggested, i've also tested out the change below and it works on my installation, however since i don't have a system with DOCKER_CLI_EXPERIMENTAL enabled, i cannot test the usage of it in the other case

      if ! DOCKER_CLI_EXPERIMENTAL=enabled; then
         if ! docker manifest inspect $insecure_registry_flag $image > /dev/null; then
           logger "Error updating service $name! Image $image does not exist or it is not available"
         fi
      else

we had an issue with our private registry (unvailable because of network maintenance) and during that period shepherd stopped/kill a few services.

The logfile clearly shows that registry access was the issue:

image ... could not be accessed on a registry to record
its digest. Each node will access ... independently,
possibly leading to different nodes running different
versions of the image.

I suspect that similar issues could be avoided by using something like

docker pull $image && \
docker service update "$service" $detach_option $registry_auth --image="$image" > /dev/null

to perform the service update, as it ensures that the registry is available
and the image is already pulled and available once service update is executed.

As a side effect, service update would become faster (for large images or slow registries)

Actual:
Actual implementation do the image cleanup only on the node running the shepherd container, not on the node running the service updated, nor on any other

Expected:
In decreasing preference order:

• Cleanup on every nodes, keeping the image digest matching the updated image (rem: image digest is immutable not the image ID)
• Cleanup on every nodes
• Cleanup on node running the service
• Limitation highlighted in documentation

Hello

I have a small feature request. Today you only get a notification if a service has been updated or it has failed to update.
But sometimes Shepherd can run a few days, without having updates to services. But you receive no notifications, so you don't know if Shepherd has run successfully and complete the check or whether it has failed.

So the request is: A notification on a completed run.

Update:

Forget about my text below. I was simply confused by the documentation.
I've added a more complete example for Docker Compose / Docker Swarm.

Cheers.

Old Text

Hi there.

I've placed a service on the ignorelist, which is recognized to be excluded.
Yet, shepherd tries to update the service.

I've looked into the code but couldn't find any hints.

Version used: mazzolino/shepherd:latest (0.7.0)

Here's my log output:

Tue Nov  1 23:08:05 CET 2022 Timezone set to Europe/Berlin
Tue Nov  1 23:08:05 CET 2022 Enabling synchronous service updates
Tue Nov  1 23:08:05 CET 2022 Excluding services: "tools_swarm-cleanup"
Tue Nov  1 23:08:10 CET 2022 Trying to update service gameservers_minecraft-1 with image itzg/minecraft-server:java17-graalvm-ce
Tue Nov  1 23:08:17 CET 2022 No updates to service gameservers_minecraft-1!
Tue Nov  1 23:08:24 CET 2022 Trying to update service tools_shepherd with image mazzolino/shepherd:latest
Tue Nov  1 23:08:31 CET 2022 No updates to service tools_shepherd!
Tue Nov  1 23:08:36 CET 2022 Trying to update service tools_swarm-cleanup with image docker:latest

yml file for shepherd:

version: "3"

services:
  shepherd:
    image: mazzolino/shepherd
    environment:
      - TZ=Europe/Berlin
      - IGNORELIST_SERVICES="tools_swarm-cleanup"
      - ROLLBACK_ON_FAILURE="true"
      - IMAGE_AUTOCLEAN_LIMIT=1
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    deploy:
      placement:
        constraints:
        - node.role == manager

Let me know, if you need more Infos.

Would be great to have a feature to add the hostname or a custom text for notifications. Currently its not visible where the update was done.