Comments (14)
nudgegoonies
commented on June 10, 2024
1
@ivanilves We havn't experienced hanging lstags processes anymore after syncing great amount of entries.
from lstags.
nudgegoonies
commented on June 10, 2024
I investigated a bit more. lstags does not work completely sequentially from the yaml files right? In the log i see, that the last image/tags in the sorted yaml is not pushed at the end. There are two more images/tags processed after that image/tags that come alphabetically and from the yaml file before that image.
When i remember right we never missed an image after the lstags sync. So i think the problem is exiting when lstags is finished. This would also explain the lots of hanging lstags processes.
from lstags.
ivanilves
commented on June 10, 2024
could U please:
- try to use latest lstags: https://github.com/ivanilves/lstags/releases/tag/v1.2.12
- add option
-w100msand also run the v.1.2.12 with it
-w, --wait-between= Time to wait between batches of requests (incl. pulls and pushes) (default: 0) [$WAIT_BETWEEN]
Yes, lstags processes yaml file entries in async non-sequential manner. 😉
So i think the problem is exiting when lstags is finished.
This is really interesting. For a research sake, could you provide your yaml file here (if it's not confidential, of course!)?
from lstags.
nudgegoonies
commented on June 10, 2024
We tried the 1.2.12 now and have the same problems with authentication.
I check if we can provide the yaml file.
from lstags.
ivanilves
commented on June 10, 2024
We tried the 1.2.12 now and have the same problems with authentication.
And rolling back to 1.2.10 fixes authentication back, right?
What kind of authentication you are using with Artifactory?
BASIC or TOKEN?
from lstags.
nudgegoonies
commented on June 10, 2024
Sorry for the long delay, we use BASIC AUTH with username/password and not the artifactory token.
from lstags.
ivanilves
commented on June 10, 2024
This is strange indeed. There were no changes on BASIC AUTH side in lstags since almost the very beginning 🤔
Call it a blunt, naive suggestion - but from the machine/VM/container you are running lstags on - could you please ensure you can actually do docker pull / docker push to your Artifactory?
from lstags.
nudgegoonies
commented on June 10, 2024
Yes, i can pull and push images normally.
from lstags.
ivanilves
commented on June 10, 2024
Just released a new version https://github.com/ivanilves/lstags/releases/tag/v1.2.13 with better tracing.
Could you please run it and share the [improved] tracing output here? 🙏
from lstags.
nudgegoonies
commented on June 10, 2024
Sorry for the delay.
I cherry-picked your 2 commits for better tracing to the latest working version 1.2.10 with the following output. Read my comments inside the two ### about what was written there:
<STATE> <DIGEST> <(local) ID> <Created At> <IMAGE>:<TAG>
ABSENT sha256:8ab8291e47460c686529dcbc1efedeb48 n/a 2018-07-20T08:07:57 docker.elastic.co/elasticsearch/elasticsearch-oss:6.3.2
ABSENT sha256:220a9a988288baf446e36d74aa93eb747 n/a 2018-12-17T22:53:36 docker.elastic.co/elasticsearch/elasticsearch-oss:6.5.4
ABSENT sha256:f59f7936d018e9d2329dfa0705b305e2c n/a 2019-01-24T12:31:25 docker.elastic.co/elasticsearch/elasticsearch-oss:6.6.0
ABSENT sha256:789e22d8b2a7ea1aacc3992da0157370e n/a 2019-02-13T18:13:42 docker.elastic.co/elasticsearch/elasticsearch-oss:6.6.1
ABSENT sha256:9c5ef68d57c747a0277f2276493057c22 n/a 2019-03-06T16:20:44 docker.elastic.co/elasticsearch/elasticsearch-oss:6.6.2
PRESENT sha256:5021b5feb63f642565d19213504d86a38 170c6c1bc829 2019-06-18T15:21:56 docker.elastic.co/elasticsearch/elasticsearch-oss:6.8.1
PRESENT sha256:316be55cedc4a1d301e57344ebed8424a e73d08f153d8 2019-07-24T17:30:47 docker.elastic.co/elasticsearch/elasticsearch-oss:6.8.2
PRESENT sha256:eadd80cfc04c7ae59f050eb23ae40931f 5756084a46f1 2019-08-29T21:12:58 docker.elastic.co/elasticsearch/elasticsearch-oss:6.8.3
-
|@URL: https://external.docker.mamdev.server.lan/v2/elasticsearch/elasticsearch-oss/tags/list
|@REQ-HEADER: Authorization = [Bearer ### Exactly the password part right of the : from the base64 decoded auth section in the config.json after decoding it ###]
|@REQ-HEADER: Accept = [application/json application/vnd.docker.distribution.manifest.v2+json]
|@RESP-HEADER: Connection = [keep-alive]
|@RESP-HEADER: Server = [Artifactory/6.16.0]
|@RESP-HEADER: X-Artifactory-Id = [################:-########:###########:-####]
|@RESP-HEADER: Docker-Distribution-Api-Version = [registry/2.0]
|@RESP-HEADER: Date = [Wed, 12 Feb 2020 14:04:10 GMT]
|@RESP-HEADER: Content-Type = [application/json]
|--- BODY BEGIN ---
|{
| "name" : "elasticsearch/elasticsearch-oss",
| "tags" : [ "6.3.2", "6.5.4", "6.6.0", "6.6.1", "6.6.2", "6.8.1", "6.8.2", "6.8.3" ]
|}
|--- BODY END ---
This is the output from your non working version 1.2.13. Read my comments inside the two ### about what was written there:
<STATE> <DIGEST> <(local) ID> <Created At> <IMAGE>:<TAG>
ABSENT sha256:8ab8291e47460c686529dcbc1efedeb48 n/a 2018-07-20T08:07:57 docker.elastic.co/elasticsearch/elasticsearch-oss:6.3.2
ABSENT sha256:220a9a988288baf446e36d74aa93eb747 n/a 2018-12-17T22:53:36 docker.elastic.co/elasticsearch/elasticsearch-oss:6.5.4
ABSENT sha256:f59f7936d018e9d2329dfa0705b305e2c n/a 2019-01-24T12:31:25 docker.elastic.co/elasticsearch/elasticsearch-oss:6.6.0
ABSENT sha256:789e22d8b2a7ea1aacc3992da0157370e n/a 2019-02-13T18:13:42 docker.elastic.co/elasticsearch/elasticsearch-oss:6.6.1
ABSENT sha256:9c5ef68d57c747a0277f2276493057c22 n/a 2019-03-06T16:20:44 docker.elastic.co/elasticsearch/elasticsearch-oss:6.6.2
PRESENT sha256:5021b5feb63f642565d19213504d86a38 170c6c1bc829 2019-06-18T15:21:56 docker.elastic.co/elasticsearch/elasticsearch-oss:6.8.1
PRESENT sha256:316be55cedc4a1d301e57344ebed8424a e73d08f153d8 2019-07-24T17:30:47 docker.elastic.co/elasticsearch/elasticsearch-oss:6.8.2
PRESENT sha256:eadd80cfc04c7ae59f050eb23ae40931f 5756084a46f1 2019-08-29T21:12:58 docker.elastic.co/elasticsearch/elasticsearch-oss:6.8.3
-
|@URL: https://external.docker.mamdev.server.lan/v2/elasticsearch/elasticsearch-oss/tags/list
|@REQ-HEADER: Authorization = [Bearer ### Broken base64 string. That is the readable part: {"typ":"JWT","alg":"RS256","kid":"#:#:#:#:#:#:#:#:#:#:#:#"} ###]
|@REQ-HEADER: Accept = [application/json application/vnd.docker.distribution.manifest.v2+json]
|@RESP-HEADER: Connection = [keep-alive]
|@RESP-HEADER: Server = [Artifactory/6.16.0]
|@RESP-HEADER: X-Artifactory-Id = [################:-########:###########:-####]
|@RESP-HEADER: Www-Authenticate = [Basic realm="Artifactory Realm"]
|@RESP-HEADER: Date = [Wed, 12 Feb 2020 13:27:43 GMT]
|@RESP-HEADER: Content-Type = [application/json;charset=ISO-8859-1]
|@RESP-HEADER: Content-Length = [101]
|--- BODY BEGIN ---
|{
| "errors" : [ {
| "status" : 401,
| "message" : "Token failed verification: signature"
| } ]
|}
|--- BODY END ---
I used the command line base64 tool to decode the string and got the error '': invalid input". But for me the json looks complete. At least there is { and }.
See also the missing Docker-Distribution-Api-Version header and the added Www-Authenticate header.
from lstags.
nudgegoonies
commented on June 10, 2024
I think our recent conversation belongs more to #214
from lstags.
ivanilves
commented on June 10, 2024
True! moved to #214
from lstags.
ivanilves
commented on June 10, 2024
Would like to ask you if original issue "Problem with great amount of entries" is still a thing?
Thank U!
from lstags.
ivanilves
commented on June 10, 2024
Good, closing 😄
from lstags.
Related Issues (20)
- Allow synchronization with sha syntax HOT 1
- Switch dependency tool from "dep" to "go mod" HOT 1
- Quay.io 429 too many requests / slow performance HOT 3
- Use PushTemplate to change remote push path HOT 1
- Flag '--no-ssl-verify' support
- Unauthorized on gcr.io HOT 4
- `--version` option returns `CURRENT` instead of version HOT 5
- Crash when pushing HOT 2
- 401 error while pushing since 1.2.11 HOT 7
- Requires list permissions on source repository
- Create missing repositories HOT 2
- Error when specifying docker.io as registry HOT 1
- Calls list API with page size of 1 causing rate limiting with quay.io HOT 5
- bearer response should also allow for access_token HOT 1
- panic: value method ... bearer.Token.Method called using nil *Token pointer HOT 5
- panic after 429 retry twice HOT 2
- Security: Don't package current dir HOT 3
- Crash when accessing large repository HOT 6
- Failure to pull images because of seemingly missing image digest (because of docker hub change)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from lstags.