[Feature Request] A Way to Reduce Quota Consumption for Replication on Docker Hub,about goharbor/harbor

Comments (7)

MinerYang commented on July 21, 2024

For dockerhub adapter, it will list tags while replicating. The request would be something like GET https://my.harbor.domain/v2/repositories/<my-repository>/tags. However it not the using Get manifest which supposed to not hit the Definition of limits.
Could you provide the proxy and harbor registry logs to help us triage better. Thx

from harbor.

BrettDean commented on July 21, 2024

Hi @BrettDean ,

For dockerhub adapter, it will list tags while replicating. The request would be something like GET https://my.harbor.domain/v2/repositories/<my-repository>/tags. However it not the using Get manifest which supposed to not hit the Definition of limits. Could you provide the proxy and harbor registry logs to help us triage better. Thx

Thank you for your response @MinerYang ,
I apologize for forgetting to provide the additional log files last time. This time, the replicate also consumed 4 quotas, decreasing from 200 to 196.

proxy.log:

Jun 24 07:51:18 172.28.0.1 proxy[2262]: 127.0.0.1 - "GET / HTTP/1.1" 308 171 "-" "curl/8.1.2" 0.000 - .
Jun 24 07:51:18 172.28.0.1 proxy[2262]: 172.28.0.1 - "GET /api/v2.0/replication/executions?policy_id=24&page_size=15&page=1 HTTP/1.1" 200 4997 "https://harbor.xxx.com/harbor/replications" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15" 0.011 0.010 .
Jun 24 07:51:32 172.28.0.1 proxy[2262]: 172.28.0.1 - "POST /api/v2.0/replication/executions HTTP/1.1" 201 0 "https://harbor.xxx.com/harbor/replications" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15" 0.047 0.047 .
Jun 24 07:51:32 172.28.0.1 proxy[2262]: 172.28.0.1 - "GET /api/v2.0/replication/executions?policy_id=24&page_size=15&page=1 HTTP/1.1" 200 4723 "https://harbor.xxx.com/harbor/replications" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15" 0.007 0.008 .
Jun 24 07:51:42 172.28.0.1 proxy[2262]: 172.28.0.1 - "GET /api/v2.0/replication/executions?policy_id=24&page_size=15&page=1 HTTP/1.1" 200 4723 "https://harbor.xxx.com/harbor/replications" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15" 0.009 0.009 .
Jun 24 07:51:43 172.28.0.1 proxy[2262]: 172.28.0.1 - "GET /api/v2.0/replication/executions/1531 HTTP/1.1" 200 224 "https://harbor.xxx.com/harbor/replications" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15" 0.004 0.004 .
Jun 24 07:51:43 172.28.0.1 proxy[2262]: 172.28.0.1 - "GET /api/v2.0/replication/executions/1531/tasks?page_size=50&page=1 HTTP/1.1" 200 316 "https://harbor.xxx.com/harbor/replications/1531/tasks" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15" 0.007 0.006 .
Jun 24 07:51:45 172.28.0.1 proxy[2262]: 172.28.0.1 - "GET /api/v2.0/replication/executions/1531/tasks/607/log HTTP/1.1" 200 2197 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15" 0.010 0.011 .
Jun 24 07:51:45 172.28.0.1 proxy[2262]: 172.28.0.1 - "GET /api/v2.0/replication/executions/1531/tasks/607/log HTTP/1.1" 200 2333 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15" 0.007 0.008 .
Jun 24 07:51:46 172.28.0.1 proxy[2262]: 172.28.0.1 - "GET /api/v2.0/replication/executions/1531/tasks/607/log HTTP/1.1" 200 2333 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15" 0.011 0.010 .
Jun 24 07:51:46 172.28.0.1 proxy[2262]: 172.28.0.1 - "GET /api/v2.0/replication/executions/1531/tasks/607/log HTTP/1.1" 200 2333 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15" 0.006 0.006 .
Jun 24 07:51:47 172.28.0.1 proxy[2262]: 172.28.0.1 - "GET /api/v2.0/replication/executions/1531/tasks/607/log HTTP/1.1" 200 2955 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15" 0.006 0.006 .
Jun 24 07:51:48 172.28.0.1 proxy[2262]: 172.28.0.1 - "GET /api/v2.0/replication/executions/1531/tasks/607/log HTTP/1.1" 200 2955 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15" 0.007 0.007 .
Jun 24 07:51:48 172.28.0.1 proxy[2262]: 172.28.0.1 - "GET /api/v2.0/replication/executions/1531/tasks/607/log HTTP/1.1" 200 2955 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15" 0.006 0.007 .
Jun 24 07:51:48 172.28.0.1 proxy[2262]: 127.0.0.1 - "GET / HTTP/1.1" 308 171 "-" "curl/8.1.2" 0.000 - .
Jun 24 07:51:52 172.28.0.1 proxy[2262]: 172.28.0.1 - "GET /api/v2.0/replication/executions?policy_id=24&page_size=15&page=1 HTTP/1.1" 200 4723 "https://harbor.xxx.com/harbor/replications/1531/tasks" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15" 0.012 0.012 .
Jun 24 07:51:53 172.28.0.1 proxy[2262]: 172.28.0.1 - "GET /api/v2.0/replication/executions/1531 HTTP/1.1" 200 224 "https://harbor.xxx.com/harbor/replications/1531/tasks" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15" 0.004 0.004 .
Jun 24 07:51:54 172.28.0.1 proxy[2262]: 172.28.0.1 - "GET /api/v2.0/replication/executions/1531/tasks?page_size=50&page=1 HTTP/1.1" 200 316 "https://harbor.xxx.com/harbor/replications/1531/tasks" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15" 0.006 0.006 .
Jun 24 07:52:03 172.28.0.1 proxy[2262]: 172.28.0.1 - "GET /api/v2.0/replication/executions?policy_id=24&page_size=15&page=1 HTTP/1.1" 200 4723 "https://harbor.xxx.com/harbor/replications/1531/tasks" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15" 0.011 0.012 .
Jun 24 07:52:04 172.28.0.1 proxy[2262]: 172.28.0.1 - "GET /api/v2.0/replication/executions/1531 HTTP/1.1" 200 224 "https://harbor.xxx.com/harbor/replications/1531/tasks" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15" 0.004 0.005 .
Jun 24 07:52:04 172.28.0.1 proxy[2262]: 172.28.0.1 - "GET /api/v2.0/replication/executions/1531/tasks?page_size=50&page=1 HTTP/1.1" 200 313 "https://harbor.xxx.com/harbor/replications/1531/tasks" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15" 0.005 0.006 .
Jun 24 07:52:09 172.28.0.1 proxy[2262]: 172.28.0.1 - "GET /api/v2.0/replication/executions/1531/tasks/607/log HTTP/1.1" 200 5857 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15" 0.008 0.009 .
Jun 24 07:52:10 172.28.0.1 proxy[2262]: 172.28.0.1 - "GET /api/v2.0/replication/executions/1531/tasks/607/log HTTP/1.1" 200 5857 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15" 0.007 0.007 .
Jun 24 07:52:13 172.28.0.1 proxy[2262]: 172.28.0.1 - "GET /api/v2.0/replication/executions?policy_id=24&page_size=15&page=1 HTTP/1.1" 200 4723 "https://harbor.xxx.com/harbor/replications/1531/tasks" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15" 0.012 0.011 .
Jun 24 07:52:14 172.28.0.1 proxy[2262]: 172.28.0.1 - "GET /api/v2.0/replication/executions/1531 HTTP/1.1" 200 224 "https://harbor.xxx.com/harbor/replications/1531/tasks" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15" 0.003 0.004 .
Jun 24 07:52:14 172.28.0.1 proxy[2262]: 172.28.0.1 - "GET /api/v2.0/replication/executions/1531/tasks?page_size=50&page=1 HTTP/1.1" 200 313 "https://harbor.xxx.com/harbor/replications/1531/tasks" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15" 0.006 0.005 .
Jun 24 07:52:18 172.28.0.1 proxy[2262]: 127.0.0.1 - "GET / HTTP/1.1" 308 171 "-" "curl/8.1.2" 0.000 - .
Jun 24 07:52:23 172.28.0.1 proxy[2262]: 172.28.0.1 - "GET /api/v2.0/replication/executions?policy_id=24&page_size=15&page=1 HTTP/1.1" 200 4720 "https://harbor.xxx.com/harbor/replications/1531/tasks" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15" 0.010 0.010 .
Jun 24 07:52:24 172.28.0.1 proxy[2262]: 172.28.0.1 - "GET /api/v2.0/replication/executions/1531 HTTP/1.1" 200 221 "https://harbor.xxx.com/harbor/replications/1531/tasks" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15" 0.004 0.005 .
Jun 24 07:52:24 172.28.0.1 proxy[2262]: 172.28.0.1 - "GET /api/v2.0/replication/executions/1531/tasks?page_size=50&page=1 HTTP/1.1" 200 313 "https://harbor.xxx.com/harbor/replications/1531/tasks" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15" 0.006 0.006 .
Jun 24 07:52:48 172.28.0.1 proxy[2262]: 127.0.0.1 - "GET / HTTP/1.1" 308 171 "-" "curl/8.1.2" 0.000 - .

registry.log:

Jun 24 07:51:07 172.28.0.1 registry[2262]: 172.28.0.8 - - [24/Jun/2024:07:51:07 +0000] "GET / HTTP/1.1" 200 0 "" "Go-http-client/1.1"
Jun 24 07:51:08 172.28.0.1 registry[2262]: 127.0.0.1 - - [24/Jun/2024:07:51:08 +0000] "GET / HTTP/1.1" 200 0 "" "curl/8.1.2"
Jun 24 07:51:17 172.28.0.1 registry[2262]: 172.28.0.8 - - [24/Jun/2024:07:51:17 +0000] "GET / HTTP/1.1" 200 0 "" "Go-http-client/1.1"
Jun 24 07:51:27 172.28.0.1 registry[2262]: 172.28.0.8 - - [24/Jun/2024:07:51:27 +0000] "GET / HTTP/1.1" 200 0 "" "Go-http-client/1.1"
Jun 24 07:51:37 172.28.0.1 registry[2262]: 172.28.0.8 - - [24/Jun/2024:07:51:37 +0000] "GET / HTTP/1.1" 200 0 "" "Go-http-client/1.1"
Jun 24 07:51:38 172.28.0.1 registry[2262]: 127.0.0.1 - - [24/Jun/2024:07:51:38 +0000] "GET / HTTP/1.1" 200 0 "" "curl/8.1.2"
Jun 24 07:51:40 172.28.0.1 registry[2262]: time="2024-06-24T07:51:40.447860968Z" level=info msg="authorized request" go.version=go1.22.3 http.request.host="core:8080" http.request.id=214da142-3a12-4e4a-ad4a-c9dfe9180fa1 http.request.method=HEAD http.request.remoteaddr=172.28.0.11 http.request.uri="/v2/vergoh/vnstat/manifests/sha256:72a6f2d6c5cbc8382ad39dbc50ff3f3a23260ca282555e255742fad3bd102d06" http.request.useragent=harbor-registry-client vars.name="vergoh/vnstat" vars.reference="sha256:72a6f2d6c5cbc8382ad39dbc50ff3f3a23260ca282555e255742fad3bd102d06" 
Jun 24 07:51:43 172.28.0.1 registry[2262]: time="2024-06-24T07:51:43.911884843Z" level=info msg="redis: connect redis:6379" go.version=go1.22.3 instance.id=9bd6e4a0-5225-4b3b-a070-c741100fd968 redis.connect.duration=6.691997ms service=registry version=v2.8.3.m 
Jun 24 07:51:44 172.28.0.1 registry[2262]: time="2024-06-24T07:51:44.726218673Z" level=info msg="response completed" go.version=go1.22.3 http.request.host="core:8080" http.request.id=214da142-3a12-4e4a-ad4a-c9dfe9180fa1 http.request.method=HEAD http.request.remoteaddr=172.28.0.11 http.request.uri="/v2/vergoh/vnstat/manifests/sha256:72a6f2d6c5cbc8382ad39dbc50ff3f3a23260ca282555e255742fad3bd102d06" http.request.useragent=harbor-registry-client http.response.contenttype="application/vnd.oci.image.index.v1+json" http.response.duration=4.285460864s http.response.status=200 http.response.written=3161 
Jun 24 07:51:44 172.28.0.1 registry[2262]: 172.28.0.8 - - [24/Jun/2024:07:51:40 +0000] "HEAD /v2/vergoh/vnstat/manifests/sha256:72a6f2d6c5cbc8382ad39dbc50ff3f3a23260ca282555e255742fad3bd102d06 HTTP/1.1" 200 3161 "" "harbor-registry-client"
Jun 24 07:51:45 172.28.0.1 registry[2262]: time="2024-06-24T07:51:45.095217224Z" level=info msg="authorized request" go.version=go1.22.3 http.request.host="core:8080" http.request.id=b26d2c34-e501-4d65-b3b3-aa0a430a0ccc http.request.method=HEAD http.request.remoteaddr=172.28.0.11 http.request.uri="/v2/vergoh/vnstat/manifests/sha256:72a6f2d6c5cbc8382ad39dbc50ff3f3a23260ca282555e255742fad3bd102d06" http.request.useragent=harbor-registry-client vars.name="vergoh/vnstat" vars.reference="sha256:72a6f2d6c5cbc8382ad39dbc50ff3f3a23260ca282555e255742fad3bd102d06" 
Jun 24 07:51:45 172.28.0.1 registry[2262]: 172.28.0.8 - - [24/Jun/2024:07:51:45 +0000] "HEAD /v2/vergoh/vnstat/manifests/sha256:72a6f2d6c5cbc8382ad39dbc50ff3f3a23260ca282555e255742fad3bd102d06 HTTP/1.1" 200 3161 "" "harbor-registry-client"
Jun 24 07:51:45 172.28.0.1 registry[2262]: time="2024-06-24T07:51:45.101121821Z" level=info msg="response completed" go.version=go1.22.3 http.request.host="core:8080" http.request.id=b26d2c34-e501-4d65-b3b3-aa0a430a0ccc http.request.method=HEAD http.request.remoteaddr=172.28.0.11 http.request.uri="/v2/vergoh/vnstat/manifests/sha256:72a6f2d6c5cbc8382ad39dbc50ff3f3a23260ca282555e255742fad3bd102d06" http.request.useragent=harbor-registry-client http.response.contenttype="application/vnd.oci.image.index.v1+json" http.response.duration=12.331753ms http.response.status=200 http.response.written=3161 
Jun 24 07:51:45 172.28.0.1 registry[2262]: time="2024-06-24T07:51:45.548040073Z" level=info msg="authorized request" go.version=go1.22.3 http.request.host="core:8080" http.request.id=7648778c-3ba1-403f-bc09-756de2e6361d http.request.method=HEAD http.request.remoteaddr=172.28.0.11 http.request.uri="/v2/vergoh/vnstat/manifests/sha256:6db175557098daba90ac4dd9ed0abfde62993d61d1d5fca949f011ca1dc233f4" http.request.useragent=harbor-registry-client vars.name="vergoh/vnstat" vars.reference="sha256:6db175557098daba90ac4dd9ed0abfde62993d61d1d5fca949f011ca1dc233f4" 
Jun 24 07:51:46 172.28.0.1 registry[2262]: time="2024-06-24T07:51:46.940810903Z" level=info msg="response completed" go.version=go1.22.3 http.request.host="core:8080" http.request.id=7648778c-3ba1-403f-bc09-756de2e6361d http.request.method=HEAD http.request.remoteaddr=172.28.0.11 http.request.uri="/v2/vergoh/vnstat/manifests/sha256:6db175557098daba90ac4dd9ed0abfde62993d61d1d5fca949f011ca1dc233f4" http.request.useragent=harbor-registry-client http.response.contenttype="application/vnd.oci.image.index.v1+json" http.response.duration=1.396110124s http.response.status=200 http.response.written=3161 
Jun 24 07:51:46 172.28.0.1 registry[2262]: 172.28.0.8 - - [24/Jun/2024:07:51:45 +0000] "HEAD /v2/vergoh/vnstat/manifests/sha256:6db175557098daba90ac4dd9ed0abfde62993d61d1d5fca949f011ca1dc233f4 HTTP/1.1" 200 3161 "" "harbor-registry-client"
Jun 24 07:51:47 172.28.0.1 registry[2262]: time="2024-06-24T07:51:47.380939307Z" level=info msg="authorized request" go.version=go1.22.3 http.request.host="core:8080" http.request.id=5913f786-692e-40a0-b139-7be49ff95f69 http.request.method=HEAD http.request.remoteaddr=172.28.0.11 http.request.uri="/v2/vergoh/vnstat/manifests/sha256:0a9a462a2d6f45af045e02c0e12b555ee369dc37340fc77abc6fab3d5489f5b3" http.request.useragent=harbor-registry-client vars.name="vergoh/vnstat" vars.reference="sha256:0a9a462a2d6f45af045e02c0e12b555ee369dc37340fc77abc6fab3d5489f5b3" 
Jun 24 07:51:47 172.28.0.1 registry[2262]: 172.28.0.8 - - [24/Jun/2024:07:51:47 +0000] "GET / HTTP/1.1" 200 0 "" "Go-http-client/1.1"
Jun 24 07:51:48 172.28.0.1 registry[2262]: 172.28.0.8 - - [24/Jun/2024:07:51:47 +0000] "HEAD /v2/vergoh/vnstat/manifests/sha256:0a9a462a2d6f45af045e02c0e12b555ee369dc37340fc77abc6fab3d5489f5b3 HTTP/1.1" 200 3161 "" "harbor-registry-client"
Jun 24 07:51:48 172.28.0.1 registry[2262]: time="2024-06-24T07:51:48.859474439Z" level=info msg="response completed" go.version=go1.22.3 http.request.host="core:8080" http.request.id=5913f786-692e-40a0-b139-7be49ff95f69 http.request.method=HEAD http.request.remoteaddr=172.28.0.11 http.request.uri="/v2/vergoh/vnstat/manifests/sha256:0a9a462a2d6f45af045e02c0e12b555ee369dc37340fc77abc6fab3d5489f5b3" http.request.useragent=harbor-registry-client http.response.contenttype="application/vnd.oci.image.index.v1+json" http.response.duration=1.482104416s http.response.status=200 http.response.written=3161 
Jun 24 07:51:49 172.28.0.1 registry[2262]: time="2024-06-24T07:51:49.232772614Z" level=info msg="authorized request" go.version=go1.22.3 http.request.host="core:8080" http.request.id=20ceb6d1-c9fb-4c96-bd3b-ddb061f62363 http.request.method=HEAD http.request.remoteaddr=172.28.0.11 http.request.uri="/v2/vergoh/vnstat/manifests/sha256:801dd0ecec9db208f5c84e051dd5dc970080961cb0854c1e9786aa8040a209b1" http.request.useragent=harbor-registry-client vars.name="vergoh/vnstat" vars.reference="sha256:801dd0ecec9db208f5c84e051dd5dc970080961cb0854c1e9786aa8040a209b1" 
Jun 24 07:51:51 172.28.0.1 registry[2262]: 172.28.0.8 - - [24/Jun/2024:07:51:49 +0000] "HEAD /v2/vergoh/vnstat/manifests/sha256:801dd0ecec9db208f5c84e051dd5dc970080961cb0854c1e9786aa8040a209b1 HTTP/1.1" 200 3161 "" "harbor-registry-client"
Jun 24 07:51:51 172.28.0.1 registry[2262]: time="2024-06-24T07:51:51.032964897Z" level=info msg="response completed" go.version=go1.22.3 http.request.host="core:8080" http.request.id=20ceb6d1-c9fb-4c96-bd3b-ddb061f62363 http.request.method=HEAD http.request.remoteaddr=172.28.0.11 http.request.uri="/v2/vergoh/vnstat/manifests/sha256:801dd0ecec9db208f5c84e051dd5dc970080961cb0854c1e9786aa8040a209b1" http.request.useragent=harbor-registry-client http.response.contenttype="application/vnd.oci.image.index.v1+json" http.response.duration=1.806858788s http.response.status=200 http.response.written=3161 
Jun 24 07:51:51 172.28.0.1 registry[2262]: time="2024-06-24T07:51:51.479668802Z" level=info msg="authorized request" go.version=go1.22.3 http.request.host="core:8080" http.request.id=27d3cbb1-3f7f-452a-9313-d5c4e8e6c6c1 http.request.method=HEAD http.request.remoteaddr=172.28.0.11 http.request.uri="/v2/vergoh/vnstat/manifests/sha256:b8ed1b84376584c54d6db232d9d3ac65873c2af117f4eb9b75c2990bd02fc2b2" http.request.useragent=harbor-registry-client vars.name="vergoh/vnstat" vars.reference="sha256:b8ed1b84376584c54d6db232d9d3ac65873c2af117f4eb9b75c2990bd02fc2b2" 
Jun 24 07:51:52 172.28.0.1 registry[2262]: time="2024-06-24T07:51:52.958540222Z" level=info msg="response completed" go.version=go1.22.3 http.request.host="core:8080" http.request.id=27d3cbb1-3f7f-452a-9313-d5c4e8e6c6c1 http.request.method=HEAD http.request.remoteaddr=172.28.0.11 http.request.uri="/v2/vergoh/vnstat/manifests/sha256:b8ed1b84376584c54d6db232d9d3ac65873c2af117f4eb9b75c2990bd02fc2b2" http.request.useragent=harbor-registry-client http.response.contenttype="application/vnd.docker.distribution.manifest.list.v2+json" http.response.duration=1.485548263s http.response.status=200 http.response.written=1415 
Jun 24 07:51:52 172.28.0.1 registry[2262]: 172.28.0.8 - - [24/Jun/2024:07:51:51 +0000] "HEAD /v2/vergoh/vnstat/manifests/sha256:b8ed1b84376584c54d6db232d9d3ac65873c2af117f4eb9b75c2990bd02fc2b2 HTTP/1.1" 200 1415 "" "harbor-registry-client"
Jun 24 07:51:53 172.28.0.1 registry[2262]: time="2024-06-24T07:51:53.35607824Z" level=info msg="authorized request" go.version=go1.22.3 http.request.host="core:8080" http.request.id=66726525-1c6f-462a-ba6d-641087515470 http.request.method=HEAD http.request.remoteaddr=172.28.0.11 http.request.uri="/v2/vergoh/vnstat/manifests/sha256:3ab8b0d77fb181df5cd1ab4a274b97f0e436b1d0e5b7731c897e8d23bc04ad15" http.request.useragent=harbor-registry-client vars.name="vergoh/vnstat" vars.reference="sha256:3ab8b0d77fb181df5cd1ab4a274b97f0e436b1d0e5b7731c897e8d23bc04ad15" 
Jun 24 07:51:57 172.28.0.1 registry[2262]: time="2024-06-24T07:51:57.218205423Z" level=info msg="response completed" go.version=go1.22.3 http.request.host="core:8080" http.request.id=66726525-1c6f-462a-ba6d-641087515470 http.request.method=HEAD http.request.remoteaddr=172.28.0.11 http.request.uri="/v2/vergoh/vnstat/manifests/sha256:3ab8b0d77fb181df5cd1ab4a274b97f0e436b1d0e5b7731c897e8d23bc04ad15" http.request.useragent=harbor-registry-client http.response.contenttype="application/vnd.docker.distribution.manifest.list.v2+json" http.response.duration=3.871801441s http.response.status=200 http.response.written=1415 
Jun 24 07:51:57 172.28.0.1 registry[2262]: 172.28.0.8 - - [24/Jun/2024:07:51:53 +0000] "HEAD /v2/vergoh/vnstat/manifests/sha256:3ab8b0d77fb181df5cd1ab4a274b97f0e436b1d0e5b7731c897e8d23bc04ad15 HTTP/1.1" 200 1415 "" "harbor-registry-client"
Jun 24 07:51:57 172.28.0.1 registry[2262]: 172.28.0.8 - - [24/Jun/2024:07:51:57 +0000] "GET / HTTP/1.1" 200 0 "" "Go-http-client/1.1"
Jun 24 07:51:57 172.28.0.1 registry[2262]: time="2024-06-24T07:51:57.608987118Z" level=info msg="authorized request" go.version=go1.22.3 http.request.host="core:8080" http.request.id=bb5f3f57-f3ed-4de3-9d03-8d79aa0ed747 http.request.method=HEAD http.request.remoteaddr=172.28.0.11 http.request.uri="/v2/vergoh/vnstat/manifests/sha256:79e0e777365d6af717503e09bd909c626d7a379c6d6b2215b283135e2ccddf17" http.request.useragent=harbor-registry-client vars.name="vergoh/vnstat" vars.reference="sha256:79e0e777365d6af717503e09bd909c626d7a379c6d6b2215b283135e2ccddf17" 
Jun 24 07:51:59 172.28.0.1 registry[2262]: 172.28.0.8 - - [24/Jun/2024:07:51:57 +0000] "HEAD /v2/vergoh/vnstat/manifests/sha256:79e0e777365d6af717503e09bd909c626d7a379c6d6b2215b283135e2ccddf17 HTTP/1.1" 200 1415 "" "harbor-registry-client"
Jun 24 07:51:59 172.28.0.1 registry[2262]: time="2024-06-24T07:51:59.087330041Z" level=info msg="response completed" go.version=go1.22.3 http.request.host="core:8080" http.request.id=bb5f3f57-f3ed-4de3-9d03-8d79aa0ed747 http.request.method=HEAD http.request.remoteaddr=172.28.0.11 http.request.uri="/v2/vergoh/vnstat/manifests/sha256:79e0e777365d6af717503e09bd909c626d7a379c6d6b2215b283135e2ccddf17" http.request.useragent=harbor-registry-client http.response.contenttype="application/vnd.docker.distribution.manifest.list.v2+json" http.response.duration=1.484883502s http.response.status=200 http.response.written=1415 
Jun 24 07:52:07 172.28.0.1 registry[2262]: 172.28.0.8 - - [24/Jun/2024:07:52:07 +0000] "GET / HTTP/1.1" 200 0 "" "Go-http-client/1.1"
Jun 24 07:52:08 172.28.0.1 registry[2262]: 127.0.0.1 - - [24/Jun/2024:07:52:08 +0000] "GET / HTTP/1.1" 200 0 "" "curl/8.1.2"
Jun 24 07:52:17 172.28.0.1 registry[2262]: 172.28.0.8 - - [24/Jun/2024:07:52:17 +0000] "GET / HTTP/1.1" 200 0 "" "Go-http-client/1.1"
Jun 24 07:52:27 172.28.0.1 registry[2262]: 172.28.0.8 - - [24/Jun/2024:07:52:27 +0000] "GET / HTTP/1.1" 200 0 "" "Go-http-client/1.1"
Jun 24 07:52:37 172.28.0.1 registry[2262]: 172.28.0.8 - - [24/Jun/2024:07:52:37 +0000] "GET / HTTP/1.1" 200 0 "" "Go-http-client/1.1"
Jun 24 07:52:38 172.28.0.1 registry[2262]: 127.0.0.1 - - [24/Jun/2024:07:52:38 +0000] "GET / HTTP/1.1" 200 0 "" "curl/8.1.2"
Jun 24 07:52:47 172.28.0.1 registry[2262]: 172.28.0.8 - - [24/Jun/2024:07:52:47 +0000] "GET / HTTP/1.1" 200 0 "" "Go-http-client/1.1"

worker.log

2024-06-24T07:51:36Z [INFO] [/controller/replication/transfer/image/transfer.go:139]: client for source registry [type: docker-hub, URL: https://hub.docker.com, insecure: false] created
2024-06-24T07:51:36Z [INFO] [/controller/replication/transfer/image/transfer.go:149]: client for destination registry [type: harbor, URL: http://core:8080, insecure: true] created
2024-06-24T07:51:36Z [INFO] [/controller/replication/transfer/image/transfer.go:182]: copying vergoh/vnstat:[latest,2.12,dev,2.11,2.10,2.9,2.8,2.7](source registry) to vergoh/vnstat:[latest,2.12,dev,2.11,2.10,2.9,2.8,2.7](destination registry)...
2024-06-24T07:51:36Z [INFO] [/controller/replication/transfer/image/transfer.go:210]: copying vergoh/vnstat:latest(source registry) to vergoh/vnstat:latest(destination registry)...
2024-06-24T07:51:36Z [INFO] [/controller/replication/transfer/image/transfer.go:467]: pulling the manifest of artifact vergoh/vnstat:latest ...
2024-06-24T07:51:40Z [INFO] [/controller/replication/transfer/image/transfer.go:473]: the manifest of artifact vergoh/vnstat:latest pulled
2024-06-24T07:51:44Z [INFO] [/controller/replication/transfer/image/transfer.go:226]: the artifact vergoh/vnstat:latest already exists on the destination registry, skip
2024-06-24T07:51:44Z [INFO] [/controller/replication/transfer/image/transfer.go:210]: copying vergoh/vnstat:2.12(source registry) to vergoh/vnstat:2.12(destination registry)...
2024-06-24T07:51:44Z [INFO] [/controller/replication/transfer/image/transfer.go:467]: pulling the manifest of artifact vergoh/vnstat:2.12 ...
2024-06-24T07:51:45Z [INFO] [/controller/replication/transfer/image/transfer.go:473]: the manifest of artifact vergoh/vnstat:2.12 pulled
2024-06-24T07:51:45Z [INFO] [/controller/replication/transfer/image/transfer.go:226]: the artifact vergoh/vnstat:2.12 already exists on the destination registry, skip
2024-06-24T07:51:45Z [INFO] [/controller/replication/transfer/image/transfer.go:210]: copying vergoh/vnstat:dev(source registry) to vergoh/vnstat:dev(destination registry)...
2024-06-24T07:51:45Z [INFO] [/controller/replication/transfer/image/transfer.go:467]: pulling the manifest of artifact vergoh/vnstat:dev ...
2024-06-24T07:51:45Z [INFO] [/controller/replication/transfer/image/transfer.go:473]: the manifest of artifact vergoh/vnstat:dev pulled
2024-06-24T07:51:46Z [INFO] [/controller/replication/transfer/image/transfer.go:226]: the artifact vergoh/vnstat:dev already exists on the destination registry, skip
2024-06-24T07:51:46Z [INFO] [/controller/replication/transfer/image/transfer.go:210]: copying vergoh/vnstat:2.11(source registry) to vergoh/vnstat:2.11(destination registry)...
2024-06-24T07:51:46Z [INFO] [/controller/replication/transfer/image/transfer.go:467]: pulling the manifest of artifact vergoh/vnstat:2.11 ...
2024-06-24T07:51:47Z [INFO] [/controller/replication/transfer/image/transfer.go:473]: the manifest of artifact vergoh/vnstat:2.11 pulled
2024-06-24T07:51:48Z [INFO] [/controller/replication/transfer/image/transfer.go:226]: the artifact vergoh/vnstat:2.11 already exists on the destination registry, skip
2024-06-24T07:51:48Z [INFO] [/controller/replication/transfer/image/transfer.go:210]: copying vergoh/vnstat:2.10(source registry) to vergoh/vnstat:2.10(destination registry)...
2024-06-24T07:51:48Z [INFO] [/controller/replication/transfer/image/transfer.go:467]: pulling the manifest of artifact vergoh/vnstat:2.10 ...
2024-06-24T07:51:49Z [INFO] [/controller/replication/transfer/image/transfer.go:473]: the manifest of artifact vergoh/vnstat:2.10 pulled
2024-06-24T07:51:51Z [INFO] [/controller/replication/transfer/image/transfer.go:226]: the artifact vergoh/vnstat:2.10 already exists on the destination registry, skip
2024-06-24T07:51:51Z [INFO] [/controller/replication/transfer/image/transfer.go:210]: copying vergoh/vnstat:2.9(source registry) to vergoh/vnstat:2.9(destination registry)...
2024-06-24T07:51:51Z [INFO] [/controller/replication/transfer/image/transfer.go:467]: pulling the manifest of artifact vergoh/vnstat:2.9 ...
2024-06-24T07:51:51Z [INFO] [/controller/replication/transfer/image/transfer.go:473]: the manifest of artifact vergoh/vnstat:2.9 pulled
2024-06-24T07:51:52Z [INFO] [/controller/replication/transfer/image/transfer.go:226]: the artifact vergoh/vnstat:2.9 already exists on the destination registry, skip
2024-06-24T07:51:52Z [INFO] [/controller/replication/transfer/image/transfer.go:210]: copying vergoh/vnstat:2.8(source registry) to vergoh/vnstat:2.8(destination registry)...
2024-06-24T07:51:52Z [INFO] [/controller/replication/transfer/image/transfer.go:467]: pulling the manifest of artifact vergoh/vnstat:2.8 ...
2024-06-24T07:51:53Z [INFO] [/controller/replication/transfer/image/transfer.go:473]: the manifest of artifact vergoh/vnstat:2.8 pulled
2024-06-24T07:51:57Z [INFO] [/controller/replication/transfer/image/transfer.go:226]: the artifact vergoh/vnstat:2.8 already exists on the destination registry, skip
2024-06-24T07:51:57Z [INFO] [/controller/replication/transfer/image/transfer.go:210]: copying vergoh/vnstat:2.7(source registry) to vergoh/vnstat:2.7(destination registry)...
2024-06-24T07:51:57Z [INFO] [/controller/replication/transfer/image/transfer.go:467]: pulling the manifest of artifact vergoh/vnstat:2.7 ...
2024-06-24T07:51:57Z [INFO] [/controller/replication/transfer/image/transfer.go:473]: the manifest of artifact vergoh/vnstat:2.7 pulled
2024-06-24T07:51:59Z [INFO] [/controller/replication/transfer/image/transfer.go:226]: the artifact vergoh/vnstat:2.7 already exists on the destination registry, skip
2024-06-24T07:51:59Z [INFO] [/controller/replication/transfer/image/transfer.go:204]: copy vergoh/vnstat:[latest,2.12,dev,2.11,2.10,2.9,2.8,2.7](source registry) to vergoh/vnstat:[latest,2.12,dev,2.11,2.10,2.9,2.8,2.7](destination registry) completed

from harbor.

MinerYang commented on July 21, 2024

How many tags approximately in your repository when hits the 429 rate limit ?

from harbor.

BrettDean commented on July 21, 2024

About 398 tags will consume 200 quotas, I suppose.
Because this image has 335 tags, using up 168 quotas:

proxy.log

https://raw.githubusercontent.com/BrettDean/temp/main/20240624/proxy.log

registry.log

https://raw.githubusercontent.com/BrettDean/temp/main/20240624/registry.log

worker.log

https://raw.githubusercontent.com/BrettDean/temp/main/20240624/worker.log

quota_remaining.log
https://raw.githubusercontent.com/BrettDean/temp/main/20240624/quota_remaining.log

from harbor.

stonezdj commented on July 21, 2024

The GET API for tags/list consumes quota, the max page_size is 100 in dockerhub api /v2/<repository_name>/tags/list, there is no room to reduce the quota usage.

from harbor.

BrettDean commented on July 21, 2024

Thank you for your reply.

I believe get the tags list does not require quota consumption, as shown in the script below. We can even get the tag list without logging into Docker Hub:

#!/bin/bash

userRepo=$1
tag_count=0

function list_tags() {
    local page_num=1
    local page_size=100
    
    while true; do
        local tags_json=$(curl -s "https://registry.hub.docker.com/v2/repositories/${userRepo}/tags/?page=${page_num}&page_size=${page_size}")
        
        if [[ $(echo "${tags_json}" | jq -r '.results | length') -eq 0 ]]; then
            break
        fi
        
        local tags=$(echo "${tags_json}" | jq -r '.results[].name')
        local tags_count=$(echo "${tags_json}" | jq -r '.count')
        tag_count=${tags_count}
        
        echo "${tags}"
        
        ((page_num++))
    done
}

list_tags "${userRepo}"
echo "There're ${tag_count} tags in ${userRepo}"

run:

bash printTags.sh jc21/nginx-proxy-manager

It'll return the list of tags:

https://raw.githubusercontent.com/BrettDean/temp/main/20240625/printTags.log

all_data:

https://raw.githubusercontent.com/BrettDean/temp/main/20240625/all_data_it_returns.json

It won’t count towards the quota of the current IP anonymous user. After running the script to get the tag list, the quota still remains 100.

ratelimit-remaining: 100;w=21600

from harbor.

shishir-11 commented on July 21, 2024

Hi, I'm hoping to contribute to harbor and am a bit new to the field and had some question, after setting up my development environment how do i run and test the functionality of harbor and changes I make ?

from harbor.

[Feature Request] A Way to Reduce Quota Consumption for Replication on Docker Hub about harbor HOT 7 OPEN

Comments (7)

proxy.log:

registry.log:

worker.log

proxy.log

registry.log

worker.log

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent