Comments (6)
Vad1mo
commented on June 20, 2024
1
Closing the issue for now. Difficult to implement for all different storage units. Gain is little as there is an existing workaround.
from harbor.
stonezdj
commented on June 20, 2024
same issue with #15641, it is already fix after 2.5.0
from harbor.
bdowling
commented on June 20, 2024
I did see that issue, and I understand that the cleanup task is possible. However that doesn't address the issue that there is zero visibility in the dashboard that this space is being consumed until you run out of storage or that objects are incoming for a given image.
Additionally, the default of a week for cleanup of _uploads seem exceedingly excessive. Uploads either work or they don't. If they are not progressing, leaving the folder there doesn't seem to make sense as it is not clear to me if there is a way for these uploads to resume from a new client, I believe they will get a new ID most of the time and just keep consuming more space. We had a couple dozen folders in our case when the the large uploads failed for a 8GB image, and ended up consuming the 1TB of storage we had allocated to the registry largely in failed uploads.
The original docker registry docs specifically mention using the startedat file for handling stale uploads, but that doesn't seem to be a the fix here. A delta between the startedat and the last updated timestamp on data would seem more prudent to clean these files up more quickly before it becomes a problem.
So with that, the fix in the mentioned issue does not seem sufficient.
Thanks.
from harbor.
stonezdj
commented on June 20, 2024
The registry does't expose the metrics of _upload usage.
But you could try to adjust the interval and age to mitigate the current issue:
# Enable purge _upload directories
upload_purging:
enabled: true
# remove files in _upload directories which exist for a period of time, default is one week.
age: 168h
# the interval of the purge operations
interval: 24h
dryrun: false
The age could be 12 hour, and the interval could be 1-2 hour.
from harbor.
bdowling
commented on June 20, 2024
I know this got marked wont-fix, but this is unfortunate. I think the real root cause needs to be addressed somehow because I have another day of dealing with this, this is just from 2 days of partially failed pulls piling up.
_uploads:
2.9G 0115d4e8-9dd3-4661-94bd-6ceeda268588
3.0G 04c0df98-e1bb-4c6b-8a9d-dd6cc392fd72
3.1G 05f5171c-8f94-4809-bd9c-99613a42b5a2
3.3G 06b55688-6450-4956-aa0e-9b71c9a9101b
3.1G 0e331732-ff6f-4a4c-823d-cd97f879f5f2
2.9G 0e87ad20-f410-494f-8b7a-71666b44cc89
3.3G 13943e1b-7565-46bd-8fda-0d5b7cb32bf6
3.4G 1536c235-a70b-4f83-895a-6e26cffd96c1
3.0G 1744477d-e8cc-4c3c-b268-f11ce213bc87
3.4G 17542cb6-61f2-48ac-b70c-5de20e315400
3.4G 1a39d804-bbdf-48f4-9194-41d4498b113f
2.9G 1d3ef0a1-7a63-458b-aa3b-cfbb5e8dc827
3.0G 1dde2482-039e-474e-b1eb-bfb1186bd451
3.6G 1ef389d9-e3a3-43a6-b9b4-29c2c42c80e9
2.7G 21d67a6f-f808-4e15-983b-708ae6b68981
3.0G 24480726-5ddf-48df-b9a4-5bb8e8ddab0c
3.3G 2d26fb75-6efc-4b80-80bb-bb76bed2e3cd
3.2G 2e38a783-6f68-4b81-b55f-9cf17e0b7a13
3.0G 38a6cb79-ef18-412f-b2fa-d519b8d4e620
3.1G 39d9a881-a869-43a4-be8e-01818eb33ef9
3.0G 3c387eeb-43f6-4f91-9b93-4be0166eb83c
3.4G 42700497-c211-4183-b1d1-b7441e2c66a5
3.6G 49191b19-a0ad-4f5f-9107-bd5c79fe4ed4
3.3G 4b3aac0d-d305-45ac-8850-7a9d8a3ca68f
3.1G 4fb93ef7-7714-49e4-8e46-c5b7e9bc9378
3.0G 51a9aeb9-4ef4-4798-9446-09f70f9e62a8
3.7G 550750cf-e970-4231-b8b9-5b791edcd469
3.1G 55ca73e1-67d7-48db-b397-109185a1769a
3.1G 572dbe93-0caa-40b8-b1e7-47ca7333fac9
3.3G 59b178bc-9f96-4a36-9e5c-656defea94b3
3.3G 6655a998-61a9-4045-b1e8-eb0f9832742e
3.4G 66728ef4-efe7-4a89-b5c1-702fef84f9e8
3.0G 67b98b8c-9d55-4422-b6fb-3a38f275a2c7
2.8G 78ba8eee-ea35-4a30-9e9b-54ec8d95e752
3.1G 86ee861e-4c5a-4f4f-9e84-a9887c1b0337
3.0G 8ac740e4-e15e-4ce5-bbd2-c6452a6899ad
955M 8bb61f53-673b-40e8-ae59-f013f5f1528a
3.2G 8c7e2577-689b-477f-a07d-36d89d9502e5
3.2G 8f0a33e1-b4bf-4879-b644-12bbeba1d3e8
3.3G 9159319c-45f2-4450-bace-2943cea264d1
2.2G 91d22f5d-f100-44d6-9b70-fa409bf4de80
3.2G 983ce4ee-f87b-4783-9fb5-22fc9b5bb4f6
3.5G 9a961b9f-47fb-41e5-8d73-87f87a23696b
3.4G a2a9d7a6-1e77-4ab8-902d-260f9e685ddf
3.2G a428895b-f6ba-42d2-87fc-874f2b67f077
3.3G a971f31b-981a-46fd-a0e1-a0d8384c561e
2.8G aaa28cda-bb60-4a32-a4ad-4d2788a1ee8f
3.4G ab5b84f9-29d6-4b3f-8b58-82840b4f0c75
3.4G ab790993-a3b6-4712-9df2-f4ac59db8c1b
2.9G acf0572c-acb6-4b0f-97d4-0826065477ad
3.0G ad366aee-6cac-4ed7-9f14-1f16b8bad4c7
3.7G b81a6e2a-989f-4b59-a2b2-5e70c2a815a4
3.4G b8fe0c60-4f35-416a-9582-710b22fc563b
3.4G bc245685-ed34-44e3-adca-41b7f13e0d39
3.1G bd6a315c-fbf8-4254-a947-a16273c718f2
3.6G c05f7f44-3db5-4f6b-abb4-c9ff86453ecc
3.0G c254d1d0-d4cf-4fb5-9dd4-01e7b6bb35d9
3.1G c267eb92-7cdf-41a4-ab48-f129df72f824
3.1G c2dcdf02-f314-48fb-ab6f-288fc8d0403e
3.2G c4368829-4822-4bf9-a6d5-238818163baa
3.2G c9f480da-901d-48a6-87e1-dbcb90fe4743
3.1G ca4e3250-0e28-4d78-b862-a9dc9521bc29
3.0G d0cf1c4b-7aff-49ac-9314-7fe01b102183
3.5G d1c62354-71b9-4516-97e1-5191cdbf6b33
2.7G d2409bf4-caff-465e-9077-9e90a4086f73
3.1G d30480bf-be51-4f05-9787-9280d4aece95
2.9G d8810d37-9902-41d2-b544-d20de0f480b7
3.6G db892cc8-9456-49f0-a588-dbc5230cebb5
3.6G dc9138b3-3fcf-4820-ae6f-71ca240d0212
3.2G e24f895b-29af-45ea-992b-5405348c2390
3.2G e9fa7d98-3e0b-48a3-aedd-0cdb69852453
3.1G ea8098aa-41b6-418a-b53e-b29693bfc3c3
3.4G ec4687d2-4622-42f0-9f3c-54eb2addeb8c
3.4G edd15330-2702-40e1-b5a2-f99b43e3d347
2.0G f6f0e2f8-aaa1-439d-9cc0-478b870908a7
3.0G f818e5e2-d687-4639-8bb1-d9c364072080
3.1G fe59de12-1ea8-433d-b8eb-be4a5ceadba7
245G total
from harbor.
bdowling
commented on June 20, 2024
fwiw, for anyone coming across this issue. I discovered the default 30m timeout was causing large uploads to fail. A resume capability would certainly be helpful here, but probably needs to be in registry protocol.
In helm values, for core, etc:
- extraEnvVars: []
+ extraEnvVars:
+ - name: REGISTRY_HTTP_CLIENT_TIMEOUT
+ value: "60"
from harbor.
Related Issues (20)
- Sweep job fails to cleanup REPLICATION executions (foreign key constraint violation) HOT 1
- Harbor Registry endpoint is redirecting to hardcoded URL HOT 3
- Harbor 2.6.0 can't start db keeps restarting HOT 1
- Unreasonable deletion operation logic in proxy cache
- Harbor doesn't delete blobs from S3 compatible storage (ceph) HOT 3
- [Harbor v2.11.0] error while putting blob to local repo, http status code: 500 (S3) HOT 2
- Korean language translation cannot selected in the UI
- Garbage Collection without Harbor GC
- [Bug Report] The banner message is causing the search bar to be obscured.
- Harbor Retention Policy Rules Fails for Multiple Wildcards Repositories
- Bump up helm, ORAS, CNAB, imgpkg, cosign ... version in Harbor 2.12
- Errors found when upgrading Harbor & migrating the database: 'ERROR: function jsonb_path_exists(jsonb, unknown) does not exist' HOT 2
- Scanning related inconsistent UI display issues HOT 1
- loggerSweeperDuration does not clean the /var/log/jobs files.
- Enable OpenSSF Scorecard to enhance security practices across the project
- Garbage collection job is constantly in pending status HOT 2
- use nginx proxy
- Expose ssl_protocols in harbor.yml to allow users to run harbor with only TLSv1.3 enabled
- docker login failed: login attempt to https://xxx/v2/ failed with status: 401 Unauthorized
- [Feature Request] A Way to Reduce Quota Consumption for Replication on Docker Hub
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from harbor.