When push artifacts to the project we met a lot of errors, n

hi <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="

The issue still persists <div class="highlight highlight-text-shell-session notran

I met a lot of redis: transaction failed and <code cl

<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/us

Failed to push artifacts to the project ,about goharbor/harbor

Comments (23)

chlins commented on May 24, 2024 1

The original quota update relies on the optimistic lock based on Postgres, which may affect performance under high concurrency. However, you might want to try the quota update by redis introduced since version v2.9. It uses redis to implement the optimistic lock to reduce the pressure on Postgres, refer to https://github.com/goharbor/perf/wiki/Quota-Update see more details.

from harbor.

chlins commented on May 24, 2024

Please attach the harbor database logs as well.

from harbor.

microyahoo commented on May 24, 2024

We use the Aliyun pg database, which can not be downloaded. Viewing the log is anti-human.

2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 400
-- | --
2024年1月25日14:10:39 | LOG: process 3993 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1004.259 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3904 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1004.192 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3542 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1004.649 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3989 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1004.112 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3991 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1004.017 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3992 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1003.950 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3879 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1003.868 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3983 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1003.790 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3746 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1003.715 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3980 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1004.045 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3985 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1003.606 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3984 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1003.542 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3986 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1003.467 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.

2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
-- | --
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3944 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1003.400 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3687 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1003.817 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3943 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1003.296 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3829 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1003.364 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3383 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1003.146 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3928 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1003.186 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3886 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1002.299 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3028 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1000.561 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3835 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1000.713 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3413 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1000.412 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3673 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1001.430 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3913 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1000.267 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 4010 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1000.275 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3998 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1000.187 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3997 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1000.172 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3918 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1000.121 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3909 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1000.701 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3906 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1000.109 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3907 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1000.465 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3162 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1000.106 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3559 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1000.092 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3841 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1000.086 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3978 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1000.214 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3982 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1000.107 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3977 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1000.205 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3324 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1000.071 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3782 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1000.091 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 4102 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1000.069 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3106 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1000.038 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | LOG: process 3982 acquired ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1508.515 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | LOG: process 3907 acquired ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1432.161 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | LOG: process 3324 acquired ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1379.138 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | LOG: process 3978 acquired ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1327.658 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | LOG: process 3977 acquired ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1272.214 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | LOG: process 3909 acquired ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1222.151 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5

2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
-- | --
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3877 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1028.971 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3728 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1028.456 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3843 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1027.949 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3863 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1027.438 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3722 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1026.937 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3654 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1026.453 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3842 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1025.959 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3735 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1025.499 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3933 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1024.995 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3803 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1024.544 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3571 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1024.053 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3778 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1023.613 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3157 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1023.161 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3437 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1022.729 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3387 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1022.293 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3988 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1021.872 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3435 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1021.459 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3229 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1021.054 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3336 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1020.644 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3637 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1020.227 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3798 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1019.838 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3636 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1019.461 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3613 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1019.088 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3609 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1018.739 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3058 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1018.374 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3979 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1019.701 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3271 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1017.977 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3704 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1017.619 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3103 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1017.271 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3464 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1016.940 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 4102. Wait queue: 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017, 3930,4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597, 3614,3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3503 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1016.601 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | LOG: process 4102 acquired ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1018.196 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.
2024年1月25日14:10:39 | LOG: process 3119 still waiting for ExclusiveLock on tuple (2,62) of relation 19211 of database 18754 after 1016.172 ms
2024年1月25日14:10:39 | STATEMENT: UPDATE quota_usage SET used = $1, update_time = $2, version = $3 WHERE id = $4 AND version = $5
2024年1月25日14:10:39 | DETAIL: Process holding the lock: 3524. Wait queue: 4102, 3687, 3106, 3782, 3909, 3977, 3978, 3324, 3907, 3982, 3980, 3542, 3981, 3462, 3261, 3326, 3673, 3841, 3829, 3928, 3196, 3559, 3979, 3886, 3401, 3162, 3383, 3943, 3944, 3986, 3984, 3985, 3906, 3746, 3983, 3879, 3992, 3991, 3989, 3918, 3904, 3993, 3995, 3882, 3997, 3994, 3066, 3648, 3030, 4005, 3996, 4009, 3998, 4011, 4007, 3999, 4008, 4017,3930, 4016, 4015, 4010, 4013, 4018, 4014, 4012, 4019, 4021, 4020, 3500, 4023, 4025, 4028, 4027, 4022, 4031, 3913, 3509, 3548, 3247, 3720, 3856, 3674, 3021, 3505, 3650, 3574, 3726, 3292, 3561, 3405, 3221, 3413, 3086, 3699, 3431, 3702, 3833, 3391, 3410, 3560, 3541, 3249, 3623, 3245, 3310, 3638, 3119, 3503, 3464, 3103, 3704, 3271, 3028, 3058, 3609, 3787, 3613, 3636, 3798, 3637, 3336, 3229, 3435, 3988, 3387, 3437, 3157, 3778, 3571, 3803, 3933, 3735, 3842, 3654, 3722, 3863, 3843, 3728, 3877, 3460, 3563, 3649, 3915, 3615, 3393, 3835, 3063, 3655, 3111, 3837, 3711, 3350, 3644, 3067, 3950, 3526, 3597,3614, 3772, 3282, 3211, 3429, 3328, 3899, 3747, 3826, 3874.

from harbor.

microyahoo commented on May 24, 2024

The failure rate is high when push artifacts to same project due to db lock

🍺 /home/DEEPROUTE/liangzheng/go/src/github.com/goharbor/perf ☞ git:(main) ✗ go run mage.go run push-artifacts-to-same-project      

          /\      |‾‾| /‾‾/   /‾‾/   
     /\  /  \     |  |/  /   /  /    
    /  \/    \    |     (   /   ‾‾\  
   /          \   |  |\  \ |  (‾)  | 
  / __________ \  |__| \__\ \_____/ .io

  execution: local
     script: ./scripts/test/push-artifacts-to-same-project.js
     output: -

  scenarios: (100.00%) 1 scenario, 500 max VUs, 10m30s max duration (incl. graceful stop):
           * default: 1000 iterations shared among 500 VUs (maxDuration: 10m0s, gracefulStop: 30s)

ERRO[0417] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0511, error: failed to copy: unexpected response: 500 Internal Server Error  source=console
ERRO[0424] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0355, error: failed commit on ref "manifest-sha256:17263e1b56449c2da9b46d05a0d347a2cd5cc2d4c113146c3556c072fea53b70": unexpected status: 500 Internal Server Error  source=console
ERRO[0424] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0518, error: failed commit on ref "config-sha256:ad36a672e2e7eb51fec8e3b02489569ca75f73953d349617ba6a4a930f00665c": unexpected status: 500 Internal Server Error  source=console
ERRO[0424] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0336, error: failed commit on ref "manifest-sha256:8d5ff1ef3e7f6f95ad6245a698b608a62a28125e386278f7c152bf66c82a16e4": unexpected status: 500 Internal Server Error  source=console
ERRO[0424] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0514, error: failed commit on ref "config-sha256:88d554378c8356d4a84865c20c7f2bb464f087abbcb1ee902c885925b4ffaa3a": unexpected status: 500 Internal Server Error  source=console
ERRO[0424] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0113, error: failed commit on ref "manifest-sha256:99af304c45bff60789594d2bdf1cd51702304d65380179dd6ad1eee386433495": unexpected status: 500 Internal Server Error  source=console
ERRO[0427] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0295, error: failed commit on ref "manifest-sha256:ffb8572bb947b57ccd22bd768a14c6ace293a9628501d67869937e36f90f4859": unexpected status: 500 Internal Server Error  source=console
ERRO[0429] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0345, error: failed commit on ref "manifest-sha256:3465648934477b6e21d69e221feb6b7cd2812cc1c38d1c9bb6ea518b636ce735": unexpected status: 500 Internal Server Error  source=console
ERRO[0429] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0082, error: failed commit on ref "manifest-sha256:2406c9a257ca8c0a258edd4501ebe42c6b8bbb3cd20de5ba309ef74df471f990": unexpected status: 500 Internal Server Error  source=console
ERRO[0429] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0133, error: failed commit on ref "manifest-sha256:7f5db2e3fcb83b256609ae6e8aa7853279ff493ad93644dc021d8b07447ebcc8": unexpected status: 500 Internal Server Error  source=console
ERRO[0429] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0148, error: failed commit on ref "manifest-sha256:dcd5c9f53ca1d3d6cf6393dbe0eb1249afaeddf5ba994e8eeffec47bb0668f7f": unexpected status: 500 Internal Server Error  source=console
ERRO[0429] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0305, error: failed commit on ref "manifest-sha256:0a402ed28939acc0adca283a8a1cab1871b85899257ac2c064be9adc1d39716b": unexpected status: 500 Internal Server Error  source=console
ERRO[0429] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0056, error: failed commit on ref "manifest-sha256:6e17ff3eb86d67d101c3fc6b71d6c472d5b026a59d18d2a8678d505790aa3ee5": unexpected status: 500 Internal Server Error  source=console
ERRO[0429] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0237, error: failed commit on ref "manifest-sha256:7b494be7bd831226100b1f2d0d771c2cc77034ed16f5f24c6348c6b79e114651": unexpected status: 500 Internal Server Error  source=console
ERRO[0429] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0149, error: failed commit on ref "manifest-sha256:f60af2805bb29b3f566479c2b8f0cf87cd719b3e78d9c0b98609fa28dbf86ae0": unexpected status: 500 Internal Server Error  source=console
ERRO[0429] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0327, error: failed commit on ref "manifest-sha256:86dab818e990813711640ab74a0251c5f9a180950ae32256c28334fb5398d1f1": unexpected status: 500 Internal Server Error  source=console
ERRO[0429] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0073, error: failed commit on ref "manifest-sha256:28d63a012c46712a73052bb00c893c3b01f38ad6ca4befff4e6c84c9da580888": unexpected status: 500 Internal Server Error  source=console
ERRO[0429] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0176, error: failed commit on ref "manifest-sha256:fe1cc89255037d70a765819ce8848984afa00e46c440f5e7bfc362cd349bcb07": unexpected status: 500 Internal Server Error  source=console
ERRO[0429] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0157, error: failed commit on ref "manifest-sha256:a02d47b7ed6cb628939a6c1c04a9916b8b4f5ac4c2382dc7c70db8fa929751ec": unexpected status: 500 Internal Server Error  source=console
ERRO[0429] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0131, error: failed commit on ref "manifest-sha256:fd284c2565c288aed95b987de078d58115decfc140efb96d042df94f5697a513": unexpected status: 500 Internal Server Error  source=console
ERRO[0429] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0296, error: failed commit on ref "manifest-sha256:9216ffb0362bd193b8b9ed916703781b082208969a91f554879b4a0821cc9c78": unexpected status: 500 Internal Server Error  source=console
ERRO[0429] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0101, error: failed commit on ref "manifest-sha256:678ae08528cbc780333dd1acf499190e8ce4af3d971a2623133546e3349284d2": unexpected status: 500 Internal Server Error  source=console
ERRO[0429] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0524, error: failed commit on ref "config-sha256:b8d24567e7997cb96761a951a006f4ef90da767c09dfd5642a7f5cff76b2a0ff": unexpected status: 500 Internal Server Error  source=console
ERRO[0429] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0416, error: failed commit on ref "manifest-sha256:3735ae81d5f8d761b6c52d5a4d5559de3eb9d99f59b5040ba5c8fd957ca63746": unexpected status: 500 Internal Server Error  source=console
ERRO[0429] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0180, error: failed commit on ref "manifest-sha256:1989b7538c7225adb1ae1e2994e9cd10cd53eb425f005dba3669dc65f0e28e0e": unexpected status: 500 Internal Server Error  source=console
ERRO[0429] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0323, error: failed commit on ref "manifest-sha256:87d248ce8da84a2b4333ddd27dddef08fe7e4eb0fe57dc0d06623a90fbf720ab": unexpected status: 500 Internal Server Error  source=console
ERRO[0429] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0513, error: failed commit on ref "config-sha256:f0a3d60b2db4c24226c50e80c2cd7efcfed85915a8cbb1f63a38038e27b56fbd": unexpected status: 500 Internal Server Error  source=console
ERRO[0429] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0308, error: failed commit on ref "manifest-sha256:9da1db507739d0dd64fb121989a5bc1f29c636456eae25a5f39452f326d1196e": unexpected status: 500 Internal Server Error  source=console
ERRO[0429] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0437, error: failed commit on ref "manifest-sha256:631e8203a8fc809c9894da1c2c933571679bf3882ec84cd2375b982508bf2e61": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0076, error: failed commit on ref "manifest-sha256:c1249c83ee47aaa5967daa79d3a3377cfd3699b51040dc01b2feff57ec4383bb": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0349, error: failed commit on ref "manifest-sha256:03ad104a3f8327c88555c60cf68d49a25b1ac10b9e0810186d8791ad2e1d7335": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0529, error: failed commit on ref "config-sha256:a78feb8f4f0c09a6982ed950d00d5269622a92182c9eae29cb092c022c81823b": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0543, error: failed commit on ref "config-sha256:7910a5581e482972b51cacdade4f1b3e1d428b110e69d1fa397a156b2f651917": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0187, error: failed commit on ref "manifest-sha256:18c17731ec5ce07560193a8ce787d3db018fc40aebd1519951387c3a27af596a": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0531, error: failed commit on ref "config-sha256:9bf9233f8a116e7af6d8b8adfba5ede7196625ca0501511eef106d6095024799": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0094, error: failed commit on ref "manifest-sha256:eff5657b392b0e8b3766c0df0e3f9465fac4da262002a979da95e4a993b4a3ae": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0487, error: failed commit on ref "manifest-sha256:267d5a9a002d0c31c5a892e8b58263c5aff359d50b48f94c2f6e68433b2ed91e": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0516, error: failed commit on ref "config-sha256:23eff14183848416f89001bc5cea0ae4bf2f93ec6f707f667a238751f0d223cf": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0055, error: failed commit on ref "manifest-sha256:09f281e15f37af6d7cbf9509eb71b5858d7a64d8488825d5e712e9ccd4cc2af3": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0412, error: failed commit on ref "manifest-sha256:d195036926225418dc07b1e669a17960fdca63d567ec6d403870f55d6ac99d28": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0304, error: failed commit on ref "manifest-sha256:d8445a6875f7a672123d6650b23ef53a8fa8b3eb656da9ed8ea405e60dbcad4a": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0217, error: failed commit on ref "manifest-sha256:fd5b75907618517c1aa908957d116bd62357563f8a893179c35c6ca1c11358f9": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0538, error: failed commit on ref "config-sha256:b80816852552346c218e9f103ba72b7a90ccc588b40d2fea79e046bf61d6aff3": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0464, error: failed commit on ref "manifest-sha256:1a13ff5419f9cf7adf56709493d62b7af18a09cb943f8584bf8df79ef17827bf": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0166, error: failed commit on ref "manifest-sha256:e5c976345d4518013d768d3512db45e4f8cd1be1e765677c519bd38b18536655": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0002, error: failed commit on ref "manifest-sha256:de6e4cb3feef46c16196588cc3667f6349c7a77ee8b1295d24a95e9c9bb8f9db": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0151, error: failed commit on ref "manifest-sha256:ae7c0b0a36f7025e54eb167defa4b9087bd38f7e2cad67db72bff2f9c807b2ec": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0303, error: failed commit on ref "manifest-sha256:c366ab6bab90d113c3bbf0f9c3e2a7e2347b77bfd3323d5c2cf36d8b83d7e3b5": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0322, error: failed commit on ref "manifest-sha256:f7f5bcb91f7276ad64c1e7c0435fb0fdd2f9b191dbfccc95798c8e6dc3d97f5f": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0449, error: failed commit on ref "manifest-sha256:b904d880740afef32a5c05c366c23346e8262626b14abc2270ecaa43299a8d12": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0306, error: failed commit on ref "manifest-sha256:8b5b0e3a2c85b3956f9aebe2fa30a9f9262bd48ff3625b8ffa1f29077be14183": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0520, error: failed to copy: unexpected response: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0539, error: failed to copy: unexpected response: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0052, error: failed commit on ref "manifest-sha256:78195992b34d6d4021f8491e8974790b247fc0d5d179f48dbb38b04e4b8545fd": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0326, error: failed commit on ref "manifest-sha256:f9d3926603d72ad85bc364cebcbf340099c40ee8215795ad7080c5fbbe9f243a": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0525, error: failed to copy: unexpected response: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0481, error: failed commit on ref "manifest-sha256:42f3853aed18ce4c6abbdeb8132229e4ac57c9dc67288b9da652bf6796fb4e3f": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0519, error: failed to copy: unexpected response: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0522, error: failed commit on ref "config-sha256:b8bb9cd006b016fb523370326fc73ff1987f47f1c0c264be2e4c728562aa51d8": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0179, error: failed commit on ref "manifest-sha256:c695aa6d5a5009215a71079a81b901f01d18c68bba048eae895a7431f5d1adda": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0255, error: failed commit on ref "manifest-sha256:f3671dea8b10f109543dcb6d16c5ccdd554277a9a4f4451901605939b8f12008": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0497, error: failed commit on ref "manifest-sha256:24500a461a3781f0bf36f5b9f41b68a96ccadfd0f3510a23427ae0ae5a83c019": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0542, error: failed to copy: unexpected response: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0475, error: failed commit on ref "manifest-sha256:ccf886a5691b8282c1e6cba76f4cc56b35c7686ffe1e800844ee5d790e83e836": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0120, error: failed commit on ref "manifest-sha256:839132229aa55a77c1a7c8aab8214187d3dc432cea6c2ec5ef9ebf61e662b9f9": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0234, error: failed commit on ref "manifest-sha256:eabcc03f4ec0643899a37a6f4184d5e03f7095dffb93d8af877857faf19769f3": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0049, error: failed commit on ref "manifest-sha256:a1c355f945eddbe7159cb6b3efd11f704b0d0c3d6ce8dcd0a087efdcc3000e9a": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0299, error: failed commit on ref "manifest-sha256:873c9d1759f0d80ef3cbff6a0548b39fe21a93aeb15627ce549a6018461b3fd2": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0265, error: failed commit on ref "manifest-sha256:cbc387aba1bc93903be3d2797779c0205a2ec80146174fe202920d65a9e572d7": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0271, error: failed commit on ref "manifest-sha256:01198418e69aaab9e75a8fcc39fd4344f17279049a43e9883276b70d730fb72c": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0310, error: failed commit on ref "manifest-sha256:36f0c02f9d87f81c3cb7be80113541b7fb3ae4527936527b3b4eebb12b9f8021": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0454, error: failed commit on ref "manifest-sha256:bc5f7fc520aef0b03e2571e6c0b7b05afd6c176cc0f9f658dfe3f36bbf799da6": unexpected status: 500 Internal Server Error  source=console
ERRO[0446] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0407, error: failed commit on ref "manifest-sha256:6a8ffd15bbff2180e6ed0b304cf58b77c299c99d85b7988661afc428411f3181": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0429, error: failed commit on ref "manifest-sha256:a94f01125f3a3b32dd2a7e00456af48227c749a8ca0b264da72bdb781fb5fba1": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0535, error: failed commit on ref "config-sha256:cade28afe2a5f2aaf67ff996e18290c8b2a1258a2854936a88120ccda9367e46": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0314, error: failed commit on ref "manifest-sha256:76b56af91a674692d586892a12aa408a64a73711c2e02be30f284e92a55cb76c": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0425, error: failed commit on ref "manifest-sha256:42cd412fdb1223ea83d06e27e1e54f58aa54714786e96221ec1a7666016fd904": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0391, error: failed commit on ref "manifest-sha256:a312a2252d146595d6fc71e19938d58599e6feecae79ede005b5456b584d5730": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0331, error: failed commit on ref "manifest-sha256:1680a34498bfafea9ecb499b473d6adc3aabd39efb57b467fb366acf22c89593": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0256, error: failed commit on ref "manifest-sha256:6027deb5915fae7981d9dad12e24c047f36d693cd8c29029421bd551adb5eb83": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0122, error: failed commit on ref "manifest-sha256:fce3c3d70c0ac101a02166b565cc8d867db32ebf17ffb97bc1ce77670c8896fb": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0160, error: failed commit on ref "manifest-sha256:58ffb22aade12e6509dea19ccdc9a3f14f42c71c67256ca9fa8f7b8caa760541": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0278, error: failed commit on ref "manifest-sha256:0a8661bb08501a8d07e8a6878cf3b9015c9fd86e101be1eb412948aa6c10fa92": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0540, error: failed to copy: unexpected response: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0335, error: failed commit on ref "manifest-sha256:b90164de30796fd57715862df6c398ed235fcedaeebd4c464f7c0b6a09578a34": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0275, error: failed commit on ref "manifest-sha256:503636bf93f1ef977296dbe30e1647dd22ab819be54e99de64ada91661b4ee67": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0328, error: failed commit on ref "manifest-sha256:e4f05c8f2d5d62d9a87d01ae000eb17cc70317f34c2084b58dbaf6db38eff3f0": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0526, error: failed commit on ref "config-sha256:06f3f353d3472054261c53d9a4d79042976f0cc4dc73bb71936f7709f02bb817": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0245, error: failed commit on ref "manifest-sha256:82bd67a7f9582514d7c815bfb5613db7032b561d94b557a47883492e57fabb49": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0249, error: failed commit on ref "manifest-sha256:b9af0b0dcfb58a626f58c80b76da2e10b5a3b71710709a99e9ac54d2760f047f": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0291, error: failed commit on ref "manifest-sha256:38a77a83c78b1552ebe4350b55be41c7f5ae3d531d984c3a96e8c42efbd165eb": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0512, error: failed to copy: unexpected response: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0506, error: failed commit on ref "manifest-sha256:75d7f3f1d27c6cbe165f8cf900faaa43ac1ba39823e06fa17ca927aa525b95b4": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0515, error: failed to copy: unexpected response: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0072, error: failed commit on ref "manifest-sha256:7db9376226f3c11d78a2d03f79564d7d9d486f8fbf181f1cb881cab798c919c2": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0205, error: failed commit on ref "manifest-sha256:cabbe1b9ac8ffd6999debb567966bdb7f0e7d7e11c836cbb27b76dd25a2a25ec": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0510, error: failed commit on ref "config-sha256:ca86bbaf1a0321cf39607cce6621e0c026622322e7600a68d6a630b92ed5ab9e": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0532, error: failed commit on ref "config-sha256:62b80052c371e9d92f0839aa49883877a000e8a5d7a1b7ec9ae2a1256e8f37ad": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0380, error: failed commit on ref "manifest-sha256:13786d65d87bbd45e5da5974c0949d5217a9416e18edb0655f8306f0eb68d561": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0479, error: failed commit on ref "manifest-sha256:a6b54a0f683d597337f2f13d9f1a09ed4a44d177a562178bee82b8e96e5cd2e6": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0517, error: failed to copy: unexpected response: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0408, error: failed commit on ref "manifest-sha256:a72d00fb5d185c01e14f770c296a70f5b313d787289e9b38bfe72c68d5c50360": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0435, error: failed commit on ref "manifest-sha256:1365cfc8f8f17ef6fbf0f725306f51ec02a21926e1a963928d793384a3c66556": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0541, error: failed to copy: unexpected response: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0530, error: failed commit on ref "config-sha256:11f2695280b5a0e9d48399b8f58af75440bd2ee6a051537eaf9b936bcf7718fa": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0528, error: failed commit on ref "config-sha256:2a3f90e935a390968082bfebe911327fd5d14872121aeb1450a24a52ee05f85f": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0534, error: failed to copy: unexpected response: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0041, error: failed commit on ref "manifest-sha256:4b726cc20af511d2e44181d24ff0216b5b2501dee2ad00b94282e3613861e4ea": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0459, error: failed commit on ref "manifest-sha256:f729c185311bfab22c6ee90627806a7274a33cb78a5db1c7ab2b99c8707ff8b0": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0373, error: failed commit on ref "manifest-sha256:5efc615ec0affd00c375c5eefc50577decc62ee0f70b51e023269d4b1a7a87b2": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0228, error: failed commit on ref "manifest-sha256:9ac800c13563885e545b3b7b17723def1d029ff0d6437e2231ca1edab3b20124": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0023, error: failed commit on ref "manifest-sha256:4462aff82f9e7447eeb055f62cc29ce287644a77001d7b20096e14b2a97bb09f": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0130, error: failed commit on ref "manifest-sha256:48cb2e1e38693e7e01b3ce5d46be2200feb8d80ef5fc636298754183487f81cf": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0182, error: failed commit on ref "manifest-sha256:9b2ef6b4abec6ab67703b031f8b0e434699fe5e39c52019bd43e7f8d77f23060": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0527, error: failed to copy: unexpected response: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0276, error: failed commit on ref "manifest-sha256:f1444ef7ac5a43b127c1e002a4cae0d5163acca4869fafbe7d4e4785a2997676": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0401, error: failed commit on ref "manifest-sha256:1243d172be35055f4b162761c7cf05fa5841b771301d7558f133e2bb37d4364b": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0458, error: failed commit on ref "manifest-sha256:9e1bb0dc3f5bf58a0ebd80785ecfc41c40d30abfedf97b3d97359e92b5772771": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0365, error: failed commit on ref "manifest-sha256:3f30af72412e21763e8a610d99417e33e79334fb3ccde23ddbb3d72dc1d06011": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0125, error: failed commit on ref "manifest-sha256:10428d60d448e5094bf152795811a9a9d8b2a8cfbbd47be61feb2d765983f906": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0033, error: failed commit on ref "manifest-sha256:d4605cf3ddeddeb2aa15c32cc7407ef9c71a197d65604462e661748749083589": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0274, error: failed commit on ref "manifest-sha256:5d4b0982628d52fdf89f21eecabae321a34a7aa49f72975ddf5dc17f405fb7e4": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0096, error: failed commit on ref "manifest-sha256:7578ca7756b55654ae87bd285f0246684a2ef974c9d2d8ef3457c02545539c41": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0090, error: failed commit on ref "manifest-sha256:346f2b2a22a1d50d51fcc0fac2c80334f1177e5f01f01e17524eac14f5a0a642": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0302, error: failed commit on ref "manifest-sha256:14a801e1fa90b5c43b97057a604c97a1327020a2912af31f8bfcf170fb700ca7": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0171, error: failed commit on ref "manifest-sha256:8a6eb59cc87279e05fffee4cd87cd4a03961f9509b149b23eb17758f6a0b33f1": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0162, error: failed commit on ref "manifest-sha256:a3bde802d99d84a12bbc2eeba34a3a4cab23581d54cca66741bd25331cd10596": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0357, error: failed commit on ref "manifest-sha256:bc55a87bfedd7e0d16af10a2051e61ee2f0e4c033bcbe5535968f0cc9779fc1a": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0164, error: failed commit on ref "manifest-sha256:19edaefe5f01538e9522f6b68b3b54ddc990b6012e9c1041c41e22ba223e16ef": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0226, error: failed commit on ref "manifest-sha256:7ded8785c8956bcf6ca47306de0b6488b8df9ce679db6bb3da8a9bb17144c822": unexpected status: 500 Internal Server Error  source=console
ERRO[0447] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0286, error: failed commit on ref "manifest-sha256:1b972266d14e4e5cc27cb7847ff356e86783757d7edb9e6fbd619cbf6b1f4d5e": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0246, error: failed commit on ref "manifest-sha256:c77a7b81d5bcb63d445bd7b3e8ef10d9d5ca26a73d2ac6bd47ea89ebc181ea58": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0312, error: failed commit on ref "manifest-sha256:f56ef3b70ce3b47bb657efaadf28464a9128ba0311bea14c850c15c86c0e369e": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0478, error: failed commit on ref "manifest-sha256:9c68630a1ab35d77453514cd6dee0efd6b3144efb847e1a453df72486667a841": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0088, error: failed commit on ref "manifest-sha256:5104ce5e3e058976ed18b05cc568fe38587acef648f65edcd9eba7fb74863c5f": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0521, error: failed commit on ref "config-sha256:01390157cef7afcfc05a78c3d97a08c1b0a804f149e49e51e5e6117f2a4db59b": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0290, error: failed commit on ref "manifest-sha256:215ed7a2a9368a5f1337de7a4e4c1a319251d76f4d0eaf4242c5df3fc72f6fec": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0272, error: failed commit on ref "manifest-sha256:f17c700bed48460c7b2a7e9b3f6e402261cf79d8a4f4dee17c581806dedb0f93": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0285, error: failed commit on ref "manifest-sha256:34ec8fd54aecdd994f8a5ce27372a789c13e1b242cf62797b6535896e3957774": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0165, error: failed commit on ref "manifest-sha256:62045737ce51926ebbf59f5fd157204b0f10ce1859cee335d9d9f657ccb37e22": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0198, error: failed commit on ref "manifest-sha256:9b774980bb26d8180b38d165d84bba78c4c1662613addcd4fe3ac3d93f671ea1": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0545, error: failed commit on ref "config-sha256:a43636293d914cd9fc581cbed39892db264d3d544befb06cf791f987abd61792": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0394, error: failed commit on ref "manifest-sha256:b602d44bfa1620208414f54ac21ad73945b3345122bc8e21a2406a2c4beaabfc": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0537, error: failed commit on ref "config-sha256:4ce29dd59b6e3786225b0272eaa40c5fb4d9fefc30d5d28808503f922737e53c": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0075, error: failed commit on ref "manifest-sha256:1ca997ff0b05566e4304cdaddc1086096544543b6b2ab7ba725ef1c34263ba91": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0093, error: failed commit on ref "manifest-sha256:93e52eb6ac248f03898e5b326c6249ed8e8850c10280d7a4caa2389ff84acc97": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0472, error: failed commit on ref "manifest-sha256:01dc3b40cdf6ef33320979cb77adfcc1592c5ee4dc9ecea77ebb401d9da6410c": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0186, error: failed commit on ref "manifest-sha256:b2b5c2522be33167b3ae4912d458378237e90f49debca8b8569574585ead3cbe": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0190, error: failed commit on ref "manifest-sha256:f10b0d42d4a1a112cf6806c06b034036e098c00e6dae5fc620c63843496099fe": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0546, error: failed to copy: unexpected response: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0231, error: failed commit on ref "manifest-sha256:5ece6ea8611f3d88bd76de671d5c73cffbfdcc59f7fdca7209af0904dc092dcc": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0132, error: failed commit on ref "manifest-sha256:df1e172c1444482cc1d0c03ace4f7cb4fa45152bc0f4080d709ca38d8f2a658e": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0447, error: failed commit on ref "manifest-sha256:c95fd7cc8e382c5e7775595e5a18c7a74ced2ce5f0c8a93b323037c57013971d": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0112, error: failed commit on ref "manifest-sha256:06b7f0a7174277c738f573b0f3c357e5a26b0bb7977043446cc5884d16aeb917": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0230, error: failed commit on ref "manifest-sha256:34d1edc3db4fe1aaba5b64e37f9cc38394b7ca153af96a0dd12c6b6a47843d5d": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0270, error: failed commit on ref "manifest-sha256:9b28367e7047ab231b693aa1afc1bfa25e40bd4952a99b2c799bdaa670ff2d1d": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0224, error: failed commit on ref "manifest-sha256:f2ffc822fb4c5fa512ed7ece770ea798f4377b37e9aef90fdf0d776681698362": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0207, error: failed commit on ref "manifest-sha256:f904af368c37a9aa96cdbcefe7896a5ed3b0e99fb84ce4b111d94d5367d58a8a": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0397, error: failed commit on ref "manifest-sha256:7a8eb42c3e0340cc0ded0a6dd580517607d66eaaa30edfb6dd79b038341091a3": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0533, error: failed commit on ref "config-sha256:c8c5cfce47254161078fa683caf49356490d4c75da35fe03fe52382181d609fb": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0334, error: failed commit on ref "manifest-sha256:2c591fbfcb990a540c907ac5f98754cabcb4261045de35694a4ee526326bf741": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0434, error: failed commit on ref "manifest-sha256:d94c83023b5d2beff49319aa7de2e526cd8fb1c223b9bed4d74be2adb0b65c54": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0450, error: failed commit on ref "manifest-sha256:c3b25a7263d27e5bdf6c59bcc547fcb2f81ced620e23cd30e53f91f6d7824184": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0220, error: failed commit on ref "manifest-sha256:207ada37501e2905344a1c68079d2e939be59cfe338ac3836ea3dd4385b3d6ce": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0124, error: failed commit on ref "manifest-sha256:e3daab9ffb20669e502588d5c84d9bde406d0fb6fbde5d4f9d6a614bc50a1e8e": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0037, error: failed commit on ref "manifest-sha256:adcb6d9b3a356393e8532397f6e5901a3966573b51a586c837ce096f9eda4f5a": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0211, error: failed commit on ref "manifest-sha256:65f339fb257b4266e12772d90b6cc2101f0ef76eb9c2bc72708b38fe3e9cd79b": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0463, error: failed commit on ref "manifest-sha256:fab23292a7dc82aa6dc0b41f5b259b377b859fe4bf8bae3f68ba7a5b29be2ba1": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0006, error: failed commit on ref "manifest-sha256:d4382da4591838715d4b8c2dc71c50e67fbd78afd24f116be395138b404e3e82": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0008, error: failed commit on ref "manifest-sha256:efc7be4c11b98f37052c33a57b9d05ffeb04858af3361377de45e133de0f1e4f": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0411, error: failed commit on ref "manifest-sha256:89ddc75cfc9cc6e0c11d3296781b4ea4557eee94ca329ff212992653725367be": unexpected status: 500 Internal Server Error  source=console
ERRO[0448] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0243, error: failed commit on ref "manifest-sha256:a101aeb11de4916be2c99295310798ccb5ebb8900aa59021a071a4b47a849fdd": unexpected status: 500 Internal Server Error  source=console
ERRO[0449] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0500, error: failed commit on ref "manifest-sha256:c85b3c4317837e6b76cae720ae240edc327110d5f45c3f22f40e365d502ca88a": unexpected status: 500 Internal Server Error  source=console
ERRO[0449] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0150, error: failed commit on ref "manifest-sha256:cae612d67fe702539fe8a89cbd456f3bbf238d19b9871cfce9e70a892c753339": unexpected status: 500 Internal Server Error  source=console
ERRO[0449] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0293, error: failed commit on ref "manifest-sha256:1c8a72bf2485ef8702c8d7ed27c41502ddd6fb4ee7820c6f14d6faf869997b2f": unexpected status: 500 Internal Server Error  source=console
ERRO[0449] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0204, error: failed commit on ref "manifest-sha256:b7dca79fbc95a852df48c5d357bb9aba0436ccabdc0f071cda6e9e2925537130": unexpected status: 500 Internal Server Error  source=console
ERRO[0449] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0185, error: failed commit on ref "manifest-sha256:88f8fe5dd01a941d6a8cbc04f369f27eaccc79a7c453232c1378b5bb0c1a5235": unexpected status: 500 Internal Server Error  source=console
ERRO[0449] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0523, error: failed to copy: unexpected response: 500 Internal Server Error  source=console
ERRO[0449] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0007, error: failed commit on ref "manifest-sha256:b93e0a3cc204631ed07eed20bee6a3976e10c8aeae637184fb7ca2ec9e043d55": unexpected status: 500 Internal Server Error  source=console
ERRO[0449] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0369, error: failed commit on ref "manifest-sha256:eb77e90ed9b8fab498025677bec65872b24075f0b236bf8532f15dcaf06cdb83": unexpected status: 500 Internal Server Error  source=console
ERRO[0449] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0128, error: failed commit on ref "manifest-sha256:22b833dec1971a495e78ef06c6bbe9c616981ee58517ebcc6234a3ce362535e6": unexpected status: 500 Internal Server Error  source=console
ERRO[0449] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0352, error: failed commit on ref "manifest-sha256:7b5e6c0e2e77934caa1b1b12226ed6f134f4eabd5bfabe0d4be1ae0c2d42ed4e": unexpected status: 500 Internal Server Error  source=console
ERRO[0449] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0390, error: failed commit on ref "manifest-sha256:7c50c5c00f55181f374f7837906a9ded6c8876224df92577cb272d875924d152": unexpected status: 500 Internal Server Error  source=console
ERRO[0449] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0152, error: failed commit on ref "manifest-sha256:4d6a9437ce58978c82f88f8d391734dd466c805369c2df2c00ba6edd61813f6e": unexpected status: 500 Internal Server Error  source=console
ERRO[0449] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0277, error: failed commit on ref "manifest-sha256:63465c55576e8bf750a38bc6f3646233c65523522c82e6ac22dd369ff8d8ab8c": unexpected status: 500 Internal Server Error  source=console
ERRO[0449] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0126, error: failed commit on ref "manifest-sha256:8273906aea34817fdfb611edb057ca2befef760a517ac402095d10532990f7a8": unexpected status: 500 Internal Server Error  source=console
ERRO[0449] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0547, error: failed to copy: unexpected response: 500 Internal Server Error  source=console
ERRO[0449] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0462, error: failed commit on ref "manifest-sha256:55e02acb2386b5b693982954f93670dc5a5d644abbb7ed934343c5b13086433a": unexpected status: 500 Internal Server Error  source=console
ERRO[0449] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0421, error: failed commit on ref "manifest-sha256:3b1c3df1f7d8089b38204c8d96245a1222c9d31fea4be4dbd1f14e4f9dd292cc": unexpected status: 500 Internal Server Error  source=console
ERRO[0449] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0086, error: failed commit on ref "manifest-sha256:91593695f7370af9d1a9862ab1875245af13e607c942d719e6306ac8735d11df": unexpected status: 500 Internal Server Error  source=console
ERRO[0449] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0199, error: failed commit on ref "manifest-sha256:5ff2ff065d0a62c2f746792a0a1f6f8b0498f10bfcce23e92f69788c632b5a0b": unexpected status: 500 Internal Server Error  source=console
ERRO[0449] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0413, error: failed commit on ref "manifest-sha256:d22d3c1dd7df962f92e3649361495cc765f289518bc75abe4cfa08d73ba484c8": unexpected status: 500 Internal Server Error  source=console
ERRO[0449] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0544, error: failed to copy: unexpected response: 500 Internal Server Error  source=console
ERRO[0449] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0117, error: failed commit on ref "manifest-sha256:c290ff020ef3e528425f055e13d73638301391436edbc36b3b9b159cc463fdef": unexpected status: 500 Internal Server Error  source=console
ERRO[0449] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0047, error: failed commit on ref "manifest-sha256:5594d4d4eda4def52a63e6687444ffcaba15b060f3c4940900b158b7a7b3bb6f": unexpected status: 500 Internal Server Error  source=console
ERRO[0449] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0483, error: failed commit on ref "manifest-sha256:5e90873e625feca940381d338a03260915cd837a874fbcf0f296ea23d608de4b": unexpected status: 500 Internal Server Error  source=console
ERRO[0450] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0085, error: failed commit on ref "manifest-sha256:6150348c5bc181fec5bb20a0b7eb1470c44f901446750c4a4377be4ac094d8e5": unexpected status: 500 Internal Server Error  source=console
ERRO[0450] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0536, error: failed commit on ref "config-sha256:c52862699bb2ffe24edb070873e6b3a5ab37484cd0973378e5fcb0766e78305c": unexpected status: 500 Internal Server Error  source=console
ERRO[0451] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0473, error: failed commit on ref "manifest-sha256:86c1ed5a892a7144dd31568f5fd3fc1e7817ac07da442fc097e827b528b00551": unexpected status: 500 Internal Server Error  source=console
ERRO[0451] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0548, error: failed to copy: unexpected response: 500 Internal Server Error  source=console
ERRO[0451] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0430, error: failed commit on ref "manifest-sha256:033ee48b9125e2e7080a3d596e41a58673375f06ec13c6c217af7be31de0575b": unexpected status: 500 Internal Server Error  source=console
ERRO[0451] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0502, error: failed commit on ref "manifest-sha256:d995da7ae1c0eef4db6fd7b2eddb3274095867981ae5d07b86eaa2b28e9a6f85": unexpected status: 500 Internal Server Error  source=console
ERRO[0452] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0480, error: failed commit on ref "manifest-sha256:d1692f8bcdbc2f1fe8b4f843857aa70ed1773e646e84916d2a1b95e4050916db": unexpected status: 500 Internal Server Error  source=console
ERRO[0452] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0346, error: failed commit on ref "manifest-sha256:02f247335d02ae0caece1278d2c7ac6f13a4a91cedfc93de333889f67b4c008b": unexpected status: 500 Internal Server Error  source=console
ERRO[0452] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0550, error: failed to copy: unexpected response: 500 Internal Server Error  source=console
ERRO[0452] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0038, error: failed commit on ref "manifest-sha256:e3409a0659b81ad759929672659d6b0bbe624455d6185e29cbd3144305c9a007": unexpected status: 500 Internal Server Error  source=console
ERRO[0452] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0549, error: failed commit on ref "config-sha256:b2a46b56e3601034fa0e6c7f545e595b53b9176519485689a5942ddf52fcc601": unexpected status: 500 Internal Server Error  source=console
ERRO[0452] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0206, error: failed commit on ref "manifest-sha256:19dfbd073aa9177151a23758a8775e801965d421c3d25b4c42c02286752dff06": unexpected status: 500 Internal Server Error  source=console
ERRO[0452] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0551, error: failed commit on ref "config-sha256:5d8f3041403f4bf037afcd83d5762501532707ee0087dc875902e3c0cf2b715c": unexpected status: 500 Internal Server Error  source=console
ERRO[0454] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0552, error: failed commit on ref "config-sha256:40aa6ec3788bb0d59310e6115f1a91f1308938142becf169ee0d808b9a7b5072": unexpected status: 500 Internal Server Error  source=console
ERRO[0454] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0553, error: failed commit on ref "config-sha256:3dec38f9570c170834953eecaa76a1a9555260d110389d36cc4390b1485e0c5e": unexpected status: 500 Internal Server Error  source=console
ERRO[0566] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0579, error: failed to copy: unexpected response: 500 Internal Server Error  source=console
ERRO[0569] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0615, error: failed to copy: unexpected response: 500 Internal Server Error  source=console
ERRO[0571] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0591, error: failed commit on ref "config-sha256:f48ded328cc5169b4e5ca322aa64d09219c99a6720acaa278886783ce381f82d": unexpected status: 500 Internal Server Error  source=console
ERRO[0572] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0640, error: failed commit on ref "config-sha256:971de18fc22c521d0c824107f88654ea27abb7adc85b0adf7af95ee4445571cd": unexpected status: 500 Internal Server Error  source=console
ERRO[0573] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0583, error: failed to copy: unexpected response: 500 Internal Server Error  source=console
ERRO[0573] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0664, error: failed to copy: unexpected response: 500 Internal Server Error  source=console
ERRO[0585] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0679, error: failed to copy: unexpected response: 500 Internal Server Error  source=console
ERRO[0585] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0643, error: failed commit on ref "config-sha256:e20356f33e555734b826dc11f8d64818582f17e1a407e36d6771f3c8b7de2ebe": unexpected status: 500 Internal Server Error  source=console
ERRO[0594] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0577, error: failed commit on ref "config-sha256:3d936251e9f69a0374a77773aae9516d67ccf07f0a698ef7a8969c690173eaf6": unexpected status: 500 Internal Server Error  source=console
ERRO[0621] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0675, error: failed commit on ref "config-sha256:0dab025f4d80c0ce808a4f425911db3449c5e9c273045e7a4934becc4f101ce5": unexpected status: 500 Internal Server Error  source=console
ERRO[0636] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0715, error: failed commit on ref "config-sha256:ff695f8cccf063f9ee7100795b7be641a1c682b83686d9a22570d786eee5e79b": unexpected status: 500 Internal Server Error  source=console
ERRO[0639] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0722, error: failed to copy: unexpected response: 500 Internal Server Error  source=console
ERRO[0639] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0730, error: failed commit on ref "config-sha256:75ca02f0a8faaa941af9e65d7e91c9af659af2663345fffd6e76104e17c7d7d8": unexpected status: 500 Internal Server Error  source=console
ERRO[0640] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0704, error: failed to copy: unexpected response: 500 Internal Server Error  source=console
ERRO[0709] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0891, error: failed commit on ref "manifest-sha256:31f198fd4c94ae50fa347f34d4fa3b7356d082e54f1c3989c18e61815241e3d1": no response  source=console
ERRO[0709] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0934, error: failed commit on ref "manifest-sha256:d1da42342613a6d21b5524a6489d790e27d2aae1f974ae7351797709da895581": no response  source=console
ERRO[0709] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0607, error: failed commit on ref "manifest-sha256:c1bc71e321568af0110af547c4ec6f338ee7e43dc3ee15e39496c121887021f9": no response  source=console
ERRO[0709] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0811, error: failed commit on ref "manifest-sha256:d8afd6d05f0fefc5cc96a08d5b5f498cb6de9bfc61430c40ad886fffa14bccea": no response  source=console
ERRO[0709] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0916, error: failed commit on ref "config-sha256:375955231417102363aa98e4bb85c8ff405077ac452e4a9de0b4083a6163def6": no response  source=console
ERRO[0709] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0869, error: failed commit on ref "manifest-sha256:f4c84900880d107b2279b3536fc026384ba8d63f507f1b99bc7c3acbee2929fc": no response  source=console
ERRO[0709] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0898, error: failed commit on ref "manifest-sha256:e88b4bbea7e7f59035dada5421701e97a7e15fa8e32c04cacc299631501b9ec6": no response  source=console
ERRO[0709] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0978, error: failed commit on ref "manifest-sha256:5e5a3f96d33720727eed5e4d6b126d0e9ff241eb5831a80bc9f659588489f494": no response  source=console
ERRO[0709] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0667, error: failed commit on ref "manifest-sha256:aa2310b9823cce901288550e7d9a7622ff38a53c1da049ef551a23a1f39fb060": no response  source=console
ERRO[0709] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0879, error: failed commit on ref "manifest-sha256:84dec2b04ad823c81b9ae026da763fa1adfc8661a38aa1778bf81893d6c1164f": no response  source=console
ERRO[0709] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0966, error: failed commit on ref "manifest-sha256:09f5efe408d11d60a6c16ab51586d8b81547ff6cbb3bd90c6dc7eec75c9db573": no response  source=console
ERRO[0709] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0587, error: failed commit on ref "manifest-sha256:3c77b2befa6d0ee4bcc620322be04a45ee5e84d73fb4e65e3d0fb4fc05db2b01": no response  source=console
ERRO[0709] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0570, error: failed commit on ref "manifest-sha256:8efd949c18e8bdd17bdbcb316eb2decbae398afc874e17f9efe1ef7c69ec4d62": no response  source=console
ERRO[0709] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0682, error: failed commit on ref "manifest-sha256:3619cffa7bf082abed50580bfa716015f4fd915967535199e385a22a0ae96207": no response  source=console
ERRO[0709] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0967, error: failed commit on ref "manifest-sha256:838a31bca59041022c9d41ce473c6ed777910d9f0321a93b9f3e45eaabd9ff6f": no response  source=console
ERRO[0709] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0866, error: failed commit on ref "manifest-sha256:90c9463699c7862a66787f23e2b146ccf998479445e50a0854b00e172b2f461f": no response  source=console
ERRO[0709] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0972, error: failed commit on ref "manifest-sha256:36124e47a39e9e7664583733a33cb96384eb6821c7402ead8c056377a97247ba": no response  source=console
ERRO[0709] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0782, error: failed commit on ref "manifest-sha256:3c521b1338ea6430bcc57319216d1ac8830ce768a9a0f5fe66b9ebc97822c649": no response  source=console
ERRO[0709] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0805, error: failed commit on ref "manifest-sha256:9e3558a70e5f24e85d8976822502d8a2bc2c1bcd6bda34791b1f7b6f9906d4fa": no response  source=console
ERRO[0709] failed to push reg.srv.deeproute.cn/project-020/repository-1706166862428:tag-0783, error: failed commit on ref "manifest-sha256:e76b6d8588ef16ae4673d19dd63aacdd1faeda1e3580bae8530eb504dda47732": no response  source=console


     █ setup

     █ teardown

     data_received............: 0 B    0 B/s
     data_sent................: 0 B    0 B/s
     iteration_duration.......: avg=4m9s  min=7.31s  med=4m22s max=8m56s p(90)=6m8s  p(95)=6m28s
     ✓ { group:::setup }......: avg=1m18s min=1m18s  med=1m18s max=1m18s p(90)=1m18s p(95)=1m18s
     ✓ { group:::teardown }...: avg=7.31s min=7.31s  med=7.31s max=7.31s p(90)=7.31s p(95)=7.31s
     ✓ { scenario:default }...: avg=4m9s  min=22.59s med=4m22s max=8m56s p(90)=6m8s  p(95)=6m28s
     iterations...............: 860    1.201053/s
     success..................: 73.83% ✓ 635      ✗ 225  
     vus......................: 0      min=0      max=500
     vus_max..................: 500    min=500    max=500


running (11m56.0s), 000/500 VUs, 860 complete and 140 interrupted iterations
default ✗ [===============================>------] 500 VUs  10m30.0s/10m0s  0860/1000 shared iters
ERRO[0717] GoError: failed to delete artifact project-020/repository-1706166862428:tag-0002, error: [DELETE /projects/{project_name}/repositories/{repository_name}/artifacts/{reference}][404] deleteArtifactNotFound  &{Errors:[0xc007aa6ec0]}
	at reflect.methodValueCall (native)
	at teardown (file:///home/DEEPROUTE/liangzheng/go/src/github.com/goharbor/perf/scripts/test/push-artifacts-to-same-project.js:67:30(36))  hint="script exception"
Error: running "k6-harbor run ./scripts/test/push-artifacts-to-same-project.js --no-usage-report --vus 500 --iterations 1000" failed with exit code 107
exit status 107
➜ /home/DEEPROUTE/liangzheng/go/src/github.com/goharbor/perf ☞ git:(main) ✗ cat outputs/push-artifacts-to-same-projects.summary.json| jq     
{
  "timestamp": 1706167578474,
  "description": "push-artifacts-to-same-projects",
  "avg": "249.88s",
  "min": "22.59s",
  "med": "262.71s",
  "max": "536.57s",
  "p90": "368.85s",
  "p95": "388.22s",
  "successRate": "73.83%",
  "iterationsRate": "1.20/s"
}

from harbor.

microyahoo commented on May 24, 2024

Thanks for your nice reminder, I will test it and give you feedback. Is harbor 2.10.0 report under QUOTA_UPDATE_PROVIDER="redis"? @chlins

from harbor.

microyahoo commented on May 24, 2024

hi @chlins, the success rate increased significantly to 100% after setting QUOTA_UPDATE_PROVIDER to redis. But push artifacts to different projects still has some errors. And there is no error info in redis log. It looks like the client actively terminated the context.

[root@bd-ssd07-node01 perf]# cat outputs/push-artifacts-to-different-projects.summary.json | jq
{
  "timestamp": 1706520431766,
  "description": "push-artifacts-to-different-projects",
  "avg": "22.25s",
  "min": "2.24s",
  "med": "20.38s",
  "max": "44.37s",
  "p90": "34.7s",
  "p95": "36.52s",
  "successRate": "95.8%",
  "iterationsRate": "4.82/s"
}

[root@bd-ssd07-node01 perf]# date; time go run mage.go run push-artifacts-to-different-projects; date                             [109/11816]
Mon Jan 29 17:23:42 CST 2024                                                                                                                 
                                                                                                                                             
          /\      |‾‾| /‾‾/   /‾‾/                                                                                                           
     /\  /  \     |  |/  /   /  /                                                                                                            
    /  \/    \    |     (   /   ‾‾\                                                                                                          
   /          \   |  |\  \ |  (‾)  |                                                                                                         
  / __________ \  |__| \__\ \_____/ .io                                                                                                      
                                                                                                                                             
  execution: local                                                                                                                           
     script: ./scripts/test/push-artifacts-to-different-projects.js                                                                          
     output: -                                                                                                                               
                                                                                                                                             
  scenarios: (100.00%) 1 scenario, 500 max VUs, 10m30s max duration (incl. graceful stop):                                                   
           * default: 1000 iterations shared among 500 VUs (maxDuration: 10m0s, gracefulStop: 30s)                                           

ERRO[0063] failed to push reg.srv.deeproute.cn/project-089/repository-1706520246968:tag-0508, error: failed commit on ref "config-sha256:e410ae7e8f6d2f2eb84b5de65873ed708d10f257b2f6189c0028e52991447456": unexpected status: 500 Internal Server Error  source=console
ERRO[0064] failed to push reg.srv.deeproute.cn/project-009/repository-1706520247715:tag-0523, error: failed to copy: unexpected response: 500
 Internal Server Error  source=console                                 
ERRO[0065] failed to push reg.srv.deeproute.cn/project-090/repository-1706520247472:tag-0518, error: failed commit on ref "config-sha256:a0896a329efde033ebe1583144326b65eb5d8efbbcbc86d8b074af0e6bbf94f5": unexpected status: 500 Internal Server Error  source=console
ERRO[0066] failed to push reg.srv.deeproute.cn/project-041/repository-1706520247767:tag-0524, error: failed commit on ref "config-sha256:5a311a8a5641b4c439dc767b8600e6c4123a6cef2a39b086a84432af2e553a59": unexpected status: 500 Internal Server Error  source=console
ERRO[0066] failed to push reg.srv.deeproute.cn/project-015/repository-1706520248470:tag-0538, error: failed commit on ref "config-sha256:f4bfba168847f2d2c4adbc595693556d5bca829ad7ea9bb658b0bdc0804e116a": unexpected status: 500 Internal Server Error  source=console
ERRO[0067] failed to push reg.srv.deeproute.cn/project-059/repository-1706520248651:tag-0542, error: failed commit on ref "config-sha256:9e72c1ad14f2772e723fc7f2505e5ebb73ee3365e0a4a0757857b36853021eac": unexpected status: 500 Internal Server Error  source=console
ERRO[0067] failed to push reg.srv.deeproute.cn/project-068/repository-1706520248059:tag-0530, error: failed commit on ref "config-sha256:581157990da18658c71ff6b653be1eba059dd8a262aef2a55b3adca509ae0fcf": unexpected status: 500 Internal Server Error  source=console
ERRO[0068] failed to push reg.srv.deeproute.cn/project-057/repository-1706520248516:tag-0539, error: failed commit on ref "config-sha256:349fb819b03580f298d7b1f678c51acc81728e3905f4aea5e747524057526b03": unexpected status: 500 Internal Server Error  source=console
ERRO[0070] failed to push reg.srv.deeproute.cn/project-061/repository-1706520250099:tag-0572, error: failed commit on ref "config-sha256:28a42f4ffa5ce488e6ba19cf50b102c37d066f153551117401ecbf14e3ae916a": unexpected status: 500 Internal Server Error  source=console
ERRO[0072] failed to push reg.srv.deeproute.cn/project-049/repository-1706520251723:tag-0608, error: failed commit on ref "config-sha256:540a23173af2f607450cd15f2a05231e65823edd45f48f1ac23c353062e71d3a": unexpected status: 500 Internal Server Error  source=console
...

harbor core log snippet

22695 2024-01-29T07:25:11Z [DEBUG] [/pkg/notifier/notifier.go:220]: Handle notification with Handler 'Replication' on topic 'PUSH_ARTIFACT': ID-6868, Repository-project-012/repository-1706513014512 Tags-[tag-0129] Digest-sha256:08555fb6aad81d47f4b19765f84bc0caf3b404eda1aa7cb80104b1eac919cde7 Operator-admin OccurAt-2024-01-29 07:25:11
22696 2024-01-29T07:25:11Z [DEBUG] [/pkg/allowlist/manager.go:75]: No CVE allowlist found for project 0, returning empty list.
22697 2024-01-29T07:25:11Z [DEBUG] [/pkg/cached/project_metadata/redis/manager.go:85]: get project 80 metadata from cache error: key not found, will query from database.
22698 2024-01-29T07:25:11Z [DEBUG] [/controller/p2p/preheat/enforcer.go:301]: No preheat policy matched for the artifact project-012/repository-1706513014512@sha256:08555fb6aad81d47f4b19765f84bc0caf3b404eda1aa7cb80104b1eac919cde7
22699 2024-01-29T07:25:11Z [DEBUG] [/pkg/notifier/notifier.go:220]: Handle notification with Handler 'P2PPreheat' on topic 'PUSH_ARTIFACT': ID-6868, Repository-project-012/repository-1706513014512 Tags-[tag-0129] Digest-sha256:08555fb6aad81d47f4b19765f84bc0caf3b404eda1aa7cb80104b1eac919cde7 Operator-admin OccurAt-2024-01-29 07:25:11
22700 2024/01/29 07:25:11 http: proxy error: context canceled                                             
22701 2024-01-29T07:25:11Z [WARNING] [/controller/quota/controller.go:402][requestID="4d3a7c1c-1fe9-43f0-88f8-c46668f506ac"]: rollback resources {"storage":10485760} for project 120 failed, error: context canceled
22702 2024/01/29 07:25:12 http: proxy error: context canceled                                             
22703 2024-01-29T07:25:12Z [WARNING] [/controller/quota/controller.go:402][requestID="f20e7050-77b5-4149-91b6-22b8b0941737"]: rollback resources {"storage":10485760} for project 56 failed, error: context canceled
22704 2024/01/29 07:25:12 http: proxy error: context canceled                                             
22705 2024-01-29T07:25:12Z [WARNING] [/controller/quota/controller.go:402][requestID="4f1d9a88-91d9-45e7-b87d-102b63b9be29"]: rollback resources {"storage":10485760} for project 75 failed, error: context canceled
22706 2024/01/29 07:25:12 http: proxy error: context canceled                                             
22707 2024-01-29T07:25:12Z [WARNING] [/controller/quota/controller.go:402][requestID="bc4d4329-d253-4b06-9cbc-96f3401156ed"]: rollback resources {"storage":10485760} for project 94 failed, error: context canceled
22708 2024/01/29 07:25:12 http: proxy error: context canceled                                             
22709 2024-01-29T07:25:12Z [WARNING] [/controller/quota/controller.go:402][requestID="5bf7f46c-deb5-4161-bb20-6003035a8800"]: rollback resources {"storage":10485760} for project 116 failed, error: context canceled 
22710 2024-01-29T07:25:12Z [DEBUG] [/pkg/cached/project/redis/manager.go:127]: get project project-029 from cache error: key not found:context canceled, will query from database.
22711 2024-01-29T07:25:12Z [DEBUG] [/pkg/cached/project_metadata/redis/manager.go:85]: get project 89 metadata from cache error: key not found:context canceled, will query from database.
22712 2024-01-29T07:25:12Z [DEBUG] [/server/middleware/quota/quota.go:138][action="request" middleware="quota" requestID="65b0eb58-5ec0-4f30-87cb-6fb04522254d" url="/v2/project-029/repository-1706513029439/manifests/tag-0389"]: not warning resources found
22713 2024-01-29T07:25:12Z [DEBUG] [/pkg/notifier/event/event.go:112]: event PUSH_ARTIFACT published

registry log

146100 10.233.106.119 - - [29/Jan/2024:07:25:01 +0000] "PUT /v2/project-059/repository-1706513021492/manifests/tag-0247 HTTP/1.1" 201 0 "" "containerd/1.3.2+unknown"
146101 2024/01/29 07:25:12 http: superfluous response.WriteHeader call from github.com/docker/distribution/vendor/github.com/gorilla/handlers.(*responseLogger).WriteHeader (handlers.go:125)                                                                                      
146102 time="2024-01-29T07:25:12.325658472Z" level=error msg="response completed with error" auth.user.name="harbor_registry_user" err.code=unknown err.detail="client disconnected" err.message="unknown error" go.version=go1.21.4 http.request.host=reg.srv.deeproute.cn http.request.id=d4a47f4b-d1fd-42ba-aefe-8d1c1c1a67d7 http.request.method=PUT http.request.remoteaddr=10.10.12.31 http.request.uri="/v2/project-005/repository-1706513057097/blobs/uploads/ad0c34a4-681b-4060-8812-773bace159bd?_state=THmaShtyHrYfjMIYyyz5g0coS7giPrqnzBLayP2Dz817Ik5hbWUiOiJwcm9qZWN0LTAwNS9yZXBvc2l0b3J5LTE3MDY1MTMwNTcwOTciLCJVVUlEIjoiYWQwYzM0YTQtNjgxYi00MDYwLTg4MTItNzczYmFjZTE1OWJkIiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDI0LTAxLTI5VDA3OjI1OjA4LjgzOTAyMzIwM1oifQ%3D%3D&digest=sha256%3Ad1721ef657242932b5439db9ecd6e6208e53e5ba0257715511f7f92856b1dab5" http.request.useragent="containerd/1.3.2+unknown" http.response.contenttype="application/json; charset=utf-8" http.response.duration=3.308967875s http.response.status=500 http.response.written=89 vars.name="project-005/repository-1706513057097" vars.uuid=ad0c34a4-681b-4060-8812-773bace159bd 
146103 10.233.94.55 - - [29/Jan/2024:07:25:09 +0000] "PUT /v2/project-005/repository-1706513057097/blobs/uploads/ad0c34a4-681b-4060-8812-773bace159bd?_state=THmaShtyHrYfjMIYyyz5g0coS7giPrqnzBLayP2Dz817Ik5hbWUiOiJwcm9qZWN0LTAwNS9yZXBvc2l0b3J5LTE3MDY1MTMwNTcwOTciLCJVVUlEIjoiYWQwYzM0YTQtNjgxYi00MDYwLTg4MTItNzczYmFjZTE1OWJkIiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDI0LTAxLTI5VDA3OjI1OjA4LjgzOTAyMzIwM1oifQ%3D%3D&dig       est=sha256%3Ad1721ef657242932b5439db9ecd6e6208e53e5ba0257715511f7f92856b1dab5 HTTP/1.1" 500 89 "" "containerd/1.3.2+unknown"
146104 time="2024-01-29T07:25:12.3261603Z" level=error msg="client disconnected during blob PUT" auth.user.name="harbor_registry_user" contentLength=10485760 copied=4724492 error="unexpected EOF" go.version=go1.21.4 http.request.host=reg.srv.deeproute.cn http.request.id=52525813-40a7-4d2a-ad1a-a7851db29cb0 http.request.method=PUT http.request.remoteaddr=10.10.12.31 http.request.uri="/v2/project-049/repository-1706513057321/blobs/uploads/3bfe34ec-e17e-4800-add7-3e6caa0679be?_state=JSA7glgfoiHWwTD6KaAQAumOpr6Ow3f0n6RHusuC4r57Ik5hbWUiOiJwcm       9qZWN0LTA0OS9yZXBvc2l0b3J5LTE3MDY1MTMwNTczMjEiLCJVVUlEIjoiM2JmZTM0ZWMtZTE3ZS00ODAwLWFkZDctM2U2Y2FhMDY3OWJlIiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDI0LTAxLTI5VDA3OjI1OjA4Ljg2NjMyNDI1NloifQ%3D%3D&digest=sha256%3Afeba9bf8492c45d739983df53f3a475562909172e6af2c6d9125e23357ae869d" http.request.useragent="containerd/1.3.2+unknown" vars.name="project-049/repository-1706513057321" vars.uuid=3bfe34ec-e17e-4800-add7-3e6caa0679be

from harbor.

microyahoo commented on May 24, 2024

hi @chlins, @wy65701436, happy Chinese new year! Is there any progress on this issue?

from harbor.

microyahoo commented on May 24, 2024

The issue still persists

~/go/src/github.com/goharbor/perf (main ✗) go run mage.go run push-artifacts-to-different-projects                              
                                                                                                                                             
          /\      |‾‾| /‾‾/   /‾‾/               
     /\  /  \     |  |/  /   /  /                        
    /  \/    \    |     (   /   ‾‾\                      
   /          \   |  |\  \ |  (‾)  |                     
  / __________ \  |__| \__\ \_____/ .io

  execution: local                                                                                                                           
     script: ./scripts/test/push-artifacts-to-different-projects.js                                                                          
     output: -                                                                                                                                                                                                                                                                            
  scenarios: (100.00%) 1 scenario, 500 max VUs, 10m30s max duration (incl. graceful stop):
           * default: 1000 iterations shared among 500 VUs (maxDuration: 10m0s, gracefulStop: 30s)                                                                              
ERRO[0156] failed to push reg.srv.deeproute.cn/project-072/repository-1708504916925:tag-0546, error: failed to copy: unexpected response: 500 Internal Server Error  source=console                                 
ERRO[0159] failed to push reg.srv.deeproute.cn/project-100/repository-1708504920884:tag-0582, error: failed to copy: unexpected response: 500
 Internal Server Error  source=console                                                                       
ERRO[0159] failed to push reg.srv.deeproute.cn/project-046/repository-1708504920140:tag-0575, error: failed to copy: unexpected response: 502
 Bad Gateway  source=console                                           
ERRO[0160] failed to push reg.srv.deeproute.cn/project-024/repository-1708504925567:tag-0619, error: failed commit on ref "config-sha256:b1d8
fc6db087b8ac5c064153a31a8fe6be6c105beb91359d7193e0a156afe8f7": unexpected status: 500 Internal Server Error  source=console
ERRO[0164] failed to push reg.srv.deeproute.cn/project-003/repository-1708504930727:tag-0656, error: failed to copy: unexpected response: 500
 Internal Server Error  source=console

from harbor.

microyahoo commented on May 24, 2024

I met a lot of redis: transaction failed and rollback resources {"storage":47} for project 133 failed, error: context canceled error, even through I set QUOTA_UPDATE_PROVIDER="redis". This can lead to a drastic drop in success rates, how can this be mitigated? Please help to look at again, thanks. @wy65701436 @chlins. Below is logs.

harbor.log.tar.gz

from harbor.

chlins commented on May 24, 2024

@microyahoo Hi, could you please provide the hardware configuration such as CPU/memory/disk info of the harbor deployment environment? so that can help us to reproduce the issue.

from harbor.

microyahoo commented on May 24, 2024

@microyahoo Hi, could you please provide the hardware configuration such as CPU/memory/disk info of the harbor deployment environment? so that can help us to reproduce the issue.

Thanks for your quick response. @chlins

node exporter:

redis: Aliyun cluodbox redis, below is metric when running performance test.

redis info:

🍺 /home/DEEPROUTE/liangzheng ☞ redis-cli -h <redis host> -a <password> info
# Server
redis_version:5.0.0
redis_git_sha1:3d1b465b
redis_git_dirty:1
redis_build_id:1ae5f854cb4a62e4
redis_mode:standalone
os:Linux  
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:0.0.0
process_id:19
run_id:5e782aa338a437171e51f34e2d84649fd371deb7
tcp_port:6379
uptime_in_seconds:4844273
uptime_in_days:56
hz:10
configured_hz:10
lru_clock:16236228
executable:
support_ptod:1
config_file:

# Clients
connected_clients:30
client_recent_max_input_buffer:8
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
pubsub_clients:0
clients_in_timeout_table:0

# Memory
used_memory:120468480
used_memory_human:114.89M
used_memory_rss:154132480
used_memory_rss_human:146.99M
used_memory_peak:802205936
used_memory_peak_human:765.04M
used_memory_peak_perc:15.02%
used_memory_overhead:97004728
used_memory_startup:58737976
used_memory_dataset:23463752
used_memory_dataset_perc:38.01%
allocator_allocated:121233776
allocator_active:143593472
allocator_resident:164933632
used_memory_lua:77824
used_memory_lua_human:76.00K
used_memory_scripts:7152
used_memory_scripts_human:6.98K
number_of_cached_scripts:6
maxmemory:2147483648
maxmemory_human:2.00G
maxmemory_policy:volatile-lru
allocator_frag_ratio:1.18
allocator_frag_bytes:22359696
allocator_rss_ratio:1.15
allocator_rss_bytes:21340160
rss_overhead_ratio:0.93
rss_overhead_bytes:-10801152
mem_fragmentation_ratio:1.28
mem_fragmentation_bytes:33850712
mem_not_counted_for_evict:3424
mem_replication_backlog:33554432
mem_clients_slaves:20512
mem_clients_normal:642144
mem_aof_buffer:3584
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfreed_objects:0
lazyfree_pending_objects:0
mem_overhead_hashtable_main:3687888
mem_overhead_hashtable_expires:351040
oom_err_count:0

# Stats
total_connections_received:2481509
total_commands_processed:1408303735
instantaneous_ops_per_sec:327
instantaneous_write_ops_per_sec:1
instantaneous_read_ops_per_sec:289
instantaneous_other_ops_per_sec:37
instantaneous_sync_write_ops_per_sec:0
total_net_input_bytes:208859095323
total_net_output_bytes:123982011288
total_net_repl_input_bytes:0
total_net_repl_output_bytes:12050215712
instantaneous_input_kbps:50.50
instantaneous_output_kbps:13.67
instantaneous_input_repl_kbps:0.00
instantaneous_output_repl_kbps:0.24
rejected_connections:0
sync_full:1
sync_partial_ok:0
sync_partial_err:0
expired_keys:11846344
expired_stale_perc:0.04
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:695189
evicted_keys:0
keyspace_hits:620400703
keyspace_misses:545676998
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:3991
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:190
instantaneous_error_replies_ops_per_sec:0
total_reads_processed:438853812
total_writes_processed:408688526
io_threaded_reads_processed:0
io_threaded_writes_processed:0
slot_psync_ok:0
slot_psync_err:0
total_commands_received:403594937
commands_received_per_sec:42
evicted_keys_per_sec:0
hits_per_sec:142
misses_per_sec:146
hit_rate_percentage:49.31
cmd_slowlog_count:5
traffic_control_input:0
traffic_control_input_status:0
traffic_control_output:0
traffic_control_output_status:0
stat_avg_rt:18
stat_max_rt:1250

# Replication
role:master

# CPU
used_cpu_sys:5918.144167
used_cpu_user:32298.789347
used_cpu_sys_children:8.529888
used_cpu_user_children:33.265656

# Errorstats
errorstat_ERR:count=20
errorstat_NOAUTH:count=1
errorstat_WRONGPASS:count=168
errorstat_WRONGTYPE:count=1

# Cluster
cluster_enabled:0
databases:256
nodecount:1

# Keyspace
db0:keys=3602,expires=3602,avg_ttl=28209546
db1:keys=100,expires=58,avg_ttl=56702724
db2:keys=56696,expires=0,avg_ttl=0
db5:keys=5,expires=0,avg_ttl=0

# mio
main_io:conn=18,ncommands=204923374,total_input_bytes=191627162785,total_output_bytes=103737733019,ops=42,input_kbps=50.50,output_kbps=13.67
follower_io_0:conn=3,ncommands=47556705,total_input_bytes=4087644351,total_output_bytes=4880824679,ops=0,input_kbps=0.00,output_kbps=0.00
follower_io_1:conn=4,ncommands=50548295,total_input_bytes=4443605741,total_output_bytes=5135831050,ops=0,input_kbps=0.00,output_kbps=0.00
follower_io_2:conn=2,ncommands=48177131,total_input_bytes=4195807063,total_output_bytes=4857888184,ops=0,input_kbps=0.00,output_kbps=0.00
follower_io_3:conn=6,ncommands=52389435,total_input_bytes=4504875383,total_output_bytes=5369734088,ops=0,input_kbps=0.00,output_kbps=0.00

# Capability
control_subsystem_enabled:1
latency_metric_enabled:1
mio_enabled:4
bigkey_enabled:1
hotkey_enabled:1
tde_enabled:0
ptod_enabled:1
slotmig_enabled:1
seamlessha_enabled:0
cluster_dht_tls_enabled:1

harbor core:

harbor registry:

~/go/src/deeproute/harbor-cli (main ✗) kubectl --kubeconfig ~/.kube/dr-config get nodes -o wide
NAME                                    STATUS   ROLES    AGE      VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION       CONTAINER-RUNTIME
master1                                 Ready    master   3y253d   v1.17.6   10.10.12.11   <none>        Ubuntu 18.04.4 LTS   4.15.0-197-generic   docker://18.9.7
master2                                 Ready    master   3y253d   v1.17.6   10.10.12.12   <none>        Ubuntu 18.04.4 LTS   4.15.0-142-generic   docker://18.9.7
master3                                 Ready    master   3y253d   v1.17.6   10.10.12.13   <none>        Ubuntu 18.04.4 LTS   4.15.0-191-generic   docker://18.9.7
node01.prod-k8s.bm.pd.sz.deeproute.ai   Ready    <none>   2y191d   v1.17.6   10.10.12.31   <none>        Ubuntu 18.04.5 LTS   4.15.0-156-generic   docker://18.9.7
node02.prod-k8s.bm.pd.sz.deeproute.ai   Ready    <none>   2y214d   v1.17.6   10.10.12.32   <none>        Ubuntu 18.04.5 LTS   4.15.0-153-generic   docker://18.9.7
node03.prod-k8s.bm.pd.sz.deeproute.ai   Ready    <none>   2y214d   v1.17.6   10.10.12.33   <none>        Ubuntu 18.04.5 LTS   4.15.0-197-generic   docker://18.9.7
node04.prod-k8s.bm.pd.sz.deeproute.ai   Ready    <none>   2y212d   v1.17.6   10.10.12.34   <none>        Ubuntu 18.04.5 LTS   4.15.0-153-generic   docker://18.9.7
node05.prod-k8s.bm.pd.sz.deeproute.ai   Ready    <none>   2y207d   v1.17.6   10.10.12.35   <none>        Ubuntu 18.04.5 LTS   4.15.0-153-generic   docker://18.9.7
node06.prod-k8s.bm.pd.sz.deeproute.ai   Ready    <none>   2y205d   v1.17.6   10.10.12.36   <none>        Ubuntu 18.04.5 LTS   4.15.0-197-generic   docker://18.9.7
node07.prod-k8s.bm.pd.sz.deeproute.ai   Ready    <none>   2y200d   v1.17.6   10.10.12.37   <none>        Ubuntu 18.04.5 LTS   4.15.0-197-generic   docker://18.9.7
node08.prod-k8s.bm.pd.sz.deeproute.ai   Ready    <none>   2y200d   v1.17.6   10.10.12.38   <none>        Ubuntu 18.04.5 LTS   4.15.0-197-generic   docker://18.9.7

~/go/src/deeproute/harbor-cli (main ✗) kubectl --kubeconfig ~/.kube/dr-config -n harbor-server get pods -o wide
NAME                                 READY   STATUS    RESTARTS   AGE     IP               NODE                                    NOMINATED NODE   READINESS GATES
harbor-core-66588fc7bd-8b9tn         1/1     Running   0          2d15h   10.233.106.243   node01.prod-k8s.bm.pd.sz.deeproute.ai   <none>           <none>
harbor-core-66588fc7bd-hs6qq         1/1     Running   0          2d15h   10.233.125.7     node05.prod-k8s.bm.pd.sz.deeproute.ai   <none>           <none>
harbor-core-66588fc7bd-mnqsb         1/1     Running   0          2d15h   10.233.112.80    node07.prod-k8s.bm.pd.sz.deeproute.ai   <none>           <none>
harbor-exporter-9d46fd77-r9z2p       1/1     Running   0          5d19h   10.233.102.32    node03.prod-k8s.bm.pd.sz.deeproute.ai   <none>           <none>
harbor-jobservice-64644b6477-lnvpf   1/1     Running   0          2d16h   10.233.120.146   node08.prod-k8s.bm.pd.sz.deeproute.ai   <none>           <none>
harbor-jobservice-64644b6477-n5mql   1/1     Running   0          2d16h   10.233.102.188   node03.prod-k8s.bm.pd.sz.deeproute.ai   <none>           <none>
harbor-jobservice-64644b6477-vrl2d   1/1     Running   0          2d16h   10.233.94.112    node04.prod-k8s.bm.pd.sz.deeproute.ai   <none>           <none>
harbor-portal-6ffd5f95f5-s6tpj       1/1     Running   0          2d16h   10.233.102.26    node03.prod-k8s.bm.pd.sz.deeproute.ai   <none>           <none>
harbor-registry-6df6b8c894-hs8vx     2/2     Running   0          2d16h   10.233.107.205   node06.prod-k8s.bm.pd.sz.deeproute.ai   <none>           <none>
harbor-registry-6df6b8c894-md8fd     2/2     Running   0          2d16h   10.233.120.49    node08.prod-k8s.bm.pd.sz.deeproute.ai   <none>           <none>
harbor-registry-6df6b8c894-vhs62     2/2     Running   0          2d16h   10.233.102.137   node03.prod-k8s.bm.pd.sz.deeproute.ai   <none>           <none>
harbor-trivy-0                       1/1     Running   0          5d19h   10.233.125.146   node05.prod-k8s.bm.pd.sz.deeproute.ai   <none>           <none>

registry.yaml

apiVersion: v1
kind: Pod
metadata:
  annotations:
    dep.configmap.hash/app-conf: 5098cc7a4b4970489b11ff4a4d176adaa5c2538c66e330d71c241ae199b20368
    dep.configmap.hash/jobservice-conf: a789a925b9310cb71746003deae0b4bf662f283c42853053696f2a94fa6905a6
    dep.configmap.hash/portal-conf: 4978cdfbc877e6b6015a90557f4603cd506bfe64d16986b79904ecd97a3c4297
    dep.configmap.hash/registry-conf: 7ab3b982a3c9728bff492128abd438d977e1fb0c0935c755b5d1744152d9adc9
    dep.configmap.hash/registry-ctl: 585c3753f4f7d155458b73e9501c0fd6d3002636aa3470ba079ef18ef7ecfcab
    dep.secret.hash/ceph-secret: 44a441dad46bec23b59e3ee2a4c46bcf0188d77207ee7e081c744535e725dc9d
    dep.secret.hash/core-env: 60b03d02d86d64d6de48e1341dee37862b2b9ce5042d89c54d0f80a3c014c8dc
    dep.secret.hash/exporter-env: f218b81357a05155188b3e80578e79159a5c06f62b89e65b99a5788249955a02
    dep.secret.hash/harbor-env: 7effc6edfb35fa76eb6274db2f0c25c3bd663765d61ad28111402e42ba0f76b4
    dep.secret.hash/jobservice-env: 65545ed53a39b359edcb4f5da591bbbbf6cd8b5e33522b50f57ed1fdf465b443
    dep.secret.hash/private-key: 97736853235d1705c806e9b852bb19a20278b66f6cb6e484229a6beebf2dbd2a
    dep.secret.hash/registry-env: 688328dafac719b43d1e58fcb9117639e3f9c88cb2e6e6b6da1e0ab8789b24b5
    dep.secret.hash/registry-htpwd: 3084e97b8b36708eade0a4868b6fad186b99b2bccd235c59ef4467c65fa69647
    dep.secret.hash/secret-key: d67cffe0dcf29c9a0e8a28e167a787925e223dea0499230d802b63c30e521750
    dep.secret.hash/trivy-env: 151c72f6b00798cb8c2af3d153bd14ca54d839e39d065c7bcacc783bd97d0268
  creationTimestamp: "2024-03-15T09:41:57Z"
  generateName: harbor-registry-6df6b8c894-
  labels:
    app: harbor-registry
    pod-template-hash: 6df6b8c894
    project: harbor
  name: harbor-registry-6df6b8c894-hs8vx
  namespace: harbor-server
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: harbor-registry-6df6b8c894
    uid: 8d5cbaf4-0614-451a-baea-eed958b38648
  resourceVersion: "1109916475"
  selfLink: /api/v1/namespaces/harbor-server/pods/harbor-registry-6df6b8c894-hs8vx
  uid: d191640b-fe2e-4f97-abc1-99dc9f40b115
spec:
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - podAffinityTerm:
          labelSelector:
            matchExpressions:
            - key: app
              operator: In
              values:
              - harbor-registry
          topologyKey: kubernetes.io/hostname
        weight: 100
  containers:
  - args:
    - serve
    - /etc/registry/config.yml
    env:
    - name: DR_META_K8S_CLUSTER_ENV
      value: production
    - name: DR_META_K8S_CLUSTER_NAME
      value: prod-k8s-cluster
    - name: STAKATER_REGISTRY_CONF_CONFIGMAP
      value: 72130abe884a7db5d543dfaeec65aac74b83bf4a
    envFrom:
    - secretRef:
        name: harbor-env
    - secretRef:
        name: registry-env
    image: reg.deeproute.ai/deeproute-public/registry-photon:v2.10.0
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 2
      httpGet:
        path: /
        port: registrya
        scheme: HTTP
      initialDelaySeconds: 300
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 1
    name: registry
    ports:
    - containerPort: 5000
      name: registrya
      protocol: TCP
    - containerPort: 5001
      name: registryb
      protocol: TCP
    - containerPort: 8080
      name: controller
      protocol: TCP
    - containerPort: 8001
      name: metrics
      protocol: TCP
    readinessProbe:
      failureThreshold: 2
      httpGet:
        path: /
        port: registrya
        scheme: HTTP
      initialDelaySeconds: 1
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 1
    resources:
      limits:
        cpu: "16"
        memory: 64Gi
      requests:
        cpu: "16"
        memory: 64Gi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /etc/registry//config.yml
      name: registry-conf
      subPath: config.yml
    - mountPath: /etc/registry//passwd
      name: registry-htpwd
      subPath: passwd
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-6nfll
      readOnly: true
  - env:
    - name: DR_META_K8S_CLUSTER_ENV
      value: production
    - name: DR_META_K8S_CLUSTER_NAME
      value: prod-k8s-cluster
    envFrom:
    - secretRef:
        name: harbor-env
    - secretRef:
        name: registry-env
    image: reg.deeproute.ai/deeproute-public/harbor-registryctl:v2.10.0
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 2
      httpGet:
        path: /api/health
        port: registryctl
        scheme: HTTP
      initialDelaySeconds: 300
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 1
    name: registryctl
    ports:
    - containerPort: 8080
      name: registryctl
      protocol: TCP
    readinessProbe:
      failureThreshold: 2
      httpGet:
        path: /api/health
        port: registryctl
        scheme: HTTP
      initialDelaySeconds: 1
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 1
    resources:
      limits:
        cpu: "2"
        memory: 4Gi
      requests:
        cpu: "2"
        memory: 4Gi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /etc/registry//config.yml
      name: registry-conf
      subPath: config.yml
    - mountPath: /etc/registryctl//config.yml
      name: registry-ctl
      subPath: config.yml
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-6nfll
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  imagePullSecrets:
  - name: image-pull-secret
  nodeName: node06.prod-k8s.bm.pd.sz.deeproute.ai
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - configMap:
      defaultMode: 420
      name: registry-conf
    name: registry-conf
  - configMap:
      defaultMode: 420
      name: registry-ctl
    name: registry-ctl
  - name: registry-htpwd
    secret:
      defaultMode: 420
      secretName: registry-htpwd
  - name: default-token-6nfll
    secret:
      defaultMode: 420
      secretName: default-token-6nfll

core.yaml

apiVersion: v1
kind: Pod
metadata:
  annotations:
    dep.configmap.hash/app-conf: 5098cc7a4b4970489b11ff4a4d176adaa5c2538c66e330d71c241ae199b20368
    dep.configmap.hash/jobservice-conf: a789a925b9310cb71746003deae0b4bf662f283c42853053696f2a94fa6905a6
    dep.configmap.hash/portal-conf: 4978cdfbc877e6b6015a90557f4603cd506bfe64d16986b79904ecd97a3c4297
    dep.configmap.hash/registry-conf: 7ab3b982a3c9728bff492128abd438d977e1fb0c0935c755b5d1744152d9adc9
    dep.configmap.hash/registry-ctl: 585c3753f4f7d155458b73e9501c0fd6d3002636aa3470ba079ef18ef7ecfcab
    dep.secret.hash/ceph-secret: 44a441dad46bec23b59e3ee2a4c46bcf0188d77207ee7e081c744535e725dc9d
    dep.secret.hash/core-env: 60b03d02d86d64d6de48e1341dee37862b2b9ce5042d89c54d0f80a3c014c8dc
    dep.secret.hash/exporter-env: f218b81357a05155188b3e80578e79159a5c06f62b89e65b99a5788249955a02
    dep.secret.hash/harbor-env: 7effc6edfb35fa76eb6274db2f0c25c3bd663765d61ad28111402e42ba0f76b4
    dep.secret.hash/jobservice-env: 65545ed53a39b359edcb4f5da591bbbbf6cd8b5e33522b50f57ed1fdf465b443
    dep.secret.hash/private-key: 97736853235d1705c806e9b852bb19a20278b66f6cb6e484229a6beebf2dbd2a
    dep.secret.hash/registry-env: 688328dafac719b43d1e58fcb9117639e3f9c88cb2e6e6b6da1e0ab8789b24b5
    dep.secret.hash/registry-htpwd: 3084e97b8b36708eade0a4868b6fad186b99b2bccd235c59ef4467c65fa69647
    dep.secret.hash/secret-key: d67cffe0dcf29c9a0e8a28e167a787925e223dea0499230d802b63c30e521750
    dep.secret.hash/trivy-env: 151c72f6b00798cb8c2af3d153bd14ca54d839e39d065c7bcacc783bd97d0268
  creationTimestamp: "2024-03-15T10:40:15Z"
  generateName: harbor-core-66588fc7bd-
  labels:
    app: harbor-core
    pod-template-hash: 66588fc7bd
    project: harbor
  name: harbor-core-66588fc7bd-8b9tn
  namespace: harbor-server
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: harbor-core-66588fc7bd
    uid: e5c76cb3-230b-4b45-a53d-0e5f96012d33
  resourceVersion: "1109966187"
  selfLink: /api/v1/namespaces/harbor-server/pods/harbor-core-66588fc7bd-8b9tn
  uid: 242d70b6-fa9e-4271-9c6a-cd8f724d570f
spec:
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - podAffinityTerm:
          labelSelector:
            matchExpressions:
            - key: app
              operator: In
              values:
              - harbor-core
          topologyKey: kubernetes.io/hostname
        weight: 100
  containers:
  - env:
    - name: DR_META_K8S_CLUSTER_ENV
      value: production
    - name: DR_META_K8S_CLUSTER_NAME
      value: prod-k8s-cluster
    - name: METRIC_SUBSYSTEM
      value: core
    - name: PORT
      value: "8080"
    - name: STAKATER_CORE_ENV_SECRET
      value: 59c3b877edcfe3a23bfe5112f6940687f072418c
    envFrom:
    - secretRef:
        name: harbor-env
    - secretRef:
        name: core-env
    image: reg.deeproute.ai/deeproute-public/harbor-core:v2.10.0
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 2
      httpGet:
        path: /api/v2.0/ping
        port: core
        scheme: HTTP
      initialDelaySeconds: 15
      periodSeconds: 120
      successThreshold: 1
      timeoutSeconds: 1
    name: core
    ports:
    - containerPort: 8080
      name: core
      protocol: TCP
    - containerPort: 8001
      name: metrics
      protocol: TCP
    readinessProbe:
      failureThreshold: 2
      httpGet:
        path: /api/v2.0/ping
        port: core
        scheme: HTTP
      initialDelaySeconds: 5
      periodSeconds: 60
      successThreshold: 1
      timeoutSeconds: 1
    resources:
      limits:
        cpu: "8"
        memory: 16Gi
      requests:
        cpu: "8"
        memory: 16Gi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /etc/core//app.conf
      name: app-conf
      subPath: app.conf
    - mountPath: /etc/core//private_key.pem
      name: private-key
      subPath: private_key.pem
    - mountPath: /etc/core/token
      name: psc
    - mountPath: /etc/core//key
      name: secret-key
      subPath: key
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-6nfll
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  imagePullSecrets:
  - name: image-pull-secret
  nodeName: node01.prod-k8s.bm.pd.sz.deeproute.ai
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext:
    fsGroup: 10000
    runAsUser: 10000
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 120
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - emptyDir: {}
    name: psc
  - configMap:
      defaultMode: 420
      name: app-conf
    name: app-conf
  - name: private-key
    secret:
      defaultMode: 420
      secretName: private-key
  - name: secret-key
    secret:
      defaultMode: 420
      secretName: secret-key
  - name: default-token-6nfll
    secret:
      defaultMode: 420
      secretName: default-token-6nfll

jobservice.yaml

apiVersion: v1
kind: Pod
metadata:
  annotations:
    dep.configmap.hash/app-conf: 5098cc7a4b4970489b11ff4a4d176adaa5c2538c66e330d71c241ae199b20368
    dep.configmap.hash/jobservice-conf: a789a925b9310cb71746003deae0b4bf662f283c42853053696f2a94fa6905a6
    dep.configmap.hash/portal-conf: 4978cdfbc877e6b6015a90557f4603cd506bfe64d16986b79904ecd97a3c4297
    dep.configmap.hash/registry-conf: 7ab3b982a3c9728bff492128abd438d977e1fb0c0935c755b5d1744152d9adc9
    dep.configmap.hash/registry-ctl: 585c3753f4f7d155458b73e9501c0fd6d3002636aa3470ba079ef18ef7ecfcab
    dep.secret.hash/ceph-secret: 44a441dad46bec23b59e3ee2a4c46bcf0188d77207ee7e081c744535e725dc9d
    dep.secret.hash/core-env: 60b03d02d86d64d6de48e1341dee37862b2b9ce5042d89c54d0f80a3c014c8dc
    dep.secret.hash/exporter-env: f218b81357a05155188b3e80578e79159a5c06f62b89e65b99a5788249955a02
    dep.secret.hash/harbor-env: 7effc6edfb35fa76eb6274db2f0c25c3bd663765d61ad28111402e42ba0f76b4
    dep.secret.hash/jobservice-env: 65545ed53a39b359edcb4f5da591bbbbf6cd8b5e33522b50f57ed1fdf465b443
    dep.secret.hash/private-key: 97736853235d1705c806e9b852bb19a20278b66f6cb6e484229a6beebf2dbd2a
    dep.secret.hash/registry-env: 688328dafac719b43d1e58fcb9117639e3f9c88cb2e6e6b6da1e0ab8789b24b5
    dep.secret.hash/registry-htpwd: 3084e97b8b36708eade0a4868b6fad186b99b2bccd235c59ef4467c65fa69647
    dep.secret.hash/secret-key: d67cffe0dcf29c9a0e8a28e167a787925e223dea0499230d802b63c30e521750
    dep.secret.hash/trivy-env: 151c72f6b00798cb8c2af3d153bd14ca54d839e39d065c7bcacc783bd97d0268
  creationTimestamp: "2024-03-15T09:41:57Z"
  generateName: harbor-jobservice-64644b6477-
  labels:
    app: harbor-jobservice
    pod-template-hash: 64644b6477
    project: harbor
  name: harbor-jobservice-64644b6477-lnvpf
  namespace: harbor-server
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: harbor-jobservice-64644b6477
    uid: 08ca140d-00cb-4a71-bb07-e21c22b3e9b6
  resourceVersion: "1109916751"
  selfLink: /api/v1/namespaces/harbor-server/pods/harbor-jobservice-64644b6477-lnvpf
  uid: 14962de8-4c58-497e-a0fe-a25a29e70fc2
spec:
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - podAffinityTerm:
          labelSelector:
            matchExpressions:
            - key: app
              operator: In
              values:
              - harbor-jobservice
          topologyKey: kubernetes.io/hostname
        weight: 100
  containers:
  - env:
    - name: DR_META_K8S_CLUSTER_ENV
      value: production
    - name: DR_META_K8S_CLUSTER_NAME
      value: prod-k8s-cluster
    - name: METRIC_SUBSYSTEM
      value: jobservice
    - name: STAKATER_JOBSERVICE_ENV_SECRET
      value: 94af87eb4dc056ddd0adfb26fbb4f9d3dbcbfb8a
    - name: STAKATER_JOBSERVICE_CONF_CONFIGMAP
      value: 725d96e8e74f6dc70f7edb0b544b1a525d6dbeb3
    envFrom:
    - secretRef:
        name: harbor-env
    - secretRef:
        name: jobservice-env
    image: reg.deeproute.ai/deeproute-public/harbor-jobservice:v2.10.0
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 3
      httpGet:
        path: /api/v1/stats
        port: jobservice
        scheme: HTTP
      initialDelaySeconds: 300
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 1
    name: jobservice
    ports:
    - containerPort: 8080
      name: jobservice
      protocol: TCP
    - containerPort: 8001
      name: metrics
      protocol: TCP
    readinessProbe:
      failureThreshold: 3
      httpGet:
        path: /api/v1/stats
        port: jobservice
        scheme: HTTP
      initialDelaySeconds: 20
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 1
    resources:
      limits:
        cpu: "2"
        memory: 4Gi
      requests:
        cpu: "2"
        memory: 4Gi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/log/jobs
      name: job-logs
    - mountPath: /etc/jobservice//config.yml
      name: jobservice-conf
      subPath: config.yml
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-6nfll
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  imagePullSecrets:
  - name: image-pull-secret
  initContainers:
  - command:
    - sh
    - -c
    - for i in $(seq 1 60); do nc -z -w3 harbor-core:80  && exit 0 || sleep 5; done;
      exit 1
    env:
    - name: DR_META_K8S_CLUSTER_ENV
      value: production
    - name: DR_META_K8S_CLUSTER_NAME
      value: prod-k8s-cluster
    image: reg.deeproute.ai/deeproute-public/busybox:latest
    imagePullPolicy: IfNotPresent
    name: wait-for-core
    resources:
      limits:
        cpu: 500m
        memory: 512Mi
      requests:
        cpu: 200m
        memory: 256Mi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-6nfll
      readOnly: true
  nodeName: node08.prod-k8s.bm.pd.sz.deeproute.ai
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext:
    fsGroup: 10000
    runAsUser: 10000
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 120
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - cephfs:
      monitors:
      - <mon-ip>:6789
      path: /volumes/<cephfs-path>
      secretRef:
        name: ceph-secret
      user: csi-cephfs-node
    name: job-logs
  - configMap:
      defaultMode: 420
      name: jobservice-conf
    name: jobservice-conf
  - name: default-token-6nfll
    secret:
      defaultMode: 420
      secretName: default-token-6nfll

from harbor.

microyahoo commented on May 24, 2024

Hi @chlins, I filtered the two request ids in the core log and found that it only took less than 1min to go from attach request ID to rollback. However, when review code of updating the quota usage, there will be a retry defaultRetryTimeout of 5 minutes. I am not sure whether this has an impact. Please help to confirm and give some suggestions, thank you.

~/go/src/deeproute/harbor-cli (main ✗) ag d9e69564-3e7b-4f2f-801d-066ac2e35c96 core.*
core.3
36770:2024-03-15T10:43:21Z [DEBUG] [/server/middleware/log/log.go:31]: attach request id d9e69564-3e7b-4f2f-801d-066ac2e35c96 to the logger for the request PUT /v2/project-015/repository-1710499314344/blobs/uploads/48fc0d3c-dde7-4a6c-bf44-7f94aa48356c
36912:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
36952:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
36991:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37033:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37076:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37113:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37151:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37186:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37220:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37254:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37291:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37326:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37359:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37393:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37424:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37456:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37488:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37518:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37545:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37570:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37597:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37628:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37653:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37679:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37706:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37731:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37757:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37782:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37805:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37833:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37861:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37887:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37910:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37934:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37958:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
37982:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38005:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38027:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38048:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38068:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38088:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38107:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38126:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38144:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38161:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38178:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38193:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38207:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38222:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38236:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38252:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38265:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38282:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38295:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38308:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38320:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38336:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38351:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38364:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38376:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38388:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38398:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38411:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38426:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38438:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38451:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38462:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38475:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38488:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38498:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38508:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38517:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38531:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38539:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38547:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38553:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38561:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38567:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38572:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38577:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38580:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38583:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38586:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38588:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38590:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38595:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
38602:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: failed to update the quota usage for project 133, error: redis: transaction failed
39203:2024-03-15T10:43:45Z [WARNING] [/controller/quota/controller.go:402][requestID="d9e69564-3e7b-4f2f-801d-066ac2e35c96"]: rollback resources {"storage":47} for project 133 failed, error: context canceled

~/go/src/deeproute/harbor-cli (main ✗) ag 96b18f4d-bcf4-426f-b7a2-383e24e0f65b core.*
core.3
37842:2024-03-15T10:43:21Z [DEBUG] [/server/middleware/log/log.go:31]: attach request id 96b18f4d-bcf4-426f-b7a2-383e24e0f65b to the logger for the request PUT /v2/project-015/repository-1710499314344/blobs/uploads/a7f354c6-b70c-4bae-b588-f908a753ff85
37932:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
37957:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
37981:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
38004:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
38026:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
38047:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
38066:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
38084:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
38103:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
38121:2024-03-15T10:43:21Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
38139:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
38156:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
38171:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
38185:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
38200:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
38215:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
38230:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
38246:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
38261:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
38275:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
38290:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
38305:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
38319:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
38331:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
38347:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
38362:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
38373:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
38382:2024-03-15T10:43:22Z [DEBUG] [/controller/quota/controller.go:353][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: failed to update the quota usage for project 133, error: redis: transaction failed
39516:2024-03-15T10:44:09Z [WARNING] [/controller/quota/controller.go:402][requestID="96b18f4d-bcf4-426f-b7a2-383e24e0f65b"]: rollback resources {"storage":10485760} for project 133 failed, error: context canceled

from harbor.

microyahoo commented on May 24, 2024

I try to set quota_per_project_enable to false to deactivating update project quota usage, but there is still no improvement in the success rate.

insert into properties (k, v) values('quota_per_project_enable', false);

from harbor.

chlins commented on May 24, 2024

There should be a debug log message if you disabled the quota_per_project_enable if you set the log level to debug, could you confirm it?

harbor/src/server/middleware/quota/quota.go

Line 88 in 69fc957

 logger.Debugf("quota is deactivated for %s %s, so direct to next handler", reference, referenceID) 

BTW, is the error message same with previous after you set the quota_per_project_enable to false?

from harbor.

microyahoo commented on May 24, 2024

There should be a debug log message if you disabled the quota_per_project_enable if you set the log level to debug, could you confirm it?

harbor/src/server/middleware/quota/quota.go

Line 88 in 69fc957

logger.Debugf("quota is deactivated for %s %s, so direct to next handler", reference, referenceID)

BTW, is the error message same with previous after you set the quota_per_project_enable to false?

yes, I already toggle on the core debug log, and the message for quota is deactivated for project xxx, so direct to next handler appears in large numbers in the core log. The previous error message redis: transaction failed and rollback resources {"storage":47} for project 133 failed, error: context canceled have all gone. Therefore, it can be proven that the error has nothing to do with redis. There are many 500 errors in the registry, such as Path not found, client disconnected, etc.

In addition, I found a clue, taking upload id 056f13cc-f188-4959-aa83-5c6fcadef6c7 as an example, while the registry uses s3 as the backend storage.
2024-03-18T06:50:37.686 s3 received the POST request for multipart upload and returned the s3 upload id, but did not receive the data to upload.

[root@cicd-node01 ~]# for i in `seq 141 146`; do echo 10.3.9.$i; ssh 10.3.9.$i 'ag 056f13cc-f188-4959-aa83-5c6fcadef6c7 /var/lib/rook/rook-ceph/log/ceph-client.rgw.os.*';  done
10.3.9.141
2024-03-18T06:50:36.411+0000 7fb7e8be0700  1 beast: 0x7fb750d88620: 10.10.12.33 - harbor [18/Mar/2024:06:50:36.081 +0000] "PUT /harbor/docker/registry/v2/repositories/project-062/repository-1710744554486/_uploads/056f13cc-f188-4959-aa83-5c6fcadef6c7/startedat HTTP/1.1" 200 20 - "aws-sdk-go/1.15.11 (go1.21.4; linux; amd64)" - latency=0.330001682s
2024-03-18T06:50:37.686+0000 7fb75cac8700  1 beast: 0x7fb750d88620: 10.10.12.33 - harbor [18/Mar/2024:06:50:36.458 +0000] "POST /harbor/docker/registry/v2/repositories/project-062/repository-1710744554486/_uploads/056f13cc-f188-4959-aa83-5c6fcadef6c7/data?uploads= HTTP/1.1" 200 356 - "aws-sdk-go/1.15.11 (go1.21.4; linux; amd64)" - latency=1.228006363s
2024-03-18T06:50:37.692+0000 7fb802c14700  1 beast: 0x7fb74c06f620: 10.10.12.33 - harbor [18/Mar/2024:06:50:37.688 +0000] "PUT /harbor/docker/registry/v2/repositories/project-062/repository-1710744554486/_uploads/056f13cc-f188-4959-aa83-5c6fcadef6c7/hashstates/sha256/0 HTTP/1.1" 200 108 - "aws-sdk-go/1.15.11 (go1.21.4; linux; amd64)" - latency=0.004000021s
10.3.9.142
10.3.9.143
10.3.9.144
10.3.9.145
2024-03-18T06:50:37.898+0000 7f3f06920700  1 beast: 0x7f3e51f26620: 10.10.12.36 - harbor [18/Mar/2024:06:50:37.896 +0000] "GET /harbor/docker/registry/v2/repositories/project-062/repository-1710744554486/_uploads/056f13cc-f188-4959-aa83-5c6fcadef6c7/startedat HTTP/1.1" 206 20 - "aws-sdk-go/1.15.11 (go1.21.4; linux; amd64)" bytes=0- latency=0.002000014s
2024-03-18T06:50:48.032+0000 7f3eab069700  1 beast: 0x7f3e506f6620: 10.10.12.36 - harbor [18/Mar/2024:06:50:37.940 +0000] "GET /harbor?prefix=docker%2Fregistry%2Fv2%2Frepositories%2Fproject-062%2Frepository-1710744554486%2F_uploads%2F056f13cc-f188-4959-aa83-5c6fcadef6c7%2Fdata&uploads= HTTP/1.1" 200 375 - "aws-sdk-go/1.15.11 (go1.21.4; linux; amd64)" - latency=10.092069626s
10.3.9.146

2024-03-18T06:50:48.061206954Z registry calls s3.stat to query 056f13cc-f188-4959-aa83-5c6fcadef6c7/data and finds that it does not exist and returns 500.

time="2024-03-18T06:50:48.061206954Z" level=error msg="error resolving upload: s3aws: Path not found: /docker/registry/v2/repositories/project-062/repository-1710744554486/_uploads/056f13cc-f188-4959-aa83-5c6fcadef6c7/data" auth.user.name="harbor_registry_user" go.version=go1.21.4 http.request.host=reg.srv.deeproute.cn http.request.id=6dcc0b71-ea73-490e-9002-438a39b67061 http.request.method=PUT http.request.remoteaddr=10.10.12.31 http.request.uri="/v2/project-062/repository-1710744554486/blobs/uploads/056f13cc-f188-4959-aa83-5c6fcadef6c7?_state=e4olaHxH5oH62wPrPgfZxg32hYqUOQCRz-ZhRFI7FQt7Ik5hbWUiOiJwcm9qZWN0LTA2Mi9yZXBvc2l0b3J5LTE3MTA3NDQ1NTQ0ODYiLCJVVUlEIjoiMDU2ZjEzY2MtZjE4OC00OTU5LWFhODMtNWM2ZmNhZGVmNmM3IiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDI0LTAzLTE4VDA2OjUwOjM2LjA4ODg4MjYzOVoifQ%3D%3D&digest=sha256%3Abc927d17069db75a4708a98a8e3ca8fb1a29683bff3ecbd6efae0cde2ba0f18d" http.request.useragent="containerd/1.3.2+unknown" vars.name="project-062/repository-1710744554486" vars.uuid=056f13cc-f188-4959-aa83-5c6fcadef6c7

harbor.log.tar.gz

from harbor.

microyahoo commented on May 24, 2024

hi @chlins, I notice another clue, the perf blob file is not existed when I comment out perf teardown to keep blob digest file.

from harbor.

microyahoo commented on May 24, 2024

Hi @chlins, I hacked harbor-core code and add detailed log for some middlewares, take PUT v2_manifest for example. Because I noticed that putting manifest sometimes takes a long time.

For the registry, the time spent putting manifest is negligible.

For harbor-core, The total time spent on put manifest is 1m, of which putManifest takes 17s and PutManifestMiddleware after response takes 29s.

from harbor.

microyahoo commented on May 24, 2024

I roughly counted one of the put manifest requests. The time overhead in each stage is as follows:

metric                                                  50.833120332s   2024-03-20T11:29:42
    repoproxy.DisableBlobAndManifestUploadMiddleware    1.647785287s    2024-03-20T11:29:03
    immutable.Middleware                                2.444636649s    2024-03-20T11:29:05
    quota.PutManifestMiddleware                         36.183096215s   2024-03-20T11:29:41
        blob.PutManifestMiddleware before request       208.525058ms    2024-03-20T11:29:09
        putManifest                                     6.580389936s    2024-03-20T11:29:16
        blob.PutManifestMiddleware after response       24.024166654s   2024-03-20T11:29:40
        subject.Middleware                              1.382551771s    2024-03-20T11:29:41
        cosign.SignatureMiddleware                      12.553us        2024-03-20T11:29:41

In the above, the entire request takes about 50 seconds, of which the actual registry processing time is less than 1 second.

from harbor.

chlins commented on May 24, 2024

Could you try to deploy the original registry https://github.com/distribution/distribution on your environment and run the performance push case to it? In theory, if the quota is turned off, then the harbor core and the registry only use golang's reverse proxy to forward traffic, and there should not be such a big gap.

from harbor.

microyahoo commented on May 24, 2024

Could you try to deploy the original registry https://github.com/distribution/distribution on your environment and run the performance push case to it? In theory, if the quota is turned off, then the harbor core and the registry only use golang's reverse proxy to forward traffic, and there should not be such a big gap.

hi @chlins, yes, the analysis result of the above comment is to use the original registry. I added some logs on it and recompiled and deployed it. However, the results when running with harbor perf are similar. The performance is improved, but the improvement is not significant.

from harbor.

microyahoo commented on May 24, 2024

I analyzed one of the put v2_blob_upload requests, and the specific time cost is as follows:

metric                                                  1m3.340869412s  2024-03-20T11:29:55
    quota.PutManifestMiddleware                         55.734914587s   2024-03-20T11:29:55
        blob.PutManifestMiddleware before request       682.602959ms    2024-03-20T11:29:03
        proxy                                           44.444999102s   2024-03-20T11:29:48
            s3                                          0.066000909s    2024-03-20T11:29:48
        blob.PutManifestMiddleware after response       5.063451415s    2024-03-20T11:29:55

Not sure where the extra >10s of overhead is? The above put manifest request is also similar too.

from harbor.

microyahoo commented on May 24, 2024

Hi @chlins, is it possible that the perf client is processing requests too slowly? For example, it takes 44 seconds to push 10MB of data, but the actual io.copy to s3 storage takes less than one second.

from harbor.

microyahoo commented on May 24, 2024

The following is my test result after turning off redis quota.

push-artifacts-to-different-projects

VUS	BLOB SIZE(kbyte)	Avg Duration	Min Duration	Max Duration	Med Duration	P90	P95	SUCCESS RATE	ITERATIONS RATE
50	4	4.38s	2.18s	4.79s	4.41s	4.71s	4.74s	100%	3.40/s
50	512	4.69s	4.15s	4.93s	4.71s	4.87s	4.9s	100%	3.22/s
50	1024	4.48s	3.3s	4.78s	4.49s	4.7s	4.73s	100%	3.26/s
50	5120	4.54s	3.84s	4.82s	4.53s	4.79s	4.79s	100%	2.68/s
50	10240	4.4s	3.04s	4.94s	4.38s	4.88s	4.92s	100%	2.35/s
50	20480	4.57s	2.46s	8.41s	4.29s	6.12s	6.8s	100%	1.76/s
50	40960	6.36s	2.95s	12.72s	5.41s	9.84s	11.52s	100%	1.16/s
50	102400	11.94s	5.58s	20.16s	12.83s	16.13s	16.87s	100%	0.57/s
100	4	7.15s	2.96s	8.29s	7.22s	7.87s	8.03s	100%	3.42/s
100	512	7.8s	6.03s	8.58s	7.75s	8.42s	8.47s	100%	3.33/s
100	1024	7.87s	6s	8.7s	7.7s	8.57s	8.61s	100%	3.23/s
100	5120	7.8s	4.9s	8.88s	7.73s	8.41s	8.62s	100%	2.66/s
100	10240	8s	6.44s	8.9s	7.91s	8.82s	8.84s	100%	2.27/s
100	20480	8.16s	5.46s	10.09s	8.24s	8.79s	8.94s	100%	1.75/s
100	40960	9.01s	3.57s	23.53s	8.36s	14.46s	16.77s	100%	1.21/s
100	102400	24.31s	8.45s	1m13.99s	18.2s	47.56s	59.53s	100%	0.53/s
200	4	15.45s	5.02s	18.25s	15.2s	17.13s	17.55s	100%	3.51/s
200	512	15.21s	11.42s	18.84s	15.12s	16.48s	16.91s	100%	3.37/s
200	1024	14.77s	8.94s	20.98s	14.66s	16.78s	17.56s	100%	3.47/s
200	5120	15.54s	11.99s	17.38s	15.46s	16.57s	16.68s	100%	2.72/s
200	10240	14.32s	7.93s	23.44s	14.04s	18.45s	19.5s	100%	2.29/s
200	20480	15.53s	7.24s	20.13s	15.52s	18.23s	18.75s	100%	1.63/s
200	40960	18.97s	4.07s	52.09s	15.76s	34.82s	41.62s	100%	1.14/s
200	102400	1m14.43s	22.22s	2m16.78s	1m12.56s	1m51.58s	1m59.24s	100%	0.53/s
300	4	20.44s	1.69s	30.99s	20.38s	26.34s	27.73s	99.33%	3.61/s
300	512	20.42s	7.59s	28.34s	18.78s	25.56s	26.16s	100%	3.49/s
300	1024	20.87s	1.76s	30.79s	20.89s	26.06s	27.25s	99.33%	3.43/s
300	5120	19.86s	1.87s	31.07s	20.02s	25.58s	26.34s	99.5%	2.83/s
300	10240	19.99s	2.11s	33.73s	19.37s	26.44s	28.14s	98.16%	2.45/s
300	20480	20.19s	3.59s	33.34s	20.07s	27.44s	28.92s	99.66%	1.84/s
300	40960	39.37s	10.46s	1m30.68s	38.18s	1m0.03s	1m8.77s	100%	1.19/s
300	102400	1m50.89s	17.99s	3m48.34s	1m51.11s	2m32.67s	2m52.09s	100%	0.56/s
400	4	29.85s	10.33s	41.13s	29.81s	32.08s	32.62s	100%	3.49/s
400	512	27.99s	1.12s	51.09s	27.81s	33.93s	36.2s	96.5%	5.03/s
400	1024	26.86s	1.71s	46.31s	27.06s	31.89s	32.64s	97%	4.84/s
400	5120	28.53s	1.39s	46.63s	28.75s	34.32s	36.72s	97.5%	3.87/s
400	10240	28.85s	1.21s	46.99s	28.54s	35.66s	37.6s	99.5%	3.06/s
400	20480	29.09s	1.8s	1m12.35s	28.19s	42.65s	53.6s	99.62%	2.08/s
400	40960	57.76s	7.17s	2m5.21s	55.57s	1m30.39s	1m43.55s	100%	1.15/s
400	102400	2m21.81s	12.95s	5m7.45s	2m19.01s	3m39.94s	4m3.83s	100%	0.56/s
500	4	32.18s	1.5s	54.21s	31.98s	40.73s	42.8s	97.5%	13.18/s
500	512	27.69s	3.31s	57.55s	26.03s	42.59s	46.15s	96.4%	13.91/s
500	1024	31.22s	1.72s	55.97s	31s	40.27s	43.12s	93.9%	11.70/s
500	5120	32.51s	2.27s	56.4s	32.35s	40.43s	42.34s	97.8%	6.70/s
500	10240	31.79s	1.6s	56.97s	31.12s	42.7s	46.45s	97.9%	3.17/s
500	20480	32.38s	2.94s	1m1.66s	31.76s	44.25s	45.88s	99.2%	1.94/s
500	40960	1m13s	9.46s	2m30.95s	1m10.53s	1m52.51s	2m0.67s	100%	1.14/s
500	102400	3m1.27s	13.7s	6m50.15s	2m57.85s	4m14.3s	4m40.32s	100%	0.55/s
1000	4	47.14s	307.48ms	1m42.59s	45.03s	1m18.18s	1m25.66s	85.25%	17.41/s
1000	512	52.53s	515.47ms	1m55.77s	45.88s	1m24.78s	1m32.46s	90.5%	14.31/s
1000	1024	51.41s	312.84ms	1m48.92s	45.97s	1m25.46s	1m31.33s	85.95%	13.52/s
1000	5120	49.83s	328.2ms	1m44.26s	48.69s	1m16.78s	1m23.47s	88.2%	7.82/s
1000	10240	55.11s	589.08ms	1m52.43s	52.13s	1m23.79s	1m30.69s	91.9%	5.11/s
1000	20480	1m8.97s	5.74s	2m35.85s	1m4.5s	1m47.31s	1m59.7s	98.1%	2.27/s
1000	40960	2m26.36s	17.45s	5m38.16s	2m19.13s	4m11.1s	4m41.72s	99.55%	1.34/s
1000	102400	6m28.78s	53.38s	10m25.54s	6m29.55s	9m18.23s	9m51.78s	100%	0.31/s

push-artifacts-to-same-projects

VUS	BLOB SIZE(kbyte)	Avg Duration	Min Duration	Max Duration	Med Duration	P90	P95	SUCCESS RATE	ITERATIONS RATE
50	4	4.74s	2.11s	5.21s	4.58s	5.17s	5.18s	100%	3.11/s
50	512	4.98s	3.54s	6.12s	5.18s	5.82s	5.94s	100%	2.90/s
50	1024	4.86s	2.4s	5.47s	5.1s	5.31s	5.42s	100%	2.88/s
50	5120	6.51s	3.48s	8.83s	6.16s	8.75s	8.77s	100%	2.27/s
50	10240	4.93s	2.76s	7.08s	5.32s	5.43s	5.64s	100%	2.17/s
50	20480	5.83s	2.18s	14.41s	4.77s	9.65s	11.29s	100%	1.61/s
50	40960	6.44s	2.55s	17.01s	5.55s	11s	12.82s	100%	1.11/s
50	102400	20.32s	3.85s	1m7.35s	11.06s	55.02s	57.86s	100%	0.45/s
100	4	7.99s	4.34s	9.28s	7.9s	8.61s	8.89s	100%	3.24/s
100	512	8.36s	2.87s	9.8s	8.37s	9.15s	9.31s	100%	3.25/s
100	1024	8.08s	3.86s	8.86s	8.11s	8.45s	8.56s	100%	3.04/s
100	5120	7.95s	3.51s	13.21s	7.96s	9.05s	9.55s	100%	2.75/s
100	10240	9.51s	2.66s	22.08s	8.72s	14.58s	16.27s	100%	2.20/s
100	20480	11.09s	2.91s	29.01s	9.23s	18.81s	20.51s	100%	1.60/s
100	40960	19.68s	3s	1m4.45s	13.36s	43.17s	49.85s	100%	0.94/s
100	102400	1m11.02s	4.23s	3m22.94s	35.41s	3m13.42s	3m22.8s	76.5%	0.45/s
200	4	14.96s	4.85s	26.02s	14.28s	22.12s	22.65s	100%	3.24/s
200	512	15.67s	5.43s	26.09s	15.74s	18.68s	20.89s	100%	3.12/s
200	1024	14.35s	6.17s	24.49s	14.21s	18.36s	19.33s	100%	3.18/s
200	5120	19.32s	1.88s	41.62s	18.74s	27.62s	34.76s	100%	2.21/s
200	10240	18.73s	2.56s	56.4s	16.88s	33.76s	41.05s	100%	1.90/s
200	20480	48.54s	1.94s	2m35.81s	37.72s	1m44.99s	2m5.05s	100%	1.15/s
200	40960	32.28s	3.24s	2m3.22s	25.4s	1m1.33s	1m21.78s	100%	0.95/s
200	102400	1m27.57s	15.01s	4m51.97s	1m11.8s	2m37.25s	3m18.62s	100%	0.46/s
300	4	19.45s	4.84s	31.51s	18.74s	26.25s	27.31s	100%	3.10/s
300	512	18.98s	2.84s	32.79s	18.31s	25.12s	26.45s	99.66%	3.15/s
300	1024	20.04s	4.64s	33.62s	18.78s	27.68s	28.74s	99.83%	3.25/s
300	5120	19.55s	4.68s	40.68s	18.36s	28.5s	33.17s	99.83%	2.43/s
300	10240	22.79s	3.36s	1m8.75s	20.29s	37.27s	48.88s	100%	1.90/s
300	20480	40.12s	2.34s	2m37.96s	26.65s	1m31.58s	1m48.56s	100%	1.29/s
300	40960	53.26s	7.83s	3m11.42s	44.97s	1m47.03s	2m5.72s	100%	0.95/s
300	102400	2m50.62s	8.59s	9m8.57s	1m49.71s	7m3.64s	7m44.12s	89.33%	0.47/s
400	4	25.8s	2.13s	40.64s	26.32s	31.28s	31.7s	97.62%	4.43/s
400	512	28.52s	1.09s	41.72s	28.21s	32.54s	33.16s	98.87%	11.54/s
400	1024	29.68s	1.8s	40s	29.72s	33.23s	35.72s	99.87%	4.15/s
400	5120	26.99s	1.78s	43.4s	26.17s	33.56s	34.64s	98.25%	3.67/s
400	10240	30.23s	9.67s	1m4.26s	29.8s	40.18s	44.97s	100%	2.15/s
400	20480	41.29s	2.96s	2m0.12s	35.19s	1m15.6s	1m24.8s	100%	1.53/s
400	40960	1m30.17s	2.6s	5m11.48s	1m15.42s	2m41.71s	3m22.19s	100%	0.88/s
400	102400	2m24.7s	2.89s	5m26.1s	1m57.51s	3m57.75s	3m58.01s	50.12%	0.60/s
500	4	31.58s	1.77s	1m0.5s	31.42s	39.44s	42.54s	97%	13.10/s
500	512	26.41s	1.9s	52.11s	26.45s	36.58s	38s	96.8%	9.11/s
500	1024	30.29s	555.07ms	57.54s	29.12s	43.78s	46.7s	96.2%	11.58/s
500	5120	32.11s	1.73s	57.83s	32.01s	39.29s	41.8s	97.6%	6.67/s
500	10240	31.46s	2.24s	56.39s	31.48s	39.87s	42.47s	98.4%	3.03/s
500	20480	33.39s	7.21s	1m15.49s	32.11s	50.87s	56.74s	100%	1.84/s
500	40960	1m11.91s	8.71s	2m29.33s	1m9.86s	1m48.74s	1m58.35s	100%	1.13/s
500	102400	3m4.63s	13.89s	6m49.78s	2m57.55s	4m52.42s	5m37.59s	100%	0.55/s
1000	4	51.03s	318.37ms	1m48.99s	46.51s	1m26.75s	1m34.58s	88.75%	15.81/s
1000	512	46.45s	296.48ms	1m43.93s	43.64s	1m13.06s	1m20.76s	87.65%	15.42/s
1000	1024	41.23s	314.81ms	1m30.02s	41.33s	1m2.35s	1m8.32s	87%	14.28/s
1000	5120	46.42s	331.87ms	1m40.58s	45.98s	1m10.04s	1m19.85s	86.6%	7.71/s
1000	10240	45.85s	553.95ms	1m38.19s	44.8s	1m11.57s	1m18.68s	87.15%	4.99/s
1000	20480	1m14.08s	5.68s	2m51.33s	1m10.98s	1m47.42s	2m4.8s	92.65%	2.77/s
1000	40960	2m32.44s	7.56s	5m43.41s	2m23.58s	4m21.09s	4m53.73s	96.7%	1.41/s
1000	102400	6m23.78s	42.64s	10m26.23s	6m34.98s	9m25.75s	9m46.52s	100%	0.29/s

pull-artifacts-from-different-projects

VUS	BLOB SIZE(kbyte)	Avg Duration	Min Duration	Max Duration	Med Duration	P90	P95	SUCCESS RATE	ITERATIONS RATE
50	4	905.89ms	549.64ms	1.65s	890.71ms	1.09s	1.32s	100%	0.45/s
50	512	919.14ms	561.73ms	1.5s	933.08ms	1.2s	1.4s	100%	0.51/s
50	1024	929.66ms	476.97ms	2.22s	949.02ms	1.27s	1.29s	100%	0.50/s
50	5120	1.81s	890.85ms	3.95s	1.73s	2.69s	2.8s	100%	0.47/s
50	10240	3.62s	670.83ms	9.7s	2.83s	7.17s	7.68s	100%	0.42/s
50	20480	5.37s	1.46s	16.54s	3.54s	12.04s	14.21s	100%	0.36/s
50	40960	9.59s	2.46s	29.41s	6.39s	19.88s	23.78s	100%	0.27/s
50	102400	22.71s	4.97s	1m12.59s	15.28s	58.42s	1m5.77s	100%	0.13/s
100	4	1.7s	599.71ms	3.17s	1.66s	2.54s	2.73s	100%	0.90/s
100	512	1.23s	607.68ms	2.37s	1.17s	1.7s	1.95s	100%	0.84/s
100	1024	3.51s	719.66ms	7.51s	2.35s	6s	6.24s	100%	0.79/s
100	5120	2.85s	787.43ms	7.42s	2.57s	4.81s	5.41s	100%	0.88/s
100	10240	4.23s	1.58s	9.47s	4.05s	5.79s	6.11s	100%	0.79/s
100	20480	7.83s	3.54s	18.26s	7.13s	11.89s	14.05s	100%	0.77/s
100	40960	18.9s	2.75s	57.54s	13.46s	39.95s	48.57s	100%	0.51/s
100	102400	41.33s	9.61s	2m30.11s	26.82s	1m34.66s	1m42.67s	100%	0.27/s
200	4	1.85s	803.68ms	4.46s	1.7s	2.9s	3.3s	100%	1.93/s
200	512	2.72s	896.46ms	7.02s	2.51s	4.3s	5.1s	100%	1.92/s
200	1024	2.87s	1.09s	7.12s	2.66s	4.25s	4.85s	100%	1.87/s
200	5120	6.24s	1.21s	16.72s	5.86s	9.74s	11.12s	100%	1.73/s
200	10240	7.55s	2.26s	18.95s	7.07s	11.2s	14.04s	100%	1.58/s
200	20480	14.53s	1.67s	38.99s	12.42s	24.7s	29.88s	100%	1.39/s
200	40960	27.77s	6.83s	1m11.48s	24.57s	47.95s	55.61s	100%	1.04/s
200	102400	1m12.39s	11.96s	3m25.04s	1m1.22s	2m18.94s	2m33.71s	100%	0.60/s
300	4	3.17s	981.03ms	7.2s	2.88s	5.19s	5.88s	100%	2.78/s
300	512	3.11s	857.19ms	8.77s	2.73s	5.22s	6.17s	100%	2.81/s
300	1024	3.46s	980.37ms	8.61s	3.13s	5.84s	6.88s	100%	2.80/s
300	5120	8.04s	1.45s	21.28s	7.25s	13.03s	16.1s	100%	2.47/s
300	10240	11.28s	2.63s	30.32s	9.72s	19.73s	23.72s	100%	2.10/s
300	20480	21.01s	4.73s	54.69s	18s	38.26s	43.72s	100%	1.93/s
300	40960	39.06s	5.24s	1m41.75s	33.05s	1m12.87s	1m25.23s	100%	1.42/s
300	102400	1m37.05s	22.82s	4m14.15s	1m24.01s	3m5.44s	3m33.67s	100%	0.83/s
400	4	3.55s	1s	9.44s	3.11s	6.02s	7.21s	100%	3.79/s
400	512	3.6s	998.07ms	9.53s	3.18s	6.01s	7.15s	100%	3.79/s
400	1024	6.91s	1.32s	16.66s	6.34s	12.08s	13.8s	100%	3.61/s
400	5120	10.64s	1.72s	26.9s	9.76s	18.06s	21.07s	100%	3.31/s
400	10240	15.41s	2.74s	38.7s	13.5s	27.17s	31.59s	100%	2.98/s
400	20480	27.88s	6.11s	1m10.95s	24.97s	51.12s	58.39s	100%	2.44/s
400	40960	52.82s	7.18s	2m17.31s	44.93s	1m41.49s	1m57s	100%	1.74/s
400	102400	2m8.79s	1.6s	5m37.22s	1m48.55s	4m16.98s	4m45.3s	97.12%	0.98/s
500	4	4.45s	896.87ms	11.56s	3.91s	8.03s	9.18s	100%	4.76/s
500	512	5.9s	1.36s	15.37s	5.23s	10.69s	12.41s	100%	4.42/s
500	1024	8.73s	1.86s	21.89s	7.61s	14.86s	18.06s	100%	4.21/s
500	5120	31.63s	655.32ms	1m40.06s	25.77s	1m5.96s	1m13.83s	100%	2.85/s
500	10240	48.04s	754.34ms	2m30.51s	41.05s	1m34.79s	1m47.66s	100%	2.30/s
500	20480	47.48s	34.06ms	2m27.31s	47.34s	1m23.25s	1m49.9s	58.8%	2.22/s
500	40960	1m1.87s	30.19ms	2m59.56s	53.84s	2m9.57s	2m28.35s	78.1%	1.80/s
500	102400	2m1.92s	76.68ms	6m12.34s	1m38.31s	4m30.7s	5m0.92s	74.7%	0.89/s
1000	4	9.63s	1.1s	25.27s	9.16s	16.3s	19.57s	100%	7.70/s
1000	512	10.28s	1.03s	25.95s	9.07s	19.16s	22.25s	100%	7.67/s
1000	1024	18.95s	1.46s	44.72s	18.44s	30.87s	37.54s	100%	7.07/s
1000	5120	34.57s	1.03s	1m43.5s	30.11s	1m5.84s	1m16.84s	100%	5.57/s
1000	10240	58.56s	1.02s	2m28.98s	50.64s	1m57.02s	2m7.29s	96.85%	4.62/s
1000	20480	1m8.26s	2.73s	3m15.7s	58.81s	2m18.68s	2m36.42s	100%	3.98/s
1000	40960	3m44.71s	161.81ms	9m50.72s	3m21.81s	7m46.59s	8m30.53s	84.45%	2.02/s
1000	102400	3m32.82s	2.97s	10m25.6s	2m48.56s	7m33.87s	8m59.61s	94.87%	0.72/s

pull-artifacts-from-same-project

VUS	BLOB SIZE(kbyte)	Avg Duration	Min Duration	Max Duration	Med Duration	P90	P95	SUCCESS RATE	ITERATIONS RATE
50	4	9.74s	553.59ms	22.78s	8.47s	20.16s	21.19s	100%	3.76/s
50	512	10.16s	662.95ms	24.88s	8.6s	20.91s	22.57s	100%	3.59/s
50	1024	10.98s	650.07ms	28.41s	8.93s	23.24s	25.52s	100%	3.11/s
50	5120	14.05s	618.32ms	37.42s	11.88s	27.33s	33.64s	100%	2.38/s
50	10240	18.2s	1.04s	47.6s	17.04s	35.35s	39.85s	100%	1.94/s
50	20480	23.67s	1.06s	1m4.14s	20.26s	47.78s	53.92s	100%	1.46/s
50	40960	46.57s	1.57s	2m3.46s	37.95s	1m36.6s	1m51.36s	100%	0.76/s
50	102400	1m42.44s	1.75s	4m39.63s	1m23.01s	3m49.76s	4m9.6s	100%	0.35/s
100	4	18.88s	666.62ms	45.47s	17.05s	35.84s	39.94s	100%	4.02/s
100	512	20.32s	1.3s	50.45s	17.36s	40.33s	44.3s	100%	3.59/s
100	1024	21.88s	824.46ms	53.05s	19.9s	43.02s	47.25s	100%	3.25/s
100	5120	28.8s	892.43ms	1m15.26s	26.17s	1m1.22s	1m7s	100%	2.45/s
100	10240	36.26s	679.41ms	1m33.77s	34.42s	1m13.09s	1m19.51s	100%	2.00/s
100	20480	1m0.14s	1.14s	2m43.9s	52.82s	2m8.27s	2m20.67s	100%	1.16/s
100	40960	1m36.22s	1.38s	4m30.53s	1m24.45s	3m13.94s	3m56.24s	100%	0.72/s
100	102400	1m52.76s	1.88s	4m54.07s	1m42.2s	3m48.03s	4m9.64s	100%	0.63/s
200	4	38.06s	809.6ms	1m29.51s	34.55s	1m14.06s	1m22.7s	100%	4.10/s
200	512	38.64s	849.02ms	1m31.87s	34.45s	1m15.82s	1m23.67s	100%	4.01/s
200	1024	41.66s	1.11s	1m44.51s	38.66s	1m20.33s	1m27.81s	100%	3.49/s
200	5120	59.33s	1.55s	2m28.27s	54.89s	1m55.64s	2m10.38s	100%	2.50/s
200	10240	1m4.03s	920.08ms	2m46.36s	58.22s	2m10.48s	2m28.14s	100%	2.28/s
200	20480	2m20.2s	1.61s	6m4.15s	2m6.47s	4m50.35s	5m17.46s	100%	1.04/s
200	40960	3m47.48s	1.48s	10m4.33s	3m14.51s	8m2.14s	8m45.93s	100%	0.64/s
200	102400	3m21.47s	2.35s	10m27.52s	2m41.11s	7m20.53s	8m14.15s	100%	0.33/s
300	4	55.24s	947.87ms	2m10.02s	49.55s	1m46.52s	2m0.3s	100%	4.13/s
300	512	1m0.95s	939.97ms	2m29.64s	55.73s	1m59.85s	2m12.37s	100%	3.76/s
300	1024	1m10.47s	991.62ms	3m3.87s	1m2.4s	2m21.42s	2m40.36s	100%	3.10/s
300	5120	1m55.33s	935.28ms	5m6s	1m43.41s	3m55.27s	4m26.14s	100%	1.91/s
300	10240	2m46.12s	884.21ms	7m27.34s	2m17.75s	5m49.54s	6m21.01s	100%	1.31/s
300	20480	2m50.13s	842.53ms	6m58.22s	2m28.28s	5m47.92s	6m24.18s	100%	1.39/s
300	40960	3m31.36s	1.22s	10m23.55s	2m45.32s	7m59.27s	9m18.85s	100%	0.85/s
300	102400	3m51.39s	2.06s	10m22.59s	3m22.69s	8m2.04s	9m8.46s	100%	0.68/s
400	4	1m13.45s	998.68ms	2m52.36s	1m6.08s	2m21.43s	2m35.33s	100%	4.31/s
400	512	1m20.6s	649.55ms	3m22.2s	1m12.89s	2m36.38s	2m53.99s	100%	3.76/s
400	1024	1m27.84s	1.6s	3m48.39s	1m21.08s	2m46.21s	3m7.93s	100%	3.34/s
400	5120	2m9.35s	591.15ms	5m54.09s	1m51.64s	4m18.01s	5m2.73s	100%	2.17/s
400	10240	2m37.68s	672.52ms	7m10.24s	2m10.1s	5m20.13s	6m3.23s	100%	1.80/s
400	20480	3m47.37s	1.56s	10m11.56s	3m14.86s	8m14.37s	8m57.83s	100%	0.90/s
400	40960	4m7.84s	1.3s	10m29.66s	3m31.46s	8m27.02s	9m15.26s	100%	0.55/s
400	102400	4m0.38s	2.45s	10m25s	3m33.75s	8m30.22s	9m22.19s	100%	0.35/s
500	4	1m34.57s	1.86s	3m43.29s	1m23s	3m2.69s	3m23.26s	100%	4.23/s
500	512	1m41.8s	789.48ms	4m24.4s	1m32.92s	3m15.59s	3m39.61s	100%	3.62/s
500	1024	1m58.43s	849.62ms	5m15.91s	1m46.03s	3m44.21s	4m28.05s	100%	3.04/s
500	5120	3m31.35s	1.75s	9m28.47s	2m58.93s	7m11.37s	8m13.06s	100%	1.70/s
500	10240	3m40.59s	707.84ms	10m27.8s	3m1.85s	7m36.95s	9m12.85s	100%	1.23/s
500	20480	3m54.34s	853.71ms	10m25.1s	3m22.99s	8m18.77s	9m9.16s	100%	0.86/s
500	40960	3m42.55s	1.48s	10m29.03s	3m2.46s	8m11.57s	9m29.09s	100%	0.70/s
500	102400	4m45.36s	4.15s	10m29.76s	4m31.71s	9m0.88s	9m42.84s	100%	0.33/s
1000	4	3m6.07s	1.89s	7m19.05s	2m48.86s	5m59.39s	6m34.52s	100%	4.33/s
1000	512	3m45.75s	1.04s	9m40.91s	3m29.11s	7m3.64s	8m4.6s	100%	3.31/s
1000	1024	3m57.91s	1.21s	10m18.6s	3m36.21s	7m38.78s	8m58.79s	100%	3.09/s
1000	5120	3m55.67s	550.33ms	10m28.94s	3m30.54s	7m51.64s	9m7.79s	100%	2.13/s
1000	10240	3m58.34s	625.94ms	10m27.1s	3m35.09s	8m5.12s	9m16.33s	100%	1.47/s
1000	20480	4m8.95s	1.26s	10m29.59s	3m44.75s	8m26.72s	9m29.06s	100%	1.00/s
1000	40960	4m36.18s	2.01s	10m28.81s	4m12.9s	9m7.83s	9m50.91s	100%	0.68/s
1000	102400	5m25.2s	4.75s	10m22.27s	5m19.89s	9m39.35s	10m0.14s	100%	0.22/s

from harbor.

Failed to push artifacts to the project about harbor HOT 23 OPEN

Comments (23)

push-artifacts-to-different-projects

push-artifacts-to-same-projects

pull-artifacts-from-different-projects

pull-artifacts-from-same-project

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent