goharbor / perf Goto Github PK

View Code? Open in Web Editor NEW

19.0 8.0 15.0 114 KB

performance test scripts and data

License: Apache License 2.0

Go 19.85% JavaScript 80.15%

perf's Introduction

perf

This repository includes scripts for the performance test of harbor.

If you see performance problems please open an issue in repo: goharbor/harbor

How to run the performance tests

To run performance tests for the target harbor instance, first ensure you have the prerequisites:

Go toolchain
Git

Then:

Clone this repostiroy and generate client from swagger

git clone https://github.com/goharbor/perf
cd perf

Prepare user data

export HARBOR_URL=https://admin:[email protected]
export HARBOR_SIZE=small
go run mage.go prepare

Run all tests

export HARBOR_URL=https://admin:[email protected]
export HARBOR_VUS=100
export HARBOR_ITERATIONS=200
go run mage.go

The environment variables and targets

The environment variables in the table are the configurations for the performance testing. Use VAR1=value1 VAR2=value2 go run mage.go target format to apply the variables to the testing

Variable	Description	Default value
HARBOR_URL	The url of the harbor will be tested, its format is https?://username:[email protected]
HARBOR_SIZE	The user data size to prepare for the tests	small
HARBOR_VUS	The number of virtual users for the performance test	500
HARBOR_ITERATIONS	The script total iteration limit (among all VUs) for the performance test	1000
K6_ALWAYS_UPDATE	Always install the latest xk6-harbor	false
K6_QUIET	Disable progress updates of the k6	false
K6_CSV_OUTPUT	Make k6 output detailed statistics in a CSV format	false
K6_JSON_OUTPUT	Make k6 output detailed statistics in JSON format	false
HARBOR_REPORT	Whether generate testing report	false

Experimental to send the metrics to the prometheus. (Refer to https://k6.io/docs/results-output/real-time/prometheus-remote-write for more details)

Variable	Description	Default value
K6_PROMETHEUS_RW_SERVER_URL	URL of the Prometheus remote write implementation's endpoint
K6_PROMETHEUS_RW_USERNAME	User for the HTTP Basic authentication at the Prometheus remote write endpoint
K6_PROMETHEUS_RW_PASSWORD	Password for the HTTP Basic authentication at the Prometheus remote write endpoint
K6_PROMETHEUS_RW_TREND_AS_NATIVE_HISTOGRAM	If true, it maps the all defined trend metrics as Native Histograms	false
K6_PROMETHEUS_RW_TREND_STATS	It's a comma-separated list of stats functions	min,p(90),p(95),p(99),max
K6_PROMETHEUS_RW_INSECURE_SKIP_TLS_VERIFY	If true, the HTTP client skips TLS verification on the endpoint	false

The following table includes the targets.

Target	Description	Example
prepare	Generate user data	HARBOR_SIZE=small HARBOR_URL=https://admin:[email protected] go run mage.go prepare
run	Execute a specific test	HARBOR_URL=https://admin:[email protected] go run mage.go run list-projects
all	Execute all tasks	HARBOR_URL=https://admin:[email protected] go run mage.go all
list	Print all test	go run mage.go list
compare	Compare performance	go run mage.go compare 251 252

Performance comparison

Compare the performance of harbor different versions, the format of HTML comparison bar can be generated easily by subcommand. Before you need to retain the outputs folder every time and then rename them to a meaningful name.

For example, if we want to compare the performance of harbor 2.5.1 and 2.5.2, we can just follow the steps:

Deploy harbor 2.5.1, prepare data and run tests

  export HARBOR_URL=https://admin:[email protected]
  export HARBOR_SIZE=small
  # prepare data
  go run mage.go prepare
  # run tests
  go run mage.go
  # retain the outputs result
  mv outputs 251 && mkdir outputs

Deploy harbor 2.5.2, prepare data and run tests

  export HARBOR_URL=https://admin:[email protected]
  export HARBOR_SIZE=small
  # prepare data
  go run mage.go prepare
  # run tests
  go run mage.go
  # retain the outputs result
  mv outputs 252 && mkdir outputs

Compare

 # use outputs result folder name as parameters
 go run mage.go compare 251 252
 # then you can see the comparison in the browser
 open ./outputs/api-comparison.html
 open ./outputs/pull-push-comparison.html

perf's People

Contributors

Stargazers

Watchers

Forkers

heww chlins hiteshmah-jan ekmixon wy65701436 jeremy-boo jholee wangyao2221 danielzhanghl enraiha0307 yangjiao0817 plodocus nmcostello kofj

perf's Issues

Preparation failed due to thresholds on metrics 'success' have been crossed, failed with exit code 99

Hi all @heww @chlins @reasonerjt @sagikazarmark ,

I am trying to prepare the user data for HARBOR_SIZE small and medium, I am getting below error for both the harbor size small and medium.

Please provide a solution for this.

[For HARBOR_SIZE CI it is working fine]

running (0d00h00m00.1s), 000/100 VUs, 100 complete and 0 interrupted iterations
default ✓ [======================================] 100 VUs  0d00h00m00.1s/24h0m0s  100/100 shared iters

          /\      |‾‾| /‾‾/   /‾‾/
     /\  /  \     |  |/  /   /  /
    /  \/    \    |     (   /   ‾‾\
   /          \   |  |\  \ |  (‾)  |
  / __________ \  |__| \__\ \_____/ .io

  execution: local
     script: scripts/data/03-project-member.js
     output: -

  scenarios: (100.00%) 1 scenario, 300 max VUs, 24h0m30s max duration (incl. graceful stop):
           * default: 1000 iterations shared among 300 VUs (maxDuration: 24h0m0s, gracefulStop: 30s)

ERRO[0005] failed to create project member for project project-075, error: [POST /projects/{project_name_or_id}/members][500] createProjectMemberInternalServerError  &{Errors:[0xc003cc43a0]}  source=console
ERRO[0005] failed to create project member for project project-007, error: [POST /projects/{project_name_or_id}/members][500] createProjectMemberInternalServerError  &{Errors:[0xc003cc4980]}  source=console
ERRO[0005] failed to create project member for project project-021, error: [POST /projects/{project_name_or_id}/members][500] createProjectMemberInternalServerError  &{Errors:[0xc003fc8b40]}  source=console
ERRO[0005] failed to create project member for project project-034, error: [POST /projects/{project_name_or_id}/members][500] createProjectMemberInternalServerError  &{Errors:[0xc003941020]}  source=console
ERRO[0005] failed to create project member for project project-098, error: [POST /projects/{project_name_or_id}/members][500] createProjectMemberInternalServerError  &{Errors:[0xc00496a5a0]}  source=console
ERRO[0005] failed to create project member for project project-075, error: [POST /projects/{project_name_or_id}/members][500] createProjectMemberInternalServerError  &{Errors:[0xc00496a7e0]}  source=console
ERRO[0005] failed to create project member for project project-054, error: [POST /projects/{project_name_or_id}/members][500] createProjectMemberInternalServerError  &{Errors:[0xc004668420]}  source=console
ERRO[0005] failed to create project member for project project-071, error: [POST /projects/{project_name_or_id}/members][500] createProjectMemberInternalServerError  &{Errors:[0xc0041869e0]}  source=console
ERRO[0005] failed to create project member for project project-056, error: [POST /projects/{project_name_or_id}/members][500] createProjectMemberInternalServerError  &{Errors:[0xc003eca160]}  source=console
ERRO[0005] failed to create project member for project project-054, error: [POST /projects/{project_name_or_id}/members][500] createProjectMemberInternalServerError  &{Errors:[0xc003eca360]}  source=console
ERRO[0005] failed to create project member for project project-055, error: [POST /projects/{project_name_or_id}/members][500] createProjectMemberInternalServerError  &{Errors:[0xc004584780]}  source=console
ERRO[0005] failed to create project member for project project-053, error: [POST /projects/{project_name_or_id}/members][500] createProjectMemberInternalServerError  &{Errors:[0xc004b12a20]}  source=console
ERRO[0006] failed to create project member for project project-027, error: [POST /projects/{project_name_or_id}/members][500] createProjectMemberInternalServerError  &{Errors:[0xc004187ac0]}  source=console
ERRO[0006] failed to create project member for project project-068, error: [POST /projects/{project_name_or_id}/members][500] createProjectMemberInternalServerError  &{Errors:[0xc004e58820]}  source=console
ERRO[0006] failed to create project member for project project-078, error: [POST /projects/{project_name_or_id}/members][500] createProjectMemberInternalServerError  &{Errors:[0xc0055cdda0]}  source=console
ERRO[0006] failed to create project member for project project-034, error: [POST /projects/{project_name_or_id}/members][500] createProjectMemberInternalServerError  &{Errors:[0xc004e58bc0]}  source=console
ERRO[0006] failed to create project member for project project-025, error: [POST /projects/{project_name_or_id}/members][500] createProjectMemberInternalServerError  &{Errors:[0xc004b13a80]}  source=console
ERRO[0006] failed to create project member for project project-010, error: [POST /projects/{project_name_or_id}/members][500] createProjectMemberInternalServerError  &{Errors:[0xc0058069e0]}  source=console
ERRO[0006] failed to create project member for project project-057, error: [POST /projects/{project_name_or_id}/members][500] createProjectMemberInternalServerError  &{Errors:[0xc005806c20]}  source=console
ERRO[0006] failed to create project member for project project-020, error: [POST /projects/{project_name_or_id}/members][500] createProjectMemberInternalServerError  &{Errors:[0xc0041492c0]}  source=console
ERRO[0006] failed to create project member for project project-054, error: [POST /projects/{project_name_or_id}/members][500] createProjectMemberInternalServerError  &{Errors:[0xc0044e0e00]}  source=console
ERRO[0006] failed to create project member for project project-082, error: [POST /projects/{project_name_or_id}/members][500] createProjectMemberInternalServerError  &{Errors:[0xc00518c400]}  source=console
ERRO[0006] failed to create project member for project project-080, error: [POST /projects/{project_name_or_id}/members][500] createProjectMemberInternalServerError  &{Errors:[0xc0050100a0]}  source=console
ERRO[0006] failed to create project member for project project-081, error: [POST /projects/{project_name_or_id}/members][500] createProjectMemberInternalServerError  &{Errors:[0xc00518c5e0]}  source=console
ERRO[0006] failed to create project member for project project-019, error: [POST /projects/{project_name_or_id}/members][500] createProjectMemberInternalServerError  &{Errors:[0xc005a150c0]}  source=console
ERRO[0006] failed to create project member for project project-047, error: [POST /projects/{project_name_or_id}/members][500] createProjectMemberInternalServerError  &{Errors:[0xc00560e440]}  source=console
ERRO[0006] failed to create project member for project project-049, error: [POST /projects/{project_name_or_id}/members][500] createProjectMemberInternalServerError  &{Errors:[0xc004e59a60]}  source=console
ERRO[0006] failed to create project member for project project-050, error: [POST /projects/{project_name_or_id}/members][500] createProjectMemberInternalServerError  &{Errors:[0xc004ae50e0]}  source=console
ERRO[0006] failed to create project member for project project-048, error: [POST /projects/{project_name_or_id}/members][500] createProjectMemberInternalServerError  &{Errors:[0xc004e59d00]}  source=console

     █ setup

     data_received............: 0 B    0 B/s
     data_sent................: 0 B    0 B/s
     iteration_duration.......: avg=1.57s   min=18.63ms med=798.24ms max=6.2s    p(90)=4.97s   p(95)=5.88s
     ✓ { group:::setup }......: avg=18.63ms min=18.63ms med=18.63ms  max=18.63ms p(90)=18.63ms p(95)=18.63ms
     ✓ { scenario:default }...: avg=1.57s   min=98.65ms med=798.32ms max=6.2s    p(90)=4.97s   p(95)=5.88s
     iterations...............: 1000   160.209585/s
   ✗ success..................: 97.10% ✓ 971        ✗ 29
     vus......................: 50     min=50       max=300
     vus_max..................: 300    min=300      max=300


running (0d00h00m06.2s), 000/300 VUs, 1000 complete and 0 interrupted iterations
default ✓ [======================================] 300 VUs  0d00h00m06.2s/24h0m0s  1000/1000 shared iters
ERRO[0008] thresholds on metrics 'success' have been crossed
Error: running "k6-harbor run scripts/data/03-project-member.js --no-usage-report" failed with exit code 99
exit status 99

@sagikazarmark @reasonerjt @heww @chlins

Failed to run all tests with exit code 107

Run       [======================================] setup()
default   [--------------------------------------]
ERRO[0147] GoError: failed to push reg.srv.deeproute.cn/project-060/repository-1706015003594:tag-1706015003594, error: failed commit on ref "manifest-sha256:9b15eb416511109d4d79ae529633de73b7264c0c6f6361f517cba690cc66d058": unexpected status: 502 Bad Gateway
        at reflect.methodValueCall (native)
        at setup (file:///root/go/src/github.com/goharbor/perf/scripts/test/pull-artifacts-from-different-projects.js:38:20(51))  hint="script exception"
Error: running "k6-harbor run scripts/test/pull-artifacts-from-different-projects.js --no-usage-report" failed with exit code 107
exit status 107

~/go/src/github.com/goharbor/perf (main ✗) go run mage.go run push-artifacts-to-different-projects

          /\      |‾‾| /‾‾/   /‾‾/
     /\  /  \     |  |/  /   /  /
    /  \/    \    |     (   /   ‾‾\
   /          \   |  |\  \ |  (‾)  |
  / __________ \  |__| \__\ \_____/ .io

  execution: local
     script: ./scripts/test/push-artifacts-to-different-projects.js
     output: -

  scenarios: (100.00%) 1 scenario, 500 max VUs, 24h0m30s max duration (incl. graceful stop):
           * default: 1000 iterations shared among 500 VUs (maxDuration: 24h0m0s, gracefulStop: 30s)

ERRO[0026] failed to push reg.srv.deeproute.cn/project-100/repository-1706023860548:tag-0504, error: failed commit on ref "config-sha256:64e128
9fef1160c02bd921bf3e885aca0d73812cb7156c0a34e0727178066fa5": unexpected status: 500 Internal Server Error  source=console
ERRO[0027] failed to push reg.srv.deeproute.cn/project-018/repository-1706023860548:tag-0507, error: failed commit on ref "unknown-sha256:4d8ca
5fcc0b85c46f2b8d7d735685a234ea63badc29f568248576e89439ea432": unexpected status: 500 Internal Server Error  source=console
ERRO[0028] failed to push reg.srv.deeproute.cn/project-030/repository-1706023860549:tag-0515, error: failed commit on ref "config-sha256:fce9a6
8a4ce2d7556be09ec968970ab10a8c8db41929675470a9f809c587bac6": unexpected status: 500 Internal Server Error  source=console

Why does it take less time to push artifacts to different projects than to the same project?

https://github.com/goharbor/perf/wiki/Harbor-2.5.0

prepare artifact error

Harbor deployed with http mode.

version: v2.0.1

$ HARBOR_SIZE=small HARBOR_URL=http://admin:[email protected]:31114 go run mage.go prepare

INFO[0008] GoError: failed to push 192.168.25.56:31114/project-1619059710385-013/repository-001:v01, error: failed to do request: Head "https://192.168.25.56:31114/v2/cpaas-system/repository-001/blobs/sha256:4a969632bc3938f5b516bb39eb3a9d098b698179bfde42a512b499786cb4246f": http: server gave HTTP response to HTTPS client  source=console
INFO[0008] GoError: failed to push 192.168.25.56:31114/project-1619059710385-013/repository-001:v02, error: failed to do request: Head "https://192.168.25.56:31114/v2/cpaas-system/repository-001/blobs/sha256:b4edb9e6688f4f5efe5a186240b73287c9435bd529b6ab39b509f02e7482ba05": http: server gave HTTP response to HTTPS client  source=console
INFO[0008] GoError: failed to push 192.168.25.56:31114/project-1619059710385-013/repository-001:v03, error: failed to do request: Head "https://192.168.25.56:31114/v2/cpaas-system/repository-001/blobs/sha256:9cc3a79d2828d644a90300491d6b7df84fd33bf951e8636505522f82a32ac453": http: server gave HTTP response to HTTPS client  source=console

createUserForbidden on OIDC enabled instance

Seems this tools doesn't work for OIDC enabled Harbor instances, as the local user creation would be disabled, when on OIDC.

When we run perf prepare on OIDC enabled instance we get below error [Test projects gets created fine]:
INFO[0001] GoError: failed to create user user-001, error: [POST /users][403] createUserForbidden source=console

We use admin User, so its not the User permission issue, rather not allowed of local User creation apart from the default admin user, when OIDC is enabled.

Any suggestions or workaround please?

Remaining repos not cleaned up after testing.

There are many remaining repositories without artifacts after testing, we should clean up these repositories in the case teardown phase.

Infrastructure code for creating test environments

First of all, thanks for putting this test suite together. It's really helpful and I'm glad I didn't have to write this myself.

I've spent the last couple weeks with running performance tests in various environment and I have to say it was a lot of struggle. Setting up a Harbor instance that actually works was not trivial, the tests kept randomly failing and ruining test environments (eg. prepare steps failing and not being able to rerun, tests randomly failing, etc).

I'm not trying to blame anyone or complain about the test suite, just sharing my overall experience about the path to actually using it.

To help the situation, I started to put together a couple terraform modules to make spinning up test environments easier: https://github.com/sagikazarmark/harbor-perf-test

It really made things easier for me and I hope it can be useful to others. It's far from perfect and the results from the test runs on these environments are not conclusive (ie. they still randomly fail or not work at all).

I plan to make some improvements. For example, replacing the builtin postgres and redis with Aurora and ElastiCache seems like a good idea, as postgres gave up quite a few times during the test runs.

As I said, it's far from perfect, but it ended up sparing me hours in the end.

I'd be happy to contribute it to this repo once it's in a better shape if you are interested.

go run mage.go prepare 失败

[root@harbor250 perf]# HARBOR_SIZE=small HARBOR_URL=https://192.168.2.250:443 go run mage.go prepare
mage.go:8:2: cannot find package "github.com/magefile/mage/mage" in any of:
/usr/lib/golang/src/github.com/magefile/mage/mage (from $GOROOT)
/root/go/src/github.com/magefile/mage/mage (from $GOPATH)

Add License

Can you add a License?

Unkown module error when k6 is installed globally

When k6 is installed to a global place (using brew, apt, etc) running the suite fails with the following error:

unknown module: k6/x/harbor

Tried setting K6_ALWAYS_UPDATE=true but it didn't help.

AWS EC2: panic: runtime error: invalid memory address or nil pointer dereference

Hi,
I have some problems to run perf testing from AWS EC2 instances.
After installing go, checking out a repo, setting some env variables and running go run mage.go I get following ERROR (the preparation go run mage.go prepare were allready done from local mashine):

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x10c7946]

goroutine 1 [running]:
debug/elf.(*Section).ReadAt(0xc000ba0000?, {0xc0007626c0?, 0x17?, 0x16?}, 0x16?)
	<autogenerated>:1 +0x26
archive/zip.readDirectoryEnd({0x17be540, 0xc0003f5600}, 0x210)
	/usr/local/go/src/archive/zip/reader.go:581 +0xf5
archive/zip.(*Reader).init(0xc00016cc40, {0x17be540?, 0xc0003f5600}, 0x210)
	/usr/local/go/src/archive/zip/reader.go:124 +0x5c
archive/zip.NewReader({0x17be540, 0xc0003f5600}, 0x210)
	/usr/local/go/src/archive/zip/reader.go:103 +0x5e
github.com/daaku/go%2ezipexe.zipExeReaderElf({0x17c9620?, 0xc00039a298}, 0x2cc4441)
	/home/ubuntu/go/pkg/mod/github.com/daaku/[email protected]/zipexe.go:128 +0x8b
github.com/daaku/go%2ezipexe.NewReader({0x17c9620, 0xc00039a298}, 0x0?)
	/home/ubuntu/go/pkg/mod/github.com/daaku/[email protected]/zipexe.go:48 +0x98
github.com/daaku/go%2ezipexe.OpenCloser({0xc0003a4a40?, 0xc000a67980?})
	/home/ubuntu/go/pkg/mod/github.com/daaku/[email protected]/zipexe.go:30 +0x57
github.com/GeertJohan/go%2erice.init.0()
	/home/ubuntu/go/pkg/mod/github.com/!geert!johan/[email protected]/appended.go:38 +0x5e
Error: running "k6-harbor run scripts/test/get-artifact-by-digest.js --no-usage-report --vus 100 --iterations 200" failed with exit code 2
exit status 2```


have tried it out on different instance types (m5.large, c4.4xlarge) and with differetn OS (Amazon Linux 2023 AMI 2023.0.20230503.0 x86_64 HVM kernel-6.1 and Canonical, Ubuntu, 22.04 LTS, amd64 jammy image build on 2023-03-25)

what am I missing?

How to initialize two Harbor instances?

Hi all!
I'm trying to write replication test between two Harbor instances. I tried to create them like this:

const harbor1 = new Harbor(settings.Harbor1)
const harbor2 = new Harbor(settings.Harbor2)

but when I run my test I got an error:

goerror: harbor module initialized

How I can solve this issue?

http: proxy error: context canceled

I am getting "http: proxy error: context canceled" errors in core.log when i try the PET test using perf tool :
Load details :

  VUS: getEnvInt('HARBOR_VUS', '20'),
  ProjectsCount: 1,
  RepositoriesCountPerProject: 60,
  ArtifactsCountPerRepository: 1,
  ArtifactTagsCountPerArtifact: 1,
  UsersCount: 1,
  ProjectMembersCountPerProject: 1,
  AuditLogsCount: 5000,
  BlobSize: '10 MiB',
  BlobsCountPerArtifact: 20

 data_received............: 0 B     0 B/s
 data_sent................: 0 B     0 B/s
 iteration_duration.......: avg=4m5s min=845.34ms med=3m58s max=8m34s p(90)=6m35s p(95)=7m43s
 ✓ { group:::setup }......: avg=0s   min=0s       med=0s    max=0s    p(90)=0s    p(95)=0s   
 ✓ { scenario:default }...: avg=4m9s min=56.74s   med=4m3s  max=8m34s p(90)=6m39s p(95)=7m44s
 iterations...............: 60      0.07627/s

✓ success..................: 100.00% ✓ 60 ✗ 0
vus......................: 0 min=0 max=20
vus_max..................: 20 min=20 max=20

running (0d00h13m06.7s), 00/20 VUs, 60 complete and 0 interrupted iterations
default ✓ [======================================] 20 VUs 0d00h13m05.8s/24h0m0s 60/60 shared iters

why this error messages are coming ? how can i reduce this errors

core.log
2024-02-26T06:31:51.499957197+02:00 stderr F 2024/02/26 04:31:51 http: proxy error: context canceled
2024-02-26T06:31:52.546913633+02:00 stderr F 2024/02/26 04:31:52 http: proxy error: context canceled
2024-02-26T06:31:56.366103291+02:00 stderr F 2024/02/26 04:31:56 http: proxy error: context canceled
2024-02-26T06:31:59.913463363+02:00 stderr F 2024/02/26 04:31:59 http: proxy error: context canceled
2024-02-26T06:32:04.449170056+02:00 stderr F 2024/02/26 04:32:04 http: proxy error: context canceled
2024-02-26T06:32:27.165450329+02:00 stderr F 2024/02/26 04:32:27 http: proxy error: context canceled
2024-02-26T06:32:43.077807763+02:00 stderr F 2024/02/26 04:32:43 http: proxy error: context canceled

Performance test preparation data script to increase fault tolerance

Should harbor's script for performance test preparation data consider adding the function of retry when preparing for errors in the middle. Preparing data is a time-consuming process, because an error leads to exit and it needs to re-prepare data, which is a waste of time

只能针对harbor2.x版本测试吗

测试harbor1.9.2版本
go run mage.go prepare的时候报错
INFO[0003] GoError: failed to create project project-068, error: response status code does not match any response statuses defined for this endpoint in the swagger spec (status 404): {} source=console
go run mage.go run pull-artifacts-from-same-project的时候报错
ERRO[0004] GoError: failed to push xxxxx/project-058/repository-1641782069696:tag-1641782069696, error: server message: insufficient_scope: authorization failed

测试harbor2.2.1是正常的

Cache Layer: recreate analysis with benchmark tool

Hello,
The wiki suggest use the benchmark Tool: https://github.com/rakyll/hey
but the output seams become from perf, in this case running:

go run mage.go compare AAA BBB

Please @chlins could you share the full guide and files to get our own results of Response Time, TPS and Success Rate.

Thanks in advance,