tuna / opentuna Goto Github PK

Sometimes the tunasync manager process is running(no any output in console), but the http server is not started yet. It's probably something wrong happened in tunasync.

Due to the tunasync manager is managed by ASG that has health checking for http service of manager, the unhealthy instance will be terminated then start new one. The manager works after restarting several times.

For our infra, we might do following things to help investigate it,

• enable log of tunasync manager
• collect logs via cloudwatch agent

Current conf of tunasync manager,

Proposed solution

Let's always create a new boltdb file for new instance of tunasync manager. We might follow below rules to create the boltdb file,

• create boltdb file with current timestamp and instance id
• use file regex pattern to search the existing boltdb files to copy the latest one as new one

Periodically running tasks test the availability of mirror repos.

Use Case

Testing if the mirror repo is availability or not.

Proposed Solution

• leverage serverless technology as possible, such as AWS Batch, AWS CodeBuild
• simplify the maintenance work as possible
• for software packages supporting matrix OS, such as debian, centos, ubuntu
• periodically trigger the testing
• randomly install some packages to mitigate the CDN caches
• send testing result as metrics to CloudWatch
• alarm if the metrics exceeds the threshold

Other

This is a 🚀 Feature Request

It's a follow up for #30.

Use Case

Proposed Solution

Other

• 👋 I may be able to implement this feature request

This is a 🚀 Feature Request

Use Case

• use cloudfront as CDN to distribute entire site, use the ALB as origin
• different cache policies for different content
• custom cloudfront invalidation for web portal
• support custom domain and SSL certificate(NOTE: ACM certificate not supported used by CloudFront in China regions, use IAM cert as workaround) via context parameters
• update domain records in r53
• unit tests
• update readme

Proposed Solution

Other

• 👋 I may be able to implement this feature request

This is a 🚀 Feature Request

When analyzing CloudFront logs, we found many HTTP 301s of non existent urls, but no HTTP 404 responses.

Error Log

Environment

• CDK CLI Version:
• Framework Version:
• Node.js Version:
• OS :

Other

This is 🐛 Bug Report

Is it possible to synchonize & provide service directly through S3? It would be useful for repos such as hugging-face-models and dart-pub that are hosted on S3 originally.

We put tunasync worker in public subnet to save the cost of NAT gateway. The IP address will be changed after the worker instance restarts.

The rsync upstream(see #40) might require whiltelist the IPs of rsync client.

Use Case

Proposed Solution

Other

• 👋 I may be able to implement this feature request

This is a 🚀 Feature Request

Use Case

Proposed Solution

• build mirror web as container image, I prefer it's done by upstream project
• consolidate mirror web fargate container into existing ecs cluster created in content server
• route different paths to different ecs service in shared ALB
• route /static/tunasync.json to content server that acts as proxy to tuna manager service. ALB does not support url rewrite currently, we have to archive it as workaround.
• unit tests
• readme doc

Other

• 👋 I may be able to implement this feature request

This is a 🚀 Feature Request

failed on installing dependencies

Reproduction Steps

run renew cert job,

An unexpected error occurred:
--
980 | pkg_resources.ContextualVersionConflict: (cryptography 3.1.1 (/usr/local/lib/python3.7/dist-packages), Requirement.parse('cryptography>=3.2'), {'PyOpenSSL'})
981 | Please see the logfile '/tmp/tmpwtu2kod2/log' for more details.
982 | An unexpected error occurred:
983 | pkg_resources.ContextualVersionConflict: (cryptography 3.1.1 (/usr/local/lib/python3.7/dist-packages), Requirement.parse('cryptography>=3.2'), {'PyOpenSSL'})
984 | Please see the logfile '/tmp/tmp8vmdzb23/log' for more details.
985 | An unexpected error occurred:
986 | pkg_resources.ContextualVersionConflict: (cryptography 3.1.1 (/usr/local/lib/python3.7/dist-packages), Requirement.parse('cryptography>=3.2'), {'PyOpenSSL', 'pyOpenSSL'})
987 | Please see the logfile '/tmp/tmp10x2xjnx/log' for more details.
988 | An unexpected error occurred:
989 | pkg_resources.ContextualVersionConflict: (cryptography 3.1.1 (/usr/local/lib/python3.7/dist-packages), Requirement.parse('cryptography>=3.2'), {'PyOpenSSL'})
990 | Please see the logfile '/tmp/tmpjvcxn5u6/log' for more details.
991 | An unexpected error occurred:
992 | pkg_resources.ContextualVersionConflict: (cryptography 3.1.1 (/usr/local/lib/python3.7/dist-packages), Requirement.parse('cryptography>=3.2'), {'PyOpenSSL'})
993 | Please see the logfile '/tmp/tmpho5dbotb/log' for more details.

Error Log

Environment

• CDK CLI Version:
• Framework Version:
• Node.js Version:
• OS :

Other

This is 🐛 Bug Report

Based on user request, add the following repos:

• archlinux
• archlinuxcn
• archlinuxarm

Use Case

Proposed Solution

Other

• 👋 I may be able to implement this feature request

This is a 🚀 Feature Request

Specify custom domain to application load balancer in case we share it to users from same region.

Use Case

Proposed Solution

If the service is specified as domain opentuna.cn, automatically assigning domain name <region>.opentuna.cn to ALB.

• request SSL certificate from ACM
• configure custom domain and SSL cert of ALB
• update DNS record in r53
• redirect http requests to https
• unit tests
• update readme

Other

• 👋 I may be able to implement this feature request

This is a 🚀 Feature Request

Use Case

MariaDB yum source for Amazon Linux2

Proposed Solution

Sync from http://yum.mariadb.org/$version/centos7-aarch64 ?

guide can be found here https://mariadb.com/kb/en/mirror-sites-for-mariadb/

Other

• 👋 I may be able to implement this feature request

This is a 🚀 Feature Request

The docker hub introduces the pull limit for anonymous and non-subscription users. Pulling images from docker hub might see below error,

BUILD_CONTAINER_UNABLE_TO_PULL_IMAGE: Unable to pull customer's container image. CannotPullContainerError: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

Use Case

We have below scenarios using public images from docker.io,

• pipeline
• monitor tasks

Proposed Solution

Other

This is a 🚀 Feature Request

Add a CloudWatch alarm to monitor auto scaling events and subscribe SNS topic.

Use Case

Proposed Solution

Other

• 👋 I may be able to implement this feature request

This is a 🚀 Feature Request

Use Case

Proposed Solution

Other

• 👋 I may be able to implement this feature request

This is a 🚀 Feature Request

rubygems is stored on S3, which can not provide files browsing like POSIX file system.

Use Case

Proposed Solution

Consider below options,

• disable the browsing for rubygems
• redirect to help page of rubygems

Other

This is a 🚀 Feature Request

• rubygems: a million files under one directory https://github.com/tuna/rubygems-mirror-s3
• pypi

Use Case

Proposed Solution

Other

• 👋 I may be able to implement this feature request

This is a 🚀 Feature Request

After we submitted new news of site, but we can not deploy the stack to pick up the latest content of opentuna-site.

Reproduction Steps

see above

Because CDK calculates the image hash based on the content of the folder of Dockerfile. There is nothing change in CDK's perspective.

We should explicitly refer to the commit in opentuna-site.

Error Log

Environment

• CDK CLI Version:
• Framework Version:
• Node.js Version:
• OS :

Other

This is 🐛 Bug Report

Use Case

Proposed Solution

Step Functions + AWS CodeBuild

• automatically trigger for changes of mainline branch
• pipeline itself update
• cross accounts for stages
• manual approval for prod stage

Other

This is a 🚀 Feature Request

Below links are incorrect.

Reproduction Steps

Error Log

Environment

• CDK CLI Version:
• Framework Version:
• Node.js Version:
• OS :

Other

This is 🐛 Bug Report

Use Case

Proposed Solution

• create ECS cluster
• package custom Nginx(or other http servers) image and push it to ECR
• create fargate type task definition to mount EFS file system
• create a readonly access point of EFS file system for content server. a plus feature might move to a separated feature
• collect container logs to CloudWatch
• only grant least privileges to task content
• unit tests

Some notes,

• it's a shared cluster also deployed the web container described in #7
• the ecs services will share an external application load balancer created in tuna stack described in #8

Other

• 👋 I may be able to implement this feature request

This is a 🚀 Feature Request

It's a follow up feature of #10 . Check the source ip of requests to find the Geo location based on public ip database.

Use Case

Get the Geo location of requests via source ip, for example

• AWS BJS
• AWS ZHY
• Beijing
• Shanghai
• CNET
• US

This is a 🚀 Feature Request

Customize web portal.

Use Case

• Words
• ICP# if provided
• Hide hardcode mirrors, such as AUR
• only keep Mirrors in nav bar
• News only for this site
• Hide Domains selection
• Hide Server status section

This is a 🚀 Feature Request

• use opentuna as upstream for centos base repos

Use Case

Proposed Solution

Other

• 👋 I may be able to implement this feature request

This is a 🚀 Feature Request

The metadata of debian/ubuntu might be in broken status while the upstream is syncing.

[Container] 2020/09/10 12:49:36 Running command apt update
--
23 |  
24 | WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
25 |  
26 | Get:1 http://opentuna.cn/debian testing InRelease [116 kB]
27 | Ign:2 http://opentuna.cn/debian testing/main amd64 Packages
28 | Get:2 http://opentuna.cn/debian testing/main amd64 Packages [7688 kB]
29 | Err:2 http://opentuna.cn/debian testing/main amd64 Packages
30 | File has unexpected size (7687532 != 7687940). Mirror sync in progress? [IP: 52.82.132.17 80]
31 | Hashes of expected file:
32 | - Filesize:7687940 [weak]
33 | - SHA256:e77275d6b178406cc1fe4a4108387f1bdbed4adce726c586f991f23583c4b3b3
34 | - MD5Sum:1d9663ee41239541f9a1b29bf027e069 [weak]
35 | Release file created at: Thu, 10 Sep 2020 02:12:15 +0000
36 | Fetched 116 kB in 0s (538 kB/s)
37 | Reading package lists...
38 | E: Failed to fetch http://opentuna.cn/debian/dists/testing/main/binary-amd64/Packages.xz  File has unexpected size (7687532 != 7687940). Mirror sync in progress? [IP: 52.82.132.17 80]
39 | Hashes of expected file:
40 | - Filesize:7687940 [weak]
41 | - SHA256:e77275d6b178406cc1fe4a4108387f1bdbed4adce726c586f991f23583c4b3b3
42 | - MD5Sum:1d9663ee41239541f9a1b29bf027e069 [weak]
43 | Release file created at: Thu, 10 Sep 2020 02:12:15 +0000
44 | E: Some index files failed to download. They have been ignored, or old ones used instead.

We should restart the sync job if above error happens.

Use Case

Proposed Solution

1. detect the error and set the env of codebuild
2. detect the failure and expected env to trigger lambda func to restart the job via EventBridge

• Switch debian to use two-stage-rsync or ftpsync
• Trigger Lambda when CodeBuild fails
• Call tunasync manager to sync repo immediately in lambda function

Other

• 👋 I may be able to implement this feature request

This is a 🚀 Feature Request

as title

Reproduction Steps

1. deploy stack with latest amz linux2

Error Log

    building 'multidict._multidict' extension
    creating build/temp.linux-x86_64-3.7
    creating build/temp.linux-x86_64-3.7/multidict
    gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python3.7m -c multidict/_multidict.c -o build/temp.linux-x86_64-3.7/multidict/_multidict.o -O2 -std=c99 -Wall -Wsign-compare -Wconversion -fno-strict-aliasing -pedantic
    unable to execute 'gcc': No such file or directory
    error: command 'gcc' failed with exit status 1

    ----------------------------------------
Command "/usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-6pow3cpx/multidict/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-zmha9ian-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-6pow3cpx/multidict/
WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.

Environment

• CDK CLI Version:
• Framework Version:
• Node.js Version:
• OS :

Other

This is 🐛 Bug Report

When syncing from rsync based upstream, the syncing fails on below errors,

03:59:05 cannot delete non-empty directory: pool/main/d/docker-swarm
03:59:10 cannot delete non-empty directory: pool/main/d/denyhosts
03:59:28 cannot delete non-empty directory: pool/main/f/filemanager-actions
03:59:35 cannot delete non-empty directory: pool/main/g/gversion-plugin
...

It's caused by the directories containing the non-empty tmp folder .~tmp~.

Is there something could be improved in upstream tool tunasync or we have to manually clean those tmp folders?

Reproduction Steps

Error Log

Environment

• CDK CLI Version:
• Framework Version:
• Node.js Version:
• OS :

Other

This is 🐛 Bug Report

Use separated domain name for pypi mirror, for example pypi.xxx.com.

Use Case

Due to we are sharing the doc among tuna sites, we need a separated domain name for pypi repo.

Proposed Solution

This is a 🚀 Feature Request

Using tool monitors the availability of site and send alarm if the service is unavailable.

Use Case

Below are some cases to cause the site unavailable,

• DNS misconfigured
• Domain expired
• SSL cert misconfigured/expired
• Backend service is down

Proposed Solution

• CloudWatch Synthetics

Other

• 👋 I may be able to implement this feature request

This is a 🚀 Feature Request

Use Case

Proposed Solution

Other

• 👋 I may be able to implement this feature request

This is a 🚀 Feature Request

The deployment for stage prod failed on the user data of worker EC2 exceeding 16KB limit.

Reproduction Steps

Error Log

Environment

• CDK CLI Version:
• Framework Version:
• Node.js Version:
• OS :

Other

This is 🐛 Bug Report

Reproduction Steps

1. Manager A started
2. Worker A started and registered with Manager A
3. Worker B and Manager B started, killing Worker A and Manager A
4. Worker B registered with the old Manager A, but db was not updated somehow
5. Manager B picked up the old db and don't know the ip of Worker B

Error Log

Environment

• CDK CLI Version:
• Framework Version:
• Node.js Version:
• OS :

Other

This is 🐛 Bug Report

Use cloud native analytic services to analysis the usage of site on demand.

Use Case

For example,

• mirror usage report
• analysis most used mirrors and packages
• user experience report
• error alarming

Proposed Solution

Use cloud native services to analysis usage data on demand with performance and cost optimization.

• store logs(cloudfront and alb access logs) to s3
• compress the csv log to column-storage parque
• ingest the manifest into glue catalog
• set up athena running queries(pending on cfn support of Athena landing in China regions)
• unit tests
• update readme

Generally we can follow the guidelines in this blog post https://aws.amazon.com/blogs/big-data/analyze-your-amazon-cloudfront-access-logs-at-scale/.

Other

• 👋 I may be able to implement this feature request

This is a 🚀 Feature Request

Checking the log of debian, found few errors like below,

rsync: link "/mnt/efs/opentuna/data/debian/dists/buster/contrib/by-hash/SHA256/.f61f27bd17de546264aa58f40f3aafaac7021e0ef69c17f6b1b4cd7664a037ec.22970" => dists/buster-proposed-updates/contrib/by-hash/SHA256/f61f27bd17de546264aa58f40f3aafaac7021e0ef69c17f6b1b4cd7664a037ec failed: Too many links (31)
rsync: link "/mnt/efs/opentuna/data/debian/dists/buster/contrib/by-hash/SHA256/.f61f27bd17de546264aa58f40f3aafaac7021e0ef69c17f6b1b4cd7664a037ec.22970" => dists/buster/contrib/debian-installer/binary-all/by-hash/MD5Sum/4a4dd3598707603b3f76a2378a4504aa failed: Too many links (31)

I can manually create the same symbol link. Not sure why rsync reports it.

Reproduction Steps

Error Log

Environment

• CDK CLI Version:
• Framework Version:
• Node.js Version:
• OS :

Other

This is 🐛 Bug Report

Currently the fargate platform version is using default version latest which actually points to 1.3.0. latest will be updated to point to 1.4.0 in future. It might introduce unexpected behavior(running web portal on fargate 1.4.0) when updating the stack.

Use Case

Proposed Solution

Other

• 👋 I may be able to implement this feature request

This is a 🚀 Feature Request

run below command in manager,

tunasynctl workers -p 80
[
  {
    "id": "tunasync-worker",
    "url": "",
    "token": "",
    "last_online": "2020-09-01T02:52:27.740256625Z"
  }
]

But both worker and manager are running well.

Reproduction Steps

Error Log

Environment

• CDK CLI Version:
• Framework Version:
• Node.js Version:
• OS :

Other

This is 🐛 Bug Report

is possible using cmd name restart in case the order is changed in upstream tunasync?

Originally posted by @zxkane in #103 (comment)

📊Tracking: monitoring services and mirror availability

It's an umbrella for monitoring feature requests.

• monitor web portal
• monitor the availability of repos, such as automation testing installing software packages and python packages in centos/fedora/ubuntu/debian
• monitor the healthy of tunasync manager
• monitor the healthy of tunasync worker
• monitor traffic bandwidth of content server, related to #19
• monitor mirroring tasks stuck/no progress for long time
• monitor error rate of distribution
• monitor the response time of first byte percentile

Always see rsync timeout errors from gitlab-ce and docker-ce repo. Already tune the retry attempts to 100, still failing to sync from upstream.

[generator] io timeout after 120 seconds -- exiting
rsync error: timeout in data send/receive (code 30) at io.c(195) [generator=3.1.2]
rsync error: received SIGUSR1 (code 19) at main.c(1429) [receiver=3.1.2]
rsync error: Timeout in data send/receive

Reproduction Steps

Error Log

Environment

• CDK CLI Version:
• Framework Version:
• Node.js Version:
• OS :

Other

This is 🐛 Bug Report

See https://github.com/aws/aws-cdk/releases/tag/v1.60.0

Use Case

Proposed Solution

• use native cdk support for mounting EFS in ECS Fargate
• remove current hacking

Other

This is a 🚀 Feature Request

It's an optional feature if providing a callback url of slack channel, we can forward the sns message to that channel.

Use Case

Use slack as IM channel to subscribe the alarm messages.

Proposed Solution

There is a reference implementation.

Other

This is a 🚀 Feature Request

Content servers act as HTTP server for serving the file content. Each task container has maximum capability to process the data transferring. We can use auto scaling capability to increase the total throughput of content servers and cost optimization.

Use Case

• add auto scaling for content server task
• unit tests

Based on the content server nature, the auto scaling should based on network traffic. Below are some criteria we should consider when designing the metrics for auto scaling,

• fargate instance network capability
• efs client network throughput, currently maximum throughout is 500MB/s per client(https://docs.aws.amazon.com/efs/latest/ug/limits.html#limits-client-specific)

Proposed Solution

Other

• 👋 I may be able to implement this feature request

This is a 🚀 Feature Request

Below snippets create two target group to point to same backend service. We can share the same target group for different routing conditions.

#81 enables syncing the rubygems to s3. Let's enable the doc of rubygems.

Use Case

Proposed Solution

Other

This is a 🚀 Feature Request

Due to the limitation of CloudFront and ACM in China regions, we have to import SSL cert into IAM for CloudFront.

Use free cert from Let's encrypt as workaround.

Use Case

Proposed Solution

• Use EventBridge to time based trigger
• Use Lambda/EC2 to request/renew cert from Let's encrypt
• Use DNS challenge via R53
• Upload cert to IAM
• Put new cert id to SSM parameter store
• Fetch latest cert id from SSM parameter store for open tuna stack

Other

• 👋 I may be able to implement this feature request

This is a 🚀 Feature Request

The folder apache in tuna

Use Case

Proposed Solution

Other

• 👋 I may be able to implement this feature request

This is a 🚀 Feature Request

Use Case

Proposed Solution

• generate iso json periodically
• support iso.json routing in ALB/CloudFront
• gracefully handle with the case if there is resource available

This is a 🚀 Feature Request

send notification if the pipeline state machine fails

Use Case

Proposed Solution

Other

This is a 🚀 Feature Request

Use Case

Proposed Solution

• Run CloudWatch agent in tunasync manager instance
• Redirect tunasync manager stdout to logs

Other

• 👋 I may be able to implement this feature request

This is a 🚀 Feature Request

Alpine syncing always failed due to below error,


00:11:12
rsync: readlink_stat("edge/community/armhf/.kstart-doc-4.2-r0.apk.UiVr7w" (in alpine)) failed: Permission denied (13)

00:11:12
rsync: readlink_stat("edge/community/armv7/.klettres-20.04.2-r0.apk.gouugR" (in alpine)) failed: Permission denied (13)

00:11:12
rsync: readlink_stat("edge/community/armv7/.libpulse-13.0-r10.apk.ulRZbd" (in alpine)) failed: Permission denied (13)

00:11:12
rsync: readlink_stat("edge/community/aarch64/.php7-phpdbg-7.4.8-r2.apk.lMJn88" (in alpine)) failed: Permission denied (13)

00:11:12
rsync: readlink_stat("edge/community/aarch64/.namecoin-tests-0.20.0-r1.apk.ZgAdc8" (in alpine)) failed: Permission denied (13)

Reproduction Steps

Error Log

Environment

• CDK CLI Version:
• Framework Version:
• Node.js Version:
• OS :

Other

This is 🐛 Bug Report