semaphoreci / toolbox Goto Github PK
View Code? Open in Web Editor NEWCommand line tools available in Semaphore 2.0.
Home Page: https://semaphoreci.com
Command line tools available in Semaphore 2.0.
Home Page: https://semaphoreci.com
When running checkout
on a PR-triggered workflow, I get this unhelpful error:
bash: cd: myproj: No such file or directory
Revision: 53efa4c0626c054a9c305dfe40dc929a95f1142f not found .... Exiting
After doing some digging, it seems that the underlying git clone
is failing, due to some intermittent network failures with GitHub. If I repeatedly run git clone
directly, I sometimes see:
Cloning into 'myproj'...
kex_exchange_identification: read: Connection reset by peer
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
In other words, the root cause is apparently GitHub in this case (it is dropping the connection). However, Semaphore's checkout
is hiding this error, because stderr for the git clone
command is sent to /dev/null
, like this:
Line 208 in 69e4ffa
Instead, checkout
assumes that the clone succeeded (it doesn't check the exit status), and tries to change into the cloned directory, which doesn't exist. So the error the user sees in this case is:
bash: cd: myproj: No such file or directory
checkout
should abort when a git clone
operation fails, and show a helpful error message; ideally the stderr output of the underlying failed git command.checkout
would provide a retry mechanism to gracefully recover from intermittent connection failures. Or the documentation should explain how retry
and checkout
can be used together (a naive retry checkout
doesn't work).In the case of PRs, checkout
just checks out the PR. It would be nice if there was an option to also merge the PR into the base branch, so that issues caused by changes merged into the basebranch after the PR branch was crated get surfaced. This entails that PRs with conflicts won't build at all.
Cache operations can take up a long time. For our builds, the do in fact take up the majority of the time. Right now that works roughly as follows:
For example, this is the code that actually does the restore:
toolbox/cache-cli/cmd/restore.go
Lines 99 to 124 in e7de131
The archives are a tar file, which supports streaming (de)compression. Thus, it should be possible to interleave the download and unpacking (as well as packing and upload) resulting in less overall latency.
In the bash version this would've been as simple as piping the sftp
output to tar -x
and vice-versa for uploads; in Go this might be slightly more tricky but in general possible and an easy win for faster builds.
It outputs nothing.
Other combinations work:
$ retry echo hello
hello
$ sh -c "echo hello"
hello
See https://ebpf.semaphoreci.com/jobs/fbd289c5-060a-470f-aab0-9c933613642a
[13:44 01/09/2020]: Changing 'go' to version 1.15
Version not found. Please try another one.
[13:44 01/09/2020]: Switch successful.
Trying to use sem-version python 2.5
generates the following error:
sem-version python 2.5
--
| [16:05 26/09/2018]: Changing 'python' to version 2.5
| change-python-version: command not found
| exit code: 220 duration: 0s
| Job Finished
You can see two output's in the upper screenshot:
cache list
output that fails in a weird wayAs you can see, the underlying command is providing an error message that is not handled in the cache list
command.
Would a check of the exit status after that command help with this issue?
In our test case one of the steps involves caching a docker image into the file and restoring it later.
Problem is, if we cache a tar file, the script will compress it which is a time-consuming operation which doesn't really bring a big value.
Default cache command, provided by semaphore:
semaphore@semaphore-vm:~/app_cache_dir$ time tar czPf /tmp/app_image_cache-cac3c3500fc4aa1a4f8dece0edb86e57ad36510e.tar app-image.tar
real 0m34.813s
user 0m33.855s
sys 0m1.230s
without compression:
semaphore@semaphore-vm:~/app_cache_dir$ time tar cPf /tmp/app_image_cache-cac3c3500fc4aa1a4f8dece0edb86e57ad36510e.2.tar app-image.tar
real 0m0.708s
user 0m0.049s
sys 0m0.655s
The possible solution would be to provide a flag option to cache
, like: cache store --compress=false my_file_or_directory
which would add or remove z
option from tar command.
Switching node versions to 18.16.1 doesn't work:
semaphore@semaphore-vm:~$ sem-version node 18.16.1
[16:03 22/06/2023]: Changing 'node' to version 18.16.1
Version '18.16.1' not found - try `nvm ls-remote` to browse available versions.
N/A: version "v18.16.1" is not yet installed.
You need to run `nvm install 18.16.1` to install and use it.
changed 59 packages in 619ms
4 packages are looking for funding
run `npm fund` for details
[16:03 22/06/2023]: Switch successful.
Confirming it didn't work (this should print v18.16.1
):
semaphore@semaphore-vm:~$ node -v
v18.15.0
The sem-version
command exits with status 0, so the build proceeds even though the version switch didn't work.
I would like to see a LICENSE file in the repo
We started using asdf as a runtime version manager. It's an 18k GitHub stars project. Would you mind adding asdf to your toolbox?
I see the following advantages for your users:
~/.asdf/installs
).Similar purpose example https://github.com/semaphoreci/cli/blob/master/CHANGELOG.md.
As mention in title, when I calling retry checkout
in my task, the log shows following error. So how can I solve this issue?
environment: line 11: checkout::refbased: command not found00:00
environment: line 17: checkout::metric: command not found00:00
[1/3] Execution Failed with exit status 127. Retrying.00:00
environment: line 11: checkout::refbased: command not found00:00
environment: line 17: checkout::metric: command not found00:00
[2/3] Execution Failed with exit status 127. Retrying.00:00
environment: line 11: checkout::refbased: command not found00:00
environment: line 17: checkout::metric: command not found00:00
[3/3] Execution Failed with exit status 127. No more retries.
my prologue
commands:
- sem-version go 1.15
- mkdir -vp "${SEMAPHORE_GIT_DIR}" "$(go env GOPATH)/bin"
- retry checkout
Hello!
So it looks like this commit: 147c12b broke my builds. I can see bash: cache: command not found
.
I'm not the expert in this stuff but 147c12b#diff-3ce1bafe737468bc6fc4d4f06a4b7e1fR34 doesn't whoami
return root
in docker? I think this if
statement should be in different order.
What supposed to be in ~/.toolbox
directory? Why is it supposed to be at ~/
? I cloned this repo and nothing resembles ~/.toolbox
in my setup.
Example:
sem-service start mysql 5.7
sem-service start postgres 9.4
Extend sem-version and not use this sudo update-alternatives --set gcc "/usr/bin/gcc-6"
.
https://github.com/semaphoreci/toolbox/blob/master/install-toolbox#L27-L37
@CommanderK5 we can do two things here:
sudo
as a prerequisite for docker containersI would prefer solution n.2 as it requires less documentation, and introduces less headaches.
When cache store
command is canceled or for any other reason cache archive is not uploaded successfully, temporary upload files remain in the cache and are viable witch cache list
. This might confuse users so it would be good to filter out these entries from the cache list
output.
Note: Temporary upload files have a limited lifetime and are automatically removed after 2 hours.
The checkout command is a sourced script in the environment. Exits are are causing SSH sessions to close. We should use returns with appropriate statuses.
Example:
FROM alpine:3.4
RUN apk add bash git lftp
ENTRYPOINT bash
I'm using a custom docker image for my Semaphore CI workflow. For that image I intentionally create a user to avoid executing the workflow on behalf of root. The problem now is, however, that in the beginning of the workflow, the toolbox installation is happening, which executes commands with sudo
:
Line 33 in 4441c16
Which is not possible in my context. My options now are to allow executing sudo without a password for my user or even using root for that matter now. However, this is something that I wanted to avoid in the first place.
What would my options now be? Is it possible to skip the installation of the toolbox at runtime and do that once when I build the image?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.