whiteinge / ok.sh Goto Github PK

The basic (non-GitHub, non-HTTP-request) functionality should be friendly for unit testing. The HTTP requests should be covered by a couple basic integration tests.

I don't think I'll test the actual GitHub-centric functions because there's not a ton of value there if the other things are working right. Plus I'd have to send actual requests against the GitHub API to do so. Feedback welcome.

On Kubuntu 14.04, when I try to create a repo, I get these errors:

$ octokit.sh create_repo testing_octokit
/home/user/bin/octokit.sh: 397: local: |: bad variable name
/home/user/bin/octokit.sh: 577: /home/user/bin/octokit.sh: [[: not found

I guess it is because the shebang is #!/usr/bin/env sh and sh does not support local or [[.

When I run it with bash, I get another error:

$ bash octokit.sh create_repo testing_octokit
Client Error: 422 Unprocessable Entity

Would be helpful to display the error message from GitHub at the CLI. E.g.

{
  "message": "Validation Failed",
  "errors": [
    {
      "resource": "Milestone",
      "code": "missing_field",
      "field": "title"
    },
    {
      "resource": "Milestone",
      "code": "missing_field",
      "field": "slug"
    }
  ],
  "documentation_url": "https://developer.github.com/v3/issues/milestones/#create-a-milestone"
}

These are a PITA to work with via the shell. How hard would it be to add a simple parser to ok.sh?

https://tools.ietf.org/html/rfc6570

Small functions that present a nice interface on making a request and processing the output for easy use with other common CLI tools.

I can create a release but when I try to upload a asset to release run

./scripts/ok.sh -v upload_asset $USER $REPO 13452549 javadoc.jar application/x-tar < ./build/libs/javadoc.jar

in run into

parse error: Invalid numeric literal at line 1, column 9
jq parse error; invalid JSON.
ok.sh INFO: Remaining next link follows: 50
./scripts/ok.sh: line 1652: upload_url: Upload URL could not be retrieved.

I use jq version jq-1.5

Any ideas ?

It's easy enough to mock out a local HTTP server. Should we also bounce a few tests off live GitHub?

Refs #5 and #10.

Describe the bug

As of 502011c, creating a repo with ok.sh create_repo foo fails with:

martinmac:ok.sh brightm$ ./ok.sh create_repo foo
awk: newline in string OK_SH_ACCEPT
OK_SH_U... at source line 1
awk: newline in string OK_SH_ACCEPT
OK_SH_U... at source line 1
awk: newline in string OK_SH_ACCEPT
OK_SH_U... at source line 1
Client Error: 400 Bad Request
martinmac:ok.sh brightm$

Environment (please complete the following information)

• ok.sh release: 502011c
• curl version: curl 7.54.0 (x86_64-apple-darwin18.0) libcurl/7.54.0 LibreSSL/2.6.4 zlib/1.2.11 nghttp2/1.24.1
• jq version: jq-1.6
• What operating system and version.

ProductName:	Mac OS X
ProductVersion:	10.14.3
BuildVersion:	18D109

• shell:

GNU bash, version 5.0.2(1)-release (x86_64-apple-darwin18.2.0)
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Additional context

I git bisected the problem to: 502011c.

From reading the docs and source to ok.sh, I didn't see any way to create an issue? I'm looking to file an issue for every warning in my code; I think I would be able to do so with that.

Pros:

• It's a good bit more readable.

Cons:

• Code churn for no functional reason.
• User _filter overrides can already use this syntax.

 ok.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/ok.sh b/ok.sh
index 4bf3b97..87407e4 100755
--- a/ok.sh
+++ b/ok.sh
@@ -1734,7 +1734,7 @@ list_releases() {
     #
     # Keyword arguments
     #
-    local _filter='.[] | "\(.name)\t\(.tag_name)\t\(.id)\t\(.html_url)"'
+    local _filter='.[] | [.name, .tag_name, .id, .html_url] | @tsv'
     #   A jq filter to apply to the return data.
 
     shift 2

Likely a user problem and not an issue. I am not able to get any useful output from the tool.

Following the one example in the wiki
ok.sh list_repos octocat _filter='.[] | select(.fork == false) | .clone_url'
returns nothing (with or without the filter).
adding -v

+ curl -nsSi -H Accept: application/vnd.github.v3+json -H Content-Type: application/json -X GET https://api.github.com/users/octocat/repos?AWKLIBPATH=%2Fusr%2Flib%2Fx86%5F64%2Dlinux%2Dgnu%2Fgawk
+ set +x
ok.sh INFO: Follow_next: 1
ok.sh INFO: Remaining next link follows: 50

once I add -vvv I get lots of json output but this is hardly usable.
I'm not sure if I am doing something wrong or if I misunderstood what the expected output should be. Any help would be appreciated.

Some API calls use a single user/repo arg and some use separate user & repo args. This is annoying to work with because you have to look up the docs for each. Should ok.sh normalize on one format?

Please add meta data to ok.sh.
For example

#!/bin/bash
#
# One line description for this script.
#
# VERSION       :semver
# DATE          :2017-12-31
# URL           :https://github.com/szepeviktor/debian-server-tools
# AUTHOR        :Viktor Szépe <[email protected]>
# LICENSE       :The MIT License (MIT)
# BASH-VERSION  :4.2+
# DEPENDS       :apt-get install package
# REFS          :url
# DOCS          :url
# SOURCE        :url
# UPSTREAM      :url
# LOCATION      :/usr/local/sbin/script.sh
# OWNER         :root:root
# PERMISSION    :0755
# SYMLINK       :/usr/local/sbin/alias.sh
# CRON-HOURLY   :/usr/local/sbin/script.sh
# CRON-DAILY    :/usr/local/sbin/script.sh
# CRON-WEEKLY   :/usr/local/sbin/script.sh
# CRON-MONTHLY  :/usr/local/sbin/script.sh | mailx -E -s "subject" root
# CRON.D        :2 2<-->* * *<->root<-->/usr/local/sbin/script.sh
# CONFIG        :~/.config/script/configuration

1. JQ is installed:

raspberrypi2> jq
jq - commandline JSON processor [version ]
Usage: jq [options] <jq filter> [file...]

        jq is a tool for processing JSON inputs, applying the..
            ....

Running any command:

qx@raspberrypi2 ~ $ ok.sh list_releases toyfoundry rebadge
/usr/local/bin/ok.sh: 399: local: |: bad variable name

With debug:

qx@raspberrypi2 ~ $ ok.sh org_repos toyfoundry
ok.sh DEBUG: Running command org_repos.
ok.sh DEBUG: Formatting 0 parameters as urlencoded
/usr/local/bin/ok.sh: 399: local: |: bad variable name
ok.sh DEBUG: Processing response.
ok.sh DEBUG: Response status is: 200 OK
ok.sh DEBUG: Outputting requested header 'status_code'.
ok.sh DEBUG: Outputting requested header 'status_text'.
ok.sh DEBUG: Outputting requested header 'Link_next'.
ok.sh INFO: Remaining next link follows: 50
ok.sh DEBUG: Command org_repos exited with 0.

Any ideas would be appreciated!

I used to test this against Busybox installed to a local directory, sandboxed via env -i. I know Busybox isn't exactly POSIX but it's pretty close, and keep the POSIX docs nearby when you're wondering about a particular flag or command. Anyway I can't think of a better way to get a near-POSIX environment set up. (Suggestions very welcome if someone has an idea here.)

It's been a while since I've done that and a few non-compatible things have snuck in. Busybox doesn't seem to have pr (even though it's POSIX), and there's a syntax error (unclosed }) somewhere in the file that Busybox's sh is choking on that dash is otherwise fine with.

So, either:

• Come up with a different way to make a POSIX env for testing and remove mention of Busybox from the README;
• or, Fix Busybox compatibility.

Describe the bug

Add support for github deployments API

• https://developer.github.com/v3/repos/deployments/

./ok.sh list_issues nixos/nix

...

bash -x ./ok.sh -vvv list_issues nixos/nix 2>&1 | tee "$log_file"
..

<= Recv header, 153 bytes (0x99)
0000: Link: <https://api.github.com/repositories/3386088/issues?page=2
0040: >; rel="next", <https://api.github.com/repositories/3386088/issu
0080: es?page=33>; rel="last"

+ read -r hdr val
+ '[' Link = $'\r' ']'
+ val='<https://api.github.com/repositories/3386088/issues?page=2>; rel="next", <https://api.github.com/repositories/3386088/issues?p
age=33>; rel="last"'
+ case "$hdr" in
++ printf %s '<https://api.github.com/repositories/3386088/issues?page=2>; rel="next", <https://api.github.com/repositories/3386088/issues?page=33>; rel="last"'
++ awk '
                BEGIN { RS=", "; FS="; "; OFS=": " }
                {
                    sub(/^rel="/, "", $2); sub(/"$/, "", $2)
                    sub(/^[[:space:]]*</, "", $1); sub(/>$/, "", $1)
                    print "Link_" $2, $1
                }'
+ headers='http_version: 1.1
status_code: 200
status_text: OK
Server: GitHub.com
Date: Thu, 22 Nov 2018 16:02:38 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 93284
Status: 200 OK
X-RateLimit-Limit: 60
X-RateLimit-Remaining: 55
X-RateLimit-Reset: 1542905660
Cache-Control: public, max-age=60, s-maxage=60
Vary: Accept
ETag: 9dc7bc9fda118e6e7066512ebc3da67d
X-GitHub-Media-Type: github.v3; format=json
Link_next: <https://api.github.com/repositories/3386088/issues?page=2
Link_last: <https://api.github.com/repositories/3386088/issues?page=33

realpath $(which awk )
/usr/bin/mawk

mawk -W version
mawk 1.3.3 Nov 1996, Copyright (C) Michael D. Brennan

messing about abit
i guess this is the expected result !!

printf %s '<https://api.github.com/repositories/3386088/issues?page=2>; rel="next", <https://api.github.com/repositories/3386088/issues?page=33>; rel="last"' | awk '
                BEGIN { RS=", "; FS="; "; OFS=": " }
                {
                    sub(/^rel="/, "", $2); sub(/"$/, "", $2)
                    sub(/^ *</, "", $1); sub(/>$/, "", $1)
                    print "Link_" $2, $1
                }'
Link_next: https://api.github.com/repositories/3386088/issues?page=2
Link_last: https://api.github.com/repositories/3386088/issues?page=33

... so this appears a
problem with mawk /^[[:space:]]*</

When running upload_asset, jq reports an error:

$ ./ok.sh upload_asset https://uploads.github.com/repos/ArturKlauser/raspberrypi-rstudio/releases/23039592/assets?name=foo1 ok.sh mime_type=application/octet-stream
parse error: Invalid numeric literal at line 1, column 9
jq parse error; invalid JSON.

The error goes away when -q is specified.
Some digging shows that apparently the HTTP headers are piped into jq before the actual json data:

$ printf ':\ncat -\n' > myjq; chmod 755 myjq
$ OK_SH_JQ_BIN=./myjq ~/Downloads/ok.sh upload_asset https://uploads.github.com/repos/ArturKlauser/raspberrypi-rstudio/releases/23039592/assets?name=foo2 ok.sh mime_type=application/octet-stream
HTTP/1.1 201 Created
Cache-Control: no-cache
Content-Security-Policy: default-src 'none'
Content-Type: application/json; charset=utf-8
Etag: W/"75c6982651342ea7db8fa9949fc7b3b4"
Last-Modified: Wed, 22 Jan 2020 14:59:26 GMT
Strict-Transport-Security: max-age=31557600
Vary: Accept, Authorization, Cookie, X-GitHub-OTP
X-Accepted-Oauth-Scopes: repo
X-Content-Type-Options: nosniff
X-Frame-Options: deny
X-Github-Media-Type: github.v3; format=json
X-Oauth-Scopes: repo
X-Xss-Protection: 1; mode=block
Date: Wed, 22 Jan 2020 14:59:26 GMT
Transfer-Encoding: chunked
X-GitHub-Request-Id: CE13:5182:68E29:8648C:5E28634C

{"url":"https://api.github.com/repos/ArturKlauser/raspberrypi-rstudio/releases/assets/17506513","id":17506513,"node_id":"MDEyOlJlbGVhc2VBc3NldDE3NTA2NTEz","name":"foo2","label":"","uploader":{"login":"ArturKlauser","id":4790123,"node_id":"MDQ6VXNlcjQ3OTAxMjM=","avatar_url":"https://avatars0.githubusercontent.com/u/4790123?v=4","gravatar_id":"","url":"https://api.github.com/users/ArturKlauser","html_url":"https://github.com/ArturKlauser","followers_url":"https://api.github.com/users/ArturKlauser/followers","following_url":"https://api.github.com/users/ArturKlauser/following{/other_user}","gists_url":"https://api.github.com/users/ArturKlauser/gists{/gist_id}","starred_url":"https://api.github.com/users/ArturKlauser/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ArturKlauser/subscriptions","organizations_url":"https://api.github.com/users/ArturKlauser/orgs","repos_url":"https://api.github.com/users/ArturKlauser/repos","events_url":"https://api.github.com/users/ArturKlauser/events{/privacy}","received_events_url":"https://api.github.com/users/ArturKlauser/received_events","type":"User","site_admin":false},"content_type":"application/octet-stream","state":"uploaded","size":60861,"download_count":0,"created_at":"2020-01-22T14:59:25Z","updated_at":"2020-01-22T14:59:26Z","browser_download_url":"https://github.com/ArturKlauser/raspberrypi-rstudio/releases/download/untagged-dc669712e7a6ef40f473/foo2"}

Include logs

• Run ok.sh -x [...command here...] 2> ok-shdebug.log and attach that file.

Environment (please complete the following information)

• ok.sh release
  • 0.5.1, same happens on HEAD
• curl version
  • curl 7.64.1 (x86_64-apple-darwin19.0) libcurl/7.64.1 (SecureTransport) LibreSSL/2.8.3 zlib/1.2.11 nghttp2/1.39.2, Release-Date: 2019-03-27
• jq version.
  • jq-1.6
• What operating system and version.
  • MacOS Catalina 10.15.2
• What shell.
  • GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin19)

ok-shdebug.log

So I try to use create_release with these arguments: stellaris-mods testing 5.20170918 body="* Test\n * Test2"

And I get this result:

0000: {.  "message": "Invalid request.\n\nFor 'properties/tag_name', 5
0040: .20170918 is not a string.",.  "documentation_url": "https://dev
0080: eloper.github.com/v3".}.

Sent data:

=> Send data, 109 bytes (0x6d)
0000: {"tag_name": 5.20170918, "body": "* Updated readme.md with chang
0040: es from steam.bbcode.\n* Skogfisk eplepai."}.
== Info: upload completely sent off: 109 out of 109 bytes

I get the same result if I quote the argument on the command line, like:
stellaris-mods testing '5.20170918' body="* Test\n * Test2"

I have not tried using " to quote it, and I don't have time to do so right now (because I deleted the repository already and I need to go make dinner).

Extract help-text out into generated (?) README file.

I'm struggling to find information about this. What values can I inject?

Edit: I assume it's ordering. But it should be clarified in the docs if that is the case.

Needed to avoid a couple gotchas since the README is generated and the test suite is custom.

cc #28

Couple things:

• local is a bashism and definitely not POSIX. It doesn't exist in compliant shells like ksh, yash, many others. Interestingly ok.sh is ok in oksh.

• The shebang is not POSIX, either, strictly speaking. You can begin a script with a colon instead.

ok.sh: line 200: local: not found
ok.sh: line 201: local: not found
ok.sh: line 202: local: not found
ok.sh: line 203: local: not found
ok.sh: line 204: local: not found
ok.sh: line 205: local: not found
ok.sh: line 206: local: not found
ok.sh: line 208: local: not found
ok.sh: line 154: local: not found
ok.sh: line 156: local: not found
No command given. Available commands:

$

• toybox is much more compliant than busybox, fwiw.

Now ok.sh uses .netrc file for authorization verification, This is available in most cases.

But in some special cases, this doesn't work very well. like Travis CI.

If Travis CI set a no display value environment, unable to generate .netrc file. and then call some api will return Client Error: 401 Unauthorized

Is it possible to add an environment variable?

like this

https://github.com/whiteinge/ok.sh/blob/master/ok.sh#L697

curl -nsSig \
    -H "Accept: ${OK_SH_ACCEPT}" \
    -H "Content-Type: ${content_type}" \
    ${etag:+-H "If-None-Match: \"${etag}\""} \
    ${GITHUB_TOKEN:+-H "Authorization token ${GITHUB_TOKEN}"} \
    ${has_stdin:+--data-binary @-} \
    ${trace_curl:+--trace-ascii /dev/stderr} \
    -X "${method}" \
    "${path}"

Set up GitHub Pages web site with usage & examples.

This function is inconsistent with the others. See the note in 0066212.

Hi,

I'm having a problem using the upload_asset command with a github organization.

Here is the output of -vvv: https://hastebin.com/iquqitoqep
And this is the tail end of -vvv -j: https://hastebin.com/itutoribat
The first one has been edited heavily to remove the zip binary and ssl data, just for ease of reading.
Second one is just the end, mostly so you can see this:

/home/folk/.local/bin/ok.sh: 1561: local: "url":: bad variable name

Here is the relevant line from my lua script:
oksh("-vvv", "-j", "upload_asset", userId, repo, id, zip, "filename=" .. zip, "mime_type=application/zip")
userId = stellaris-mods
repo = test for the first paste, testing on the short 2nd paste
zip = folk_testmod.zip

My head is fried at the moment, hopefully I can take a closer look soon, and come back with some more information.

Github's releases UI allows you to create new tags as part of creating a new release. Could this functionality also be exposed via the create_release command?

Create issues, create milestones, make a pull request, etc.

Allow passing in an ETag to do an If-None-Match request.

Describe the bug

A clear and concise description of what the bug is and what the expected
behavior is.

Include logs

• Run ok.sh with the -vvv flag to look for any potential problems in the
  HTTP response from GitHub. Feel free to attach that output but remember to
  remove the Authorization header first!
• Run ok.sh -x [...command here...] 2> ok-shdebug.log and attach that file.

Environment (please complete the following information)

• ok.sh release.

• curl version.

• jq version.

• What operating system and version.

• What shell.

  If you are unsure, download
  whatshell.sh and run
  /usr/bin/env sh /path/to/whatshell.sh.

Additional context

Add any other context about the problem here.