opencultureconsulting / orcli Goto Github PK

• examples folder https://github.com/DannyBen/bashly/tree/master/examples
• jupyter notebooks
• example data https://github.com/opencultureconsulting/openrefine-batch/tree/master/examples/powerhouse-museum and https://github.com/opencultureconsulting/openrefine-task-runner

provide binder config and badge

• bash_kernel
• jupyter-server-proxy
• see https://github.com/opencultureconsulting/openrefine-task-runner/tree/main/binder
• see https://github.com/felixlohmeier/bibtag21-automatisierung-openrefine

• parse given json file(s) and post every operation to it's individual endpoint (e.g. command/core/text-transform instead of command/core/apply-operations)
• parse historyEntry in response

https://gitpod.io/#https://github.com/opencultureconsulting/orcli/issues/5

prevent OpenRefine from opening the browser

include some lines of OpenRefine's server log in error messages

improve src/lib/error.sh:

echo >&2 "[$(date +'%Y-%m-%dT%H:%M:%S')]: ERROR $1"
shift
for err in "$@"; do echo >&2 "$err"; done
exit 1
}

remove exit statements in commands

explore ways to support a dry-run

• option 1: set -x (resolves variables)
• option 2: trap 'echo "# $BASH_COMMAND"' DEBUG (commands remain unchanged)

if it runs well, could you possibly generate a standalone script with it by excluding output?

• trap 'echo "# $BASH_COMMAND" >&2' DEBUG
• ./test.sh > /dev/null

flag: --split

• help: one file per record; filename = first column (needs to be unique)
• --splitPrefix (optional)
• --splitSuffix (optional)
• add ":::::ORCLI:::::" + row.columnNames[0] + "\n" to row template

cat test | gawk -v prefix="$splitPrefix" -v suffix="$splitSuffix" '/^:::::ORCLI:::::/{if(file){close(file)};file=prefix$2suffix;next}{print > file}'

• dependency: GNU awk (gawk command)
• performance?

• https://github.com/opencultureconsulting/openrefine-client/blob/master/tests-cli.sh
• https://bashly.dannyb.co/usage/testing-your-scripts/

https://gitpod.io/#https://github.com/opencultureconsulting/orcli/issues/7

new syntax:

# multiple projects
orcli batch << EOF
import csv "https://git.io/fj5hF" --projectName "duplicates"
import csv "https://git.io/fj5hF" --projectName "copy"
info "duplicates"
info "copy"
export tsv "duplicates"
EOF

# multiple projects running in parallel #25
orcli batch << EOF
import csv "https://git.io/fj5hF" --projectName "duplicates" &
import csv "https://git.io/fj5hF" --projectName "copy" &
wait
export csv "duplicates" --output duplicates.csv &
export tsv "duplicates" --output duplicates.tsv &
wait
EOF

# shell commands
orcli batch << EOF
import csv "https://git.io/fj5hF" --projectName "duplicates" &
import csv "https://git.io/fj5hF" --projectName "copy" &
wait
echo "finished import"
export tsv "duplicates" --output duplicates.csv &
export tsv "duplicates" --output duplicates.tsv &
wait
wc duplicates*
echo "finished in \$SECONDS seconds"
EOF

implementation:

• use bash -e to execute heredoc / file
• use export for variables OPENREFINE_URL, tmpdir and openrefine_pid
• enhance heredoc/file with sed before exec to support #22, #39 and #43
• explicit flag —parallel #25 is obsolete?

args: file(s) or url(s)
out: project id

replace working directory with script directory

provide a docker container with openrefine and orcli run entrypoint

• build with github actions?
• based on Debian?
• weekly rebuild?
• multi-arch?

calculate starting time and run time of each step
print RAM and CPU usage for each step

see bash-refine templates and openrefine-batch.sh

provide a small getting started tutorial in README.md chapter "Usage" (instead of help screen)

start

• flags: port, workspace directory, memory, autosave
• set env OPENREFINE_URL to http://localhost:${port}
• -x refine.autosave

stop

• flags: port, kill (to use signal -9)
• dependency: lsof (or fuser?)

PID=$(lsof -t -i:${PORT})
kill $PID
while ps -p $PID > /dev/null; do sleep 1; done

print only "real" errors to STDERR

provide row template as FILE, URL or STDIN

create https://asciinema.org recording of basic features and convert it to svg with https://github.com/marionebl/svg-term-cli to enrich README.md

create documentation with examples of how to upgrade to orcli from...

• openrefine-batch
• openrefine-client

maybe directly in README.md of the old repos (before read-only switching)

features:

• import (without sub-command) should derive format from first arg
• export (without sub-command) should derive format from file ext of output parameter (and default to tsv?)

approach:

• collect all args via bashly catch-all setting
• search ${other_args[@]} for file ext (.*) and save unique items (sort -u) in array ext
• error if ${ext[0]} is blank (e.g. stdin)
• error if ${ext[1]} is non-blank
• call orcli import ${ext[0]} ${other_args[@]}

caveat:

• bashly does not support args for command groups (root.commands[0] cannot have both commands and args) so we have to drop sub-commands and put all commands in top-level (which may be overwhelming)
  • import
  • import-csv
  • import-tsv
  • ...

• consistent formatting with shfmt
• shellcheck requires shebang in each file

• remove OPENREFINE_CSRF env in bashly.yml
• improve src/lib/get_csrf.sh

  if ! [[ "${response}" == '{"token":"'* ]]; then
    local version
    version=$(curl -fs http://localhost:3333/command/core/get-version | jq -r '.version' | cut -c 1-3)
  

  • error "no OpenRefine reachable/running at ${OPENREFINE_URL}" if $version is blank
  • pass if $version is in 3.2 3.1 3.0 2.8 2.7
  • else error "getting CSRF token failed!"

get metadata for a project (see Utilities in https://code.opencultureconsulting.com/opencultureconsulting/bash-refine/src/branch/master/templates.sh)

https://gitpod.io/#https://github.com/opencultureconsulting/orcli/issues/9

option 1: args

orcli batch
orcli list
orcli import FILE…
orcli transform PROJECT FILE…
orcli export PROJECT COLUMN…
orcli show PROJECT
orcli delete PROJECT

option 2: flags

orcli batch
orcli list
orcli import FILE…
orcli transform FILE… --project
orcli export COLUMN… --project
orcli show --project
orcli delete --project

support short syntax if ${other_args[@} does not contain any option --projectName and only one import command

example: orcli batch import-csv "https://git.io/fj5hF" export-tsv

approach:

• get projectid from import (stdout?) or call list command after import?
• add projectid to (the end of) command group arrays that start with info transform export (...)

• args: file.sh (executed in a subshell with bash -e)
• flags: port, memory
• start with tmp dir, autosave 2500M
• trap for removing tmp files and killing openrefine java process

MYTMPDIR="$(mktemp -d)"
trap 'rm -rf -- "$MYTMPDIR"' EXIT

https://gitpod.io/#https://github.com/opencultureconsulting/orcli/issues/6

this is essentially a shortcut for

orcli export tsv PROJECT --facets '[ { "type": "list", "expression": "grel:filter(row.columnNames,cn,cells[cn].value.find(/TERM/).length()>0).length()>0", "columnName": "", "selection": [ { "v": { "v": true } } ] } ]'

orcli search PROJECT TERM - apply regex to each column individually and show only matching rows

• flag: match COLUMN
  • optional, repeatable
  • replaces row.columnNames in filter expression
  • no column(s) = all columns
• flag: select COLUMN
  • optional, repeatable
  • filter result set to one or more columns
  • no column(s) = all columns
• flag: mode (rows/records, default rows)
• output: write tsv to stdout

https://bashly.dannyb.co/advanced/bash-completion/

• recommend install via homebrew?

brew install bash jq

• different behavior of macOS (BSD) utilities and GNU core utilities?
• curl version?
• ~/.local/bin/ in $PATH?

support running commands in parallel

approach:

• all command groups will be run as background jobs
• wait function (that checks exit codes) between import-transform and transform-export
• user may add additional waits

example:

orcli batch --parallel \
import a.csv --projectName p1 \
import b.csv --projectName p2 \
transform a.json p1 \
transform a.json p2 \
wait \
transform b.json p1 \
export 1

implement remaining export options

bashly.yml snippet:

        flags:
          - long: --select
            help: filter result set to one or more columns
            repeatable: true
          - long: --facets
            help: filter result set by providing an OpenRefine facets config in json
          - long: --mode
            help: set operation mode
            arg: mode
            default: "row-based"
          - long: --noHeader
            help: do not output column header
          - long: --blankRows
            help: output blank rows
          - long: --quoteAll
            help: quote all cells
          - long: --preview
            help: limit export to 10 rows/records
          - long: --separator
            help: character(s) that separates columns
            arg: separator
            default: "\t"
          - long: --lineSeparator
            help: character(s) that separates rows/records
            arg: lineSeparator
            default: "\n"