Giter Club home page Giter Club logo

commandunit's Introduction

The commandunit testing framework.

The commandunit is a testing framework intended to verrify your command line tools. You can define your tests in JSON or YAML formats. It is also powered by jq-front processor, with which you can write your JSON files using inheritance and references.

Usage

Runs tests.

Usage: commandunit [OPTION]... [SUBCOMMAND]...

Sub-commands:

  • preprocess: Preprocesses test definition files (yaml++, yaml, and json++) and convert them into "executable JSON test" files
  • run: Runs tests under a directory specified by --test-workdir and writes a report file: testreport.json under a directory specified by --test-reportdir.
  • report: Reads a file testreport.json under directory specified by --test-reportdir and renders a report file (testreport.adoc) under the same directory.

If no sub-command is given, it will be interpreted as a sequence of preprocess , run, and report.

Options:

  • -h, --help: show this help
  • -p, --parallel: execute the tests in parallel
  • -f, --filter: filter tests with the specified regular expression(default: {regexany})
  • --configdir: directory to store config files (default: current {directory/.commandunit})
  • --test-srcdir: write test reports under the specified directory (default: current directory)
  • --test-workdir: write test reports under the specified directory (default: value for --test-srcdir)
  • --test-reportdir: write test reports under the specified directory (default: value for --test-workdir)

Examples:

commandunit Run tests found under current directory in sequential mode.

commandunit --srcdir=DIR Run tests found under DIR in sequential mode.

commandunit --srcdir=DIR -p Run tests found under DIR in parallel mode.

Installation

Place the following script in one of your directories on PATH environment variable entry with 755 permission.

#!/usr/bin/env bash

function __commandunit_exec_commandunit() {
  export PROJECT_BASE_DIR="${PWD}"
  export COMMANDUNIT_PWD="${PROJECT_BASE_DIR}" # Referenced by commandunit itself.
  export COMMANDUNIT_SOURCE_DIR="${PROJECT_BASE_DIR}/src/dependencies/commandunit"
  export COMMANDUNIT_DEFAULT_VERSION="${COMMANDUNIT_DEFAULT_VERSION:-v1.29}"
  export BUD_DEBUG=enabled
  function __commandunit_user_option() {
    case "$(uname -sr)" in

    Darwin*)
      echo -n "-u" "1000:1000"
      ;;

    Linux*Microsoft*)
      echo -n "-u" "$(id -u):$(id -g)"
      ;;

    Linux*)
      echo -n "-u" "$(id -u):$(id -g)"
      ;;

    CYGWIN* | MINGW* | MSYS*)
      echo -n "-u" "$(id -u):$(id -g)"
      ;;

    # Add here more strings to compare
    # See correspondence table at the bottom of this answer

    *)
      echo -n "-u" "$(id -u):$(id -g)"
      ;;
    esac
  }
  function __commandunit_exec_commandunit_docker() {
    local _image_name="${1}"
    shift
    # shellcheck disable=SC2086
    # shellcheck disable=SC2046
    docker run \
      $(__commandunit_user_option) \
      --env COMMANDUNIT_PWD="${_project_basedir}" \
      --env COMMANDUNIT_LOGLEVEL="${_loglevel}" \
      --env TMPDIR="/tmp" \
      -v "${_project_basedir}:${_hostfsroot_mountpoint}${_project_basedir}" \
      ${_entrypoint} \
      -i "${_image_name}" \
      "${@}"
  }
  function __commandunit_clean_cloned_commandunit() {
    local _version_name="${1}" _source_dir="${2:-${COMMANDUNIT_SOURCE_DIR}}"
    #      +-- Safeguard for a bug, where the variable becomes empty.
    #     |    Because this function removes everything under the dir.
    #     V
    if [[ "${_source_dir}/${_version_name}" == *"/src/dependencies/"* ]]; then
      if [[ -d "${_source_dir}/${_version_name}" ]]; then
        rm -fr "${_source_dir"?"}/${_version_name:?}"
      fi
    fi
  }
  function __commandunit_clone_commandunit() {
    local _git_tag_option="${1}" _version_name="${2}"
    local _out
    # shellcheck disable=SC2086
    _out="$(git clone --depth 1 ${_git_tag_option} https://github.com/dakusui/commandunit.git "${COMMANDUNIT_SOURCE_DIR}/${_version_name}" 2>&1)" || {
      echo "Failed to clone<: ${_out}>" >&2
      return 1
    }
  }
  function __commandunit_exec_commandunit_native() {
    local _version_name="${1}"
    shift
    "${COMMANDUNIT_SOURCE_DIR}/${_version_name}/src/main/scripts/bin/commandunit-main" "${@}"
  }

  local _project_basedir="${PROJECT_BASE_DIR}"
  local _hostfsroot_mountpoint="/var/lib/commandunit"
  local _docker_repo_name="dakusui/commandunit"
  local _entrypoint=""
  local _native="no"
  local _loglevel="${COMMANDUNIT_LOGLEVEL:-ERROR}"
  local _image_name _image_version="${COMMANDUNIT_DEFAULT_VERSION}" _branch_name="main" _version_name=${COMMANDUNIT_DEFAULT_VERSION} _args _show_image_name=false _i _s _quit=false _help=false
  _args=()
  _s=to_func
  for _i in "${@}"; do
    if [[ "${_s}" == to_func ]]; then
      if [[ $_i == "--version="* ]]; then
        local _v="${_i#*=}"
        if [[ "${_v}" == "snapshot" ]]; then
          # _image_version="${COMMANDUNIT_VERSION%.*}.$((${COMMANDUNIT_VERSION##*.} + 1))"
          local _default_version="${COMMANDUNIT_DEFAULT_VERSION}" # e.g. v1.24
          _image_version="${_default_version%.*}.$((${_default_version##*.} + 1))"
        else
          _branch_name="${_v}"
          _image_version="${_v}"
        fi
        _version_name="${_v}"
        # --version=      snapshot               v1.99          (none)
        # _version_name   snapshot               v1.99           v1.24
        # _image_version  v1.25                  v1.99           v1.24          (private)
        # _image_name     dakusui:v1.25-snapshot dakusui:v1.99   dakusui:v1.24
        # _branch_name    main                   v1.99           v1.24
      elif [[ $_i == "--native" ]]; then
        _native="yes"
      elif [[ $_i == "--debug-shell" ]]; then
        _entrypoint="--entrypoint=/bin/bash"
      elif [[ $_i == "--show-image-name" ]]; then
        _show_image_name=true
      elif [[ $_i == "--quit" ]]; then
        _quit=true
      elif [[ $_i == "--help" ]]; then
        _help=true
        _args+=("${_i}")
      elif [[ $_i == "--" ]]; then
        _s=to_container
      else
        _args+=("${_i}")
      fi
    else
      _args+=("${_i}")
    fi
  done
  _image_name="${_docker_repo_name}:${_image_version}"
  if [[ "${_version_name}" == "snapshot" ]]; then
    _image_name="${_image_name}-${_v}"
  fi
  if ${_show_image_name}; then
    echo "${_image_name}"
  fi
  if ${_help}; then
    echo "Usage: commandunit [WRAPPER OPTION]... [--] [OPTION]... [SUBCOMMAND]..."
    echo ""
    echo "A wrapper function for 'commandunit' to invoke its docker image.".
    echo "Followings are switching options to control the wrapper's behaviour."
    echo "Options not listed here or ones after the separator (--) are passed to the commandunit implementation directly."
    echo ""
    echo "Wrapper options:"
    echo "--native            Use the 'native' version of commandunit."
    echo "--version={VERSION} Use the specified version of commandunit. If 'snapshot' is given, a version under development is used (default: ${COMMANDUNIT_DEFAULT_VERSION})."
    echo "--debug-shell       Get the shell of the docker image. Type Ctrl-D to quit it. Development purpose only."
    echo "--show-image-name   Print the image name. Useful to figure out the version."
    echo "--quit              Quit before running the image. With --show-image-name, useful to figure out the image version"
    echo "--help              Show this help and pass the --help option to the docker image."
    echo "--                  A separator to let this wrapper know the options after it should be passed directly to the image"
    echo ""
  fi
  ${_quit} && return 0
  if [[ "${_native}" == "no" ]]; then
    __commandunit_exec_commandunit_docker "${_image_name}" "${_args[@]}"
  else
    which jq-front > /dev/null || {
      function jq-front() {
        docker run --rm -i \
          -v "${HOME}:/var/lib/jf/${HOME}" \
          -e JF_PATH_BASE="/var/lib/jf" \
          -e JF_PATH="${JF_PATH}" \
          -e JF_DEBUG=${JF_DEBUG:-disabled} \
          -e JF_CWD="$(pwd)" \
          -e COMMANDUNIT_DEPENDENCIES_ROOT="${COMMANDUNIT_DEPENDENCIES_ROOT:?Required environment variable is not set.}" \
          -e COMMANDUNIT_BUILTIN_ROOT="${COMMANDUNIT_DEPENDENCIES_ROOT:?Required environment variable is not set.}" \
          dakusui/jq-front:"${JF_DOCKER_TAG:-v0.53}" "${@}"
      } &&
      export -f jq-front
    }
    if [[ "${_version_name}" ==  "snapshot" ]]; then
      # Remove already loaded one, then... do clone
      __commandunit_clean_cloned_commandunit "${_version_name}"
    fi
    if [[ ! -e "${COMMANDUNIT_SOURCE_DIR}/${_version_name}" ]]; then
      __commandunit_clone_commandunit "--branch ${_branch_name}" "${_version_name}"
    fi
    __commandunit_exec_commandunit_native "${_version_name}" "${_args[@]}"
  fi
}

function main() {
  __commandunit_exec_commandunit "${@}"
}

main "${@}"

If you are on macOS, make sure ~/.bashrc is read by ~/.bash_profile.

Building commandunit

Dependencies

  • jq-front: Native version of jq-front somewhere on the PATH.

Authors

Support

License

License

commandunit's People

Contributors

dakusui avatar

Stargazers

 avatar

commandunit's Issues

Update commandunit function

Update commandunit function.
In the commandunit-example project, there is an improved version of commandunit function and it should be imported to the main project side.

Improve help message

commandunit print help message as follows:

Usage: commandunit [OPTION]... [SUBCOMMAND]...

Runs tests.

Sub-commands:
  preprocess:
    Preprocesses test definition files (yaml++, yaml, and json++) and convert them into executable JSON test files
  run:
    Runs tests under a directory specified by --test-workdir and writes a report file: testreport.json under a 
    directory specified by --test-reportdir.
  report:
    Reads a file testreport.json under directory specified by --test-reportdir and renders a report file (testreport.adoc)
    under the same directory.

 -h, --help            show this help
 -p, --parallel        execute the tests in parallel
 -f, --filter          filter tests with the specified regular expression(default:'.*')
     --commandunit-dir directory to store config and data files (default: current {directory/.commandunit})
     --project-name    set project name of the test execution. used as the report's title(default:'unknown')
     --test-srcdir     write test reports under the specified directory (default: current directory)
     --test-workdir    write test reports under the specified directory (default: value for --test-srcdir + .commandunit/work)
     --test-reportdir  write test reports under the specified directory (default: value for --test-workdir)
     --ignore-mtime    ignore mtime and forcibly compile tests
     --clean           clean working directory
     --tapview         show test progress with 'tapview'

Examples:
  commandunit                 Run tests found under current directory in sequential mode.
  commandunit --srcdir=DIR    Run tests found under DIR in sequential mode.
  commandunit --srcdir=DIR -p Run tests found under DIR in parallel mode.

- documentation: <https://dakusui.github.io/commandunit/>
- github project: <https://github.com/dakusui/bud>

This will be better.

Usage: commandunit [OPTION]... [SUBCOMMAND]...

Runs tests.

Sub-commands:
  preprocess:
    Preprocesses test definition files (yaml++, yaml, and json++) and convert them into executable JSON test files
  run:
    Runs tests under a directory specified by --test-workdir and writes a report file: testreport.json under a 
    directory specified by --test-reportdir.
  report:
    Reads a file testreport.json under directory specified by --test-reportdir and renders a report file (testreport.adoc)
    under the same directory.

 -h, --help            show this help
 -p, --parallel        execute the tests in parallel
 -f, --filter          filter tests with the specified regular expression(default:'.*')
     --commandunit-dir directory to store config and data files (default: current {directory/.commandunit})
     --project-name    set project name of the test execution. used as the report's title(default:'unknown')
     --test-srcdir     execute tests under the specified directory (default: current directory)
     --test-workdir    set a working directory to the specified directory (default: value for --test-srcdir + .commandunit/work)
     --test-reportdir  write test reports under the specified directory (default: value for --test-workdir)
     --ignore-mtime    ignore mtime and forcibly compile tests
     --clean           clean working directory
     --tapview         show test progress with 'tapview'

Examples:
  commandunit                 Run tests found under current directory in sequential mode.
  commandunit --srcdir=DIR    Run tests found under DIR in sequential mode.
  commandunit --srcdir=DIR -p Run tests found under DIR in parallel mode.

- documentation: <https://dakusui.github.io/commandunit/>
- github project: <https://github.com/dakusui/bud>

commandunit fails on macOS in non-native mode

commandunit fails with the following output:

INFO: Local repository path that has SSH Git URL '/Users/scott.tiger/Desktop/somewhere'
mkdir: cannot create directory '/tmp/bud': No space left on device
ERROR:(exit code:1, pipe status: 1): 
  at /app/dependencies/bud/lib/core.rc:54 (abort)

Some other users can successfully use commandunit (non-native) on macOS.
The only known difference is

If TMPDIR is not set -> This failure is observed.
If TMPDIR is set (Something like /var/folders/c9/ndzj29354pn6h9hcjqz6gy15j6nqy4/T/) -> This failure is not observed.

We don't have knowledge about what component is (should be) responsible for defining TMPDIR environment variable in macOS.
(Help needed)

Directory for test target cannot be created as there is no permission to do it

How to reproduce

  1. Install commandunit in the former procedure written in https://github.com/dakusui/commandunit#installation.
  2. Try to run tests by commandunit run command in the directory where commandunit tests are defined.
  3. The following error was logged and the execution failed

$ commandunit run mkdir: cannot create directory '/var/lib/commandunit/Users/<user_name>/Desktop/dev/gspci-shared-resources/gspci/src/test/target': Permission denied ERROR:(exit code:1, pipe status: 1): at /app/dependencies/bud/lib/core.rc:54 (abort) at /app/bin/commandunit:138 (main) at /app/bin/commandunit:193 (main)

Consider having capability to install another command that is not natively installed

While I was adding tests using commandunit, one of them requires curl command to be installed.
However, the docker image used for commandunit (You can see it here, which is originated in the image in jq-front).

I also got below error when I was trying to install curl command in the docker image that is launched by commandunit --debug-shell command.

I have no name!@8e93788c6fc1:/$ apt-get install curl E: Could not open lock file /var/lib/dpkg/lock-frontend - open (13: Permission denied) E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), are you root?

If I add sudo to the beginning, I got this.

I have no name!@8e93788c6fc1:/$ sudo apt-get install curl bash: sudo: command not found

I'm not sure whether curl should be natively installed, but at least I'd like to have a capability to install another command required for the test.

commandunit requires COMMANDUNIT_PWD to be set

commandunit fails with the following output.

$ commandunit --native
/home/hiroshi/Documents/github/commandunit-example/src/dependencies/commandunit/src/main/scripts/bin/commandunit: line 77: COMMANDUNIT_PWD: unbound variable
ERROR:(exit code:1, pipe status: 1): 
  at src/dependencies/commandunit/src/main/scripts/dependencies/bud/lib/core.rc:54 (abort)
  at src/dependencies/commandunit/src/main/scripts/bin/commandunit:77 (main)
  at src/dependencies/commandunit/src/main/scripts/bin/commandunit:193 (main)

Command unit should set an appropriate for the environment variable no to bother uses to set it.

Sourcing a file inside the sourced file is not working

I have a script file(scriptA.sh) which sources scriptB.sh;
scriptA.sh

#!/usr/bin/env bash
  source scriptB.sh

when I create commandunit script such as below and source scriptA.sh, commandunit cannot find scriptB.sh file and giving ".../scriptA.sh: line 6: /dev/scriptB.sh: No such file or directory" error message.
test.yaml

"$extends":
  - some.json
when:
  source: 
   -scriptA.sh
  cmd:
then:
  exitCode:
    - EQUAL
    - 0

Remove unnecessary link in the test report

Currently, the generated report (testreport.adoc) contains a link (inclusion) shown below:

  [%collapsible]  
 ====
 [source, json]
 ----
 include::./core/test-cat.json[]
 ----

But this is not necessary and doesn't render with the following error:

This element should go away.

Unresolved directive in testreport.adoc - include::./core/test-cat.json[]

Change default values of --xyzdir options

Change default values of --xyzdir options because the behavior is not intuitive.

That is, I always use the following setting and it would be easier to understand the behavior

commandunit --test-srcdir=./src/test/scripts/ --test-workdir=./commandunit-out/work  --test-reportdir=./commandunit-out/report

Make it possible to reference environment variables from `environmentVariables` section.

Make it possible to reference environment variables from environmentVariables section.

That is, what we want to do is the following, but the SCRIPTS_DIR environment variable cannot be defined based on other environment variables.

---
"$extends":
  - base/normal.json
when:
  environmentVariables:
    - SCRIPTS_DIR: "${COMMANDUNIT_HOSTFSROOT_MOUNTPOINT}/${COMMANDUNIT_PWD}/src/main/scripts"
  source:
    - ${COMMANDUNIT_DEPENDENCIES_ROOT}/bud/lib/core.rc
    - ${COMMANDUNIT_HOSTFSROOT_MOUNTPOINT}/${COMMANDUNIT_PWD}/src/main/scripts/target_lib.rc
  cmd: cat
  args:
    - ${SCRIPTS_DIR}/hello.txt
then:
  exitCode:
    - EQUAL
    - 0
  stdout:
    present:
      - REGEX:Hello world!
  stderr:
    absent:
      - REGEX:.+
      - ```

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.