Giter Club home page Giter Club logo

htcompact's Introduction

HTCompact

Build Status codecov Codacy Badge

Search through HTCondor log files to extract information, identify patterns, and even build graphs (such as resource utilization).

Installation

HTCompact can be installed directly from GitHub using pip:

pip install git+https://github.com/psyinfra/htcompact.git

I recommend using a virtual environment, (if you're not already); it makes life a lot tidier.

python3 -m venv ~/htcompact
source ~/htcompact/bin/activate
pip install git+https://github.com/psyinfra/htcompact.git

A config file (setup.conf) is also provided, to control the default behavior of htcompact. This is especially useful when installing system-wide, or for adjusting various warning/error thresholds. See CONFIG for more information

Notice:

  • The script is using the python module htcondor 8.9.8, which is not working on MacOS devices.

Getting Started

  • htcompact --help for detailed description
  • general use: htcompact \[files] \[directories] \[config_file] \[args]

Let's assume that we have a directory called logs/ containing these files:

logs >
    395_2.log

    job_5991_0.err
    job_5991_0.log
    job_5991_0.out

    job_5992_23.err
    job_5992_23.log
    job_5992_23.out

    ...

possible configurations:

htcompact -h (show a detailed description to all functionalities)

htcompact path_to_logs/job_5991_0.log

htcompact path_to_logs/job_5991_0 path_to_logs/job_5992_23.log

htcompact path_to_logs (run through all files inside the logs directory)

htcompact path_to_logs/job_5991_* -s  (summarize all files starting with: job_5991_)

htcompact path_to_logs/395_2.log --table-format=pretty

lets consider we also have a config file (see: CONFIG)
a default setup.conf should already exist inside the project folder

possible configurations could be reduced to something like:

htcompact setup.conf
or
htcompact [files/directories] setup.conf (ignores files/directories set inside the config file)

Testing:

you'll find the tests in: /tests \ to simply run the tests:

  • pytest tests More details see: TESTS

where all arguments, files and directories can be set inside that config file
see: CONFIG

Examples:

  • default mode:

    The job procedure of : ../logs/job_5991_0.log
    +-------------------+--------------------+
    | Executing on Host |      cpu: 3        |
    |       Port        |       96186        |
    |      Runtime      |      0:00:04       |
    | Termination State | Normal termination |
    |   Return Value    |         0          |
    +-------------------+--------------------+
    +------------+-------+-----------+-----------+
    | Rescources | Usage | Requested | Allocated |
    +------------+-------+-----------+-----------+
    |    Cpu     |   0   |     1     |     1     |
    |    Disk    | 5000  |   5000    |  3770642  |
    |   Memory   |   0   |   6000    |   6016    |
    +------------+-------+-----------+-----------+
    
  • summarizer mode:

    Example

  • analyser mode:

    Example

  • analysed summary mode:

    Example

htcompact's People

Contributors

adityaj7 avatar aqw avatar mathisloevenich avatar tobiaskadelka avatar

Stargazers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.