git clone https://github.com/Tikquuss/eulascript

• PyPDF2 and PyMuPDF: for reading pdf files
• python-docx : for reading docx files
• wget : for model downloading
• pandas : to write the result in csv files
• validators : to check the validity of the urls
• ktrain : for loading models. It is a duplication of amaiya/ktrain modified to install tensorflow-cpu (instead of tensorflow-2.1.0-cp36-cp36m-manylinux2010_x86_64.whl) and tqdm>=4.29.1.

pip install -r eulascript/requirements.txt

• model_folder : directory (or url of the directory) where the model is located (must contain the following three files: tf_model.preproc, config.json and tf_model.h5). In the case of a url the three previous files are downloaded automatically. You can use the pre-trained models directly from huggingface, but this notebook illustrates how to fine-tune these models (bert, distilbert, albert, roberta, xlnet) on our dataset with the ktrain library.
• output_dir : folder in which the csv file(s) containing the results (in the format: clause, label, probability) will be stored (the name of the created file starts with the name, without extension, of the original file containing the license, followed optionally by a number to avoid file collisions)
• path_to_eula : comma-separated list of documents (txt, md, pdf and docx) containing the licenses to be analyzed
• logistic_regression : this parameter can be provided at the expense of model_folder in order to use one of the pre-trained logistic regression models (must be obligatorily made from these three models: bag_of_word, tf_idf, bert or distilbert). This parameter is ignored if it is passed at the same time as model_folder. This notebook illustrates the process of obtaining the production.pth file.

model_folder=my/model_dir_or_url
output_dir=my/output_folder
path_to_eula=my/eula.txt,my/eula.md,my/eula.pdf,my/eula.docx

python eulascript/eula.py --model_folder $model_folder --path_to_eula $path_to_eula --output_dir $output_dir

logistic_regression=bag_of_word
output_dir=my/output_folder
path_to_eula=my/eula.txt,my/eula.md,my/eula.pdf,my/eula.docx

python eulascript/eula.py --logistic_regression $logistic_regression --path_to_eula $path_to_eula --output_dir $output_dir

Note:

• the samples folder contains some user licenses and a notebook illustrating all.
• The associated web application is available here.