Giter Club home page Giter Club logo

machawk1 / wail Goto Github PK

View Code? Open in Web Editor NEW
344.0 14.0 32.0 851.76 MB

:whale2: Web Archiving Integration Layer: One-Click User Instigated Preservation

Home Page: https://matkelly.com/wail

License: MIT License

Shell 3.53% XSLT 0.08% Perl 0.28% Java 7.38% HTML 13.45% Python 3.15% Batchfile 1.61% Dockerfile 0.03% C 7.93% Objective-C 0.07% C++ 2.96% CSS 5.40% Roff 50.26% JavaScript 3.86%
web-archiving wayback python heritrix gui warc openwayback pyinstaller

wail's Introduction

WAIL logo

"One-Click User Instigated Preservation"


Web Archiving Integration Layer (WAIL) is a graphical user interface (GUI) atop multiple web archiving tools intended to be used as an easy way for anyone to preserve and replay web pages.

Tools included and accessible through the GUI are Heritrix 3.2.0 and OpenWayback 2.4.0. Support packages include Apache Tomcat, pyinstaller, and MemGator.

WAIL is written in Python and compiled to a native executable using PyInstaller.

NOTE: WAIL has also been reimagined in WAIL-Electron. The reimagined version enables high fidelity web archiving using modern capture and replay tools as well as collection-based archive organization. For natively using conventional institutional-grade Web archiving tools like OpenWayback and Heritrix, the repo where you reside is the place to be.

Installing WAIL

WAIL is an application that runs either on your macOS (.app) or Windows (.exe) system.

To install WAIL:

  1. Download the appropriate package for your file system (.dmg for macOS or .zip for Windows) or see more releases.
  2. Open the package (.dmg or .zip).
  3. Follow the instructions to drag-and-drop the WAIL application to the correct location on your file system.

Alternatively, WAIL can be installed with the homebrew package manager on macOS with:

brew install wail

Running WAIL

This section is intended only to run WAIL from source. To download the compiled application, see the downloads section.

End-user execution is meant to be accessed through the binary file, either WAIL.app on macOS or WAIL.exe on Windows. To run it using Python for testing, run the following from the root of the WAIL source directory:

python ./bundledApps/WAIL.py

Since Wayback and Heritrix configurations rely on absolute paths on the system, checks and interactions with services may not work in debugging mode unless a binary of WAIL (e.g., WAIL.app) currently exists in directory specific to your operating system (see below).

Python is not required to be installed for end-users, just double-click (see above) and go!

Development

To build WAIL from source to a native system executable (.app or .exe), open a terminal or console session and move into the root of the WAIL source directory then:

macOS

sh bundledApps/MAKEFILE.sh
This will create /Applications/WAIL.app on macOS.

Generating App Icons

On macOS, application icons are stored in a .icns file. To generate this:

  1. Create a directory to store images: mkdir wail_blue.iconset
  2. Create a PNG of your high resolution (1024x1024) icon, currently based on /build/icons/wail-blue-logo.psd, name it icon_1024x1024.png and place it in the wail_blue.iconset
  3. In a terminal, cd wail_blue.iconset then run the script to generate the other images needed for an .icns with ../build/resize.sh. There will then be 11 PNGs in the directory.
  4. Navigate to the parent directory with cd ../ and run iconutil -c icns wail_blue.iconset. This will create wail_blue.icns.
  5. Store wail_blue.icns in WAIL/build/icons/ when running building WAIL using MAKEFILE.sh on macOS.

Windows

From the Windows shell:
bundledApps\MAKEFILE.bat
then move the WAIL source directory to the root of your C drive (thus making C:\WAIL\).

Linux

Linux support is currently in-development using Docker (see #2). From the root of the WAIL source working directory, run:
docker build -t machawk1/wail .

then

docker container run -d -it --rm -p 5920:5920 --name wail machawk1/wail

To log into the container to view the WAIL interface point a VNC Client to localhost:5920. The container can also be accessed via the command-line with:

docker exec -it wail /bin/bash

Once in the container, run the WAIL executable with:

./WAIL

There still exists some issues to be resolved for #2, as will be evident in the reports on the console.

Problems? Questions?

Please see the Frequently Asked Questions page.

Contact

WAIL is a project of the Web Science and Digital Libraries (WS-DL) Research Group at Old Dominion University (ODU), created by Mat Kelly with additional extensive contributions by John Berlin.

For support e-mail [email protected] or tweet to us at @machawk1 and/or @WebSciDL.

wail's People

Contributors

ibnesayeed avatar machawk1 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

wail's Issues

Port WAIL to Linux

The question remains: where is the most appropriate location that is system independent where WAIL could reside on a Linux system? C:\WAIL is used for Windows, /Applications/WAIL on MacOS X.

JRE/JDK checks prevent use on Windows

Please define either JAVA_HOME or JAVACMD or make sure java.exe is in PATH
https://lorem:ipsum@localhost:8443/engine/job/1363023499

[XAMPP]: Searching for JDK or JRE HOME with reg query ...
ERROR: The system was unable to find the specified registry key or value.
. [XAMPP]: Could not find 32 bit or 64 bit JDK
. [XAMPP]: Looking for 32 bit JDK on 64 bit machine
ERROR: The system was unable to find the specified registry key or value.
. [XAMPP]: Could not find 32 bit JDK
. [XAMPP]: Looking for 32 bit or 64 bit JRE
ERROR: The system was unable to find the specified registry key or value.
. [XAMPP]: Could not find 32 bit or 64 bit JRE
. [XAMPP]: Looking for 32 bit JRE on 64 bit machine

HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\JavaSoft\Java Runtime Environment
Java7FamilyVersion REG_SZ 1.7.0_15
CurrentVersion REG_SZ 1.7
BrowserJavaVersion REG_SZ 10.15.2

HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\JavaSoft\Java Runtime Environment\1.7
JavaHome REG_SZ C:\Program Files (x86)\Java\jre7
RuntimeLib REG_SZ C:\Program Files (x86)\Java\jre7\bin\client\jvm.dll
MicroVersion REG_SZ 0

HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\JavaSoft\Java Runtime Environment\1.7.0_
15
JavaHome REG_SZ C:\Program Files (x86)\Java\jre7
MicroVersion REG_SZ 0
RuntimeLib REG_SZ C:\Program Files (x86)\Java\jre7\bin\client\jvm.dll

HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\JavaSoft\Java Runtime Environment\1.7.0_
15\MSI
JU REG_SZ 1
OEMUPDATE REG_SZ
MODE REG_SZ C
JQS REG_SZ
FROMVERSION REG_SZ
FROMVERSIONFULL REG_SZ
PRODUCTVERSION REG_SZ 7.0.150
INSTALLDIR REG_SZ C:\Program Files (x86)\Java\jre7
PATCHDIR REG_SZ
WEB_JAVA_SECURITY_LEVEL REG_SZ
WEB_JAVA REG_SZ
EULA REG_SZ 0
IEXPLORER REG_SZ 1
MOZILLA REG_SZ 1
JAVAUPDATE REG_SZ 1
AUTOUPDATECHECK REG_SZ 1
AUTOUPDATEDELAY REG_SZ
ImageCkSum REG_SZ 799880791
FullVersion REG_SZ 1.7.0_15-b03

[XAMPP]: Using JRE

[XAMPP]: Seems fine!
[XAMPP]: Set JRE_HOME : C:\Program Files (x86)\Java\jre7
[XAMPP]: Set CATALINA_HOME : C:\WAIL\tomcat

Could Not Find C:\WAIL\tomcat\logs\catalina.pid
Using CATALINA_BASE: "C:\WAIL\tomcat"
Using CATALINA_HOME: "C:\WAIL\tomcat"
Using CATALINA_TMPDIR: "C:\WAIL\tomcat\temp"
Using JRE_HOME: "C:\Program Files (x86)\Java\jre7"
Using CLASSPATH: "C:\WAIL\tomcat\bin\bootstrap.jar;C:\WAIL\tomcat\bin\tomc
at-juli.jar"
https://lorem:ipsum@localhost:8443/engine/job/1363023499
TypeError: 'undefined' is not an object (evaluating 'build.click')

phantomjs://webpage.evaluate():4
phantomjs://webpage.evaluate():5
phantomjs://webpage.evaluate():5
Mar 11, 2013 10:38:25 AM org.apache.catalina.startup.Catalina stopServer
SEVERE: Catalina.stop:
java.net.ConnectException: Connection refused: connect
at java.net.DualStackPlainSocketImpl.connect0(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(Unknown Source)
at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
at java.net.AbstractPlainSocketImpl.connect(Unknown Source)
at java.net.PlainSocketImpl.connect(Unknown Source)
at java.net.SocksSocketImpl.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at java.net.Socket.(Unknown Source)
at java.net.Socket.(Unknown Source)
at org.apache.catalina.startup.Catalina.stopServer(Catalina.java:499)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.catalina.startup.Bootstrap.stopServer(Bootstrap.java:371)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:453)

TypeError: 'undefined' is not an object (evaluating 'launch.click')

phantomjs://webpage.evaluate():4
phantomjs://webpage.evaluate():5
phantomjs://webpage.evaluate():5

Unify the packaged dependencies into one maintainable package

Currently, there are deviations in the OS X and Windows version that must be hard-coded to the OS (e.g., the paths specified in the wayback.xml configuration file). Once these xml files are able to be created on first execution / app-setup, make it so that two separate directories don't need to be maintained for each OS.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.