Gathers metadata and transforms it to learning object metadata.

Dependencies listed for each interface. Currently tested for Python 2.7 and 3.6. Tests fail for 3.4 and 3.5 (test process goes to sleep in mysql execution).

• lxml
• PyMySQL
• pylom
• prettytable for admin

• NLTK with stopwords and punkt tokenizer
• readability-score
• mwparserfromhell
• requests

• aniso8601
• google-api-python-client
• socksipy for httplib2 proxy

• requests
• RDFLib
• google-api-python-client for purger

Data can be gathered through different supported interfaces. An interface, or module, can be configured with a specific configuration. For example, multiple Wikipedia instances can be harvested with a single Mediawiki module and different configurations. The module can be generic, while the config file contains the specific configurations for that module.

The mangrove-crawler start harvesting a source by providing:

./mangrove-crawler.py -s <source>

Sources can be defined in the config file as sections. The source is equal to the config section header, and the configuration part needs to correspond to the collection name in the collections database table. Usually, the section header is equal to the collection name. Here's an example for source "wikipedia_nl" with comments:

{
    "common": {
        "proxy_host": "123.123.23.23",
        "proxy_port": "3128",
        "db_host": "localhost",
        "db_user": "root",
        "db_passwd": "******",
        "db_name": "mangrove",
        "fs_recordbase": "/var/mangrove_records",
        "fs_workbase": "/var/mangrove_workdir"
    },
    "wikipedia_nl": {
        "proxy_use": false,
        "module": "mediawiki",
        "wiki": "nlwiki",
        "host": "https://nl.wikipedia.org/wiki/",
        "download_path": "http://dumps.wikimedia.org/nlwiki/latest/",
        "context_static": "PO",
        "context_dynamic": false,
        "age_range": "8-12",
        "list_prefix": "Lijst van"
    }
}

The following things to do for a 1.0

• setup.py
• debug output optional
• oaiprovider/uniform database solution for metadata storage
• fix reporter, for generic data storage
• command for new source, generating config file, and database entry
• unittests
• better exceptions and logging