[TOC]

# Default libraries
import time
import inspect
import threading
import datetime
import traceback
import re
# Need Install
import cachetools
import bs4
import lxml
import selenium

list of techniques used in the project

• distributing packages
• decorator
• multithreading
• caching

list of optimization used

• design pattern
• logging and verbose
• test suite

Below is the file structure of this release / version:

./
    folderTree.py
    README.md
    testground.py
    testNativeBrowser.py
    testWebMaster.py
    myengine/
        chromedriver.exe
        geckodriver.exe
        MicrosoftWebDriver.exe
    myscraper/
        HKUSites.py
        NativeBrowser.py
        webmaster.py
    myutil/
        testsuit.py
        weberror.py
        webutil.py

As we can see a big restructure was performed. Below is the details of each part:

• weberror.py: the web error that used in the below packages
• webutil.py: useful functions that often used
• testsuit.py: testing classes that help testing

• chromedriver.exe: webdriver of Chrome
• geckodriver.exe: webdriver of Firefox
• MicrosoftWebDriver.exe: webdriver of Edge

• NativeBrowser.py: selenium browser upgraded
• HKUSites.py: websites of HKU
• webmaster.py: webmaster that controlls all the things

The testcase are good examples on how to use them. In general, you should only use webmaster with credentials. (see testground.py).

1. import library

   from myscraper.webmaster import WebMaster

2. make a webmaster instance

   B = WebMaster(username='username', password='password')

3. query it and get a result

   result = B.query('Moodle', 'get_sitemap')

4. terminate the thread when no longer in use

   B.cancel()

• Get the record of the webmaster

  for line in B.get_record():
     print(line)

  Result:

  [ 2020-04-06 20:31:36.576511 ] Moodle               > get_sitemap           :

  This is the called website, called function and called function arguments in string format.

You can pass other settings into the webmaster like this:

B = WebMaster(username='username', password='password', verbose=2)

Below is a list of options:

• verbose ==Integer==: the depth of debugging level you want to see, 3 is enough to see all.
• headless==Boolean==: if False, then turn off headless mode and show browser screen
• init_setting==String==: 3 options: 'All', 'Only Portal' and 'On Demand'
• browser_name ==String==: 3 options: 'Chrome', 'Firefox' and 'Edge'
• options: browser options in list
• path: browser path

These files are created for testing:

• testBrowser.py: test the functionalities of NativeBrowser.py
• testWebMaster.py: test the functionalities of WebMaster.py
• testground.py: testing the functionalities of HKUSites.py

You should test with the above order.

To run the test file, simple command python <filename>

run python testNativeBrowser.py in terminal and you should get the following result:

D:\jupyterbook\Web Scraping\version\Version 9>python testNativeBrowser.py
======Test for NewFirefox         ======
------Test  0: create browser     ------
------Test  1: get url            ------
------Test  2: get page_source    ------
------Test  3: simple wait        ------
------Test  4: advanced wait      ------
------Test  5: create tab         ------
------Test  6: switch tab         ------
------Test  7: destroy tab        ------
------Test  8: destroy browser    ------
========================================

- Use sematic python class annotations
- remove browser cache functions
- add advanced selected HTML function for better parser caching