Scrape job websites into a single spreadsheet with no duplicates.
License: MIT License
I'm Paul, I like to build software tools and hardware gadgets.
Failed to install funnel on ubuntu 19.10. Am I missing something? Are there any prerequisite for funnel installation.
$ pip3 install git+https://github.com/PaulMcInnis/JobFunnel.git
Error logs
ERROR: Command errored out with exit status 1:
command: /home/xyz/.asdf/installs/python/3.8.0/bin/python3.8 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-n91obe4e/scikit-learn/setup.py'"'"'; __file__='"'"'/tmp/pip-install-n91obe4e/scikit-learn/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-dbbevzm6/install-record.txt --single-version-externally-managed --compile
cwd: /tmp/pip-install-n91obe4e/scikit-learn/
Complete output (90 lines):
Partial import of sklearn during the build process.
blas_opt_info:
blas_mkl_info:
customize UnixCCompiler
libraries mkl_rt not found in ['/home/xyz/.asdf/installs/python/3.8.0/lib', '/usr/local/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
NOT AVAILABLE
blis_info:
libraries blis not found in ['/home/xyz/.asdf/installs/python/3.8.0/lib', '/usr/local/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
NOT AVAILABLE
openblas_info:
libraries openblas not found in ['/home/xyz/.asdf/installs/python/3.8.0/lib', '/usr/local/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
NOT AVAILABLE
atlas_3_10_blas_threads_info:
Setting PTATLAS=ATLAS
libraries tatlas not found in ['/home/xyz/.asdf/installs/python/3.8.0/lib', '/usr/local/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
NOT AVAILABLE
atlas_3_10_blas_info:
libraries satlas not found in ['/home/xyz/.asdf/installs/python/3.8.0/lib', '/usr/local/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
NOT AVAILABLE
atlas_blas_threads_info:
Setting PTATLAS=ATLAS
libraries ptf77blas,ptcblas,atlas not found in ['/home/xyz/.asdf/installs/python/3.8.0/lib', '/usr/local/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
NOT AVAILABLE
atlas_blas_info:
libraries f77blas,cblas,atlas not found in ['/home/xyz/.asdf/installs/python/3.8.0/lib', '/usr/local/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
NOT AVAILABLE
accelerate_info:
NOT AVAILABLE
/home/xyz/.asdf/installs/python/3.8.0/lib/python3.8/site-packages/numpy/distutils/system_info.py:1896: UserWarning:
Optimized (vendor) Blas libraries are not found.
Falls back to netlib Blas library which has worse performance.
A better performance should be easily gained by switching
Blas library.
if self._calc_info(blas):
blas_info:
libraries blas not found in ['/home/xyz/.asdf/installs/python/3.8.0/lib', '/usr/local/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
NOT AVAILABLE
/home/xyz/.asdf/installs/python/3.8.0/lib/python3.8/site-packages/numpy/distutils/system_info.py:1896: UserWarning:
Blas (http://www.netlib.org/blas/) libraries not found.
Directories to search for the libraries can be specified in the
numpy/distutils/site.cfg file (section [blas]) or by setting
the BLAS environment variable.
if self._calc_info(blas):
blas_src_info:
NOT AVAILABLE
/home/xyz/.asdf/installs/python/3.8.0/lib/python3.8/site-packages/numpy/distutils/system_info.py:1896: UserWarning:
Blas (http://www.netlib.org/blas/) sources not found.
Directories to search for the sources can be specified in the
numpy/distutils/site.cfg file (section [blas_src]) or by setting
the BLAS_SRC environment variable.
if self._calc_info(blas):
NOT AVAILABLE
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-install-n91obe4e/scikit-learn/setup.py", line 290, in <module>
setup_package()
File "/tmp/pip-install-n91obe4e/scikit-learn/setup.py", line 286, in setup_package
setup(**metadata)
File "/home/xyz/.asdf/installs/python/3.8.0/lib/python3.8/site-packages/numpy/distutils/core.py", line 137, in setup
config = configuration()
File "/tmp/pip-install-n91obe4e/scikit-learn/setup.py", line 174, in configuration
config.add_subpackage('sklearn')
File "/home/xyz/.asdf/installs/python/3.8.0/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 1033, in add_subpackage
config_list = self.get_subpackage(subpackage_name, subpackage_path,
File "/home/xyz/.asdf/installs/python/3.8.0/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 999, in get_subpackage
config = self._get_configuration_from_setup_py(
File "/home/xyz/.asdf/installs/python/3.8.0/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 941, in _get_configuration_from_setup_py
config = setup_module.configuration(*args)
File "sklearn/setup.py", line 66, in configuration
config.add_subpackage('utils')
File "/home/xyz/.asdf/installs/python/3.8.0/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 1033, in add_subpackage
config_list = self.get_subpackage(subpackage_name, subpackage_path,
File "/home/xyz/.asdf/installs/python/3.8.0/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 999, in get_subpackage
config = self._get_configuration_from_setup_py(
File "/home/xyz/.asdf/installs/python/3.8.0/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 941, in _get_configuration_from_setup_py
config = setup_module.configuration(*args)
File "sklearn/utils/setup.py", line 8, in configuration
from Cython import Tempita
ModuleNotFoundError: No module named 'Cython'
----------------------------------------
ERROR: Command errored out with exit status 1: /home/xyz/.asdf/installs/python/3.8.0/bin/python3.8 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-n91obe4e/scikit-learn/setup.py'"'"'; __file__='"'"'/tmp/pip-install-n91obe4e/scikit-learn/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-dbbevzm6/install-record.txt --single-version-externally-managed --compile Check the logs for full command output.
It would be great to not search by city, as even if the filter can be set for remote, the results returned are still based on city indicated.
funnel load -s my_settings.yamlSuccessfully scrape and generate .csv
Traceback and ValueError if 'province_or_state' or 'city' is not included
Is your feature request related to a problem? Please describe.
We are missing some setup items for PyPi:
https://medium.com/@joel.barmettler/how-to-upload-your-python-package-to-pypi-65edc5fe9c56
Describe the solution you'd like
We should make this package conform to PyPi specs so that we can offer a more accessible installation
Describe alternatives you've considered
We currently offer installation via github URL, this works totally fine, but I would also like to offer it on PyPi
| option | notes |
|---|---|
workmate or jobmate |
to long? |
slacker or slack |
my favourite but portrays the opposite, ironic? |
workaholic |
accurate but maybe not the best, to long? |
opus or operis |
both mean work in Latin |
Let me know what you guys think, and maybe propose your own suggestions :)
Currently the scraper is still retrieving blurbs for jobs that have been filtered out by the pre_filter method.
Please include a summary of the issue.
Please include the steps to reproduce.
List any additional libraries that are affected.
The scraper should remove jobs identified by the by the pre_filter, and only obtain blurbs for the remaining jobs.
The scraper retrieves blurbs for all jobs whether they were filtered out or not.
To fix the issue, the order of the creation of the scrape_list and call to the pre_filter method would have to be switched. The screenshot below highlights the issue within the code and the debugger output :
Although this could've of been fixed in a pull request, making this fix would break date_filter called by the pre_filter method in the main JobFunnel class.
Currently we get the second page of glassdoor via the URL of the 2 button, but this no longer works as it redirects you to the first page. This is the case wether we use the webdriver or not.
We get to the second page of jobs
We are redirected to the first page during the GET, which leads to every single page of jobs being a duplicate of the first page, with loads of TFIDF duplicate detection hits.
If you click the 2 button yourself, you will get toast RE: subscribing to email notifications, which will then navigate you to the second page.
the log path variable isn't being set. I tried setting the log_path in settings too, but not sure what the issue is.
Funnel runs
$ funnel
ConfigError: 'log_path' has an invalid value
Glassdoor is missing from the search results.
Seems like only data from the last call in run.py will ever make it into the master list CSV.
addressing this in branch: https://github.com/PaulMcInnis/JobPy/tree/fix_job_providers
Perhaps we should default the configuration to a shallower scrape?
Hi, is this tool compatible with european versions of those sites? (monster.co.uk, etc)
Hey everyone, I was gonna cut us a new release, but I noticed an issue:
Currently you get blurbs like below for all jobs scraped from Monster with GlassdoorStatic scraper:
.css-1noe2rc *{color:#505863;line-height:1.4em;}.css-1noe2rc .ecgq1xb1{padding-left:0;}.css-1noe2rc .ecgq1xb1 .ecgq1xb0{margin:0 0 8px 0;}.css-1noe2rc ol,.css-1noe2rc ul{padding-left:32px;}.css-1noe2rc li{margin:10px;margin-bottom:5px;margin-left:20px;line-height:1.4em;}.css-58vpdc{margin-bottom:24px;}.css-58vpdc ul{margin:5px 0 10px 20px;}.css-58vpdc ul > br{display:none;}.css-58vpdc ul > li{margin-left:0;}.css-58vpdc li{padding:0;}PlayStation isn't just the Best Place to Play it's also the Best Place to Work. We've thrilled gamers since 1994, when we launched the original ...
Since glassdoor runs first most of the duplicates are in other job sites and as a result most of the jobs I scrape now have blurbs like the one above.
I was just checking out GlassDoorDynamic and it seems to work well but it misses the date and blurb fields for jobs. As a side note, watching the browser windows go by made me feel like I was in the matrix 😎
Perhaps it is easier for us to purge some of the CSS from these blurbs in the GlassDoorStatic scraper with a regex for longest string in the raw scrape? Open to suggestions.
Alternatively we could just:
Easily replicable on the stock YAMl on current master with command funnel -kw Engineer, resulting ./search/masterlist.csv will contain aforementioned results.
blurb should not contain CSS
blurb contains CSS.
b30b28453a0f3528095166d0cbbe871726929b64If I need to search for jobs in New-Zealand, it won't work because url formation always adds domain as prefix.
URL formation is developed in indeed.py
It will be great if it provides support for URLs which have domain in prefix.
First of, great idea which could definitely be useful for a lot of people!
Trying to use the application does raise a few questions for me though. From the README and the demo it is unclear to me what type of search terms I can use. The demo provides the province, city, domain and radius for the region search term.
To be concrete:
domain mean?funnel : The term 'funnel' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of
the name, or if a path was included, verify that the path is correct and try again.
At line:1 char:1
+ CategoryInfo : ObjectNotFound: (funnel:String) [], CommandNotFoundException
+ FullyQualifiedErrorId : CommandNotFoundException
A cost of releasing version 3.0.0 is a signifcant loss of test coverage.
git clone https://github.com/PaulMcInnis/JobFunnel
cd JobFunnel
pytest
We should cover Jobfunnel and scraper get/set methods with unit testing via pyest. Existing codecov was set to around 60%
I've added Notes for all the modules that need testing coverage in a kanban here: https://github.com/PaulMcInnis/JobFunnel/projects/2
Please keep this up-to-date so that we don't duplicate eachother's work on upping the test coverage 👍
Ran
$ funnel -s /home/danny/JobFunnel/jobfunnel/config/settings.yaml
and got
jobfunnel initialized at 2020-01-05
jobfunnel indeed to pickle running @ 2020-01-05
Starting new HTTP connection (1): www.indeed.ie
http://www.indeed.ie:80 "GET /jobs?q=security&l=xxxx%2C+None&radius=25&limit=50&filter=0 HTTP/1.1" 301 366
Starting new HTTP connection (1): ie.indeed.com
http://ie.indeed.com:80 "GET /jobs?q=security&l=xxxx%2C+None&radius=25&limit=50&filter=0 HTTP/1.1" 301 None
http://ie.indeed.com:80 "GET /jobs?q=security&l=xxxx,+None&radius=25&limit=50&filter=0 HTTP/1.1" 301 0
Starting new HTTPS connection (1): ie.indeed.com
https://ie.indeed.com:443 "GET /jobs?q=security&l=xxxx,+None&radius=25&limit=50&filter=0 HTTP/1.1" 200 None
failed to scrape Indeed: 'NoneType' object has no attribute 'contents'
....
When I run any funnel command (--help, -s) I get the following errors:
Traceback (most recent call last):
File "/home/pi/.local/bin/funnel", line 6, in
from jobfunnel.main import main
File "/home/pi/.local/lib/python3.7/site-packages/jobfunnel/main.py", line 11, in
from .jobfunnel import JobFunnel
File "/home/pi/.local/lib/python3.7/site-packages/jobfunnel/jobfunnel.py", line 20, in
from .tools.delay import delay_alg
File "/home/pi/.local/lib/python3.7/site-packages/jobfunnel/tools/delay.py", line 9, in
from scipy.special import expit
File "/home/pi/.local/lib/python3.7/site-packages/scipy/special/init.py", line 633, in
from . import _ufuncs
ImportError: libf77blas.so.3: cannot open shared object file: No such file or directory
Raspbian OS
Raspberry Pi 4
Python 2 has been deprecated for some time now, and it's end of life is 2020. We should NOT support it.
Proxy support would be nice if you were somewhere that requires them.
E.g. in indeed.py
http_proxy = "http.foo.com:8000" https_proxy = "http.foo.com:8000" proxyDict = { "http" : http_proxy, "https" : https_proxy } request_HTML = get(search, headers=self.headers, proxies=proxyDict)
For example on indeed when you want to search for an exact phrase (multiple words) as keyword you put this phrase between double quotes.
When I want to use this feature on funnel it removes the double quotes and it returns wrong results.
Normally when you write this keywords on indeed website this is the URL that is generated:
https://www.indeed.com/jobs?q=%22data+distribution+service%22&l=Saratoga%2C+CA&radius=25
But funnel generates this url:
getting indeed page 0 : http://www.indeed.com/jobs?q=Data Distribution Service&l=Saratoga%2C+CA&radius=25&limit=50&filter=0&start=0
*Windows 10 Home
Having a requirements.txt file is helpful for creating virtual environments.
I get an error when running funnel, an exception gets caugh then scraping Indeed and then moves on to Monster...
$ funnel -s JobFunnel/jobfunnel/config/settings.yaml
jobfunnel initialized at 2020-01-05
jobfunnel indeed to pickle running @ 2020-01-05
Starting new HTTP connection (1): www.indeed.ie
http://www.indeed.ie:80 "GET /jobs?q=security&l=xxxx%2C+&radius=25&limit=50&filter=0 HTTP/1.1" 301 362
Starting new HTTP connection (1): ie.indeed.com
http://ie.indeed.com:80 "GET /jobs?q=security&l=xxxx%2C+&radius=25&limit=50&filter=0 HTTP/1.1" 301 None
http://ie.indeed.com:80 "GET /jobs?q=security&l=xxxx,&radius=25&limit=50&filter=0 HTTP/1.1" 301 0
Starting new HTTPS connection (1): ie.indeed.com
https://ie.indeed.com:443 "GET /jobs?q=security&l=xxxx,&radius=25&limit=50&filter=0 HTTP/1.1" 200 None
Found 291 indeed results for query=security
getting indeed page 0 : http://www.indeed.ie/jobs?q=security&l=xxxx%2C+&radius=25&limit=50&filter=0&start=0
getting indeed page 1 : http://www.indeed.ie/jobs?q=security&l=xxxx%2C+&radius=25&limit=50&filter=0&start=50
Starting new HTTP connection (1): www.indeed.ie
Starting new HTTP connection (1): www.indeed.ie
getting indeed page 2 : http://www.indeed.ie/jobs?q=security&l=xxxx%2C+&radius=25&limit=50&filter=0&start=100
Starting new HTTP connection (1): www.indeed.ie
getting indeed page 3 : http://www.indeed.ie/jobs?q=security&l=xxxx%2C+&radius=25&limit=50&filter=0&start=150
Starting new HTTP connection (1): www.indeed.ie
getting indeed page 4 : http://www.indeed.ie/jobs?q=security&l=xxxx%2C+&radius=25&limit=50&filter=0&start=200
Starting new HTTP connection (1): www.indeed.ie
getting indeed page 5 : http://www.indeed.ie/jobs?q=security&l=xxxx%2C+&radius=25&limit=50&filter=0&start=250
Starting new HTTP connection (1): www.indeed.ie
http://www.indeed.ie:80 "GET /jobs?q=security&l=xxxx%2C+&radius=25&limit=50&filter=0&start=50 HTTP/1.1" 301 375
http://www.indeed.ie:80 "GET /jobs?q=security&l=xxxx%2C+&radius=25&limit=50&filter=0&start=0 HTTP/1.1" 301 374
http://www.indeed.ie:80 "GET /jobs?q=security&l=xxxx%2C+&radius=25&limit=50&filter=0&start=200 HTTP/1.1" 301 376
http://www.indeed.ie:80 "GET /jobs?q=security&l=xxxx%2C+&radius=25&limit=50&filter=0&start=150 HTTP/1.1" 301 376
Starting new HTTP connection (1): ie.indeed.com
http://www.indeed.ie:80 "GET /jobs?q=security&l=xxxx%2C+&radius=25&limit=50&filter=0&start=100 HTTP/1.1" 301 376
Starting new HTTP connection (1): ie.indeed.com
Starting new HTTP connection (1): ie.indeed.com
Starting new HTTP connection (1): ie.indeed.com
http://www.indeed.ie:80 "GET /jobs?q=security&l=xxxx%2C+&radius=25&limit=50&filter=0&start=250 HTTP/1.1" 301 376
Starting new HTTP connection (1): ie.indeed.com
Starting new HTTP connection (1): ie.indeed.com
http://ie.indeed.com:80 "GET /jobs?q=security&l=xxxx%2C+&radius=25&limit=50&filter=0&start=0 HTTP/1.1" 301 None
http://ie.indeed.com:80 "GET /jobs?q=security&l=xxxx%2C+&radius=25&limit=50&filter=0&start=250 HTTP/1.1" 301 None
http://ie.indeed.com:80 "GET /jobs?q=security&l=xxxx%2C+&radius=25&limit=50&filter=0&start=50 HTTP/1.1" 301 None
http://ie.indeed.com:80 "GET /jobs?q=security&l=xxxx%2C+&radius=25&limit=50&filter=0&start=150 HTTP/1.1" 301 None
http://ie.indeed.com:80 "GET /jobs?q=security&l=xxxx%2C+&radius=25&limit=50&filter=0&start=100 HTTP/1.1" 301 None
http://ie.indeed.com:80 "GET /jobs?q=security&l=xxxx%2C+&radius=25&limit=50&filter=0&start=200 HTTP/1.1" 301 None
http://ie.indeed.com:80 "GET /jobs?q=security&l=xxxx,&radius=25&limit=50&filter=0&start=0 HTTP/1.1" 301 0
http://ie.indeed.com:80 "GET /jobs?q=security&l=xxxx,&radius=25&limit=50&filter=0&start=250 HTTP/1.1" 301 0
http://ie.indeed.com:80 "GET /jobs?q=security&l=xxxx,&radius=25&limit=50&filter=0&start=50 HTTP/1.1" 301 0
Starting new HTTPS connection (1): ie.indeed.com
http://ie.indeed.com:80 "GET /jobs?q=security&l=xxxx,&radius=25&limit=50&filter=0&start=100 HTTP/1.1" 301 0
Starting new HTTPS connection (1): ie.indeed.com
http://ie.indeed.com:80 "GET /jobs?q=security&l=xxxx,&radius=25&limit=50&filter=0&start=150 HTTP/1.1" 301 0
http://ie.indeed.com:80 "GET /jobs?q=security&l=xxxx,&radius=25&limit=50&filter=0&start=200 HTTP/1.1" 301 0
Starting new HTTPS connection (1): ie.indeed.com
Starting new HTTPS connection (1): ie.indeed.com
Starting new HTTPS connection (1): ie.indeed.com
Starting new HTTPS connection (1): ie.indeed.com
https://ie.indeed.com:443 "GET /jobs?q=security&l=xxxx,&radius=25&limit=50&filter=0&start=250 HTTP/1.1" 200 None
https://ie.indeed.com:443 "GET /jobs?q=security&l=xxxx,&radius=25&limit=50&filter=0&start=0 HTTP/1.1" 200 None
https://ie.indeed.com:443 "GET /jobs?q=security&l=xxxx,&radius=25&limit=50&filter=0&start=100 HTTP/1.1" 200 None
https://ie.indeed.com:443 "GET /jobs?q=security&l=xxxx,&radius=25&limit=50&filter=0&start=150 HTTP/1.1" 200 None
https://ie.indeed.com:443 "GET /jobs?q=security&l=xxxx,&radius=25&limit=50&filter=0&start=50 HTTP/1.1" 200 None
https://ie.indeed.com:443 "GET /jobs?q=security&l=xxxx,&radius=25&limit=50&filter=0&start=200 HTTP/1.1" 200 None
failed to scrape Indeed: invalid literal for int() with base 10: 'Just poste'
jobfunnel monster to pickle running @ : 2020-01-05
Starting new HTTPS connection (1): www.monster.ie
https://www.monster.ie:443 "GET /jobs/search/?q=security&whe
Notes that I've changed the location to xxxx for posting purposes.
The settings file looks like this
# This is the default settings file. Do not edit.
# All paths are relative to this file.
# Paths.
output_path: 'search'
# Providers from which to search (case insensitive)
providers:
- 'Indeed'
- 'Monster'
- 'GlassDoor' # This takes ~10x longer to run than the other providers
# Filters.
search_terms:
region:
province: ''
city: 'xxxx'
domain: 'ie'
radius: 25
keywords:
- 'security'
# Black-listed company names
black_list:
- 'yyyyyyyyyy'
# Logging level options are: critical, error, warning, info, debug, notset
log_level: 'debug'
~
pauls-mbp $ funnel --no_scrape
jobfunnel initialized at 2019-07-07
Traceback (most recent call last):
File "/usr/local/bin/funnel", line 11, in <module>
load_entry_point('JobFunnel', 'console_scripts', 'funnel')()
File "/Users/paulmcinnis/JobFunnel/jobfunnel/__main__.py", line 30, in main
jp.load_pickle()
File "/Users/paulmcinnis/JobFunnel/jobfunnel/jobfunnel.py", line 62, in load_pickle
pickle_filepath = os.path.join(args['data_path'], 'scraped',
NameError: name 'args' is not defined
pauls-mbp $
I just successfully installed it via:
pip install git+https://github.com/PaulMcInnis/JobFunnel.git
However, when I entered funnel --help it gave me the error:
AttributeError: module 'os' has no attribute 'add_dll_directory'
Description
It would be really nice to have some kind of web-app to use jobfunnel with, perhaps even just to provide a good demo experience.
i.e. https://pages.github.com/
Describe the solution you'd like
Ideally the user could run jobfunnel in-browser and review results with a simple web-app.
Describe alternatives you've considered
It may be more desirable to make something that runs on a user's local machine, limitations of github pages could prove a blocker.
I guess at some points dicts get converted to .csv so maybe we could have a --json flag which exports the jobs in JSON format instead of CSV.
I seem to have botched this in the release... oops!
Everything seems to be working fine for the demo scrape, but suddenly you get no jobs added because we update it to be Nothing!
JobFunnel/jobfunnel/backend/jobfunnel.py
Line 244 in 25ce96e
funnel load -s demo/settings.yaml
We should be updating to accumulate incoming jobs.
We never accumulate anything! but we get nice progress bars (and scrape goes fine)
I attempted to adjust the 'providers' in the settings.yaml, but I found a few that raised errors. The following would be great additions to impact the tool:
Google Chrome Driver not defined
Replace line 158 of jobfunnel/tools/tools.py
webdriver.Chrome(ChromeDriverManager().install())
with
driver = webdriver.Chrome(ChromeDriverManager().install())
so function get_webdriver() has a value to return.
Why not a pull request? Too small of a fix
Standard search produces web scrape error
Standard search with
Results of query
Traceback (most recent call last):
File "C:\Users\phcre\AppData\Local\Programs\Python\Python38\Scripts\funnel-script.py", line 11, in <module>
load_entry_point('JobFunnel', 'console_scripts', 'funnel')()
File "c:\users\phcre\documents\jobs\jobfunnel\jobfunnel\__main__.py", line 55, in main
jf.update_masterlist()
File "c:\users\phcre\documents\jobs\jobfunnel\jobfunnel\jobfunnel.py", line 330, in update_masterlist
tfidf_filter(self.scrape_data, masterlist)
File "c:\users\phcre\documents\jobs\jobfunnel\jobfunnel\tools\filters.py", line 118, in tfidf_filter
duplicate_ids = tfidf_filter(cur_dict)
File "c:\users\phcre\documents\jobs\jobfunnel\jobfunnel\tools\filters.py", line 90, in tfidf_filter
similarities = cosine_similarity(vectorizer.fit_transform(query_words))
File "c:\users\phcre\appdata\local\programs\python\python38\lib\site-packages\sklearn\feature_extraction\text.py", line 1840, in fit_transform
X = super().fit_transform(raw_documents)
File "c:\users\phcre\appdata\local\programs\python\python38\lib\site-packages\sklearn\feature_extraction\text.py", line 1198, in fit_transform
vocabulary, X = self._count_vocab(raw_documents,
File "c:\users\phcre\appdata\local\programs\python\python38\lib\site-packages\sklearn\feature_extraction\text.py", line 1129, in _count_vocab
raise ValueError("empty vocabulary; perhaps the documents only"
ValueError: empty vocabulary; perhaps the documents only contain stop words
query_words is empty therefore cannot be fit_transform by vectorizer
GET http://127.0.0.1:50081/session/02b7e485dd5ae5ae4fb5c16bf406267a/source {}
http://127.0.0.1:50081 "GET /session/02b7e485dd5ae5ae4fb5c16bf406267a/source HTTP/1.1" 200 381722
Finished Request
Found 8 glassdoor results for query=Advertising-Marketing-Coordinator-Account-Agency
GET http://127.0.0.1:50081/session/02b7e485dd5ae5ae4fb5c16bf406267a/url {}
http://127.0.0.1:50081 "GET /session/02b7e485dd5ae5ae4fb5c16bf406267a/url HTTP/1.1" 200 144
Finished Request
getting glassdoor page 1 : https://www.glassdoor.com/Job/allen-advertising-marketing-coordinator-account-agency-jobs-SRCH_IL.0,5_IC1139946_KE6,54.htm?radius=25
POST http://127.0.0.1:50081/session/02b7e485dd5ae5ae4fb5c16bf406267a/url {"url": "https://www.glassdoor.com/Job/allen-advertising-marketing-coordinator-account-agency-jobs-SRCH_IL.0,5_IC1139946_KE6,54.htm?radius=25"}
http://127.0.0.1:50081 "POST /session/02b7e485dd5ae5ae4fb5c16bf406267a/url HTTP/1.1" 200 14
Finished Request
GET http://127.0.0.1:50081/session/02b7e485dd5ae5ae4fb5c16bf406267a/source {}
http://127.0.0.1:50081 "GET /session/02b7e485dd5ae5ae4fb5c16bf406267a/source HTTP/1.1" 200 381666
Finished Request
DELETE http://127.0.0.1:50081/session/02b7e485dd5ae5ae4fb5c16bf406267a/window {}
http://127.0.0.1:50081 "DELETE /session/02b7e485dd5ae5ae4fb5c16bf406267a/window HTTP/1.1" 200 14
Finished Request
found 8 unique job ids and 0 duplicates from glassdoor
removed 0 jobs present in filter-list
removed 0 jobs in blacklist from master-list
Calculating delay...
Done! Starting scrape!
delay of 0.00s, getting glassdoor search: https://www.glassdoor.com/partner/jobListing.htm?pos=101&ao=68087&s=58&guid=00000172a6913e06ac92fffcddc5bb23&src=GD_JOB_AD&t=SR&extid=1&exst=EL&ist=&ast=EL&slr=true&cs=1_c26c12d6&cb=1591932436271&jobListingId=3596513699
POST http://127.0.0.1:50081/session/02b7e485dd5ae5ae4fb5c16bf406267a/url {"url": "https://www.glassdoor.com/partner/jobListing.htm?pos=101&ao=68087&s=58&guid=00000172a6913e06ac92fffcddc5bb23&src=GD_JOB_AD&t=SR&extid=1&exst=EL&ist=&ast=EL&slr=true&cs=1_c26c12d6&cb=1591932436271&jobListingId=3596513699"}
http://127.0.0.1:50081 "POST /session/02b7e485dd5ae5ae4fb5c16bf406267a/url HTTP/1.1" 404 770
Finished Request
delay of 22.19s, getting glassdoor search: https://www.glassdoor.com/partner/jobListing.htm?pos=102&ao=85058&s=58&guid=00000172a6913e06ac92fffcddc5bb23&src=GD_JOB_AD&t=SR&extid=1&exst=EL&ist=&ast=EL&slr=true&cs=1_4b2ba71c&cb=1591932436271&jobListingId=3593859227
POST http://127.0.0.1:50081/session/02b7e485dd5ae5ae4fb5c16bf406267a/url {"url": "https://www.glassdoor.com/partner/jobListing.htm?pos=102&ao=85058&s=58&guid=00000172a6913e06ac92fffcddc5bb23&src=GD_JOB_AD&t=SR&extid=1&exst=EL&ist=&ast=EL&slr=true&cs=1_4b2ba71c&cb=1591932436271&jobListingId=3593859227"}
http://127.0.0.1:50081 "POST /session/02b7e485dd5ae5ae4fb5c16bf406267a/url HTTP/1.1" 404 899
Finished Request
delay of 22.34s, getting glassdoor search: https://www.glassdoor.com/partner/jobListing.htm?pos=103&ao=58033&s=58&guid=00000172a6913e06ac92fffcddc5bb23&src=GD_JOB_AD&t=SR&extid=1&exst=EL&ist=&ast=EL&slr=true&cs=1_9dd170bc&cb=1591932436271&jobListingId=3319079566
POST http://127.0.0.1:50081/session/02b7e485dd5ae5ae4fb5c16bf406267a/url {"url": "https://www.glassdoor.com/partner/jobListing.htm?pos=103&ao=58033&s=58&guid=00000172a6913e06ac92fffcddc5bb23&src=GD_JOB_AD&t=SR&extid=1&exst=EL&ist=&ast=EL&slr=true&cs=1_9dd170bc&cb=1591932436271&jobListingId=3319079566"}
http://127.0.0.1:50081 "POST /session/02b7e485dd5ae5ae4fb5c16bf406267a/url HTTP/1.1" 404 899
Finished Request
delay of 24.76s, getting glassdoor search: https://www.glassdoor.com/partner/jobListing.htm?pos=104&ao=926135&s=58&guid=00000172a6913e06ac92fffcddc5bb23&src=GD_JOB_AD&t=SR&extid=1&exst=EL&ist=&ast=EL&slr=true&cs=1_3f575b87&cb=1591932436271&jobListingId=3582441465
POST http://127.0.0.1:50081/session/02b7e485dd5ae5ae4fb5c16bf406267a/url {"url": "https://www.glassdoor.com/partner/jobListing.htm?pos=104&ao=926135&s=58&guid=00000172a6913e06ac92fffcddc5bb23&src=GD_JOB_AD&t=SR&extid=1&exst=EL&ist=&ast=EL&slr=true&cs=1_3f575b87&cb=1591932436271&jobListingId=3582441465"}
http://127.0.0.1:50081 "POST /session/02b7e485dd5ae5ae4fb5c16bf406267a/url HTTP/1.1" 404 899
Finished Request
delay of 27.24s, getting glassdoor search: https://www.glassdoor.com/partner/jobListing.htm?pos=105&ao=85058&s=58&guid=00000172a6913e06ac92fffcddc5bb23&src=GD_JOB_AD&t=SR&extid=1&exst=EL&ist=&ast=EL&slr=true&cs=1_98a55e04&cb=1591932436271&jobListingId=3584976096
POST http://127.0.0.1:50081/session/02b7e485dd5ae5ae4fb5c16bf406267a/url {"url": "https://www.glassdoor.com/partner/jobListing.htm?pos=105&ao=85058&s=58&guid=00000172a6913e06ac92fffcddc5bb23&src=GD_JOB_AD&t=SR&extid=1&exst=EL&ist=&ast=EL&slr=true&cs=1_98a55e04&cb=1591932436271&jobListingId=3584976096"}
http://127.0.0.1:50081 "POST /session/02b7e485dd5ae5ae4fb5c16bf406267a/url HTTP/1.1" 404 899
Finished Request
delay of 29.04s, getting glassdoor search: https://www.glassdoor.com/partner/jobListing.htm?pos=106&ao=85058&s=58&guid=00000172a6913e06ac92fffcddc5bb23&src=GD_JOB_AD&t=SR&extid=1&exst=EL&ist=&ast=EL&slr=true&cs=1_ca4062d5&cb=1591932436271&jobListingId=3579768726
POST http://127.0.0.1:50081/session/02b7e485dd5ae5ae4fb5c16bf406267a/url {"url": "https://www.glassdoor.com/partner/jobListing.htm?pos=106&ao=85058&s=58&guid=00000172a6913e06ac92fffcddc5bb23&src=GD_JOB_AD&t=SR&extid=1&exst=EL&ist=&ast=EL&slr=true&cs=1_ca4062d5&cb=1591932436271&jobListingId=3579768726"}
http://127.0.0.1:50081 "POST /session/02b7e485dd5ae5ae4fb5c16bf406267a/url HTTP/1.1" 404 899
Finished Request
delay of 29.64s, getting glassdoor search: https://www.glassdoor.com/partner/jobListing.htm?pos=107&ao=60939&s=58&guid=00000172a6913e06ac92fffcddc5bb23&src=GD_JOB_AD&t=SR&extid=1&exst=EL&ist=&ast=EL&slr=true&cs=1_64a152f4&cb=1591932436271&jobListingId=3504589748
POST http://127.0.0.1:50081/session/02b7e485dd5ae5ae4fb5c16bf406267a/url {"url": "https://www.glassdoor.com/partner/jobListing.htm?pos=107&ao=60939&s=58&guid=00000172a6913e06ac92fffcddc5bb23&src=GD_JOB_AD&t=SR&extid=1&exst=EL&ist=&ast=EL&slr=true&cs=1_64a152f4&cb=1591932436271&jobListingId=3504589748"}
http://127.0.0.1:50081 "POST /session/02b7e485dd5ae5ae4fb5c16bf406267a/url HTTP/1.1" 404 899
Finished Request
delay of 18.15s, getting glassdoor search: https://www.glassdoor.com/partner/jobListing.htm?pos=108&ao=60939&s=58&guid=00000172a6913e06ac92fffcddc5bb23&src=GD_JOB_AD&t=SR&extid=1&exst=EL&ist=&ast=EL&slr=true&cs=1_81ad2932&cb=1591932436272&jobListingId=3543437733
POST http://127.0.0.1:50081/session/02b7e485dd5ae5ae4fb5c16bf406267a/url {"url": "https://www.glassdoor.com/partner/jobListing.htm?pos=108&ao=60939&s=58&guid=00000172a6913e06ac92fffcddc5bb23&src=GD_JOB_AD&t=SR&extid=1&exst=EL&ist=&ast=EL&slr=true&cs=1_81ad2932&cb=1591932436272&jobListingId=3543437733"}
http://127.0.0.1:50081 "POST /session/02b7e485dd5ae5ae4fb5c16bf406267a/url HTTP/1.1" 404 899
Finished Request
glassdoor scrape job took 173.619s
removed 0 jobs present in filter-list
removed 0 jobs in blacklist from master-list
removed 0 jobs present in filter-list
removed 0 jobs in blacklist from master-list
Traceback (most recent call last):
File "C:\Users\phcre\AppData\Local\Programs\Python\Python38\Scripts\funnel-script.py", line 11, in <module>
load_entry_point('JobFunnel', 'console_scripts', 'funnel')()
File "c:\users\asdf\documents\jobs\jobfunnel\jobfunnel\__main__.py", line 55, in main
jf.update_masterlist()
File "c:\users\asdf\documents\jobs\jobfunnel\jobfunnel\jobfunnel.py", line 330, in update_masterlist
tfidf_filter(self.scrape_data, masterlist)
File "c:\users\asdf\documents\jobs\jobfunnel\jobfunnel\tools\filters.py", line 118, in tfidf_filter
duplicate_ids = tfidf_filter(cur_dict)
File "c:\users\asdf\documents\jobs\jobfunnel\jobfunnel\tools\filters.py", line 90, in tfidf_filter
similarities = cosine_similarity(vectorizer.fit_transform(query_words))
File "c:\users\asdf\appdata\local\programs\python\python38\lib\site-packages\sklearn\feature_extraction\text.py", line 1840, in fit_transform
X = super().fit_transform(raw_documents)
File "c:\users\asdf\appdata\local\programs\python\python38\lib\site-packages\sklearn\feature_extraction\text.py", line 1198, in fit_transform
vocabulary, X = self._count_vocab(raw_documents,
File "c:\users\asdf\appdata\local\programs\python\python38\lib\site-packages\sklearn\feature_extraction\text.py", line 1129, in _count_vocab
raise ValueError("empty vocabulary; perhaps the documents only"
ValueError: empty vocabulary; perhaps the documents only contain stop words
webdriver manager returning 404 errors?
prev_dictNone
cur_dict.values()odict_values([{'status': 'new', 'title': 'Account Manager Digital Marketing - Professional Services - Entertainment and Media Industry Opportunity', 'company': 'Gannett', 'location': 'Plano, TX', 'date': '', 'blurb': '', 'tags': '', 'link': 'https://www.glassdoor.com/partner/jobListing.htm?pos=101&ao=68087&s=58&guid=00000172a6913e06ac92fffcddc5bb23&src=GD_JOB_AD&t=SR&extid=1&exst=EL&ist=&ast=EL&slr=true&cs=1_c26c12d6&cb=1591932436271&jobListingId=3596513699', 'id': '3596513699', 'provider': 'glassdoor', 'query': 'Advertising-Marketing-Coordinator-Account-Agency'}, {'status': 'new', 'title': 'Account Coordinator - Marketing', 'company': 'The Point Group', 'location': 'Dallas, TX', 'date': '', 'blurb': '', 'tags': '', 'link': 'https://www.glassdoor.com/partner/jobListing.htm?pos=102&ao=85058&s=58&guid=00000172a6913e06ac92fffcddc5bb23&src=GD_JOB_AD&t=SR&extid=1&exst=EL&ist=&ast=EL&slr=true&cs=1_4b2ba71c&cb=1591932436271&jobListingId=3593859227', 'id': '3593859227', 'provider': 'glassdoor', 'query': 'Advertising-Marketing-Coordinator-Account-Agency'}, {'status': 'new', 'title': 'Marketing Coordinator', 'company': 'Gourmet Marketing LLC', 'location': 'Plano, TX', 'date': '', 'blurb': '', 'tags': '', 'link': 'https://www.glassdoor.com/partner/jobListing.htm?pos=103&ao=58033&s=58&guid=00000172a6913e06ac92fffcddc5bb23&src=GD_JOB_AD&t=SR&extid=1&exst=EL&ist=&ast=EL&slr=true&cs=1_9dd170bc&cb=1591932436271&jobListingId=3319079566', 'id': '3319079566', 'provider': 'glassdoor', 'query': 'Advertising-Marketing-Coordinator-Account-Agency'}, {'status': 'new', 'title': 'Account Coordinator - Client Service', 'company': 'RKD Group, Inc.', 'location': 'Richardson, TX', 'date': '', 'blurb': '', 'tags': '', 'link': 'https://www.glassdoor.com/partner/jobListing.htm?pos=104&ao=926135&s=58&guid=00000172a6913e06ac92fffcddc5bb23&src=GD_JOB_AD&t=SR&extid=1&exst=EL&ist=&ast=EL&slr=true&cs=1_3f575b87&cb=1591932436271&jobListingId=3582441465', 'id': '3582441465', 'provider': 'glassdoor', 'query': 'Advertising-Marketing-Coordinator-Account-Agency'}, {'status': 'new', 'title': 'COLLEGE GRADS & INTERNS - Entry Level Marketing & Advertising', 'company': 'Millennium Events Management', 'location': 'Dallas, TX', 'date': '', 'blurb': '', 'tags': '', 'link': 'https://www.glassdoor.com/partner/jobListing.htm?pos=105&ao=85058&s=58&guid=00000172a6913e06ac92fffcddc5bb23&src=GD_JOB_AD&t=SR&extid=1&exst=EL&ist=&ast=EL&slr=true&cs=1_98a55e04&cb=1591932436271&jobListingId=3584976096', 'id': '3584976096', 'provider': 'glassdoor', 'query': 'Advertising-Marketing-Coordinator-Account-Agency'}, {'status': 'new', 'title': 'Senior Account Executive (Marketing/Advertising)', 'company': 'The Point Group', 'location': 'Dallas, TX', 'date': '', 'blurb': '', 'tags': '', 'link': 'https://www.glassdoor.com/partner/jobListing.htm?pos=106&ao=85058&s=58&guid=00000172a6913e06ac92fffcddc5bb23&src=GD_JOB_AD&t=SR&extid=1&exst=EL&ist=&ast=EL&slr=true&cs=1_ca4062d5&cb=1591932436271&jobListingId=3579768726', 'id': '3579768726', 'provider': 'glassdoor', 'query': 'Advertising-Marketing-Coordinator-Account-Agency'}, {'status': 'new', 'title': 'Account Coordinator - Client Service', 'company': 'RKD Group', 'location': 'Richardson, TX', 'date': '', 'blurb': '', 'tags': '', 'link': 'https://www.glassdoor.com/partner/jobListing.htm?pos=107&ao=60939&s=58&guid=00000172a6913e06ac92fffcddc5bb23&src=GD_JOB_AD&t=SR&extid=1&exst=EL&ist=&ast=EL&slr=true&cs=1_64a152f4&cb=1591932436271&jobListingId=3504589748', 'id': '3504589748', 'provider': 'glassdoor', 'query': 'Advertising-Marketing-Coordinator-Account-Agency'}, {'status': 'new', 'title': 'Digital Account Coordinator', 'company': 'RKD Group', 'location': 'Richardson, TX', 'date': '', 'blurb': '', 'tags': '', 'link': 'https://www.glassdoor.com/partner/jobListing.htm?pos=108&ao=60939&s=58&guid=00000172a6913e06ac92fffcddc5bb23&src=GD_JOB_AD&t=SR&extid=1&exst=EL&ist=&ast=EL&slr=true&cs=1_81ad2932&cb=1591932436272&jobListingId=3543437733', 'id': '3543437733', 'provider': 'glassdoor', 'query': 'Advertising-Marketing-Coordinator-Account-Agency'}])
query_ids['3596513699', '3593859227', '3319079566', '3582441465', '3584976096', '3579768726', '3504589748', '3543437733']
query_words['', '', '', '', '', '', '', '']
beautifulsoup4>=4.6.3 (4.9.1)
lxml>=4.2.4 (4.5.1)
requests>=2.19.1 (2.23.0)
python-dateutil>=2.8.0 (2.8.1)
PyYAML>=5.1 (5.3.1)
scikit-learn>=0.21.2 (0.23.1)
nltk>=3.4.1 (3.5)
scipy>=1.4.1 (1.4.1)
selenium>=3.141.0 (3.141.0)
webdriver-manager>=2.4.0 (3.1.0)
soupsieve>1.2 (2.0.1)
certifi>=2017.4.17 (2020.4.5.2)
urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (1.25.9)
chardet<4,>=3.0.2 (3.0.4)
idna<3,>=2.5 (2.9)
six>=1.5 (1.15.0)
threadpoolctl>=2.0.0 (2.1.0)
joblib>=0.11 (0.15.1)
numpy>=1.13.3 (1.18.5)
click (7.1.2)
tqdm(4.46.1)
atomicwrites>=1.0; (1.4.0)
packaging (20.4)
pluggy<1.0,>=0.12 (0.13.1)
Just today I discovered that when scraping Glassdoor.com, JobFunnel fails.
Please include the steps to reproduce.
List any additional libraries that are affected.
settings.yaml as such: # - 'Indeed'
# - 'Monster'
- 'GlassDoor'
funnel -s settings.yamlScrape Glassdoor.com and store jobs in master_list.csv
JobFunnel output:
jobfunnel initialized at 2020-05-05
no master-list, filter-list was not updated
jobfunnel glassdoor to pickle running @ 2020-05-05
failed to scrape GlassDoor: 'NoneType' object has no attribute 'text'
Traceback (most recent call last):
File "/usr/local/bin/funnel", line 11, in <module>
load_entry_point('JobFunnel==2.1.6', 'console_scripts', 'funnel')()
File "/usr/local/lib/python3.6/dist-packages/jobfunnel/__main__.py", line 55, in main
jf.update_masterlist()
File "/usr/local/lib/python3.6/dist-packages/jobfunnel/jobfunnel.py", line 291, in update_masterlist
raise ValueError('No scraped jobs, cannot update masterlist')
I discovered this while inspecting glassdoor.py for testing. I will try my best to tackle this issue in the upcoming days. Hopefully we'll fix it soon!
Cheers!
We need to manually update the code to make it work if anything related to the html elements changes, right?
Currently we remove duplicates everywhere but we only remove duplicates by description (TFIDF) between the masterlist and all scrape data.
We should allow masterlist to perform a content match to itself.
We should be running TFIDF inter-scrape data and inter-master csv
Only duplicates in the ncoming dict are identified based on master CSV
Hi all,
Installing the latest version of JobFunnel and running funnel --help takes a few seconds when opening the first time. Later calls are slightly faster but are arguable still a tad slow.
I know I could have looked into the cause of the problem right now but I'm really low on time as of lately. If this issue still persists when I have more time I will come back to this (kind of a reminder for myself then).
funnel --helpA few ms delay before showing CLI help information.
Significant delay between function call and presented info.
pip installer fails to install because it can't find pipenv.
Currently none of the installation instructions on the readme work.
Is your feature request related to a problem? Please describe.
We currently only offer CANADA_ENGLISH and USA_ENGLISH scrapers, but we should also implement some european locales
Describe the solution you'd like
Implement some european locales such as UK_ENGLISH, FRANCE_FRENCH
Describe alternatives you've considered
n/a
Additional context
Existing request #45
Currently we are using the JobFunnel class for to much, I want to break it down into the following:
Job(object):
def __init__(self, title: str, company: str, location: str, tags: List[str], post_date: datetime.date, key_id: str, url: str) -> None:
...
Scraper(ABC):
@abstractmethod
def scrape(self) -> List[Job]:
pass
main():
# instantiate scrapers
# run filter on list of Job
# dump pickle
# writeout CSV
Note: if I get to it, I'd also like our filters to be an ABC.
This is a structural technical debt issue. (n/a)
Abstract base class should not be halfway abstract, Need seperation between JobFunnel and main() and inherited scrapers.
JobFunnel being monolithic and half-abstract has allowed us to implement three script-like scrapers which share too many methods, without an actual Job object.
n/a
Job Object
Support for Internationalization
BaseScraper with get/set scraping logic
New YAML and CLI implemented
Schema Validation with Cerberus
Caching
Filtering with lists
Indeed
Monster
GlassDoorStatic (works but seems like it has bugs so fixing this).
Wage Scraping
GlassDoor Dynamic/Driven
Duplicates list file support
Integrate TFIDF similarity filter (special case filter)
Prevent writing out empty CSVs in --no-scrape mode
Prevent delayed get/set for jobs which fail filters
Fix multi-page Monster scraping
handle duplicated jobs special case
Make JobFilter class
Add TAG scraping to Monster
Implement job filtering as own class
Fix paths from -s yaml being overwritten with defaults with CLI
Fix concurrency issue with dependencies for get/set
Monkey / general usability testing
Update main README
Update other READMEs + tutorials
Add versioning to cache files (i.e wrapper for dict with metadata)
Review various FIXMES in-code
Fix build (Travis CI)
Test setup.py
Fix demo GIF
Document how to write new scrapers with localization
We currently pass keywords verbatim, even if they include “+” or other special characters, we should be encoding these so as not to disrupt the url formation.
We would get a query containing “%” encoded values in place of +s
We build a search string with erroneous + chars that breaks the search query url (query is not as intended)
Is your feature request related to a problem? Please describe.
Currently we do not scrape Remote field for Indeed or Monster, which are providing many jobs.
Indeed provides this fairly visibly, it's just a bit buried in tags.
Glassdoor has this implemented already (tags as well)
Monster does not seem to have this capability, but it may be worth looking into if we can infer it.
Describe the solution you'd like
We should scrape Remote = True if a job is fully-remote on Indeed. Temporarily remote doesn't count, sorry!
Describe alternatives you've considered
n/a
Additional context
When changing the status to applied and rerunning JobFunnel master_list.csv has no fields!
Yikes!
Every time running JubFunnel thereafter will result in an empty list of jobs.
If you delete master_list.csv a new list will be created next time running JobFunnel but the previous jobs are gone.
We need two things:
master_list.csv in case this ever happens again.I think by default I will have the software search for existing pickles by changing:
load_pickle(self, args) to load_pickles(self, args).
This way, all pickles are important and not just today's pickle.
If the user provides an invalid configuration file, then the program will throw unclear errors. I think it's good practice to check all settings of the config dictionary and show a helpful error message in case the config is invalid.
settings.yaml)funnel -s settings.yaml
I am opening this issue because I think we need a folder pycharm for debugging this project.
PyCharm has become a very popular IDE and can really help with debugging this project.
If we can make a folder that gets developers up and running in debug mode in just a few simple steps then I think it is worth doing.
Main benefits of using PyCharm for developing this project:
Since a few weeks, testing has been introduced to the JobFunnel software but it only covers a fraction of the entire code base. I think increasing the quality of the testing framework with unit and integration tests can make it easier for the reviewers to assess new pull requests and gives new contributors more feedback whether their changes to the code base does not break functionality.
I came up with two ideas to provide a clear and productive environment to increase test coverage:
Note that checking whether a function is 'covered' by tests can be a bit tricky because some functions are rather long, e.g.parse_config. Hence, true unit tests are very difficult for these long functions but can be easily implemented for shorter functions. Realising that unit tests are complicated for certain functions might also be a sign that these functions should be broken into smaller parts
glassdoor.pyconvert_radiusget_search_urlsearch_page_for_job_soupssearch_joblink_for_blurbparse_blurbscrapeindeed.pyconvert_radiusget_search_urlsearch_page_for_job_soupssearch_joblink_for_blurbget_blurb_with_delayparse_blurbscrapemonster.pyconvert_radiusget_search_urlsearch_page_for_job_soupssearch_joblink_for_blurbget_blurb_with_delayparse_blurbscrapejobfunnel.pyinit_loggingload_pickleload_picklesdump_pickleread_csvwrite_csvremove_jobs_in_filterlistremove_blacklisted_companiesupdate_filterjsonpre_filterdelay_threaderupdate_masterlistdelay.py_c_delay_lin_delay_sig_delaydelay_algfilters.pyid_filtertfidf_filtertools.pyfilter_non_printablespost_date_from_relative_post_agesplit_urlproxy_dict_to_urlchange_nested_dictconfig_factoryparser.pyparse_clicli_to_yamlupdate_yamlrecursive_check_config_typescheck_config_typesparse_configvalidate.pyvalidate_regionvalidate_delayvalidate_configDo you like this project and do you want to make it even better? Feel free to discuss below if you want to contribute to this project. All help is welcome 👍.
Want to start with something (relatively) easy? The functions in validate.py, tools.py and delay.py are relatively easy to test (at first inspection)!
Is your feature request related to a problem? Please describe.
Currently Glassdoor performs wage scraping, but Indeed and monster should do this as well.
Describe the solution you'd like
Add scraping for JobField.WAGE to indeed and monster scrapers
There are currently a number of state-of-the-art algorithms to summarize long passages of text.
We should apply one of these methods to produce the currently-unused short_description JobField
Example algorithms (whatever we use must provide a good license):
n/a
We should populate short_description with a condensed (1-3 sentences?) description generated from the longer Job.description
short_description is un-implemented.
n/a
I have 2 identical .yaml files. Only difference is the location the information is stored and one looks for CFO positions and the other Human Resource. However, the Human Resource has the following error where as the CFO does not:
Traceback (most recent call last):
File "/home/pittsie/.local/bin/funnel", line 8, in
sys.exit(main())
File "/home/pittsie/.local/lib/python3.8/site-packages/jobfunnel/main.py", line 28, in main
job_funnel.run()
File "/home/pittsie/.local/lib/python3.8/site-packages/jobfunnel/backend/jobfunnel.py", line 86, in run
self.master_jobs_dict = self.read_master_csv()
File "/home/pittsie/.local/lib/python3.8/site-packages/jobfunnel/backend/jobfunnel.py", line 405, in read_master_csv
locale = locale.UNKNOWN
AttributeError: 'NoneType' object has no attribute 'UNKNOWN'
Both files have been attached in a zip
Run
Error shown above only on Bill.yaml not 2Mom.yaml
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.