federicohaag / linkedinscraping Goto Github PK

Scraping of LinkedIn Profiles: Creates an Excel file containing the personal data and the last job position of all the provided LinkedIn profiles.

Python 100.00%

linkedinscraping's Introduction

LinkedIn Scraping with Python

Create an Excel file containing personal data and last job position of specified people.

Scraping can be currently done only providing the LinkedIn profile url of the target person. If you don't have yet the LinkedIn profile urls, see below how to "Automatically get LinkedIn Profiles Urls".

Doubts? Reach me out on LinkedIn.

Prerequisites

You must have installed in your machine:

Google Chrome (last version)
Python (last version)

Installing

These instructions will get you a copy of the project up and running on your local machine for development and testing purposes.

Download the whole repository.
Place the directory LinkedinScraping in your python environment.
Navigate to the directory LinkedinScraping and run the following:

pip install -r requirements.txt

Run LinkedinScraping\configurator.py following the prompted instructions.

If this is the first time, choose the suggested configuration. In any time in the future you can easily run again the configuration to apply changes.

Executing

There are two ways you can run the code: headless execution and normal one.

In both cases, be careful (especially when you scrap a lot of profiles) because your computer may enter sleep mode. In sleep mode the scraping could not work. For MacOS I suggest Amphetamine.

Normal execution

In this mode the script will do scraping opening a real Chrome window.

Pros: In this case you will be able - if prompted - to satisfy the Captcha check and to proceed the scraping: the python script is trained on this situation and will perfectly manage it alerting you.

Cons: Be aware that if you choose this mode you can not loose the focus on the window, otherwise no data will be scraped.

To run in normal mode:

python do_scraping.py

Headless execution

In this mode the script will do scraping without opening a real Chrome window.

Pros: The scraping process is distributed into many threads to speed up to 4 times the performance. Moreover, in this way you can keep on doing your regular business on your computer as you don't have to keep the focus on any specific window.

Cons: If you scrap many profiles (more than hundreds) and/or in unusual times (in the night) LinkedIn may prompt a Captcha to check that you are not a human. If this happens, there is no way for you to fill in the Captcha. The script will detect this particular situation and terminate the scraping with an alert: you will have hence to run the script in normal mode, do the captcha, and then you can proceed in scraping the profiles that were left.

To run in headless mode:

python do_scraping.py HEADLESS

Examples

LinkedIn URLs:

https://www.linkedin.com/in/federicohaag/
https://www.linkedin.com/in/someoneelse/

When the Chrome page closes, it means the program ended. You can find inside the LinkedInScraping folder the extracted data in the results file results_profiles.xlsx. The filename will get concatenated to the current timestamp if the configuration was set as suggested.

Common problems in Running

Human check freezing the scraping

It may happen that while scraping the script will warn you about the need to perform a Human Check (a Google Captcha) even if it's not prompted for real.

This happens when you inserted in the input file a Profile URL which is not correctly formatted.

Here some tips:

The profile URL should always end with /
Open a browser window and navigate to such URL. Wait for the page to load. Is the URL currently in the browser navigation bar the same as the one you initially inserted? If not, you should insert in the input file the one you see now at the navigation bar.

Customizing

You can customize the configurations easily re-running configurator.py.

You can also customize the code in many ways:

The easy one is changing the order how the data is inserted in the excel file, or renaming the excel file headers.
The harder one is to do scraping of additional data: have a look at the Acknowledgments down here or feel free to reach me out to propose new code.

Authors

Federico Haag - LinkedIn - Medium

Disclaimer

The repository is intended to be used as a reference to learn more on Python and to perform scraping for personal usage. Every country has different and special regulations on usage of personal information, so I strongly recommend you to check your national legislation before using / sharing / selling / elaborating the scraped information. I decline any responsibility on the usage of scraped information. Cloning this repository and executing the included scripts you declare and confirm the responsibility of the scraped data usage is totally up on you.

linkedinscraping's People

Contributors

Stargazers

Watchers

linkedinscraping's Issues

A new version will be updated ??

Hello,

is it possible to update the version please? It doesn't work well...

Thanks

Roles

Hello, if a person has had more than one role or position within the same company, the script fails. Any idea how to add this functionality? Thank you

Jobs divided into multiple functions are not scraped

When a job is divided into multiple functions like on the screenshot above it is not recognized by the script.

Code needs to be updated as LinkedIn changed layout

This code is no longer able get names of the person's from their LinkedIn profile. Also LinkedIn added "Remember me on browser" screen and after that the code fails.

Please add new code to address these issues.

Scrapping Multiple Profiles Mix data

Hi! Im trying to scrape multiple person data in a for loop but even closing the driver, this library mix the data in the fields. Any suggestion?

Code Implemented:

from linkedin_scraper import Person, Company, actions
from selenium import webdriver
Personas = []

for i in range(5):
driver2 = webdriver.Chrome()
email = "[email protected]"
password = "Angostura!2020"
actions.login(driver2, email, password) # if email and password isnt given, it'll prompt in terminal
p= Personas.append(Person(str(final_test_short[i]).replace('['',"").replace('']',""), scrape=True,driver=driver2))
del driver2
sleep(2)

Chrome version

Hi. I was giving it a try for an OSINT project, and I get this:

selenium.common.exceptions.SessionNotCreatedException: Message: session not created: This version of ChromeDriver only supports Chrome version 80

My current chrome version is 83.0.4103.106

Thanks for sharing anyway.

Update JS functions to use document.querySelector for shorter queries

I was just look at this repo to use it for an upcoming project and was about to extend it to find phone numbers. Before that, I noticed that the current way to scrape an email and similar is relatively verbose. Is there any interest in updating to querySelector?

As an example, that'd change this:

email = self.browser.execute_script(
     "return (function(){try{for (i in document.getElementsByClassName('pv-contact-info__contact-type')){ "
                "let el = document.getElementsByClassName('pv-contact-info__contact-type')[i]; if("
                "el.className.includes( 'ci-email')){ return el.children[2].children[0].innerText; } }} catch(e){"
                "return '';}})()")

to this:

email = self.browser.execute_script(
                "return (function(){try{return document.querySelector('.pv-contact-info__contact-type.ci-email a')"
                ".innerText;} catch(e){return '';}})()")

Which I think is likely to be more readable long term?

Happy to take it on if there's interest in that, as well as adding some extra methods on Scraper for other information available in the Contact Info tab.

.

Scraping profiles by name

Hi Federico, thanks for this super useful tool! I completed the configuration and came to this point:

Configuration completed. You can now do scraping.
To scrape profile by url: execute do_scraping.py
To search profiles by name: execute search_profiles_by_name.py

I'm interested in collecting data from a list of names, but it doesn't look like there is an option to do it by name yet? (As an aside, it'd be awesome if you could also restrict it to geographic area, in case of duplicate names.)

i try to use but get me this error

configparser.NoSectionError: No section: 'profiles_data'

Scraping profiles by hashtags

Hi
I am Rizwan, I am working on a bot that can be used for marketing.
Phases include
1.Scraping emails of persons with specific Hashtags on their post
2.Perform automation to send emails in bulk to those people
I am working on the ist part I want to start working right aways making additions in your code.
I have tried to reach out you on LinkedIn kindly look into that please

Error

"Scraping Ended" after login

[WDM] - Current google-chrome version is 85.0.4183
[WDM] - Get LATEST driver version for 85.0.4183
[WDM] - Driver [/Users/.../.wdm/drivers/chromedriver/mac64/85.0.4183.87/chromedriver] found in cache
Scraping Ended