Comments (6)
know this is a semi-long time coming but once the chrome-remote-interface-extra-intergration branch is merged this and a hole lot more things will be possible using Squidwarc
PS spread the word, you dont need puppeteer to simply use the CDP
https://github.com/N0taN3rd/chrome-remote-interface-extra ;)
from squidwarc.
Hey y'all, If you want to start test running things today this feature is living in the chrome-remote-interface-extra-intergration branch.
The entry point to make changes like this is the chromeCustomizer.js file.
Puppeteer CI is failing currently and chrome-remote-interface-extra's CI is good except for an pesky net::ERR_NAME_NOT_RESOLVED vs net::ERR_NAME_RESOLUTION_FAILED error message that happens on travis for some reason and using google chrome canary....
CI link: https://travis-ci.com/N0taN3rd/chrome-remote-interface-extra
Full documentation for the more you can do with this library than with puppeteer is found here https://n0tan3rd.github.io/chrome-remote-interface-extra/.
I'm gona add redis frontier support and frontier customization functions before this feature gets merged into master (I'm tired of in memory frontiers)
from squidwarc.
This is a good suggestion for an option, @peterk. http://ws-dl.blogspot.com/2018/03/2018-03-21-cookies-are-why-your.html provides some examples of weirdness in language detection via IA submission. It would be interesting to test this from different IPs and Accept-Language
values to see if the effects are replicable.
from squidwarc.
This issue is up next on the big list of things to do
from squidwarc.
Hey y'all I finally got node-warc and chrome-remote-interface-extra in a position to support this feature request.
I am thinking the API for this is as follows:
You can, like you do for supplying a user script that is run before WARC generation, supply a function that is passed as its only argument the page object of chrome-remote-interface-extra, puppeteer or the chrome-remote-interface client object in order to customize the behavior of the browser.
Example when using chrome-remote-interface-extra (type definitions for the arguments of pageOrClient.setGeolocation is not valid JS but provided for your convince)
module.exports = async function chromeCustomizer (pageOrClient) {
// set the download path of files downloaded by the browser
await pageOrClient.setDownloadBehavior('<path to new downloads folder>')
// set the Accept-Language HTTP header
await pageOrClient.setAcceptLanguage('<new language>')
// set navigator.platform
await pageOrClient.setNavigatorPlatform('<new platform>')
// set new geolocation
await pageOrClient.setGeolocation({longitude: number, latitude: number, accuracy: (number|undefined)})
}
For both chrome-remote-interface-extra and puppeteer the connection to the browser tab is found on pageOrClient._client
if you need more fine tuned customization and as always please consult the CDP documentation for details.
Please let me know if there are any suggestions or concerns about how to make this as user friendly as possible.
from squidwarc.
Documentation on the upcoming chrome-remote-interface-extra integration https://n0tan3rd.github.io/chrome-remote-interface-extra/
from squidwarc.
Related Issues (20)
- TypeError: input.on is not a function HOT 6
- Add docker file
- Allow crawl config to contain user scripts for a page HOT 16
- Instagram pages not recorded properly? HOT 3
- Squidwarc on Windows 10 (using npm) HOT 1
- Generated WARC files fail indexing with OpenWayback cdx-indexer HOT 11
- Exception immediately thrown on first-run HOT 5
- Make option for gzipped WARC:s? HOT 1
- Describe requirements in the README HOT 1
- Unable to reuse local Chrome user dir/cookies HOT 7
- Append config seem working not correctly. HOT 3
- Feature request: ignore sets
- Feature request: following links HOT 5
- Error: options.stripFragment is renamed to options.stripHash HOT 1
- Installation fails with missing node-warc submodule HOT 3
- TypeError in node-warc causes Squidwarc to crash HOT 2
- Exception with defaults using Docker in Puppeteer config HOT 2
- Setting mode in conf.json is ignored HOT 1
- Fatal error when starting crawl HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from squidwarc.