website-scraper / demo Goto Github PK
View Code? Open in Web Editor NEWDemo app for website-scraper module
Demo app for website-scraper module
Hi, I'm using the library to scrape a site and I need all the assets on their root folders:
css/[all CSS here]
js/[all JS here]
Turns out that it works, but in some cases it creates subdirectories with the very same structure. I want the scrapper to put all configured assets (no matter the deep page it belongs to) to be on the root folder.
What file do I change to enable the iframe links to work?
When I open scraped page, like this, and click Back
button I should be returned on list
page, not main
.
Really weird occurance. Scraper ran, clicked on the new window button to preview, everything looks normal.
Download the file, the background images have disappeared..
Another example:
https://scraper.nepochataya.pp.ua/static/files/www.riemtex.de-1594223643199/
After download:
502 Bad Gateway
Some websites will determine the country of the IP address.
I think, dependencies update are required.
Also, I get bower question, when I trying install app:
Unable to find a suitable version for angular, please choose one:
1) angular#1.2.29 which resolved to 1.2.29 and is required by angular-cookies#1.2.29, angular-route#1.2.29
2) angular#~1.3.15 which resolved to 1.3.20 and is required by web-scraper
3) angular#1.3.20 which resolved to 1.3.20 and is required by angular-resource#1.3.20
where is the variable in the javascript that stores the html of the fetched scraped page? need to find it so I can modify the content.
version: [result of npm ls website-scraper --depth 0
command]
options: [provide your full options object]
i tried your demo app and the problem is that there is only the main url which is not downloaded under url for example if the url is a.html it will not download a / 2.html
[Description of the issue]
Expected behavior: [What you expect to happen]
Actual behavior: [What actually happens]
[Any additional information, configuration or data that might be necessary to reproduce the issue]
I'm learning your script and looking at the demo app. And noticed that some sites are being opened in full screen not iframe for example this page "http://www.delfi.lt/", which is a news page.
I'm not sure but the problem might be here
Can't Load URL: The domain of this URL isn't included in the app's domains. ping?client_id=161041023936278&domain=localhost&origin=1&redirect_uri=http%3A%2F%2Fstaticxx.faceboo…:1 To be able to load this URL, add all domains and sub-domains of your app to the App Domains field in your app settings.
OneSignal: Could not load iFrame with URL https://delfi.onesignal.com/webPushIframe. Please check that your 'subdomainName' matches that on your OneSignal Chrome platform settings. Also please check that your Site URL on your Chrome platform settings is a valid reachable URL pointing to your site.
Demo is gone, https://scraper.nepochataya.pp.ua/
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.