andrejgajdos / link-preview-generator Goto Github PK
View Code? Open in Web Editor NEWGet preview data (a title, description, image, domain name) from a url. Library uses puppeteer headless browser to scrape the web site.
License: MIT License
Get preview data (a title, description, image, domain name) from a url. Library uses puppeteer headless browser to scrape the web site.
License: MIT License
Hi there, firstly, I love this link preview generator. Your blog post spells it out very cleanly.
Anyway, it works beautifully locally but then when we tried to launch it live (in our docker) it errors out.
We don't get the error locally, do we really have to add a bunch of packages like this article suggests?
This is a minor bug but h2 tags are never used as a title.
See the code below from index.js, where both parts parse the h1 header :
const h1 = document.querySelector("h1").innerHTML;
if (h1 != null && h1.length > 0) {
return h1;
}
const h2 = document.querySelector("h1").innerHTML;
if (h2 != null && h2.length > 0) {
return h2;
}
I'm not sure how useful it is to get h2 tags, but I thought it couldn't hurt to fix it.
Is there a way to use this in a front-end application? E.g. using webpack and ES6 imports? The current package seems to only work in node.
Do
I have added this in one of my APIs with this logic
const { link } = req.body;
try {
const previewData = await linkPreviewGenerator(link, [
'--no-sandbox',
'--disable-setuid-sandbox'
]);
return res.json(previewData);
} catch (error) {
console.error(error);
return res.status(500).send(error.message);
}
And In the Dockerfile, I have added all the dependencies Puppeteer will require to launch chrome.
with this code:
RUN apt-get update \
&& apt-get install -yq --no-install-recommends \
ca-certificates fonts-liberation gconf-service libappindicator1 \
libasound2 libatk-bridge2.0-0 libatk1.0-0 libc6 libcairo2 libcups2 \
libdbus-1-3 libexpat1 libfontconfig1 libgbm1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 \
libglib2.0-0 libgtk-3-0 libnspr4 libnss3 libpango-1.0-0 \
libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 \
libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 \
libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 lsb-release wget xdg-utils
But when I call the API, it is throwing URL is not defined
I would love, if you can help mw with this.
src
is never defined in
link-preview-generator/index.js
Line 69 in 75cded0
paragraphs
isn't defined either at
link-preview-generator/index.js
Line 127 in 75cded0
Thanks for the efforts and work you gave to make this.
I'm facing a isssue when i send google meet link as url to the function and the error is :
(node:15759) [fs-extra-WARN0003] Warning: fs.realpath.native is not a function. Is fs being monkey-patched?
please can you provide any insight to this .
could you provide the heroku code found here https://link-preview-generator.herokuapp.com/ ?
Best,
This is all very new to me. I saw you deployed the demo to Heroku, but it seems that to use this as an API for a mobile app in production, it would have to be deployed to someplace like AWS. I looked at the documentation for npm and I couldn't find out how this would be used to create a business-use API. Can you direct me in the right direction?
I want to get a favicon as preview data. Can you merge a PR if I implement the following?
// index.js
const getFavicon = async (page, uri) => { // New
...
return favicon
}
module.exports = async (
uri,
puppeteerArgs = [],
puppeteerAgent = "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)",
executablePath
) => {
puppeteer.use(pluginStealth());
...
const obj = {};
obj.title = await getTitle(page);
obj.description = await getDescription(page);
obj.domain = await getDomainName(page, uri);
obj.img = await getImg(page, uri);
ojb.favicon = await getFavicon(page, uri) // New
await browser.close();
return obj;
};
Hi,
thanks for your work so far.
I've trouble with quite some links and I thought you might could have a look. Thanks!
https://en.bem.info/methodology/
(node:300075) UnhandledPromiseRejectionWarning: TypeError [ERR_INVALID_URL]: Invalid URL: /methodology/
https://www.der-informationsdesigner.de/agentur-blog/allgemein/persuasive-design-psychologie-im-webdesign
(node:300075) UnhandledPromiseRejectionWarning: Error: Evaluation failed: Error [ERR_TLS_CERT_ALTNAME_INVALID]: Hostname/IP does not match certificate's altnames: Host: der-informationsdesigner.de. is not in the cert's altnames: DNS:*.goserver.host, DNS:goserver.host
https://web.dev/one-line-layouts/
(node:300075) UnhandledPromiseRejectionWarning: TimeoutError: Navigation timeout of 30000 ms exceeded
This dependency was not found:
* ws in ./node_modules/puppeteer/lib/WebSocketTransport.js
To install it, you can run: npm install --save ws
used in Vuejs project
even after installing ws , still same error!
Thanks for your library. It works well, just this link: https://yarnpkg.com doesn't.
Here is some of the stack trace:
Error: Evaluation failed: ReferenceError: uri is not defined\n at <anonymous>:48:37\n at Array.forEach (<anonymous>)\n at <anonymous>:46:14\n at ExecutionContext._evaluateInternal (./node_modules/puppeteer/lib/ExecutionContext.js:122:13)\n at runMicrotasks (<anonymous>)\n at processTicksAndRejections (node:internal/process/task_queues:94:5)\n at async ExecutionContext.evaluate (./node_modules/puppeteer/lib/ExecutionContext.js:48:12)\n at async getImg (./node_modules/link-preview-generator/index.js:18:15)\n at async module.exports (./node_modules/link-preview-generator/index.js:176:13)
I'm not sure if it's link-preview-generator
or puppeteer
which needs a fix. The error also appears in the provided heroku demo
Thanks for sharing this handy preview generator. It works on most links, but a few are failing:
Error: Evaluation failed: TypeError: Cannot read property 'src' of undefined
at puppeteer_evaluation_script:50:22
at ExecutionContext._evaluateInternal ((...)\node_modules\puppeteer\lib\ExecutionContext.js:122:13)
at process._tickCallback (internal/process/next_tick.js:68:7)
-- ASYNC --
at ExecutionContext. ((...)\node_modules\puppeteer\lib\helper.js:111:15)
at DOMWorld.evaluate ((...)\node_modules\puppeteer\lib\DOMWorld.js:112:20)
at process._tickCallback (internal/process/next_tick.js:68:7)
-- ASYNC --
at Frame. ((...)\node_modules\puppeteer\lib\helper.js:111:15)
at Page.evaluate ((...)\node_modules\puppeteer\lib\Page.js:860:43)
at Page. ((...)\node_modules\puppeteer\lib\helper.js:112:23)
at getImg ((...)\node_modules\link-preview-generator\index.js:18:26)
at module.exports ((...)\node_modules\link-preview-generator\index.js:174:19)
at process._tickCallback (internal/process/next_tick.js:68:7)
I am using it in Windows 10 nodejs and got the following error while CPU is heavily busy
(node:5176) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 exit listeners added. Use emitter.setMaxListeners() to increase limit
(node:5176) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 SIGINT listeners added. Use emitter.setMaxListeners() to increase limit
(node:5176) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 SIGTERM listeners added. Use emitter.setMaxListeners() to increase limit
(node:5176) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 SIGHUP listeners added. Use emitter.setMaxListeners() to increase limit
(node:5176) UnhandledPromiseRejectionWarning: Error: Evaluation failed: TypeError: Cannot read property 'innerHTML' of null
at puppeteer_evaluation_script:14:44
Demos are not working.
After running:
const linkPreviewGenerator = require("link-preview-generator");
const previewData = linkPreviewGenerator("https://www.youtube.com/watch?v=8mqqY2Ji7_g");
console.log(previewData);
Promise { }
This hangs out here:
(node:12888) UnhandledPromiseRejectionWarning: TimeoutError: Navigation timeout of 30000 ms exceeded at C:\lpg\node_modules\puppeteer\lib\cjs\puppeteer\common\LifecycleWatch er.js:106:111
Any ideas of how to make it work?
Hello
i try to use your package but i got this error :
"Evaluation failed: TypeError: Cannot read property 'innerHTML' of null\n at puppeteer_evaluation_script:14:44" when trying this url:
https://www.laprovence.com/article/om/6063890/om-leonardo-balerdi-a-tire-daile.html
thanks for your help
I am sorry but it is not working for
Possible to fix it?
Update the npm package (https://www.npmjs.com/package/link-preview-generator)
Hey, Is there any ready-to-use files for shared hosting so we can make our own link preview?
Hi Everyone, i have tried this with Walmart products but this package doesn't work with link that are CAPTCHA protected. Any solution for this issue? Thanks
I can't use this with error
/root/bot/node_modules/puppeteer/lib/cjs/puppeteer/node/BrowserRunner.js:193
reject(new Error([
^
Error: Failed to launch the browser process!
[0729/104939.603734:ERROR:zygote_host_impl_linux.cc(90)] Running as root without --no-sandbox is not supported. See https://crbug.com/638180.
TROUBLESHOOTING: https://github.com/puppeteer/puppeteer/blob/main/docs/troubleshooting.md
at onClose (/root/bot/node_modules/puppeteer/lib/cjs/puppeteer/node/BrowserRunner.js:193:20)
at Interface.<anonymous> (/root/bot/node_modules/puppeteer/lib/cjs/puppeteer/node/BrowserRunner.js:183:68)
at Interface.emit (node:events:539:35)
at Interface.close (node:internal/readline/interface:529:10)
at Socket.onend (node:internal/readline/interface:258:10)
at Socket.emit (node:events:539:35)
at endReadableNT (node:internal/streams/readable:1345:12)
at processTicksAndRejections (node:internal/process/task_queues:83:21)
Node.js v17.9.0
Hey,
First of all, thank you for this great library.
I use your demo website to generate preview for my website and it seams to be down since a couple of days. The website is not really par of this repository but it was a way to notice you.
Alex
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.