We get unscrapable responses trying to access YouTube video .../watchv= URLs (via a cors-proxy).

Or rather, ten times out of eleven. The content shown in browser dev tools (network) is almost all <script> and no meta tags.

In PostMan, if I clear the user-agent header, I get consistently usable data. I want to try this for metascraper, but I can't see any way to tell it to use a popsicle instance with options (i.e. custom headers) specified.

Is there any way to override popsicle defaults for metascraper? If not, it may make sense to be able to pass-through either a prebuilt popsicle object, or popsicle options. Is there quick workaround to over-ride the user-agent header when using metascraper from a browser?

EDIT:

Probably pointless, unless an expert can advise otherwise. Trying to set the user-agent header on popsicle just gives Refused to set unsafe header "user-agent" which is presumably enforced at the browser level. I've been trying to hold off on implementing a custom back-end cors/scrape-proxy, but maybe that's the only option.

It'd be nicer on users if it were written in ES5 promises. Related to #7. Current implementation relies on a regenerator runtime from Babel to support async/await.

• http://tubularinsights.com/download-youtube-thumbnail-images-creators-tip-120/

when I try to get tags from github, and many other sites, I get a similar error to the one below:

XMLHttpRequest cannot load https://github.com/ianstormtaylor/metascraper/issues/new. 
Redirect from 'https://github.com/ianstormtaylor/metascraper/issues/new' to 
  'https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fianstormtaylor%2Fmetascraper%2Fissues%2Fnew' 
has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. 
Origin 'http://localhost:4201' is therefore not allowed access.

Is there anyway to ignore this? or a method of retrieving the page like a browser would?

There are a few different metadata syntaxes out there for tags, would return an array of them. Maybe even join across syntaxes to get the full list. And then unique with case insensitivity

Hi

I would be interested in language property. These are the places I know where the language can be stored:

• html lang
• og:locale
• hreflang (https://moz.com/learn/seo/hreflang-tag) - but this part is needs most work and from what I see there could be errors here (many people only use href lang for other languanges than currently selected)

Thanks in advance.

This particular scenario throws an exception:
const MY_RULES = {
title: [
myPreferredTitleRule,
myFallbackTitleRule,
mySuperLastResortTitleRule,
]
}
scrapeMetadatum does not seem to anticipate receiving fallback rules and throws an exception on line 95: var next = rule($, url); when "rule" is an array of functions instead of a function itself.

I am really looking forward to trying this project out. My node version is v5.5.0. I am getting the following syntax error:

I believe this is because the ES6 default parameters are not supported in my version of node.

Node 8 is great and everybody should update ASAP, but unfortunately, it's not always that easy.

Node 6.x is still Active LTS status and some great services don't support 8.x yet.

Also, adoption isn't widespread yet:

In this, you can see the three release lines represented by solid green (Node.js 4 "Argon"), solid blue (Node.js 6 “Boron”), and solid purple (Node.js 8 “Carbon”).

Throughout the year, you can see Node.js 6 reigning supreme—peaking at 388,417 downloads on October 25th. Unlike last year with Node.js 6 and Node.js 4, the downloads of Node.js 8 didn’t surpass the downloads of the Node.js 6 LTS. This is an interesting trend that we should watch into the new year. It seems that the majority of downloads of Node.js are still Node.js 6, even though Node.js 8 has been trending upward for the past few months.

Given the data, it may be safe to assume that the delayed LTS-adoption trend will continue, and as Node.js usage continues to grow, adoption of new LTS versions could very well take longer and longer with each new LTS cycle. We’ll be able to confirm or reassess this assumption with next year’s data, though!

Source: NodeSource

I found that the library aimed to support Node 4.x and browser back in the old days by transpiling via Babel but I couldn't find a notice about dropping support to these environments on Changelog/Release Notes (Although there is a Remove browser mention commit).

Cloud Function providers are probably going to support 8.x soon as this is The Right Thing™ but they're moving slowly.

So, would maintainers consider accepting a PR adding back Babel to enable this great library at least on 6.x runtime in the meantime or while marked as LTS?

Thanks for working on OSS projects!

Hi,

I set up a small test page (with nearly no metadata) to play around with metascraper and noticed two unexpected things:

1. image will be returned null if the only available on the page has a relative URL (works fine with absolute URLs)
2. url is always null (both for the direct URL to the page and short URLs)

The test page can be found here: http://x.manuel-heidrich.com/tests/browse/
Short URLs for the test page: http://bit.ly/2zVanQ9 and https://goo.gl/gTYg35

HTML of the test page: http://sharedby.manuel-heidrich.com/Cs6cno/3QJaQeuE
Response from metascraper: http://sharedby.manuel-heidrich.com/WAH4q/1HE7KFYF

In case of the images I guess it would be great if in case of relative URLs imagescraper would return absolute URLs.

Concerning the URL: Is it a bug or intended behaviour? I think url should never be null.

Happy to help if I can!

Would favor canonical URLs, but fall all the way back to window.location.href.

@blakeembrey pointed out that New York Times ends up perma redirecting without one! We should switch to popsicle anyways, since i think it has built-in support.

i saw the #27
i did you say that, but cors error...
error: XMLHttpRequest cannot load http://www.naver.com/. Redirect from 'http://www.naver.com/' to 'https://www.naver.com/' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://www.lululala.co.kr:8000' is therefore not allowed access.

i'm a front-end developer.. so i must solve the cors in front...
please tell me... the solve
how can i do?
please.... help me.

Currently the build actually relies on the end-user having Babel installed and all of Babel core. It'd be nicer on us users if we could use it without Babel.

trying to run the basic example however it is logging TypeError: Cannot read property 'parent' of null

    import Metascraper from 'metascraper';

    Metascraper
      .scrapeUrl('http://www.bloomberg.com/news/articles/2016-05-24/as-zenefits-stumbles-gusto-goes-head-on-by-selling-insurance')
      .then((metadata) => {
        console.log('meta: ', metadata);
      })
      .catch((error) => {
        console.log('error: ', error);
      });

Embedly for example can be given keywords. Could you add such functionality?

I think it would be simple to do, and kind of nice to allow rules to be defined with just a single function, instead of always needing to be an array. Such that as simple case could look like...

title: async (window) => window.document.title.textContent

Or the complex ones could be:

title: [
  // tons of rules...
]

@blakeembrey that sound good to you?

any idea why some url like this one http://www.concordmonitor.com/on-new-hampshire-s-antique-alley-it-s-out-with-the-old-in-with-the-new-9234789 gives this error while calling:

Metascraper.scrapeUrl(results[j].url)

Returning the cleaned body of the article.

Are there any examples of a custom rule set in a project? The documentation, to me, isn't clear enough. Where do I save the custom ruleset? How do I require it in the rc file?

I'm running docker + node.

I noticed this uses async rules, any particular reason? Seems all the current rules are completely synchronous.

FWIW, if you just want to compose async functions I wrote https://github.com/blakeembrey/throwback. It takes a chain of functions that return promises and any can either return value or return next() to continue execution. Might be a nice middleware pattern here if you want to keep async middleware.

It would be nice if you could specify a timeout and reject the returning promise after said timeout.

Right now I think jsdom is being included regardless. But it would be good to make it easier for certain pieces to be used without it.

Add a new property for supporting video (mp4, avi, webm) and gif embed.

Rules from og meta tags

• og:video:secure_url
• og:video:url
• og:video

(a lot of good examples here!)

Rules from twitter meta tags

• twitter:player:stream

Not sure if twitter support property tags to target video. Need to check:

• https://davidwalsh.name/twitter-cards
• https://developer.twitter.com/en/docs/tweets/optimize-with-cards/overview/player-card

Rules from sailthru meta tags

???

Rules from HTML

• <video>
• <source>

Generate a thumbail of the important part of a video based on video property added by metascraper-video

https://github.com/MrYakobo/video-thumbnail

This is the code in my express router:

console.log('Scraping metadata...', req.query.url);
  Metascraper.scrapeUrl(req.query.url)
  .then((metadata) => {
    console.log(metadata);
    res.send(metadata);
  })
  .catch((err) => {
    console.log(err);
    res.send(err);
  });

It returns this error when trying to scrape http://facebook.com

{ [Error: incorrect header check] errno: -3, code: 'Z_DATA_ERROR' }

Any thoughts why this is happening?

I have a package published that uses metascraper, yet it doesn't seem to draw from either a metascraperrc or the metascraper section of package.json, when it's in node_modules. It works when testing in its own directory, but not as a package. Am I doing something wrong, or is this a scoping issue? It could also be that I'm using a local rule, such as ./rule.js, which it might be looking for in the root directory of the project that has it as its dependency.

In https://github.com/ianstormtaylor/metascraper/blob/master/lib/rules/author.js. Maybe a better check is /^\S+\s+\S+/?

Would be cool to have, and helpful in evaluating them

Getting this error:

ERROR in ./node_modules/cheerio/index.js
Module not found: Error: Can't resolve './package' in '....../node_modules/cheerio'

Hi @ianstormtaylor, I'm not sure if this is completely out of scope for this library - if yes, apologies.

But in case, it isn't, it would be amazing to treat product pages as distinct from articles by getting product specific information from the sites (atleast the main ones have it standardized). Here is a library (though a bit outdated) I found which does some of that - https://github.com/hulihanapplications/fletcher/blob/master/lib/fletcher/models/

Thank you - your library looks great! :)

JSDOM is kind of heavy here. Would you consider a PR that replaces JSDOM with Cheerio instead? Since it's really only using getAttribute and querySelector like APIs, maybe use Cheerio for all text parsing use-cases and have a wrapper for the browser window use-case that defers to the document.* methods?

Here is the full error:

web_1         | ERROR in ./~/cheerio/index.js
web_1         | Module not found: Error: Can't resolve './package' in '/src/app/node_modules/cheerio'
web_1         |  @ ./~/cheerio/index.js 11:18-38
web_1         |  @ ./~/metascraper/dist/index.js

trying to build with webpack.

When I try to require('metascraper') I got this error:

/home/user/workspace/node_modules/metascraper/lib/index.js:2
let RULES = require('./rules')
^^^

SyntaxError: Block-scoped declarations (let, const, function, class) not yet supported outside strict mode

My suggestion is to add the 'use strict'; on top of each file containing a block-scoped declaration.

I'm running into issues with the image value not being the main image for metascraper-amazon. There are actually multiple .a-dyanmic-image classes on the screen as seen in the attached photo. Can we create some rules with priority over this like wrapUrl($ => $('#landingImage').attr('src')) or wrapUrl($ => $('.a-dynamic-image').first().attr('src'))?

sometimes the meta image returns something like "//images.solecollector.com/complex/images/w_1200/viukziox1lihfkmhn2i5/off-white-air-jordan-1"

instead of "https://images.solecollector.com/complex/images/w_1200/viukziox1lihfkmhn2i5/off-white-air-jordan-1"

I am having some difficulties creating my own rule. For test purposes, I copied the metascraper-publisher folder and renamed it to "metascraper-test". I changed the appropriate names in the package.json file. I then added "metascraper-test" to the Default_Rules section in the "load-rules.js" file. However, I keep getting a "metascraper-test" not found error, even though it is present.

Any ideas on what could be causing this?

Thanks!

Can you support it, global or from plugins?
<meta http-equiv="Content-Type" content="text/html; charset=windows-1251" />

Hi,

It seem that the published npm packge isn't transpiled by Babel.

Are there any reasons why application/ld+json is not being utilized? It seems to be contained in the majority of large websites and includes information such as headline, description, date, authors, publisher and more!

An example site that utilizes this format is https://www.theverge.com/2017/11/16/16667366/tesla-semi-truck-announced-price-release-date-electric-self-driving.

I can work on this if others think it would be helpful. Just wanted to reach out first to see if anybody else has already done this, or if there is a reason not too.

Thanks!

Prerequisites

• I'm using the last version.
• [N/A] My node version is the same as declared as package.json.

Subject of the issue

We're trying to use Metascraper in an Angular 5 application since the README.md indicates Metascraper supports browser usage. However, we get Module not found: Error: Can't resolve 'module' in require-from-string and resolve-from from Angular CLI when calling Metascraper from our code:

ERROR in ./node_modules/metascraper/node_modules/require-from-string/index.js
Module not found: Error: Can't resolve 'module' in '/path-to/node_modules/metascraper/node_modules/require-from-string'
 @ ./node_modules/metascraper/node_modules/require-from-string/index.js 3:13-30
 @ ./node_modules/metascraper/node_modules/cosmiconfig/dist/loadRc.js
 @ ./node_modules/metascraper/node_modules/cosmiconfig/dist/createExplorer.js
 @ ./node_modules/metascraper/node_modules/cosmiconfig/dist/index.js
 @ ./node_modules/metascraper/src/load-rules.js
 @ ./node_modules/metascraper/src/index.js
 @ ./clients/browser/app/library/content-linked-edit/content-linked-edit.component.ts
 @ ./clients/browser/app/library/library.module.ts
 @ ./clients/browser/$$_lazy_route_resource lazy
 @ ./node_modules/@angular/core/esm5/core.js
 @ ./clients/browser/main.ts
 @ multi webpack-dev-server/client?http://0.0.0.0:0 ./clients/browser/main.ts

ERROR in ./node_modules/metascraper/node_modules/resolve-from/index.js
Module not found: Error: Can't resolve 'module' in '/path-to/node_modules/metascraper/node_modules/resolve-from'
 @ ./node_modules/metascraper/node_modules/resolve-from/index.js 3:15-32
 @ ./node_modules/metascraper/src/load-rules.js
 @ ./node_modules/metascraper/src/index.js
 @ ./clients/browser/app/library/content-linked-edit/content-linked-edit.component.ts
 @ ./clients/browser/app/library/library.module.ts
 @ ./clients/browser/$$_lazy_route_resource lazy
 @ ./node_modules/@angular/core/esm5/core.js
 @ ./clients/browser/main.ts
 @ multi webpack-dev-server/client?http://0.0.0.0:0 ./clients/browser/main.ts

The following warning precedes those errors:

WARNING in ./node_modules/metascraper/src/load-rules.js
52:15-34 Critical dependency: the request of a dependency is an expression
 @ ./node_modules/metascraper/src/load-rules.js
 @ ./node_modules/metascraper/src/index.js
 @ ./clients/browser/app/library/content-linked-edit/content-linked-edit.component.ts
 @ ./clients/browser/app/library/library.module.ts
 @ ./clients/browser/$$_lazy_route_resource lazy
 @ ./node_modules/@angular/core/esm5/core.js
 @ ./clients/browser/main.ts
 @ multi webpack-dev-server/client?http://0.0.0.0:0 ./clients/browser/main.ts

Does Metascraper work in a browser environment?

XMLHttpRequest cannot load #27. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://www.lululala.co.kr:8000' is therefore not allowed access.

:8000/#/:1 Uncaught (in promise)
PopsicleError {cause: undefined, code: "EUNAVAILABLE", popsicle: Request, message: "Unable to connect to "#27"", name: "PopsicleError"…}

how should i do???

It appears that there is an unintended 'a' in line 50 of the author package. In my test's this is causing poor results for the author.

strict(wrap($ => getValue($, $('[class*="author"] a')))),

Using this url, you will find that metascraper returns "Shakespeare Videos" for author
http://www.sparknotes.com/shakespeare/macbeth/quotes.html

Removing the random "a" on line 50, returns the correct author, "William Shakespeare"

strict(wrap($ => getValue($, $('[class*="author"]')))),

However, there are other URL's that only successfuly extract the Authors name when that 'a' is present. What exactly is that "a" doing?

Thanks,

I believe this is an issue of how metascraper use of "require()" works with webpack design. I wish to ask if you can offer any direction to fix these errors I'm getting when using metascraper with vue.js, which uses webpack. Below is something I learned about webpack by referencing an example of how webpack expects require() to work:

var module_path= "./dir/"+somevariable+".js";
var foo= require(module_path);

• require("./dir/"+somevaribale+".js") is statically analyzable, so it works with webpack
• require(module_path) is not statically analyzable, so it doesn't work with webpack.
Webpack calls this feature "require context" and its behavior is by design.

Referencing one of the warnings -- warning in ./node_modules/metascraper/src/load-rules.js

• 52:15-34 Critical dependency: the request of a dependency is an expression
• the offending code in load-rules.js seems to be with all the "require()" statements that are referencing variables.

QUESTION: Is there a way to resolve this issue so metascraper can be used with webpack?

Thanks... Jim

Prerequisites

• [ Yes ] I'm using the last version of metascraper.
• [ 5.6.0 ] My npm version is the same as declared as package.json.
• [ v6.11.1 ] My node version is the same as declared as package.json.
• "vue": "^2.5.2",
• "webpack": "^3.6.0",
• "webpack-bundle-analyzer": "^2.9.0",
• "webpack-dev-server": "^2.9.1",
• "webpack-merge": "^4.1.0"

Critical dependency: the request of a dependency is an expression

PLEASE SEE THE SEVEN WARNINGS BELOW.

# npm run dev

 WARNING  Compiled with 7 warnings                                 

 warning  in ./node_modules/keyv/src/index.js

18:14-40 Critical dependency: the request of a dependency is an expression

 warning  in ./node_modules/metascraper/src/load-rules.js

52:15-34 Critical dependency: the request of a dependency is an expression

 warning  in ./node_modules/lazy-debug-legacy/src/functions.js

62:15-40 Critical dependency: the request of a dependency is an expression

 warning  in ./node_modules/lazy-debug-legacy/src/functions.js

87:8-28 Critical dependency: the request of a dependency is an expression

 warning  in ./node_modules/css/node_modules/source-map/lib/source-map/source-map-generator.js

8:45-52 Critical dependency: require function is used in a way in which dependencies cannot be statically extracted

 warning  in ./node_modules/css/node_modules/source-map/lib/source-map/source-map-consumer.js

8:45-52 Critical dependency: require function is used in a way in which dependencies cannot be statically extracted

 warning  in ./node_modules/css/node_modules/source-map/lib/source-map/source-node.js

8:45-52 Critical dependency: require function is used in a way in which dependencies cannot be statically extracted

We are working on a v2 iteration.

TODO

• Close all issues related with HTTP problems.
• Migrate documentation.
• Update dependencies.
• Migrate benchmark.
• Add contributor guidelines.

Breaking Changes

The major breaking changes in this new version are that HTTP layers are moved out of the library.

From now, metascraper will be a main function that receives HTML and URL for extracting the data:

const metascraper = require('metascraper')
const got = require('got')

const targetUrl = 'http://www.bloomberg.com/news/articles/2016-05-24/as-zenefits-stumbles-gusto-goes-head-on-by-selling-insurance'

;(async () => {
  const {body: html, url} = await got(targetUrl)
  const metadata = await metascraper({html, url})
  console.log(metadata)
})()

Because this iteration refactor a lot of code, is not possible create a PR, so I'm going to use this issue as a tracking list of things to close before release the new version.

Improvements

• Add new logo field.
• Updated tests, now with at least top50 popular internet sites.
• The codebase was simplified and ready for supporting plugins in 3.0.0.

Cheerio results often return empty strings when no matches occur, so it seems best to not match on that instead and keep going down the fallbacks array

A bit more to think about after we nail the main API, but I was thinking that it feels weird for the bundled default rules to have such prominence, when they are for a pretty specific use case. Wondering instead if it might be nicer to make rules more of a plugin-system.

It could even maintain the same (source, [rules]) API if that's still the simplest, although maybe it becomes a constructor with new Metascraper(rules) and then .scrapeHtml()... instead.

This would allow bundles of rules. The current set might be something like metascraper-plugin-articles but there could easily be a simple Open Graph one that doesn't handle fallbacks, as metascraper-plugin-open-graph, or whatever!

cc @blakeembrey for thoughts!

Hi @ianstormtaylor! Does/can the library also return the favicon distinct from the image?

Would be awesome have a logo for identifying the project.

Ideas are welcome 😄

This is more of a question than an issue.

In the author's plugin,
wrap($ => $('meta[property="article:author"]').attr('content')),
is listed second in the module export. Does this mean that it has a higher priority than the other methods of extracting an author?

Because I have noticed that if "article: author" is equal to a website url, then it is not used and a method further down the list is being used to extract the author.

How is this working? I do not see any code that is checking to see if its an URL or an actual authors name.