Comments (3)
Hey @eposjk
There isn't only JSONLD - structured data can be provided also in the microdata notation, and: good news - there is a project which parses microdata and converts it to the same data structure as JSONLD: https://github.com/yusufkandemir/microdata-parser
So it should be possible to use both and treat it just like an additional JSONLD block!
The parser looks good. But it supports only ^8.1
- that would require either a major release as it blocks 7.4 (will be dropped once it phased out a bit more) and 8.0 (still in use and security-supported). I've had a quick look over the lib and couldn't spot a reason why 8.0 wouldn't work...
Internally this project uses an own DOM document class derived from DOMDocument. It has a function to import a DOMDocument - but Symphonys response class doesn't allow to access the DOMDocument.
Passing the DOMDocument in directly would be preferred for sure. I'll have a look around to see if I can find a way to extract it without re-parsing.
What makes sense?
Adding separate PHPScraper functions for JSONLD and microdata? Or mixing both automatically? (my opinion: mixing)How should support for microdata look like? Adding the other project to PHPScraper? Extending the existing classes or porting the whole functionality to PHPScraper?
If a native solution is simple enough, I usually add it in directly. Here an external lib might make more sense. I guess the approach depends also the question if we can get the DOMDocument out of the packages used.
Cheers,
Peter
from phpscraper.
I've looked around and getting to a usable and maintainable approach seems to be tricky. My best approach atm would be something similar like the code below. It's placed in GoutteClient.php
(extension of \Goutte\Client
like in the other PR) with a method replacing createCrawlerFromContent
:
protected function createCrawlerFromContent(string $uri, string $content, string $type): ?Crawler
{
$previously = libxml_use_internal_errors(true);
$dom = (new HTML5)->loadHTML($content);
libxml_clear_errors();
libxml_use_internal_errors($previously);
// Get the base url.
$baseElements = $dom->getElementsByTagName('base');
// Crawler
$crawler = new Crawler(
null,
$uri,
$baseElements->count() > 0 ? $baseElements[0]->getAttribute('href') : $uri
);
$crawler->addDocument($dom);
return $crawler;
}
But this keeps failing with an error in the base href tests... It might be cleaner to re-parse it after all.
from phpscraper.
yusufkandemir/microdata-parser#4
This will cause issues if you want to do it with a DOM object.
from phpscraper.
Related Issues (20)
- Allow to set cookies
- TypeError HOT 3
- get http status code HOT 7
- Idea: Discovery Sets
- Idea: Implement low-level util to access the web. HOT 1
- Idea: Directly exposing received headers HOT 1
- What location PHPSCrapper based on? HOT 1
- Docker Composer Install Error HOT 12
- [Request] Add robots.txt parsing HOT 3
- [Request] Sitemap Index Files HOT 2
- Syntax Error when i tried using PHP 7.3 HOT 3
- fabpot/goutte HOT 14
- Spanish web content not displayed correctly '?' is putted instead of the correct character HOT 1
- Fix problems reported by PHPStan HOT 5
- psr/http-message 2.0 compatibility HOT 2
- issue about php scraping api HOT 1
- Scraping a site with CloudFlare protection/redirect returns no results HOT 2
- upgrade path for 3.0 HOT 5
- No way to get whole html content of page to process further. HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from phpscraper.