Comments (3)
pandzel-zz
commented on September 26, 2024
David,
I tried it myself by harvesting ioos site into the local folder and an instance of the geoportal catalog. I've got sold 300 records/min for the folder and 200 records/min for catalog. Not bad.
At this moment, without solid evidence and perhaps spending some time on profiling this endpoint, I can only conclude that the problem is NOT on the harvester side.
from geoportal-server-harvester.
valentinedwv
commented on September 26, 2024
If you leave "ignore robots.txt" unchecked, then you get a 10 second delay in CrawlLocker
from geoportal-server-harvester.
mhogeweg
commented on September 26, 2024
yep, check out:
http://search.geothermaldata.org/robots.txt
https://data.ioos.us/robots.txt
it lists the crawl delay. We respect the robots.txt settings by default (geoportal is a good bot).
from geoportal-server-harvester.
Related Issues (20)
- Harvester not removing content from geoportal that has been removed from source WAF HOT 5
- Item type of tiled image layers in ArcGIS Image not properly maintained when harvesting into ArcGIS Portal/Online HOT 1
- Harvester Issue to ArcGIS Portal - The size of each typeKeyword cannot be more than 256 characters
- Translation for AGOL/Portal HOT 1
- Harvester CKAN Broker Iterator Error for Data.gov
- Upgrading to 2.7 issue HOT 2
- Parse markdown to HTML in metadata XML
- Associate harvested metadata to existing sub-layers HOT 1
- Enable ArcGIS Online/Portal authentication in the harvester HOT 2
- Support for records in ISO 19115-3? HOT 2
- Enable layers option on ArcGIS Portal input broker. HOT 2
- Use title as output file name
- include reference to source metadata when publishing fails
- Harvest full XML from ArcGIS Server services and layers when available HOT 1
- Use ArcGIS Server layer metadata if available
- translate metadata when harvesting into geoportal
- translate locale information when harvesting to ArcGIS Online/Portal HOT 1
- support harvesting from OGC API: Records
- give CSW input broker option to switch http client
- include explicit sign out from web app HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from geoportal-server-harvester.