Comments (1)
Google announced proposed standards for robots.txt which have been submitted to the IETF. As part of this work they have looked at non-standard rules such as crawl-delay and decided to no longer support this as of September 1st, 2019 (https://webmasters.googleblog.com/2019/07/a-note-on-unsupported-rules-in-robotstxt.html). Whilst other search engines MAY support crawl-delay, it's more than likely that the'll adopt the proposed standards, especially if they're formalized by the IETF.
I therefore propose that we no longer add support for crawl-delay and I'm closing this issue. We can reopen it if we later decide that we want to support it for other search engines or if it's added to the standards.
from all-in-one-seo-pack.
Related Issues (20)
- Meta title and description NULL in DB, and not preset in published page's source code HOT 4
- Orphaned post left-overs in aioseo_posts table HOT 13
- Uncaught TypeError: array_replace_recursive() with PHP 8.0 HOT 1
- Feature Request: Separate Social Image Sharing settings for Pages vs Posts
- Critical bug: AIOSEO initialize $wp_roles global variable prematurely and broke wordpress role management HOT 4
- Bug: AIOSEO removes the "dashicons" stylesheet from WP frontend HOT 2
- The editor has encountered an unexpected error. HOT 2
- meta og:site_name generated despite being left blank HOT 2
- Robots.php error with PHP 8.1 HOT 2
- Call to Undefined Method - When saving a menu HOT 2
- Ping not working HOT 1
- The aiosp_opengraph_meta filter isn't working – cannot re-write og:url... HOT 2
- Updating link editor modal styles for WordPress 6.1 HOT 2
- Remove version number from generator meta tag HOT 1
- Version 4.3.3 broke query loop to cpt, pages, post after updating to WordPress 6.2 HOT 3
- Person/sameAs invalid type issue HOT 1
- Custom Archive and Polylang HOT 1
- Please remove "table.wp-list-table .column-name" css rule HOT 2
- Warning: Undefined array key "dashboardWidgets" after plugin update HOT 1
- Deprecated: Function print_emoji_styles in /wp-admin/index.php?page=aioseo-setup-wizard HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from all-in-one-seo-pack.