viasite / site-audit-seo Goto Github PK

Hi @popstas ,

Hope you are all well !

It would be awesome to generate a powerpoint with stats/charts from the seo analysis results.

More precisely was thinking about such template (we can probably find a free template):
https://www.sketchbubble.com/en/presentation-website-seo-audit.html

What do you think about that feature ?

Cheers,
Luc Michalski

Это очень полезный инструмент и будет очень удобно запускать его в докер контейнере.

I see the license is listed as ISC. I would recommend using a copyleft license instead as successful permissively licensed projects tend to be used by companies within proprietary projects and they typically never give back. If you go with copyleft, I would recommend AGPL-3.0-or-later or at least GPL-3.0-or-later. If you stick with a permissive license, I would recommend Apache-2.0.

I would further recommend making a license file in the root directory of the repo. If you create a LICENSE file, this information will be displayed within the first page view for the project. GitHub walks you through the steps after naming a new file LICENSE.

Hi guys,

Hope you are all well !

Is it possible to add a webservice or webhook to collect data from a website ?

Something like [POST] http://localhost:3000/collect/?url=example.com

Thanks for any insights or inputs on that question.

Cheers,
Luc Michalski

Is this package still being maintained?

I just did a fresh install and got the following warnings:

npm WARN deprecated [email protected]: Critical security vulnerability fixed in v0.21.1. For more information, see https://github.com/axios/axios/pull/3410
npm WARN deprecated [email protected]: request-promise has been deprecated because it extends the now deprecated request package, see https://github.com/request/request/issues/3142
npm WARN deprecated [email protected]: request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN deprecated [email protected]: < 18.1.0 is no longer supported
npm WARN deprecated [email protected]: this library is no longer supported
npm WARN deprecated [email protected]: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
npm WARN deprecated [email protected]: Please upgrade to @sentry/node. See the migration guide https://bit.ly/3ybOlo7
npm WARN deprecated [email protected]: We've written a new parser that's 6x faster and is backwards compatible. Please use @formatjs/icu-messageformat-parser
npm WARN deprecated [email protected]: request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN deprecated [email protected]: Deprecated due to CVE-2021-21366 resolved in 0.5.0
npm WARN deprecated [email protected]: this library is no longer supported
npm WARN deprecated [email protected]: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
npm WARN deprecated [email protected]: Debug versions >=3.2.0 <3.2.7 || >=4 <4.3.1 have a low-severity ReDos regression when used in a Node.js environment. It is recommended you upgrade to 3.2.7 or 4.3.1. (https://github.com/visionmedia/debug/issues/797)

Hi I keep getting this error after the command: site-audit-seo -u https://example.com --lighthouse
error: node:internal/process/promises:288
triggerUncaughtException(err, true /* fromPromise */);
^

Error: connect ECONNREFUSED ::1:57468
at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1300:16) {
errno: -4078,
code: 'ECONNREFUSED',
syscall: 'connect',
address: '::1',
port: 57468
}

Node.js v18.12.1

dont plan to use it unless its a .ru site

Hi guys,

Hope you are all well !

Would it be possible to send your metrics into influxdb so we can display them in a grafana dashboard ?

Just check this example, and you ll get what I mean: https://github.com/FeliceGeracitano/webperf-dashboard

That would be awesome, as it would be easy to integrate the site-audit-seo-viewer into a dashboard.

I can help :-)

Thanks for any insights or inputs on that.

Cheers,
Luc Michalski

Hi,

Hope you are all well ! And merry Christmas first of all !

I was playing to today with site-audit-seo and I was missing some features like a robots.txt parser to find available sitemaps for a website and the related sitemap extractor (handling sitemap indexes also).

Do you think it is possible to add these 2 components easily ?

Please fin below some references that I found:

• https://github.com/seantomburke/sitemapper
• https://github.com/evanderkoogh/node-sitemap-stream-parser#readme
• https://github.com/b4dnewz/robots-parse
• https://github.com/ChristopherAkroyd/robots-txt-parser

Thanks for any insights or inputs on that.

Ps. Do you have a telegram account as I have some questions for you and do not want to pollute this thread ?
My handle is "deepocrates"

Cheers,
Luc Michalski

Hey Guys,

I really like your solution to get some statistics about homepages and SEO-Stuff.

I tried your provides solution and it works perfectly.

After it I tried to run the docker-compose section and got these errors:

$ docker-compose up -d

ERROR: build path /var/docker/config/pagespeed-insights-lighthouse/site-audit-seo/data/front either does not exist, is not accessible, or is not a valid URL.

So I created the folder-path
$ mkdir -p data/front

$ docker-compose up -d

sas-backend is always restarting
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e1510237a6a4 viasite/site-audit-seo:latest "docker-entrypoint.s…" 16 seconds ago Restarting (1) 1 second ago sas-backend

$ docker logs sas-backend

[email protected] server /app
node src/server.js

loaded plugins: export-influxdb
Create empty package.json in data
internal/fs/utils.js:307
throw err;
^

Error: EACCES: permission denied, copyfile './package-data.json' -> 'data/package.json'
at Object.copyFileSync (fs.js:1991:3)
at Object.exports.initDataDir (/app/src/utils.js:22:8)
at Object. (/app/src/server.js:17:7)
at Module._compile (internal/modules/cjs/loader.js:1063:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1092:10)
at Module.load (internal/modules/cjs/loader.js:928:32)
at Function.Module._load (internal/modules/cjs/loader.js:769:14)
at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:72:12)
at internal/main/run_main_module.js:17:47 {
errno: -13,
syscall: 'copyfile',
code: 'EACCES',
path: './package-data.json',
dest: 'data/package.json'
}
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] server: node src/server.js
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] server script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR! /home/node/.npm/_logs/2021-06-09T15_23_14_901Z-debug.log

I added the the App Path in docker-compose File
volumes:
- .:/app <------------
- ./data:/app/data
- ./data/reports:/app/data/reports
- ./data/db-docker.json:/app/data/db.json

$ docker-compose down
$ docker-compose up -d
$ docker logs sas-backend

After this I get this error

[email protected] server /app
node src/server.js

internal/modules/cjs/loader.js:883
throw err;
^

Error: Cannot find module 'lowdb'
Require stack:

• /app/src/server.js
  at Function.Module._resolveFilename (internal/modules/cjs/loader.js:880:15)
  at Function.Module._load (internal/modules/cjs/loader.js:725:27)
  at Module.require (internal/modules/cjs/loader.js:952:19)
  at require (internal/modules/cjs/helpers.js:88:18)
  at Object. (/app/src/server.js:2:15)
  at Module._compile (internal/modules/cjs/loader.js:1063:30)
  at Object.Module._extensions..js (internal/modules/cjs/loader.js:1092:10)
  at Module.load (internal/modules/cjs/loader.js:928:32)
  at Function.Module._load (internal/modules/cjs/loader.js:769:14)
  at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:72:12) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [ '/app/src/server.js' ]
  }
  npm ERR! code ELIFECYCLE
  npm ERR! errno 1
  npm ERR! [email protected] server: node src/server.js
  npm ERR! Exit status 1
  npm ERR!
  npm ERR! Failed at the [email protected] server script.
  npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
  npm WARN Local package.json exists, but node_modules missing, did you mean to install?

npm ERR! A complete log of this run can be found in:
npm ERR! /home/node/.npm/_logs/2021-06-09T15_26_01_533Z-debug.log

Now I'm without ideas, I hope you can help me or fix the image :)

Thanks for your great work :)

Hi @popstas ,

Hope you are all well ! And Happy New Year to all !

I had a play with site-audit-seo and figured out some things to add:

• Slack notification (I already started to implement that one) but something more pluggable would be nice (Slack,Jira,Email,...)
• Estimate the audit time if follow sitemap is enabled with an average time of 1s per page
• Create a queue of urls ro crawl in redis or kafka per audits if the server restart to resume tasks
• Log errors while a request failed, it can be helpful to identify failing url rewriting rules

And again, you did an amazing job with this project. Thanks again and again :-)

Thanks for any insights or inputs on that

Cheers,
Luc Michalski

Im running a scan with:

site-audit-seo -u https://domain.com -c 3 --preset=seo -m 1000

It never gets to 1000 urls because it just stops after some time and then nothing happens.
Sometimes its after 200 urls, sometimes its after just 10 urls.
Theres no error or other info, so I just have to kill the process and try again.

How can I debug this?

JSON file: http://localhost:3001/data.json

Dev viewer: http://localhost:3000/?url=http://localhost:3001/data.json

Online viewer: https://viasite.github.io/site-audit-seo-viewer/?url=http://localhost:3001/data.json

Finish: 7.9 mins (0.14 sec per page)
events.js:287
throw er; // Unhandled 'error' event
^

Error: listen EADDRINUSE: address already in use :::3001
at Server.setupListenHandle [as _listen2] (net.js:1313:16)
at listenInCluster (net.js:1361:12)
at Server.listen (net.js:1449:7)
at Function.listen (/usr/local/lib/node_modules/site-audit-seo/node_modules/express/lib/application.js:618:24)
at module.exports (/usr/local/lib/node_modules/site-audit-seo/src/actions/startViewer.js:43:9)
at finishScan (/usr/local/lib/node_modules/site-audit-seo/src/scrap-site.js:476:13)
at async tryFinish (/usr/local/lib/node_modules/site-audit-seo/src/scrap-site.js:491:7)
at async module.exports (/usr/local/lib/node_modules/site-audit-seo/src/scrap-site.js:509:3)
at async start (/usr/local/lib/node_modules/site-audit-seo/src/index.js:217:5)
Emitted 'error' event on Server instance at:
at emitErrorNT (net.js:1340:8)
at processTicksAndRejections (internal/process/task_queues.js:84:21) {
code: 'EADDRINUSE',
errno: 'EADDRINUSE',
syscall: 'listen',
address: '::',
port: 3001
}