This project is not optimized. We should use esbuild to minify the code.

Example: https://gateway.us1.storjshare.io/

401 is Unauthorized. It should be a valid link.

The jest test does not terminate due to asynchronous operations.

${ was fixed in #69 , but $ separately should be skipped and { separately.

This project can be slow on large projects. Right now, it checks one link at a time to prevent hitting rate limits. But there should be an option to increase the amount of links are checked at once.

This was brought up here. It would be cool if link-inspector runs like a github action.

When link-inspector finds localhost, it returns a false positive. localhost should be skipped instead.

Any url that has "api." is probably a false positive. So maybe they should all be skipped?

Examples:

https://api.pythagora.io/telemetry

This project should have tests.

The program scans images for links. Returning false positives and slowing down the program. Images should be skipped over.

The project has grown immensely since the original tests where written. The tests should be updated to include false positives such as placeholder urls, redirect urls, and urls deemed to be skipped.

The link:

https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume?url=https://github.com/geekan/MetaGPT

The error:

node:_http_client:183
throw new ERR_INVALID_PROTOCOL(protocol, expectedProtocol);
^

TypeError [ERR_INVALID_PROTOCOL]: Protocol "vscode:" not supported. Expected "http:"
at new ClientRequest (node:_http_client:183:11)
at Object.request (node:http:103:10)
at ne.send_request (/home/justin/Git/link-inspector/dist/index.js:24:429723)
at ClientRequest. (/home/justin/Git/link-inspector/dist/index.js:24:430695)
at Object.onceWrapper (node:events:634:26)
at ClientRequest.emit (node:events:519:28)
at HTTPParser.parserOnIncomingClient [as onIncoming] (node:_http_client:693:27)
at HTTPParser.parserOnHeadersComplete (node:_http_common:119:17)
at TLSSocket.socketOnData (node:_http_client:535:22)
at TLSSocket.emit (node:events:519:28) {
code: 'ERR_INVALID_PROTOCOL'
}

This is a valid link on Jan 18 09:39 PST. But is returned as a broken link.

https://fonts.gstatic.com is a false positive

Currently link-inspector . output shows broken links like this:

$ link-inspector .
Broken Link: https://raw.githubusercontent.com/Homebrew/install/master/install
Broken Link: https://github.com/bitcoin/bips/blob/master/bip-0040.mediawiki
Broken Link: https://github.com/Joinmarket-Org/joinmarket-clientserver/blob/master/test/ygrunner.py
...

Would be useful if it showed also file and line number where broken URL is found, otherwise I need to do extra grep -R url steps afterwards for each URL.

.xml files are much more likely to return a false positive opposed to an actual broken link. We might consider skipping these files.

actions/checkout@v3 and actions/setup-node@v3 use node 16, which is deprecated. We should move to V4.

Documentation: https://github.blog/changelog/2023-09-22-github-actions-transitioning-from-node-16-to-node-20/.

https://goo.gl/ was a service google used to have to shorten urls.

Example: https://goo.gl/fbAQLP --> https://jestjs.io/docs/snapshot-testing

These return false positives, and should be fixed.

This returns a FORBIDDEN error. This should not be marked as a broken link.

https://fonts.googleapis.com/ is a false positive

https://example.com/ is a popular placeholder domain in projects. https://example.com/anything returns as a broken link, while it should be ignored.

FATAL ERROR: v8::ToLocalChecked Empty MaybeLocal
----- Native stack trace -----

 1: 0xde0c10 node::Abort() [node]
 2: 0xcb1f63 node::OOMErrorHandler(char const*, v8::OOMDetails const&) [node]
 3: 0x102a180 v8::api_internal::ToLocalEmpty() [node]
 4: 0xde801c  [node]
 5: 0x7fdf9ddceadd 

----- JavaScript stack trace -----

1: readFileSync (node:fs:453:20)
2: vs (/home/justin/Git/link-inspector/dist/index.js:2:1052)
3: /home/justin/Git/link-inspector/dist/index.js:13:504
4: Ir (/home/justin/Git/link-inspector/dist/index.js:13:458)
5: u (/home/justin/Git/link-inspector/dist/index.js:13:681)
6: /home/justin/Git/link-inspector/dist/index.js:13:708
7: processTicksAndRejections (node:internal/process/task_queues:95:5)


Aborted (core dumped)

This error happens when running https://github.com/ubccsss/ubccsss.org on a Intel i5-8365U with 16G of ram. This is a out of RAM error. The solution is to use pipes.

Documentation: https://nodejs.org/docs/v0.5.10/api/streams.html#stream.pipe
Stack over flow on the issue: https://stackoverflow.com/questions/7883698/node-js-is-running-out-of-memory-on-large-bit-by-bit-file-read

While the README.md is great for learning how to use link-inspector, a website with in-depth documentation will aid in developing link-inspector. Plus, a github pages website is free.

Documentation:
https://pages.github.com/
https://about.readthedocs.com/?ref=readthedocs.com

In python files, % is used to put variables in urls. The regex should be updated to skip urls that end with %( , just like #70.

Example: https://build.getmonero.org/downloads/monero-%(gitversion)s-%(platform)s.tar.gz

Example: https://nodejs.org/

This is a valid link that redirects to https://nodejs.org/en The program thinks this is a broken link. Urls that require redirects return false positives.

https://example.org is also a popular placeholder domain. It should be skipped just like https://example.com.

Very similar fix to #79

Program gives this error if a file being scanned is larger than 512MB. 0x1fffffe8 is exactly 512MB.

Here is a stack overflow with possible solutions: https://stackoverflow.com/questions/68230031/cannot-create-a-string-longer-than-0x1fffffe8-characters-in-json-parse

Example: https://react-builds.vercel.app/commits/${headSha}/files/${result.path}?compare=${baseSha}

${} is used in languages like JavaScript to include variables in the url. We should update the Regex to omit links that are followed by "${" to reduce false positives.

Example: https://registry.npmjs.org/chalk/-/chalk-3.0.0.tgz

This is a valid link, it downloads a file. But the program acts as if this is a broken link.

The program terminates before it can write broken link to a file. This is the problem with async. Sometimes a broken link will be missed.

Example: https://github.com/ubccsss/ubccsss.org.git

This is a valid link that redirects to https://github.com/ubccsss/ubccsss.org The program thinks this is a broken link.

Summery

Many programming languages will have the url as a string and add to the url using +. We should update the regex so that urls followed by " + are skipped.

Example

string url = "https://github.com/nzgamer41/tpgamemods/raw/master/" + m.GUID + ".zip";

Files go to "output/" + path/to/file/or/directory. Must fix.

Links after xsi are usually apart of the deep net and not the surface net. These are usually working links, but they come up as broken in this project. To reduce the number of false positives, links after xsi:* should be skipped.

Example: xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"

This is very similar to PR #66

This project uses link-check as a dependency. link-check is bloated and has features we don't need. We should move over to needle.