It should try to fetch those project files directly via url and pull dependency details with help of parsers.

Lessons learned from CpanCrawler:
If product has many submodules, then it seriously slows down saving data and the same data is duplicated.

TODO:

Save each subpackage as module in the Product model and also in version model as users may remove submodules between each release.

The Hex API is limited. We need an API key and the Hex crawler needs to be refactored that they that it's using an API key for every request to the Hex API. If possible the crawler should check how many requests remain and wait for N minutes if the rate limit is used of.

I noticed that most of CrawlWorker are duplicating the work method, which could be moved into parent class like the GosearchVersionWorker does: https://github.com/versioneye/crawl_r/blob/master/lib/versioneye/workers/crates_crawl_worker.rb

Bring the new Hex crawler to production.

It uses a different file name that is not included inside the crawler: COPYING

I suggested to rename the file to current github standards but no respnse so far.
icalendar/icalendar#170

Maybe the crawler should also look for COPYING file names.

The Github search also points out a lot more file names that could be helpful.
https://github.com/search?utf8=%E2%9C%93&q=COPYING+in%3Apath&type=Code&ref=searchresults

• All extensions for the current filenames not ony .txt, .md, but also .gpl, ...
• Sometimes the License is inside gpl.txt, gpl.md, apache.txt, ... {license name}.ending
• also often inside sub folders copying

What would be interesting here is how many of the crawled repositories are missing a license at the moment?

See e.g. https://github.com/copiousfreetime/hitimes/blob/master/LICENSE

Many packages on Nuget only provide a link to license but no license name. During the crawling we should check if the license link is matching with a well known source. I found for example 64K license objects in our db where the url was http://www.apache.org/licenses/LICENSE-2.0.html and the license name was nil. I updated them all and set the name to Apache-2.0. This logic should be part of the Nuget crawler. Check at least for this urls:

http://www.apache.org/licenses/LICENSE-2.0.html
https://www.apache.org/licenses/LICENSE-2.0.html
https://opensource.org/licenses/MIT

I got email notification for a Rust package i follow and i noticed it is now missing license details.

I digged into the problem and i noticed that the response model from the API has been changed - they moved all the licenses onto versions;

curl https://crates.io/api/v1/crates/tokio-core?api_key=XYZ | jq

docs: https://hackage.haskell.org/

nb! use Accept header to get JSON data;

System.Linq, System.Collections are still stuck at 4.0.11, but they should have 4.3.0

I refactored the Cpan crawler. Remove all Threads and using RabbitMQ directly. In generell everything seems to work. But somehow there is not a single Perl dependency in the db. Even not after 2000 packages have been crawled. Maybe a bug?

Some, specially older, Perl packages have no license information, but if you look into their file directory, there usually exists license file. Quite many of them are using a name Artistic;

Also research how those old packages refer to dependencies;

Here's example for Curses: https://metacpan.org/source/GIRAFFED/Curses-1.36

It's one of reasons why specs are slow - it must rebuild the token index each time and it takes time;

Fastest solutions is to move everything into own repo; But it doesnt speed up creation of index;

Better approach: use proper tools, refactor it into 2 processes: training/match, where the training-process will build and save model; and match-process will re-use already existing model;

@timgluz Every Rust package is available through a unique URL.

https://crates.io/crates/<PACKAGE>

For example:

https://crates.io/crates/aseprite

The crawler should create this links automatically so that we can display it on the VersionEye pages.

In the logs of the cpan crawler I can see stuff like this:

[wtf] Failed to parse JSON response from https://fastapi.metacpan.org/v1/release/FLORA/Catalyst-Runtime-5.80009?join=author - execution expired (pid:197)

But if I try the URL in the browser I'm getting back a valid JSON in less then 1 second. Maybe a redirect issue?

It would be simpler and more robust to use persistent MQs when paginating over releases.

According to go-search, there're 2000 (low priority!!) packages hosted on Bitbucket and it would nice if we could pull Go dependencies from there too like we're doing for Github;

I found out that the url of API for CPAN crawler has been changed and old url doesnt work anymore.

ToDO:

• change url
• check has data from API changed
• update response fixtures in the spec/fixtures/files/cpan
• run crawler for extended period of time and check does scrolling work

API docs: https://github.com/metacpan/metacpan-api/blob/master/docs/API-docs.md

License file: https://github.com/sparklemotion/nokogiri/blob/master/LICENSE.txt
Somehow the algorithm has problems with the file contents.

Caused by line breaks possibly:

to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,

https://github.com/versioneye/crawl_r/blob/master/lib/versioneye/crawlers/license_crawler.rb, line 370

The matcher should be made more robust to line breaks.

Hello,

Here's my current use case: I use versioneye to track the dependencies of internal projects (mostly PHP-related). Obviously we do not use all the packages available on packagist. We would also like not to depend on VersionEye public API to obtain packages information. However, crawling all of packagist while we're probably interested in 500-1000 packages seems a bit excessive.

Thus, it would probably make sense to have tracked projects dictate which packages should be crawled. Then, the crawler/producer would only have to walk through these packages in order to fetch a minimal list of metadata.

The complete test suite quiet slow and some tests are failing. I guess the tests for bower are slow. Please review & refactor them. Make the test work again and if possible improve the speed.

Here's a rough idea for futher research:

use some dockerized Go pkg-manager to generate project file with dependencies and version label/commit sha by giving an url of any git repo.

@timgluz One of the newly added tests is failing:

  2) PythonLicenseDetector detecting standard spdx license texts ignores popular non-sense
     Failure/Error: expect(spdx_id).to eq('GNU')

       expected: "GNU"
            got: nil

       (compared using ==)
     # ./spec/versioneye/utils/python_license_detector_spec.rb:58:in `block (3 levels) in <top (required)>'

For some packages the release dates seem to be wrong. For example for this one it is wrong. Release 116 years ago.

@timgluz Why we don't have Author information for Rust packages? Is it not included in the REST API?