joshdk / licensor Goto Github PK

Many projects use an abbreviated form of certain licenses (specifically Apache 2.0). Add additional (non-SPDX?) examples for training/classification.

The goal is to classify licenses correctly, not enforce the license body perfectionism.

It is nice to have spdx.License.JenValue() for locality reasons, but it's ugly to have in the public API. Consider moving this helper somewhere, as the spdx.License struct won't change very often.

Using regex to split on whitespace is slow. Write benchmarks for current implementation and then optimize. Possible optimizations:

• Build set as map[[]byte]struct{} instead of map[string]struct{}
• Build set concurrently with multiple indices scanning text.
• (For SPDX data) Create index and persist to source at go generate time

Raw data currently pollutes the SPDX namespace, specifically, with auto-generated package comments. Move it into an internal sub-package.

license
 `- spdx
     `- internal
         `- index
             |- mit.gen.go
             |- apache.gen.go
             `- bsd.gen.go

Given an arbitrary license body, attempt to extract something that looks like a copyright notice, for attribution purposes.

Importing the project as license often conflicts during usage. Consider:

licenses := license.All()

for _, license := range licenses { // Oops, can't do that
    ...
}
``

Other libraries do this. Given the root directory of some project, look for files that look like LICENSE, LICENSE.txt, COPYING.txt, etc.

Would make more sense for this to be local to the module it's generating, and not at project top-level.

Explore alternate text comparison methods. Benchmark different methods. Possible optimizations include:

• https://en.wikipedia.org/wiki/Overlap_coefficient
• Concurrent licence confidence calculation
• Strip "bad" characters from source text (License probably doesn't contain runes like [{;:#+ etc)

Current index file is sufficiently large that Github refuses to display it (not counting raw mode). Split this file into multiple files for clarity. May also speed up incremental compilation? Benchmark the resulting increase in init() functions.

For example; If you only consider matches with > 0.9 confidence as "acceptable", you probably don't need to compare additional licenses after finding the first that meets the confidence bar.