hlibbabii / log-recommender Goto Github PK
View Code? Open in Web Editor NEWSupporting logging activities by mining software repositories
License: MIT License
Supporting logging activities by mining software repositories
License: MIT License
Currently all the words that contain non-asci chars are considered as non-English. However there are English words like naïve and Café that contain non-asci chars.
Possible solution is to remove accents.
E.g.
Café -> Café
Naïve -> naive
logrec.dataprep.vocabsize.py
Currently, the vocabulary of the preprocessed datasets is calculated as the first step of language model training. Recently some logic has been added for calculation the vocab size using that script without training the whole data model. However, it is still quite a cumbersome procedure.
That's why it totally makes sense to have a separate script for this. An additional feature could be saving information about vocab size for 1,2,5, 15, 50 etc. percents of the dataset. Moreover, multiple files should be processed in parallel.
Along with vocabulary size, words that comprise the vocabulary and how frequently they occur should be returned. Saving this information for certain percentage of data is not necessary. Only the number of oov words (non-asci for the beginning) should be saved for different percents.
So, the subtasks would be:
logrec vocab build
) hlibbabii/log-recommender-cli#2Currently, the following line:
logger.info("The value is " + val);
will be parsed as
[ProcessableToken("logger"), '.', ProcessableToken('info'), StringLiteral([ ProcessableToken("The"), ProcessableToken("value"), ProcessableToken("is"), '+', ProcessableToken('val') ])]
In other words, logger
, info
are seen the same as any other identifier.
Since we are interested in logging statements, it would make sense to treat them in a special way.
So, instead, the original expression could be parsed as something like:
LogStatement(level=info, text=[ProcessableToken("The"), ProcessableToken("value"), ProcessableToken("is"), VarPlaceholder()], vars=[ProcessableToken('val')])
The dictionaries are stored under dicts/
directory.
It would be a good idea to review them, check where they are coming from, double-check the licenses under which they are distributed and if neccessary replace/remove them or add new dictionaries.
Examples:
'einstance', 'destructee', 'executionspecification', 'combinedfragment', 'interactionoperand', 'coregion', 'durationobservation', 'timeobservation', 'commentlink', 'constraintlink'
'propertyeditor', 'propertyeditorclass', 'readmethod', 'writemethod', 'customizerclass', 'lnsep', 'zic', 'zoneinfo', 'csyntax', 'representatives'
Namely: do initialization in one place; make use of loggers hierarchy (for now everything is root
)
Currently when training the model, we are feeding code to it from left to right.
It would be good to try doing it additionally from right to left
total tokens code
total tokens code (unique)
non-eng tokens code
non-eng tokens code (unique)
total tokens strings
total tokens strings(unique)
non-eng tokens strings
non-eng tokens strings (unique)
total tokens comments
total tokens comments (unique)
non-eng tokens comments
non-eng tokens comments (unique)
non-eng examples code
non-eng examples strings
non-eng examples comments
[x] Save information about file names parsed:
<project_name>.filenames files contain the information about filenames that were parsed into <project_name>.parsed projects (Line n in .<project_name>.filenames file corresponds to line n in <project_name>.parsed file)
e.g. if we have
`s my awesome class s`
we lose infornation about how this words were originally connected
myAwesomeClass
OR
my_awesome_class
OR
my_awesomeClass
Possible option is to prefix all files with <chunk>_
where <chunk>
is 0..999, e.g.:
70_proj5
70_proj5.parsed.repr
70_proj5.label
Currently, there is a bunch of projects. There are to possible reasons for that:
The statistics about 1) the percentage of projects without logs at all, 2) the proportion of files with and without logs within the projects that has some logging - should be continuously monitored.
import java.mypackage
int a() {
log.warn("!");
}
int b() {
log.info("?");
}
Given the file, the following positive case should be generated (consisting of 2 subcases for different directions):
--> import java.mypackage int a() {
} int b() { log.info("?"); } <--
Negative cases
--> import java.mypackage int a()
{ } int b() { log.info("?"); } <--
During parsing, some files fail to parse because of UnicodeDecodeError.
A possible reason is that those files are encoded not in UTF-8. If it is so, maybe there is a way to 'guess' the encoding, if not, we should calculate at least the percentage fo such files to include into the report
See travis build # 25
Currently, if we split some word as e.g. MyFavoriteClass
, we represent it as my <cc_sep> favorite <cc_sep> class
There is an idea to try an alternative way to mark split words:
<s> my favorite class <s>
Subtasks:
Currently travis ci builds take around 12 mins to run. The reason for that is the downloading of anaconda packages. It could be improved if there were a possibility to somehow cache anaconda dependencies
Such kind of logs can be found in different scripts (parsing, preprocessing, vocab calculation etc.)
INFO:root:Time elapsed: 568.12 s, estimated time until completion: 97.
Although they are fine if the execution starts from the beginning if the task has been interrupted and restarted, the predicted time is far ffrom true.
multilang_100_percent/train/1/android-sdk --> multilang_100_percent/train/android-sdk
See “Neural Machine Translation of Rare Words with Subword Units”
Sennrich, Haddow, Birch, ACL 2016 for details
We split camel case and underscore identifiers, mark (sub)words boundaries, so that bpe doesn't merge them, split everything into characters, give target vocabulary size as input to stop when it is reached. Maybe, we need one more condition to be met along with vocab size, so that frequent words that need to be represented as whole words are included in the final dictionary. As a result, we have a list of merges. At the test time, we split a word into characters and then reassemble it using that list.
Advantage of bpe over the training of a nn thaining: we can specify exactly which vocab size we want to get!
Examples from dataset:
'add$connection', 'short$connection', 'create$connection', 'short$pattern', '$feature', '$diagram'
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.