asah / codesmell Goto Github PK

Currently, Keras code uses Lambda functions for encoding the characters, which breaks two things:

1. can't run keras-js (in browser prediction): https://github.com/transcranial/keras-js
2. trained models (.hdf5) are specific to the given encoding

The code to replace:

def binarize(x, sz=97):
import tensorflow as tf2
return tf2.to_float(tf2.one_hot(x, sz, on_value=1, off_value=0, axis=-1))

def binarize_outshape(in_shape, sz=97):
return in_shape[0], in_shape[1], sz
...
embedded = Lambda(binarize, output_shape=binarize_outshape)(line_of_code_input)

Currently, saved models (.hdf5 files) are hard-coded to a given --max-len-per-line and --max-lines-per-commit setting, because these values are embedded in the tensor-shapes in the neural net architecture.

Subtly, we can't just use a large value for these settings because of GPU RAM requirements.

This issue will "go away" in <1 year as GPU hardware radically improves.

TBD if we can find enough training data.

there's plenty of code, but the problem is separating browser JS from backend (Node) JS.

one solution: manually identify frontend vs backend repos... e.g. create two language-directories, backend_js (with node.js libraries) and browser_js (with frontend frameworks). Common libraries can be duplicated in each.

esp useful for running predictions... here's what I'm seeing so far: (El Capitan, 10.11.6)

$ cat c_and_cpp/*0 c_and_cpp/*0|bunzip2 -c | ./codesmell.py -n 100 --confusion --no-array --no-details -c example-models/example-c_and_cpp--ep32--va76.hdf5 predict
/bin/sh: nvidia-smi: command not found
('Checkpoint:', 'example-models/example-c_and_cpp--ep32--va76.hdf5')
RuntimeError: module compiled against API version 0xa but this version of numpy is 0x9
Traceback (most recent call last):
File "./codesmell.py", line 402, in
import pandas as pd, numpy as np, tensorflow as tf
File "/usr/local/lib/python2.7/site-packages/tensorflow/init.py", line 24, in
from tensorflow.python import *
File "/usr/local/lib/python2.7/site-packages/tensorflow/python/init.py", line 49, in
from tensorflow.python import pywrap_tensorflow
File "/usr/local/lib/python2.7/site-packages/tensorflow/python/pywrap_tensorflow.py", line 52, in
raise ImportError(msg)
ImportError: Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/tensorflow/python/pywrap_tensorflow.py", line 41, in
from tensorflow.python.pywrap_tensorflow_internal import *
File "/usr/local/lib/python2.7/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in
_pywrap_tensorflow_internal = swig_import_helper()
File "/usr/local/lib/python2.7/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
_mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
ImportError: numpy.core.multiarray failed to import

Failed to load the native TensorFlow runtime.

See https://www.tensorflow.org/install/install_sources#common_installation_problems

for some common reasons and solutions. Include the entire stack trace
above this error message when asking for help.