mldl@ub1604:/ub16_prj/coarse2fine$ bash run.sh atis 0
Traceback (most recent call last):
File "preprocess.py", line 6, in
from path import Path
ImportError: No module named 'path'
Traceback (most recent call last):
File "train.py", line 11, in
import table
File "/home/mldl/ub16_prj/coarse2fine/logical/table/init.py", line 2, in
import table.Models
File "/home/mldl/ub16_prj/coarse2fine/logical/table/Models.py", line 10, in
from table.Utils import aeq, sort_for_pack
File "/home/mldl/ub16_prj/coarse2fine/logical/table/Utils.py", line 2, in
from path import Path
ImportError: No module named 'path'
Traceback (most recent call last):
File "evaluate.py", line 8, in
import table
File "/home/mldl/ub16_prj/coarse2fine/logical/table/init.py", line 2, in
import table.Models
File "/home/mldl/ub16_prj/coarse2fine/logical/table/Models.py", line 10, in
from table.Utils import aeq, sort_for_pack
File "/home/mldl/ub16_prj/coarse2fine/logical/table/Utils.py", line 2, in
from path import Path
ImportError: No module named 'path'
head: cannot open '/home/mldl/ub16_prj/coarse2fine/data_model/atis/dev_best.txt' for reading: No such file or directory
Traceback (most recent call last):
File "evaluate.py", line 8, in
import table
File "/home/mldl/ub16_prj/coarse2fine/logical/table/init.py", line 2, in
import table.Models
File "/home/mldl/ub16_prj/coarse2fine/logical/table/Models.py", line 10, in
from table.Utils import aeq, sort_for_pack
File "/home/mldl/ub16_prj/coarse2fine/logical/table/Utils.py", line 2, in
from path import Path
ImportError: No module named 'path'
mldl@ub1604:/ub16_prj/coarse2fine$

Hi,

I am very interested in your work
I wanted to know if it is possible to train the model on another dataset to convert text to java code?
if so how can i proceed?

thank you.

HI Li,
Currently the "lay" field gets whether or not there is condition in the required SQL query and if yes, how many conditions and which conditions (i.e. = , > , <) and this decides the "sketch" of the query,correct?

Now If I want to add LIMIT functionality can I append "lim" to the sketch so that "lay" field becomes say ['=','>','lim'] and then have a separate "limit decoder" just like cond decoder to calculate limit value?

Your thoughts/guidance is greatly appreciated

Thank you,
Shruti

Hi,
when I run preprocess.py , I got
File "preprocess.py", line 70, in <module>
main()
File "preprocess.py", line 65, in main
torch.save(train, open(os.path.join(opt.save_data, 'train.pt'), 'wb'))
File "/Users/SC/anaconda2/envs/python3/lib/python3.6/site-packages/torch/serialization.py", line 161, in save
return _with_file_like(f, "wb", lambda f: _save(obj, f, pickle_module, pickle_protocol))
File "/Users/SC/anaconda2/envs/python3/lib/python3.6/site-packages/torch/serialization.py", line 118, in _with_file_like
return body(f)
File "/Users/SC/anaconda2/envs/python3/lib/python3.6/site-packages/torch/serialization.py", line 161, in <lambda>
return _with_file_like(f, "wb", lambda f: _save(obj, f, pickle_module, pickle_protocol))
File "/Users/SC/anaconda2/envs/python3/lib/python3.6/site-packages/torch/serialization.py", line 232, in _save
pickler.dump(obj)
AttributeError: Can't pickle local object 'TableDataset.__init__.<locals>.filter_pred'

It happens when the code try to save train.pt in line 65:
torch.save(train, open(os.path.join(opt.save_data, 'train.pt'), 'wb'))
I found there is a comment in the code:# Can't save fields, so remove/reconstruct at training time. But I didn't understand how should I modify the code when training. Could you please tell me what I should do？
Thanks,

Hi Dong,

I am new to the semantic parsing research. But I have several issues related to your dataset.

I see in the "geoquery" dataset, you replaced the entity with numbers. Wouldn't that be a problem when comparing with other works as I see most works would just keep the entity in the utterance?

And where do you get the geoquery data annotated with the lambda meaning representation? I searched online and found most works use the geo data annotated with something like "( A , ( _capital ( A ) , _loc ( A , B ) , _const ( B , _stateid ( Iowa ) ) ) )". The two datasets, though, have the same utterance but with different types of annotations.

Thanks if you can resolve these issues for me.

Hi Li,Stephen

Could you please shed some light on cond_span_l, cond_span_r and lay fields as to what they are used for and how are they modelled(classifier or rnn etc) ? I would like to use the way the in which condition value is predicted for predicting LIMIT value.

Any insights would be greatly helpful!

Thank you,
Shruti

Hi Mr.Li,

I have download the entire code package. I would like to train a model for my own SQL table. From what I understand I need to-

1. Annotate my table using annotate.py
   2)Then use preprocess.py,train.py,evaluate.py.

But I am confused about the input data to annotate.py. In what format should I give my SQL table to annotate.py? And once that part runs through rest scripts should technically run ok, correct? Or do I need to make any other format changes?

Any guidance is greatly appreciated! Thank you!

Hello,

I am trying to run wikisql/preprocess.py and I get the error-
File "/home/coarse2fine/wikisql/table/ModelConstructor.py", line 13, in
from lib.query import agg_ops, cond_ops
When I looked in query.py-agg_ops and cond_ops are lists, how can we "import" lists?or am I misunderstanding something? Actually I could not find "lib" module in your repository hence i am using the one in WIKISQL(https://github.com/salesforce/WikiSQL) Is your query.py difference than the WikiSQL one?

Thank you,
Shruti

Hi Li,

when I want to preprocess some data new for the model, I have encountered an error which warns me about missing the "ent" field. I think there may be a omit in wikisql/annotate.py, and here is my solution:

def annotate(sentence, lower=True):
    global client
    if client is None:
        client = CoreNLPClient(default_annotators=['ssplit', 'tokenize', 'pos'])
    words, gloss, after, ents = [], [], [], []
    for s in client.annotate(sentence):
        for t in s:
            words.append(t.word)
            gloss.append(t.originalText)
            after.append(t.after)
            ents.append(t.pos)
    if lower:
        words = [w.lower() for w in words]
    return {
        'gloss': gloss,
        'words': words,
        'after': after,
        'ent': ents
    }

Hope it could help others, thanks :-D

Hi,
The annotate script for wikisql doesn't seem to add an ent field for questions, which seems to be expected by the TableDataset class. I added this by using a pos annotator with CoreNLPClient, however in your paper I see "We appended 10-dimensional part-of-speech tag vectors to em-
beddings of the question words in WIKI SQL. The part-of-speech tags were obtained by the spaCy
toolkit." - is this related to the ent field, and is there a different approach I should use to populate it?

Hi Li.

I'm having trouble with running the project.
While running preprocess.py, I got error when torch.save 'train Dataset' as below.
Could you tell me the solution if you know what it is?
I cannot find any generator objects in the dataset class.

Hi Li,

In the paper you mention that you used a bidirectional encoder with LSTM units for encoding inputs. But I do not see anything like say -> lstm = nn.LSTM(3, 3) in the code. Am I missing something?

Thank you,
Shruti

Hi Li,

Could you provide me with the Pytorch version in your project? I have a segmentation fault when running the code.

Please add license file to this repository.

mldl@ub1604:/ub16_prj/coarse2fine$ bash run.sh atis 0
Preparing training ...
Building Training...
Building Valid...
Building Test...
Building Vocab...
Saving train/valid/fields
run.sh: line 23: 25247 Segmentation fault (core dumped) CUDA_VISIBLE_DEVICES=$GPU_ID python3 train.py -root_dir "$WORK_DIR/data_model/" -dataset $DATANAME -rnn_size 250 -word_vec_size 200 -decoder_input_size 150 -layers 1 -start_checkpoint_at 60 -learning_rate 0.005 -start_decay_at 0 -epochs 100 -global_attention "dot" -attn_hidden 0 -lock_dropout -dropout 0.5 -dropout_i 0.5
head: cannot open '/home/mldl/ub16_prj/coarse2fine/data_model/atis/dev_best.txt' for reading: No such file or directory
mldl@ub1604:/ub16_prj/coarse2fine$

Hi Li,
I understand the use of cond_span_l and con_span_r but doesn't the loss get calculated for these variables during training? Then why do we need cond_span_l_loss and cond_span_r_loss fields?

Also, is the variable len_tgt_list (declared in wikisql/table/IO.py line 181) useful? I see it declared but I cannot see it used anywhere.
Thank you,
Shruti

I got the following error for batch size 5 and lower. I am trynig to use this for batch_size = 1 and I got the same thing. Any idea why?

python evaluate.py -model_path ../pretrained/pretrain.pt -batch_size 4
Evaluating model on the test set.
Loading model
Traceback (most recent call last):
  File "evaluate.py", line 89, in <module>
    main()
  File "evaluate.py", line 65, in main
    for batch in test_data:
  File "/home/ubuntu/anaconda3/envs/pytorch_p36_copy/lib/python3.6/site-packages/torchtext/data/iterator.py", line 151, in __iter__
    self.train)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36_copy/lib/python3.6/site-packages/torchtext/data/batch.py", line 27, in __init__
    setattr(self, name, field.process(batch, device=device, train=train))
  File "/home/ubuntu/anaconda3/envs/pytorch_p36_copy/lib/python3.6/site-packages/torchtext/data/field.py", line 188, in process
    tensor = self.numericalize(padded, device=device, train=train)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36_copy/lib/python3.6/site-packages/torchtext/data/field.py", line 310, in numericalize
    arr = self.tensor_type(arr)
RuntimeError: given sequence has an invalid size of dimension 2: 0

os information: the same with yours
cuda : 8.0
cudnn : 5.1

** issue**:
run.sh: line 27: 17688 Segmentation fault (core dumped) CUDA_VISIBLE_DEVICES=$GPU_ID python train.py -root_dir "$WORK_DIR/data_model/" -dataset $DATANAME -rnn_size 250 -word_vec_size 200 -decoder_input_size 150 -layers 1 -start_checkpoint_at 60 -learning_rate 0.005 -start_decay_at 0 -epochs 100 -global_attention "dot" -attn_hidden 0 -lock_dropout -dropout 0.5 -dropout_i 0.5

can you tell me what happened ? and how can i address it ?

I wonder whether the pretrain.pt is the pretrained model that can be used directly? I tried to run evaluate.py via loading it but got this error:

  File "/logical/table/Translator.py", line 112, in __init__
    checkpoint = torch.load(opt.model)
  File "/opt/conda/lib/python3.6/site-packages/torch/serialization.py", line 303, in load
    return _load(f, map_location, pickle_module)
  File "/opt/conda/lib/python3.6/site-packages/torch/serialization.py", line 469, in _load
    result = unpickler.load()
ModuleNotFoundError: No module named 'table'

Do you have any idea about it? Thanks a lot.

Hi Li,

I run the code myself and reproduced the results on Django and WikiSQL without any problem.

However, I wasn't able to reproduce the results on GEO and ATIS by running the commands given.

I got acc 86.4 on GEO and 86.2 on ATIS. Would you please take a look at this issue? Thanks.

Hi Mr. Li,
According to the paper(Fig 3) sketch encoding is connected to cond_decoder in the neural net.

But in Models.py line 440: lay_out = self.lay_classifier(q_ht)
after getting the output of lay classifier-> lay_out, lay_out is not used in the rest of the function for decoding conditions. Then how can we say that encoded layout is connected to condition decoder?

Also for lay classifier you used lay.vocab() but for lay_encoder you used cond_op.vocab. Both fields hold the same value. Any particular for using these different field names?

Thank you,
Shruti

I would like to understand how to use it database other than WikiSQL, I'm new to ML and would like to use it for querying attendance data. Can you please provide instructions to implement it?

Thanks for sharing this interesting framework. I am trying to run the codes to understand it better. In the (wikisql) evaluation function, based on what I understood, you call ParseResult.eval(pred, gold, sql_gold) to calculate the accuracy of the model. Inside the eval function, when you want to execute the pred_query, you change the conditions with the gold ones using recover_cond_to_gloss() function. Why do you do that? Is the resulted query still representing the predicted_query?

Hi Li,

Apologies to bother you with so many questions. I understand the output dim of 6 for agg_classifier since we get probabilties for 6 possible agg options.

But why is the output dimn of sel_match 1 and output dim of lay_classifier 2? Aren't there like 35 sketches to classify from? I have the models below
(sel_match): MatchScorer (
(score_layer): Sequential (
(0): Dropout (p = 0.5)
(1): Linear (500 -> 64)
(2): Tanh ()
(3): Linear (64 -> 1)
)
(log_sm): LogSoftmax ()

(lay_classifier): Sequential (
(0): Dropout (p = 0.5)
(1): Linear (250 -> 2)
(2): LogSoftmax ()
)

Thank you,
Shruti

Here is what I get when I run "python train.py" for WikiSQL. Any idea?

Traceback (most recent call last):
  File "train.py", line 205, in <module>
    main()
  File "train.py", line 201, in main
    train_model(model, train, valid, fields, optim)
  File "train.py", line 106, in train_model
    train_stats = trainer.train(epoch, report_func)
  File "/home/ubuntu/seq2sql/coarse2fine/wikisql/table/Trainer.py", line 143, in train
    loss, batch_stats = self.forward(batch, self.train_loss)
  File "/home/ubuntu/seq2sql/coarse2fine/wikisql/table/Trainer.py", line 107, in forward
    q, q_len, batch.ent, tbl, tbl_len, batch.tbl_split, batch.tbl_mask, cond_op, cond_op_len, batch.cond_col, batch.cond_span_l, batch.cond_span_r, batch.lay)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36_copy/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ubuntu/seq2sql/coarse2fine/wikisql/table/Models.py", line 459, in forward
    cond_context, _, _ = self.cond_decoder(emb, q_all, q_state)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36_copy/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ubuntu/seq2sql/coarse2fine/wikisql/table/Models.py", line 239, in forward
    outputs = torch.stack(outputs)
TypeError: stack(): argument 'tensors' (position 1) must be tuple of Tensors, not Tensor