error when do evaluating dev in training procedure about spert HOT 4 CLOSED

It looks as if your dev dataset contains samples with empty sentences, i.e. sentences without real tokens. Is this the case? If so, please remove these samples from your dataset.

from spert.

I used the program to check the data to find anything with empty token list, nothing was found.
then I use the following code to monitor if the input is empty, but nothing was found, do you have any sense why?
def _create_eval_sample(doc, max_span_size, context_size):
encoding = doc.encoding
token_count = len(doc.tokens)
if token_count == 0:
print ("EMPTY TOKENS")
for token in doc._tokens:
print (token._phrase)

# create entity candidates
entity_spans = []
entity_masks = []
entity_sizes = []
#print ("max:", max_span_size + 1)
for size in range(1, max_span_size + 1):
    #print ((token_count - size) + 1)
    for i in range(0, (token_count - size) + 1):
        #print ("size:", size)
        #print ("i:", i)
        span = doc.tokens[i:i + size].span
        entity_spans.append(span)
        entity_masks.append(create_entity_mask(*span, context_size))
        entity_sizes.append(size)

#print ("mask len:", len(entity_masks))
if len(entity_masks) == 0:
    print ("ZERO ENTITY MASKS")
    for token in doc._tokens:
       print (token._phrase)

from spert.

This is a bit hard to investigate without seeing your dataset. However, the exception "RuntimeError: stack expects a non-empty TensorList" should only occur if the entity_masks list is empty. Are you sure that len(entity_masks) is greater than 0 when the exception occurs? Can you please set 'sampling_processes = 0' in your configuration file, rerun the experiment and send me the output of your print statements when the exception occurs?

from spert.

Thanks for your suggestions, I already figured it out, there are empty tokens.

from spert.