About "tcmalloc: large alloc" message and training aborted about ncrfpp HOT 3 CLOSED

Awesome! Thanks for sharing this knowledge.

from ncrfpp.

You can refer to this: huggingface/transformers#4668

from ncrfpp.

Thank you for the reply. I managed to work out the problem. Sharing below my experience.

I tried running it on my local machine with 40GB ram and the issue persisted. It asked for a significantly large RAM which seemed unreasonable.

I then ran the code on the sample data and looked in the DATA SUMMARY STAT section where it showed the Label alphabet size: 18. But as in my screenshot below, from the run on my data, my Label alphabet size is almost the same as the Word alphabet size.

The problem was in my test data. I had not attached default O labels to the words. So perhaps the program was considering the words as the labels themselves. As an example, the former test file contained data as below:

Quantitative
Regular
Expressions
for
Arrhythmia
Detection
Algorithms

Logics
and
Games
for
True
Concurrency

And I fixed this with:

Quantitative O
Regular O
Expressions O
for O
Arrhythmia O
Detection O
Algorithms O

Logics O
and O
Games O
for O
True O
Concurrency O

This then showed an accurate Label alphabet size: 30 and no memory issues. :)

from ncrfpp.