error message suddenly popped up about papermill HOT 13 CLOSED

This may be related to a bug introduced in the latest IPython. If this is the problem, the fix is merged but not released from IPython. What version of python and IPython are you using?

from papermill.

Hi MSeal,

I was using python 3.6.4 and now 3.6.5. For IPython it has always been 6.1.0. Thanks for your help.

from papermill.

Hmm this is something else then since that bug we hit was introduced in 6.3.0. I wonder if nbconvert is assuming we're on the 6.2.x+ display version with the changed output format. What's your nbconvert version?

from papermill.

Nbconvert is 5.3.1. The odd thing is that I didn't update any packages. All of a sudden it went from working beautifully to giving me an error message every time.

from papermill.

Do you have an example input notebook / code snippet which reproduced the issue?

The comment that it started generating this error without upgrades suggests the notebook your parsing is of a different (or malformed) format. What tools is building the notebooks you're parsing? Since IPython changed its internal formatting some it might be that the notebook is saving with a slight format change we're not parsing correctly under the conditions you're in.

from papermill.

I have a for loop in which I pass the 3 variables (Sample, read1, read2) to the batch notebook. The parameterized notebook is renamed (output) and the executed notebook converted using nbcovert.

output = Sample + "_" + now + ".ipynb"
pm.execute_notebook('batch_042718.ipynb', output, dict(Sample=Sample, read1 = reads[0], read2 = reads[1]))
!jupyter nbconvert --to html $output

If there is any change I can think of of the notebooks I am parsing, it was that at the end of the notebook that I run to parse, originally I use pm.display to display a string ('prediction') that I recorded in the notebook. This does coincide with the error message.

(in the notebooks being parsed)
pm.display('prediction', prediction)

(in the notebook parsing the above notebooks)
nb = pm.read_notebook(output)
nb.display_output("prediction")

This is rather ugly so I later changed it to:

(in the notebooks being parsed)
pm.record('prediction', prediction)

(in the notebook parsing the above notebooks)
nb = pm.read_notebook(output)
print(nb.dataframe.iloc[3, 1])

from papermill.

cc @mpacer ^^ might be good to check this one out

from papermill.

Is this error still occurring for you @florathecat ? We had difficulty reproducing and there were some isues in ipython / nbconvert that have been merged since this was opened.

from papermill.

Hi @MSeal

Yes, it is still occurring for me. Every time I run the notebook it still gives me the same error message no matter I do it at home with a Ubuntu addition on a Virtual box or at work with a Ubuntu on a VMware player.

Thanks,

from papermill.

Could you post the notebook contents (or a paired down version which exhibits the behavior)? Also could you post your pip freeze? On the off chance it helps, are you executing this in a fresh virtualenv or conda environment (assuming so based on the multiple machines comment)?

I'm pretty sure something is damaged in the metadata but I can't get anything to reproduce. I generated notebooks locally with all combinations of inputs and processing I could think of based on the above inputs but never saw that warning. I'm also on a Ubuntu machine as well, but I don't think it's OS related.

from papermill.

Hi,
All my work is done from a conda environment (Anaconda 3.5.0.1 + python 3.6). I just created another 2 really simple notebooks (see below) in my windows 7 host (also a conda environment) and could replicate the same error. I agree with you 100% that something is damaged. The problem is I can't figure out what went wrong.

Notebook 1 'test.ipynb'
c = a+b
import papermill as pm
pm.record("c", c)
print(c)

Notebook 2 'papermill_test.ipynb'
import papermill as pm
pm.execute_notebook('test.ipynb', "test_after.ipynb", dict(a = 2, b = 3))
!jupyter nbconvert --to html test_after.ipynb
nb = pm.read_notebook('test_after.ipynb')
print(nb.dataframe.iloc[2, 1], '\n')

And sure enough it gives me exactly the same error message:

Input Notebook: test.ipynb
Output Notebook: test_after.ipynb
100%|████████████████████████████████████████████| 4/4 [00:09<00:00, 2.49s/it]
[NbConvertApp] Converting notebook test_after.ipynb to html
C:\ProgramData\Anaconda3\lib\site-packages\nbconvert\filters\datatypefilter.py:41: UserWarning: Your element with mimetype(s) dict_keys(['application/papermill.record+json']) is not able to be represented.
mimetypes=output.keys())
[NbConvertApp] Writing 252670 bytes to test_after.html

from papermill.

Like I said before the problem may come from the function pm.record(). I also created the following 2 similar notebooks and they don't seem to have a problem.

Notebook 1 'test.ipynb'
c = a+b
import papermill as pm
pm.display("c", c)
print(c)

Notebook 2 'papermill_test.ipynb'
import papermill as pm
pm.execute_notebook('test.ipynb', "test_after2.ipynb", dict(a = 2, b = 3))
!jupyter nbconvert --to html test_after2.ipynb
nb = pm.read_notebook('test_after2.ipynb')
nb.display_output('c')

No error message given during the execution.
Input Notebook: test.ipynb
Output Notebook: test_after2.ipynb
100%|████████████████████████████████████████████| 4/4 [00:09<00:00, 2.50s/it]
[NbConvertApp] Converting notebook test_after2.ipynb to html
[NbConvertApp] Writing 252786 bytes to test_after2.html

from papermill.

Going to close this issue since there's not been an update in a year and record isn't in papermill anymore.

from papermill.