juliainterop / nbinclude.jl Goto Github PK

I'm able to add and use the NBInclude package on my system. But when I try to add the NBInclude package in travis-ci.com, it core dumps. You can see the problem at https://github.com/PsuAstro528/HomeworkTemplate/tree/v0.7
and
https://travis-ci.com/PsuAstro528/HomeworkTemplate/builds/93116947

Note that since this is for classroom use, student's won't be creating a package for every assignment. Hence I wrote a simpler parser to have travis add packages that are used for the exercise based on a REQUIRE file.

Here's the .travis.yml file
language: julia
os:

• linux
  julia:
• 1.0

script:

• julia test/install_required_pacakges.jl
• julia test/runtests.jl

In essence the test/install_required_packages.jl file
is just doing
import Pkg
Pkg.add("Glob")
Pkg.add("NBInclude")

Is this something that can be fixed by NBInclude? Or is it deeper? Any suggestions?
Thanks.

First of all, I love the new soft global scope. But is NBInclude.jl using it the same was as IJulia?

Here is my test case. Create a cell with

n = 4
for i=1:2
    @show i
    n+= i
end
n

Then another, separate, code cell beneath it with just

n

If you run the two cells in Jupyter, it works great. No globals required, and the second n correctly captures the modified value, as you need for any notebook style programming.

But if I run that same notebook with NBInclude.jl it says

julia> @nbinclude("Untitled.ipynb")

ERROR: LoadError: UndefVarError: n not defined
Stacktrace:
 [1] top-level scope at /mnt/c/Users/jlperla/Documents/projects/Untitled.ipynb:In[4]:3 [inlined]
 [2] top-level scope at ./none:0
 [3] include_string(::Module, ::String, ::String) at ./loading.jl:1002
 [4] my_include_string(::Module, ::String, ::String, ::Nothing) at /home/jlperla/.julia/packages/NBInclude/YVOMm/src/NBInclude.jl:31
 [5] #nbinclude#1(::Bool, ::UnitRange{Int64}, ::Regex, ::typeof(identity), ::Function, ::Module, ::String) at /home/jlperla/.julia/packages/NBInclude/YVOMm/src/NBInclude.jl:83
 [6] nbinclude(::Module, ::String) at /home/jlperla/.julia/packages/NBInclude/YVOMm/src/NBInclude.jl:54
 [7] top-level scope at none:0
in expression starting at /mnt/c/Users/jlperla/Documents/projects/Untitled.ipynb:In[4]:2

For my packages

(v1.0) pkg> st
    Status `~/.julia/environments/v1.0/Project.toml`
  [c52e3926] Atom v0.7.4
  [0c46a032] DifferentialEquations v5.3.0
  [31c24e10] Distributions v0.16.2
  [2fe49d83] Expectations v1.0.1
  [7073ff75] IJulia v1.10.0
  [a98d9a8b] Interpolations v0.8.0
  [0db19996] NBInclude v2.0.1
  [429524aa] Optim v0.17.0
  [d96e819e] Parameters v0.9.2
  [91a5bcdd] Plots v0.20.1
  [295af30f] Revise v0.7.3

(I tried to upload the notebook, but github won't let me)

I think it is viable to make this work with Revise.jl,
so editting another notebook causes it to reinclude it.

I would require writing
a custom function to do the tracking,
based on applying
https://github.com/timholy/Revise.jl/blob/dd6e970a23e0b998883f814742723bf53b88f57f/src/Revise.jl#L505-L516

just to the code cells.
but that wouldn't be too hard.

This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

When generating and evaluating notebooks with https://github.com/fredrikekre/Literate.jl via a scrip which is not in the output directory, then @nbinclude generates a path which contains the path of the Literate script and not the path of the notebook-directory. I'm pretty sure that this is a NBInclude issue, but happy to move to Literate.jl if that is not the case.

To reproduce I setup a repo: https://github.com/mauro3/NBinclude-issue-tmp.jl

After cloning that repo, reproduce with:

$ julia --project

julia> using Pkg; Pkg.instantiate()

julia> include("gen-notebook.jl")

Backtrace:

julia> include("gen-notebook.jl")
[ Info: generating notebook from `~/julia/issues/literate-nbinclude/scripts/00.jl`
[ Info: executing notebook `00.ipynb`
[ Info: writing result to `~/julia/issues/literate-nbinclude/notebooks/00.ipynb`
[ Info: generating notebook from `~/julia/issues/literate-nbinclude/scripts/lit-script.jl`
[ Info: executing notebook `lit-script.ipynb`
┌ Error: error when executing notebook based on input file: `~/julia/issues/literate-nbinclude/scripts/lit-script.jl`
└ @ Literate ~/.julia/packages/Literate/VQn4b/src/Literate.jl:748
ERROR: LoadError: LoadError: SystemError: opening file "~/julia/issues/literate-nbinclude/00.jl": No such file or directory
in expression starting at ~/julia/issues/literate-nbinclude/notebooks/lit-script.ipynb:2
when executing the following code block from inputfile `~/julia/issues/literate-nbinclude/scripts/lit-script.jl`

```julia
using NBInclude
@nbinclude("00.jl")
```


Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] execute_block(sb::Module, block::String; inputfile::String, fake_source::String)
    @ Literate ~/.julia/packages/Literate/VQn4b/src/Literate.jl:866
  [3] execute_notebook(nb::Dict{Any, Any}; inputfile::String, fake_source::String)
    @ Literate ~/.julia/packages/Literate/VQn4b/src/Literate.jl:764
  [4] (::Literate.var"#38#40"{Dict{String, Any}})()
    @ Literate ~/.julia/packages/Literate/VQn4b/src/Literate.jl:744
  [5] cd(f::Literate.var"#38#40"{Dict{String, Any}}, dir::String)
    @ Base.Filesystem ./file.jl:112
  [6] jupyter_notebook(chunks::Vector{Literate.Chunk}, config::Dict{String, Any})
    @ Literate ~/.julia/packages/Literate/VQn4b/src/Literate.jl:743
  [7] notebook(inputfile::String, outputdir::String; config::Dict{Any, Any}, kwargs::Base.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:execute,), Tuple{Bool}}})
    @ Literate ~/.julia/packages/Literate/VQn4b/src/Literate.jl:680
  [8] top-level scope
    @ ~/julia/issues/literate-nbinclude/gen-notebook.jl:6
  [9] include(fname::String)
    @ Base.MainInclude ./client.jl:476
 [10] top-level scope
    @ REPL[7]:1
in expression starting at ~/julia/issues/literate-nbinclude/gen-notebook.jl:6

julia>

What happens:

• Literate.jl tries to evaluate the generated notebook's cell via https://github.com/fredrikekre/Literate.jl/blob/60c08dc23e3f6dc6fb2ae04571f9f6e8a0c6d2fa/src/Literate.jl#L862 Of note is that it gets executed within the right directory, namely notebooks for this example, see https://github.com/fredrikekre/Literate.jl/blob/60c08dc23e3f6dc6fb2ae04571f9f6e8a0c6d2fa/src/Literate.jl#L743
• this then goes to NBInclude.jl to evaluate the @nbinclude call, at https://github.com/stevengj/NBInclude.jl/blob/d1cff61be648d2f8400f3eb4bc0509f622917088/src/NBInclude.jl#L65 the bug occurs. The path variable changes from "00.jl" to ".../literate-nbinclude/00.jl". I'm not sure what Base._include_dependency(m, path) is supposed to do but here it should return ".../literate-nbinclude/notebooks/00.jl"

I've encountered it a few times now that I wanted to include a notebook which produces plots (with PyPlot), but the include is not for the plots it produces but for the functions it contains. Is there a way to suppress the plot output in the included location without having to comment them out in the source notebook?

Hi there.

With JupyterLabs capability to add arbitrary metadata-tags to cells, I think it would be cool if we could filter included cells based on these Tags.

I have actually already made a modified version of the module with this feature implemented through two new keywords in the nbinclude-function, excludeTags and includeOnlyTags, complete with documentation and tests. But since I dont know much about how Github works I am not quite sure how to properly propose these changes. For now, I will include the changed function in this Issue.

Any Feedback is appreciated.

function nbinclude(m::Module, path::AbstractString;
                   renumber::Bool=false,
                   counters = 1:typemax(Int),
                   regex::Regex = r"",
                   anshook = identity,
                   softscope::Bool=false,
                   includeOnlyTags::_TagListType = nothing,
                   excludeTags::_TagListType =nothing,
    )
   # act like include(path), in that path is relative to current file:
   # for precompilation, invalidate the cache if the notebook changes:
    path, prev = @static if VERSION >= v"0.7.0-DEV.3483" # julia#25455
        Base._include_dependency(m, path)
    else
        Base._include_dependency(path)
    end
    
    includeOnlyTags = isnothing(includeOnlyTags) ? nothing : string.(includeOnlyTags) |> Set
    excludeTags = isnothing(excludeTags) ? nothing : string.(excludeTags) |> Set

    # similar to julia#22588, we assume that all nodes
    # where you are running nbinclude can access the filesystem
    nb = open(JSON.parse, path, "r")

    # check for an acceptable notebook:
    nb["nbformat"] == 4 || error("unrecognized notebook format ", nb["nbformat"])
    lang = lowercase(nb["metadata"]["language_info"]["name"])
    lang == "julia" || error("notebook is for unregognized language $lang")

    ret = nothing
    counter = 0 # keep our own cell counter to handle un-executed notebooks.
    for cell in nb["cells"]
        if cell["cell_type"] == "code" && !isempty(cell["source"])
            s = join(cell["source"])
            isempty(strip(s)) && continue # Jupyter doesn't number empty cells
            counter += 1
            occursin(shell_or_help, s) && continue
            cellnum = renumber ? string(counter) :
                      cell["execution_count"] == nothing ? string('+',counter) :
                      string(cell["execution_count"])
            counter in counters && occursin(regex, s) || continue
            
            if excludeTags ≠ nothing && haskey(cell["metadata"],"tags")
                if length(excludeTags ∩ cell["metadata"]["tags"]) > 0
                    continue
                end
            end
            
            if includeOnlyTags ≠ nothing 
                haskey(cell["metadata"],"tags") || continue
                if (includeOnlyTags ∩ cell["metadata"]["tags"]) ≠ includeOnlyTags
                    continue
                end
            end
            
            ret = Base.task_local_storage(_in_nbinclude, true) do
                my_include_string(m, s, string(path, ":In[", cellnum, "]"), prev, softscope)
            end
            anshook(ret)
        end
    end
    return ret
end

I've started using NBInclude for regression testing and it works well so far, since @test is interpreted normally, but I'd like to make a few improvements to nbinclude. NBInclude is currently quite lean and minimal, so the first question is whether you're interested in this at all, or if I should fork it into NBTesting.jl. I'm considering:

• Add a fresh_module=false option, that runs the notebook in a new module
• print_headers=false: print all markdown cells that start with # (useful for long-running tests)
• Perhaps skipping all code cells that start with #NBSKIP. That's already possible via the regex option, but it seems like a useful default.

What do you think? I can do the first and third without modifying NBinclude, so if you OK the second, I could just write the rest as a convenience function on my side.

I don't know enough about Julia's internals to know if it's possible, but it would be awesome if errors gave stack traces that pointed to the appropriate notebook and cell.

It would be nice to have that functionality, based on this post (to preserve markdown formatting, unlike Jupyter's built-in export): https://discourse.julialang.org/t/export-jupyter-to-jl-does-not-preserve-markdown-cells/57556/2?u=stevengj

@nbinclude("n1.ipynb",softscope=true)

gives an error on Julia 1.2.0 (and 1.3.0-rc4.1) on Win10:
ERROR: syntax: invalid keyword argument name "Array{Any, (2,)}[ :softscope, true]"

Changing to @nbinclude("n1.ipynb";softscope=true) (notice: semi-colon ;)
solves the problem.

Suggestion: change the README.md to reflect this (or else, change the code)

I use NBInclude in several of my packages, but lately it has started to fail because it is not compatible with Compat v3, for example

https://travis-ci.org/JuliaPOMDP/POMDPModels.jl/jobs/647754939?utm_medium=notification&utm_source=github_status

Seems like fixing this will require upgrading to Project.toml, etc. For now I think I'm just going to have to disable the tests that use NBInclude :(

Consider this attached file (please rename it as "tester.ipynb"): tester.txt.

It has just 2 cells, with 6 lines of code in total. If you open it in Jupyter, it runs fine. But if you try to read it with the following command, you get an error.

@nbinclude("tester.ipynb")

The error says,

UndefVarError: s not defined

Stacktrace:
 [1] top-level scope at C:\Users\loew\loewCloud\CALO\2019-10\tester.ipynb:In[+2]:4
 [2] include_string(::Module, ::String, ::String) at .\loading.jl:1064
 [3] my_include_string(::Module, ::String, ::String, ::Nothing, ::Bool) at C:\Users\loew\.julia\packages\NBInclude\m4rfj\src\NBInclude.jl:29
 [4] #nbinclude#1(::Bool, ::UnitRange{Int64}, ::Regex, ::typeof(identity), ::Bool, ::typeof(nbinclude), ::Module, ::String) at C:\Users\loew\.julia\packages\NBInclude\m4rfj\src\NBInclude.jl:82
 [5] nbinclude(::Module, ::String) at C:\Users\loew\.julia\packages\NBInclude\m4rfj\src\NBInclude.jl:53
 [6] top-level scope at In[2]:1

I have Julia 1.2.0, with IJulia v1.20.0 and NBInclude v2.1.0.

I hope there is enough information here to decide if this is a bug in your package and how to fix it. If the bug is in my understanding/expectations instead, then I will be grateful to learn more. Either way, I have something extra to thank you for ... in addition to this useful module!

It would be nice to have a function available in notebooks to check if code is being run by NBInclude. I currently do something like this:

in_nbinclude() = contains(@__FILE__, r".:In\[.*\]$")

# then in later cells:

...
if !in_nbinclude()
    display(...)  # This won't display anything when run with nbinclude
end
...

This is related to #20. However using cell exclusions like in #20 requires setting regex in every @nbinclude call.

On Ubuntu-lts 32-bit, I get the following errors (for testing I deleted the .julia directory):

Version 0.5.0-dev+697 (2015-10-09 21:38 UTC)
Commit 7dcc01b* (1 day old master)

julia> Pkg.add("NBInclude")
INFO: Initializing package repository /home/mom/.julia/v0.5
INFO: Cloning METADATA from https://github.com/JuliaLang/METADATA.jl
ERROR: Base.Pkg.PkgError("unknown package NBInclude")
[inlined code] from pkg/entry.jl:53
in anonymous at task.jl:443
in sync_end at ./task.jl:409
[inlined code] from task.jl:418
in add at pkg/entry.jl:57
in add at pkg/entry.jl:78
[inlined code] from promotion.jl:225
in anonymous at pkg/dir.jl:31
in cd at file.jl:22
in cd at pkg/dir.jl:31
[inlined code] from pkg/dir.jl:25
in add at pkg.jl:27

Thanks.

Update: Fwiw, the Clone command works Pkg.clone("https://github.com/stevengj/NBInclude.jl.git"), so this may be related to metadata?