Related to <a href="https://docs.julialang.org/en/v1/devdocs/precompile_hang/" rel="no

Precompilation hangs indefinitely in 1.10 about jobschedulers.jl HOT 5 CLOSED

jClugstor commented on June 18, 2024

Precompilation hangs indefinitely in 1.10

from jobschedulers.jl.

Comments (5)

jClugstor commented on June 18, 2024 1

The issue happens for me when trying to precompile any package that uses JobSchedulers.jl.

The minimum working example that I have found is to make a new project in Julia

(@v1.10) pkg> generate JobSchedPrecompExample
  Generating  project JobSchedPrecompExample:
    JobSchedPrecompExample/Project.toml
    JobSchedPrecompExample/src/JobSchedPrecompExample.jl

Then I go in to the project I just created and activate it

shell> cd JobSchedPrecompExample

  Activating project at `~/Documents/Work/dev/JobSchedPrecompExample`

Then I add JobSchedulers

   Resolving package versions...
    Updating `~/Documents/Work/dev/JobSchedPrecompExample/Project.toml`
  [eeff360b] + JobSchedulers v0.8.2
    Updating `~/Documents/Work/dev/JobSchedPrecompExample/Manifest.toml`
  [1520ce14] + AbstractTrees v0.4.4
  [da1fd8a2] + CodeTracking v1.3.5
  [34da2185] + Compat v4.10.1
  [a8cc5b0e] + Crayons v4.1.1
  [9a962f9c] + DataAPI v1.15.0
  [e2d170a0] + DataValueInterfaces v1.0.0
  [ffbed154] + DocStringExtensions v0.9.3
  [5789e2e9] + FileIO v1.16.2
  [48062228] + FilePathsBase v0.9.21
  [eafb193a] + Highlights v0.5.2
  [82899510] + IteratorInterfaceExtensions v1.0.0
  [033835bb] + JLD2 v0.4.41
  [682c06a0] + JSON v0.21.4
  [eeff360b] + JobSchedulers v0.8.2
  [b964fa9f] + LaTeXStrings v1.3.1
  [1914dd2f] + MacroTools v0.5.12
  [1c23619d] + MyterialColors v0.3.0
  [bac558e1] + OrderedCollections v1.6.3
  [d96e819e] + Parameters v0.12.3
  [69de0a69] + Parsers v2.8.1
  [ef544631] + Pipelines v0.10.5
  [aea7be01] + PrecompileTools v1.2.0
  [21216c6a] + Preferences v1.4.1
  [08abe8d2] + PrettyTables v2.3.1
  [33c8b6b6] + ProgressLogging v0.1.4
  [189a3867] + Reexport v1.2.2
  [ae029012] + Requires v1.3.0
  [892a3eda] + StringManipulation v0.3.4
  [3783bdb8] + TableTraits v1.0.1
  [bd369af6] + Tables v1.11.1
  [22787eb5] + Term v2.0.5
  [5a1048b7] + Terming v0.2.6
  [3bb67fe8] + TranscodingStreams v0.10.2
  [3a884ed6] + UnPack v1.0.2
  [1cfade01] + UnicodeFun v0.4.1
  [0dad84c5] + ArgTools v1.1.1
  [56f22d72] + Artifacts
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [f43a241f] + Downloads v1.6.0
  [7b1f6079] + FileWatching
  [b77e0a4c] + InteractiveUtils
  [b27032c2] + LibCURL v0.6.4
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [a63ad114] + Mmap
  [ca575930] + NetworkOptions v1.2.0
  [44cfe95a] + Pkg v1.10.0
  [de0858da] + Printf
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA v0.7.0
  [9e88b42a] + Serialization
  [6462fe0b] + Sockets
  [fa267f1f] + TOML v1.0.3
  [a4e569a6] + Tar v1.10.0
  [8dfed614] + Test
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode
  [e66e0078] + CompilerSupportLibraries_jll v1.0.5+1
  [deac9b47] + LibCURL_jll v8.4.0+0
  [e37daf67] + LibGit2_jll v1.6.4+0
  [29816b5a] + LibSSH2_jll v1.11.0+1
  [c8ffd9c3] + MbedTLS_jll v2.28.2+1
  [14a3606d] + MozillaCACerts_jll v2023.1.10
  [4536629a] + OpenBLAS_jll v0.3.23+2
  [83775a58] + Zlib_jll v1.2.13+1
  [8e850b90] + libblastrampoline_jll v5.8.0+1
  [8e850ede] + nghttp2_jll v1.52.0+1
  [3f19e933] + p7zip_jll v17.4.0+2
Precompiling project...
  1 dependency successfully precompiled in 1 seconds. 40 already precompiled.

In the project I then just have using JobSchedulers

module JobSchedPrecompExample

using JobSchedulers

end # module JobSchedPrecompExample

Then I try precompiling and I get

(JobSchedPrecompExample) pkg> precompile
Precompiling project...
        Info Given JobSchedPrecompExample was explicitly requested, output will be shown live 

[pid 10555] waiting for IO to finish:
 Handle type        uv_handle_t->data
 timer              0x188ed40->0x7fc88d965600
This means that a package has started a background task or event source that has not finished running. For precompilation to complete successfully, the event source needs to be closed explicitly. See the developer documentation on fixing precompilation hangs for more help.

This has happened to me after clearing the precompilation cache, and I have tried it on two different machines.

The message points towards the docs at https://docs.julialang.org/en/v1/devdocs/precompile_hang/. So I think the problem is that in Julia 1.10, if a task is started during precompile, the precompilation will wait until any tasks are concluded. So if you start the scheduler in __init__ it will start a task that won't complete.

Actually the docs have an alternate solution. If you put
ccall(:jl_generating_output, Cint, ()) == 1 && return nothing at the start of the __init__ function, it will prevent it from initializing in the precompile, which should fix the problem, while allowing the scheduler to start at using JobSchedulers. Which I just tested and it seems to work fine that way, so I'll add it to my PR.

from jobschedulers.jl.

cihga39871 commented on June 18, 2024

Hi, thank you for the issue and pull request. Could you show your environment? I tried to repeat the precompilation hang using julia v1.10, but failed.

Here is my code:

               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.10.0 (2023-12-25)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

(@v1.10) pkg> status
Status `~/.julia/environments/v1.10/Project.toml`
  [91a5bcdd] Plots v1.39.0

(@v1.10) pkg> add JobSchedulers
   Resolving package versions...
    Updating `~/.julia/environments/v1.10/Project.toml`
  [eeff360b] + JobSchedulers v0.8.2
    Updating `~/.julia/environments/v1.10/Manifest.toml`
  [1520ce14] + AbstractTrees v0.4.4
  [da1fd8a2] + CodeTracking v1.3.5
  [a8cc5b0e] + Crayons v4.1.1
  [e2d170a0] + DataValueInterfaces v1.0.0
  [5789e2e9] + FileIO v1.16.2
  [48062228] + FilePathsBase v0.9.21
  [eafb193a] + Highlights v0.5.2
  [82899510] + IteratorInterfaceExtensions v1.0.0
  [033835bb] + JLD2 v0.4.41
  [eeff360b] + JobSchedulers v0.8.2
  [1c23619d] + MyterialColors v0.3.0
  [d96e819e] + Parameters v0.12.3
  [ef544631] + Pipelines v0.10.4
  [08abe8d2] + PrettyTables v2.3.1
  [33c8b6b6] + ProgressLogging v0.1.4
  [892a3eda] + StringManipulation v0.3.4
  [3783bdb8] + TableTraits v1.0.1
  [bd369af6] + Tables v1.11.1
  [22787eb5] + Term v2.0.5
  [5a1048b7] + Terming v0.2.6
  [3a884ed6] + UnPack v1.0.2
Precompiling project...
  2 dependencies successfully precompiled in 2 seconds. 168 already precompiled.

julia> using JobSchedulers

julia> scheduler_status()
┌ Info: Scheduler is running.
│   SCHEDULER_MAX_CPU = 32
│   SCHEDULER_MAX_MEM = 121309492838
│   SCHEDULER_UPDATE_SECOND = 0.05
│   JOB_QUEUE_MAX_LENGTH = 10000
└   SCHEDULER_TASK[] = Task (runnable) @0x00007f6240e48330
:running

julia> versioninfo()
Julia Version 1.10.0
Commit 3120989f39b (2023-12-25 18:01 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 32 × 13th Gen Intel(R) Core(TM) i9-13900K
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, goldmont)
  Threads: 1 on 32 virtual cores

from jobschedulers.jl.

jClugstor commented on June 18, 2024

Hey, thanks for the response.

This is very strange, when I first got the error, it was in a bigger project. Yesterday, I was able to replicate the hang just by creating a new project, adding JobSchedulers.jl and doing Pkg.precompile().

When I did the exact same thing today, made a new project, added JobSchedulers.jl and ran Pkg.precompile() it worked perfectly.

But when I run precompile on the project that I made yesterday, it does not work...

I suspect I might need to delete my precompilation cache for the original project...

from jobschedulers.jl.

cihga39871 commented on June 18, 2024

Thank you, Jadon. Could you please get back to me whether deleting the cache works? If not, I can install the similar dependencies of your project and test it. Eric

…

________________________________ From: Jadon Clugston ***@***.***> Sent: Thursday, January 4, 2024 1:15:17 PM To: cihga39871/JobSchedulers.jl ***@***.***> Cc: Jiacheng Chuan ***@***.***>; Assign ***@***.***> Subject: Re: [cihga39871/JobSchedulers.jl] Precompilation hangs indefinitely in 1.10 (Issue #11) Hey, thanks for the response. This is very strange, when I first got the error, it was in a bigger project. Yesterday, I was able to replicate the hang just by creating a new project, adding JobSchedulers.jl and doing Pkg.precompile(). When I did the exact same thing today, made a new project, added JobSchedulers.jl and ran Pkg.precompile() it worked perfectly. But when I run precompile on the project that I made yesterday, it does not work... I suspect I might need to delete my precompilation cache for the original project... — Reply to this email directly, view it on GitHub<#11 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AE4TY4GWPMSTYJN4CPGRIP3YM3PSLAVCNFSM6AAAAABBL4YADKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZXGQ3TKMJTGE>. You are receiving this because you were assigned.Message ID: ***@***.***>

from jobschedulers.jl.

cihga39871 commented on June 18, 2024

Thank you so much for investigating into this. Your pull request was merged into the main branch! Thanks for your time and effort!

from jobschedulers.jl.

Precompilation hangs indefinitely in 1.10 about jobschedulers.jl HOT 5 CLOSED

Comments (5)

Related Issues (11)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent