jeffkinnison / shadho Goto Github PK

Bug: shadho attempts to copy work_queue.py to the same location.

How to Reproduce:

1. conda create --name shadho-test //create a new environment to reproduce this bug
2. conda activate shadho-test //activate this environment
3. conda install -c conda-forge ndcctools //install cctools from conda channel conda-forge
4. conda install pip //let conda handle pip install
5. python -m pip install shadho //pip install shadho to shadho-test environment
6. python -m shadho.installers.workqueue --prefix $CONDA_PREFIX //"glue" shadho and cctools

Error:
Traceback (most recent call last):
File "/afs/crc.nd.edu/user/t/tphung/anaconda3/envs/shadho-test/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/afs/crc.nd.edu/user/t/tphung/anaconda3/envs/shadho-test/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/afs/crc.nd.edu/user/t/tphung/anaconda3/envs/shadho-test/lib/python3.9/site-packages/shadho/installers/workqueue.py", line 209, in
main()
File "/afs/crc.nd.edu/user/t/tphung/anaconda3/envs/shadho-test/lib/python3.9/site-packages/shadho/installers/workqueue.py", line 202, in main
install_shadho_files(prefix, sp_prefix)
File "/afs/crc.nd.edu/user/t/tphung/anaconda3/envs/shadho-test/lib/python3.9/site-packages/shadho/installers/workqueue.py", line 176, in install_shadho_files
shutil.copy(os.path.join(source, 'work_queue.py'), sp_prefix)
File "/afs/crc.nd.edu/user/t/tphung/anaconda3/envs/shadho-test/lib/python3.9/shutil.py", line 418, in copy
copyfile(src, dst, follow_symlinks=follow_symlinks)
File "/afs/crc.nd.edu/user/t/tphung/anaconda3/envs/shadho-test/lib/python3.9/shutil.py", line 244, in copyfile
raise SameFileError("{!r} and {!r} are the same file".format(src, dst))
shutil.SameFileError: '/afs/crc.nd.edu/user/t/tphung/anaconda3/envs/shadho-test/lib/python3.9/site-packages/work_queue.py' and '/afs/crc.nd.edu/user/t/tphung/anaconda3/envs/shadho-test/lib/python3.9/site-packages/work_queue.py' are the same file

Temporary Solution: commenting out lines 176 and 177 in file https://github.com/jeffkinnison/shadho/blob/master/shadho/installers/workqueue.py solves the problem.

See title. Need documentation for all of this.

Problem: shadho module cannot be imported.
How to reproduce: (assuming that we have conda or miniconda)

conda create --name shadho-env
conda activate shadho-env
conda install pip
pip install shadho
python -c "import shadho"

The last command throws this error:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "~/anaconda3/envs/shadho-env/lib/python3.9/site-packages/shadho/shadho.py", line 8, in <module>
    from .configuration import ShadhoConfig
ImportError: attempted relative import with no known parent package

It looks like an import error, which hopefully is easy to fix.

Shadho seems to insist on finding its config file as there's no option for me to tell shadho to use the default configuration. It'd be nice if there's an option to do so, maybe in Shadho() object instantiation. Also shadho looks for its working directory .shadho when it doesn't need one. This is because I installed shadho via pip, and I installed pip via conda, so .shadho already lives in a conda environment, instead of the home directory.

See title. Need good docs up and running.

Per title. This is because shadho depends on package pyrameter, which uses object Sequence from library collections, which is deprecated since python3.3 and removed since python3.9. This issue is reproducible by running shadho-test.sh as specified in this issue, but changing the python version from 3.8 to 3.10 when creating the conda environment. Hope this helps!

See title. What are you doing with your life, Jeff???

Create an installed executable to install/uninstall Work Queue and the .shadho directory.

I was interested in checking out this tool, but when I found my way here I was disappointed to see that the documentation link seems to be broken. Can it be updated or fixed?

Currently, SHADHO requires a lot of manual configuration. Much of this can be pushed behind-the-scenes (e.g. Work Queue config) or be automated (e.g. a worker type function for running the training code).

There should be a default use mode that requires no input from the user besides which function to optimize.

Currently, trees are assigned to ccs under the assumption that the number of trees will be greater than the number of ccs. This will not always hold, and for the sake of robustness the assignment process must work in all cases.

Installing from PyPI does not create a .shadhorc file in $HOME. This should be created automatically on first use of SHADHO if it does not exist.

The code in both of these files is waaaaay less organized than in the rest of the package. Clean it up, Jeff.

max_tasks parameter in shadho object does not equate to the total number of runs.

For example, passed max_tasks = 2 but received 7 successes:

Experiment nn_mnist_test
-------------------------------------------------------------------
106 Trials	99 Pending	7 Successes	0 Errors
-------------------------------------------------------------------