Can path to .bidscoin be configured? about bidscoin HOT 62 CLOSED

Thanks for reporting this, I will try to release a new version today (I think the best and most robust version so far)

from bidscoin.

FYI, I changed BIDSCOIN_CONFIG to BIDSCOIN_CONFIGDIR to better reflect what happens / avoid config conflicts. Another way of seeing it is that the templates are also config files, not just config.toml

from bidscoin.

It's stated when you run:

$ bidscoin
usage: bidscoin [-h] [-l] [-p] [-i INSTALL [INSTALL ...]] [-u UNINSTALL [UNINSTALL ...]] [-d DOWNLOAD] [-t [TEST]] [-b BIDSMAPTEST]
                [--tracking {yes,no,show}] [-v]

BIDScoin is a toolkit to convert and organize raw data-sets according to the Brain Imaging Data Structure (BIDS)

The basic workflow is to run these two tools:

  $ bidsmapper sourcefolder bidsfolder        # This produces a study bidsmap and launches a GUI
  $ bidscoiner sourcefolder bidsfolder        # This converts your data to BIDS according to the study bidsmap

Default settings and template bidsmaps are stored in the ".bidscoin" folder in your home directory (you can modify
them to your needs with any plain text editor)

It's also stated in the docs (at least in the latest), but that can always be improved of course. The path to the config is now defined like this (in __init__.py):

# Create a BIDScoin user configuration directory if needed and load the BIDScoin user settings
configfile     = Path.home()/'.bidscoin'/__version__/'config.toml'
templatefolder = configfile.parent/'templates'

I don't see an easy way to configure the path to the config :-p. Where would you like it? Note that the user specific templates are copied from the central install, so if you change it there then all your users will have that version (and they can modify it individually). Or what is your use case?

from bidscoin.

I guess I can add a few lines to check if home exists, otherwise use the central location

from bidscoin.

My use case is a fully containerized conversion pipeline. In Nextflow, the home is not mounted by default and that causes the problem. I can mount it, so you don't have to change anything. I simply wasn't aware of these changes and need to document that myself. Wasn't BIDScoin writing to code directory before and is it still doing so?

from bidscoin.

It certainly writes to the code directory, all the log are stored there (not ideal, but I want to keep that close to the data and then there is no alternative). In fact, you may perhaps like the new bidscoiner.tsv file in there :-)

from bidscoin.

Also note that bidscoiner can now run in parallel. I don't know if you already did so in nextflow, but I guess that would lead to concurrency issues (e.g. multiple workers writing to the same participants.tsv file)

from bidscoin.

I don't see ant tsv file. Which version introduce it? I'm using 4.2.1.
I'm executing it for one participant in isolation.

from bidscoin.

No, it will be in the upcoming release. Since I plan to release it today or tomorrow, you have to be quick if you want to put something in :-)

from bidscoin.

Oh. Thanks a lot for this kind offer :-)

With config files, I usually see some priority lists, like config in your home, an env variable set to point to your config, config in /etc/..., etc. python-gitlab has a nice example: https://python-gitlab.readthedocs.io/en/stable/cli-usage.html#configuration-files

But I don't want to push this one. I simply didn't want to mount the entire home in the container.

from bidscoin.

That's a good example, I can add a check for environment variable that contains the path to the configuration folder (but the folder needs to be writable)

from bidscoin.

That was easy :-), you can now use export BIDSCOIN_CONFIG=/writable/path/to/config.toml if you like to use a different config file

from bidscoin.

I've seen other files in .bidscoin directory. Is the change for the config file enough?

/home/mpawlik/.bidscoin/
├── 4.2.1
│   ├── config.toml
│   └── templates
│       ├── bidsmap_dccn.yaml
│       ├── bidsmap_sst.yaml
│       └── schema.json
└── 4.3.0
    ├── config.toml
    ├── templates
    │   ├── bidsmap_bids2bids.yaml
    │   ├── bidsmap_dccn.yaml
    │   ├── bidsmap_sst.yaml
    │   └── schema.json
    └── usage
        └── bidscoin_4a39a33d0ba0040bb243175c70854fa5.db

from bidscoin.

I think so, but I haven't tested it :-)

from bidscoin.

It seems the configuration consists of more files.

(venv) mpawlik@erdwolf:~/Remote/anc/anc-data-acquisition (main) $ echo $BIDSCOIN_CONFIG
/home/mpawlik/Remote/anc/anc-data-acquisition/assets/bidscoin_config.toml
(venv) mpawlik@erdwolf:~/Remote/anc/anc-data-acquisition (main) $ bidscoin -t assets/bidscoin_map.yaml 
Creating BIDScoin configuration:
-> /home/mpawlik/.bidscoin/4.3.0/config.toml
-> /home/mpawlik/.bidscoin/4.3.0/templates/bidsmap_sst.yaml
-> /home/mpawlik/.bidscoin/4.3.0/templates/bidsmap_dccn.yaml
-> /home/mpawlik/.bidscoin/4.3.0/templates/bidscoin_map.yaml
-> /home/mpawlik/.bidscoin/4.3.0/templates/bidsmap_bids2bids.yaml
-> /home/mpawlik/.bidscoin/4.3.0/templates/schema.json
...

from bidscoin.

I don't like it but I could bind any host directory to $HOME in the container. This should do the trick. Playing with env variables and container is not optimal but I could also pass that.

from bidscoin.

It seems the configuration consists of more files.

Yes, but they are all automatically stored in the same folder as the config file. I just tested it and fixed a minor bug (so it should now work)

from bidscoin.

$ export BIDSCOIN_CONFIG=~/tmp/config.toml
$ bidsmapper raw bids
$ tree ~/tmp
/home/marzwi/tmp
├── config.toml
├── templates
│   ├── bidsmap_bids2bids.yaml
│   ├── bidsmap_dccn.yaml
│   ├── bidsmap_sst.yaml
│   └── schema.json
└── usage
    ├── bidscoin_2feb5e843ccb54a41416f963a6c88a5d.bak
    ├── bidscoin_2feb5e843ccb54a41416f963a6c88a5d.dat
    └── bidscoin_2feb5e843ccb54a41416f963a6c88a5d.dir

from bidscoin.

I installed the newest version with pip install --upgrade git+https://github.com/Donders-Institute/bidscoin.git@4f130416305c74d9f7884034e853b89fd6f32ad3 and unfortunately can't reproduce:

(venv) mpawlik@erdwolf:~/Remote/anc/anc-data-acquisition (main) $ echo $BIDSCOIN_CONFIG
/home/mpawlik/Remote/anc/anc-data-acquisition/assets/bidscoin_config.toml
(venv) mpawlik@erdwolf:~/Remote/anc/anc-data-acquisition (main) $ rm -r ~/.bidscoin/
rm: cannot remove '/home/mpawlik/.bidscoin/': No such file or directory
(venv) mpawlik@erdwolf:~/Remote/anc/anc-data-acquisition (main) $ bidscoin --version
Creating BIDScoin configuration:
-> /home/mpawlik/.bidscoin/4.3.0/config.toml
-> /home/mpawlik/.bidscoin/4.3.0/templates/bidsmap_sst.yaml
-> /home/mpawlik/.bidscoin/4.3.0/templates/bidsmap_dccn.yaml
-> /home/mpawlik/.bidscoin/4.3.0/templates/bidscoin_map.yaml
-> /home/mpawlik/.bidscoin/4.3.0/templates/bidsmap_bids2bids.yaml
-> /home/mpawlik/.bidscoin/4.3.0/templates/schema.json
BIDS-version:           1.9.0
BIDScoin-version:       4.3.0, NB: Your BIDScoin version is NOT up-to-date: 4.3.0 -> 4.2.1
(venv) mpawlik@erdwolf:~/Remote/anc/anc-data-acquisition (main) $ tree assets/
assets/
├── a003_bidscoin_map.yaml
├── bidscoin_config.toml
└── bidscoin_map.yaml

from bidscoin.

Despite the flag, that doesn't upgrade when installing from github (if the version nr doesn't change), try:

pip install --force-reinstall git+https://github.com/Donders-Institute/bidscoin

from bidscoin.

Building wheels for collected packages: bidscoin
  Building wheel for bidscoin (PEP 517) ... error
  ERROR: Command errored out with exit status 1:
   command: /home/mpawlik/Remote/anc/anc-data-acquisition/venv/bin/python3 /tmp/tmpdrqpb91d_in_process.py build_wheel /tmp/tmp30mkamn8                                     
       cwd: /tmp/pip-req-build-5evrj7jp                                                                                                                                    
  Complete output (77 lines):                                                                                                                                              
  No `packages` or `py_modules` configuration, performing automatic discovery.                                                                                             
  `flat-layout` detected -- analysing .                                                                                                                                    
  discovered packages -- ['bidscoin', 'bidscoin.heuristics', 'bidscoin.utilities', 'bidscoin.cli', 'bidscoin.bidsapps', 'bidscoin.plugins', 'bidscoin.schema', 'bidscoin.schema.objects', 'bidscoin.schema.meta', 'bidscoin.schema.rules', 'bidscoin.schema.rules.tabular_data', 'bidscoin.schema.rules.checks', 'bidscoin.schema.rules.files', 'bidscoin.schema.rules.sidecars', 'bidscoin.schema.rules.tabular_data.derivatives', 'bidscoin.schema.rules.files.deriv', 'bidscoin.schema.rules.files.common', 'bidscoin.schema.rules.files.raw', 'bidscoin.schema.rules.sidecars.derivatives']                                                                                                             
  running bdist_wheel                                                                                                                                                      
  running build                                                                                                                                                            
  running build_py                                                                                                                                                         
  running build_manpages                                                                                                                                                   
  generating man/bidscoin.1                                                                                                                                                
  Traceback (most recent call last):                                                                                                                                       
    File "/tmp/tmpdrqpb91d_in_process.py", line 280, in <module>                                                                                                           
      main()                                                                                                                                                               
    File "/tmp/tmpdrqpb91d_in_process.py", line 263, in main                                                                                                               
      json_out['return_val'] = hook(**hook_input['kwargs'])                                                                                                                
    File "/tmp/tmpdrqpb91d_in_process.py", line 204, in build_wheel                                                                                                        
      return _build_backend().build_wheel(wheel_directory, config_settings,                                                                                                
    File "/tmp/pip-build-env-y0z6mxxs/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 404, in build_wheel                                              
      return self._build_with_temp_dir(                                                                                                                                    
    File "/tmp/pip-build-env-y0z6mxxs/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 389, in _build_with_temp_dir                                     
      self.run_setup()                                                                                                                                                     
    File "/tmp/pip-build-env-y0z6mxxs/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 311, in run_setup                                                
      exec(code, locals())                                                                                                                                                 
    File "<string>", line 3, in <module>                                                                                                                                   
    File "/tmp/pip-build-env-y0z6mxxs/overlay/lib/python3.9/site-packages/setuptools/__init__.py", line 103, in setup                                                      
      return distutils.core.setup(**attrs)                                                                                                                                 
    File "/tmp/pip-build-env-y0z6mxxs/overlay/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 185, in setup                                               
      return run_commands(dist)                                                                                                                                            
    File "/tmp/pip-build-env-y0z6mxxs/overlay/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 201, in run_commands                                        
      dist.run_commands()                                                                                                                                                  
    File "/tmp/pip-build-env-y0z6mxxs/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands                                        
      self.run_command(cmd)                                                                                                                                                
    File "/tmp/pip-build-env-y0z6mxxs/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 963, in run_command                                                    
      super().run_command(command)                                                                                                                                         
    File "/tmp/pip-build-env-y0z6mxxs/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command                                         
      cmd_obj.run()                                                                                                                                                        
    File "/tmp/pip-build-env-y0z6mxxs/overlay/lib/python3.9/site-packages/wheel/bdist_wheel.py", line 368, in run                                                          
      self.run_command("build")                                                                                                                                            
    File "/tmp/pip-build-env-y0z6mxxs/overlay/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command                                          
      self.distribution.run_command(command)                                                                                                                               
    File "/tmp/pip-build-env-y0z6mxxs/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 963, in run_command
      super().run_command(command)
    File "/tmp/pip-build-env-y0z6mxxs/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/tmp/pip-build-env-y0z6mxxs/overlay/lib/python3.9/site-packages/setuptools/_distutils/command/build.py", line 131, in run
      self.run_command(cmd_name)
    File "/tmp/pip-build-env-y0z6mxxs/overlay/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
      self.distribution.run_command(command)
    File "/tmp/pip-build-env-y0z6mxxs/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 963, in run_command
      super().run_command(command)
    File "/tmp/pip-build-env-y0z6mxxs/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/tmp/pip-build-env-y0z6mxxs/overlay/lib/python3.9/site-packages/build_manpages/build_manpages.py", line 151, in run
      self.run_command(DEFAULT_CMD_NAME)
    File "/tmp/pip-build-env-y0z6mxxs/overlay/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
      self.distribution.run_command(command)
    File "/tmp/pip-build-env-y0z6mxxs/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 963, in run_command
      super().run_command(command)
    File "/tmp/pip-build-env-y0z6mxxs/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/tmp/pip-build-env-y0z6mxxs/overlay/lib/python3.9/site-packages/build_manpages/build_manpages.py", line 127, in run
      parser = get_parser(data['import_type'], data['import_from'], data['objname'], data['objtype'], data.get('prog', None))
    File "/tmp/pip-build-env-y0z6mxxs/overlay/lib/python3.9/site-packages/argparse_manpage/tooling.py", line 65, in get_parser
      return get_parser_from_file(import_from, objname, objtype, prog=prog)
    File "/tmp/pip-build-env-y0z6mxxs/overlay/lib/python3.9/site-packages/argparse_manpage/tooling.py", line 52, in get_parser_from_file
      module_loaded = load_file_as_module(filename)
    File "/tmp/pip-build-env-y0z6mxxs/overlay/lib/python3.9/site-packages/argparse_manpage/compat.py", line 48, in load_file_as_module
      return load_py_file(filename)
    File "/usr/lib/python3.9/runpy.py", line 268, in run_path
      return _run_module_code(code, init_globals, run_name,
    File "/usr/lib/python3.9/runpy.py", line 97, in _run_module_code
      _run_code(code, mod_globals, init_globals,
    File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
      exec(code, run_globals)
    File "bidscoin/cli/_bcoin.py", line 28, in <module>
      from bidscoin import check_version, __version__, bidsversion, bidsmap_template
    File "./bidscoin/__init__.py", line 79, in <module>
      bidsmap_template = Path(config['bidscoin']['bidsmap_template'])
  KeyError: 'bidscoin'
  ----------------------------------------
  ERROR: Failed building wheel for bidscoin
Failed to build bidscoin
ERROR: Could not build wheels for bidscoin which use PEP 517 and cannot be installed directly

uninstall and install also didn't help.

from bidscoin.

generating man/bidscoin.1

Ah, that's due to a bug (not mine) when building manpages (see here)
Remove that and it should work?

from bidscoin.

A fresh venv didn't help and gave the same error. I did:

(venv) mpawlik@erdwolf:~/Remote/anc/anc-data-acquisition (main) $ deactivate 
mpawlik@erdwolf:~/Remote/anc/anc-data-acquisition (main) $ rm -rf venv/
mpawlik@erdwolf:~/Remote/anc/anc-data-acquisition (main) $ python3 -m venv venv                                             
mpawlik@erdwolf:~/Remote/anc/anc-data-acquisition (main) $ source venv/bin/activate
(venv) mpawlik@erdwolf:~/Remote/anc/anc-data-acquisition (main) $ pip install git+https://github.com/Donders-Institute/bidscoin

I can't figure out where are the man pages. This is what I should delete, right?

Any of these?

$ man -w
/usr/local/man:/usr/local/share/man:/usr/share/man

from bidscoin.

Yeah, that's not my expertise either, I think it also depends on your distro. In my case they where build in the cwd (CentOS) or somewhere in my venv (OpenSuse). What if you do:

$ find -name bidscoin.1

from bidscoin.

I don't find it anywhere in my home :-( Neither /usr/local/ nor /usr/share/

from bidscoin.

I tried previous commits and pip install git+https://github.com/Donders-Institute/bidscoin.git@8ed9a53ce2202d2ef7520a84eb6642d4e519e91c succeeds without deleting any man pages, which I cannot find anywhere.

from bidscoin.

I don't think it's the ma pages issue here. Here's why installing the newest commits is failing:

684e293

File "./bidscoin/__init__.py", line 66, in <module>                                                                                                                    
      templatefolder = configfile.parent/'templates'                                                                                                                       
  AttributeError: 'str' object has no attribute 'parent'

89fb796

File "./bidscoin/__init__.py", line 66, in <module>                                                                                                                    
      templatefolder = configfile.parent/'templates'                                                                                                                       
  AttributeError: 'str' object has no attribute 'parent'

4f13041 (newest)

File "./bidscoin/__init__.py", line 79, in <module>                                                                                                                    
      bidsmap_template = Path(config['bidscoin']['bidsmap_template'])                                                                                                      
  KeyError: 'bidscoin'

from bidscoin.

And this is because my config file pointed by BIDSCOIN_CONFIG env variable was empty and therefore KeyError: 'bidscoin'. It is weird that this error is triggered when installing bidscoin. After filling in my config file, it installs.

from bidscoin.

AttributeError: 'str' object has no attribute 'parent'

Yes, I saw and fixed that

4f13041 (newest)

That bidscoin key is written in the config.toml file, could it be that writing the config.toml file failed?

from bidscoin.

Ah, you set an empty path, yes, that probably fails. Try unset BIDSCOIN_CONFIG

from bidscoin.

No, my config.toml file was empty and therefore the key could not be found in it. Once I added content to my config.toml file, bidscoin installed. I don't know what happens during installation but the bidscoin code is executed.

from bidscoin.

I have no idea what happened, but I now added some extra robustness to the code. I hope that solves it :-/

from bidscoin.

No, my config.toml file was empty and therefore the key could not be found in it. Once I added content to my config.toml file, bidscoin installed. I don't know what happens during installation but the bidscoin code is executed.

I'm getting more errors now, like:

$ bidscoin --version
/home/mpawlik/Remote/anc/anc-data-acquisition/venv/lib/python3.9/site-packages/bidscoin/__init__.py:81: RuntimeWarning: Missing template bidsmap: /home/mpawlik/.bidscoin/4.3.0/templates/bidsmap_dccn.yaml (see /home/mpawlik/Remote/anc/anc-data-acquisition/assets/bidscoin_config.toml)
  warnings.warn(f"Missing template bidsmap: {bidsmap_template} (see {configfile})", RuntimeWarning)

It still expects files in my home. But the installation copied the template files to the parent directory of BIDSCOIN_CONFIG. It seems that you copy these files during the installation but later expect the path in home.

I'm sorry for causing all this trouble and thank you for taking care of that so quickly.

from bidscoin.

No, thanks for testing this, I really appreciate it. I'll have a look

from bidscoin.

I'm wandering why do you need to copy all these files. The config file should be enough to configure bidscoin. If more files are necessary, how about add their paths to the config file instead of copying them.

from bidscoin.

In the center where I work, there is a single bidscoin installation that all users share. Copying the default templates to every user's home allows them to edit the default templates to their needs

from bidscoin.

It seems that you copy these files during the installation but later expect the path in home.

Only if BIDSCOIN_CONFIG is unset / empty?

from bidscoin.

When BIDSCOIN_CONFIG was set before the installation, the template files were copied to the parent directory of the file pointed by BIDSCOIN_CONFIG. However, errors occurred when executing bidscoin #226 (comment)

When BIDSCOIN_CONFIG was NOT set before the installation, the installation created ~/.bidscoin directory with all the files.

from bidscoin.

So in my container use case (Apptainer), bidscoin installation in the image created /root/.bidscoin/4.2.1/) directory with all the files. However, when executed (even with --no-home), bidscoin expects the files in my home. Shouldn't bidscoin internally use the path where it copies the files to, instead of $HOME?

Another weird thing is that I get these errors with dicomsort, which does not rely on .bidscoin directory.

I'm also confused now. For bidsmapper, should I use the config file to set my custom map file, or the --template option? What's the priority here?

from bidscoin.

Just to make sure your expectations are in line, using the environment variable is not permanent, bidscoin only uses that config file when the variable is non-empty, otherwise it will use the default

from bidscoin.

Another weird thing is that I get these errors with dicomsort, which does not rely on .bidscoin directory.

Perhaps that's a bug, let me check

from bidscoin.

bidscoin installation in the image created /root/.bidscoin/4.2.1/) directory with all the files.

These file are only created when bidscoin is imported (it is all done in __init__.py), i.e. when a bidscoin function is run. The pip installation itself really should not create anything? Running apptainer should be no problem if you pass the BIDSCOIN_CONFIG every time you run a bidscoin function

from bidscoin.

Do you get the same issue if you don't use apptainer?

from bidscoin.

bidscoin installation in the image created /root/.bidscoin/4.2.1/) directory with all the files.

These file are only created when bidscoin is imported (it is all done in __init__.py), i.e. when a bidscoin function is run. The pip installation itself really should not create anything? Running apptainer should be no problem if you pass the BIDSCOIN_CONFIG every time you run a bidscoin function

But it does 🤔

Bidscoin in Apptainer certainly doesn't remember where it saved the files. It keeps recreating .bidscoin directory if home is not mounted in. But it tries to do that in $HOME and errors out. I solved it with --scratch $HOME Apptainer option.

Relevant Apptainer issue: apptainer/apptainer#388

from bidscoin.

Do you get the same issue if you don't use apptainer?

Is #226 (comment) an answer?

from bidscoin.

But it does 🤔

Ah, I think building the manpages imports bidscoin, that's when it saves the config file

from bidscoin.

Nevertheless, it shouldn't matter, you can have multiple locations where config files are stored. As long as there is nothing in the environment, then the home directory is used, otherwise the location from the environment variable is used. Perhaps there is a weird interaction somewhere that I don't see, but otherwise I don't see how this doesn't work. I'll test a bit more.

from bidscoin.

It seems that my problems were triggered mainly by Apptainer and the hard-coded path to store .bidscoin directory.

I'll try to summarize a bit.

Outside of the Apptainer

It seems to work as intended now. Template files are copied to the parent of BIDSCOIN_CONFIG.

Inside of the container

version 4.2.1

I fix it with --scratch $HOME when Apptainer is executed with --no-home, because bidscoin keeps looking up $HOME although it isn't mounted.

current master version (cd409e1)

Adding --env BIDSCOIN_CONFIG=/root/.bidscoin/4.3.0/config.toml to Apptainer command should help, because bidscoin doesn't know where is .bidscoin without BIDSCOIN_CONFIG set. If it is not set, bidscoin expects .bidscoin in HOME, which for installation in Apptainer is not true (HOME always points to host home).

If config.toml needs to be overwritten, it can be mounted additionally (--bind /path/to/my/config.toml:/root/.bidscoin/4.3.0/config.toml). If BIDSCOIN_CONFIG is set to something else than /root/.bidscoin/4.3.0/config.toml, the parent directory has to be mounted.

In general

I should not need to set BIDSCOIN_CONFIG and care about the .bidscoin directory, especially if I use bidsmapper with --template option. The config file only sets this template file and the tracking. Usually, I can set the options in a config file, or provide them with CLI, with some priorities. Bidscoin could remember where it created .bidscoin directory, whether BIDSCOIN_CONFIG is set or not. Then, it could preset the options from the config file in there, and optionally overwrite them with values provided with CLI. This would make the behavior consistent.

from bidscoin.

Inside of the container

version 4.2.1

I fix it with --scratch $HOME when Apptainer is executed with --no-home, because bidscoin keeps looking up $HOME although it isn't mounted.

Ok, that's not good, I will make a fallback-fix for this (in $TMPDIR?)

current master version (cd409e1)

Adding --env BIDSCOIN_CONFIG=/root/.bidscoin/4.3.0/config.toml to Apptainer command should help, because bidscoin doesn't know where is .bidscoin without BIDSCOIN_CONFIG set.

So you pass a config file, that should work, but pointing it to /root puzzles me.

If it is not set, bidscoin expects .bidscoin in HOME, which for installation in Apptainer is not true (HOME always points to host home).

If it's not set, then bidscoin will create it in home, but I'll make that fallback-fix that should cover this

If config.toml needs to be overwritten, it can be mounted additionally (--bind /path/to/my/config.toml:/root/.bidscoin/4.3.0/config.toml). If BIDSCOIN_CONFIG is set to something else than /root/.bidscoin/4.3.0/config.toml, the parent directory has to be mounted.

Sorry, I'm not that used to working with containers, but I don't see why you have to bind that root path? Isn't passing --env BIDSCOIN_CONFIG=/some/mounted/folder/bidscoin/config.toml sufficient?

In general

I should not need to set BIDSCOIN_CONFIG and care about the .bidscoin directory, especially if I use bidsmapper with --template option. The config file only sets this template file and the tracking.

I agree that the config folder shouldn't be a user concern and I'm surely open for suggestions to improve things here..

Usually, I can set the options in a config file, or provide them with CLI, with some priorities.

Well, that sounds great but on the other hand, it is also very common for applications to simply keep their settings in a hidden file or directory in home (e.g. dcm2niix).

Bidscoin could remember where it created .bidscoin directory, whether BIDSCOIN_CONFIG is set or not.

I don't see how bidscoin can remember that without storing it somewhere on disk. Which is what the config file is

Then, it could preset the options from the config file in there, and optionally overwrite them with values provided with CLI. This would make the behavior consistent.

Currently I don't provide a tool or CLI options for editing/overruling the config file, the user needs to do that him/herself using a text editor (which is very easy and works very well). The behaviour should consistently be: Use the default config (in home or /tmp) or use the config specified in the environment variable.

Do you perhaps know what Path.home() returns in an apptainer that runs without $HOME?

from bidscoin.

Do you perhaps know what Path.home() returns in an apptainer that runs without $HOME?

Never mind, I found the answer. The problem I have is that tempfile.gettempdir() doesn't necessarily return a usable result...

from bidscoin.

If your container has a tempdir (doesn't it?) then I hope my last commit works as expected...

from bidscoin.

I just tested it in my own apptainer and there things now work (latest version from github) without setting any environment variable. So I guess that settles it, or?

from bidscoin.

INFO:    fuse2fs not found, will not be able to mount EXT3 filesystems
Creating BIDScoin configuration:
-> /tmp/.bidscoin/4.3.0/config.toml
-> /tmp/.bidscoin/4.3.0/templates/bidsmap_bids2bids.yaml
-> /tmp/.bidscoin/4.3.0/templates/bidsmap_dccn.yaml
-> /tmp/.bidscoin/4.3.0/templates/bidsmap_sst.yaml
-> /tmp/.bidscoin/4.3.0/templates/schema.json
usage: bidscoin [-h] [-l] [-p] [-i INSTALL [INSTALL ...]] [-u UNINSTALL [UNINSTALL ...]] [-d DOWNLOAD] [-t [TEST]] [-b BIDSMAPTEST] [-c CREDITS [CREDITS ...]] [--tracking {yes,no,show}] [-v]

BIDScoin is a toolkit to convert raw data-sets according to the Brain Imaging Data Structure (BIDS)

The basic workflow is to run these two tools:

  $ bidsmapper sourcefolder bidsfolder     # This produces a study bidsmap and launches a GUI
  $ bidscoiner sourcefolder bidsfolder     # This converts your data to BIDS according to the study bidsmap

Default settings and template bidsmaps are stored in the `.bidscoin` folder in your home directory
(you can modify them to your needs with any plain text editor)

Set the environment variable `BIDSCOIN_DEBUG=TRUE` to run BIDScoin in a more verbose logging mode and
`BIDSCOIN_CONFIG=/writable/path/to/config.toml` if you like to use a different config file.
Citation reports can be generated with the help of duecredit (https://github.com/duecredit/duecredit)

For more documentation see: https://bidscoin.readthedocs.io

options:
  -h, --help            show this help message and exit
  -l, --list            List all executables (i.e. the apps, bidsapps and utilities)
  -p, --plugins         List all installed plugins and template bidsmaps
  -i INSTALL [INSTALL ...], --install INSTALL [INSTALL ...]
                        A list of template bidsmaps and/or bidscoin plugins to install
  -u UNINSTALL [UNINSTALL ...], --uninstall UNINSTALL [UNINSTALL ...]
                        A list of template bidsmaps and/or bidscoin plugins to uninstall
  -d DOWNLOAD, --download DOWNLOAD
                        Download tutorial MRI data to the DOWNLOAD folder
  -t [TEST], --test [TEST]
                        Test the bidscoin installation and template bidsmap
  -b BIDSMAPTEST, --bidsmaptest BIDSMAPTEST
                        Test the run-items and their bidsnames of all normal runs in the study bidsmap. Provide the bids-folder or the bidsmap filepath
  -c CREDITS [CREDITS ...], --credits CREDITS [CREDITS ...]
                        Show duecredit citations for your BIDS repository. You can also add duecredit summary arguments (without dashes), e.g. `style {apa,harvard1}` or `format {text,bibtex}`.
  --tracking {yes,no,show}
                        Show the usage tracking info {show}, or set usage tracking to {yes} or {no}
  -v, --version         Show the installed version and check for updates

examples:
  bidscoin -l
  bidscoin -d data/bidscoin_tutorial
  bidscoin -t
  bidscoin -t my_template_bidsmap
  bidscoin -b my_study_bidsmap
  bidscoin -i data/my_template_bidsmap.yaml downloads/my_plugin.py
  bidscoin -c myproject/bids
  bidscoin -c myproject/bids format bibtex
  bidscoin --tracking show

INFO:    fuse2fs not found, will not be able to mount EXT3 filesystems
Creating BIDScoin configuration:
-> /home/marzwi/.bidscoin/4.3.0/config.toml
-> /home/marzwi/.bidscoin/4.3.0/templates/bidsmap_bids2bids.yaml
-> /home/marzwi/.bidscoin/4.3.0/templates/bidsmap_dccn.yaml
-> /home/marzwi/.bidscoin/4.3.0/templates/bidsmap_sst.yaml
-> /home/marzwi/.bidscoin/4.3.0/templates/schema.json
usage: bidscoin [-h] [-l] [-p] [-i INSTALL [INSTALL ...]] [-u UNINSTALL [UNINSTALL ...]] [-d DOWNLOAD] [-t [TEST]] [-b BIDSMAPTEST] [-c CREDITS [CREDITS ...]] [--tracking {yes,no,show}] [-v]

BIDScoin is a toolkit to convert raw data-sets according to the Brain Imaging Data Structure (BIDS)

The basic workflow is to run these two tools:

  $ bidsmapper sourcefolder bidsfolder     # This produces a study bidsmap and launches a GUI
  $ bidscoiner sourcefolder bidsfolder     # This converts your data to BIDS according to the study bidsmap

Default settings and template bidsmaps are stored in the `.bidscoin` folder in your home directory
(you can modify them to your needs with any plain text editor)

Set the environment variable `BIDSCOIN_DEBUG=TRUE` to run BIDScoin in a more verbose logging mode and
`BIDSCOIN_CONFIG=/writable/path/to/config.toml` if you like to use a different config file.
Citation reports can be generated with the help of duecredit (https://github.com/duecredit/duecredit)

For more documentation see: https://bidscoin.readthedocs.io

options:
  -h, --help            show this help message and exit
  -l, --list            List all executables (i.e. the apps, bidsapps and utilities)
  -p, --plugins         List all installed plugins and template bidsmaps
  -i INSTALL [INSTALL ...], --install INSTALL [INSTALL ...]
                        A list of template bidsmaps and/or bidscoin plugins to install
  -u UNINSTALL [UNINSTALL ...], --uninstall UNINSTALL [UNINSTALL ...]
                        A list of template bidsmaps and/or bidscoin plugins to uninstall
  -d DOWNLOAD, --download DOWNLOAD
                        Download tutorial MRI data to the DOWNLOAD folder
  -t [TEST], --test [TEST]
                        Test the bidscoin installation and template bidsmap
  -b BIDSMAPTEST, --bidsmaptest BIDSMAPTEST
                        Test the run-items and their bidsnames of all normal runs in the study bidsmap. Provide the bids-folder or the bidsmap filepath
  -c CREDITS [CREDITS ...], --credits CREDITS [CREDITS ...]
                        Show duecredit citations for your BIDS repository. You can also add duecredit summary arguments (without dashes), e.g. `style {apa,harvard1}` or `format {text,bibtex}`.
  --tracking {yes,no,show}
                        Show the usage tracking info {show}, or set usage tracking to {yes} or {no}
  -v, --version         Show the installed version and check for updates

examples:
  bidscoin -l
  bidscoin -d data/bidscoin_tutorial
  bidscoin -t
  bidscoin -t my_template_bidsmap
  bidscoin -b my_study_bidsmap
  bidscoin -i data/my_template_bidsmap.yaml downloads/my_plugin.py
  bidscoin -c myproject/bids
  bidscoin -c myproject/bids format bibtex
  bidscoin --tracking show

INFO:    fuse2fs not found, will not be able to mount EXT3 filesystems
Creating BIDScoin configuration:
-> /home/marzwi/somefolder/config.toml
-> /home/marzwi/somefolder/templates/bidsmap_bids2bids.yaml
-> /home/marzwi/somefolder/templates/bidsmap_dccn.yaml
-> /home/marzwi/somefolder/templates/bidsmap_sst.yaml
-> /home/marzwi/somefolder/templates/schema.json
usage: bidscoin [-h] [-l] [-p] [-i INSTALL [INSTALL ...]] [-u UNINSTALL [UNINSTALL ...]] [-d DOWNLOAD] [-t [TEST]] [-b BIDSMAPTEST] [-c CREDITS [CREDITS ...]] [--tracking {yes,no,show}] [-v]

BIDScoin is a toolkit to convert raw data-sets according to the Brain Imaging Data Structure (BIDS)

The basic workflow is to run these two tools:

  $ bidsmapper sourcefolder bidsfolder     # This produces a study bidsmap and launches a GUI
  $ bidscoiner sourcefolder bidsfolder     # This converts your data to BIDS according to the study bidsmap

Default settings and template bidsmaps are stored in the `.bidscoin` folder in your home directory
(you can modify them to your needs with any plain text editor)

Set the environment variable `BIDSCOIN_DEBUG=TRUE` to run BIDScoin in a more verbose logging mode and
`BIDSCOIN_CONFIG=/writable/path/to/config.toml` if you like to use a different config file.
Citation reports can be generated with the help of duecredit (https://github.com/duecredit/duecredit)

For more documentation see: https://bidscoin.readthedocs.io

options:
  -h, --help            show this help message and exit
  -l, --list            List all executables (i.e. the apps, bidsapps and utilities)
  -p, --plugins         List all installed plugins and template bidsmaps
  -i INSTALL [INSTALL ...], --install INSTALL [INSTALL ...]
                        A list of template bidsmaps and/or bidscoin plugins to install
  -u UNINSTALL [UNINSTALL ...], --uninstall UNINSTALL [UNINSTALL ...]
                        A list of template bidsmaps and/or bidscoin plugins to uninstall
  -d DOWNLOAD, --download DOWNLOAD
                        Download tutorial MRI data to the DOWNLOAD folder
  -t [TEST], --test [TEST]
                        Test the bidscoin installation and template bidsmap
  -b BIDSMAPTEST, --bidsmaptest BIDSMAPTEST
                        Test the run-items and their bidsnames of all normal runs in the study bidsmap. Provide the bids-folder or the bidsmap filepath
  -c CREDITS [CREDITS ...], --credits CREDITS [CREDITS ...]
                        Show duecredit citations for your BIDS repository. You can also add duecredit summary arguments (without dashes), e.g. `style {apa,harvard1}` or `format {text,bibtex}`.
  --tracking {yes,no,show}
                        Show the usage tracking info {show}, or set usage tracking to {yes} or {no}
  -v, --version         Show the installed version and check for updates

examples:
  bidscoin -l
  bidscoin -d data/bidscoin_tutorial
  bidscoin -t
  bidscoin -t my_template_bidsmap
  bidscoin -b my_study_bidsmap
  bidscoin -i data/my_template_bidsmap.yaml downloads/my_plugin.py
  bidscoin -c myproject/bids
  bidscoin -c myproject/bids format bibtex
  bidscoin --tracking show

from bidscoin.

Do you perhaps know what Path.home() returns in an apptainer that runs without $HOME?

Never mind, I found the answer. The problem I have is that tempfile.gettempdir() doesn't necessarily return a usable result...

Yes, I know. It points to host HOME. I will cite the relevant issue again ;-) apptainer/apptainer#388

from bidscoin.

If your container has a tempdir (doesn't it?) then I hope my last commit works as expected...

It does. But depending on the execution parameters, it may be mounted from the host or not.

from bidscoin.

As long as tempfile.gettempdir() returns a result then things are ok. BIDScoin needs some space to write temporary stuff (also for unpacking zipped data, etc)

from bidscoin.

Inside of the container

version 4.2.1

I fix it with --scratch $HOME when Apptainer is executed with --no-home, because bidscoin keeps looking up $HOME although it isn't mounted.

Ok, that's not good, I will make a fallback-fix for this (in $TMPDIR?)

I like it.

current master version (cd409e1)

Adding --env BIDSCOIN_CONFIG=/root/.bidscoin/4.3.0/config.toml to Apptainer command should help, because bidscoin doesn't know where is .bidscoin without BIDSCOIN_CONFIG set.

So you pass a config file, that should work, but pointing it to /root puzzles me.

When you build an Apptainer image (especially from a Docker image), there is only root user, and therefore there are no user homes. During building of the container, HOME points to /root. In order to reuse the files created by BIDScoin's installation, I point to where they were created.

If it is not set, bidscoin expects .bidscoin in HOME, which for installation in Apptainer is not true (HOME always points to host home).

If it's not set, then bidscoin will create it in home, but I'll make that fallback-fix that should cover this

Ok. This seems to be on Apptainer and --no-home flag. HOME points to home in the host but --no-home doesn't mount it.

If config.toml needs to be overwritten, it can be mounted additionally (--bind /path/to/my/config.toml:/root/.bidscoin/4.3.0/config.toml). If BIDSCOIN_CONFIG is set to something else than /root/.bidscoin/4.3.0/config.toml, the parent directory has to be mounted.

Sorry, I'm not that used to working with containers, but I don't see why you have to bind that root path? Isn't passing --env BIDSCOIN_CONFIG=/some/mounted/folder/bidscoin/config.toml sufficient?

In this example (second sentence of my comment), you still have to mount --bind /some/mounted/folder/bidscoin.

In general

I should not need to set BIDSCOIN_CONFIG and care about the .bidscoin directory, especially if I use bidsmapper with --template option. The config file only sets this template file and the tracking.

I agree that the config folder shouldn't be a user concern and I'm surely open for suggestions to improve things here.

It may be solved with your newest changes. I'll test it in a bit.

Usually, I can set the options in a config file, or provide them with CLI, with some priorities.

Well, that sounds great but on the other hand, it is also very common for applications to simply keep their settings in a hidden file or directory in home (e.g. dcm2niix).

Agree. It is fine if BIDScoin creates itself a directory. Though, in my opinion it should create it where it's installed. If I install it in the container, .bindcoin should be in the container and not recreate it in the host. No?

Bidscoin could remember where it created .bidscoin directory, whether BIDSCOIN_CONFIG is set or not.

I don't see how bidscoin can remember that without storing it somewhere on disk. Which is what the config file is

Good point. Maybe the problem is really only this Apptainer issue.

Then, it could preset the options from the config file in there, and optionally overwrite them with values provided with CLI. This would make the behavior consistent.

Currently I don't provide a tool or CLI options for editing/overruling the config file, the user needs to do that him/herself using a text editor (which is very easy and works very well). The behaviour should consistently be: Use the default config (in home or /tmp) or use the config specified in the environment variable.

I like the tmp solution. In my particular case, however, it won't work. I mount a subdirectory of my home in the container (absolute path). It means that Path.home().exists() returns true in the container, but it is not writable. One could use os.access('my_folder', os.W_OK) to additionally check permission. But I really don't want to complicate it anymore 😭

>>> Path.home().exists()
True
>>> os.access(Path.home(), os.W_OK)
False

from bidscoin.

Many thanks for dealing with that, although it wasn't on you. I learned a lot.

from bidscoin.

Agree. It is fine if BIDScoin creates itself a directory. Though, in my opinion it should create it where it's installed. If I install it in the container, .bindcoin should be in the container and not recreate it in the host. No?

No, because then it is not writable

It means that Path.home().exists() returns true in the container

I didn't know that, I'll add an even more thorough test then

I like the tmp solution. In my particular case, however, it won't work

You mean in your case tempfile.gettempdir() does not return a writable tmpdir? If so, then I am afraid I cannot fix that

But I really don't want to complicate it anymore

No, I really appreciate it, it makes bidscoin more robust

from bidscoin.

One could use os.access('my_folder', os.W_OK) to additionally check permission.

Done!

from bidscoin.

Agree. It is fine if BIDScoin creates itself a directory. Though, in my opinion it should create it where it's installed. If I install it in the container, .bindcoin should be in the container and not recreate it in the host. No?

No, because then it is not writable

Right 🙈

I like the tmp solution. In my particular case, however, it won't work

You mean in your case tempfile.gettempdir() does not return a writable tmpdir? If so, then I am afraid I cannot fix that

/tmp is fine:

>>> Path(tempfile.gettempdir())
PosixPath('/tmp')
>>> os.access(Path(tempfile.gettempdir()), os.W_OK)
True

I meant that it won't work for me if you only check for existence of HOME but not if it's writable. If you check both and resort on fail to tmp, it should work for me too.

But I really don't want to complicate it anymore

No, I really appreciate it, it makes bidscoin more robust

:-)

from bidscoin.

That is actually great :-) I agree. Thanks.

from bidscoin.