wiki page for shared volumes about rocker HOT 15 CLOSED

I agree, easy folder sharing is crucial to making rocker containers a practical day-to-day analysis tool.

I tested and added the instructions for windows folder sharing on the RStudio wiki page yesterday, but I can move them over to your new page and leave a cross-reference, what do you think?

I'll do some more testing of permissions, I didn't notice anything unexpected, but perhaps haven't yet exhausted the options. I haven't yet tried to edit and save a doc on my desktop and simultaneously in RStudio in docker.

I agree with your point about enabling diverse workflows as much as possible. For myself and my students we do almost everything in RStudio, to keep the toolkit simple (but when the students aren't looking I use emacs ;). We do some markdown writing in a web editor (http://dillinger.io/ or https://stackedit.io/) because they give word counts a nice live rendering, and we produce the final doc in RStudio with knitr. Word counts is on the feature request list for RStudio, hopefully coming soon. Editing in the browser is handy with docker/rstudio because then we just move between browser tabs.

from rocker.

Thanks, yes, the windows linking instructions look great. Curious that you
mount to /data in the example; seems like RStudio wouldn't have write
permissions to a directory on the container mounted below ~, or can you
write as well as read files in /data from RStudio like that?

Yeah, moving them into the new page and just linking is probably good, will
help keep the RStudio wiki page brief.

On Mon, Nov 3, 2014 at 4:28 PM, Ben Marwick [email protected]
wrote:

I agree, easy folder sharing is crucial to making rocker containers a
practical day-to-day analysis tool.

I tested and added the instructions for windows folder sharing on the RStudio
wiki page
https://github.com/rocker-org/rocker/wiki/Using-the-RStudio-image#folder-sharing
yesterday, but I can move them over to your new page and leave a
cross-reference, what do you think?

I'll do some more testing of permissions, I didn't notice anything
unexpected, but perhaps haven't yet exhausted the options. I haven't yet
tried to edit and save a doc on my desktop and simultaneously in RStudio in
docker.

I agree with your point about enabling diverse workflows as much as
possible. For myself and my students we do almost everything in RStudio, to
keep the toolkit simple (but when the students aren't looking I use emacs
;). We do some markdown writing in a web editor (http://dillinger.io/ or
https://stackedit.io/) because they give word counts a nice live
rendering. That's great with docker because then we just move between
browser tabs.

—
Reply to this email directly or view it on GitHub
#74 (comment).

Carl Boettiger
UC Santa Cruz
http://carlboettiger.info/

from rocker.

I've done a bit of testing folder sharing on windows. Seems like the shortest workflow is to move or create files on the host side of the shared folder (ie. data files), then change their permissions on the container side using bash, then we can read and write bidirectionally. However, files created in the host side of the shared folder do not appear to inherit permissions from the container and their permissions need to be changed in the container (so move everything you need in at the start, then chmod). Files created on the container side will be editable on the host side also (live updating) with no further changes (so it's fine to create scripts and output in RStudio as you go).

Notes on tests, starting from docker run -dp 8787:8787 rocker/rstudio:

• can't create new file or make new folder in /data in RStudio
• can create new file in explorer at \192.168.59.103\data
• file created on host in /data is read-only in RStudio
• can access shell via RStudio GUI, need to sudo to chmod, but cannot, password 'rstudio' doesn't work
• can access shell via docker run -it --volumes-from my-data rocker/rstudio /bin/bash or docker exec -it "id of running container" bash and then chmod -R a+rwX data (without sudo) to enable read and write permissions on data folder
• after the chmod then running docker run -dp 8787:8787 rocker/rstudio I can create new files and folders in RStudio, and edit files on the host also
• after chmod, new files created on the host side of the share are not immediately writable in RStudio, and chmod needs to be done again in the container.

So, some outstanding questions:

• sudo using the shell via RStudio, what's the password?
• streamline changing permissions on the data folder? Can some steps be combined? docker run -itdp 8787:8787 --volumes-from my-data rocker/rstudio /bin/bash doesn't seem to start RStudio or give a shell prompt, but perhaps there is a similar one-liner that will?
• create a file on the host that inherits permissions from the container?

This is my current shortest workflow:

# open a regular terminal and assuming b2d is installed already:  
boot2docker ssh

# folder sharing on
docker run -v /data --name my-data busybox true
docker run --rm -v /usr/local/bin/docker:/docker -v /var/run/docker.sock:/docker.sock svendowideit/samba my-data

# change folder permissions
docker run -it --volumes-from my-data rocker/rstudio /bin/bash # or docker exec -it "id of running container" bash
chmod -R a+rwX /data # to change permissions for the shared data folder, access this on the desktop via \\192.168.59.103\data in explorer
exit # to exit bash and get back to docker prompt

# start my container & do stuff in RStudio with shared folder
docker run -itdp 8787:8787 --volumes-from my-data rocker/rstudio # connect this container to shared folder
# go to http://192.168.59.103:8787/ in the browser, log in with rstudio/rstudio

# finish
exit # in boot2docker window, to quit boot2docker

from rocker.

Right, this looks like what I would expect; on the container /data is owned by root by default because it's below the level of the user's home directory (e.g. /home/rstudio). On the linux side I've simply resorted to mounting data in the user's home directory instead, which seems more natural than editing permissions. Can you just replace /data with /home/rstudio/data, e.g.

docker run -v /home/rstudio/data --name my-data busybox true
docker run --rm -v /usr/local/bin/docker:/docker -v /var/run/docker.sock:/docker.sock  svendowideit/samba my-data

?

from rocker.

Thanks for the suggestion, it would be good if it could be that simple :)

I get unable to connect to service when I try http://192.168.59.103:8787/ (so no RStudio server at all) and Windows cannot access \\192.168.59.103\home\rstudio\data or any variant of that path when I try to connect on the host side. Seems that this might not be how windows is expecting the share to operate, I don't have any specific ideas to move ahead, but will keep exploring (suggestions welcome!).

from rocker.

Hmm, well, you're using the volume syntax to specify only the location on the container (e.g. no colon). Perhaps we could make Windows browser part happy by doing:

docker run -v /data:/home/rstudio/data --name my-data busybox true

maybe?

If that works and you can access: 192.168.59.103\data then maybe we can debug why you can't connect to RStudio server next... (that one might be due to an old version of rocker/rstudio, lemme know what hash your image has)

from rocker.

(Or perhaps we might need to also run the busybox container as a non-root user to avoid the rstudio unable to connect to service issue)

from rocker.

Okay, haven't tested on Windows yet but I think this will work:

docker run -v /data:/home/docker/data --name my-data busybox true
docker run --rm -v /usr/local/bin/docker:/docker -v /var/run/docker.sock:/docker.sock  svendowideit/samba my-data

docker run -itdp 8787:8787 -e USER=docker --volumes-from my-data rocker/rstudio # connect this container to shared folder

And login as user/pw set docker/rstudio.

I think I see why this isn't working when user is rstudio instead of docker, testing a fix now.

from rocker.

Okay, just pushed a fix so it will work with the rstudio user or any custom user. Also noticed a typo in the command I gave earlier (forgot to use /data:/home/rstudio/data as I had suggested). So with the new version on sandbox (not yet merged) I'm thinking this should work:

docker run -v /data:/home/rstudio/data --name my-data busybox true

docker run --rm -v /usr/local/bin/docker:/docker -v /var/run/docker.sock:/docker.sock  svendowideit/samba my-data

docker run -itdp 8787:8787 --volumes-from my-data rocker/rstudio 

Test with changing the /home/rstudio to /home/docker and using -e USER=docker on the last call until we merge sandbox...

from rocker.

p.s. is the samba method still the most robust option for windows? From the boot2docker docs, it looks like VirtualBox Guest Additions is standard with boot2docker now, https://github.com/boot2docker/boot2docker#folder-sharing, so that you should just be able to link directly with something like:

docker run -d -p 8787:8787 -v /c/Users/foobar:/home/rstudio/foobar rocker/rstudio

Sounds more streamlined than samba, but no idea how well it works.

from rocker.

Thanks very much for all those suggestions, I've given them all a thorough workout and it looks like the winner is your last one using Guest Additions. I also tried the method here, but no luck, and it's more complex, so I'm happy to ignore that in favour of the GA method. I'll do some editing of the wiki to indicate the GA method as the way to go, though would you mind to look at the message I'm getting in the console (see bottom)?

Notes on testing:

docker run -v /home/docker/data --name my-data busybox true
docker run --rm -v /usr/local/bin/docker:/docker -v /var/run/docker.sock:/docker.sock  svendowideit/samba my-data

docker run -itdp 8787:8787 -e USER=docker --volumes-from my-data rocker/rstudio # connect this container to shared folder

And login as user/pw set docker/rstudio.

• can connect to RStudio and see /data folder, but cannot access it (save to)

######################

docker run -v /data:/home/docker/data --name my-data busybox true

docker run --rm -v /usr/local/bin/docker:/docker -v /var/run/docker.sock:/docker.sock  svendowideit/samba my-data

docker run -itdp 8787:8787 --volumes-from my-data  -e USER=docker rocker/rstudio 

And login as user/pw set docker/rstudio.

• can connect to RStudio and see /data folder, but cannot access it (save to)
• I get 'no such file or directory'

#######################

docker run -d -p 8787:8787 -v /c/Users/marwick:/home/rstudio/marwick rocker/rstudio

• log in with rstudio/rstudio
• can connect to RStudio and see contents of my host folders, and can save a new file to them from RStudio
• I can edit and save files already on the host in RStudio
• I get this message about 50 times in my RStudio console each time open the shared folder or navigate to or from the top level, which is rather distracting

05 Nov 2014 18:36:57 [rsession-rstudio] ERROR system error 71 (Protocol error) [path=/home/rstudio/marwick/Templates]; OCCURRED AT: bool core::FilePath::exists() const /home/ubuntu/rstudio/src/cpp/core/FilePath.cpp:307; LOGGED FROM: bool core::FilePath::exists() const /home/ubuntu/rstudio/src/cpp/core/FilePath.cpp:307

from rocker.

In case 2 (with -v data:/home/docker/data), on the container (that is, from
within RStudio) you say that you can see but cannot access the /data
folder. Are you referring to /home/docker/data that you can see but
cannot write to, or to /data? (The rstudio file browser should open up
in /home/docker and show the folder /data).

If you can't write to /home/docker/data, can you check that you have the
very latest rocker/rstudio (e.g. docker pull rocker rstudio)? Can you
also run in RStudio:

system("ls -l /home/docker/")

and maybe:

system("ls -l /home")

and paste the results here? (e.g. does user 'docker' own these
directories?)

otherwise, if /home/docker/data doesn't exist, that's a different issue.

Glad the Guest Additions pretty much works out of the box with one command.

Very weird regarding that error message. what's with the 'ubuntu' user
appearing in that path anyway? (none of the containers you use are based
on ubuntu...) Can you try logging in setting the custom user name to
ubuntu instead of defaulting to rstudio and see what happens?

On Wed, Nov 5, 2014 at 11:50 PM, Ben Marwick [email protected]
wrote:

Thanks very much for all those suggestions, I've given them all a thorough
workout and it looks like the winner is your last one using Guest
Additions. I also tried the method here
http://blog.tutum.co/2014/11/05/how-to-use-docker-on-windows/, but no
luck, and it's more complex, so I'm happy to ignore that in favour of the
GA method.

Notes on testing:

docker run -v /home/docker/data --name my-data busybox true
docker run --rm -v /usr/local/bin/docker:/docker -v /var/run/docker.sock:/docker.sock svendowideit/samba my-data

docker run -itdp 8787:8787 -e USER=docker --volumes-from my-data rocker/rstudio # connect this container to shared folder

And login as user/pw set docker/rstudio.

• can connect to RStudio and see /data folder, but cannot access it
  (save to)

######################

docker run -v /data:/home/docker/data --name my-data busybox true

docker run --rm -v /usr/local/bin/docker:/docker -v /var/run/docker.sock:/docker.sock svendowideit/samba my-data

docker run -itdp 8787:8787 --volumes-from my-data -e USER=docker rocker/rstudio

And login as user/pw set docker/rstudio.

• can connect to RStudio and see /data folder, but cannot access it
  (save to)
• I get 'no such file or directory'

#######################

docker run -d -p 8787:8787 -v /c/Users/marwick:/home/rstudio/marwick rocker/rstudio

• log in with rstudio/rstudio
• can connect to RStudio and see contents of my host folders, and can
  save a new file to them from RStudio
• I can edit and save files already on the host in RStudio
• I get this message about 50 times in my RStudio console each time
  open the shared folder or navigate to or from the top level, which is
  rather distracting

05 Nov 2014 18:36:57 [rsession-rstudio] ERROR system error 71 (Protocol error) [path=/home/rstudio/marwick/Templates]; OCCURRED AT: bool core::FilePath::exists() const /home/ubuntu/rstudio/src/cpp/core/FilePath.cpp:307; LOGGED FROM: bool core::FilePath::exists() const /home/ubuntu/rstudio/src/cpp/core/FilePath.cpp:307

—
Reply to this email directly or view it on GitHub
#74 (comment).

Carl Boettiger
UC Santa Cruz
http://carlboettiger.info/

from rocker.

bump. @benmarwick did we ever resolve this?

from rocker.

Thanks for the reminder, now that we've got folder sharing basically working I've been enjoying using rocker as my regular R environment, and neglected the tests, sorry about that. Here's a bit of follow-up:

For this setup:

docker run -v /data:/home/docker/data --name my-data busybox true
docker run --rm -v /usr/local/bin/docker:/docker -v /var/run/docker.sock:/docker.sock  svendowideit/samba my-data
docker run -itdp 8787:8787 --volumes-from my-data  -e USER=docker rocker/rstudio 

I get:

> system("ls -l /home/docker/")
total 0
drwxr-xr-x 2 root root 40 Nov 23 22:14 data
> system("ls -l /home")
total 4
drwxr-xr-x 4 docker docker 4096 Nov 23 22:29 docker

And if remove -e USER=docker:

docker run -v /data:/home/docker/data --name my-data busybox true
docker run --rm -v /usr/local/bin/docker:/docker -v /var/run/docker.sock:/docker.sock  svendowideit/samba my-data
docker run -itdp 8787:8787 --volumes-from my-data  rocker/rstudio 

I get

> system("ls -l /home/docker/")
total 0
drwxr-xr-x 2 root root 40 Nov 23 22:14 data
> system("ls -l /home")
total 8
drwxr-xr-x 3 rstudio rstudio 4096 Nov 23 22:26 docker
drwxr-xr-x 3 rstudio rstudio 4096 Nov 23 22:27 rstudio

For comparison, by not invoking sharing at all with docker run -dp 8787:8787 -e ROOT=TRUE rocker/rstudio I get:

> system("ls -l /home/docker/")
total 0
> system("ls -l /home")
total 8
drwxr-xr-x 2 rstudio rstudio 4096 Nov 21 03:15 docker
drwxr-xr-x 3 rstudio rstudio 4096 Nov 23 22:02 rstudio

And using my current favourite: docker run -dp 8787:8787 -e ROOT=TRUE -v /c/Users/marwick:/home/rstudio/ rocker/hadleyverse I get:

> system("ls -l /home/docker/")
total 0   
>     system("ls -l /home")
total 24
drwxr-xr-x 2 rstudio rstudio  4096 Nov 21 03:15 docker
drwxrwxrwx 1 rstudio staff   20480 Nov 27  2014 rstudio

as well as access to the main parts of my windows filesystem and this message periodically scrolling down the console, seems like it's related to file access operations:

23 Nov 2014 22:09:04 [rsession-rstudio] ERROR system error 71 (Protocol error) [path=/home/rstudio/Templates]; OCCURRED AT: bool core::FilePath::exists() const /home/ubuntu/rstudio/src/cpp/core/FilePath.cpp:307; LOGGED FROM: bool core::FilePath::exists() const /home/ubuntu/rstudio/src/cpp/core/FilePath.cpp:307

A similar error has been reported to RStudio support here, I've posted mine, so let's see if they have any ideas.

from rocker.

Okay, thanks a lot for the report. So it looks like the samba sharing makes the directory it shares (e.g. /home/$USER/data) owned by root no matter what, and we loose write permissions. Of course now that you can enable root inside RStudio, one could chown -R $USER:$USER /home/$USER to fix things. But it looks like in general we should instruct people to link directly with the -v /c/Users/... and not samba, and things work as expected (and with fewer commands. win). Looks like you've also listed this as the prefered method on the wiki, so I think we're all set -- nice work!

Think we can close this out then. Thanks again for all the testing.

from rocker.