hpc / give Goto Github PK

Currently large files being given will take a long time using cp. I'm not sure we want to encourage the reproduction of large files that are already stored, so this may be a non-issue.

We should at least check file size before "giving", maybe print an error and exit if the file is too large? Or let the user know the give is going to take too long. If we want give to allow large file sharing, we need to parallelize the copy or let the user know it could take a very long time.

@dayr @jennfshr thoughts?

When using python 3, take reports file sizes in YB:

$> python --version
Python 2.7.5
$> take
 jgreen has given:
   189  B May 28 09:36 yum.out
     9 MB Jul 17 14:55 zerolibs.txt
   2 file(s)
You have 2 file(s) to take.
$> module load python/3.6 
$> python --version
Python 3.6.3 :: Anaconda, Inc.
$> take
 jgreen has given:
     0 YB May 28 09:36 yum.out
     0 YB Jul 17 14:55 zerolibs.txt
   2 file(s)
You have 2 file(s) to take.

This is with give version 3.1-4 (as reported via rpm).

I believe the issue is in the list_a_file method in take. It uses a set of conditionals to decide what units to display file sizes in. Between python 2 and 3, the / operator changed from integer to floating point division, so all of the conditionals now return a value greater than 0. This means the first condition evaluates to true all of the time, which leads to YB being the unit.

Here is the division that list_a_file relies on done in both python 2 and 3 to illustrate what is going on:

$> python --version
Python 2.7.5
$> python -c "print(20 / 2**80)"
0
$> module load python/3.6
$> python --version
Python 3.6.3 :: Anaconda, Inc.
$> python -c "print(20 / 2**80)"
1.6543612251060553e-23

I can see two possible ways of fixing this:

1. Change all of the / operators in the conditionals to // operators (recovers integer division in python 3, and works in python 2.7).
2. Change all of the the > 0 to > 1 in the conditionals.

There might be other solutions, too.

Thanks,
David

Create src, docs, contrib, and so on then put files in appropriate places. Placeholder so I don't forget about this...

Need to address python3. Either explicitly require python2, print a friendly message, and exit or make give.py python3 compatible.

Currently, for a user with python3 in their env, give crashes at the print and produces a syntax error.

@dayr has this come up as an issue at LLNL?
@jennfshr @dayr any preference on implementation?

I will merge Ryan's previous changes into master. If we switch to python3 compatibility, we should branch and version up. If we explicitly require python 2, we should minor version up.