Giter Club home page Giter Club logo

cmssh's People

Contributors

neggert avatar vkuznet avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Forkers

vkuznet neggert

cmssh's Issues

Add pycurl manager to handle HTTP(s) session

PyCurl can significantly speed up client part if used properly (see DAS/DBS3). I need to add pycurl HTTP manager and use it for all remote calls, to do so I need to keep around single instance of curl object.

How to change dbs_instance

I'm not sure if this is a bug or if I just don't know what I'm doing. I'm trying to change the dbs instance. I assume that I should be able to do :

dbs_instance cms_dbs_ph_analysis_01

but that gives me an invalid DBS instance message. Is this not working properly or am I doing something wrong. Implementing dbs_instance --help might be useful in the future.

turn off text.usetex

Hi,
I get these warnings when I start cmssh:

/Users/nic/cms/soft/CMSSW/osx106_amd64_gcc421/external/py2-matplotlib/1.0.1-cms3/lib/python2.6/site-packages/matplotlib/__init__.py:397: UserWarning: matplotlibrc text.usetex option can not be used unless TeX-3.1415 or later is installed on your system
  'installed on your system') % tex_req)
/Users/nic/cms/soft/CMSSW/osx106_amd64_gcc421/external/py2-matplotlib/1.0.1-cms3/lib/python2.6/site-packages/matplotlib/__init__.py:403: UserWarning: matplotlibrc text.usetex can not be used with *Agg backend unless dvipng-1.5 or later is installed on your system
  warnings.warn( 'matplotlibrc text.usetex can not be used with *Agg '
/Users/nic/cms/soft/CMSSW/osx106_amd64_gcc421/external/py2-matplotlib/1.0.1-cms3/lib/python2.6/site-packages/matplotlib/__init__.py:417: UserWarning: matplotlibrc text.usetex can not be used unless ghostscript-7.07 or later is installed on your system
  'installed on your system') % gs_req)

This happens because I have my matplotlibrc set up to use latex. You might consider forcing the system to use a matplotlibrc that you have written rather than picking up the one in the home directory to prevent problems like this. I've written a matplotlibrc that looks somewhat like the official CMS plot style, if you'd like me to send it to you.

Handle root properly

Right now I use root_init in setup.sh. Instead I need to wrap root call based on SCRAM_ARCH. Remove root_init from setup.sh, modify cms_root to do root_init call.

Improve int. lumi calculation via usage of golden JSON file

I can improve integration lumi calculation by adding intermediate step to check if run_lumi dict are within boundaries of golden JSON file. To do so I may extend cms_cmds::run_lumi_info function and check presence of this file
/afs/cern.ch/cms/CAF/CMSCOMM/COMM_DQM/certification/Collisions12/8TeV/Prompt/Cert_190456-202305_8TeV_PromptReco_Collisions12_JSON.txt

If so, I need to add helper function to load golden JSON from it and use it for comparison with found run lumi dict.

Limit number of simultaneous background cp operations

Hi,
I just (foolishly) tried to cp every file in a dataset to my computer in the background. Java ate all my memory and I had to reboot my computer. It would be nice if there was a queue of some sort, so that only 4 or so downloads are running at once.

Nic

Correctly detect libreadline5 on ubuntu

I'm trying to install cmssh on ubuntu and I'm running into a problem with the detection of the readline library.

[~] python cmssh_install.py --help
Traceback (most recent call last):
  File "cmssh_install.py", line 149, in <module>
    DEF_SCRAM_ARCH = get_scram_arch()
  File "cmssh_install.py", line 97, in get_scram_arch
    raise Exception(msg)
Exception: cmssh on Linux requires readline5. Please verify that you have it installed on your system. So far we found
/usr/lib/libreadline.so -> /usr/lib/libreadline.so

I do, in fact have libreadline installed, but it's not finding it because it's in a different place. Here are the contents of the libreadline5 package in ubuntu.

[~] dpkg -L libreadline5/.
/etc
/lib
/lib/x86_64-linux-gnu
/lib/x86_64-linux-gnu/libreadline.so.5.2
/lib/x86_64-linux-gnu/libhistory.so.5.2
/usr
/usr/share
/usr/share/doc
/usr/share/doc/libreadline5
/usr/share/doc/libreadline5/USAGE
/usr/share/doc/libreadline5/inputrc.arrows
/usr/share/doc/libreadline5/examples
/usr/share/doc/libreadline5/examples/Inputrc
/usr/share/doc/libreadline5/copyright
/usr/share/doc/libreadline5/changelog.Debian.gz
/usr/share/doc/libreadline5/README.Debian
/lib/x86_64-linux-gnu/libreadline.so.5
/lib/x86_64-linux-gnu/libhistory.so.5

If I can make a suggestion, a better way to check for libreadline might be

/sbin/ldconfig -p | grep libreadline

or the equivalent python command.

cmsenv problem

I get an error on cmsenv (see below). I guess, it's my crazy machine again.. Haven't tested lxplus on yet.
cheers, Heiner

cms-sh|3> cmsenv

ERROR: OSError, fail to execute cmd=['eval', 'scramv1', 'runtime', '-sh'], kwds={}, error=[Errno 2] No such file or directory

fix background mode

When I submit LFN transfer in background mode, it tries xrdcp method and fails sometimes since file cannot be transferred via xrdcp. So I need to fallback into lcg/srm mode somehow.

cp by lfn doesn't work

I'm having a problem copying by lfn. See the error message below. This file is in cms_dbs_ph_analysis_01, so maybe that has something to do with the problem. It looks fine if I prepend the site name to the lfn, i.e. T1_US_FNAL_Buffer:/store/user/...

cp /store/user/neggert/TT_TuneZ2_7TeV-mcatnlo/MCTSusy_Skim_Mar2012//7b5af1bfe3424f60f0db5b5f14cf327a/MCTSusySkimMar2012_100_1_kYt.root .
Traceback (most recent call last):
File "/Users/nic/cms/soft/cmssh/src/cmssh/cms_cmds.py", line 487, in cms_cp
status = copy_lfn(src, dst, verbose, background)
File "/Users/nic/cms/soft/cmssh/src/cmssh/filemover.py", line 610, in copy_lfn
status = FM_SINGLETON.copy(lfn, dst, verbose, background)
File "/Users/nic/cms/soft/cmssh/src/cmssh/filemover.py", line 456, in copy
for cmd in srmcp("srm-copy", lfn, dst, verbose):
File "/Users/nic/cms/soft/cmssh/src/cmssh/filemover.py", line 307, in srmcp
raise Exception(msg)
Exception: LFN: /store/user/neggert/TT_TuneZ2_7TeV-mcatnlo/MCTSusy_Skim_Mar2012//7b5af1bfe3424f60f0db5b5f14cf327a/MCTSusySkimMar2012_100_1_kYt.root
No replicas found
{u'phedex': {u'request_date': u'2012-04-06 18:09:06 UTC', u'request_timestamp': 1333735746.04251, u'call_time': u'0.01102', u'instance': u'prod', u'request_call': u'fileReplicas', u'request_url': u'http://cmsweb.cern.ch:7001/phedex/datasvc/json/prod/fileReplicas', u'request_version': u'2.3.8', u'block': []}}
Wrong argument '/store/user/neggert/TT_TuneZ2_7TeV-mcatnlo/MCTSusy_Skim_Mar2012//7b5af1bfe3424f60f0db5b5f14cf327a/MCTSusySkimMar2012_100_1_kYt.root .'

Installer doesn't work

This installer was just downloaded today. I tried manually creating the directory it's complaining about, but it didn't help.

[cms] python cmssh_install.py -i -d cmssh                                                  11:23:29 
Checking CMSSW
Bootstrap CMSSW
Traceback (most recent call last):
  File "cmssh_install.py", line 770, in <module>
    main()
  File "cmssh_install.py", line 352, in main
    exe_cmd(sdir, cmd, debug, 'Bootstrap CMSSW', log='bootstrap.log')
  File "cmssh_install.py", line 203, in exe_cmd
    os.chdir(idir)
OSError: [Errno 2] No such file or directory: 'cmssh/soft/CMSSW'

History/provenance via git

I can hook-up git tool into cmssh and perform commits all user actions. Since git provides all versioning information and excellent logging capabilities it will means that whole history/provenance of user actions can easily captured. If user will be willing to setup git repository on github that it can be portable and distributable.

CRAB_2_8_1 is not available. Update install script to use CRAB_2_8_4.patch1

CRAB_2_8_1 is not available. Please update installation script to use CRAB_2_8_4_patch1.

Install CRAB
Traceback (most recent call last):
File "cmssh_install.py", line 1183, in
main()
File "cmssh_install.py", line 618, in main
get_file(url, 'crab.tar.gz', path, debug)
File "cmssh_install.py", line 357, in get_file
tar_file.write(getdata(url, debug))
File "cmssh_install.py", line 335, in getdata
data = urllib2.urlopen(req)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 126, in urlopen
return _opener.open(url, data, timeout)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 400, in open
response = meth(req, response)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 513, in http_response
'http', request, response, code, msg, hdrs)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 432, in error
result = self._call_chain(_args)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 372, in _call_chain
result = func(_args)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 619, in http_error_302
return self.parent.open(new, timeout=req.timeout)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 400, in open
response = meth(req, response)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 513, in http_response
'http', request, response, code, msg, hdrs)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 438, in error
return self._call_chain(_args)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 372, in _call_chain
result = func(_args)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 521, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 403: Forbidden

Won't load on lxplus


~: /afs/cern.ch/work/v/valya/public/soft/bin/cmssh                                                                                                                                                        õ
Welcome to cmssh! Loading configuration, please wait ...
Loading external/apt ... DONE
Loading lcg/root ... DONE
Loading external/python ... DONE
Loading external/xz ... DONE
Loading external/pcre ... DONE
Loading external/py2-matplotlib ... DONE
Loading external/py2-numpy ... DONE
Loading external/libpng ... DONE
Loading external/lapack ... DONE
Loading external/libjpg ... DONE
Loading external/libtiff ... DONE
Loading external/libungif ... DONE
Traceback (most recent call last):
  File "/afs/cern.ch/user/v/valya/workspace/public/soft/install/bin/ipython", line 8, in <module>
    load_entry_point('ipython==0.12.1', 'console_scripts', 'ipython')()
  File "/afs/cern.ch/user/v/valya/workspace/public/soft/install/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/pkg_resources.py", line 318, in load_entry_point
  File "/afs/cern.ch/user/v/valya/workspace/public/soft/install/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/pkg_resources.py", line 2221, in load_entry_point
  File "/afs/cern.ch/user/v/valya/workspace/public/soft/install/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/pkg_resources.py", line 1954, in load
  File "/afs/cern.ch/user/v/valya/workspace/public/soft/install/lib/python2.6/site-packages/IPython/__init__.py", line 43, in <module>
    from .config.loader import Config
  File "/afs/cern.ch/user/v/valya/workspace/public/soft/install/lib/python2.6/site-packages/IPython/config/loader.py", line 26, in <module>
    from IPython.external import argparse
  File "/afs/cern.ch/user/v/valya/workspace/public/soft/install/lib/python2.6/site-packages/IPython/external/argparse/__init__.py", line 10, in <module>
    from _argparse import *
  File "/afs/cern.ch/user/v/valya/workspace/public/soft/install/lib/python2.6/site-packages/IPython/external/argparse/_argparse.py", line 98, in <module>
    from gettext import gettext as _
  File "/afs/cern.ch/cms/slc5_amd64_gcc462/external/python/2.6.4-cms/lib/python2.6/gettext.py", line 49, in <module>
    import locale, copy, os, re, struct, sys
  File "/afs/cern.ch/user/v/valya/workspace/public/soft/install/lib64/python2.6/locale.py", line 202, in <module>
    import re, operator
ImportError: /afs/cern.ch/user/v/valya/workspace/public/soft/install/lib64/python2.6/lib-dynload/operator.so: undefined symbol: _PyUnicodeUCS2_AsDefaultEncodedString

das queries with filters don't behave as expected

Hi,

I'm trying to run a DAS query with a filter and I'm getting an error. This works just fine on the web interface:

cms-sh|35> find file dataset=/DYToEE_M-10To20_TuneZ2_7TeV-pythia6/neggert-MCTSusy_Skim_Mar2012_DYToEE_M10To20-daf9758207e31d09324b57b0328014ab/USER | grep file.nevents>= 1

ValueError Traceback (most recent call last)
/Users/nic/cms/MCTSusy/notebooks/Data/tW/ in ()
----> 1 get_ipython().magic(u'find file dataset=/DYToEE_M-10To20_TuneZ2_7TeV-pythia6/neggert-MCTSusy_Skim_Mar2012_DYToEE_M10To20-daf9758207e31d09324b57b0328014ab/USER | grep file.nevents>= 1')

/Users/nic/cms/soft/install/lib/python2.7/site-packages/IPython/core/interactiveshell.pyc in magic(self, arg_s, next_input)
1996 self._magic_locals = sys._getframe(1).f_locals
1997 with self.builtin_trap:
-> 1998 result = fn(magic_args)
1999 # Ensure we're not keeping object references around:

2000 self._magic_locals = {}

/Users/nic/cms/soft/cmssh/src/cmssh/cms_cmds.pyc in cms_find(arg)
137 Perform lookup of given query in CMS data-services.
138 """
--> 139 lookup(arg)
140
141 def cms_du(arg):

/Users/nic/cms/soft/cmssh/src/cmssh/cms_cmds.pyc in lookup(arg)
159 res = apply_filter(flt.strip(), gen)
160 RESMGR.assign(res)
--> 161 list_results(res, debug)
162
163 def verbose(arg):

/Users/nic/cms/soft/cmssh/src/cmssh/utils.pyc in list_results(res, debug)
77 return
78 if isinstance(res, list) or isinstance(res, GeneratorType):
---> 79 for row in res:
80 if not debug:
81 print row

/Users/nic/cms/soft/cmssh/src/cmssh/cmsfs.pyc in apply_filter(flt, gen)
66 def apply_filter(flt, gen):
67 """Apply given filter to a given set of results"""
---> 68 flt_func, flt_name = flt.split()
69 if isinstance(gen, list) or isinstance(gen, GeneratorType):
70 if flt_func == 'grep':

ValueError: too many values to unpack

Use system VO_CMS_SW_DIR if it exists

I need to modify installer to use existing CMSSW installation on given node. If VO_CMS_SW_DIR exists, I can create a link to this area, e.g. CMSSW -> $VO_CMS_SW_DIR. May be I'll need to create bunch of links, etc.

Protect cmssh loader from possible SiteDB outage (originally: does not load)

I'm getting the following error upon loading cmssh:

Loading external/libtiff ... DONE
Loading external/libungif ... DONE

cmssh+pylab Python environment [backend: Agg].

[TerminalIPythonApp] Error in loading extension: cmssh_extension

Check your config files in /home/joosep/cmssh/soft/.ipython/profile_cmssh

BadStatusLine Traceback (most recent call last)
/home/joosep/cmssh/soft/install/lib/python2.6/site-packages/IPython/core/extensions.pyc in load_extension(self, module_str)
88 if module_str not in sys.modules:
89 with prepended_to_syspath(self.ipython_extension_dir):
---> 90 import(module_str)
91 mod = sys.modules[module_str]
92 return self._call_load_ipython_extension(mod)

/home/joosep/cmssh/soft/.ipython/extensions/cmssh_extension.py in ()
25 from cmssh.iprint import PrintManager, print_error, print_warning, print_info
26 from cmssh.debug import DebugManager
---> 27 from cmssh.cms_cmds import dbs_instance, Magic, cms_find, cms_du
28 from cmssh.cms_cmds import cms_ls, cms_cp, verbose, cms_dqueue, cmscrab
29 from cmssh.cms_cmds import cms_rm, cms_rmdir, cms_mkdir, cms_root, cms_xrdcp

/home/joosep/cmssh/soft/cmssh/src/cmssh/cms_cmds.py in ()
21 from cmssh.iprint import msg_red, msg_green, msg_blue
22 from cmssh.iprint import print_warning, print_error, print_status, print_info
---> 23 from cmssh.filemover import copy_lfn, rm_lfn, mkdir, rmdir, list_se, dqueue
24 from cmssh.utils import list_results, check_os, unsupported_linux
25 from cmssh.utils import osparameters, check_voms_proxy, run, user_input

/home/joosep/cmssh/soft/cmssh/src/cmssh/filemover.py in ()
28 from cmssh.utils import PrintProgress, qlxml_parser
29 from cmssh.url_utils import get_data
---> 30 from cmssh.sitedb import SITEDBMGR
31
32 def get_dbs_se(lfn):

/home/joosep/cmssh/soft/cmssh/src/cmssh/sitedb.py in ()
74
75 # Singleton
---> 76 SITEDBMGR = SiteDBManager()

/home/joosep/cmssh/soft/cmssh/src/cmssh/sitedb.py in init(self, url, threshold)
47 self.timestamp = time.time()
48 self.threshold = threshold # in sec, default 3 hours
---> 49 self.init()
50
51 def init(self):

/home/joosep/cmssh/soft/cmssh/src/cmssh/sitedb.py in init(self)
54 url = self.url + '/site-names'
55 names = {}
---> 56 with get_data_and_close(url) as data:
57 for row in parser(data.read()):
58 names[row['site_name']] = row['alias']

/opt/software/cms/slc5_amd64_gcc462/external/python/2.6.4/lib/python2.6/contextlib.pyc in enter(self)
14 def enter(self):
15 try:
---> 16 return self.gen.next()
17 except StopIteration:
18 raise RuntimeError("generator didn't yield")

/home/joosep/cmssh/soft/cmssh/src/cmssh/url_utils.pyc in get_data_and_close(url, headers)
161 opener = urllib2.build_opener(handler)
162 urllib2.install_opener(opener)
--> 163 data = urllib2.urlopen(req)
164 try:
165 yield data

/opt/software/cms/slc5_amd64_gcc462/external/python/2.6.4/lib/python2.6/urllib2.pyc in urlopen(url, data, timeout)
122 if _opener is None:
123 _opener = build_opener()
--> 124 return _opener.open(url, data, timeout)
125
126 def install_opener(opener):

/opt/software/cms/slc5_amd64_gcc462/external/python/2.6.4/lib/python2.6/urllib2.pyc in open(self, fullurl, data, timeout)
387 req = meth(req)
388
--> 389 response = self._open(req, data)
390
391 # post-process response

/opt/software/cms/slc5_amd64_gcc462/external/python/2.6.4/lib/python2.6/urllib2.pyc in _open(self, req, data)
405 protocol = req.get_type()
406 result = self._call_chain(self.handle_open, protocol, protocol +
--> 407 '_open', req)
408 if result:
409 return result

/opt/software/cms/slc5_amd64_gcc462/external/python/2.6.4/lib/python2.6/urllib2.pyc in _call_chain(self, chain, kind, meth_name, _args)
365 func = getattr(handler, meth_name)
366
--> 367 result = func(_args)
368 if result is not None:
369 return result

/home/joosep/cmssh/soft/cmssh/src/cmssh/url_utils.pyc in https_open(self, req)
41 # a reference to a function which, for all intents and purposes,
42 # will behave as a constructor
---> 43 return self.do_open(self.get_connection, req)
44
45 def get_connection(self, host, timeout=300):

/opt/software/cms/slc5_amd64_gcc462/external/python/2.6.4/lib/python2.6/urllib2.pyc in do_open(self, http_class, req)
1117 try:
1118 h.request(req.get_method(), req.get_selector(), req.data, headers)
-> 1119 r = h.getresponse()
1120 except socket.error, err: # XXX what error?
1121 raise URLError(err)

/opt/software/cms/slc5_amd64_gcc462/external/python/2.6.4/lib/python2.6/httplib.pyc in getresponse(self)
972 method=self._method)
973
--> 974 response.begin()
975 assert response.will_close != _UNKNOWN
976 self.__state = _CS_IDLE

/opt/software/cms/slc5_amd64_gcc462/external/python/2.6.4/lib/python2.6/httplib.pyc in begin(self)
389 # read until we get a non-100 response
390 while True:
--> 391 version, status, reason = self._read_status()
392 if status != CONTINUE:
393 break

/opt/software/cms/slc5_amd64_gcc462/external/python/2.6.4/lib/python2.6/httplib.pyc in _read_status(self)
353 # Presumably, the server closed the connection before
354 # sending a valid response.
--> 355 raise BadStatusLine(line)
356 try:
357 [version, status, reason] = line.split(None, 2)

BadStatusLine:

slow response to keypress

Sometime in the last few days, cmssh became very slow to respond to keypresses and also drops them if you type too quickly. I've tried reinstalling the newest dev version, rebooting my computer, and running it both in iTerm and Terminal.app. The only other oddity I've noticed is that I get an extraneous print asking for my grid password. Here's what it looks like:

[cms] soft/bin/cmssh                                                  10:36:30 
Enter GRID pass phrase:
Your identity: /DC=org/DC=doegrids/OU=People/CN=Nicholas S Eggert 114717
Creating temporary proxy ......................................... Done
Contacting  lcg-voms.cern.ch:15002 [/DC=ch/DC=cern/OU=computers/CN=lcg-voms.cern.ch] "cms" Done
Creating proxy ........................................ Done
Your proxy is valid until Thu Apr 26 10:36:39 2012

Welcome to pylab, a matplotlib-based Python environment [backend: MacOSX].
For more information, type 'help(pylab)'.
Welcome to cmssh:
[python 2.6.4 (r264:75706, Sep  6 2011, 23:47:05) , ipython 0.12.1]
Darwin Kernel Version 11.3.0: Thu Jan 12 18:47:41 PST 2012; root:xnu-1699.24.23~1/RELEASE_X86_64

Available cmssh commands:
find         search CMS meta-data (query DBS/Phedex/SiteDB)
dbs_instance show/set DBS instance, default is DBS global instance
mkdir/rmdir  mkdir/rmdir command, e.g. mkdir /path/foo or rmdir T3_US_Cornell:/store/user/foo
ls           list file/LFN, e.g. ls local.file or ls /store/user/file.root
rm           remove file/LFN, e.g. rm local.file or rm T3_US_Cornell:/store/user/file.root
cp           copy file/LFN, e.g. cp local.file or cp /store/user/file.root .
info         provides detailed info about given CMS entity, e.g. info run=160915
dqueue       status of download queue, list files which are in progress.
root         invoke ROOT
du           display disk usage for given site, e.g. du T3_US_Cornell

Available CMSSW commands (once you install any CMSSW release):
releases     list available CMSSW releases
install      install CMSSW release, e.g. install CMSSW_5_0_0
cmsrel       switch to given CMSSW release and setup its environment
arch         show or switch to given CMSSW architecture
scram        CMSSW scram command
cmsRun       cmsRun command for release in question

Available GRID commands: <cmd> either grid or voms
<cmd>init     setup your proxy (aka <cmd>-proxy-init)
<cmd>info     show your proxy info (aka <cmd>-proxy-info)

Query results are accessible via results() function:
   find dataset=/*Zee*
   for r in results(): print r, type(r)

Help is accessible via cmshelp <command>

To install python software use pip <search|(un)install> <package>

cms-sh|1> Enter GRID pass phrase:

ImportError for module routes when launching

When I start cmssh, I get the following output:

ImportError Traceback (most recent call last)
/Library/Frameworks/EPD64.framework/Versions/7.1/lib/python2.7/site-packages/IPython/core/extensions.py in load_extension(self, module_str)
85 if module_str not in sys.modules:
86 with prepended_to_syspath(self.ipython_extension_dir):
---> 87 import(module_str)
88 mod = sys.modules[module_str]
89 return self._call_load_ipython_extension(mod)

/Users/nic/cms/soft/.ipython/extensions/cmssh_extension.py in ()
19 from cmssh.iprint import PrintManager
20 from cmssh.debug import DebugManager
---> 21 from cmssh.cms_cmds import dbs_instance, Magic
22 from cmssh.cms_cmds import lookup, cms_ls, cms_cp, verbose
23 from cmssh.cms_cmds import cms_rm, cms_rmdir, cms_mkdir, cms_root, cms_xrdcp

/Users/nic/cms/soft/cmssh/src/cmssh/cms_cmds.py in ()
15 from cmssh.filemover import copy_lfn, rm_lfn, mkdir, rmdir, list_se
16 from cmssh.utils import list_results
---> 17 from cmssh.cmsfs import dataset_info, block_info, file_info, site_info
18 from cmssh.cmsfs import CMSFS, apply_filter, validate_dbs_instance
19 from cmssh.cms_urls import dbs_url

/Users/nic/cms/soft/cmssh/src/cmssh/cmsfs.py in ()
10 import re
11 import json
---> 12 import routes
13 import urllib
14 import urllib2

ImportError: No module named routes

dqueue-summary

The output of dqueue when lots of jobs are in the queue is somewhat long and messy. Perhaps add a way to get just a summary of the number of jobs running, waiting, and finished.

I can think of two approaches for this.

  1. Change the current dqueue command to print something like:
Waiting:
job1
job2
job3
job4
job5
...
+ 73 other jobs (use dqueue --full to see all jobs)
  1. Leave dqueue as it is. Add a new command dqueue-summary or dqueue --summary which just prints the number of jobs in the in progress, waiting, and finshed states.

Enter PEM pass phrase:

I'm often prompted to enter a PEM pass phrase when trying to copy a file from a storage element. Sometimes I'm asked for every file. This is quite annoying when I'm trying to copy every file in a dataset using a for loop.

Try xrdls

Eric suggested to try use xrdls in addition or as replacement of srm/lcg interfaces

Setup magic functions for edm utilties

I need to add a hook to add into magic functions a list of CMSSW utitilies, e.g. edmDump, etc. when user switch from one release to another. The hook should keep track of them, and discard old ones. Can be done via internal dict.

crash with plt.savefig('test.pdf')

Apparently something wrong in the way we compile matplotlib that it cause crash when user try to save figure into PDF format, all other formats are fine

try to use CMSFS

In multi-user environment, if system provides /cvmfs/cms.cern.ch, it's better to detect it and use it instead of /afs.

Implement du SE:/path

I can implement du SE:/path since it provides me enough information about given path. First I need to create an objects which will hold dir structure. So, it can be DirObject which has the following attributes, list of files with their file size and list of sub-dirs. Then it would be easy to implement sum/total function.

Check code to look-up libreadline5

One user suggested that cmsh_install.py may incorrectly check presence of libreadline5 on a system. The /sbin/ldconfig -p may return a list where the first match may not be proper readline

Switch between architectures

Switch SCRAM_ARCH and install new/older releases. For example CMSSW_5_2 is using
export SCRAM_ARCH=slc5_amd64_gcc462, while CMSSW_5_0 based on gcc461 arch., etc.

macosx driver & plotting window

CMSSH matplotlib library does not ship with macosx driver. If I use the one I compile myself the plotting window is not showing, while graphics still can be save via savefig.

linux installation lacks two packages

Hello,

I just installed on lxplus. Two pip packages were missing to make it work:
pytz
pycrypto

I just added then in line 825 of cmssh_install.py to the std_pkgs list.

cheers, Heiner

Add eos support

source /afs/cern.ch/project/eos/installation/pro/etc/setup.sh

eos mkdir
eos ls -l

eos help for more details.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.