dmwm / cmssh Goto Github PK
View Code? Open in Web Editor NEWInteractive shell for CMS experiment
Home Page: http://cms.cern.ch/
Interactive shell for CMS experiment
Home Page: http://cms.cern.ch/
PyCurl can significantly speed up client part if used properly (see DAS/DBS3). I need to add pycurl HTTP manager and use it for all remote calls, to do so I need to keep around single instance of curl object.
To do so I need to add numpy, matplotlib, zeromq, tornado packages in place.
I'm not sure if this is a bug or if I just don't know what I'm doing. I'm trying to change the dbs instance. I assume that I should be able to do :
dbs_instance cms_dbs_ph_analysis_01
but that gives me an invalid DBS instance message. Is this not working properly or am I doing something wrong. Implementing dbs_instance --help might be useful in the future.
errorReply to [email protected]
Hi,
I get these warnings when I start cmssh:
/Users/nic/cms/soft/CMSSW/osx106_amd64_gcc421/external/py2-matplotlib/1.0.1-cms3/lib/python2.6/site-packages/matplotlib/__init__.py:397: UserWarning: matplotlibrc text.usetex option can not be used unless TeX-3.1415 or later is installed on your system
'installed on your system') % tex_req)
/Users/nic/cms/soft/CMSSW/osx106_amd64_gcc421/external/py2-matplotlib/1.0.1-cms3/lib/python2.6/site-packages/matplotlib/__init__.py:403: UserWarning: matplotlibrc text.usetex can not be used with *Agg backend unless dvipng-1.5 or later is installed on your system
warnings.warn( 'matplotlibrc text.usetex can not be used with *Agg '
/Users/nic/cms/soft/CMSSW/osx106_amd64_gcc421/external/py2-matplotlib/1.0.1-cms3/lib/python2.6/site-packages/matplotlib/__init__.py:417: UserWarning: matplotlibrc text.usetex can not be used unless ghostscript-7.07 or later is installed on your system
'installed on your system') % gs_req)
This happens because I have my matplotlibrc set up to use latex. You might consider forcing the system to use a matplotlibrc that you have written rather than picking up the one in the home directory to prevent problems like this. I've written a matplotlibrc that looks somewhat like the official CMS plot style, if you'd like me to send it to you.
Right now I use root_init in setup.sh. Instead I need to wrap root call based on SCRAM_ARCH. Remove root_init from setup.sh, modify cms_root to do root_init call.
error
I can improve integration lumi calculation by adding intermediate step to check if run_lumi dict are within boundaries of golden JSON file. To do so I may extend cms_cmds::run_lumi_info function and check presence of this file
/afs/cern.ch/cms/CAF/CMSCOMM/COMM_DQM/certification/Collisions12/8TeV/Prompt/Cert_190456-202305_8TeV_PromptReco_Collisions12_JSON.txt
If so, I need to add helper function to load golden JSON from it and use it for comparison with found run lumi dict.
Hi,
I just (foolishly) tried to cp every file in a dataset to my computer in the background. Java ate all my memory and I had to reboot my computer. It would be nice if there was a queue of some sort, so that only 4 or so downloads are running at once.
Nic
Nic reported problem with cmssh, see https://gist.github.com/2503811
Warner example shows that user needs to type several times GRID pass phrase.
I'm trying to install cmssh on ubuntu and I'm running into a problem with the detection of the readline library.
[~] python cmssh_install.py --help
Traceback (most recent call last):
File "cmssh_install.py", line 149, in <module>
DEF_SCRAM_ARCH = get_scram_arch()
File "cmssh_install.py", line 97, in get_scram_arch
raise Exception(msg)
Exception: cmssh on Linux requires readline5. Please verify that you have it installed on your system. So far we found
/usr/lib/libreadline.so -> /usr/lib/libreadline.so
I do, in fact have libreadline installed, but it's not finding it because it's in a different place. Here are the contents of the libreadline5 package in ubuntu.
[~] dpkg -L libreadline5/.
/etc
/lib
/lib/x86_64-linux-gnu
/lib/x86_64-linux-gnu/libreadline.so.5.2
/lib/x86_64-linux-gnu/libhistory.so.5.2
/usr
/usr/share
/usr/share/doc
/usr/share/doc/libreadline5
/usr/share/doc/libreadline5/USAGE
/usr/share/doc/libreadline5/inputrc.arrows
/usr/share/doc/libreadline5/examples
/usr/share/doc/libreadline5/examples/Inputrc
/usr/share/doc/libreadline5/copyright
/usr/share/doc/libreadline5/changelog.Debian.gz
/usr/share/doc/libreadline5/README.Debian
/lib/x86_64-linux-gnu/libreadline.so.5
/lib/x86_64-linux-gnu/libhistory.so.5
If I can make a suggestion, a better way to check for libreadline might be
/sbin/ldconfig -p | grep libreadline
or the equivalent python command.
I get an error on cmsenv (see below). I guess, it's my crazy machine again.. Haven't tested lxplus on yet.
cheers, Heiner
cms-sh|3> cmsenv
ERROR: OSError, fail to execute cmd=['eval', 'scramv1', 'runtime', '-sh
'], kwds={}, error=[Errno 2] No such file or directory
When I submit LFN transfer in background mode, it tries xrdcp method and fails sometimes since file cannot be transferred via xrdcp. So I need to fallback into lcg/srm mode somehow.
I'm having a problem copying by lfn. See the error message below. This file is in cms_dbs_ph_analysis_01, so maybe that has something to do with the problem. It looks fine if I prepend the site name to the lfn, i.e. T1_US_FNAL_Buffer:/store/user/...
cp /store/user/neggert/TT_TuneZ2_7TeV-mcatnlo/MCTSusy_Skim_Mar2012//7b5af1bfe3424f60f0db5b5f14cf327a/MCTSusySkimMar2012_100_1_kYt.root .
Traceback (most recent call last):
File "/Users/nic/cms/soft/cmssh/src/cmssh/cms_cmds.py", line 487, in cms_cp
status = copy_lfn(src, dst, verbose, background)
File "/Users/nic/cms/soft/cmssh/src/cmssh/filemover.py", line 610, in copy_lfn
status = FM_SINGLETON.copy(lfn, dst, verbose, background)
File "/Users/nic/cms/soft/cmssh/src/cmssh/filemover.py", line 456, in copy
for cmd in srmcp("srm-copy", lfn, dst, verbose):
File "/Users/nic/cms/soft/cmssh/src/cmssh/filemover.py", line 307, in srmcp
raise Exception(msg)
Exception: LFN: /store/user/neggert/TT_TuneZ2_7TeV-mcatnlo/MCTSusy_Skim_Mar2012//7b5af1bfe3424f60f0db5b5f14cf327a/MCTSusySkimMar2012_100_1_kYt.root
No replicas found
{u'phedex': {u'request_date': u'2012-04-06 18:09:06 UTC', u'request_timestamp': 1333735746.04251, u'call_time': u'0.01102', u'instance': u'prod', u'request_call': u'fileReplicas', u'request_url': u'http://cmsweb.cern.ch:7001/phedex/datasvc/json/prod/fileReplicas', u'request_version': u'2.3.8', u'block': []}}
Wrong argument '/store/user/neggert/TT_TuneZ2_7TeV-mcatnlo/MCTSusy_Skim_Mar2012//7b5af1bfe3424f60f0db5b5f14cf327a/MCTSusySkimMar2012_100_1_kYt.root .'
This installer was just downloaded today. I tried manually creating the directory it's complaining about, but it didn't help.
[cms] python cmssh_install.py -i -d cmssh 11:23:29
Checking CMSSW
Bootstrap CMSSW
Traceback (most recent call last):
File "cmssh_install.py", line 770, in <module>
main()
File "cmssh_install.py", line 352, in main
exe_cmd(sdir, cmd, debug, 'Bootstrap CMSSW', log='bootstrap.log')
File "cmssh_install.py", line 203, in exe_cmd
os.chdir(idir)
OSError: [Errno 2] No such file or directory: 'cmssh/soft/CMSSW'
I can hook-up git tool into cmssh and perform commits all user actions. Since git provides all versioning information and excellent logging capabilities it will means that whole history/provenance of user actions can easily captured. If user will be willing to setup git repository on github that it can be portable and distributable.
CRAB_2_8_1 is not available. Please update installation script to use CRAB_2_8_4_patch1.
Install CRAB
Traceback (most recent call last):
File "cmssh_install.py", line 1183, in
main()
File "cmssh_install.py", line 618, in main
get_file(url, 'crab.tar.gz', path, debug)
File "cmssh_install.py", line 357, in get_file
tar_file.write(getdata(url, debug))
File "cmssh_install.py", line 335, in getdata
data = urllib2.urlopen(req)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 126, in urlopen
return _opener.open(url, data, timeout)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 400, in open
response = meth(req, response)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 513, in http_response
'http', request, response, code, msg, hdrs)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 432, in error
result = self._call_chain(_args)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 372, in _call_chain
result = func(_args)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 619, in http_error_302
return self.parent.open(new, timeout=req.timeout)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 400, in open
response = meth(req, response)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 513, in http_response
'http', request, response, code, msg, hdrs)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 438, in error
return self._call_chain(_args)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 372, in _call_chain
result = func(_args)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 521, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 403: Forbidden
~: /afs/cern.ch/work/v/valya/public/soft/bin/cmssh õ
Welcome to cmssh! Loading configuration, please wait ...
Loading external/apt ... DONE
Loading lcg/root ... DONE
Loading external/python ... DONE
Loading external/xz ... DONE
Loading external/pcre ... DONE
Loading external/py2-matplotlib ... DONE
Loading external/py2-numpy ... DONE
Loading external/libpng ... DONE
Loading external/lapack ... DONE
Loading external/libjpg ... DONE
Loading external/libtiff ... DONE
Loading external/libungif ... DONE
Traceback (most recent call last):
File "/afs/cern.ch/user/v/valya/workspace/public/soft/install/bin/ipython", line 8, in <module>
load_entry_point('ipython==0.12.1', 'console_scripts', 'ipython')()
File "/afs/cern.ch/user/v/valya/workspace/public/soft/install/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/pkg_resources.py", line 318, in load_entry_point
File "/afs/cern.ch/user/v/valya/workspace/public/soft/install/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/pkg_resources.py", line 2221, in load_entry_point
File "/afs/cern.ch/user/v/valya/workspace/public/soft/install/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/pkg_resources.py", line 1954, in load
File "/afs/cern.ch/user/v/valya/workspace/public/soft/install/lib/python2.6/site-packages/IPython/__init__.py", line 43, in <module>
from .config.loader import Config
File "/afs/cern.ch/user/v/valya/workspace/public/soft/install/lib/python2.6/site-packages/IPython/config/loader.py", line 26, in <module>
from IPython.external import argparse
File "/afs/cern.ch/user/v/valya/workspace/public/soft/install/lib/python2.6/site-packages/IPython/external/argparse/__init__.py", line 10, in <module>
from _argparse import *
File "/afs/cern.ch/user/v/valya/workspace/public/soft/install/lib/python2.6/site-packages/IPython/external/argparse/_argparse.py", line 98, in <module>
from gettext import gettext as _
File "/afs/cern.ch/cms/slc5_amd64_gcc462/external/python/2.6.4-cms/lib/python2.6/gettext.py", line 49, in <module>
import locale, copy, os, re, struct, sys
File "/afs/cern.ch/user/v/valya/workspace/public/soft/install/lib64/python2.6/locale.py", line 202, in <module>
import re, operator
ImportError: /afs/cern.ch/user/v/valya/workspace/public/soft/install/lib64/python2.6/lib-dynload/operator.so: undefined symbol: _PyUnicodeUCS2_AsDefaultEncodedString
test body
Hi,
I'm trying to run a DAS query with a filter and I'm getting an error. This works just fine on the web interface:
ValueError Traceback (most recent call last)
/Users/nic/cms/MCTSusy/notebooks/Data/tW/ in ()
----> 1 get_ipython().magic(u'find file dataset=/DYToEE_M-10To20_TuneZ2_7TeV-pythia6/neggert-MCTSusy_Skim_Mar2012_DYToEE_M10To20-daf9758207e31d09324b57b0328014ab/USER | grep file.nevents>= 1')
/Users/nic/cms/soft/install/lib/python2.7/site-packages/IPython/core/interactiveshell.pyc in magic(self, arg_s, next_input)
1996 self._magic_locals = sys._getframe(1).f_locals
1997 with self.builtin_trap:
-> 1998 result = fn(magic_args)
1999 # Ensure we're not keeping object references around:
2000 self._magic_locals = {}
/Users/nic/cms/soft/cmssh/src/cmssh/cms_cmds.pyc in cms_find(arg)
137 Perform lookup of given query in CMS data-services.
138 """
--> 139 lookup(arg)
140
141 def cms_du(arg):
/Users/nic/cms/soft/cmssh/src/cmssh/cms_cmds.pyc in lookup(arg)
159 res = apply_filter(flt.strip(), gen)
160 RESMGR.assign(res)
--> 161 list_results(res, debug)
162
163 def verbose(arg):
/Users/nic/cms/soft/cmssh/src/cmssh/utils.pyc in list_results(res, debug)
77 return
78 if isinstance(res, list) or isinstance(res, GeneratorType):
---> 79 for row in res:
80 if not debug:
81 print row
/Users/nic/cms/soft/cmssh/src/cmssh/cmsfs.pyc in apply_filter(flt, gen)
66 def apply_filter(flt, gen):
67 """Apply given filter to a given set of results"""
---> 68 flt_func, flt_name = flt.split()
69 if isinstance(gen, list) or isinstance(gen, GeneratorType):
70 if flt_func == 'grep':
ValueError: too many values to unpack
Tab completion works for local files, but I possibly I can do it as well for SE paths.
I need to modify installer to use existing CMSSW installation on given node. If VO_CMS_SW_DIR exists, I can create a link to this area, e.g. CMSSW -> $VO_CMS_SW_DIR. May be I'll need to create bunch of links, etc.
I'm getting the following error upon loading cmssh:
Loading external/libtiff ... DONE
Loading external/libungif ... DONE
cmssh+pylab Python environment [backend: Agg].
[TerminalIPythonApp] Error in loading extension: cmssh_extension
BadStatusLine Traceback (most recent call last)
/home/joosep/cmssh/soft/install/lib/python2.6/site-packages/IPython/core/extensions.pyc in load_extension(self, module_str)
88 if module_str not in sys.modules:
89 with prepended_to_syspath(self.ipython_extension_dir):
---> 90 import(module_str)
91 mod = sys.modules[module_str]
92 return self._call_load_ipython_extension(mod)
/home/joosep/cmssh/soft/.ipython/extensions/cmssh_extension.py in ()
25 from cmssh.iprint import PrintManager, print_error, print_warning, print_info
26 from cmssh.debug import DebugManager
---> 27 from cmssh.cms_cmds import dbs_instance, Magic, cms_find, cms_du
28 from cmssh.cms_cmds import cms_ls, cms_cp, verbose, cms_dqueue, cmscrab
29 from cmssh.cms_cmds import cms_rm, cms_rmdir, cms_mkdir, cms_root, cms_xrdcp
/home/joosep/cmssh/soft/cmssh/src/cmssh/cms_cmds.py in ()
21 from cmssh.iprint import msg_red, msg_green, msg_blue
22 from cmssh.iprint import print_warning, print_error, print_status, print_info
---> 23 from cmssh.filemover import copy_lfn, rm_lfn, mkdir, rmdir, list_se, dqueue
24 from cmssh.utils import list_results, check_os, unsupported_linux
25 from cmssh.utils import osparameters, check_voms_proxy, run, user_input
/home/joosep/cmssh/soft/cmssh/src/cmssh/filemover.py in ()
28 from cmssh.utils import PrintProgress, qlxml_parser
29 from cmssh.url_utils import get_data
---> 30 from cmssh.sitedb import SITEDBMGR
31
32 def get_dbs_se(lfn):
/home/joosep/cmssh/soft/cmssh/src/cmssh/sitedb.py in ()
74
75 # Singleton
---> 76 SITEDBMGR = SiteDBManager()
/home/joosep/cmssh/soft/cmssh/src/cmssh/sitedb.py in init(self, url, threshold)
47 self.timestamp = time.time()
48 self.threshold = threshold # in sec, default 3 hours
---> 49 self.init()
50
51 def init(self):
/home/joosep/cmssh/soft/cmssh/src/cmssh/sitedb.py in init(self)
54 url = self.url + '/site-names'
55 names = {}
---> 56 with get_data_and_close(url) as data:
57 for row in parser(data.read()):
58 names[row['site_name']] = row['alias']
/opt/software/cms/slc5_amd64_gcc462/external/python/2.6.4/lib/python2.6/contextlib.pyc in enter(self)
14 def enter(self):
15 try:
---> 16 return self.gen.next()
17 except StopIteration:
18 raise RuntimeError("generator didn't yield")
/home/joosep/cmssh/soft/cmssh/src/cmssh/url_utils.pyc in get_data_and_close(url, headers)
161 opener = urllib2.build_opener(handler)
162 urllib2.install_opener(opener)
--> 163 data = urllib2.urlopen(req)
164 try:
165 yield data
/opt/software/cms/slc5_amd64_gcc462/external/python/2.6.4/lib/python2.6/urllib2.pyc in urlopen(url, data, timeout)
122 if _opener is None:
123 _opener = build_opener()
--> 124 return _opener.open(url, data, timeout)
125
126 def install_opener(opener):
/opt/software/cms/slc5_amd64_gcc462/external/python/2.6.4/lib/python2.6/urllib2.pyc in open(self, fullurl, data, timeout)
387 req = meth(req)
388
--> 389 response = self._open(req, data)
390
391 # post-process response
/opt/software/cms/slc5_amd64_gcc462/external/python/2.6.4/lib/python2.6/urllib2.pyc in _open(self, req, data)
405 protocol = req.get_type()
406 result = self._call_chain(self.handle_open, protocol, protocol +
--> 407 '_open', req)
408 if result:
409 return result
/opt/software/cms/slc5_amd64_gcc462/external/python/2.6.4/lib/python2.6/urllib2.pyc in _call_chain(self, chain, kind, meth_name, _args)
365 func = getattr(handler, meth_name)
366
--> 367 result = func(_args)
368 if result is not None:
369 return result
/home/joosep/cmssh/soft/cmssh/src/cmssh/url_utils.pyc in https_open(self, req)
41 # a reference to a function which, for all intents and purposes,
42 # will behave as a constructor
---> 43 return self.do_open(self.get_connection, req)
44
45 def get_connection(self, host, timeout=300):
/opt/software/cms/slc5_amd64_gcc462/external/python/2.6.4/lib/python2.6/urllib2.pyc in do_open(self, http_class, req)
1117 try:
1118 h.request(req.get_method(), req.get_selector(), req.data, headers)
-> 1119 r = h.getresponse()
1120 except socket.error, err: # XXX what error?
1121 raise URLError(err)
/opt/software/cms/slc5_amd64_gcc462/external/python/2.6.4/lib/python2.6/httplib.pyc in getresponse(self)
972 method=self._method)
973
--> 974 response.begin()
975 assert response.will_close != _UNKNOWN
976 self.__state = _CS_IDLE
/opt/software/cms/slc5_amd64_gcc462/external/python/2.6.4/lib/python2.6/httplib.pyc in begin(self)
389 # read until we get a non-100 response
390 while True:
--> 391 version, status, reason = self._read_status()
392 if status != CONTINUE:
393 break
/opt/software/cms/slc5_amd64_gcc462/external/python/2.6.4/lib/python2.6/httplib.pyc in _read_status(self)
353 # Presumably, the server closed the connection before
354 # sending a valid response.
--> 355 raise BadStatusLine(line)
356 try:
357 [version, status, reason] = line.split(None, 2)
BadStatusLine:
Sometime in the last few days, cmssh became very slow to respond to keypresses and also drops them if you type too quickly. I've tried reinstalling the newest dev version, rebooting my computer, and running it both in iTerm and Terminal.app. The only other oddity I've noticed is that I get an extraneous print asking for my grid password. Here's what it looks like:
[cms] soft/bin/cmssh 10:36:30
Enter GRID pass phrase:
Your identity: /DC=org/DC=doegrids/OU=People/CN=Nicholas S Eggert 114717
Creating temporary proxy ......................................... Done
Contacting lcg-voms.cern.ch:15002 [/DC=ch/DC=cern/OU=computers/CN=lcg-voms.cern.ch] "cms" Done
Creating proxy ........................................ Done
Your proxy is valid until Thu Apr 26 10:36:39 2012
Welcome to pylab, a matplotlib-based Python environment [backend: MacOSX].
For more information, type 'help(pylab)'.
Welcome to cmssh:
[python 2.6.4 (r264:75706, Sep 6 2011, 23:47:05) , ipython 0.12.1]
Darwin Kernel Version 11.3.0: Thu Jan 12 18:47:41 PST 2012; root:xnu-1699.24.23~1/RELEASE_X86_64
Available cmssh commands:
find search CMS meta-data (query DBS/Phedex/SiteDB)
dbs_instance show/set DBS instance, default is DBS global instance
mkdir/rmdir mkdir/rmdir command, e.g. mkdir /path/foo or rmdir T3_US_Cornell:/store/user/foo
ls list file/LFN, e.g. ls local.file or ls /store/user/file.root
rm remove file/LFN, e.g. rm local.file or rm T3_US_Cornell:/store/user/file.root
cp copy file/LFN, e.g. cp local.file or cp /store/user/file.root .
info provides detailed info about given CMS entity, e.g. info run=160915
dqueue status of download queue, list files which are in progress.
root invoke ROOT
du display disk usage for given site, e.g. du T3_US_Cornell
Available CMSSW commands (once you install any CMSSW release):
releases list available CMSSW releases
install install CMSSW release, e.g. install CMSSW_5_0_0
cmsrel switch to given CMSSW release and setup its environment
arch show or switch to given CMSSW architecture
scram CMSSW scram command
cmsRun cmsRun command for release in question
Available GRID commands: <cmd> either grid or voms
<cmd>init setup your proxy (aka <cmd>-proxy-init)
<cmd>info show your proxy info (aka <cmd>-proxy-info)
Query results are accessible via results() function:
find dataset=/*Zee*
for r in results(): print r, type(r)
Help is accessible via cmshelp <command>
To install python software use pip <search|(un)install> <package>
cms-sh|1> Enter GRID pass phrase:
Since I ship cmssh with pyROOT, I need to check how easy/hard will be analysis of ROOT files in cmssh. First step would be to check loading ROOT file content via PyROOT.
When I start cmssh, I get the following output:
ImportError Traceback (most recent call last)
/Library/Frameworks/EPD64.framework/Versions/7.1/lib/python2.7/site-packages/IPython/core/extensions.py in load_extension(self, module_str)
85 if module_str not in sys.modules:
86 with prepended_to_syspath(self.ipython_extension_dir):
---> 87 import(module_str)
88 mod = sys.modules[module_str]
89 return self._call_load_ipython_extension(mod)
/Users/nic/cms/soft/.ipython/extensions/cmssh_extension.py in ()
19 from cmssh.iprint import PrintManager
20 from cmssh.debug import DebugManager
---> 21 from cmssh.cms_cmds import dbs_instance, Magic
22 from cmssh.cms_cmds import lookup, cms_ls, cms_cp, verbose
23 from cmssh.cms_cmds import cms_rm, cms_rmdir, cms_mkdir, cms_root, cms_xrdcp
/Users/nic/cms/soft/cmssh/src/cmssh/cms_cmds.py in ()
15 from cmssh.filemover import copy_lfn, rm_lfn, mkdir, rmdir, list_se
16 from cmssh.utils import list_results
---> 17 from cmssh.cmsfs import dataset_info, block_info, file_info, site_info
18 from cmssh.cmsfs import CMSFS, apply_filter, validate_dbs_instance
19 from cmssh.cms_urls import dbs_url
/Users/nic/cms/soft/cmssh/src/cmssh/cmsfs.py in ()
10 import re
11 import json
---> 12 import routes
13 import urllib
14 import urllib2
ImportError: No module named routes
Once release is installed I need a new magic function to run cmsShow.
The versions of these libraries that are installed with CMSSW are somewhat old. Could the cmssh installer pull down the newest versions?
The output of dqueue
when lots of jobs are in the queue is somewhat long and messy. Perhaps add a way to get just a summary of the number of jobs running, waiting, and finished.
I can think of two approaches for this.
dqueue
command to print something like:Waiting:
job1
job2
job3
job4
job5
...
+ 73 other jobs (use dqueue --full to see all jobs)
dqueue
as it is. Add a new command dqueue-summary
or dqueue --summary
which just prints the number of jobs in the in progress, waiting, and finshed states.I'm often prompted to enter a PEM pass phrase when trying to copy a file from a storage element. Sometimes I'm asked for every file. This is quite annoying when I'm trying to copy every file in a dataset using a for loop.
my body
I need to add:
Eric suggested to try use xrdls in addition or as replacement of srm/lcg interfaces
I need to add a hook to add into magic functions a list of CMSSW utitilies, e.g. edmDump, etc. when user switch from one release to another. The hook should keep track of them, and discard old ones. Can be done via internal dict.
Apparently something wrong in the way we compile matplotlib that it cause crash when user try to save figure into PDF format, all other formats are fine
In multi-user environment, if system provides /cvmfs/cms.cern.ch, it's better to detect it and use it instead of /afs.
I can implement du SE:/path since it provides me enough information about given path. First I need to create an objects which will hold dir structure. So, it can be DirObject which has the following attributes, list of files with their file size and list of sub-dirs. Then it would be easy to implement sum/total function.
One user suggested that cmsh_install.py may incorrectly check presence of libreadline5 on a system. The /sbin/ldconfig -p may return a list where the first match may not be proper readline
Switch SCRAM_ARCH and install new/older releases. For example CMSSW_5_2 is using
export SCRAM_ARCH=slc5_amd64_gcc462, while CMSSW_5_0 based on gcc461 arch., etc.
It would be nice to have ability to kill transfer job
CMSSH matplotlib library does not ship with macosx driver. If I use the one I compile myself the plotting window is not showing, while graphics still can be save via savefig.
Hello,
I just installed on lxplus. Two pip packages were missing to make it work:
pytz
pycrypto
I just added then in line 825 of cmssh_install.py to the std_pkgs list.
cheers, Heiner
source /afs/cern.ch/project/eos/installation/pro/etc/setup.sh
eos mkdir
eos ls -l
eos help for more details.
my error
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.