Giter Club home page Giter Club logo

cctbx / cctbx_project Goto Github PK

View Code? Open in Web Editor NEW
210.0 42.0 111.0 126.93 MB

Computational Crystallography Toolbox

Home Page: https://cci.lbl.gov/docs/cctbx

License: Other

Python 79.44% C++ 15.84% C 2.67% Shell 0.09% TeX 0.24% HTML 0.42% Mathematica 0.50% Fortran 0.26% MATLAB 0.01% Scilab 0.01% CSS 0.01% Cuda 0.33% Makefile 0.02% Batchfile 0.01% Parrot 0.01% Java 0.02% NSIS 0.02% DTrace 0.01% GAP 0.01% JavaScript 0.10%
crystallography cctbx phenix cryo-em

cctbx_project's Introduction

Computational Crystallography Toolbox

Build Status Conda Version Nightly conda package tests Conda Platforms DOI

Table of Contents

The Computational Crystallography Toolbox (cctbx) is being developed as the open source component of the Phenix project. The goal of the Phenix project is to advance automation of macromolecular structure determination. Phenix depends on the cctbx, but not vice versa. This hierarchical approach enforces a clean design as a reusable library. The cctbx is therefore also useful for small-molecule crystallography and even general scientific applications.

The cctbx also provides some of the key component of the Olex 2 software. Olex 2 is dedicated to the workflow of small molecule crystallographic studies. It features a powerful and flexible refinement engine, olex2.refine, which is developed as part of the cctbx, in the smtbx top-module.

To maximize reusability and, maybe even more importantly, to give individual developers a notion of privacy, the cctbx is organized as a set of smaller modules. This is very much like a village (the cctbx project) with individual houses (modules) for each family (groups of developers, of any size including one).

The cctbx code base is available without restrictions and free of charge to all interested developers, both academic and commercial. The entire community is invited to actively participate in the development of the code base. A sophisticated technical infrastructure that enables community based software development is provided by GitHub. This service is also free of charge and open to the entire world.

The cctbx is designed with an open and flexible architecture to promote extendability and easy incorporation into other software environments. The package is organized as a set of ISO C++ classes with Python bindings. This organization combines the computational efficiency of a strongly typed compiled language with the convenience and flexibility of a dynamically typed scripting language in a strikingly uniform and very maintainable way.

Use of the Python interfaces is highly recommended, but optional. The cctbx can also be used purely as a C++ class library.

The easiest way to install cctbx is through the Conda package manager. We recommend the Miniforge installers since they provide a minimal environment and default to the conda-forge channel.

There are two packages available, cctbx and cctbx-base. The cctbx package is cctbx-base with some additional GUI packages (e.g. wxpython, pyside2, ipython).

With the conda command available, a new cctbx-base environment named my_env can be created with

conda create -n my_env -c conda-forge cctbx-base

To choose a specific version of Python, add the python package with the specific version

conda create -n my_env -c conda-forge cctbx-base python=3.8

Then the environment can be activated with

conda activate my_env

To install cctbx-base into the currently active environment, use

conda install -c conda-forge cctbx-base

The python package with a specific version can be added to change the version of python that is already installed in the active environment.

  1. Download https://raw.githubusercontent.com/cctbx/cctbx_project/master/libtbx/auto_build/bootstrap.py in the directory where the cctbx and its dependencies shall be installed
  2. Run python bootstrap.py (you may want to run it with the --help option first to discover the available options).
  • For better compatibility with newer operating systems, conda packages can be used for dependencies. Add the --use-conda flag and the command becomes python bootstrap.py --use-conda. This will run the miniconda installer if conda cannot be found. The environment with the dependencies will be located in the conda_base directory. See the description of the --use-conda flag from the --help output for more details.

The installation will take a long while but the script will verbosely describe what it does.

For a more detailed description on how to contribute to the cctbx please visit our contribution guide.

Build Status Conda Version Conda Platforms

A nightly build of the conda packages are available on the cctbx-nightly channel. To use these packages, prepend -c cctbx-nightly as a channel to the commands above. For example, the command to create a new my_env environment would become


conda create -n my_env -c cctbx-nightly -c conda-forge cctbx-base

This will use the cctbx-base package from the cctbx-nightly channel, but pull the remaining dependencies from conda-forge.

Nightly builds are only updated if there are additional commits from the previous build.

A subset of tests is run on the current cctbx-base packages from the conda-forge and cctbx-nightly channels every night (10 pm Pacific) to test compatibility with the latest packages from conda-forge. Additional source files for fable and antlr3 are needed for the tests. The nightly test details can be viewed by clicking the "Nightly conda package tests" badge near the beginning of this README.

cctbx_project's People

Contributors

alyubimov avatar anthchirp avatar bkpoon avatar dagewa avatar dcliebschner avatar dermen avatar dwpaley avatar gbunkoczi avatar graeme-winter avatar hattne avatar irees avatar irisdyoung avatar jjheadd avatar jmp1985 avatar luc-j-bourhis avatar monarin avatar naderm avatar natechols avatar nksauter avatar nwmoriarty avatar oeffner avatar olegsobolev avatar pafonine avatar phyy-nx avatar phzwart avatar rjgildea avatar rwgk avatar terwill avatar youdar avatar zeldebeest avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

cctbx_project's Issues

bootstrap broken on mac

1c6d48f broke mac builds

build log snippet:

(...)
: 
: Python build finished, but the necessary bits to build these modules were not found:
: _bsddb             dl                 gdbm
: imageop            linuxaudiodev      ossaudiodev
: readline           spwd               sunaudiodev
: To find the necessary bits, look in setup.py in detect_modules() for the module's name.
: 
: running build_scripts
: running install_lib
(...)
: running install_scripts
(..)
: running install_egg_info
(..)
: build
: Building './Python Launcher.app'
: Copying files
: Done.
: test -d "/Applications/Python 2.7" || mkdir -p "/Applications/Python 2.7"
: test -d "/Applications/Python 2.7/Python Launcher.app" && rm -r "/Applications/Python 2.7/Python Launcher.app"
: make[2]: [install] Error 1 (ignored)
: /bin/cp -r "Python Launcher.app" "/Applications/Python 2.7"
: cp: /Applications/Python 2.7/Python Launcher.app: Permission denied
: cp: Python Launcher.app: unable to copy extended attributes to /Applications/Python 2.7/Python Launcher.app: Permission denied
: cp: /Applications/Python 2.7/Python Launcher.app/Contents: No such file or directory
: cp: Python Launcher.app/Contents: unable to copy extended attributes to /Applications/Python 2.7/Python Launcher.app/Contents: No such file or directory
: cp: /Applications/Python 2.7/Python Launcher.app/Contents/Info.plist: No such file or directory
: cp: /Applications/Python 2.7/Python Launcher.app/Contents/MacOS: No such file or directory
: cp: Python Launcher.app/Contents/MacOS: unable to copy extended attributes to /Applications/Python 2.7/Python Launcher.app/Contents/MacOS: No such file or directory
: cp: /Applications/Python 2.7/Python Launcher.app/Contents/MacOS/Python Launcher: No such file or directory
: cp: /Applications/Python 2.7/Python Launcher.app/Contents/PkgInfo: No such file or directory
: cp: /Applications/Python 2.7/Python Launcher.app/Contents/Resources: No such file or directory
: cp: Python Launcher.app/Contents/Resources: unable to copy extended attributes to /Applications/Python 2.7/Python Launcher.app/Contents/Resources: No such file or directory
: cp: /Applications/Python 2.7/Python Launcher.app/Contents/Resources/English.lproj: No such file or directory
: cp: Python Launcher.app/Contents/Resources/English.lproj: unable to copy extended attributes to /Applications/Python 2.7/Python Launcher.app/Contents/Resources/English.lproj: No such file or directory
: cp: /Applications/Python 2.7/Python Launcher.app/Contents/Resources/English.lproj/Credits.rtf: No such file or directory
: cp: /Applications/Python 2.7/Python Launcher.app/Contents/Resources/English.lproj/MainMenu.nib: No such file or directory
: cp: Python Launcher.app/Contents/Resources/English.lproj/MainMenu.nib: unable to copy extended attributes to /Applications/Python 2.7/Python Launcher.app/Contents/Resources/English.lproj/MainMenu.nib: No such file or directory
: cp: /Applications/Python 2.7/Python Launcher.app/Contents/Resources/English.lproj/MainMenu.nib/classes.nib: No such file or directory
: cp: /Applications/Python 2.7/Python Launcher.app/Contents/Resources/English.lproj/MainMenu.nib/info.nib: No such file or directory
: cp: /Applications/Python 2.7/Python Launcher.app/Contents/Resources/English.lproj/MainMenu.nib/objects.nib: No such file or directory
: cp: /Applications/Python 2.7/Python Launcher.app/Contents/Resources/English.lproj/MyDocument.nib: No such file or directory
: cp: Python Launcher.app/Contents/Resources/English.lproj/MyDocument.nib: unable to copy extended attributes to /Applications/Python 2.7/Python Launcher.app/Contents/Resources/English.lproj/MyDocument.nib: No such file or directory
: cp: /Applications/Python 2.7/Python Launcher.app/Contents/Resources/English.lproj/MyDocument.nib/classes.nib: No such file or directory
: cp: /Applications/Python 2.7/Python Launcher.app/Contents/Resources/English.lproj/MyDocument.nib/info.nib: No such file or directory
: cp: /Applications/Python 2.7/Python Launcher.app/Contents/Resources/English.lproj/MyDocument.nib/objects.nib: No such file or directory
: cp: /Applications/Python 2.7/Python Launcher.app/Contents/Resources/English.lproj/PreferenceWindow.nib: No such file or directory
: cp: Python Launcher.app/Contents/Resources/English.lproj/PreferenceWindow.nib: unable to copy extended attributes to /Applications/Python 2.7/Python Launcher.app/Contents/Resources/English.lproj/PreferenceWindow.nib: No such file or directory
: cp: /Applications/Python 2.7/Python Launcher.app/Contents/Resources/English.lproj/PreferenceWindow.nib/classes.nib: No such file or directory
: cp: /Applications/Python 2.7/Python Launcher.app/Contents/Resources/English.lproj/PreferenceWindow.nib/info.nib: No such file or directory
: cp: /Applications/Python 2.7/Python Launcher.app/Contents/Resources/English.lproj/PreferenceWindow.nib/objects.nib: No such file or directory
: cp: /Applications/Python 2.7/Python Launcher.app/Contents/Resources/factorySettings.plist: No such file or directory
: cp: /Applications/Python 2.7/Python Launcher.app/Contents/Resources/PythonCompiled.icns: No such file or directory
: cp: /Applications/Python 2.7/Python Launcher.app/Contents/Resources/PythonLauncher.icns: No such file or directory
: cp: /Applications/Python 2.7/Python Launcher.app/Contents/Resources/PythonSource.icns: No such file or directory
: make[2]: *** [install] Error 1
: make[1]: *** [install_PythonLauncher] Error 2
: make: *** [frameworkinstallapps] Error 2
Traceback (most recent call last):
  File "modules/cctbx_project/libtbx/auto_build/install_base_packages.py", line 1480, in <module>
    installer(args=sys.argv, log=sys.stdout)
  File "modules/cctbx_project/libtbx/auto_build/install_base_packages.py", line 217, in __init__
    self.build_dependencies(packages=packages)
  File "modules/cctbx_project/libtbx/auto_build/install_base_packages.py", line 683, in build_dependencies
    getattr(self, 'build_%s'%i)()
  File "modules/cctbx_project/libtbx/auto_build/install_base_packages.py", line 748, in build_python
    self.call('make install', log=log, cwd=python_dir)
  File "modules/cctbx_project/libtbx/auto_build/install_base_packages.py", line 307, in call
    return call(args, log=log, verbose=self.verbose, **kwargs)
  File "/Users/dlshudson/jenkins_slave/workspace/dials_bootstrap_platforms/label/dials-mac-mini/build_dials/modules/cctbx_project/libtbx/auto_build/installer_utils.py", line 81, in call
    raise RuntimeError("Call to '%s' failed with exit code %d" % (args, rc))
RuntimeError: Call to 'make install' failed with exit code 2
Performing actions: base

excessive memory usage opening large files

Hi folks,

I have a multi-image file, a copy of which is at /dls/tmp/david/Lys2_grid3_diffraction_lamella2_noSA_1.ser. This is a 7.4G file containing 117 diffraction images. Starting dials.image_viewer with this file is okay - the process uses 711M virtual memory on my computer. When I click 'Next' this goes up to 793M. Click 'Next' again and it settles at 1049M. As I keep clicking 'Next' this total increases. By image 20 it is at 5017M.

If I try to jump from image 20 to 117 the image viewer greys out, the virtual memory usage leaps to 10.9G and eventually there is a traceback in the terminal:

Traceback (most recent call last):
  File "/home/david/sw/cctbx/modules/dials/util/image_viewer/spotfinder_frame.py", line 202, in OnChooseImage
    super(SpotFrame, self).OnChooseImage(event)
  File "/home/david/sw/cctbx/modules/cctbx_project/rstbx/viewer/frame.py", line 328, in OnChooseImage
    self.image_chooser.GetSelection()))
  File "/home/david/sw/cctbx/modules/dials/util/image_viewer/spotfinder_frame.py", line 470, in load_image
    show_untrusted=show_untrusted)
  File "/home/david/sw/cctbx/modules/cctbx_project/rstbx/slip_viewer/frame.py", line 345, in load_image
    get_raw_data=get_raw_data)
  File "/home/david/sw/cctbx/modules/cctbx_project/rstbx/slip_viewer/tile_generation.py", line 250, in set_image
    self.raw_image.set_raw_data(get_raw_data(self.raw_image))
  File "/home/david/sw/cctbx/modules/dials/util/image_viewer/spotfinder_frame.py", line 737, in get_raw_data
    raw_data = image.get_raw_data()
  File "/home/david/sw/cctbx/modules/cctbx_project/rstbx/slip_viewer/frame.py", line 37, in get_raw_data
    self._raw_data = self.image_set[self.index]
  File "/home/david/sw/cctbx/modules/cctbx_project/dxtbx/imageset.py", line 1155, in __getitem__
    return self.get_corrected_data(item)
  File "/home/david/sw/cctbx/modules/cctbx_project/dxtbx/imageset.py", line 570, in get_corrected_data
    data = self.get_raw_data(index)
  File "/home/david/sw/cctbx/modules/cctbx_project/dxtbx/imageset.py", line 559, in get_raw_data
    image = self.reader().read(self._indices[index])
  File "/home/david/sw/cctbx/modules/cctbx_project/dxtbx/imageset.py", line 181, in read
    return self.get_format().get_raw_data(index)
  File "/home/david/sw/cctbx/modules/cctbx_project/dxtbx/format/FormatSEReBIC.py", line 73, in get_raw_data
    raw_data = super(FormatSEReBIC, self).get_raw_data(index)
  File "/home/david/sw/cctbx/modules/cctbx_project/dxtbx/format/FormatSER.py", line 145, in get_raw_data
    d['CalibrationOffsetX'] = struct.unpack('<d', f.read(8))[0]
  File "/home/david/sw/cctbx/modules/cctbx_project/dxtbx/filecache.py", line 314, in read
    data, self._seek = self._cache_object.pass_read(start=self._seek, maxbytes=size)
  File "/home/david/sw/cctbx/modules/cctbx_project/dxtbx/filecache.py", line 211, in pass_read
    self._cache_up_to(start + maxbytes)
  File "/home/david/sw/cctbx/modules/cctbx_project/dxtbx/filecache.py", line 116, in _cache_up_to
    self._cache_object.write(data)
OverflowError: length too large


Why is this happening like this? Is it something to do with the dxtbx file cache?

Cheers
David

dxtbx DataBlockTemplateImporter format_kwargs bug

When dials.import is run using the template importer, there is an extra "format_kwargs" appearing in the resulting datablock.json, which means the dynamic_shadowing option gets ignored:

$ dials.import $(libtbx.find_in_repositories dials_regression)/centroid_test_data/centroid_000*.cbf dynamic_shadowing=True output.datablock=1.json
$ dials.import template=$(libtbx.find_in_repositories dials_regression)/centroid_test_data/centroid_####.cbf dynamic_shadowing=True output.datablock=2.json
$ grep -A 2 -B 2 dynamic_shadowing *.json
1.json-        "scan": 0, 
1.json-        "params": {
1.json:          "dynamic_shadowing": true, 
1.json-          "multi_panel": false
1.json-        }
--
--
2.json-        "params": {
2.json-          "format_kwargs": {
2.json:            "dynamic_shadowing": true
2.json-          }
2.json-        }

phil parameter disambiguation

(originally reported in dials/dials/issues/265)

I assert we can do better than

Sorry: Ambiguous parameter definition: global_threshold = 20.0
Best matches:
  spotfinder.threshold.xds.global_threshold
  spotfinder.threshold.helen.global_threshold

Suggest distinguishing two levels of ambiguity.

  • If one of the matches has a minimum expert_level, ie which is lower than that of all alternative matches, then print a warning: You said xx. I think you meant 'banana' and I will proceed under this assumption..
  • If two or more matches have the same minimum expert_levels, fail. Potentially list the more likely options first.

Update Python to 2.7.13

We tried updating Python to 2.7.12 before, and had to revert to 2.7.8 when we came across SSL issues on macOS and Debian/Ubuntu, cf. 22b7727

This needs more investigating

phenix.map_value_at_point (or eight_point_interpolation) does not work with ccp4 maps

As discussed more than one year ago in http://phenix-online.org/pipermail/cctbxbb/2015-December/001181.html, phenix.map_value_at_point (or eight_point_interpolation in maptbx) does not work with CCP4 (MRC) maps from phenix.maps.

$ phenix.fetch_pdb --mtz 3eml
$ phenix.maps 3eml.pdb 3eml-sf.cif

# Just to use the latest svn version
$ phenix.map_value_at_point 3eml_2mFo-DFc_map.ccp4 point="1.0 2.0 3.0"
Default params::

  label = None
  point = None
  resolution_factor = 0.25
  scale = *sigma volume
  resolution = None
  low_resolution = None

Map values at specified points:
Traceback (most recent call last):
  File "/misc/home/app/phenix-1.11.1-2575/build/../modules/cctbx_project/mmtbx/command_line/map_value_at_point.py", line 162, in <module>
    run(sys.argv[1:])
  File "/misc/home/app/phenix-1.11.1-2575/build/../modules/cctbx_project/mmtbx/command_line/map_value_at_point.py", line 146, in run
    "%10.3f"%map_3d.eight_point_interpolation(point_frac)).strip()
Boost.Python.ArgumentError: Python argument types in
    cctbx_maptbx_ext.eight_point_interpolation(double, tuple)
did not match C++ signature:
    eight_point_interpolation(scitbx::af::const_ref<double, scitbx::af::c_grid_padded<3ul, unsigned long> >, scitbx::vec3<double>)

However, it does work with maps generated by FFT in CCP4.

$ fft hklin 3eml_map_coeffs.mtz mapout fft.ccp4 <<EOF
LABIN F1=2FOFCWT PHI=PH2FOFCWT
END
EOF

$ phenix.map_value_at_point fft.ccp4 point="1.0 2.0 3.0"
Default params::

  label = None
  point = None
  resolution_factor = 0.25
  scale = *sigma volume
  resolution = None
  low_resolution = None

Map values at specified points:
  Input point: (1.000,2.000,3.000) Fractionalized: (0.034,0.026,0.035) Map value: -0.009

All done.

Boost 1.65: boost::python::numeric removed

As per http://www.boost.org/users/history/version_1_65_1.html:

The boost::python::numeric API has been removed, as it is being obsoleted by boost::python::numpy.

This is used in (at least) scitbx/.../numpy_bridge.hpp and thus anything that depends on it. I tried literally just renaming the interface and providing compatibility macros (ndevenish/cctbx_project@1d25c89 was the very rough first attempt), and that wasn't sufficient, but didn't get time to delve any deeper.

More of an "in the future" problem since we don't use it at the moment - since this was just discussed I thought I'd add it here to track. I don't think this is the only issue with Boost 1.65.

The new numpy API was only added in Boost 1.63, so potentially moving to that as a minimum requirement first would mean that we don't have to conditionally adhere to both APIs, but I don't know how realistic that is.

Build failing on macs: urllib2.urlopen error while getting HDF5

base is failing on all macs as of a few days. The full traceback from buildbot is below. This appears to be a bug in the ssl libraries we are building on mac before we build python. Possible solutions:

  1. Get HDF5 from cci.lbl.gov instead of the HDF5 webserver
  2. Apply a wonky patch to our ssl libraries. Stack overflow has a few ideas
  3. ???

Installing HDF5...
log file is /Users/builder/slave/phenix-nightly-mac-intel-osx-x86_64/base_tmp/HDF5_install_log
getting package hdf5-1.10.0-patch1.tar.bz2...
downloading from https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.10/hdf5-1.10.0-patch1/src/ : installer(args=sys.argv, log=sys.stdout)
File "modules/cctbx_project/libtbx/auto_build/install_base_packages.py", line 217, in init
self.build_dependencies(packages=packages)
File "modules/cctbx_project/libtbx/auto_build/install_base_packages.py", line 647, in build_dependencies
getattr(self, 'build_%s'%i)()
File "modules/cctbx_project/libtbx/auto_build/install_base_packages.py", line 987, in build_hdf5
hdf5pkg = self.fetch_package(pkg_name=HDF5_PKG, pkg_url=BASE_HDF5_PKG_URL)
File "/Users/builder/slave/phenix-nightly-mac-intel-osx-x86_64/modules/cctbx_project/libtbx/auto_build/package_defs.py", line 214, in call
size = self.toolbox.download_to_file(full_url, output_file, log=self.log)
File "/Users/builder/slave/phenix-nightly-mac-intel-osx-x86_64/modules/cctbx_project/libtbx/auto_build/bootstrap.py", line 268, in download_to_file
socket = urllib2.urlopen(url_request)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 127, in urlopen
return _opener.open(url, data, timeout)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 404, in open
response = self._open(req, data)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 422, in _open
'_open', req)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 382, in _call_chain
result = func(*args)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1222, in https_open
return self.do_open(httplib.HTTPSConnection, req)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1184, in do_open
raise URLError(err)
urllib2.URLError: <urlopen error [Errno 8] _ssl.c:504: EOF occurred in violation of protocol>

MTZ column selection in phenix.map_value_at_point

As reported a year ago in cctbxbb, phenix.map_value_at_point has a bug in MTZ column selection.

When there are multiple columns in an mtz file and the specified label does not match any of them, the last column is used without warning.

https://github.com/cctbx/cctbx_project/blob/master/mmtbx/command_line/map_value_at_point.py#L78
should be something like this:

    found = False
    for ma in miller_arrays:
      if(ma.info().labels[0].lower() == params.label.lower()):
        found = True
        break
    if not found:
      raise Sorry("Specified label not found.")

(details in http://phenix-online.org/pipermail/cctbxbb/2015-December/001180.html and http://phenix-online.org/pipermail/cctbxbb/2015-December/001183.html)

Could someone please verify the above and commit the fix?

building in jenkins

I'm building with
python bootstrap.py
from within a jenkins job and it errors on the python not having the ssl module

Installing Python...
  log file is /var/lib/jenkins/workspace/cctbx/cctbx/base_tmp/Python_install_log
  getting package Python-2.7.8_cci.tar.gz...
    downloading from http://cci.lbl.gov/cctbx_dependencies : 14.3 MB
    [0%.........20%.........40%.........60%.........80%.........100%]
  installing Python-2.7.8_cci.tar.gz...
Using Python: /var/lib/jenkins/workspace/cctbx/cctbx/base/bin/python
  verifying Python installation in /var/lib/jenkins/workspace/cctbx/cctbx/base/bin/python OK
  verifying zlib installation in /var/lib/jenkins/workspace/cctbx/cctbx/base/bin/python OK
  verifying _ssl installation in /var/lib/jenkins/workspace/cctbx/cctbx/base/bin/pythonTraceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named _ssl



********************************************************************************

This python does not have _ssl - Secure Socket Library installed

********************************************************************************



Traceback (most recent call last):
  File "modules/cctbx_project/libtbx/auto_build/install_base_packages.py", line 1450, in <module>
    installer(args=sys.argv, log=sys.stdout)
  File "modules/cctbx_project/libtbx/auto_build/install_base_packages.py", line 200, in __init__
    self.build_dependencies(packages=packages)
  File "modules/cctbx_project/libtbx/auto_build/install_base_packages.py", line 630, in build_dependencies
    getattr(self, 'build_%s'%i)()
  File "modules/cctbx_project/libtbx/auto_build/install_base_packages.py", line 722, in build_python
    self.set_python(op.abspath(python_exe))
  File "modules/cctbx_project/libtbx/auto_build/install_base_packages.py", line 395, in set_python
    self.check_python_dependencies()
  File "modules/cctbx_project/libtbx/auto_build/install_base_packages.py", line 387, in check_python_dependencies
    raise e
RuntimeError: Call to '/var/lib/jenkins/workspace/cctbx/cctbx/base/bin/python -c 'import _ssl'' failed with exit code 1
===== Running in modules: extracting files from clipper.gz

  removing .pyc files in /var/lib/jenkins/workspace/cctbx/cctbx/modules
  removed 0 files
===== Running in .: base
Process failed with return code 1

anyone run into this? Is there a work around?

.alias phil attribute weirdness

Does anyone use the .alias attribute for phil scopes? Like the one in grand_master_phil_str in mmtbx/monomer_library/pdb_interpretaion.py? It seems like the attribute causes issues when getting the diff between 2 phil scopes.

If you run the code pasted below, you'll get a difference between the same, unmodified phil scope objects. The code works if the .alias lines are removed from grand_master_phil_str. It is also weird that only the secondary_structure.enabled parameter is always set to True when the default value is False.

from mmtbx.monomer_library.pdb_interpretation import master_params_str
from mmtbx.monomer_library.pdb_interpretation import grand_master_phil_str
from iotbx.phil import parse

if (__name__ == '__main__'):

  # works normally
  master_phil = parse(master_params_str, process_includes=True)
  working_phil = parse(master_params_str, process_includes=True)
  diff = master_phil.fetch_diff(source=working_phil)
  print 'No .alias (should be empty)'
  print '=' * 79
  print 'secondary_structure.enabled'
  print 'master value: %s' % master_phil.extract().secondary_structure.enabled
  print 'working value: %s' % working_phil.extract().secondary_structure.enabled
  print 'diff:'
  diff.show()
  print

  # works weirdly
  grand_master_phil = parse(grand_master_phil_str, process_includes=True)
  grand_working_phil = parse(grand_master_phil_str, process_includes=True)
  grand_diff = grand_master_phil.fetch_diff(source=grand_working_phil)
  print 'With .alias (should be empty)'
  print '=' * 79
  print 'pdb_interpretation.secondary_structure.enabled'
  print 'master value: %s' % grand_master_phil.extract().pdb_interpretation.\
      secondary_structure.enabled
  print 'working value: %s' % grand_working_phil.extract().pdb_interpretation.\
      secondary_structure.enabled
  print 'diff:'
  grand_diff.show()
  print

Code examples

Write some code examples for models, maps, reflections, flex arrays and select scitbx topics

Rigaku Saturn: respect _all_ elements in image header

i.e. even

CCD_SPATIAL_DISTORTION_INFO=518.3300  514.7750 0.0900 0.0900;
CCD_SPATIAL_DISTORTION_TYPE=Simple_spatial;
CCD_SPATIAL_DISTORTION_VECTORS=0 -1 1 0;

which switch the definition of the fast, slow vectors

electron diffraction intensities in ring patterns

I don't know if this is the right place to ask this question but I am having trouble using cctbx to simulate the profile of electron diffraction ring patterns.

I wrote a program to integrate electron diffraction rings in TEM for ease of analysis and I have been using external software like gdis or crystal maker to simulate the expected electron diffraction pattern.

I was exited to find that cctbx had a table for electron scattering and I have been trying to use it to get structure factors from .cif files. After following this documentation, I using the electron scattering tables.

I get:
gdis and cctbx

The electron diffraction intensities don't fall off as fast as I would expect. I was wondering if these intensities include other corrections such as debye-waller or what else I might need.

The normalized intensities from a gdis electron and x-ray simulation are also included. As far as I can tell the gdis electron simulation uses the Mott-Bethe on the x-ray form factors and it fits the data quite nicely.

glib build issue

I'm hitting a new error. This is with a fresh build folder and fresh wget https://raw.githubusercontent.com/cctbx/cctbx_project/master/libtbx/auto_build/bootstrap.py. Building on a CentOS 6 box: 2.6.32-573.7.1.el6.centos.plus.x86_64 #1 SMP Wed Sep 23 03:02:55 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux and gcc 6.3.

Had built successfully with the same box and script Dec 22, 2016. No changes to this box or compiler in the duration.

Installing glib...
  log file is /home/roger/Software/cctbx/base_tmp/glib_install_log
  getting package glib-2.46.2.tar.gz...
    downloading from http://cci.lbl.gov/cctbx_dependencies : 14.1 MB
    [0%.........20%.........40%.........60%.........80%.........100%]
  installing glib-2.46.2.tar.gz...
Traceback (most recent call last):
  File "modules/cctbx_project/libtbx/auto_build/install_base_packages.py", line 1450, in <module>
    installer(args=sys.argv, log=sys.stdout)
  File "modules/cctbx_project/libtbx/auto_build/install_base_packages.py", line 200, in __init__
    self.build_dependencies(packages=packages)
  File "modules/cctbx_project/libtbx/auto_build/install_base_packages.py", line 630, in build_dependencies
    getattr(self, 'build_%s'%i)()
  File "modules/cctbx_project/libtbx/auto_build/install_base_packages.py", line 1098, in build_glib
    log=pkg_log)
  File "modules/cctbx_project/libtbx/auto_build/install_base_packages.py", line 501, in configure_and_build
    self.call("make -j %d %s" % (nproc, " ".join(list(make_args))), log=log)
  File "modules/cctbx_project/libtbx/auto_build/install_base_packages.py", line 295, in call
    return call(args, log=log, verbose=self.verbose, **kwargs)
  File "/home/roger/Software/cctbx/modules/cctbx_project/libtbx/auto_build/installer_utils.py", line 81, in call
    raise RuntimeError("Call to '%s' failed with exit code %d" % (args, rc))
RuntimeError: Call to 'make -j 1 ' failed with exit code 2
Process failed with return code 1

Update Boost to 1.56

Issue to track boost 1.56 update.

DIALS 1.5 is now branched, so proceed when ready.

FlexImage needs to support passing a mask array for display of masked pixels

Currently dials.image_viewer "show_mask" functionality relies on setting masked pixels to -2 in order for FlexImage to colour masked pixels red. This is not ideal as it can result in some non-masked pixels being mistakenly displayed as masked in the image viewer for some format classes.

if (has_pilatus_inactive_flag && raw[idx_ij]==-2){

https://github.com/dials/dials/blob/master/util/image_viewer/spotfinder_frame.py#L1014

cctbx/maptbx/tst_resolution_from_map_and_model.py broken

Collectively, commits @d796909, @6ce3004, @875d30a and @79300f6 broke the build, test cctbx/maptbx/tst_resolution_from_map_and_model.py.
cf. https://github.com/cctbx/cctbx_project/commits/master

(...)
15.1 0.838744251443 15.0
15.2 0.826037717701 15.0
15.3 0.826037717701 15.0
15.4 0.825890196726 15.0
15.5 0.825890196726 15.0
20 8.0 <> 15.0 8.1 <> 0.999331342895
approx_equal eps: 1e-06
approx_equal multiplier: 10000000000.0
20 approx_equal ERROR
15.0 approx_equal ERROR
Traceback (most recent call last):
  File "/scratch/jenkins_slave/workspace/cctbx_tests/build_dials/modules/cctbx_project/cctbx/maptbx/tst_resolution_from_map_and_model.py", line 487, in <module>
    assert approx_equal(b, b_result)
AssertionError

conda distribution?

hi,

is there any plan to build and distribute via conda?
It's very popular now. We (AMBER) are thinking about distributing AmberTools binary via conda too.

iotbx.xds.xparm reads too much

This line

lines = file_handle.readlines()

in xparm.py causes problems for e.g. dials.import, because it reads the entire file into memory. This happens when FormatXDS is trying to understand an image file - bad news in the case I've just looked at, when that file is a 7.5GB multi-image file!

wxtbx utils: Gtk-CRITICAL... assertion 'width >= -1' failed messages

On Ubuntu at least, dials.reciprocal_lattice_viewer emits a number of annoying messages to the terminal when it starts up:

Gtk-Message: Failed to load module "overlay-scrollbar"
Gtk-Message: Failed to load module "atk-bridge"
Gtk-Message: Failed to load module "unity-gtk-module"
Gtk-Message: Failed to load module "canberra-gtk-module"

(reciprocal_lattice_viewer.py:28128): Gtk-CRITICAL **: IA__gtk_widget_set_size_request: assertion 'width >= -1' failed

(reciprocal_lattice_viewer.py:28128): Gtk-CRITICAL **: IA__gtk_widget_set_size_request: assertion 'width >= -1' failed

(reciprocal_lattice_viewer.py:28128): Gtk-CRITICAL **: IA__gtk_widget_set_size_request: assertion 'width >= -1' failed

(reciprocal_lattice_viewer.py:28128): Gtk-CRITICAL **: IA__gtk_widget_set_size_request: assertion 'width >= -1' failed

(reciprocal_lattice_viewer.py:28128): Gtk-CRITICAL **: IA__gtk_widget_set_size_request: assertion 'width >= -1' failed

(reciprocal_lattice_viewer.py:28128): Gtk-CRITICAL **: IA__gtk_widget_set_size_request: assertion 'width >= -1' failed

I have found that the last 6 of these messages go away with this simple patch:

$ git diff
diff --git a/wxtbx/utils.py b/wxtbx/utils.py
index 17e1bd7..0cb2139 100644
--- a/wxtbx/utils.py
+++ b/wxtbx/utils.py
@@ -106,6 +106,7 @@ class SettingsPanel (wx.Panel, SettingsToolBase) :
     self.panel_sizer = wx.BoxSizer(wx.VERTICAL)
     self.SetSizer(self.panel_sizer)
     self.add_controls()
+    self.panel_sizer.SetSizeHints(self)
     self.panel_sizer.Layout()
 
 def bold_text (parent, label) :

Now, I don't pretend to understand what this line actually does or what effect it has on other projects. I discovered this by trial and error. Could somebody with a better knowledge of wxPython comment here?

cctbx.crystal.reindex.reindexing_operators() sometimes fails

Here is the code to reproduce the problem:

from cctbx.crystal import reindex
from cctbx import crystal
xs1=crystal.symmetry((40.3,81.7,199.9,93.2,95.7,104.1), 1)
xs2=crystal.symmetry((40.1,81.4,189.6,92.6,89.0,103.8), 1)
reindex.reindexing_operators(xs1, xs2)

and we get:

Traceback (most recent call last):
  File "./test.py", line 5, in 
    reindex.reindexing_operators(xs1, xs2)
  File "cctbx_build/src/cctbx/crystal/reindex.py", line 59, in __init__
    try: common_lattice_group.expand_smx(s)
AttributeError: 'NoneType' object has no attribute 'expand_smx'

Clearly we need break after replacing common_lattice_group with None https://github.com/cctbx/cctbx_project/blob/master/cctbx/crystal/reindex.py#L61

I will commit this if nobody objects.

Remaining side effects for Eiger data sets following dxtbx refactor

e.g.

Graemes-MacBook-Pro-5:index graeme$ dials.generate_mask  'input.datablock="SWEEP1_datablock.json"' 'output.mask="1_mask.pickle"' 'output.datablock="1_SWEEP1_datablock.json"' '1_mask.phil'
Traceback (most recent call last):
  File "/Users/graeme/svn/cctbx/build/../modules/dials/command_line/generate_mask.py", line 117, in <module>
    halraiser(e)
  File "/Users/graeme/svn/cctbx/build/../modules/dials/command_line/generate_mask.py", line 115, in <module>
    script.run()
  File "/Users/graeme/svn/cctbx/build/../modules/dials/command_line/generate_mask.py", line 74, in run
    params, options = self.parser.parse_args(show_diff_phil=True)
  File "/Users/graeme/svn/cctbx/modules/dials/util/options.py", line 783, in parse_args
    quick_parse=quick_parse)
  File "/Users/graeme/svn/cctbx/modules/dials/util/options.py", line 454, in parse_args
    track_unused_definitions=True)
  File "/Users/graeme/svn/cctbx/modules/cctbx_project/libtbx/phil/__init__.py", line 1857, in fetch
    skip_incompatible_objects=skip_incompatible_objects)
  File "/Users/graeme/svn/cctbx/modules/cctbx_project/libtbx/phil/__init__.py", line 1881, in fetch
    source=candidate).as_str()
  File "/Users/graeme/svn/cctbx/modules/cctbx_project/libtbx/phil/__init__.py", line 1137, in extract_format
    return self.format(python_object=source.extract())
  File "/Users/graeme/svn/cctbx/modules/cctbx_project/libtbx/phil/__init__.py", line 1122, in extract
    return self._type_from_words()(self.words, master=self)
  File "/Users/graeme/svn/cctbx/modules/dials/util/phil.py", line 51, in from_words
    return self.from_string(libtbx.phil.str_from_words(words=words))
  File "/Users/graeme/svn/cctbx/modules/dials/util/phil.py", line 47, in from_string
    check_format=self._check_format))
  File "/Users/graeme/svn/cctbx/modules/cctbx_project/dxtbx/datablock.py", line 920, in from_json_file
    return DataBlockFactory.from_json(infile.read(), check_format)
  File "/Users/graeme/svn/cctbx/modules/cctbx_project/dxtbx/datablock.py", line 910, in from_json
    check_format)
  File "/Users/graeme/svn/cctbx/modules/cctbx_project/dxtbx/datablock.py", line 898, in from_dict
    importer = DataBlockDictImporter(obj, check_format)
  File "/Users/graeme/svn/cctbx/modules/cctbx_project/dxtbx/datablock.py", line 650, in __init__
    self.datablocks = self._load_datablocks(obj, check_format)
  File "/Users/graeme/svn/cctbx/modules/cctbx_project/dxtbx/datablock.py", line 663, in _load_datablocks
    return [self._load_datablocks(dd, check_format) for dd in obj]
  File "/Users/graeme/svn/cctbx/modules/cctbx_project/dxtbx/datablock.py", line 737, in _load_datablocks
    format_kwargs=format_kwargs)
  File "/Users/graeme/svn/cctbx/modules/cctbx_project/dxtbx/imageset.py", line 1149, in make_sweep
    filenames = [template_format % (i+1) for i in range(*array_range)]
TypeError: Please report this error to [email protected]: unsupported operand type(s) for %: 'NoneType' and 'int'

Working now

cctbx.image_viewer: offsets of "Picture" and "Readout" coordinates

As discussed on the [email protected] mailing list, the image viewer coordinates displayed in the status bar do not match with the dxtbx Panel pixel coordinate system. The effect seems to vary with format. FormatCBFMiniPilatus has a single panel with coincident "Picture" and "Readout" coordinates. However, these are shifted by 1 whole pixel in both fast and slow directions.

If dials_regression is present, this can be demonstrated as follows:

dials.import $(libtbx.find_in_repositories dials_regression)/centroid_test_data/centroid_0001.cbf slow_fast_beam_centre=0,0
dials.image_viewer datablock.json

Although this example uses the dials.image_viewer, the behaviour comes from the cctbx.image_viewer upstream. The intention was to place the cross at 0,0 in the Panel's pixel coordinate system. It is displayed at a "Readout" coordinate of 0,0, but this overlays the point 1,1 in the pixel coordinate system.

corner

Other formats have different issues. For example

dials.import 1_000001.img slow_fast_beam_centre=0,0
dials.image_viewer datablock.json

This time the cross is where we expect, but the Picture and Readout coordinates are offset by 0.5 pixels

corner2

It even matters if the image is imported or viewed directly. For example,
dials.image_viewer 1_000001.img opens the same image, but the Picture and Readout coordinates no longer coincide in the top left corner.

corner

Basic expectations discussed on the mailing list

Hovering in the center of the origin pixel shows:

  • Readout: (0.5, 0.5) px (no parallax)
  • Picture: (0.5, 0.5) px (no parallax)
  • ADU: intensity for this pixel
  • Resolution: resolution given panel.get_pixel_lab_coord(readout) with parallax applied (angstroms)

Hovering near the corner of the origin pixel closest to the corner of the panel shows:

  • Readout: (0.1, 0.1) px (no parallax)
  • Picture, ADU and resolution as above given this readout.
  • Same result for single or multi-panel images

The Picture and Readout values can in general differ, as the Picture involves an orthographic projection from the laboratory coordinate system. However, these frames should be aligned such that (0,0) of the Picture coincides with (0,0) of the Readout for Panel 0.

The display should be consistent whether the image viewer is passed an image file or a datablock.json

ADSC JCSG images have incorrectly flagged bad pixels (all pixels with value -2)

This later results in almost all reflections failing in integration due to failures in background modelling.

$ dials.import /dls/metrix/metrix/anomalous_data/1VKN/b5_6_001.img 
DIALS 1.dev.1524-gc2f4a1e
The following parameters have been modified:

input {
  datablock = <image files>
}

--------------------------------------------------------------------------------
  format: <class 'dxtbx.format.FormatSMVADSCSN445.FormatSMVADSCSN445'>
  num images: 1
  num sweeps: 1
  num stills: 0
--------------------------------------------------------------------------------
Writing datablocks to datablock.json

$ dials.export_bitmaps show_mask=True datablock.json 
The following parameters have been modified:

show_mask = True
input {
  datablock = datablock.json
}

Exporting ./image0001.png

image0001

Binary installers built on CentOS fail on Ubuntu

It looks like the update to Python 2.7.13 is causing the binary installers built on CentOS to fail on Ubuntu during installation. This is due to the OpenSSL libraries having different names in the two distributions and some step in the installation is importing something with ssl.

I have a patch where OpenSSL is compiled for Linux as well as macOS and the installer built on CentOS works on Ubuntu again. I'm testing the build on our CentOS and Ubuntu virtual machines before checking it in.

I added the diff for libtbx/auto_build/install_base_packages.py and libtbx/env_config.py below, but the basic changes are,

  1. Build OpenSSL and certifi whenever Python is built
  2. Change check between "configure" and "Configure" to between "configure" and "config". When building OpenSSL, "config" automatically determines the platform.
  3. On Linux, patch "Modules/Setup.dist" to look in the base directory for the OpenSSL installation when building Python (https://bugs.python.org/issue21541). There may be a configuration flag in the future to simplify this. This is not necessary on macOS.
  4. Set SSL_CERT_FILE for root certificates in dispatchers. Previously, this was only done on macOS.

Let me know if there are any issues. Otherwise, testing should be done early next week and the changes will be checked in.

diff --git a/libtbx/auto_build/install_base_packages.py b/libtbx/auto_build/install_base_packages.py
index f7358f786..f86165549 100644
--- a/libtbx/auto_build/install_base_packages.py
+++ b/libtbx/auto_build/install_base_packages.py
@@ -249,14 +249,9 @@ class installer (object) :
     if self.python_exe:
       self.set_python(self.python_exe)
     else:
-      packages += ['python']
-
-    if (self.flag_is_mac and get_os_version() in ("10.11", "10.12")) or options.download_only:
-      # Apple no longer ships openssl headers, therefore need to provide our own
-      # https://forums.developer.apple.com/thread/3897
-      # http://lists.apple.com/archives/macnetworkprog/2015/Jun/msg00025.html
-      # Also, root certificates (certifi) are needed for OpenSSL to work
-      packages += ['openssl', 'certifi']
+      # for better platform independence, build OpenSSL for Linux and macOS
+      # whenever Python is built
+      packages += ['python', 'openssl', 'certifi']
 
     # Always build hdf5 and numpy.
     packages += ['cython', 'hdf5', 'numpy', 'setuptools', 'pip', 'pythonextra', 'docutils']
@@ -510,7 +505,7 @@ Installation of Python packages may fail.
 
   def configure_and_build(self, config_args=(), log=None, make_args=(), limit_nproc=None) :
     # case sensitive file system workaround
-    configure = filter(os.path.exists, ('configure', 'Configure'))
+    configure = filter(os.path.exists, ('configure', 'config'))
     assert configure, 'No configure script found'
     self.call("./%s %s" % (configure[0], " ".join(list(config_args))), log=log)
     self.workarounds()
@@ -700,6 +695,20 @@ Installation of Python packages may fail.
         configure_args.append("--enable-shared")
         configure_args.append("LDFLAGS=-Wl,-rpath=\$$ORIGIN/../lib")
         #configure_args.append("--enable-unicode=ucs4")
+
+      # patch Modules/Setup.dist to find custom OpenSSL
+      targets = ['#SSL=/usr/local/ssl',
+                 '#_ssl _ssl.c \\',
+                 '#\t-DUSE_SSL -I$(SSL)/include -I$(SSL)/include/openssl \\',
+                 '#\t-L$(SSL)/lib -lssl -lcrypto']
+      replacements = ['SSL=%s' % self.base_dir,
+                      '_ssl _ssl.c \\',
+                      '  -DUSE_SSL -I$(SSL)/include -I$(SSL)/include/openssl \\',
+                      '  -L$(SSL)/lib -lssl -lcrypto']
+      for target,replacement in zip(targets,replacements):
+        self.patch_src(src_file='Modules/Setup.dist',
+                       target=target, replace_with=replacement)
+
       self.call([os.path.join(python_dir, 'configure')] + configure_args,
         log=log,
         cwd=python_dir,
@@ -1028,9 +1037,7 @@ _replace_sysconfig_paths(build_time_vars)
     if self.check_download_only(pkg_name): return
     self.untar_and_chdir(pkg=pkg, log=pkg_log)
     self.configure_and_build(
-      config_args=["darwin64-x86_64-cc", self.prefix,
-                   "no-hw", "--openssldir=share",
-                   ],
+      config_args=[self.prefix, "-fPIC", "no-hw", "--openssldir=share"],
       log=pkg_log, limit_nproc=1) # openssl is not parallel buildable
     self.include_dirs.append(op.join(self.base_dir, "include", "openssl"))
 
diff --git a/libtbx/env_config.py b/libtbx/env_config.py
index 351901054..cfe72bcc6 100644
--- a/libtbx/env_config.py
+++ b/libtbx/env_config.py
@@ -949,14 +949,13 @@ Wait for the command to finish, then try again.""" % vars())
   def write_bin_sh_dispatcher(self,
         source_file, target_file, source_is_python_exe=False):
 
-    # set SSL_CERT_FILE if certifi is available (macOS 10.11 and later)
+    # set SSL_CERT_FILE if certifi is available
     cert_file = None
-    if (sys.platform == 'darwin'):
-      try:  # only builds on macOS >= 10.11 will have certifi installed
-        import certifi
-        cert_file = self.as_relocatable_path(certifi.where())
-      except ImportError:
-        pass
+    try:
+      import certifi
+      cert_file = self.as_relocatable_path(certifi.where())
+    except ImportError:
+      pass
 
     # determine LC_ALL from environment (Python UTF-8 compatibility in Linux)
     LC_ALL = os.environ.get('LC_ALL')     # user setting

Move to UCS4

There are a number of reasons to enable UCS4 in all builds.

  1. It's the current standard.
  2. It's the standard in most UNIX OS releases.
  3. I want to use the Mogul Python API which is UCS4, by default. ๐Ÿ˜Ž

Disable user's site directory

Users with a per-user site packages directory can fail install_base_packages in the verification step because their site packages directory will take precedence over the base folder being constructed.

From @graeme-winter:

"https://www.python.org/dev/peps/pep-0370/

The user site directory can be suppressed with a new option -s or the environment variable PYTHONNOUSERSITE . The feature can be disabled globally by setting site.ENABLE_USER_SITE to the value False . It must be set by editing site.py . It can't be altered in sitecustomize.py or later."

From @dagewa, we can do this by adding '-S' to the call at line 465 of install_base_packages.py (inside the verify_python_module function).

-s and -S seem to do different things. Which is correct?

dxtbx's read_uint32 does not

Quite possible I'm just getting confused, but my expectation of the script below is that read_uint32 should read the value 4294967295 here, not -1.

import struct, binascii

# maximum value of unsigned int
uint_max = 4294967295
assert binascii.hexlify(struct.pack('<I', uint_max)) == 'ffffffff'

with open("onebyte.bin","wb") as f:
  f.write(struct.pack('<I', uint_max))

# external check:
# $ hexdump -C onebyte.bin 
# 00000000  ff ff ff ff                                       |....|
# 00000004

from boost.python import streambuf
from dxtbx import read_uint32

with open("onebyte.bin", "rb") as f:
  data = read_uint32(streambuf(f), 1)

print list(data)
# [-1]

Cannot address ExperimentList with a negative number

This used to be possible:

del experiments[-1]

After the changes to move ExperimentList to C++ it no longer works:

>>> from dxtbx.model import Experiment, ExperimentList
>>> el=ExperimentList()
>>> el.append(Experiment())
>>> el.append(Experiment())
>>> el[-1]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: can't convert negative value to unsigned

I think we'd like to keep the Python list-like behaviour, so could we get experiments[-1] working again please?

Add option to store transmission to dxtbx

In some cases where we want to analyse screening images or scale multiple data sets together, having transmission as well as exposure time would be very helpful. For backwards compatibility, default value of 1 should be used; guess this is a property of the scan?

Mac builds: don't embed absolute path in python executable

no idea where/why this happens. The linux python executable does not contain an absolute path of where python was originally built. The mac python executable does.
This messes up relocation. To reproduce use a build moved from the original build location and run

$ base/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python -c "import site; print site.getsitepackages()"

or

$ python -c "import site; print site.getsitepackages()"

On Linux this returns 2 absolute, relocated paths.
On Mac this returns 2 absolute, non-relocated paths.

iotbx.merging_statistics crashing with empty bins

Think small molecule world with small unit cells, few reflections, and high resolution:

>>> import iotbx.merging_statistics as ims
>>> ims.dataset_statistics(i_obs=ims.select_data('/dls/mx-scratch/dials/cctbx-issue-17/AUTOMATIC_DEFAULT_scaled_unmerged.mtz', data_labels=None), n_bins=20)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/scratch/wra62962/files/dials/modules/cctbx_project/iotbx/merging_statistics.py", line 540, in __init__
    cc_one_half_method=cc_one_half_method)
  File "/scratch/wra62962/files/dials/modules/cctbx_project/iotbx/merging_statistics.py", line 239, in __init__
    "(%g - %g)") % (self.d_max, self.d_min))
RuntimeError: No reflections within specified resolution range (0.965792 - 0.940519)

Failure in tst_monitor.py

This test fails with this error. Maybe because of no reduce in the cctbx build.

Please take a look @pafonine

Standard error:
Traceback (most recent call last):
File "/home/builder/slave/cctbx-nightly-intel-linux-2_6-x86_64-centos5/modules/cctbx_project/mmtbx/refinement/real_space/tst_monitor.py", line 224, in
exercise()
File "/home/builder/slave/cctbx-nightly-intel-linux-2_6-x86_64-centos5/modules/cctbx_project/mmtbx/refinement/real_space/tst_monitor.py", line 219, in exercise
sm.show(prefix="start")
File "/home/builder/slave/cctbx-nightly-intel-linux-2_6-x86_64-centos5/modules/cctbx_project/mmtbx/refinement/real_space/init.py", line 467, in show
prefix, format_value("%-5.2f", mso.cis_proline),
AttributeError: 'geometry_no_grm' object has no attribute 'cis_proline'

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.