Hello,

I get the message below when I try and use star.py.

Traceback (most recent call last):
File "/home/user/pyem/pyem/star.py", line 26, in
from pyem.geom import e2r_vec
ImportError: No module named pyem.geom

How can I get the module?

Thank you,
Dominique

Dear Daniel,

I did not have this issue with the previous versions of csparc2star.py.
Am I missing something in my new pyem or Eman installation?

pyem was installed with
pip install git+https://github.com/asarnow/pyem

I am using python that comes with Eman2.2

Thank you,

Peter

./csparc2star.py cryosparc_exp000873_008.csv cryosparc_exp000873_008.star
Traceback (most recent call last):
File "./csparc2star.py", line 171, in
sys.exit(main(parser.parse_args()))
File "./csparc2star.py", line 108, in main
if df.columns.intersection(star.Relion.ANGLES).size == len(star.Relion.ANGLES):
AttributeError: 'module' object has no attribute 'Relion'

Hi there,

I posted this issue on the Scipion github but it turns out to be a small issue with star.py. If the particles.star file has a leading space before the _rlnColumnNames it fails with

File "pandas/_libs/parsers.pyx", line 545, in pandas._libs.parsers.TextReader.cinit
pandas.errors.EmptyDataError: No columns to parse from file
It appears that the string is not stripped before parsing
https://github.com/asarnow/pyem/blob/master/pyem/star.py#L260

Would it be possible to update this so it can handle star files with a space before the column names?

Here is the github thread if interested:
scipion-em/scipion-em-relion#91

Thank you

Andrea

Hello,

I started messing around with your star parser. It is very helpful, it took me almost no time to start making custom plots of what I was interested in from my star files. Thank you!

The star parser would be even more helpful if it was its own library that one could install with pip or conda. Would you consider publishing it on either package registry?

Alternatively, making all of pyem a pip or conda package would work too. This would also have the side effect of making installation of the entire pyem somewhat easier, as people wouldn't need to clone the git repo anymore. I don't know how much the star parser depends on the other modules in pyem, maybe it would be easier to package the whole thing, maybe not.

Hello,
I am getting the message below when I run pyem/pyem/star.py --copy-micrograph-coordinates particles.star cryosparc.star cryosparc_withcoords.star. Should I have done anything to the script? All I did was git clone it to my directory.

Traceback (most recent call last):
File "pyem/pyem/star.py", line 30, in
from pyem.algo import query_connected
ImportError: No module named pyem.algo

Best,
Dominique

Hi, Asarnow,

Thanks for this wonderful program. I tried ii on three datasets processed with cryosparc, after converting the .csv files to .star and computing maps with relion_reconstruct, one dataset worked fine but maps of other two datsets were incorrect. Furthermore, for the working dataset, I revised the .star file to point to the same datset but with phase_flipped, then computed a map with relion_reconstruct, but the map is incorrect. Any suggestions would be greatly appreciated. Thanks a lot.

Best regards,
Xing

The angle convention is wrong so reconstructions after applying a transformation return garbage.

Hi Daniel,
I have used pyem/0.5 csparc2star.py to converted a particle .cs file to a relion-3.1 star file. it works, however, the relion cannot use the star file to run classification. I have checked converted "_rlnImageName", and found the actual particle stack file is "xxxx.mrcs"; but star file records "xxxx.mrc". After I replaced all the .mrc to .mrcs in the star file, it works in relion-3.1. I guess that is a small bug.

Hi,

I have recently been attempting to run recenter.py (looks very handy with all the centring modes implemented now!). However, I have not been able to succesfully do so, and I am wondering if you could possibly provide an example of intended usage? I gave it a shot at centring based on 2D classes in the following way:

recenter.py --class-2d Class2D/mrcs_stack_with_2Dclasses.mrcs Class2D/data_star_file_with_rlnClassNumber.star test_recenter.star

Then got an error along the lines of

File "/csbdatalocal/ra3014/pyem/recenter.py", line 47, in main
com = Vec2f(*im.phase_cog()[:2])
NameError: global name 'Vec2f' is not defined

Which I got around by adding Vec2f to the imports from EMAN2. But now Im getting the following

Traceback (most recent call last):
File "/csbdatalocal/ra3014/pyem/recenter.py", line 104, in
sys.exit(main(args))
File "/csbdatalocal/ra3014/pyem/recenter.py", line 57, in main
for ptcl in star.rows:
File "/csb/home/ra3014/anaconda2/lib/python2.7/site-packages/pandas/core/generic.py", line 2744, in getattr
return object.getattribute(self, name)
AttributeError: 'DataFrame' object has no attribute 'rows'

Any help would be appreciated! Thanks a lot in advance!

I found a small issue on csparc2star.py，that now relion data_optics contains _rlnImageSize information。Could you please add this info into the next version of csparc2star.py？

Hello
I'm trying to export particules coordinates form cryospark 2.5 to Relion3
I install pyem using miniconda (I first download and instal it)
, create symlink in ~/bin
but when I try csparc2star I got the following errors :

csparc2star.py particles_selected.cs test.star
Traceback (most recent call last):
File "/home/emmanuel/bin/csparc2star.py", line 28, in
from pyem import metadata
File "/home/emmanuel/Software/pyem/pyem/init.py", line 3, in
from . import metadata
File "/home/emmanuel/Software/pyem/pyem/metadata.py", line 25, in
from . import star
File "/home/emmanuel/Software/pyem/pyem/star.py", line 31, in
from pyem.util import rot2euler
File "/home/emmanuel/Software/pyem/pyem/util/init.py", line 3, in
from .util import *
File "/home/emmanuel/Software/pyem/pyem/util/util.py", line 25, in
from .. import geom
File "/home/emmanuel/Software/pyem/pyem/geom/init.py", line 4, in
from .geom import *
File "/home/emmanuel/Software/pyem/pyem/geom/geom.py", line 22, in
from .quat_numba import qslerp
File "/home/emmanuel/Software/pyem/pyem/geom/quat_numba.py", line 153, in
"(m)->(m), (m), (m), (m)", nopython=True, cache=False)
File "/home/emmanuel/miniconda2/lib/python2.7/site-packages/numba/npyufunc/decorators.py", line 179, in wrap
guvec.add(fty)
File "/home/emmanuel/miniconda2/lib/python2.7/site-packages/numba/npyufunc/ufuncbuilder.py", line 194, in add
self.nb_func, targetoptions, sig)
File "/home/emmanuel/miniconda2/lib/python2.7/site-packages/numba/npyufunc/ufuncbuilder.py", line 128, in _compile_element_wise_function
cres = nb_func.compile(sig, **targetoptions)
File "/home/emmanuel/miniconda2/lib/python2.7/site-packages/numba/npyufunc/ufuncbuilder.py", line 77, in compile
return self._compile_core(sig, flags, locals)
File "/home/emmanuel/miniconda2/lib/python2.7/site-packages/numba/npyufunc/ufuncbuilder.py", line 112, in _compile_core
flags=flags, locals=locals)
File "/home/emmanuel/miniconda2/lib/python2.7/site-packages/numba/compiler.py", line 926, in compile_extra
return pipeline.compile_extra(func)
File "/home/emmanuel/miniconda2/lib/python2.7/site-packages/numba/compiler.py", line 374, in compile_extra
return self._compile_bytecode()
File "/home/emmanuel/miniconda2/lib/python2.7/site-packages/numba/compiler.py", line 857, in _compile_bytecode
return self._compile_core()
File "/home/emmanuel/miniconda2/lib/python2.7/site-packages/numba/compiler.py", line 844, in _compile_core
res = pm.run(self.status)
File "/home/emmanuel/miniconda2/lib/python2.7/site-packages/numba/compiler_lock.py", line 32, in _acquire_compile_lock
return func(*args, **kwargs)
File "/home/emmanuel/miniconda2/lib/python2.7/site-packages/numba/compiler.py", line 255, in run
raise patched_exception
TypeError: Failed in nopython mode pipeline (step: analyzing bytecode)

I also try the general installation with the same result.

Daniel, i'm sorry to bother you. I'm having trouble (again/still) with csparc2star.py. I followed your installation instructions (detailed below), but I'm getting an error when converting and exported set of particles. The particles were originally from relion.

Help will be appreciated. Michael

csparc2star.py P4_J176_particles_exported.cs P4_J176_particles_exported.star --loglevel debug
Detected CryoSPARC 2+ .cs file
Reading primary file
Collecting particle parameters from most likely classes
Columns must be same length as key
Traceback (most recent call last):
File "/srv/local/progs/pyem-release/csparc2star.py", line 42, in main
df = metadata.parse_cryosparc_2_cs(cs, passthroughs=args.input[1:], minphic=args.minphic, boxsize=args.boxsize, swapxy=args.swapxy)
File "/srv/local/progs/pyem-release/pyem/metadata.py", line 390, in parse_cryosparc_2_cs
df = cryosparc_2_cs_model_parameters(cs, df, minphic=minphic)
File "/srv/local/progs/pyem-release/pyem/metadata.py", line 333, in cryosparc_2_cs_model_parameters
[cs[names[c]][i] for i, c in enumerate(cls)]))
File "/srv/local/progs/miniconda3/envs/pyem/lib/python3.7/site-packages/pandas/core/frame.py", line 2935, in setitem
self._setitem_array(key, value)
File "/srv/local/progs/miniconda3/envs/pyem/lib/python3.7/site-packages/pandas/core/frame.py", line 2961, in _setitem_array
raise ValueError("Columns must be same length as key")
ValueError: Columns must be same length as key
Required fields could not be mapped. Are you using the right input file(s)?

Installation on Ubuntu 18.04

download pyem-release branch

https://github.com/asarnow/pyem

unzip in /srv/local/progs

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh

bash Miniconda3-latest-Linux-x86_64.sh

You currently have a PYTHONPATH environment variable set. This may cause
unexpected behavior when running the Python interpreter in Miniconda3.
For best results, please verify that your PYTHONPATH only points to
directories of packages that are compatible with the Python interpreter
in Miniconda3: /srv/local/progs/miniconda3
Do you wish the installer to initialize Miniconda3
by running conda init? [yes|no]
yes

log out > log in

unset PYTHONPATH

. /srv/local/progs/miniconda3/etc/profile.d/conda.sh

conda create -n pyem

conda activate pyem

conda install numpy scipy matplotlib seaborn numba pandas natsort

conda install -c conda-forge pyfftw healpy pathos

unzip pyem-release

cd pyem-release

pip install --no-dependencies -e .
Obtaining file:///srv/local/progs/pyem-release
Installing collected packages: pyem
Running setup.py develop for pyem
Successfully installed pyem

nano /etc/bash.bashrc

pyem-release

PATH=:$PATH:/srv/local/progs/pyem-release

source /etc/bash.bashrc

csparc2star.py --help
usage: csparc2star.py [-h] [--boxsize BOXSIZE] [--class CLS]
[--minphic MINPHIC] [--stack-path STACK_PATH]
[--micrograph-path MICROGRAPH_PATH]
[--copy-micrograph-coordinates COPY_MICROGRAPH_COORDINATES]
[--swapxy] [--cached] [--transform TRANSFORM]
[--loglevel LOGLEVEL]
[input [input ...]] output

positional arguments:
input Cryosparc metadata .csv (v0.6.5) or .cs (v2+) files
output Output .star file

optional arguments:
-h, --help show this help message and exit
--boxsize BOXSIZE Cryosparc refinement box size (if different from
particles)
--class CLS Keep this class in output, may be passed multiple
times
--minphic MINPHIC Minimum posterior probability for class assignment
--stack-path STACK_PATH
Path to single particle stack
--micrograph-path MICROGRAPH_PATH
Replacement path for micrographs
--copy-micrograph-coordinates COPY_MICROGRAPH_COORDINATES
Source for micrograph paths and particle coordinates
(file or quoted glob)
--swapxy Swap X and Y axes when converting particle coordinates
--cached Keep paths from the Cryosparc 2+ cache when merging
coordinates
--transform TRANSFORM
Apply rotation matrix or 3x4 rotation plus translation
matrix to particles (Numpy format)
--loglevel LOGLEVEL, -l LOGLEVEL
Logging level and debug output

Dear Dan,
If I want to export the coordinates of particles when I convert from cs to star, which particles cs files should I use?
"--copy-micrograph-coordinates", "Source for micrograph paths and particle coordinates (file or quoted glob)",
What do you mean here "quoted glob"? Thanks!
Best regards,
Wei

I'm just curious whether
git pull
is enough to use the latest pyem version.

Otherwise,
do I need to run something else as well after
git pull
in my local pyem git repo?

Thank you

@asarnow Hi Daniel,

I am trying to convert cs file from cryosparc2 into star files and encountered several question:

1. The first line in csparc2star.py is: #!/usr/bin/env python2.7
   Does that mean that I have to use python 2.7 as assigned version? I have updated my mac with python3.7. I am not sure if python3.7 caused the problem.

2. A very frequent error (I tried to install on different computers and this happened a lot) that happened when I want to use csparc2.star is: csparc2star.py: command not found
   Do you know why this happened and how to solve this problem?

3. Sometimes, it seems that csparc2star.py was about to run and I have encountered many problems regarding numpy. One of the problems is ImportError: No module named numpy. Another error is ImportError: numpy.core.multiarray failed to import. PS: I have updated numpy to 1.15.4
   I also followed the suggestion from this post: tensorflow/tensorflow#559
   I deleted and re-updated all numpy that I could find on my laptop. The problem was still not solved.

Thank you very much in advance for your time and help.

Best,
Zhen

I get the following error when running csparc2star.py:

File "csparc2star.py", line 103, in
sys.exit(main(parser.parse_args()))
File "csparc2star.py", line 42, in main
df = metadata.parse_cryosparc_2_cs(cs, passthrough=args.passthrough, minphic=args.minphic)
File "/usr/lib/python3.5/site-packages/pyem/metadata.py", line 313, in parse_cryosparc_2_cs
star.simplify_star_ucsf(df)
File "/usr/lib/python3.5/site-packages/pyem/star.py", line 364, in simplify_star_ucsf
lambda x: "%.6d" % (x + 1)).str.cat(df[UCSF.IMAGE_PATH], sep="@")
File "/usr/lib64/python3.5/site-packages/pandas/core/strings.py", line 2320, in cat
res = str_cat(data, others=others, sep=sep, na_rep=na_rep)
File "/usr/lib64/python3.5/site-packages/pandas/core/strings.py", line 110, in str_cat
cats = [sep.join(tup) for tup in tuples]
File "/usr/lib64/python3.5/site-packages/pandas/core/strings.py", line 110, in
cats = [sep.join(tup) for tup in tuples]
TypeError: sequence item 1: expected str instance, bytes found

Hi,

Just installed pyem with EMan2.2 but csparc2star.py gives an error:

Traceback (most recent call last):
File "./csparc2star.py", line 25, in
import pandas as pd
File "/progs/EMAN2/lib/python2.7/site-packages/pandas/init.py", line 42, in
from pandas.core.api import *
File "/progs/EMAN2/lib/python2.7/site-packages/pandas/core/api.py", line 10, in
from pandas.core.groupby import Grouper
File "/progs/EMAN2/lib/python2.7/site-packages/pandas/core/groupby.py", line 49, in
from pandas.core.frame import DataFrame
File "/progs/EMAN2/lib/python2.7/site-packages/pandas/core/frame.py", line 74, in
from pandas.core.series import Series
File "/progs/EMAN2/lib/python2.7/site-packages/pandas/core/series.py", line 3042, in
import pandas.plotting._core as _gfx # noqa
File "/progs/EMAN2/lib/python2.7/site-packages/pandas/plotting/init.py", line 17, in
from pandas.plotting._core import boxplot
File "/progs/EMAN2/lib/python2.7/site-packages/pandas/plotting/_core.py", line 40, in
if _mpl_ge_1_5_0():
File "/progs/EMAN2/lib/python2.7/site-packages/pandas/plotting/_compat.py", line 40, in _mpl_ge_1_5_0
return (matplotlib.version >= LooseVersion('1.5') or
File "/progs/EMAN2/lib/python2.7/distutils/version.py", line 296, in cmp
return cmp(self.version, other.version)
AttributeError: 'unicode' object has no attribute 'version'

Hi Daniel,

I was using csparc2star.py to convert cs file from cryosparc v2 to relion. I got the following error during the conversion:

Traceback (most recent call last):
File "/usr/local/packages/pyem_v2/pyem/csparc2star.py", line 103, in
sys.exit(main(parser.parse_args()))
File "/usr/local/packages/pyem_v2/pyem/csparc2star.py", line 42, in main
df = metadata.parse_cryosparc_2_cs(cs, passthrough=args.passthrough, minphic=args.minphic)
File "/Data_HDD2/Programs/packages/pyem_v2/pyem/pyem/metadata.py", line 384, in parse_cryosparc_2_cs
axis=1, raw=True, result_type='broadcast'))
File "/home/whuang/miniconda2/lib/python2.7/site-packages/pandas/core/frame.py", line 4255, in apply
return self._apply_raw(f, axis)
File "/home/whuang/miniconda2/lib/python2.7/site-packages/pandas/core/frame.py", line 4284, in _apply_raw
result = np.apply_along_axis(func, axis, self.values)
File "/home/whuang/miniconda2/lib/python2.7/site-packages/numpy/lib/shape_base.py", line 116, in apply_along_axis
res = asanyarray(func1d(inarr_view[ind0], *args, **kwargs))
File "/home/whuang/miniconda2/lib/python2.7/site-packages/pandas/core/frame.py", line 4239, in f
return func(x, *args, **kwds)
TypeError: () got an unexpected keyword argument 'result_type'

Could you help me figure out what causes the problem? Please let me know if additional information is needed. Thanks!

Best,
Wei

I notice that pyem have an update and gonna to install the latest version. However, it said that Python version >= 3.5 required. but in the wiki, not only the pip command but also Miniconda tell us we need use python2. When I use python3's pip to install the software, it also cannot open. that make me so confuse. Which python should I indeed use for pyem?

pyem-csparc2star XXXX_passthrough_particles.cs Stars/XXXXp.star
raise ValueError("Columns must be same length as key") ValueError: Columns must be same length as key Required fields could not be mapped. Are you using the right input file(s)?

Hi, @asarnow thanks for having this nice repo.
We are actually using it and we were converting cs output to relion star file using this (actually what is in pipy).

Later we did a reconstruction with relion and xmipp and both didn't seem right. Our refinement in CS has a D7 symmetry whereas both reconstructions (relion and xmipp) showed a D14 symmetry (in both cases we specified D7 symmetry fro the reconstruction.

Does it ring any bell?

Alternatively....do CS offer an equivalent to relion_reconstruct, I mean asking CS to make a reconstruction using angle in a star file?

Dear Daniel,

I am having some issues importing results from 2D back into relion for further processing.
I used to use this command in the past which I suppose is now obsolete:
/data/CRYOSPARC/pyem/csparc2star.py --copy-micrograph-coordinates /data/XXX/Extract/job41/particles.star /data/CRYOSPARC/USERS/Omid/P1/J154/particles_selected.cs from_cryosparc.star

I now get the following error:
Traceback (most recent call last):
File "/data/CRYOSPARC/pyem/csparc2star.py", line 28, in
from pyem import metadata
File "/data/CRYOSPARC/pyem/pyem/init.py", line 3, in
from . import metadata
File "/data/CRYOSPARC/pyem/pyem/metadata.py", line 401
ptdf = util.dataframe_from_records_mapped(pt, {**general, **micrograph})

I have also updated conda and pyem. The particles were originally extracted in Relion. I removed the top part (data_optics section) as suggested in the forums in order to import it into cryosparc. Could that be part of the problem? I assume that pyem will re-import this info back from cryosparc.

I saw your recent post (https://discuss.cryosparc.com/t/csparc2star-py-rlnmicrographname-error/3750/3) as well on this issue but it was not clear to me what the command should be to get around this problem or if this is the same problem at all.

Best,
Omid

I exported a .cs file from a cryosparc v2 homo-refinment and converted to .star with csparc2star.py, but the output rlnImageName field isn't a valid path. The reason seems to be that the paths in the .cs file have a leading >, e.g. >J01/extract/particles.mrc, and it is not removed in the conversion to .star, yielding image names like 000001@>J01/extract/particles.mrc. Removing the > with sed fixes this, but I figured I'd raise the issue anyway, just in case this is unexpected behaviour. (I don't know if a leading > is typical of the current cryosparc v2 version I'm using, or just an idiosyncrasy of my installation.)

Hi and thanks for the script. I (attempted) to follow the installation instructions, but I'm get the "no module name pandas" error. Belwo is what I did. I will appreciate your help finding my error/problem. Thanks, Mike

$ which python
/usr/bin/python
$ python --version
Python 2.7.6
$which pip
/usr/bin/pip

unzipped pyem-master.zip in /srv/local/progs
$cd pyem-master
$ sudo pip install -r requirements.txt
Requirement already satisfied (use --upgrade to upgrade): future in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): numpy in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 2))
Requirement already satisfied (use --upgrade to upgrade): scipy in /usr/lib/python2.7/dist-packages (from -r requirements.txt (line 3))
Requirement already satisfied (use --upgrade to upgrade): matplotlib in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 4))
Requirement already satisfied (use --upgrade to upgrade): seaborn in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 5))
Requirement already satisfied (use --upgrade to upgrade): pandas in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 6))
Requirement already satisfied (use --upgrade to upgrade): pathos in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 7))
Requirement already satisfied (use --upgrade to upgrade): six>=1.10 in /usr/local/lib/python2.7/dist-packages (from matplotlib->-r requirements.txt (line 4))
Requirement already satisfied (use --upgrade to upgrade): python-dateutil>=2.0 in /usr/local/lib/python2.7/dist-packages (from matplotlib->-r requirements.txt (line 4))
Requirement already satisfied (use --upgrade to upgrade): backports.functools-lru-cache in /usr/local/lib/python2.7/dist-packages (from matplotlib->-r requirements.txt (line 4))
Requirement already satisfied (use --upgrade to upgrade): subprocess32 in /usr/local/lib/python2.7/dist-packages (from matplotlib->-r requirements.txt (line 4))
Requirement already satisfied (use --upgrade to upgrade): pytz in /usr/local/lib/python2.7/dist-packages (from matplotlib->-r requirements.txt (line 4))
Requirement already satisfied (use --upgrade to upgrade): cycler>=0.10 in /usr/local/lib/python2.7/dist-packages (from matplotlib->-r requirements.txt (line 4))
Requirement already satisfied (use --upgrade to upgrade): pyparsing>=2.0.1,!=2.0.4,!=2.1.2,!=2.1.6 in /usr/local/lib/python2.7/dist-packages (from matplotlib->-r requirements.txt (line 4))
Requirement already satisfied (use --upgrade to upgrade): ppft>=1.6.4.7 in /usr/local/lib/python2.7/dist-packages (from pathos->-r requirements.txt (line 7))
Requirement already satisfied (use --upgrade to upgrade): dill>=0.2.7 in /usr/local/lib/python2.7/dist-packages (from pathos->-r requirements.txt (line 7))
Requirement already satisfied (use --upgrade to upgrade): pox>=0.2.3 in /usr/local/lib/python2.7/dist-packages (from pathos->-r requirements.txt (line 7))
Requirement already satisfied (use --upgrade to upgrade): multiprocess>=0.70.5 in /usr/local/lib/python2.7/dist-packages (from pathos->-r requirements.txt (line 7))
Cleaning up...

$ sudo python setup.py install
/usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'install_requires'
warnings.warn(msg)
running install
running build
running build_py
running install_lib
running install_egg_info
Removing /usr/local/lib/python2.7/dist-packages/pyem-0.1.egg-info
Writing /usr/local/lib/python2.7/dist-packages/pyem-0.1.egg-info

$csparc2star.py cryosparc_exp000107_iter20_selected_from_000108.csv csparc-2D.star
Traceback (most recent call last):
File "/srv/local/progs/pyem-master/csparc2star.py", line 25, in
import pandas as pd
ImportError: No module named pandas

After performing "Select 2D" in cryosparc, I export the output and there are two files P9_J165_particles_selected_exported.cs and P9_J165_particles_selected_exported.csg.
I ran below command line to convert cryosparc data into relion format. There is a issue.

$ csparc2star.py P9_J165_particles_selected_exported.cs P9_J165_particles_selected_exported.star
/home/z/sw/pyem/pyem/metadata.py:330: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=obj
ect' when creating the ndarray
df[model[k]] = pd.DataFrame(np.array(
Columns must be same length as key
Traceback (most recent call last):
File "/home/z/sw/pyem/csparc2star.py", line 42, in main
df = metadata.parse_cryosparc_2_cs(cs, passthroughs=args.input[1:], minphic=args.minphic, boxsize=args.boxsize, swapxy=args.swapxy)
File "/home/z/sw/pyem/pyem/metadata.py", line 398, in parse_cryosparc_2_cs
df = cryosparc_2_cs_model_parameters(cs, df, minphic=minphic)
File "/home/z/sw/pyem/pyem/metadata.py", line 330, in cryosparc_2_cs_model_parameters
df[model[k]] = pd.DataFrame(np.array(
File "/home/z/.conda/envs/pyem/lib/python3.8/site-packages/pandas/core/frame.py", line 3037, in setitem
self._setitem_array(key, value)
File "/home/z/.conda/envs/pyem/lib/python3.8/site-packages/pandas/core/frame.py", line 3063, in _setitem_array
raise ValueError("Columns must be same length as key")
ValueError: Columns must be same length as key
Required fields could not be mapped. Are you using the right input file(s)?

How can I figure it out? Thank you!

Hi Dan,
I am trying to use csparc2star.py on the particles.cs file output from an old NU-refine job. The NU-refine job was run using particles imported from relion 3.0.x; the cryosparc version at the time was 2.5.2. When I run the job, I get an error complaining that it can't find the OpticsGroup column. Any ideas as to what I am doing wrong?

Cheers
Oli

(pyem) user@narwhal:~$ csparc2star.py cryosparc_P16_J504_007_particles.cs cryosparc_P16_J504_007_particles.star Traceback (most recent call last): File "/home/user/software/pyem/csparc2star.py", line 110, in <module> sys.exit(main(parser.parse_args())) File "/home/user/software/pyem/csparc2star.py", line 83, in main star.write_star(args.output, df, resort_records=True) File "/home/user/software/pyem/pyem/star.py", line 419, in write_star gb = df.groupby(Relion.OPTICSGROUP) File "/home/user/software/miniconda3/envs/pyem/lib/python3.8/site-packages/pandas/core/frame.py", line 5801, in groupby return groupby_generic.DataFrameGroupBy( File "/home/user/software/miniconda3/envs/pyem/lib/python3.8/site-packages/pandas/core/groupby/groupby.py", line 402, in __init__ grouper, exclusions, obj = get_grouper( File "/home/user/software/miniconda3/envs/pyem/lib/python3.8/site-packages/pandas/core/groupby/grouper.py", line 598, in get_grouper raise KeyError(gpr) KeyError: 'rlnOpticsGroup'

Trailing blank lines in a cryosparc metadata break csparc2csv.py.

Fixed by adding skip_blank_lines=True to pandas.read_csv() call. Supposed to be the default, but apparently not in some versions of Pandas.

Hi,

I recently found this repository and the functionality provided by recenter.py looked very interesting. I was wondering if there is an updated finished version?

Best wishes,

Rafa

There is a new bug while I export particles to cryosparc then export it back to relion using relion-3.1.
When I export star file from relion, the OpticsGroup number is "1,2,3" and when I using csparc2star go back, it turns to "2,3,4" so that relion cannot reextract those particles. Now I just use vim to replace OpticsNumbers to the right one. Could we have a parameter fix it? Thanks!

Hi,

I ran into an issue recently that I was able to solve. I converted a selected 2D class from cryosparc to a star file using the c2sparc2star.py command. I then ran multiple 2D classifications and 3D classifications without any issues. Though during 3D auto refine, I got a segment error. All of the _rlnRandomSubset column was set to 1 instead of a range of 1 and 2 values. By deleting the column, 3D auto refine worked fine. Is this something that occurs from a pyem conversion. I am using Relion 3.1.0-beta and the newest version of cryosparc. Here is a copy of the particle star file before I deleted the column:

version 30001

data_optics

loop_
_rlnVoltage #1
_rlnSphericalAberration #2
_rlnAmplitudeContrast #3
_rlnOpticsGroup #4
_rlnImagePixelSize #5
_rlnImageDimensionality #6
_rlnImageSize #7
300.000000 2.700000 0.070000 1 1.060000 2 200

version 30001

data_particles

loop_
_rlnImageName #1
_rlnMicrographName #2
_rlnCoordinateX #3
_rlnCoordinateY #4
_rlnAnglePsi #5
_rlnOriginXAngst #6
_rlnOriginYAngst #7
_rlnDefocusU #8
_rlnDefocusV #9
_rlnDefocusAngle #10
_rlnPhaseShift #11
_rlnCtfBfactor #12
_rlnRandomSubset #13
_rlnClassNumber #14
_rlnOpticsGroup #15
_rlnGroupNumber #16
_rlnAngleRot #17
_rlnAngleTilt #18
_rlnNormCorrection #19
_rlnLogLikeliContribution #20
_rlnMaxValueProbDistribution #21
_rlnNrOfSignificantSamples #22
000014@J774/extract/FoilHole_1159715_Data_31204767_31204768_20190201_1934_aligned_DW_particles.mrcs J764/imported/FoilHole_1159715_Data_31204767_31204768_20190201_1934_aligned_DW.mrc 2334.000000 2788.000000 94.120187 5.421603 1.181603 29223.787109 28601.591797 25.456038 0.000000 0.000000 1 4 1 1 9.204747 89.815355 0.700938 17112.210941 0.274805 105
000074@J774/extract/FoilHole_1159715_Data_31204767_31204768_20190201_1934_aligned_DW_particles.mrcs J764/imported/FoilHole_1159715_Data_31204767_31204768_20190201_1934_aligned_DW.mrc 3416.000000 2316.000000 145.879813 3.301603 4.361603 29223.787109 28601.591797 25.456038 0.000000 0.000000 1 4 1 1 -170.79525 90.184645 0.663777 16773.422483 0.340087 80
000033@J774/extract/FoilHole_1159715_Data_31204767_31204768_20190201_1934_aligned_DW_particles.mrcs J764/imported/FoilHole_1159715_Data_31204767_31204768_20190201_1934_aligned_DW.mrc 665.000000 2217.000000 -6.38271 -8.35840 1.181603 29223.787109 28601.591797 25.456038 0.000000 0.000000 1 4 1 1 -30.24124 93.786117 0.673539 16900.963475 0.197940 39
000161@J774/extract/FoilHole_1159715_Data_31204773_31204774_20190201_1935_aligned_DW_particles.mrcs J764/imported/FoilHole_1159715_Data_31204773_31204774_20190201_1935_aligned_DW.mrc 1090.000000 2175.000000 22.117371 -0.93840 -4.11840 29848.451172 29340.681641 14.153535 0.000000 0.000000 1 4 1 2 -58.69878 96.867049 0.685048 16806.612919 0.400166 39
000009@J774/extract/FoilHole_1159715_Data_31204773_31204774_20190201_1935_aligned_DW_particles.mrcs J764/imported/FoilHole_1159715_Data_31204773_31204774_20190201_1935_aligned_DW.mrc 433.000000 620.000000 -146.16819 9.661603 1.181603 29848.451172 29340.681641 14.153535 0.000000 0.000000 1 4 1 2 -176.40750 85.390328 0.722245 16913.232501 0.242369 49
000046@J774/extract/FoilHole_1159715_Data_31204773_31204774_20190201_1935_aligned_DW_particles.mrcs J764/imported/FoilHole_1159715_Data_31204773_31204774_20190201_1935_aligned_DW.mrc 1414.000000 248.000000 -12.50878 0.121603 7.541603 29848.451172 29340.681641 14.153535 0.000000 0.000000 1 4 1 2 37.330451 27.313210 0.647260 16311.495712 0.620180 16
000040@J774/extract/FoilHole_1159715_Data_31204773_31204774_20190201_1935_aligned_DW_particles.mrcs J764/imported/FoilHole_1159715_Data_31204773_31204774_20190201_1935_aligned_DW.mrc 3362.000000 2151.000000 141.712664 1.181603 8.601602 29848.451172 29340.681641 14.153535 0.000000 0.000000 1 4 1 2 -75.18663 90.818378 0.669095 16823.168263 0.197942 28
000148@J774/extract/FoilHole_1159715_Data_31204773_31204774_20190201_1935_aligned_DW_particles.mrcs J764/imported/FoilHole_1159715_Data_31204773_31204774_20190201_1935_aligned_DW.mrc 2697.000000 1555.000000 -110.20394 9.661603 11.781603 29848.451172 29340.681641 14.153535 0.000000 0.000000 1 4 1 2 -163.22947 37.342240 0.693711 16715.405393 0.695409 16
000112@J774/extract/FoilHole_1159715_Data_31204807_31204808_20190201_1935_aligned_DW_particles.mrcs J764/imported/FoilHole_1159715_Data_31204807_31204808_20190201_1935_aligned_DW.mrc 3385.000000 2779.000000 94.645874 6.481603 2.241603 26461.250000 26245.857422 0.341052 0.000000 0.000000 1 4 1 3 15.177866 114.005953 0.670025 16754.361465 0.753241 40
000046@J774/extract/FoilHole_1159715_Data_31204807_31204808_20190201_1935_aligned_DW_particles.mrcs J764/imported/FoilHole_1159715_Data_31204807_31204808_20190201_1935_aligned_DW.mrc 672.000000 165.000000 -128.52164 14.961603 -6.23840 26461.250000 26245.857422 0.341052 0.000000 0.000000 1 4 1 3 150.669899 105.664212 0.696432 16769.565261 0.827122 20
000030@J774/extract/FoilHole_1159715_Data_31204807_31204808_20190201_1935_aligned_DW_particles.mrcs J764/imported/FoilHole_1159715_Data_31204807_31204808_20190201_1935_aligned_DW.mrc 3331.000000 3019.000000 -75.03671 -7.29840 -1.99840 26461.250000 26245.857422 0.341052 0.000000 0.000000 1 4 1 3 140.361580 116.746192 0.670294 16866.984542 0.556135 35
000020@J774/extract/FoilHole_1159715_Data_31204807_31204808_20190201_1935_aligned_DW_particles.mrcs J764/imported/FoilHole_1159715_Data_31204807_31204808_20190201_1935_aligned_DW.mrc 796.000000 529.000000 82.197054 0.121603 11.781603 26461.250000 26245.857422 0.341052 0.000000 0.000000 1 4 1 3 -51.38736 73.065729 0.671986 16825.010930 0.710078 28
000009@J774/extract/FoilHole_1159715_Data_31204807_31204808_20190201_1935_aligned_DW_particles.mrcs J764/imported/FoilHole_1159715_Data_31204807_31204808_20190201_1935_aligned_DW.mrc 1376.000000 2771.000000 -0.30295 -6.23840 -4.11840 26461.250000 26245.857422 0.341052 0.000000 0.000000 1 4 1 3 -51.38736 73.065729 0.623831 16442.850967 0.858759 24
000138@J774/extract/FoilHole_1159715_Data_31204807_31204808_20190201_1935_aligned_DW_particles.mrcs J764/imported/FoilHole_1159715_Data_31204807_31204808_20190201_1935_aligned_DW.mrc 2922.000000 3623.000000 -106.14044 -3.05840 6.481603 26461.250000 26245.857422 0.341052 0.000000 0.000000 1 4 1 3 -84.77253 70.512775 0.659289 16795.785255 0.227446 65
000011@J774/extract/FoilHole_1159715_Data_31204818_31204819_20190201_1932_aligned_DW_particles.mrcs J764/imported/FoilHole_1159715_Data_31204818_31204819_20190201_1932_aligned_DW.mrc 1476.000000 3143.000000 -71.33932 -9.41840 -5.17840 31429.634766 31092.513672 -33.23296 0.000000 0.000000 1 4 1 4 3.562630 114.080872 0.660083 16631.900281 0.761788 5
000265@J774/extract/FoilHole_1159715_Data_31204818_31204819_20190201_1932_aligned_DW_particles.mrcs J764/imported/FoilHole_1159715_Data_31204818_31204819_20190201_1932_aligned_DW.mrc 997.000000 3350.000000 50.622266 -1.99840 -8.35840 31429.634766 31092.513672 -33.23296 0.000000 0.000000 1 4 1 4 25.220518 64.303222 0.683167 16812.392686 0.284285 17
000037@J774/extract/FoilHole_1159715_Data_31204818_31204819_20190201_1932_aligned_DW_particles.mrcs J764/imported/FoilHole_1159715_Data_31204818_31204819_20190201_1932_aligned_DW.mrc 1345.000000 761.000000 164.964778 0.121603 4.361603 31429.634766 31092.513672 -33.23296 0.000000 0.000000 1 4 1 4 -48.06638 107.306982 0.687698 16786.084987 0.299265 8
000008@J774/extract/FoilHole_1159715_Data_31204818_31204819_20190201_1932_aligned_DW_particles.mrcs J764/imported/FoilHole_1159715_Data_31204818_31204819_20190201_1932_aligned_DW.mrc 672.000000 620.000000 -73.89048 2.241603 2.241603 31429.634766 31092.513672 -33.23296 0.000000 0.000000 1 4 1 4 -29.80035 84.202158 0.664798 16753.889008 0.557087 9
000175@J774/extract/FoilHole_1159715_Data_31204818_31204819_20190201_1932_aligned_DW_particles.mrcs J764/imported/FoilHole_1159715_Data_31204818_31204819_20190201_1932_aligned_DW.mrc 2288.000000 877.000000 73.978363 2.241603 6.481602 31429.634766 31092.513672 -33.23296 0.000000 0.000000 1 4 1 4 150.669899 105.664212 0.691363 16796.244649 0.636259 60
000085@J774/extract/FoilHole_1159715_Data_31204818_31204819_20190201_1932_aligned_DW_particles.mrcs J764/imported/FoilHole_1159715_Data_31204818_31204819_20190201_1932_aligned_DW.mrc 1237.000000 761.000000 -37.12173 -7.29840 -13.65840 31429.634766 31092.513672 -33.23296 0.000000 0.000000 1 4 1 4 20.333782 22.008140 0.676979 16737.898359 0.440337 9
000087@J774/extract/FoilHole_1159715_Data_31204818_31204819_20190201_1932_aligned_DW_particles.mrcs J764/imported/FoilHole_1159715_Data_31204818_31204819_20190201_1932_aligned_DW.mrc 1886.000000 2705.000000 -151.90412 1.181603 -4.11840 31429.634766 31092.513672 -33.23296 0.000000 0.000000 1 4 1 4 161.336217 95.362307 0.671897 16694.344017 0.399231 22
000113@J774/extract/FoilHole_1159715_Data_31204818_31204819_20190201_1932_aligned_DW_particles.mrcs J764/imported/FoilHole_1159715_Data_31204818_31204819_20190201_1932_aligned_DW.mrc 394.000000 2928.000000 -169.12019 -10.47840 14.961603 31429.634766 31092.513672 -33.23296 0.000000 0.000000 1 4 1 4 -170.79525 90.184645 0.693297 16645.338821 0.158388 23
000012@J774/extract/FoilHole_1159715_Data_31204818_31204819_20190201_1932_aligned_DW_particles.mrcs J764/imported/FoilHole_1159715_Data_31204818_31204819_20190201_1932_aligned_DW.mrc 2234.000000 3019.000000 178.095881 -5.17840 -4.11840 31429.634766 31092.513672 -33.23296 0.000000 0.000000 1 4 1 4 161.336217 95.362307 0.672823 16829.892019 0.403285 6
000287@J774/extract/FoilHole_1159715_Data_31204818_31204819_20190201_1932_aligned_DW_particles.mrcs J764/imported/FoilHole_1159715_Data_31204818_31204819_20190201_1932_aligned_DW.mrc 2010.000000 860.000000 -1.62088 0.121603 5.421603 31429.634766 31092.513672 -33.23296 0.000000 0.000000 1 4 1 4 -66.44722 55.694646 0.703808 16781.008687 0.406659 27
000245@J774/extract/FoilHole_1159715_Data_31204818_31204819_20190201_1932_aligned_DW_particles.mrcs J764/imported/FoilHole_1159715_Data_31204818_31204819_20190201_1932_aligned_DW.mrc 3115.000000 3466.000000 173.190312 7.541603 -3.05840 31429.634766 31092.513672 -33.23296 0.000000 0.000000 1 4 1 4 123.495382 112.458862 0.649564 16615.933570 0.457630 24
000050@J774/extract/FoilHole_1159715_Data_31204825_31204826_20190201_1933_aligned_DW_particles.mrcs J764/imported/FoilHole_1159715_Data_31204825_31204826_20190201_1933_aligned_DW.mrc 3038.000000 1208.000000 -165.46711 5.421603 2.241603 30977.580078 30530.318359 -66.56420 0.000000 0.000000 1 4 1 5 49.236360 102.894297 0.637815 16598.085598 0.348380 14
000007@J774/extract/FoilHole_1159715_Data_31204833_31204834_20190201_1934_aligned_DW_particles.mrcs J764/imported/FoilHole_1159715_Data_31204833_31204834_20190201_1934_aligned_DW.mrc 332.000000 3433.000000 30.413799 1.181603 -0.93840 30219.453125 29600.865234 -64.16928 0.000000 0.000000 1 4 1 6 -40.94039 83.622055 0.696446 16951.494990 0.267894 14
000004@J774/extract/FoilHole_1159715_Data_31204833_31204834_20190201_1934_aligned_DW_particles.mrcs J764/imported/FoilHole_1159715_Data_31204833_31204834_20190201_1934_aligned_DW.mrc 2891.000000 2936.000000 164.859986 5.421603 1.181603 30219.453125 29600.865234 -64.16928 0.000000 0.000000 1 4 1 6 -45.49771 68.286185 0.668709 16844.960660 0.247165 32
000122@J774/extract/FoilHole_1159715_Data_31204833_31204834_20190201_1934_aligned_DW_particles.mrcs J764/imported/FoilHole_1159715_Data_31204833_31204834_20190201_1934_aligned_DW.mrc 1669.000000 2390.000000 -38.54981 -7.29840 -8.35840 30219.453125 29600.865234 -64.16928 0.000000 0.000000 1 4 1 6 149.307907 76.630152 0.674488 16948.558810 0.474562 35
000049@J774/extract/FoilHole_1159715_Data_31204833_31204834_20190201_1934_aligned_DW_particles.mrcs J764/imported/FoilHole_1159715_Data_31204833_31204834_20190201_1934_aligned_DW.mrc 734.000000 2680.000000 88.298871 4.361603 5.421603 30219.453125 29600.865234 -64.16928 0.000000 0.000000 1 4 1 6 -38.93834 38.881156 0.696787 16919.660236 0.368154 15
000113@J774/extract/FoilHole_1159715_Data_31204833_31204834_20190201_1934_aligned_DW_particles.mrcs J764/imported/FoilHole_1159715_Data_31204833_31204834_20190201_1934_aligned_DW.mrc 1785.000000 3094.000000 -162.68707 6.481603 -1.99840 30219.453125 29600.865234 -64.16928 0.000000 0.000000 1 4 1 6 -12.97327 79.913974 0.657007 16727.464291 0.547840 11
000008@J774/extract/FoilHole_1159715_Data_31204842_31204843_20190201_1936_aligned_DW_particles.mrcs J764/imported/FoilHole_1159715_Data_31204842_31204843_20190201_1936_aligned_DW.mrc 1615.000000 3284.000000 7.369394 2.241603 -12.59840 29487.628906 29163.509766 -71.80595 0.000000 0.000000 1 4 1 7 -52.73862 92.491214 0.713132 16674.150538 0.642377 19
000062@J774/extract/FoilHole_1159715_Data_31204842_31204843_20190201_1936_aligned_DW_particles.mrcs J764/imported/FoilHole_1159715_Data_31204842_31204843_20190201_1936_aligned_DW.mrc 2211.000000 2895.000000 67.604065 -3.05840 -4.11840 29487.628906 29163.509766 -71.80595 0.000000 0.000000 1 4 1 7 -47.42354 97.629942 0.689446 16674.383150 0.749773 44
000028@J774/extract/FoilHole_1159715_Data_31204842_31204843_20190201_1936_aligned_DW_particles.mrcs J764/imported/FoilHole_1159715_Data_31204842_31204843_20190201_1936_aligned_DW.mrc 1353.000000 1175.000000 85.321867 -14.71840 13.901603 29487.628906 29163.509766 -71.80595 0.000000 0.000000 1 4 1 7 70.852618 18.975993 0.692606 16769.116234 0.450483 60
000127@J774/extract/FoilHole_1159715_Data_31204842_31204843_20190201_1936_aligned_DW_particles.mrcs J764/imported/FoilHole_1159715_Data_31204842_31204843_20190201_1936_aligned_DW.mrc 618.000000 2539.000000 -68.28734 -7.29840 -17.89840 29487.628906 29163.509766 -71.80595 0.000000 0.000000 1 4 1 7 -75.18663 90.818378 0.648401 16646.380029 0.228377 41
000020@J774/extract/FoilHole_1159715_Data_31204842_31204843_20190201_1936_aligned_DW_particles.mrcs J764/imported/FoilHole_1159715_Data_31204842_31204843_20190201_1936_aligned_DW.mrc 982.000000 819.000000 -120.84833 -1.99840 2.241603 29487.628906 29163.509766 -71.80595 0.000000 0.000000 1 4 1 7 -65.49940 111.100889 0.698358 16732.493381 0.542817 84
000125@J774/extract/FoilHole_1159715_Data_31204854_31204855_20190201_1933_aligned_DW_particles.mrcs J764/imported/FoilHole_1159715_Data_31204854_31204855_20190201_1933_aligned_DW.mrc 897.000000 405.000000 125.028314 2.241603 -1.99840 29240.873047 28953.306641 -84.93554 0.000000 0.000000 1 4 1 8 20.199164 79.960531 0.649479 16720.890494 0.366092 22
000052@J774/extract/FoilHole_1159715_Data_31204854_31204855_20190201_1933_aligned_DW_particles.mrcs J764/imported/FoilHole_1159715_Data_31204854_31204855_20190201_1933_aligned_DW.mrc 379.000000 1737.000000 144.991240 6.481603 -3.05840 29240.873047 28953.306641 -84.93554 0.000000 0.000000 1 4 1 8 -159.58553 90.344513 0.682161 16798.190565 0.294840 55
000021@J774/extract/FoilHole_1159715_Data_31204854_31204855_20190201_1933_aligned_DW_particles.mrcs J764/imported/FoilHole_1159715_Data_31204854_31204855_20190201_1933_aligned_DW.mrc 665.000000 1795.000000 174.810053 -11.53840 0.121603 29240.873047 28953.306641 -84.93554 0.000000 0.000000 1 4 1 8 -158.87860 60.947769 0.707236 16776.818391 0.158857 71

Thanks!

Jake

Hi,

I'm using the pyem command to generate a star file for one of my refinement jobs in cryosparc since I want to carry out further processing using Relion. the particles were picked and extracted in WARP and processed in cryosparc. the command for pyem is: ~ketam/pyem/csparc2star.py J130/cryosparc_P3_J130_008_particles.cs (space) J130/P3_J130_passthrough_particles.cs (space) J130/cryosparc_P3_J130_008_particles_test.star (space) --boxsize 290

The star file seems to be generated but it's missing the information for X/Y coordinates. I then tried using the star.py command to get missing parameters but I get the following error (this is for an example dataset):

~ketanm/pyem/star.py --copy-micrograph-coordinates Warp/EIF3-2/goodparticles_BoxNet2Mask_20180918.star test_cryosparc_P3_J146_particles.star cryosparc_P3_J146_particles_with_coords_dataset2.star
/lmb/home/ketanm/pyem/pyem/star.py:531: FutureWarning: Columnar iteration over characters will be deprecated in future releases.
df[UCSF.IMAGE_INDEX], df[UCSF.IMAGE_PATH] =
/lmb/home/ketanm/pyem/pyem/star.py:539: FutureWarning: Columnar iteration over characters will be deprecated in future releases.
df[UCSF.IMAGE_ORIGINAL_INDEX], df[UCSF.IMAGE_ORIGINAL_PATH] =
Traceback (most recent call last):
File "/lmb/home/ketanm/pyem/star.py", line 363, in
sys.exit(main(parser.parse_args()))
File "/lmb/home/ketanm/pyem/star.py", line 149, in main
df = star.smart_merge(df, coord_star, fields=star.Relion.MICROGRAPH_COORDS, key=args.merge_key)
File "/lmb/home/ketanm/pyem/pyem/star.py", line 143, in smart_merge
s2 = s2.set_index(key, drop=False)
File "/lmb/home/ketanm/miniconda3/envs/pyem/lib/python3.8/site-packages/pandas/core/frame.py", line 4548, in set_index
raise KeyError(f"None of {missing} are in the columns")
KeyError: 'None of [None] are in the columns’

It will be helpful to know what I'm doing wrong so that I can get the coordinates back into the star file.

Hi!

I want to get ride of particle duplicates in my .cs file using star.py.
To do that, first I am using cspar2star.py to transform aligned particles.cs file (from 2D-classification job) to .star file using:
$csparc2star.py particles.cs passthrough_particles.cs export.star
but when I load the star file generated using csparc2star.py into cryosparc (to verify coordinates), I see that the coordinates are shifted (shift of 128px in Y and 64px in X) and I get weird 2D classes (attached).

I tried to extract particles with the shifted coordinates using bigger box to include the particles but the 2D alignment also didnt work.
Is there anyway to prevent this shift from happening? or maybe to re-shift the coordinates back to the original position?
any help would be very appreciated,
thanks in advance,
ALI-AHMAD.

Hi, after I reinstalled latest version of pyem, I found csparc2star.py not work. Did you have any suggestions? Here are the loglevel:
Detected CryoSPARC 2+ .cs file
Reading passthrough file
Particle passthrough detected
Concatenating passthrough fields: alignments2D/split, alignments2D/shift, alignments2D/pose, alignments2D/psize_A, alignments2D/error, alignments2D/error_min, alignments2D/resid_pow, alignments2D/slice_pow, alignments2D/image_pow, alignments2D/cross_cor, alignments2D/alpha, alignments2D/weight, alignments2D/pose_ess, alignments2D/shift_ess, alignments2D/class_posterior, alignments2D/class, alignments2D/class_ess
Creating particle DataFrame from recarray
Directly copied fields: rlnDefocusAngle, rlnDetectorPixelSize, rlnCtfFigureOfMerit, rlnSphericalAberration, rlnAmplitudeContrast, rlnCtfMaxResolution, rlnVoltage, rlnDefocusU, rlnPhaseShift, rlnDefocusV, rlnImageName, rlnMagnification
Converting DEFOCUSANGLE from degrees to radians
Converting PHASESHIFT from degrees to radians
Collecting particle parameters from most likely classes
Columns must be same length as key
A passthrough file may be required (check inside the cryoSPARC 2+ job directory)
Columns must be same length as key
Traceback (most recent call last):
File "/opt/pyem/csparc2star.py", line 42, in main
df = metadata.parse_cryosparc_2_cs(cs, passthrough=args.passthrough, minphic=args.minphic)
File "/opt/pyem/pyem/metadata.py", line 356, in parse_cryosparc_2_cs
[cs[names[c]][i] for i, c in enumerate(cls)]))
File "/usr/lib64/python2.7/site-packages/pandas/core/frame.py", line 2352, in setitem
self._setitem_array(key, value)
File "/usr/lib64/python2.7/site-packages/pandas/core/frame.py", line 2376, in _setitem_array
raise ValueError('Columns must be same length as key')
ValueError: Columns must be same length as key
And here are the dependency versions:
future==0.17.1
numba==0.42.0
numpy==1.15.3 (I tried 1.14.0, 1.15.4 and 1.16.0, none of them worked)
scipy==1.2.0
matplotlib==2.2.3
seaborn==0.9.0
pandas==0.23.4
pathos==0.2.2.1
pyfftw==0.11.1
And we used latest pyem version now.
Looking for your reply.

Hi I am trying to convert a cs file from cryosparc 2 to relion with your script and I am getting:

python csparc2star.py particles_selected.cs test.star
Traceback (most recent call last):
File "csparc2star.py", line 115, in
sys.exit(main(parser.parse_args()))
File "csparc2star.py", line 67, in main
df = metadata.parse_cryosparc_2_cs(cs, passthrough=args.passthrough, minphic=args.minphic)
File "/home/david.haselbach/Downloads/pyem-master/pyem/metadata.py", line 249, in parse_cryosparc_2_cs
df[star.Relion.DEFOCUSANGLE] = np.rad2deg(np.arctan2(df[star.Relion.DEFOCUSV], df[star.Relion.DEFOCUSU]))
File "/home/david.haselbach/anaconda2/lib/python2.7/site-packages/pandas/core/frame.py", line 2685, in getitem
return self._getitem_column(key)
File "/home/david.haselbach/anaconda2/lib/python2.7/site-packages/pandas/core/frame.py", line 2692, in _getitem_column
return self._get_item_cache(key)
File "/home/david.haselbach/anaconda2/lib/python2.7/site-packages/pandas/core/generic.py", line 2486, in _get_item_cache
values = self._data.get(item)
File "/home/david.haselbach/anaconda2/lib/python2.7/site-packages/pandas/core/internals.py", line 4115, in get
loc = self.items.get_loc(item)
File "/home/david.haselbach/anaconda2/lib/python2.7/site-packages/pandas/core/indexes/base.py", line 3065, in get_loc
return self._engine.get_loc(self._maybe_cast_indexer(key))
File "pandas/_libs/index.pyx", line 140, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/index.pyx", line 162, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/hashtable_class_helper.pxi", line 1492, in pandas._libs.hashtable.PyObjectHashTable.get_item
File "pandas/_libs/hashtable_class_helper.pxi", line 1500, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'rlnDefocusV'

Any Idea what could be the issue?

Traceback (most recent call last):
  File "../csparc2star.py", line 115, in <module>
    sys.exit(main(parser.parse_args()))
  File "../csparc2star.py", line 73, in main
    phic.columns = [int(h[21]) for h in meta.columns if "phiC" in h]
ValueError: invalid literal for int() with base 10: 'i'

Reason is that instead of single alignments.model.phiC, we have multiple alignments.model-0.phiC, and same for alignment parameters (test for r.0 etc currently simply ignores multi-class equivalents).

Need to use alignment parameters for maximum phiC above the threshold, and convert the maximum phiC to the right relion field (rlnLogLikelihoodContribution?).

Hello,

we succesfully use csparc2star.py to carry out refinement in relion, but when we try to re-extract the data to then do a movie refinement it doesn't work.
Do you know how we could that?
Classification in relion takes forever and we use cryosparc for everything but the movie refinement only works in relion. We are trying to squeeze as much from the data as possible.
Thanks for any advice you can give me.
Best,
Chiara

Hi Dan,

Not sure if this is an avoidable issue without input from cryoSPARC devs, but I have noticed that if I take particles from a refinement that was performed with a different box size (and hence pixel size) to the original particles, then the offsets in the output star file are wrong (or correct for whatever pixel size was used in the refinement, but wrong for the pixel size in the output star file).

So for example, if I have binned particles to 4Å/pixel but then refine interpolating to 2Å per pixel, the output offsets are wrong (and will give bad results if used for recentering particles).

Cheers
Oli

I downloaded a .cs file in v2.4.0 (the job was computed with v2.3.2) and tried to convert it to a star file with pyem. The strings in 'ctf/type' and 'blob/path' came out as bytes (b'stringvalue'), so I had to modify pyem to decode this. I'd like to share the fix here:

def simplify_star_ucsf(df, inplace=True):
    df = df if inplace else df.copy()
    if UCSF.IMAGE_ORIGINAL_INDEX in df and UCSF.IMAGE_ORIGINAL_PATH in df:
        df[Relion.IMAGE_ORIGINAL_NAME] = df[UCSF.IMAGE_ORIGINAL_INDEX].map(
            lambda x: "%.6d" % (x + 1)).str.cat(df[UCSF.IMAGE_ORIGINAL_PATH],
                                                sep="@")
# the 3 lines below are the fix
    if UCSF.IMAGE_INDEX in df and UCSF.IMAGE_PATH in df: 
        if type(df[UCSF.IMAGE_PATH][0]) in set([np.bytes_, bytes]) : 
            df[UCSF.IMAGE_PATH] = df[UCSF.IMAGE_PATH].apply(lambda x: x.decode('UTF-8')) 
            
        df[Relion.IMAGE_NAME] = df[UCSF.IMAGE_INDEX].map(
            lambda x: "%.6d" % (x + 1)).str.cat(df[UCSF.IMAGE_PATH], sep="@")
    df.drop([c for c in df.columns if "ucsf" in c or "eman" in c],
            axis=1, inplace=True)
    if "index" in df.columns:
        df.set_index("index", inplace=True)
        df.sort_index(inplace=True, kind="mergesort")
    return df

Hi,
I'm now install the pyem using the miniconda, and when I try to run the csparc2star.py, it shows the following error:

Traceback (most recent call last):
File "/Users/robinchen/PycharmProjects/pyem/csparc2star.py", line 27, in
from pyem import metadata
File "/Users/robinchen/PycharmProjects/pyem/pyem/init.py", line 3, in
from . import metadata
File "/Users/robinchen/PycharmProjects/pyem/pyem/metadata.py", line 22, in
from . import star
File "/Users/robinchen/PycharmProjects/pyem/pyem/star.py", line 31, in
from pyem.util import rot2euler
File "/Users/robinchen/PycharmProjects/pyem/pyem/util/init.py", line 5, in
from .quat_numba import *
File "/Users/robinchen/PycharmProjects/pyem/pyem/util/quat_numba.py", line 30, in
"(m)->(m)", nopython=True, cache=True)
File "/Users/robinchen/anaconda/lib/python2.7/site-packages/numba/npyufunc/decorators.py", line 165, in wrap
guvec.add(fty)
File "/Users/robinchen/anaconda/lib/python2.7/site-packages/numba/npyufunc/ufuncbuilder.py", line 169, in add
self.nb_func, targetoptions, sig)
File "/Users/robinchen/anaconda/lib/python2.7/site-packages/numba/npyufunc/ufuncbuilder.py", line 92, in _compile_element_wise_function
cres = nb_func.compile(sig, **targetoptions)
File "/Users/robinchen/anaconda/lib/python2.7/site-packages/numba/npyufunc/ufuncbuilder.py", line 63, in compile
self.targetdescr.options.parse_as_flags(flags, topt)
File "/Users/robinchen/anaconda/lib/python2.7/site-packages/numba/targets/options.py", line 26, in parse_as_flags
opt.from_dict(options)
File "/Users/robinchen/anaconda/lib/python2.7/site-packages/numba/targets/options.py", line 19, in from_dict
raise KeyError(fmt % k)
KeyError: "Does not support option: 'cache'"

I try different version of python and update every package, but still don't fix it.

Please let me know if I can try other way to fix it

Thank you!

Robin

Hi, I am trying to apply a 3x4 transformation matrix obtained from chimera on a particles.star file generated in relion 3.1, but am getting the following error:

(pyem) [kapoor@kapoor-linux1 n-1]$ star.py --transform '0.9 0.4 0.1 -121.1; -0.4 0.9 -0.1 139.7; -0.2 0.03 0.9 56.4' j206_particles_n-1_temp.star tt.star
/home/kapoor/pyem/pyem/star.py:531: FutureWarning: Columnar iteration over characters will be deprecated in future releases.
df[UCSF.IMAGE_INDEX], df[UCSF.IMAGE_PATH] =
/home/kapoor/pyem/pyem/star.py:539: FutureWarning: Columnar iteration over characters will be deprecated in future releases.
df[UCSF.IMAGE_ORIGINAL_INDEX], df[UCSF.IMAGE_ORIGINAL_PATH] =
Traceback (most recent call last):
File "/home/kapoor/bin/star.py", line 360, in
sys.exit(main(parser.parse_args()))
File "/home/kapoor/bin/star.py", line 129, in main
r = np.array(json.loads(args.transform))
File "/home/kapoor/miniconda3/envs/pyem/lib/python3.8/json/init.py", line 357, in loads
return _default_decoder.decode(s)
File "/home/kapoor/miniconda3/envs/pyem/lib/python3.8/json/decoder.py", line 340, in decode
raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 1 column 5 (char 4)

Is my matrix input wrong? Any help would be much appreciated. Particle star file is pasted below (first few particles only as a test).
Thanks,
MW

version 30001

data_optics

loop_
_rlnOpticsGroupName #1
_rlnOpticsGroup #2
_rlnMtfFileName #3
_rlnMicrographOriginalPixelSize #4
_rlnVoltage #5
_rlnSphericalAberration #6
_rlnAmplitudeContrast #7
_rlnImagePixelSize #8
_rlnImageSize #9
_rlnImageDimensionality #10
_rlnMicrographPixelSize #11
opticsGroup1 1 MTF_K2_200kV.star 5.200000 200.000000 2.700000 0.100000 5.200000 128 2 5.200000
opticsGroup3 2 MTF_K2_200kV.star 5.200000 200.000000 2.700000 0.100000 5.200000 128 2 5.200000
opticsGroup6 3 MTF_K2_200kV.star 5.200000 200.000000 2.700000 0.100000 5.200000 128 2 5.200000
opticsGroup2 4 MTF_K2_300kV.star 5.200000 300.000000 2.700000 0.100000 5.200000 128 2 5.200000
opticsGroup4 5 MTF_K2_300kV.star 5.200000 300.000000 2.700000 0.100000 5.200000 128 2 5.200000
opticsGroup5 6 MTF_K2_300kV.star 5.200000 300.000000 2.700000 0.100000 5.200000 128 2 5.200000

version 30001

data_particles

loop_
_rlnCoordinateX #1
_rlnCoordinateY #2
_rlnAnglePsi #3
_rlnImageName #4
_rlnMicrographName #5
_rlnOpticsGroup #6
_rlnCtfMaxResolution #7
_rlnCtfFigureOfMerit #8
_rlnDefocusU #9
_rlnDefocusV #10
_rlnDefocusAngle #11
_rlnCtfBfactor #12
_rlnCtfScalefactor #13
_rlnPhaseShift #14
_rlnAngleRot #15
_rlnAngleTilt #16
_rlnOriginXAngst #17
_rlnOriginYAngst #18
_rlnClassNumber #19
_rlnNormCorrection #20
_rlnLogLikeliContribution #21
_rlnMaxValueProbDistribution #22
_rlnNrOfSignificantSamples #23
_rlnGroupName #24
_rlnRandomSubset #25
_rlnGroupNumber #26
571.369000 982.462000 -115.51571 000014@Extract/job032/Micrographs_bin/20190430_Arctica_52A/grafix_0001_52A.mrcs Micrographs_bin/20190430_Arctica_52A/grafix_0001_52A.mrc 1 10.481889 0.068258 10853.190430 9793.325195 -36.83827 0.000000 1.000000 0.000000 127.483969 82.975249 -0.95396 10.946037 1 0.845320 43329.678841 0.090501 37 group_02 2 1
793.031000 509.877000 -14.26509 000023@Extract/job032/Micrographs_bin/20190430_Arctica_52A/grafix_0001_52A.mrcs Micrographs_bin/20190430_Arctica_52A/grafix_0001_52A.mrc 1 10.481889 0.068258 10853.190430 9793.325195 -36.83827 0.000000 1.000000 0.000000 131.749059 69.428113 7.946037 0.546037 1 0.885231 43202.085594 0.039006 58 group_02 1 1
332.554000 765.831000 -36.29312 000016@Extract/job032/Micrographs_bin/20190430_Arctica_52A/grafix_0003_52A.mrcs Micrographs_bin/20190430_Arctica_52A/grafix_0003_52A.mrc 1 9.716788 0.082300 14850.016602 14850.016602 49.183117 0.000000 1.000000 0.000000 -128.29879 131.967749 -0.95396 -2.45396 1 0.868942 44453.415240 0.075998 19 group_03 1 2
305.308000 719.492000 164.233498 000014@Extract/job032/Micrographs_bin/20190430_Arctica_52A/grafix_0003_52A.mrcs Micrographs_bin/20190430_Arctica_52A/grafix_0003_52A.mrc 1 9.716788 0.082300 14850.016602 14850.016602 49.183117 0.000000 1.000000 0.000000 -39.83175 108.013521 -18.75396 -8.35396 1 0.911413 44442.767064 0.088459 18 group_03 2 2
262.000000 112.846000 -175.25574 000004@Extract/job032/Micrographs_bin/20190430_Arctica_52A/grafix_0005_52A.mrcs Micrographs_bin/20190430_Arctica_52A/grafix_0005_52A.mrc 1 8.904347 0.091428 12476.629883 11975.007812 -73.42570 0.000000 1.000000 0.000000 -170.70390 48.441091 -16.55396 -11.35396 1 0.887011 43983.068081 0.098374 70 group_02 2 1

Hi,

I saw related threats regarding the panda-module, but these fixes didn't work for me.

When I try to execute:
csparc2star.py /P6/J46/passthrough_particles_selected.cs /P6/J46/passthrough_particles_selected.star
I get the following error:

Traceback (most recent call last):
  File "/root/pyem/csparc2star.py", line 26, in <module>
    import pandas as pd
ImportError: No module named pandas

When I try:

# which pip
/usr/local/EMAN2/bin/pip
# which python
/usr/local/EMAN2/bin/python
# pip --version
pip 10.0.1 from /usr/local/EMAN2/lib/python2.7/site-packages/pip (python 2.7)

So I guess that's what the problem is, as also mentioned in the other threads.
I don't know how to "hide EMAN python", as was mentioned elsewhere. I'd like to use both EMAN2 and csparc2star if at all possible.

The strange thing is, it worked before. Or at least, the command csparc2star ran smoothly, but then I couldn't open the particles in relion, as it was unable to find micropgraphXX.mrc, or something related to that. But one problem at the time ;-)

Maybe unrelated, but I installed pyem as root in: /root/pyem/. Could that cause issues?

Many thanks in advance for your help, much appreciated.
Regards,
Maarten

Hi I'm trying to export particles from cryosparc to then import into relion. I'm having an issue when running the csparc2star.py command. I get the following message:

[hansenbry@D01868462 J86]$ csparc2star.py cryosparc_P4_J86_005_particles.cs cryosparc_P4_J86_005_particles.star
Traceback (most recent call last):
File "/home/hansenbry/EMsoftware/pyem/csparc2star.py", line 28, in
from pyem import metadata
File "/home/hansenbry/EMsoftware/pyem/pyem/init.py", line 3, in
from . import metadata
File "/home/hansenbry/EMsoftware/pyem/pyem/metadata.py", line 24, in
from . import star
File "/home/hansenbry/EMsoftware/pyem/pyem/star.py", line 31, in
from pyem.util import rot2euler
File "/home/hansenbry/EMsoftware/pyem/pyem/util/init.py", line 3, in
from .util import *
File "/home/hansenbry/EMsoftware/pyem/pyem/util/util.py", line 25, in
from .. import geom
File "/home/hansenbry/EMsoftware/pyem/pyem/geom/init.py", line 4, in
from .geom import *
File "/home/hansenbry/EMsoftware/pyem/pyem/geom/geom.py", line 20, in
from .quat import distq
File "/home/hansenbry/EMsoftware/pyem/pyem/geom/quat.py", line 27, in
qconj = np.vectorize(_qconj, signature="(m),(m)->(m)")
TypeError: init() got an unexpected keyword argument 'signature'

I get this message if I try to run it without any arguments as well. I just did a git clone and build today (1/16/19) so I'm assuming this is the newest code base. Thanks for any tips

Currently, trying to run pyem on Linux 3.7.10-1.28 via pip install git+https://github.com/asarnow/pyem, however, getting error due to pyem/util/quat.py.

The solution has been to go through and remove all instances (3) of , signature = ; e.g., changing the first of three lines to qconj = np.vectorize(_qconj) #, signature="(m),(m)->(m)") works when done for all 3.

To note, everything works fine on macOS.

Hi,
I tried to use the pyem package command of csparc2star.py to convert the .cs file of cryosparc into a relion 3.1 star file and didn’t work for me.

python csparc2star.py --copy-micrograph-coordinates particles.star particles_selected.cs PC2.star
Traceback (most recent call last):
File "csparc2star.py", line 28, in
from pyem import metadata
File "/home/greg/pyem/pyem/init.py", line 3, in
from . import metadata
File "/home/greg/pyem/pyem/metadata.py", line 399
ptdf = util.dataframe_from_records_mapped(pt, {**general, **micrograph})
^
SyntaxError: invalid syntax

python --version
Python 2.7.15 :: Anaconda, Inc.
python -c 'import pandas; print pandas.version'
0.21.0
any idea about how to solve this problem ?

Dear @asarnow,

Thank you first of all for an amazing script. I have no issues importing relion particles into cryosparc and back. However, I am having some problems converting particles from 2D classification (originally extracted from cryosparc) into relion .star file.

I ran the following command: /data/CRYOSPARC/pyem/csparc2star.py --copy-micrograph-coordinates --passthrough /Cryosparc_job_2D/passthrough_selected.cs /data/CRYOSPARC/USERS/Omid/2D_job/particles_selected.cs from_cryosparc.star

The command seem to give me an output, but in relion I cannot get the particles imported properly. I think part of the issue is that the particles were extracted in cryosparc from a CTF corrected images there, and the coordinate system is different than in the CTF corrected images in relion.

For each image it says 0 particles get extracted in relion.
Is there a way to correct or get around the problem?

I am using cryosparc 2.9 and relion 3.0

Best,
Omid

Hi,
My goal is to either export particles from cryosparc into relion, or alternatively to export only the locations of the particles, so something like .box files, which I can then use to extract the selected particles in relion.

I've successfully created a .star of particles after 2D classification and selection, using:
csparc2star.py /P6/J46/passthrough_particles_selected.cs /P6/J46/passthrough_particles_selected.star

I can also import it in relion, using the Import job, linking to "passthrough_particles_selected.star", with Node type: Particles STAR file (.star), but when I try to open some particles for inspection, it opens micrographs instead.

I have the feeling I'm missing something obvious here, but can't figure it out.

I've opened the .star file, and I get the following info:

data_images
loop_
_rlnMicrographName #1 
_rlnCoordinateX #2 
_rlnCoordinateY #3 
_rlnAnglePsi #4 
_rlnOriginX #5 
_rlnOriginY #6 
_rlnVoltage #7 
_rlnMagnification #8 
_rlnDefocusU #9 
_rlnDefocusV #10 
_rlnDefocusAngle #11 
_rlnSphericalAberration #12 
_rlnPhaseShift #13 
_rlnAmplitudeContrast #14 
_rlnClassNumber #15 
_rlnRandomSubset #16 

followed by all the data.

If I open another random particles.star file created by relion itself, I get:

# RELION; version 3.0

data_

loop_ 
_rlnCoordinateX #1 
_rlnCoordinateY #2 
_rlnImageName #3 
_rlnMicrographName #4 
_rlnMagnification #5 
_rlnDetectorPixelSize #6 
_rlnCtfMaxResolution #7 
_rlnCtfFigureOfMerit #8 
_rlnVoltage #9 
_rlnDefocusU #10 
_rlnDefocusV #11 
_rlnDefocusAngle #12 
_rlnSphericalAberration #13 
_rlnCtfBfactor #14 
_rlnCtfScalefactor #15 
_rlnPhaseShift #16 
_rlnAmplitudeContrast #17 

I have the suspicion the error is in the absence of _rlnImageName, although I'm not sure.

I've played around with --copy-micrograph-coordinates, and with --passthrough, although some of the online help forums were a couple of years old, so I'm not sure if all of these are still valid.

I feel like all the information is in the .star file, as all I really want are the locations, but now I need relion to properly read them as particles, instead of micrographs.

cheers,
Maarten

Dear Daniel,

csparc2star generates this error: TypeError: () got an unexpected keyword argument 'result_type'

Am I missing something on my system?

Peter

I'm trying to install the latest pyem on a RHEL7 system. I get the following error:

pip install git+https://github.com/asarnow/pyem
Collecting git+https://github.com/asarnow/pyem
Cloning https://github.com/asarnow/pyem to /tmp/pip-req-build-bNGQwH
Requirement already satisfied (use --upgrade to upgrade): pyem==0.3 from git+https://github.com/asarnow/pyem in ./EMAN2/lib/python2.7/site-packages/pyem-0.3-py2.7.egg
Requirement already satisfied: future in ./EMAN2/lib/python2.7/site-packages (from pyem==0.3) (0.16.0)
Requirement already satisfied: numba in ./EMAN2/lib/python2.7/site-packages (from pyem==0.3) (0.40.1)
Requirement already satisfied: numpy in ./EMAN2/lib/python2.7/site-packages (from pyem==0.3) (1.9.3)
Requirement already satisfied: scipy in ./EMAN2/lib/python2.7/site-packages (from pyem==0.3) (1.0.0)
Requirement already satisfied: matplotlib in ./EMAN2/lib/python2.7/site-packages (from pyem==0.3) (1.4.3)
Requirement already satisfied: seaborn in ./EMAN2/lib/python2.7/site-packages (from pyem==0.3) (0.9.0)
Requirement already satisfied: pandas in ./EMAN2/lib/python2.7/site-packages (from pyem==0.3) (0.22.0)
Requirement already satisfied: pathos in ./EMAN2/lib/python2.7/site-packages (from pyem==0.3) (0.2.2.1)
Collecting pyfftw (from pyem==0.3)
Using cached https://files.pythonhosted.org/packages/c2/2e/b25edc6960fc837e915eb1b38e5f0e3013e32e90aff14a1d0f4556b3d145/pyFFTW-0.10.4.tar.gz
Requirement already satisfied: singledispatch in ./EMAN2/lib/python2.7/site-packages (from numba->pyem==0.3) (3.4.0.3)
Requirement already satisfied: enum34 in ./EMAN2/lib/python2.7/site-packages (from numba->pyem==0.3) (1.1.6)
Requirement already satisfied: funcsigs in ./EMAN2/lib/python2.7/site-packages (from numba->pyem==0.3) (1.0.2)
Requirement already satisfied: llvmlite>=0.25.0dev0 in ./EMAN2/lib/python2.7/site-packages (from numba->pyem==0.3) (0.25.0)
Requirement already satisfied: six>=1.4 in ./EMAN2/lib/python2.7/site-packages (from matplotlib->pyem==0.3) (1.10.0)
Requirement already satisfied: python-dateutil in ./EMAN2/lib/python2.7/site-packages (from matplotlib->pyem==0.3) (2.6.1)
Requirement already satisfied: pytz in ./EMAN2/lib/python2.7/site-packages (from matplotlib->pyem==0.3) (2017.2)
Requirement already satisfied: pyparsing>=1.5.6 in ./EMAN2/lib/python2.7/site-packages (from matplotlib->pyem==0.3) (2.0.3)
Requirement already satisfied: nose>=0.11.1 in ./EMAN2/lib/python2.7/site-packages (from matplotlib->pyem==0.3) (1.3.7)
Collecting mock (from matplotlib->pyem==0.3)
Using cached https://files.pythonhosted.org/packages/e6/35/f187bdf23be87092bd0f1200d43d23076cee4d0dec109f195173fd3ebc79/mock-2.0.0-py2.py3-none-any.whl
Requirement already satisfied: dill>=0.2.8.2 in ./EMAN2/lib/python2.7/site-packages (from pathos->pyem==0.3) (0.2.8.2)
Requirement already satisfied: ppft>=1.6.4.8 in ./EMAN2/lib/python2.7/site-packages (from pathos->pyem==0.3) (1.6.4.8)
Requirement already satisfied: multiprocess>=0.70.6.1 in ./EMAN2/lib/python2.7/site-packages (from pathos->pyem==0.3) (0.70.6.1)
Requirement already satisfied: pox>=0.2.4 in ./EMAN2/lib/python2.7/site-packages (from pathos->pyem==0.3) (0.2.4)
Collecting pbr>=0.11 (from mock->matplotlib->pyem==0.3)
Using cached https://files.pythonhosted.org/packages/01/0a/1e81639e7ed6aa51554ab05827984d07885d6873e612a97268ab3d80c73f/pbr-4.3.0-py2.py3-none-any.whl
Building wheels for collected packages: pyem, pyfftw
Running setup.py bdist_wheel for pyem ... done
Stored in directory: /tmp/pip-ephem-wheel-cache-8s0TU6/wheels/cd/76/2c/0c50f235945adaaf47111c47856feb4dd796b5bfbf03fd7785
Running setup.py bdist_wheel for pyfftw ... error
Complete output from command /progs/EMAN2/bin/python -u -c "import setuptools, tokenize;file='/tmp/pip-install-Us0CBA/pyfftw/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" bdist_wheel -d /tmp/pip-wheel-eqq5Qf --python-tag cp27:
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/pyfftw
copying pyfftw/version.py -> build/lib.linux-x86_64-2.7/pyfftw
copying pyfftw/init.py -> build/lib.linux-x86_64-2.7/pyfftw
creating build/lib.linux-x86_64-2.7/pyfftw/builders
copying pyfftw/builders/_utils.py -> build/lib.linux-x86_64-2.7/pyfftw/builders
copying pyfftw/builders/init.py -> build/lib.linux-x86_64-2.7/pyfftw/builders
copying pyfftw/builders/builders.py -> build/lib.linux-x86_64-2.7/pyfftw/builders
creating build/lib.linux-x86_64-2.7/pyfftw/interfaces
copying pyfftw/interfaces/cache.py -> build/lib.linux-x86_64-2.7/pyfftw/interfaces
copying pyfftw/interfaces/scipy_fftpack.py -> build/lib.linux-x86_64-2.7/pyfftw/interfaces
copying pyfftw/interfaces/_utils.py -> build/lib.linux-x86_64-2.7/pyfftw/interfaces
copying pyfftw/interfaces/init.py -> build/lib.linux-x86_64-2.7/pyfftw/interfaces
copying pyfftw/interfaces/numpy_fft.py -> build/lib.linux-x86_64-2.7/pyfftw/interfaces
running build_ext
building 'pyfftw.pyfftw' extension
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/tmp
creating build/temp.linux-x86_64-2.7/tmp/pip-install-Us0CBA
creating build/temp.linux-x86_64-2.7/tmp/pip-install-Us0CBA/pyfftw
creating build/temp.linux-x86_64-2.7/tmp/pip-install-Us0CBA/pyfftw/pyfftw
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/tmp/pip-install-Us0CBA/pyfftw/include -I/tmp/pip-install-Us0CBA/pyfftw/pyfftw -I/progs/EMAN2/lib/python2.7/site-packages/numpy/core/include -I/progs/EMAN2/include/python2.7 -c /tmp/pip-install-Us0CBA/pyfftw/pyfftw/pyfftw.c -o build/temp.linux-x86_64-2.7/tmp/pip-install-Us0CBA/pyfftw/pyfftw/pyfftw.o
In file included from /progs/EMAN2/lib/python2.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1804:0,
from /progs/EMAN2/lib/python2.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:17,
from /progs/EMAN2/lib/python2.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,
from /tmp/pip-install-Us0CBA/pyfftw/pyfftw/pyfftw.c:268:
/progs/EMAN2/lib/python2.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
#warning "Using deprecated NumPy API, disable it by "
^
/tmp/pip-install-Us0CBA/pyfftw/pyfftw/pyfftw.c:272:19: fatal error: fftw3.h: No such file or directory
#include "fftw3.h"
^
compilation terminated.
error: command 'gcc' failed with exit status 1

Failed building wheel for pyfftw
Running setup.py clean for pyfftw
Successfully built pyem
Failed to build pyfftw
Installing collected packages: pyfftw, pbr, mock
Running setup.py install for pyfftw ... error
Complete output from command /progs/EMAN2/bin/python -u -c "import setuptools, tokenize;file='/tmp/pip-install-Us0CBA/pyfftw/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-record-KeDpa7/install-record.txt --single-version-externally-managed --compile:
running install
running build
running build_py
running build_ext
building 'pyfftw.pyfftw' extension
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/tmp
creating build/temp.linux-x86_64-2.7/tmp/pip-install-Us0CBA
creating build/temp.linux-x86_64-2.7/tmp/pip-install-Us0CBA/pyfftw
creating build/temp.linux-x86_64-2.7/tmp/pip-install-Us0CBA/pyfftw/pyfftw
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/tmp/pip-install-Us0CBA/pyfftw/include -I/tmp/pip-install-Us0CBA/pyfftw/pyfftw -I/progs/EMAN2/lib/python2.7/site-packages/numpy/core/include -I/progs/EMAN2/include/python2.7 -c /tmp/pip-install-Us0CBA/pyfftw/pyfftw/pyfftw.c -o build/temp.linux-x86_64-2.7/tmp/pip-install-Us0CBA/pyfftw/pyfftw/pyfftw.o
In file included from /progs/EMAN2/lib/python2.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1804:0,
from /progs/EMAN2/lib/python2.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:17,
from /progs/EMAN2/lib/python2.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,
from /tmp/pip-install-Us0CBA/pyfftw/pyfftw/pyfftw.c:268:
/progs/EMAN2/lib/python2.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
#warning "Using deprecated NumPy API, disable it by "
^
/tmp/pip-install-Us0CBA/pyfftw/pyfftw/pyfftw.c:272:19: fatal error: fftw3.h: No such file or directory
#include "fftw3.h"
^
compilation terminated.
error: command 'gcc' failed with exit status 1

----------------------------------------

Command "/progs/EMAN2/bin/python -u -c "import setuptools, tokenize;file='/tmp/pip-install-Us0CBA/pyfftw/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-record-KeDpa7/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-install-Us0CBA/pyfftw/

Hi Daniel,

Thank you for developing these very useful tools. I have been trying to import particle coordinates from csparc (2.15.0) to relion (3.1) using the very handy csparc2star.py, but have been running into the following issues:

1. The coordinates are flipped about the X-axis (not the diagonal, so I don't expect --swapxy to fix it).
2. The coordinates seem to be scaled by a small factor. When inspecting picks, one can tell that applying a small shift to center some particles will make other particles fall off center further. A small shift in addition to scaling cannot be discarded.

The X-axis flipping is easy to fix, but I can't tell what the correct scaling and/or shifting is, so perhaps it's worth figuring out why the coordinates are not being correctly parsed by csparc2star.py.

My current workflow:

The particles come from blob picking in csparc, (diameter 20 to 75A, max resolution to consider 4A), followed by curation, extraction in a 256pix box (0.57angpix, no scaling), 2D classification and 2D template selection.

I give csparc2star.py the passthrough_particles_selected.cs file from the 2D selection job, and get a star file with coordinates and micrograph paths (paths have to be edited to point at the relion-run motcor2 instead of the csparc inputs). Csparc and relion micrographs are the same dimensions (3710x3838).

I split the coordinates by micrograph name in different *.box-like files in order to import them to relion with the "Particle coordinates" node type. These particles, when extracted and inspected, are X-flipped and shifted/scaled by a small factor. This is further evidenced by the fact that 2D classification in RELION from these particles will yield mostly noise, when csparc produced clean classes.

Any idea where the issued could be coming from? Let me know if any additional information would be helpful. Your help is very much appreciated.

Thank you,
Benjamin

Dear Daniel Asarnow @asarnow

I am using your csparc2star.py program. It is very useful for me. However I encounter a fundamental problem with the coordinates in my case. The coordinates that I get from your program are correct but the way relion3-beta and cryosparc read micrographs is different in my hands. Basically there is a y-flip on the coordinates (for tif files) on cryosparc compared to relion3-beta. The movie pixel information on cryosparc shows movie_blob/shape: [ 40 3710 3838] and apparently 3710 is the y dimension. When I subtract the y coordinate reported on the star file from your program from 3710, I get the correct value. If I am not mistaken, do you think you can easily implement this?

Thank you for your contribution to the community.

Alpay