thepetabyteproject / your Goto Github PK
View Code? Open in Web Editor NEWYour Unified Reader
Home Page: https://thepetabyteproject.github.io/your/
License: GNU General Public License v3.0
Your Unified Reader
Home Page: https://thepetabyteproject.github.io/your/
License: GNU General Public License v3.0
Do we need the fix_empty_psrfits branch?
I think the changes were pushed to main branch with 42e10ff.
Describe the bug
16 bit data in psrfits can be int16, but is always written as uint16 in Filterbanks. Would be good to add a warning about this.
It would be nice to add in option to dedisperse the data and visualize RFI mitigation, given relevant inputs, in your_viewer
. Once decimation is implemented, can also add in the option to display decimated data.
Describe the bug
your_writer.py
writes data in .
instead of ./
when no output-dir is given
To Reproduce
your_writer.py -f test.fits -t fil
Expected behaviour
should create test.fil
instead of .test.fil
The file has 66 isub(s)
. I think this bit of the code needs to be checked.
2019-11-11 17:17:56,463 - to_dada -your.dada - DEBUG - Data read is 6291456, Data step is 32768
2019-11-11 17:17:56,463 - get_data -your - DEBUG - Reading from 32768 samples from sample 6291456
2019-11-11 17:17:56,463 - get_data -your.psrfits - DEBUG - Startsub 384, endsub 386
2019-11-11 17:17:56,463 - get_data -your.psrfits - DEBUG - isub is 384
2019-11-11 17:17:56,464 - get_data -your.psrfits - DEBUG - file id is 1
2019-11-11 17:17:56,464 - get_data -your.psrfits - DEBUG - Reading file: /hyrule/data/grbfrb/P3231D/puppi_58763_J1935+2154_0296_0006.fits
2019-11-11 17:17:56,464 - get_data -your.psrfits - DEBUG - Using: [<astropy.io.fits.hdu.image.PrimaryHDU object at 0x7f4691973b38>, <astropy.io.fits.hdu.table.BinTableHDU object at 0x7f469196d390>]
2019-11-11 17:17:56,464 - read_subint -your.psrfits - WARNING - Polarization is IQUV, just using Stokes I
2019-11-11 17:17:56,515 - get_data -your.psrfits - DEBUG - isub is 385
2019-11-11 17:17:56,515 - get_data -your.psrfits - DEBUG - file id is 1
2019-11-11 17:17:56,515 - get_data -your.psrfits - DEBUG - Reading file: /hyrule/data/grbfrb/P3231D/puppi_58763_J1935+2154_0296_0006.fits
2019-11-11 17:17:56,515 - get_data -your.psrfits - DEBUG - Using: [<astropy.io.fits.hdu.image.PrimaryHDU object at 0x7f4691973b38>, <astropy.io.fits.hdu.table.BinTableHDU object at 0x7f469196d390>]
2019-11-11 17:17:56,515 - read_subint -your.psrfits - WARNING - Polarization is IQUV, just using Stokes I
2019-11-11 17:17:56,571 - get_data -your.psrfits - DEBUG - isub is 386
2019-11-11 17:17:56,571 - get_data -your.psrfits - DEBUG - file id is 1
2019-11-11 17:17:56,571 - get_data -your.psrfits - DEBUG - Reading file: /hyrule/data/grbfrb/P3231D/puppi_58763_J1935+2154_0296_0006.fits
2019-11-11 17:17:56,571 - get_data -your.psrfits - DEBUG - Using: [<astropy.io.fits.hdu.image.PrimaryHDU object at 0x7f4691973b38>, <astropy.io.fits.hdu.table.BinTableHDU object at 0x7f469196d390>]
Process p1:
Traceback (most recent call last):
File "/home/dagarwal/soft/conda/envs/numba_gpu/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/home/dagarwal/soft/conda/envs/numba_gpu/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/home/dagarwal/soft/conda/envs/numba_gpu/lib/python3.6/site-packages/your-0.4.1-py3.6.egg/your/dada.py", line 114, in to_dada
data_input = self.your_object.get_data(data_read, self.data_step)
File "/home/dagarwal/soft/conda/envs/numba_gpu/lib/python3.6/site-packages/your-0.4.1-py3.6.egg/your/__init__.py", line 65, in get_data
return PsrfitsFile.get_data(self, nstart, nsamp)
File "/home/dagarwal/soft/conda/envs/numba_gpu/lib/python3.6/site-packages/your-0.4.1-py3.6.egg/your/psrfits.py", line 254, in get_data
data.append(self.read_subint(int(isub % self.nsubints)))
File "/home/dagarwal/soft/conda/envs/numba_gpu/lib/python3.6/site-packages/your-0.4.1-py3.6.egg/your/psrfits.py", line 138, in read_subint
sdata = self.fits['SUBINT'].data[isub]['DATA']
File "/home/dagarwal/soft/conda/envs/numba_gpu/lib/python3.6/site-packages/astropy/io/fits/fitsrec.py", line 492, in __getitem__
out = self.view(np.recarray)[key]
File "/home/dagarwal/soft/conda/envs/numba_gpu/lib/python3.6/site-packages/numpy/core/records.py", line 506, in __getitem__
obj = super(recarray, self).__getitem__(indx)
IndexError: index 66 is out of bounds for axis 0 with size 66
When I run your_heimdall.py, I have memory and cuda issues, and can not get .cand file. Please see the below message.
(psrdada and psrdada-python was already installed.)
your_heimdall.py -dm 560 570 -f *.fits
Created DADA data block with nbufs=8 bufsz=536870912 nread=1
Created DADA header block with nhdrs = 8, hdrsz = 4096 bytes, nread=1
ipcbuf_lock: shmctl (shmid, SHM_LOCK): Cannot allocate memory
Could not lock DADA data block into RAM
Reading... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0% -:--:--Createing PSRDADA client
Connecting to ring buffer
Waiting for next header / data
Reading... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0% -:--:--processing beam 1
Reading... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0% -:--:--Could not setCudaDevice to 0: forward compatibility was attempted on non supported HW
ERROR: Pipeline creation failed
Unknown error. Please contact the author(s).
Reading... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
Could you tell me how to fix these errors?
Versions
Need tests for the decimation
In to_fits
in Writer
, it would be useful to add in the option to select a range of channels and spectra to write. This is already there in to_fil
(inputs: c
, nstart
, nsamp
)
Some small issues and typos (with suggestions) I found in the documentation:
The docs do not explicitly state the target audience as required for a statement of need. I will suggest adding one or two sentences in the README explaining use for a general audience. Also, hyperlinking sigproc, psrfits, etc to their proper references might be a good idea.
filterbanks -> filterbank, go -> convert
your
to differentiate from your. appropritate -> appropriateyour
your
, it can also be added to the statement of need in README.get_chunk()
will extract data from the source file.from candidate import crop
results in ModuleNotFoundErroryour
or just drop it in this case.your_heimdall.py
description is not very clear.This issue is part of my JOSS review.
A filterbank file containing custom keywords crashes the reader.
Consider, e.g., https://data.caltech.edu/records/1647:
We note that we use a custom filterbank header that includes the following keys in addition to the standard filterbank header keys:
MJD_hour - The hour (in MJD-7hr) that the (typically 5 hr long) observing track started.
...
start_sample - The sample of the first time sample contained in the filterbank file.
When trying to read this filterbank file with YOUR:
import your
fil_file = 'candidate_ovro_20200428.fil'
dobj = your.Your(fil_file)
KeyError: 'start_sample'
Expected behavior: the reader should provide a warning and continue reading the file.
e.g.,
"Unknown keyword: start_sample"
It's not ideal, but crashing with KeyError is worse.
[bowser:/hyrule/data/users/dagarwal/ALFABURST_CAND/S1/pdevs1] dagarwal% your_header.py -f p3231.20190504.pointing1.b0s1g0.0000*.fits
usage: your_header.py [-h] -f FILES
your_header.py: error: unrecognized arguments: p3231.20190504.pointing1.b0s1g0.00001.fits p3231.20190504.pointing1.b0s1g0.00002.fits p3231.20190504.pointing1.b0s1g0.00003.fits p3231.20190504.pointing1.b0s1g0.00004.fits p3231.20190504.pointing1.b0s1g0.00005.fits p3231.20190504.pointing1.b0s1g0.00006.fits p3231.20190504.pointing1.b0s1g0.00007.fits p3231.20190504.pointing1.b0s1g0.00008.fits p3231.20190504.pointing1.b0s1g0.00009.fits
When running the plotting functions, this warning pops up:
MatplotlibDeprecationWarning: Support for setting the 'text.latex.preamble' or
'pgf.preamble' rcParam to a list of strings is deprecated since 3.3 and will be
removed two minor releases later; set it to a single string instead.
self[key] = other[key]
It would be good to fix this warning. I suppose putting all LaTex commands in a single string will do.
This issue is part of my JOSS review.
Apologies if this is a simple misunderstanding, a lot of the PSRFITS/PSRCHIVE documentation is offline/incomplete making this hard to debug.
I was trying to open the psrchive files from Oostrum et al. 2020, which are in *.ar
format by using psrconv
to convert to PSRFITS and then using your
The following error gets thrown (ValueError: File 'R1_B01.fits' does not appear to be PSRFITS!
) and is due to is_PSRFITS
reading the "OBS_MODE" field of the fits file being set to "PSR" instead of "SEARCH". I'm not sure what the difference is, but bypassing this condition to force your to try and read it results in other format errors.
If this is a problem with the data itself or a bug in psrconv feel free to close.
Is there some other way to read *.ar
files with your?
your/utils/rfi.py:77: DeprecationWarning: median_absolute_deviation
is deprecated, use median_abs_deviation
instead!
To preserve the existing default behavior, use
scipy.stats.median_abs_deviation(..., scale=1/1.4826)
.
The value 1.4826 is not numerically precise for scaling
with a normal distribution. For a numerically precise value, use
scipy.stats.median_abs_deviation(..., scale='normal')
.
std = 1.4826 * stats.median_absolute_deviation(sk_c)
It would be useful to add in the option to do subband search in your_heimdall
either based on number of subbands or by channel ranges.
Add a tool to visualize the data and some basic statistics from the command line. Could optionally show bandpass and time series of the relevant chunk. Can also do dedispersion, maybe RFI flagging too?
Dependabot couldn't authenticate with https://pypi.python.org/simple/.
You can provide authentication details in your Dependabot dashboard by clicking into the account menu (in the top right) and selecting 'Config variables'.
your_object = Your("/hyrule/data/grbfrb/P3231_ALFA/test/4bit-p3231.20191118.J2229+2643.b0g0_0001.fits")
your_object.get_data(0,1024)
IndexError Traceback (most recent call last)
<ipython-input-8-926a6d8290ce> in <module>
----> 1 your_object.get_data(0,1024)
~/soft/conda/envs/numba_gpu/lib/python3.6/site-packages/your-0.4.2-py3.6.egg/your/__init__.py in get_data(self, nstart, nsamp)
64 return SigprocFile.get_data(self, nstart, nsamp)
65 else:
---> 66 return PsrfitsFile.get_data(self, nstart, nsamp)
67
68 def __repr__(self):
~/soft/conda/envs/numba_gpu/lib/python3.6/site-packages/your-0.4.2-py3.6.egg/your/psrfits.py in get_data(self, nstart, nsamp)
265 logger.debug(f'Reading subint {fsub} in file {self.filename}')
266 try:
--> 267 data.append(self.read_subint(fsub))
268 except KeyError:
269 logger.warn(f"Encountered KeyError, maybe mmap'd object was delected")
~/soft/conda/envs/numba_gpu/lib/python3.6/site-packages/your-0.4.2-py3.6.egg/your/psrfits.py in read_subint(self, isub, apply_weights, apply_scales, apply_offsets)
140 if self.nbits < 8: # Unpack the bytes data
141 if (shp[0] != self.nsamp_per_subint) and \
--> 142 (shp[1] != self.nchan * self.nbits / 8):
143 sdata = sdata.reshape(self.nsamp_per_subint,
144 int(self.nchan * self.nbits / 8))
IndexError: tuple index out of range
can improve tests for candidate, esp for different cases in get_chunk
some functions of dada writer can now be moved to Writer class.
Describe the bug
A clear and concise description of what the bug is.
The bug comes from running your_combine_mocks.py normally.
Apparently the nsamp variable variable defined in line 303 is defined as a method instead of a number.
To Reproduce
Steps to reproduce the behavior:
This error was produced when running:
your_combine_mocks.py -f1 freq1/* -f2 freq2/*
The original files cannot be included since they are 2 files of 2GB size each
However this problem should be reproducable for any 2 frequency split file.
Expected behavior
A clear and concise description of what you expected to happen.
I expected it to be able to combine both frequency files and not get stuck on line 308
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):
Smartphone (please complete the following information):
-OS Centos8
Additional context
Add any other context about the problem here.
This should be fixed by changing line 303 from :
nsamp = lowband_obj.nspectra
to:
nsamp = lowband_obj.nspectra()
Bandpass fitter tests are failing for some reason. Here is the link.
Should we consider moving to another CI service like Circle or Travis? Github Actions keep breaking.
I ran the action associated with the commit 136e2cd in a brand new conda env, and it works just fine.
Describe the bug
Can't read a bunch of fits files together for some reason?
To Reproduce
your_header.py -f /hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145*fits -v
[02/20/21 11:00:17] DEBUG 2021-02-20 11:00:17,100 - __init__ -your.your - DEBUG - Reading the file(s): your.py:70
['/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0001.fits',
'/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0002.fits',
'/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0003.fits',
'/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0004.fits',
'/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0005.fits',
'/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0006.fits',
'/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0007.fits',
'/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0008.fits',
'/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0009.fits',
'/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0010.fits',
'/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0011.fits',
'/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0012.fits',
'/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0013.fits',
'/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0014.fits',
'/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0015.fits',
'/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0016.fits',
'/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0017.fits',
'/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0018.fits',
'/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0019.fits',
'/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0020.fits',
'/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0021.fits',
'/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0022.fits',
'/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0023.fits',
'/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0024.fits',
'/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0025.fits',
'/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0026.fits']
WARNING 2021-02-20 11:00:17,136 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 1(GBT)! psrfits.py:477
WARNING 2021-02-20 11:00:17,154 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 2(GBT)! psrfits.py:477
WARNING 2021-02-20 11:00:17,171 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 3(GBT)! psrfits.py:477
WARNING 2021-02-20 11:00:17,189 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 4(GBT)! psrfits.py:477
WARNING 2021-02-20 11:00:17,206 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 5(GBT)! psrfits.py:477
WARNING 2021-02-20 11:00:17,223 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 6(GBT)! psrfits.py:477
WARNING 2021-02-20 11:00:17,241 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 7(GBT)! psrfits.py:477
WARNING 2021-02-20 11:00:17,258 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 8(GBT)! psrfits.py:477
WARNING 2021-02-20 11:00:17,275 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 9(GBT)! psrfits.py:477
WARNING 2021-02-20 11:00:17,292 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 10(GBT)! psrfits.py:477
WARNING 2021-02-20 11:00:17,309 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 11(GBT)! psrfits.py:477
WARNING 2021-02-20 11:00:17,326 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 12(GBT)! psrfits.py:477
WARNING 2021-02-20 11:00:17,343 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 13(GBT)! psrfits.py:477
WARNING 2021-02-20 11:00:17,360 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 14(GBT)! psrfits.py:477
WARNING 2021-02-20 11:00:17,377 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 15(GBT)! psrfits.py:477
WARNING 2021-02-20 11:00:17,394 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 16(GBT)! psrfits.py:477
WARNING 2021-02-20 11:00:17,411 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 17(GBT)! psrfits.py:477
WARNING 2021-02-20 11:00:17,428 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 18(GBT)! psrfits.py:477
WARNING 2021-02-20 11:00:17,445 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 19(GBT)! psrfits.py:477
WARNING 2021-02-20 11:00:17,462 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 20(GBT)! psrfits.py:477
WARNING 2021-02-20 11:00:17,479 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 21(GBT)! psrfits.py:477
WARNING 2021-02-20 11:00:17,495 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 22(GBT)! psrfits.py:477
WARNING 2021-02-20 11:00:17,512 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 23(GBT)! psrfits.py:477
WARNING 2021-02-20 11:00:17,529 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 24(GBT)! psrfits.py:477
WARNING 2021-02-20 11:00:17,546 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 25(GBT)! psrfits.py:477
Traceback (most recent call last):
File "/home/dagarwal/soft/conda/envs/your_env/bin/your_header.py", line 104, in <module>
read_header(values.files, values.no_table)
File "/home/dagarwal/soft/conda/envs/your_env/bin/your_header.py", line 55, in read_header
y = Your(f)
File "/home/dagarwal/soft/conda/envs/your_env/lib/python3.8/site-packages/your/your.py", line 72, in __init__
self.formatclass.__init__(self, self.your_file)
File "/home/dagarwal/soft/conda/envs/your_env/lib/python3.8/site-packages/your/formats/psrfits.py", line 112, in __init__
self.specinfo = SpectraInfo(psrfitslist)
File "/home/dagarwal/soft/conda/envs/your_env/lib/python3.8/site-packages/your/formats/psrfits.py", line 557, in __init__
first_subint = subint_hdu.data[0]
File "/home/dagarwal/.local/lib/python3.8/site-packages/astropy/io/fits/fitsrec.py", line 512, in __getitem__
out = self.view(np.recarray)[key]
File "/home/dagarwal/soft/conda/envs/your_env/lib/python3.8/site-packages/numpy/core/records.py", line 519, in __getitem__
obj = super(recarray, self).__getitem__(indx)
IndexError: index 0 is out of bounds for axis 0 with size 0
Describe the bug
your_writer.py gives a value error.
To Reproduce
Steps to reproduce the behavior:
your_writer.py -f *fits -t fil -o ./cands -r -sksig 4 -sgsig 4 -sgfw 15 -zero_dm_subt
Traceback (most recent call last):
File "/home/rat0022/anaconda3/envs/new/bin/your_writer.py", line 4, in <module>
__import__('pkg_resources').run_script('your==0.5.1', 'your_writer.py')
File "/home/rat0022/anaconda3/envs/new/lib/python3.6/site-packages/pkg_resources/__init__.py", line 667, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/home/rat0022/anaconda3/envs/new/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1464, in run_script
exec(code, namespace, namespace)
File "/home/rat0022/anaconda3/envs/new/lib/python3.6/site-packages/your-0.5.1-py3.6.egg/EGG-INFO/scripts/your_writer.py", line 70, in <module>
w.to_fil()
File "/home/rat0022/anaconda3/envs/new/lib/python3.6/site-packages/your-0.5.1-py3.6.egg/your/writer.py", line 167, in to_fil
if nsamps % interval != 0:
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
Describe the bug
Writer doesn't correctly write data from 4 pol fits files.
To Reproduce
>> your_writer.py -f 4_pol_fits_file -t fits -nstart 0 -nsamp 5000 -o ./ -name test
>> your_header.py -f test.fits
This will show that the output file has only 1 pol, but its size would be 4 times larger.
>> your_viewer.py -f test.fits
.
.
.
Traceback (most recent call last):
File "/home/kshitij/anaconda3/envs/grbfrb/bin/your_viewer.py", line 4, in <module>
__import__('pkg_resources').run_script('your==0.6.3', 'your_viewer.py')
File "/home/kshitij/anaconda3/envs/grbfrb/lib/python3.6/site-packages/pkg_resources/__init__.py", line 650, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/home/kshitij/anaconda3/envs/grbfrb/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1446, in run_script
exec(code, namespace, namespace)
File "/home/kshitij/anaconda3/envs/grbfrb/lib/python3.6/site-packages/your-0.6.3-py3.6.egg/EGG-INFO/scripts/your_viewer.py", line 375, in <module>
values.files, values.start, values.gulp, values.chan_std
File "/home/kshitij/anaconda3/envs/grbfrb/lib/python3.6/site-packages/your-0.6.3-py3.6.egg/EGG-INFO/scripts/your_viewer.py", line 139, in load_file
self.read_data()
File "/home/kshitij/anaconda3/envs/grbfrb/lib/python3.6/site-packages/your-0.6.3-py3.6.egg/EGG-INFO/scripts/your_viewer.py", line 270, in read_data
self.data = self.your_obj.get_data(self.start_samp, self.gulp_size).T
File "/home/kshitij/anaconda3/envs/grbfrb/lib/python3.6/site-packages/your-0.6.3-py3.6.egg/your/your.py", line 240, in get_data
data = self.formatclass.get_data(self, nstart, nsamp, pol=pol)[:, 0, :]
File "/home/kshitij/anaconda3/envs/grbfrb/lib/python3.6/site-packages/your-0.6.3-py3.6.egg/your/formats/psrfits.py", line 392, in get_data
data.append(self.read_subint(fsub, pol=pol))
File "/home/kshitij/anaconda3/envs/grbfrb/lib/python3.6/site-packages/your-0.6.3-py3.6.egg/your/formats/psrfits.py", line 267, in read_subint
data = data.reshape((self.nsamp_per_subint, self.nchan)).astype(np.float32)
ValueError: cannot reshape array of size 81920000 into shape (5000,4096)
This is because it actually wrote 4 pol data to the fits file, but the header info in the fits file is incorrect.
Describe the bug
Zero DM subtraction is leading to overflow when the data is to be written in integer format.
To Reproduce
your_writer.py -f 28.fits -t fil -r -zero_dm_subt
Expected behavior
It should just create a fil file with zero DM subtracted data. But that would not be possible if the output data format is integer. After 0 DM subtraction mean will be 0, so some data will be negative, and that overflows when typecasted into unsigned integer.
The submitted draft looks great! Few comments:
Other suggestions:
dedisp
)This issue is part of my JOSS review.
it would be useful if your_viewer
can optionally dedisperse and RFI flag the data as well.
Candidate tests seem to be failing, maybe because of low tolerance in the comparison.
Need to fix that.
Describe the bug
cands.csv is not properly parsed using pandas.read_csv in the line: cand_pars = pd.read_csv(values.cand_param_file)
The first entry is treated as a header and ignored. Something like this could be better:
cand_pars = pd.read_csv(values.cand_param_file, names=["file", "snr", "stime", "dm", "width", "label"])
The keys "file", "snr", "width", "dm", "label", "stime", "chan_mask_path", and "num_files" are undefined otherwise. This creates problems later on in the code.
See:
process_list.append(
[row['file'], row['snr'], 2 ** row['width'], row['dm'], row['label'], row['stime'],
row['chan_mask_path'], row['num_files'], values, gpu_id])
To Reproduce
Traceback (most recent call last):
File "/home/pearlman/miniconda3/envs/fetch/lib/python3.7/site-packages/pandas/core/indexes/base.py", line 4736, in get_value
return libindex.get_value_box(s, key)
File "pandas/_libs/index.pyx", line 51, in pandas._libs.index.get_value_box
File "pandas/_libs/index.pyx", line 47, in pandas._libs.index.get_value_at
File "pandas/_libs/util.pxd", line 98, in pandas._libs.util.get_value_at
File "pandas/_libs/util.pxd", line 83, in pandas._libs.util.validate_indexer
TypeError: 'str' object cannot be interpreted as an integer
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/pearlman/.local/bin/your_candmaker.py", line 4, in <module>
__import__('pkg_resources').run_script('your==0.4.9', 'your_candmaker.py')
File "/home/pearlman/miniconda3/envs/fetch/lib/python3.7/site-packages/pkg_resources/__init__.py", line 666, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/home/pearlman/miniconda3/envs/fetch/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1453, in run_script
exec(code, namespace, namespace)
File "/home/pearlman/.local/lib/python3.7/site-packages/your-0.4.9-py3.7.egg/EGG-INFO/scripts/your_candmaker.py", line 176, in <module>
[row['file'], row['snr'], 2 ** row['width'], row['dm'], row['label'], row['stime'],
File "/home/pearlman/miniconda3/envs/fetch/lib/python3.7/site-packages/pandas/core/series.py", line 1068, in __getitem__
result = self.index.get_value(self, key)
File "/home/pearlman/miniconda3/envs/fetch/lib/python3.7/site-packages/pandas/core/indexes/base.py", line 4744, in get_value
raise e1
File "/home/pearlman/miniconda3/envs/fetch/lib/python3.7/site-packages/pandas/core/indexes/base.py", line 4730, in get_value
return self._engine.get_value(s, k, tz=getattr(series.dtype, "tz", None))
File "pandas/_libs/index.pyx", line 80, in pandas._libs.index.IndexEngine.get_value
File "pandas/_libs/index.pyx", line 88, in pandas._libs.index.IndexEngine.get_value
File "pandas/_libs/index.pyx", line 131, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/hashtable_class_helper.pxi", line 1607, in pandas._libs.hashtable.PyObjectHashTable.get_item
File "pandas/_libs/hashtable_class_helper.pxi", line 1614, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'file'
Expected behavior
your_candmaker.py should parse the cands.csv and generate .h5 files exactly like candmaker.py.
In some cases your_heimdall either doesn't clear the shared memory when exiting, or gives an error if shared memory is already used by something.
can mention somewhere that this can be used to clear the shared memory:
https://roadha.us/2011/09/kill-all-shared-memory-segments-in-linux.html
Fix the telescope name, and other such header parameters written out by Writer
Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
When a title is not defined in output, it chooses .fil as the name.
Describe the solution you'd like
A clear and concise description of what you want to happen.
I would like a default name, whether chosen based on the original filenames, or just a default name like combined.fits, when no name is present in output definition.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
I've considered leaving it as is and leaving the naming to the user, however, it could be a good improvement for users that might be using the program for intermediate file products for a quick check of the data.
Additional context
Add any other context or screenshots about the feature request here.
your_viewer.py -f /gbo/20m/57889/Skynet_57889_psr_b1822_09_28142_28500_0002.fits
Leads to
Traceback (most recent call last):
File "/home/dagarwal/soft/conda/envs/your_env/bin/your_viewer.py", line 4, in <module>
__import__('pkg_resources').run_script('your==0.5.6', 'your_viewer.py')
File "/home/dagarwal/soft/conda/envs/your_env/lib/python3.8/site-packages/pkg_resources/__init__.py", line 665, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/home/dagarwal/soft/conda/envs/your_env/lib/python3.8/site-packages/pkg_resources/__init__.py", line 1463, in run_script
exec(code, namespace, namespace)
File "/home/dagarwal/soft/conda/envs/your_env/lib/python3.8/site-packages/your-0.5.6-py3.8.egg/EGG-INFO/scripts/your_viewer.py", line 295, in <module>
app.load_file(values.files, values.start, values.gulp, values.chan_std) # load file with user params
File "/home/dagarwal/soft/conda/envs/your_env/lib/python3.8/site-packages/your-0.5.6-py3.8.egg/EGG-INFO/scripts/your_viewer.py", line 125, in load_file
self.read_data()
File "/home/dagarwal/soft/conda/envs/your_env/lib/python3.8/site-packages/your-0.5.6-py3.8.egg/EGG-INFO/scripts/your_viewer.py", line 228, in read_data
self.data = self.your_obj.get_data(self.start_samp, self.gulp_size).T
File "/home/dagarwal/soft/conda/envs/your_env/lib/python3.8/site-packages/your-0.5.6-py3.8.egg/your/your.py", line 204, in get_data
data = self.formatclass.get_data(self, nstart, nsamp, pol=pol)[:, 0, :]
File "/home/dagarwal/soft/conda/envs/your_env/lib/python3.8/site-packages/your-0.5.6-py3.8.egg/your/formats/psrfits.py", line 384, in get_data
data.append(self.read_subint(fsub, pol=pol))
File "/home/dagarwal/soft/conda/envs/your_env/lib/python3.8/site-packages/your-0.5.6-py3.8.egg/your/formats/psrfits.py", line 259, in read_subint
data = data.reshape((self.nsamp_per_subint, self.nchan)).astype(np.float32)
ValueError: cannot reshape array of size 262144 into shape (32,1024)
Which method is used in your to get the periodicity of the pulsar signal?
While doing a fresh install from the git repo as suggested in docs,
pip install git+https://github.com/thepetabyteproject/your.git
I get the following error:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-req-build-ee84oxd9/setup.py", line 5, in <module>
import your
File "/tmp/pip-req-build-ee84oxd9/your/__init__.py", line 1, in <module>
from your.writer import *
File "/tmp/pip-req-build-ee84oxd9/your/writer.py", line 9, in <module>
from rich.progress import Progress
ModuleNotFoundError: No module named 'rich'
----------------------------------------
The easy solution is to add more info to the installation instructions (e.g. to use the requirements.txt). A long-term fix can be to define the version either in setup.py or in a separate configuration file.
This issue is part of my review for JOSS.
I am processing AO327 data which is in a 4-bit data format. I used your_writer to convert those fits file to filterbank format so that I can later decimate it using sigproc. But the filterbank files produced by your_writer.py, when run on sigproc for decimation is showing " ERROR: input data to decimate is not in the filterbank format" and as a result I am not able to decimate those files.
To cross check I converted those files from fits to filterbank format using psrfits2fil.py tool of PRESTO. The filterbank files generated using this are accepted by sigproc for decimation and not showing any error.
On looking at the details of the filterbank file produced by psrfits2fil and your_writer.py using readfile command of PRESTO. I found that many values of frequency and other things are mentioned wrong in the filterbank file produced by your_writer.py
To check if the issue is only with the 4-bit data file or not. I did the same process on 16-bit data files and I am getting the same error.
To Reproduce
your can simply go inside /hyrule/data/ao327/4bitdata and run it on any file.
for some 16 bit files you can find it in /hyrule/data/users/satri/16_bit_test/ in the bowser
Expected behavior
your_writer.py must convert the files from fits to filterbank format properly so that it show all the details of frequencies correct and the filterbank format is proper and Accepted by sigproc for decimation.
Versions (please provide the versions of the the following packages):
Additional context
Let me know if any other information is required from my side.
to_fil
in Writer
currently overwrites the fil file, if a file with outname is already present. This can lead to a lot of confusion. A solution for this would be useful. Could either be a warning/Error or a bit different way to write fil files.
It would be useful to add in instructions on how to add new formats to the reader and writer. It would first require a somewhat significant change in the Your
class to get rid of isfil
and isfits
attributes, and manage it in a cleaner way.
Describe the bug
I am using YOUR to read in and process Arecibo 4 bit data, On using your_header.py I am getting wrong values of fc1 (lowest frequency), Bandwidth and channel width. After having a look at the frequency vs time plot of one file using your_h5plotter.py , I found that the bandpass is being read in wrong. We can see the correct values by simplly using readfile command of presto.
Expected behavior
I am using YOUR to run heimdall, psrdada,fetch etc to recover pulsars. I ran YOUR on some known pulsar file and I could recover those pulsars as well, but the DM value found in the plot is very much deflected from the real value of the pulsar. In one specific pulsar case the real DM is 50 but using YOUR I am getting DM around 34 ( B0525+21 pulsar.). I am expecting all values to be read in correctly and as a result the DM also must be closer to the actual value.
Screenshots
Here you can compare the values produced my your_header.py and readfile (presto), most of the values are being read in wrong here by your_header.py, which eventually is affecting my results. You Can see in the final plot that all values are wrong and the DM is also deflected alot from the real value. The screenshot are of a data file of B2110+27 pulsar.
The correct DM of this pulsar is 25.11 but from the plot we can see the value we are getting is too much deflected.
Versions (please provide the versions of the the following packages):
These are versions of packages in the specific conda environment I am using.
Additional context
I suspect that it is specific to 4 bit data files
Hi,
I wanted to convert a two column ascii file into psrfits format. You have mentioned how to write a filter file from scratch. Is the process similar or I need do something else?
After conversion, I want to run psrchive commands on the psrfits file. So I also need to know how to add parameters to the file while I convert it.
Let me know how this issue can be solved...
Clear Skies,
Pranav
Hello there,
It seems as if the centre frequency reported by your_header is incorrect for filterbank files.
First some outputs of 'readfile' and 'digihdr' (what should be correct):
mark@ebur:/scratch2/mark/sgrl58$ readfile sgrl58_wb_no0025_IFall_s1_vdif_pol2.fil
Assuming the data is a SIGPROC filterbank file.
1: From the SIGPROC filterbank file 'sgrl58_wb_no0025_IFall_s1_vdif_pol2.fil':
Telescope = Fake
Source Name = B1933+16
Obs Date String = 2020-08-19T19:53:14
MJD start time = 59080.82863425926189
RA J2000 = 19:35:47.8000
RA J2000 (deg) = 293.949166666667
Dec J2000 = 16:16:39.9000
Dec J2000 (deg) = 16.27775
Tracking? = True
Azimuth (deg) = 0
Zenith Ang (deg) = 0
Number of polns = 2 (summed)
Sample time (us) = 256
Central freq (MHz) = 1323.49
Low channel (MHz) = 1259.5525
High channel (MHz) = 1387.4275
Channel width (MHz) = 0.125
Number of channels = 1024
Total Bandwidth (MHz) = 128
Beam = 0 of 1
Beam FWHM (deg) = 1.000
Spectra per subint = 2400
Starting subint = 1661206552
Subints per file = 0
Spectra per file = 2328112
Time per subint (sec) = 0.6144
Time per file (sec) = 595.996672
bits per sample = 2
bytes per spectra = 256
samples per spectra = 1024
bytes per subint = 614400
samples per subint = 2457600
zero offset = 0
Invert the band? = False
bytes in file header = 349
and
mark@ebur:/scratch2/mark/sgrl58$ digihdr sgrl58_wb_no0025_IFall_s1_vdif_pol2.fil
data file sgrl58_wb_no0025_IFall_s1_vdif_pol2.fil opened
ndat Number of time samples 2328112
nchan Number of frequency channels 1024
npol Number of polarizations 1
ndim Number of data dimensions 1
nbit Number of bits per datum 2
type Observation type Pulsar
site Telescope name Fake
name Source name B1933+16
coord Source coordinates 19:35:47.800+16:16:39.90
freq Centre frequency (MHz) 1323.49
bw Bandwidth (MHz) -128
dm Dispersion measure (pc/cm^3) 0
rm Rotation measure (rad/m^2) 0
scale Data units 1
state Data state Intensity
machine Name of machine/backend FAKE
And now the output of your_header:
(yourenv) mark@ebur:/scratch2/mark/sgrl58$ python --version
Python 3.9.2
(yourenv) mark@ebur:/scratch2/mark/sgrl58$ python
Python 3.9.2 (default, Mar 3 2021, 20:02:32)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import your
>>> your.__version__
'0.6.3'
>>> myobj = your.Your("sgrl58_wb_no0025_IFall_s1_vdif_pol2.fil")
>>> myobj.your_header
Unified Header:
"basename": "sgrl58_wb_no0025_IFall_s1_vdif_pol2"
"bw": -128.0
"center_freq": 1323.4275
"dec_deg": 16.277749999999997
"dtype": "uint8"
"fch1": 1387.4275
"filelist": [
"sgrl58_wb_no0025_IFall_s1_vdif_pol2.fil"
]
"filename": "sgrl58_wb_no0025_IFall_s1_vdif_pol2.fil"
"foff": -0.125
"format": "fil"
"frequency_decimation_factor": 1
"gb": -2.09253356015966
"gl": -127.56414485356551
"native_foff": -0.125
"native_nchans": 1024
"native_nspectra": 2328112.0
"native_tsamp": 0.000256
"nbits": 2
"nchans": 1024
"npol": 1
"nspectra": 2328112
"ra_deg": 293.94916666666666
"source_name": "B1933+16"
"time_decimation_factor": 1
"tsamp": 0.000256
"tstart": 59080.82863425926
"tstart_utc": "2020-08-19T19:53:14.000"
I think this is because of the following line:
self.cfreq = self.fch1 + self.bw / 2
(line 91 of /your/formats/pysigproc.py).
The definition of the fch1 of sigproc filterbank files is: "fch1 (double): centre frequency (MHz) of first filterbank channel" (page 4 of http://sigproc.sourceforge.net/sigproc.pdf). Therefore, the cfreq is off by half a channel width.
I hope this clarifies things! Thank you for making your, it makes it much easier to play with data.
During a0bcf90, I created a bug, only one of the filters is being used right now.
Describe the bug
printing your_header gives a type error
To Reproduce
Steps to reproduce the behavior:
your_object = Your("/hyrule/data/users/dagarwal/ALFABURST_CAND/S1/pdevs1/p3231.20190504.B1855+09.b0s0g0.00000.fits")
print(your_object.your_header)
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-48-e1e51a42fd26> in <module>
----> 1 print(your_object.your_header)
~/soft/conda/envs/numba_gpu/lib/python3.6/site-packages/your-0.4.9-py3.6.egg/your/your.py in __str__(self)
419 def __str__(self):
420 hdr = vars(self)
--> 421 return json.dumps(hdr, indent=2)
~/soft/conda/envs/numba_gpu/lib/python3.6/json/__init__.py in dumps(obj, skipkeys, ensure_ascii, check_circular, allow_nan, cls, indent, separators, default, sort_keys, **kw)
236 check_circular=check_circular, allow_nan=allow_nan, indent=indent,
237 separators=separators, default=default, sort_keys=sort_keys,
--> 238 **kw).encode(obj)
239
240
~/soft/conda/envs/numba_gpu/lib/python3.6/json/encoder.py in encode(self, o)
199 chunks = self.iterencode(o, _one_shot=True)
200 if not isinstance(chunks, (list, tuple)):
--> 201 chunks = list(chunks)
202 return ''.join(chunks)
203
~/soft/conda/envs/numba_gpu/lib/python3.6/json/encoder.py in _iterencode(o, _current_indent_level)
428 yield from _iterencode_list(o, _current_indent_level)
429 elif isinstance(o, dict):
--> 430 yield from _iterencode_dict(o, _current_indent_level)
431 else:
432 if markers is not None:
~/soft/conda/envs/numba_gpu/lib/python3.6/json/encoder.py in _iterencode_dict(dct, _current_indent_level)
402 else:
403 chunks = _iterencode(value, _current_indent_level)
--> 404 yield from chunks
405 if newline_indent is not None:
406 _current_indent_level -= 1
~/soft/conda/envs/numba_gpu/lib/python3.6/json/encoder.py in _iterencode(o, _current_indent_level)
435 raise ValueError("Circular reference detected")
436 markers[markerid] = o
--> 437 o = _default(o)
438 yield from _iterencode(o, _current_indent_level)
439 if markers is not None:
~/soft/conda/envs/numba_gpu/lib/python3.6/json/encoder.py in default(self, o)
178 """
179 raise TypeError("Object of type '%s' is not JSON serializable" %
--> 180 o.__class__.__name__)
181
182 def encode(self, o):
TypeError: Object of type 'type' is not JSON serializable
Expected behavior
prints your_header
Add details for RFI and Candidate modules.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.