In to_fits in Writer, it would be useful to add in the option to select a range of channels and spectra to write. This is already there in to_fil (inputs: c, nstart, nsamp)

your/utils/rfi.py:77: DeprecationWarning: median_absolute_deviation is deprecated, use median_abs_deviation instead!

To preserve the existing default behavior, use
scipy.stats.median_abs_deviation(..., scale=1/1.4826).
The value 1.4826 is not numerically precise for scaling
with a normal distribution. For a numerically precise value, use
scipy.stats.median_abs_deviation(..., scale='normal').

std = 1.4826 * stats.median_absolute_deviation(sk_c)

https://app.lucidchart.com/documents/view/b60d135f-ba86-4c5f-a159-a7e6601ec24e/0_0

it would be useful if your_viewer can optionally dedisperse and RFI flag the data as well.

Fix the telescope name, and other such header parameters written out by Writer

Some small issues and typos (with suggestions) I found in the documentation:

• update the hyperlinks (e.g., https://github.com/devanshkv/your/issues). I guess this is due to the transfer of repo ownership.

Getting Started

• The docs do not explicitly state the target audience as required for a statement of need. I will suggest adding one or two sentences in the README explaining use for a general audience. Also, hyperlinking sigproc, psrfits, etc to their proper references might be a good idea.

• filterbanks -> filterbank, go -> convert

Contributing Guide

• May be highlight or italicize your to differentiate from your. appropritate -> appropriate
• making -> make (in code of conduct)

Reading files

• the a -> a, the filterbanks -> a filterbank file
• highlight your
• fits -> psrfits, fil -> filterbank, as as -> as

Candidates

• some context re how to obtain these candidate parameters (e.g., Heimdall) would be great. Since this is also an important part of your, it can also be added to the statement of need in README.
• candiate -> candidate, paramters -> parameters, approimate -> approximate, -SNR -> SNR, DM Time -> DM-Time
• this will take data from the filterbank file -> get_chunk() will extract data from the source file.
• frequency-time -> frequency-time data, Single pulse search pipelines lines -> Single-pulse search pipelines
• from candidate import crop results in ModuleNotFoundError
• h5 files -> HDF format file

RFI Mitigation

• mitigation -> RFI Mitigation, usefull -> useful, filterbank filterbank -> filterbank
• the PSR B0329+54 single-pulse data is not part of the package. Maybe including the data file as a test case will be helpful to the users.

Command Line Interface

• fits -> psrfits, your header -> either highlight your or just drop it in this case.
• your_heimdall.py description is not very clear.
• I will suggest to add some more description to these cmd line tools.

API - Your

• exactly device the -> divide the. This note needs more clarification.
• time_decimation_factor -> two different definitions used (decimate in time with this factor and Number of time samples to average). I like the second one. Same goes for frequency_decimation_factor
• pre decimation -> pre-decimation

This issue is part of my JOSS review.

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

When a title is not defined in output, it chooses .fil as the name.

Describe the solution you'd like
A clear and concise description of what you want to happen.

I would like a default name, whether chosen based on the original filenames, or just a default name like combined.fits, when no name is present in output definition.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

I've considered leaving it as is and leaving the naming to the user, however, it could be a good improvement for users that might be using the program for intermediate file products for a quick check of the data.

Additional context
Add any other context or screenshots about the feature request here.

Which method is used in your to get the periodicity of the pulsar signal?

It would be useful to add in instructions on how to add new formats to the reader and writer. It would first require a somewhat significant change in the Your class to get rid of isfil and isfits attributes, and manage it in a cleaner way.

A filterbank file containing custom keywords crashes the reader.

Consider, e.g., https://data.caltech.edu/records/1647:

We note that we use a custom filterbank header that includes the following keys in addition to the standard filterbank header keys:
MJD_hour - The hour (in MJD-7hr) that the (typically 5 hr long) observing track started.
...
start_sample - The sample of the first time sample contained in the filterbank file.

When trying to read this filterbank file with YOUR:
import your
fil_file = 'candidate_ovro_20200428.fil'
dobj = your.Your(fil_file)

KeyError: 'start_sample'

Expected behavior: the reader should provide a warning and continue reading the file.
e.g.,

"Unknown keyword: start_sample"

It's not ideal, but crashing with KeyError is worse.

Describe the bug
printing your_header gives a type error

To Reproduce
Steps to reproduce the behavior:

your_object = Your("/hyrule/data/users/dagarwal/ALFABURST_CAND/S1/pdevs1/p3231.20190504.B1855+09.b0s0g0.00000.fits")
print(your_object.your_header)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-48-e1e51a42fd26> in <module>
----> 1 print(your_object.your_header)

~/soft/conda/envs/numba_gpu/lib/python3.6/site-packages/your-0.4.9-py3.6.egg/your/your.py in __str__(self)
    419     def __str__(self):
    420         hdr = vars(self)
--> 421         return json.dumps(hdr, indent=2)

~/soft/conda/envs/numba_gpu/lib/python3.6/json/__init__.py in dumps(obj, skipkeys, ensure_ascii, check_circular, allow_nan, cls, indent, separators, default, sort_keys, **kw)
    236         check_circular=check_circular, allow_nan=allow_nan, indent=indent,
    237         separators=separators, default=default, sort_keys=sort_keys,
--> 238         **kw).encode(obj)
    239 
    240 

~/soft/conda/envs/numba_gpu/lib/python3.6/json/encoder.py in encode(self, o)
    199         chunks = self.iterencode(o, _one_shot=True)
    200         if not isinstance(chunks, (list, tuple)):
--> 201             chunks = list(chunks)
    202         return ''.join(chunks)
    203 

~/soft/conda/envs/numba_gpu/lib/python3.6/json/encoder.py in _iterencode(o, _current_indent_level)
    428             yield from _iterencode_list(o, _current_indent_level)
    429         elif isinstance(o, dict):
--> 430             yield from _iterencode_dict(o, _current_indent_level)
    431         else:
    432             if markers is not None:

~/soft/conda/envs/numba_gpu/lib/python3.6/json/encoder.py in _iterencode_dict(dct, _current_indent_level)
    402                 else:
    403                     chunks = _iterencode(value, _current_indent_level)
--> 404                 yield from chunks
    405         if newline_indent is not None:
    406             _current_indent_level -= 1

~/soft/conda/envs/numba_gpu/lib/python3.6/json/encoder.py in _iterencode(o, _current_indent_level)
    435                     raise ValueError("Circular reference detected")
    436                 markers[markerid] = o
--> 437             o = _default(o)
    438             yield from _iterencode(o, _current_indent_level)
    439             if markers is not None:

~/soft/conda/envs/numba_gpu/lib/python3.6/json/encoder.py in default(self, o)
    178         """
    179         raise TypeError("Object of type '%s' is not JSON serializable" %
--> 180                         o.__class__.__name__)
    181 
    182     def encode(self, o):

TypeError: Object of type 'type' is not JSON serializable

Expected behavior
prints your_header

We should Black our code.

Here is why!

It would be nice to add in option to dedisperse the data and visualize RFI mitigation, given relevant inputs, in your_viewer. Once decimation is implemented, can also add in the option to display decimated data.

can improve tests for candidate, esp for different cases in get_chunk

During a0bcf90, I created a bug, only one of the filters is being used right now.

Describe the bug
Writer doesn't correctly write data from 4 pol fits files.

To Reproduce

>> your_writer.py -f 4_pol_fits_file -t fits -nstart 0 -nsamp 5000 -o ./ -name test
>> your_header.py -f test.fits

This will show that the output file has only 1 pol, but its size would be 4 times larger.

>> your_viewer.py -f test.fits
.
.
.
Traceback (most recent call last):
  File "/home/kshitij/anaconda3/envs/grbfrb/bin/your_viewer.py", line 4, in <module>
    __import__('pkg_resources').run_script('your==0.6.3', 'your_viewer.py')
  File "/home/kshitij/anaconda3/envs/grbfrb/lib/python3.6/site-packages/pkg_resources/__init__.py", line 650, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/home/kshitij/anaconda3/envs/grbfrb/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1446, in run_script
    exec(code, namespace, namespace)
  File "/home/kshitij/anaconda3/envs/grbfrb/lib/python3.6/site-packages/your-0.6.3-py3.6.egg/EGG-INFO/scripts/your_viewer.py", line 375, in <module>
    values.files, values.start, values.gulp, values.chan_std
  File "/home/kshitij/anaconda3/envs/grbfrb/lib/python3.6/site-packages/your-0.6.3-py3.6.egg/EGG-INFO/scripts/your_viewer.py", line 139, in load_file
    self.read_data()
  File "/home/kshitij/anaconda3/envs/grbfrb/lib/python3.6/site-packages/your-0.6.3-py3.6.egg/EGG-INFO/scripts/your_viewer.py", line 270, in read_data
    self.data = self.your_obj.get_data(self.start_samp, self.gulp_size).T
  File "/home/kshitij/anaconda3/envs/grbfrb/lib/python3.6/site-packages/your-0.6.3-py3.6.egg/your/your.py", line 240, in get_data
    data = self.formatclass.get_data(self, nstart, nsamp, pol=pol)[:, 0, :]
  File "/home/kshitij/anaconda3/envs/grbfrb/lib/python3.6/site-packages/your-0.6.3-py3.6.egg/your/formats/psrfits.py", line 392, in get_data
    data.append(self.read_subint(fsub, pol=pol))
  File "/home/kshitij/anaconda3/envs/grbfrb/lib/python3.6/site-packages/your-0.6.3-py3.6.egg/your/formats/psrfits.py", line 267, in read_subint
    data = data.reshape((self.nsamp_per_subint, self.nchan)).astype(np.float32)
ValueError: cannot reshape array of size 81920000 into shape (5000,4096)

This is because it actually wrote 4 pol data to the fits file, but the header info in the fits file is incorrect.

Hi,
I wanted to convert a two column ascii file into psrfits format. You have mentioned how to write a filter file from scratch. Is the process similar or I need do something else?
After conversion, I want to run psrchive commands on the psrfits file. So I also need to know how to add parameters to the file while I convert it.

Let me know how this issue can be solved...

Clear Skies,
Pranav

Describe the bug
Can't read a bunch of fits files together for some reason?

To Reproduce

your_header.py -f /hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145*fits -v
[02/20/21 11:00:17] DEBUG    2021-02-20 11:00:17,100 - __init__ -your.your - DEBUG - Reading the file(s):                                                                                         your.py:70
                             ['/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0001.fits',
                             '/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0002.fits',
                             '/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0003.fits',
                             '/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0004.fits',
                             '/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0005.fits',
                             '/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0006.fits',
                             '/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0007.fits',
                             '/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0008.fits',
                             '/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0009.fits',
                             '/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0010.fits',
                             '/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0011.fits',
                             '/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0012.fits',
                             '/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0013.fits',
                             '/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0014.fits',
                             '/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0015.fits',
                             '/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0016.fits',
                             '/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0017.fits',
                             '/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0018.fits',
                             '/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0019.fits',
                             '/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0020.fits',
                             '/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0021.fits',
                             '/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0022.fits',
                             '/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0023.fits',
                             '/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0024.fits',
                             '/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0025.fits',
                             '/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0026.fits']
                    WARNING  2021-02-20 11:00:17,136 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 1(GBT)!                                   psrfits.py:477
                    WARNING  2021-02-20 11:00:17,154 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 2(GBT)!                                   psrfits.py:477
                    WARNING  2021-02-20 11:00:17,171 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 3(GBT)!                                   psrfits.py:477
                    WARNING  2021-02-20 11:00:17,189 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 4(GBT)!                                   psrfits.py:477
                    WARNING  2021-02-20 11:00:17,206 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 5(GBT)!                                   psrfits.py:477
                    WARNING  2021-02-20 11:00:17,223 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 6(GBT)!                                   psrfits.py:477
                    WARNING  2021-02-20 11:00:17,241 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 7(GBT)!                                   psrfits.py:477
                    WARNING  2021-02-20 11:00:17,258 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 8(GBT)!                                   psrfits.py:477
                    WARNING  2021-02-20 11:00:17,275 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 9(GBT)!                                   psrfits.py:477
                    WARNING  2021-02-20 11:00:17,292 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 10(GBT)!                                  psrfits.py:477
                    WARNING  2021-02-20 11:00:17,309 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 11(GBT)!                                  psrfits.py:477
                    WARNING  2021-02-20 11:00:17,326 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 12(GBT)!                                  psrfits.py:477
                    WARNING  2021-02-20 11:00:17,343 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 13(GBT)!                                  psrfits.py:477
                    WARNING  2021-02-20 11:00:17,360 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 14(GBT)!                                  psrfits.py:477
                    WARNING  2021-02-20 11:00:17,377 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 15(GBT)!                                  psrfits.py:477
                    WARNING  2021-02-20 11:00:17,394 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 16(GBT)!                                  psrfits.py:477
                    WARNING  2021-02-20 11:00:17,411 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 17(GBT)!                                  psrfits.py:477
                    WARNING  2021-02-20 11:00:17,428 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 18(GBT)!                                  psrfits.py:477
                    WARNING  2021-02-20 11:00:17,445 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 19(GBT)!                                  psrfits.py:477
                    WARNING  2021-02-20 11:00:17,462 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 20(GBT)!                                  psrfits.py:477
                    WARNING  2021-02-20 11:00:17,479 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 21(GBT)!                                  psrfits.py:477
                    WARNING  2021-02-20 11:00:17,495 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 22(GBT)!                                  psrfits.py:477
                    WARNING  2021-02-20 11:00:17,512 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 23(GBT)!                                  psrfits.py:477
                    WARNING  2021-02-20 11:00:17,529 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 24(GBT)!                                  psrfits.py:477
                    WARNING  2021-02-20 11:00:17,546 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 25(GBT)!                                  psrfits.py:477
Traceback (most recent call last):
  File "/home/dagarwal/soft/conda/envs/your_env/bin/your_header.py", line 104, in <module>
    read_header(values.files, values.no_table)
  File "/home/dagarwal/soft/conda/envs/your_env/bin/your_header.py", line 55, in read_header
    y = Your(f)
  File "/home/dagarwal/soft/conda/envs/your_env/lib/python3.8/site-packages/your/your.py", line 72, in __init__
    self.formatclass.__init__(self, self.your_file)
  File "/home/dagarwal/soft/conda/envs/your_env/lib/python3.8/site-packages/your/formats/psrfits.py", line 112, in __init__
    self.specinfo = SpectraInfo(psrfitslist)
  File "/home/dagarwal/soft/conda/envs/your_env/lib/python3.8/site-packages/your/formats/psrfits.py", line 557, in __init__
    first_subint = subint_hdu.data[0]
  File "/home/dagarwal/.local/lib/python3.8/site-packages/astropy/io/fits/fitsrec.py", line 512, in __getitem__
    out = self.view(np.recarray)[key]
  File "/home/dagarwal/soft/conda/envs/your_env/lib/python3.8/site-packages/numpy/core/records.py", line 519, in __getitem__
    obj = super(recarray, self).__getitem__(indx)
IndexError: index 0 is out of bounds for axis 0 with size 0

Apologies if this is a simple misunderstanding, a lot of the PSRFITS/PSRCHIVE documentation is offline/incomplete making this hard to debug.

I was trying to open the psrchive files from Oostrum et al. 2020, which are in *.ar format by using psrconv to convert to PSRFITS and then using your

The following error gets thrown (ValueError: File 'R1_B01.fits' does not appear to be PSRFITS!) and is due to is_PSRFITS reading the "OBS_MODE" field of the fits file being set to "PSR" instead of "SEARCH". I'm not sure what the difference is, but bypassing this condition to force your to try and read it results in other format errors.

If this is a problem with the data itself or a bug in psrconv feel free to close.
Is there some other way to read *.ar files with your?

Candidate tests seem to be failing, maybe because of low tolerance in the comparison.
Need to fix that.

Describe the bug
Zero DM subtraction is leading to overflow when the data is to be written in integer format.

To Reproduce
your_writer.py -f 28.fits -t fil -r -zero_dm_subt

Expected behavior
It should just create a fil file with zero DM subtracted data. But that would not be possible if the output data format is integer. After 0 DM subtraction mean will be 0, so some data will be negative, and that overflows when typecasted into unsigned integer.

Screenshots
After 0 DM subtraction:

After typecasting to unsigned int:

Describe the bug

your_writer.py gives a type error.

To Reproduce
Steps to reproduce the behavior:
your_writer.py -v -f *fits -t fil -r -sksig 4 -sgsig 4 -sgfw 15

Screenshots

Should we consider moving to another CI service like Circle or Travis? Github Actions keep breaking.
I ran the action associated with the commit 136e2cd in a brand new conda env, and it works just fine.

Hello there,

It seems as if the centre frequency reported by your_header is incorrect for filterbank files.

First some outputs of 'readfile' and 'digihdr' (what should be correct):

mark@ebur:/scratch2/mark/sgrl58$ readfile sgrl58_wb_no0025_IFall_s1_vdif_pol2.fil 
Assuming the data is a SIGPROC filterbank file.

1: From the SIGPROC filterbank file 'sgrl58_wb_no0025_IFall_s1_vdif_pol2.fil':
                  Telescope = Fake
                Source Name = B1933+16
            Obs Date String = 2020-08-19T19:53:14
             MJD start time = 59080.82863425926189
                   RA J2000 = 19:35:47.8000
             RA J2000 (deg) = 293.949166666667 
                  Dec J2000 = 16:16:39.9000
            Dec J2000 (deg) = 16.27775         
                  Tracking? = True
              Azimuth (deg) = 0
           Zenith Ang (deg) = 0
            Number of polns = 2 (summed)
           Sample time (us) = 256              
         Central freq (MHz) = 1323.49          
          Low channel (MHz) = 1259.5525        
         High channel (MHz) = 1387.4275        
        Channel width (MHz) = 0.125            
         Number of channels = 1024
      Total Bandwidth (MHz) = 128              
                       Beam = 0 of 1
            Beam FWHM (deg) = 1.000
         Spectra per subint = 2400
            Starting subint = 1661206552
           Subints per file = 0
           Spectra per file = 2328112
      Time per subint (sec) = 0.6144
        Time per file (sec) = 595.996672
            bits per sample = 2
          bytes per spectra = 256
        samples per spectra = 1024
           bytes per subint = 614400
         samples per subint = 2457600
                zero offset = 0                
           Invert the band? = False
       bytes in file header = 349

and

mark@ebur:/scratch2/mark/sgrl58$ digihdr sgrl58_wb_no0025_IFall_s1_vdif_pol2.fil 
data file sgrl58_wb_no0025_IFall_s1_vdif_pol2.fil opened
ndat             Number of time samples         2328112
nchan            Number of frequency channels   1024
npol             Number of polarizations        1
ndim             Number of data dimensions      1
nbit             Number of bits per datum       2
type             Observation type               Pulsar
site             Telescope name                 Fake
name             Source name                    B1933+16
coord            Source coordinates             19:35:47.800+16:16:39.90
freq             Centre frequency (MHz)         1323.49
bw               Bandwidth (MHz)                -128
dm               Dispersion measure (pc/cm^3)   0
rm               Rotation measure (rad/m^2)     0
scale            Data units                     1
state            Data state                     Intensity
machine          Name of machine/backend        FAKE

And now the output of your_header:

(yourenv) mark@ebur:/scratch2/mark/sgrl58$ python --version
Python 3.9.2
(yourenv) mark@ebur:/scratch2/mark/sgrl58$ python
Python 3.9.2 (default, Mar  3 2021, 20:02:32) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import your
>>> your.__version__
'0.6.3'
>>> myobj = your.Your("sgrl58_wb_no0025_IFall_s1_vdif_pol2.fil")
>>> myobj.your_header
Unified Header:
  "basename": "sgrl58_wb_no0025_IFall_s1_vdif_pol2"
  "bw": -128.0
  "center_freq": 1323.4275
  "dec_deg": 16.277749999999997
  "dtype": "uint8"
  "fch1": 1387.4275
  "filelist": [
    "sgrl58_wb_no0025_IFall_s1_vdif_pol2.fil"
  ]
  "filename": "sgrl58_wb_no0025_IFall_s1_vdif_pol2.fil"
  "foff": -0.125
  "format": "fil"
  "frequency_decimation_factor": 1
  "gb": -2.09253356015966
  "gl": -127.56414485356551
  "native_foff": -0.125
  "native_nchans": 1024
  "native_nspectra": 2328112.0
  "native_tsamp": 0.000256
  "nbits": 2
  "nchans": 1024
  "npol": 1
  "nspectra": 2328112
  "ra_deg": 293.94916666666666
  "source_name": "B1933+16"
  "time_decimation_factor": 1
  "tsamp": 0.000256
  "tstart": 59080.82863425926
  "tstart_utc": "2020-08-19T19:53:14.000"

I think this is because of the following line:
self.cfreq = self.fch1 + self.bw / 2 (line 91 of /your/formats/pysigproc.py).
The definition of the fch1 of sigproc filterbank files is: "fch1 (double): centre frequency (MHz) of first filterbank channel" (page 4 of http://sigproc.sourceforge.net/sigproc.pdf). Therefore, the cfreq is off by half a channel width.

I hope this clarifies things! Thank you for making your, it makes it much easier to play with data.

Bandpass fitter tests are failing for some reason. Here is the link.

I am processing AO327 data which is in a 4-bit data format. I used your_writer to convert those fits file to filterbank format so that I can later decimate it using sigproc. But the filterbank files produced by your_writer.py, when run on sigproc for decimation is showing " ERROR: input data to decimate is not in the filterbank format" and as a result I am not able to decimate those files.

To cross check I converted those files from fits to filterbank format using psrfits2fil.py tool of PRESTO. The filterbank files generated using this are accepted by sigproc for decimation and not showing any error.

On looking at the details of the filterbank file produced by psrfits2fil and your_writer.py using readfile command of PRESTO. I found that many values of frequency and other things are mentioned wrong in the filterbank file produced by your_writer.py

To check if the issue is only with the 4-bit data file or not. I did the same process on 16-bit data files and I am getting the same error.

To Reproduce
your can simply go inside /hyrule/data/ao327/4bitdata and run it on any file.
for some 16 bit files you can find it in /hyrule/data/users/satri/16_bit_test/ in the bowser

Expected behavior
your_writer.py must convert the files from fits to filterbank format properly so that it show all the details of frequencies correct and the filterbank format is proper and Accepted by sigproc for decimation.

Screenshots

Versions (please provide the versions of the the following packages):

• your -0.6.6

Additional context
Let me know if any other information is required from my side.

Describe the bug
your_writer.py gives a value error.

To Reproduce
Steps to reproduce the behavior:

your_writer.py -f *fits -t fil -o ./cands -r -sksig 4 -sgsig 4 -sgfw 15 -zero_dm_subt 

Traceback (most recent call last):
  File "/home/rat0022/anaconda3/envs/new/bin/your_writer.py", line 4, in <module>
    __import__('pkg_resources').run_script('your==0.5.1', 'your_writer.py')
  File "/home/rat0022/anaconda3/envs/new/lib/python3.6/site-packages/pkg_resources/__init__.py", line 667, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/home/rat0022/anaconda3/envs/new/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1464, in run_script
    exec(code, namespace, namespace)
  File "/home/rat0022/anaconda3/envs/new/lib/python3.6/site-packages/your-0.5.1-py3.6.egg/EGG-INFO/scripts/your_writer.py", line 70, in <module>
    w.to_fil()
  File "/home/rat0022/anaconda3/envs/new/lib/python3.6/site-packages/your-0.5.1-py3.6.egg/your/writer.py", line 167, in to_fil
    if nsamps % interval != 0:
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

Add a tool to visualize the data and some basic statistics from the command line. Could optionally show bandpass and time series of the relevant chunk. Can also do dedispersion, maybe RFI flagging too?

Describe the bug
your_writer.py writes data in . instead of ./ when no output-dir is given

To Reproduce
your_writer.py -f test.fits -t fil

Expected behaviour
should create test.fil instead of .test.fil

When I run your_heimdall.py, I have memory and cuda issues, and can not get .cand file. Please see the below message.
(psrdada and psrdada-python was already installed.)

your_heimdall.py -dm 560 570 -f *.fits

Created DADA data block with nbufs=8 bufsz=536870912 nread=1
Created DADA header block with nhdrs = 8, hdrsz = 4096 bytes, nread=1
ipcbuf_lock: shmctl (shmid, SHM_LOCK): Cannot allocate memory
Could not lock DADA data block into RAM
Reading... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0% -:--:--Createing PSRDADA client
Connecting to ring buffer
Waiting for next header / data
Reading... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0% -:--:--processing beam 1
Reading... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0% -:--:--Could not setCudaDevice to 0: forward compatibility was attempted on non supported HW
ERROR: Pipeline creation failed
Unknown error. Please contact the author(s).
Reading... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00

Could you tell me how to fix these errors?

Versions

• your '0.6.7'

The submitted draft looks great! Few comments:

• fast radio transients like pulsars -> I do not think pulsars should be classified as transients; maybe add giant pulses or single-pulse from pulsars
• "streamline the pipeline" -> search pipeline?
• "candidate events:" -> candidate events,
• Fix citation for Astropy (missing @ in astropy:2018)

Other suggestions:

• citation for Heimdall -> I prefer B. R. Barsdell thesis as the more appropriate citation for Heimdall than the usually cited reference in the literature (which has more to do with dedisp)

This issue is part of my JOSS review.

In some cases your_heimdall either doesn't clear the shared memory when exiting, or gives an error if shared memory is already used by something.

can mention somewhere that this can be used to clear the shared memory:
https://roadha.us/2011/09/kill-all-shared-memory-segments-in-linux.html

Add details for RFI and Candidate modules.

your_viewer.py -f /gbo/20m/57889/Skynet_57889_psr_b1822_09_28142_28500_0002.fits

Leads to

Traceback (most recent call last):
  File "/home/dagarwal/soft/conda/envs/your_env/bin/your_viewer.py", line 4, in <module>
    __import__('pkg_resources').run_script('your==0.5.6', 'your_viewer.py')
  File "/home/dagarwal/soft/conda/envs/your_env/lib/python3.8/site-packages/pkg_resources/__init__.py", line 665, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/home/dagarwal/soft/conda/envs/your_env/lib/python3.8/site-packages/pkg_resources/__init__.py", line 1463, in run_script
    exec(code, namespace, namespace)
  File "/home/dagarwal/soft/conda/envs/your_env/lib/python3.8/site-packages/your-0.5.6-py3.8.egg/EGG-INFO/scripts/your_viewer.py", line 295, in <module>
    app.load_file(values.files, values.start, values.gulp, values.chan_std)  # load file with user params
  File "/home/dagarwal/soft/conda/envs/your_env/lib/python3.8/site-packages/your-0.5.6-py3.8.egg/EGG-INFO/scripts/your_viewer.py", line 125, in load_file
    self.read_data()
  File "/home/dagarwal/soft/conda/envs/your_env/lib/python3.8/site-packages/your-0.5.6-py3.8.egg/EGG-INFO/scripts/your_viewer.py", line 228, in read_data
    self.data = self.your_obj.get_data(self.start_samp, self.gulp_size).T
  File "/home/dagarwal/soft/conda/envs/your_env/lib/python3.8/site-packages/your-0.5.6-py3.8.egg/your/your.py", line 204, in get_data
    data = self.formatclass.get_data(self, nstart, nsamp, pol=pol)[:, 0, :]
  File "/home/dagarwal/soft/conda/envs/your_env/lib/python3.8/site-packages/your-0.5.6-py3.8.egg/your/formats/psrfits.py", line 384, in get_data
    data.append(self.read_subint(fsub, pol=pol))
  File "/home/dagarwal/soft/conda/envs/your_env/lib/python3.8/site-packages/your-0.5.6-py3.8.egg/your/formats/psrfits.py", line 259, in read_subint
    data = data.reshape((self.nsamp_per_subint, self.nchan)).astype(np.float32)
ValueError: cannot reshape array of size 262144 into shape (32,1024)

Dependabot couldn't authenticate with https://pypi.python.org/simple/.

You can provide authentication details in your Dependabot dashboard by clicking into the account menu (in the top right) and selecting 'Config variables'.

View the update logs.

to_fil in Writer currently overwrites the fil file, if a file with outname is already present. This can lead to a lot of confusion. A solution for this would be useful. Could either be a warning/Error or a bit different way to write fil files.

Do we need the fix_empty_psrfits branch?

I think the changes were pushed to main branch with 42e10ff.

Describe the bug
A clear and concise description of what the bug is.

The bug comes from running your_combine_mocks.py normally.
Apparently the nsamp variable variable defined in line 303 is defined as a method instead of a number.

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

This error was produced when running:
your_combine_mocks.py -f1 freq1/* -f2 freq2/*

The original files cannot be included since they are 2 files of 2GB size each
However this problem should be reproducable for any 2 frequency split file.

Expected behavior
A clear and concise description of what you expected to happen.

I expected it to be able to combine both frequency files and not get stuck on line 308

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Smartphone (please complete the following information):

• Device: [e.g. iPhone6]
• OS: [e.g. iOS8.1]
• Browser [e.g. stock browser, safari]
• Version [e.g. 22]

-OS Centos8

Additional context
Add any other context about the problem here.

This should be fixed by changing line 303 from :
nsamp = lowband_obj.nspectra
to:
nsamp = lowband_obj.nspectra()

Need tests for the decimation

It would be useful to add in the option to do subband search in your_heimdall either based on number of subbands or by channel ranges.

When running the plotting functions, this warning pops up:

MatplotlibDeprecationWarning: Support for setting the 'text.latex.preamble' or 
'pgf.preamble' rcParam to a list of strings is deprecated since 3.3 and will be
removed two minor releases later; set it to a single string instead.
    self[key] = other[key]

It would be good to fix this warning. I suppose putting all LaTex commands in a single string will do.
This issue is part of my JOSS review.

While doing a fresh install from the git repo as suggested in docs,
pip install git+https://github.com/thepetabyteproject/your.git
I get the following error:

    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-req-build-ee84oxd9/setup.py", line 5, in <module>
        import your
      File "/tmp/pip-req-build-ee84oxd9/your/__init__.py", line 1, in <module>
        from your.writer import *
      File "/tmp/pip-req-build-ee84oxd9/your/writer.py", line 9, in <module>
        from rich.progress import Progress
    ModuleNotFoundError: No module named 'rich'
    ----------------------------------------

The easy solution is to add more info to the installation instructions (e.g. to use the requirements.txt). A long-term fix can be to define the version either in setup.py or in a separate configuration file.

This issue is part of my review for JOSS.

Describe the bug
I am using YOUR to read in and process Arecibo 4 bit data, On using your_header.py I am getting wrong values of fc1 (lowest frequency), Bandwidth and channel width. After having a look at the frequency vs time plot of one file using your_h5plotter.py , I found that the bandpass is being read in wrong. We can see the correct values by simplly using readfile command of presto.

Expected behavior
I am using YOUR to run heimdall, psrdada,fetch etc to recover pulsars. I ran YOUR on some known pulsar file and I could recover those pulsars as well, but the DM value found in the plot is very much deflected from the real value of the pulsar. In one specific pulsar case the real DM is 50 but using YOUR I am getting DM around 34 ( B0525+21 pulsar.). I am expecting all values to be read in correctly and as a result the DM also must be closer to the actual value.

Screenshots

Here you can compare the values produced my your_header.py and readfile (presto), most of the values are being read in wrong here by your_header.py, which eventually is affecting my results. You Can see in the final plot that all values are wrong and the DM is also deflected alot from the real value. The screenshot are of a data file of B2110+27 pulsar.
The correct DM of this pulsar is 25.11 but from the plot we can see the value we are getting is too much deflected.

Versions (please provide the versions of the the following packages):
These are versions of packages in the specific conda environment I am using.

• your 0.6.5
• matplotlib 3.3.4
• numpy 1.19.5
• h5py 1.10.6
• scikit-image 0.17.2
• scipy 1.5.4
• numba 0.53.1
• astropy 4.1
• pandas 1.1.5
• rich 10.4.0

Additional context
I suspect that it is specific to 4 bit data files

Describe the bug

cands.csv is not properly parsed using pandas.read_csv in the line: cand_pars = pd.read_csv(values.cand_param_file)

The first entry is treated as a header and ignored. Something like this could be better:

cand_pars = pd.read_csv(values.cand_param_file, names=["file", "snr", "stime", "dm", "width", "label"])

The keys "file", "snr", "width", "dm", "label", "stime", "chan_mask_path", and "num_files" are undefined otherwise. This creates problems later on in the code.

See:

process_list.append(
            [row['file'], row['snr'], 2 ** row['width'], row['dm'], row['label'], row['stime'],
             row['chan_mask_path'], row['num_files'], values, gpu_id])

To Reproduce

Traceback (most recent call last):
  File "/home/pearlman/miniconda3/envs/fetch/lib/python3.7/site-packages/pandas/core/indexes/base.py", line 4736, in get_value
    return libindex.get_value_box(s, key)
  File "pandas/_libs/index.pyx", line 51, in pandas._libs.index.get_value_box
  File "pandas/_libs/index.pyx", line 47, in pandas._libs.index.get_value_at
  File "pandas/_libs/util.pxd", line 98, in pandas._libs.util.get_value_at
  File "pandas/_libs/util.pxd", line 83, in pandas._libs.util.validate_indexer
TypeError: 'str' object cannot be interpreted as an integer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pearlman/.local/bin/your_candmaker.py", line 4, in <module>
    __import__('pkg_resources').run_script('your==0.4.9', 'your_candmaker.py')
  File "/home/pearlman/miniconda3/envs/fetch/lib/python3.7/site-packages/pkg_resources/__init__.py", line 666, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/home/pearlman/miniconda3/envs/fetch/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1453, in run_script
    exec(code, namespace, namespace)
  File "/home/pearlman/.local/lib/python3.7/site-packages/your-0.4.9-py3.7.egg/EGG-INFO/scripts/your_candmaker.py", line 176, in <module>
    [row['file'], row['snr'], 2 ** row['width'], row['dm'], row['label'], row['stime'],
  File "/home/pearlman/miniconda3/envs/fetch/lib/python3.7/site-packages/pandas/core/series.py", line 1068, in __getitem__
    result = self.index.get_value(self, key)
  File "/home/pearlman/miniconda3/envs/fetch/lib/python3.7/site-packages/pandas/core/indexes/base.py", line 4744, in get_value
    raise e1
  File "/home/pearlman/miniconda3/envs/fetch/lib/python3.7/site-packages/pandas/core/indexes/base.py", line 4730, in get_value
    return self._engine.get_value(s, k, tz=getattr(series.dtype, "tz", None))
  File "pandas/_libs/index.pyx", line 80, in pandas._libs.index.IndexEngine.get_value
  File "pandas/_libs/index.pyx", line 88, in pandas._libs.index.IndexEngine.get_value
  File "pandas/_libs/index.pyx", line 131, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 1607, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 1614, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'file'

Expected behavior

your_candmaker.py should parse the cands.csv and generate .h5 files exactly like candmaker.py.

Describe the bug
16 bit data in psrfits can be int16, but is always written as uint16 in Filterbanks. Would be good to add a warning about this.

The file has 66 isub(s). I think this bit of the code needs to be checked.

2019-11-11 17:17:56,463 - to_dada -your.dada - DEBUG - Data read is 6291456, Data step is 32768
2019-11-11 17:17:56,463 - get_data -your - DEBUG - Reading from 32768 samples from sample 6291456
2019-11-11 17:17:56,463 - get_data -your.psrfits - DEBUG - Startsub 384, endsub 386
2019-11-11 17:17:56,463 - get_data -your.psrfits - DEBUG - isub is 384
2019-11-11 17:17:56,464 - get_data -your.psrfits - DEBUG - file id is 1
2019-11-11 17:17:56,464 - get_data -your.psrfits - DEBUG - Reading file: /hyrule/data/grbfrb/P3231D/puppi_58763_J1935+2154_0296_0006.fits
2019-11-11 17:17:56,464 - get_data -your.psrfits - DEBUG - Using: [<astropy.io.fits.hdu.image.PrimaryHDU object at 0x7f4691973b38>, <astropy.io.fits.hdu.table.BinTableHDU object at 0x7f469196d390>]
2019-11-11 17:17:56,464 - read_subint -your.psrfits - WARNING - Polarization is IQUV, just using Stokes I
2019-11-11 17:17:56,515 - get_data -your.psrfits - DEBUG - isub is 385
2019-11-11 17:17:56,515 - get_data -your.psrfits - DEBUG - file id is 1
2019-11-11 17:17:56,515 - get_data -your.psrfits - DEBUG - Reading file: /hyrule/data/grbfrb/P3231D/puppi_58763_J1935+2154_0296_0006.fits
2019-11-11 17:17:56,515 - get_data -your.psrfits - DEBUG - Using: [<astropy.io.fits.hdu.image.PrimaryHDU object at 0x7f4691973b38>, <astropy.io.fits.hdu.table.BinTableHDU object at 0x7f469196d390>]
2019-11-11 17:17:56,515 - read_subint -your.psrfits - WARNING - Polarization is IQUV, just using Stokes I
2019-11-11 17:17:56,571 - get_data -your.psrfits - DEBUG - isub is 386
2019-11-11 17:17:56,571 - get_data -your.psrfits - DEBUG - file id is 1
2019-11-11 17:17:56,571 - get_data -your.psrfits - DEBUG - Reading file: /hyrule/data/grbfrb/P3231D/puppi_58763_J1935+2154_0296_0006.fits
2019-11-11 17:17:56,571 - get_data -your.psrfits - DEBUG - Using: [<astropy.io.fits.hdu.image.PrimaryHDU object at 0x7f4691973b38>, <astropy.io.fits.hdu.table.BinTableHDU object at 0x7f469196d390>]
Process p1:
Traceback (most recent call last):
  File "/home/dagarwal/soft/conda/envs/numba_gpu/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/home/dagarwal/soft/conda/envs/numba_gpu/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/home/dagarwal/soft/conda/envs/numba_gpu/lib/python3.6/site-packages/your-0.4.1-py3.6.egg/your/dada.py", line 114, in to_dada
    data_input = self.your_object.get_data(data_read, self.data_step)
  File "/home/dagarwal/soft/conda/envs/numba_gpu/lib/python3.6/site-packages/your-0.4.1-py3.6.egg/your/__init__.py", line 65, in get_data
    return PsrfitsFile.get_data(self, nstart, nsamp)
  File "/home/dagarwal/soft/conda/envs/numba_gpu/lib/python3.6/site-packages/your-0.4.1-py3.6.egg/your/psrfits.py", line 254, in get_data
    data.append(self.read_subint(int(isub % self.nsubints)))
  File "/home/dagarwal/soft/conda/envs/numba_gpu/lib/python3.6/site-packages/your-0.4.1-py3.6.egg/your/psrfits.py", line 138, in read_subint
    sdata = self.fits['SUBINT'].data[isub]['DATA']
  File "/home/dagarwal/soft/conda/envs/numba_gpu/lib/python3.6/site-packages/astropy/io/fits/fitsrec.py", line 492, in __getitem__
    out = self.view(np.recarray)[key]
  File "/home/dagarwal/soft/conda/envs/numba_gpu/lib/python3.6/site-packages/numpy/core/records.py", line 506, in __getitem__
    obj = super(recarray, self).__getitem__(indx)
IndexError: index 66 is out of bounds for axis 0 with size 66

some functions of dada writer can now be moved to Writer class.