eLWA

This directory contains a collection of utilities for correlating LWA data with the VLA 4-band system.

createConfigFile.py

Given a collection of LWA and VLA data files or a directory containing LWA/VLA data files, create a configuration file for superCorrelator.py. The configuration file is written to standard out and contains:

the phase center,
the file names with full path information,
the file types,
the antenna locations in a LWA-centric reference frame, and
the polarizations

plotConfig.py

Simple script to plot up where all of the antennas are located in a configuration file.

superCorrelator.py

Given a configuration file, correlate LWA and/or VLA according to that configuration.
There are various command line options to control the FFT window size and the correlator dump time. The output of superCorrelator.py is a collection of .npz files, one for integration, that contain the visibility data.

superPulsarCorrelator.py

Similar to superCorrelator.py but implements pular binning using a .polyco file.

plotUVCoverage.py

Script that takes in a collection of .npz files generated by superCorrelator.py and makes a plot of the uv coverage.

plotFringes2.py

Script to generate plots of visibility amplitude and phase as a function of time and frequency.

plotSniffer.py

Script to generated a VLBA sniffer-style plot for all baselines to a specified antenna.

fringeSearch.py

Given a collection of .npz files generated by superCorrelator.py, perform a course fringe search to determine the antenna clock offsets. Prints out the peak delay and fringe rate determined by the search for each baseline examined.

buildIDI.py

Given a collection of .npz files generated by superCorrelator.py, convert the .npz files into one or more FITS IDI files for loading into AIPS.

buildMultiBandIDI.py

Similar to buildIDI.py but useful for LWA-only observations so that both tunings can be packed into a single set of files using. Each tuning is created as a seperate IF.

vdifCheckTimetags.py

Simple script to check the flow of time in a VDIF file. This can be useful for checking for missing frames/polarizations.

vdifFileCheck.py

Simple script to check the power levels in a VDIF file.

vdifSpectra.py

Simple script to generate spectra for each thread in a VDIF file.

vdifWaterfall.py

hdfWaterfall.py-style script to take in a VDIF file and build a HDF5 files containing dynamic spectra. This file can be viewed/interacted with using the plotHDF.py utility in the LSL Commissioning extension.

Correlator MemoryError [OVRO to LWA of NM]

Adding an issue to host work on understanding why superCorrelator.py has a memory issue when attempting to correlate Jupiter Observation (24/03/04) between OVRO and LWA1/LWA-SV for future reference.

Brief Summary:
Observations of Jupiter were taken and attempts to correlate the data along the baselines from NM to OVRO has not yielded any high SNR results. Previous pulsar observations indicated that the timing delay between OVRO and LWA1 was roughly 650-700ms, however this delay did not result in success. Instead we attempted to fit the new data and found a course timing delay of 100ms. Attempts to correlate using a delay around 100ms lead to a MemoryError on startup of correlation, w/ the following error:

NOTE: Set FFT length to 2048 and dump time to 0.250 s per user defined configuration
Antennas:
  Antenna 1: Stand 54, Pol. 0 (100000.000 us offset)
  Antenna 2: Stand 54, Pol. 1 (100000.000 us offset)
  Antenna 3: Stand 51, Pol. 0 (-25.800 us offset)
  Antenna 4: Stand 51, Pol. 1 (-25.800 us offset)
  Antenna 5: Stand 52, Pol. 0 (-28.500 us offset)
  Antenna 6: Stand 52, Pol. 1 (-28.500 us offset)
Skipping forward 10.000 s
-> 1709576999.567359 (2024-03-04 18:29:59.567359)
-> 1709577009.567033 (2024-03-04 18:30:09.567032)
Skipping forward 11.033 s
-> 1709576999.989445 (2024-03-04 18:29:59.989445)
-> 1709577011.022314 (2024-03-04 18:30:11.022313)
Skipping forward 11.349 s
-> 1709576999.673730 (2024-03-04 18:29:59.673729)
-> 1709577011.021322 (2024-03-04 18:30:11.021321)
Shifted beam 1 data by 27848 frames (1.4549 s)
Shifted beam 2 data by 0 frames (0.0000 s)
Shifted beam 2 data by 12 frames (0.0006 s)
Filename: 060373_183000000000299a202
  Type/Reader: lsl.reader.drx
  Date of First Frame: 2024-03-04 18:30:11.022784
  Sample Rate: 19600000 Hz
  Tuning 1: 15999999.992 Hz
  Tuning 2: 31999999.983 Hz
  Bit Depth: 8
Filename: 060373_011247659
  Type/Reader: lsl.reader.drx
  Date of First Frame: 2024-03-04 18:30:11.022313
  Sample Rate: 19600000 Hz
  Tuning 1: 16078125.000 Hz
  Tuning 2: 31390623.996 Hz
  Bit Depth: 8
Filename: 060373_009684440
  Type/Reader: lsl.reader.drx
  Date of First Frame: 2024-03-04 18:30:11.022575
  Sample Rate: 19600000 Hz
  Tuning 1: 16078125.000 Hz
  Tuning 2: 31390623.996 Hz
  Bit Depth: 8
  ===
  Phase Center:
    Name: Jupiter
    RA: moving
    Dec: moving
  ===
  Data Read Time: 1.000 s
  Data Reads in File: 240
 
Processing 0 VDIF and 3 DRX input streams
 
DRX Frames/s: 4785.156250
DRX Frames/Integration: 4785
 
DRX Transform Size: 2048
 
Correlating DRX tuning #2

 Sub-integration time is: 0.010 s
Integration (dump) time is: 0.250 s
 
RR - Read finished in 1.337 s for 1.000s of data
TT - Start [[1709577011, 0.02153061224489796], [1709577011, 0.022312795918367347], [1709577011, 0.022575510204081634]]
TT - Offsets [20480, 5149, 0]
TT - Adjusted [[1709577011, 0.022575510204081634], [1709577011, 0.022575500000000002], [1709577011, 0.022575510204081634]]
TT - Residual ['10.2 ns', '0.0 ns', '10.2 ns']
(6, 19599360) 195993 20480
[1.03153758e-01 1.03153758e-01 2.04200000e-04 2.04200000e-04
 7.86483599e-07 7.86483599e-07]
sigs (6, 195993)
sig inx 6
freq (2048,)
delays1 (6, 2048)
lfft 2048
overlap 1
srate 19600000.0
Traceback (most recent call last):
  File "/home/eLWA/tools/superCorrelator.py", line 843, in <module>
    main(args)
  File "/home/eLWA/tools/superCorrelator.py", line 619, in main
    freqD, feoD, veoD, deoD = multirate.fengine(dataDSub, antennas[2*nVDIFInputs:], LFFT=drxLFFT,
  File "/home/eLWA/tools/multirate.py", line 167, in fengine
    signalsF1, validF1 = _core.FEngine(signals, freq, delays1, LFFT=LFFT, overlap=overlap, sample_rate=sample_rate, clip_level=clip_level, window=window)
MemoryError: Cannot create output array

Following this I attempted to tweak the FFT Length, sub-integration length, and also a shorter dump time. None of these resulted in noticeable change from what I can tell at this point. However, when I tested ranges of input timing delay I found that the program will execute on delays smaller than 6.6ms. Above 7.0ms up to the 100ms value that I was interested in testing all fail in the above manner.

Other clues might be that around 6800us, but below 7000us, of delay, but BELOW 7000us,there is a different error that presents which is the following:

Traceback (most recent call last):
  File "/home/eLWA/tools/superCorrelator.py", line 843, in <module>
    main(args)
  File "/home/eLWA/tools/superCorrelator.py", line 725, in main
    nWin = min([nWin, numpy.argmax(numpy.cumsum(veoD.sum(axis=0)))+1])
  File "<__array_function__ internals>", line 180, in argmax
  File "/usr/local/lib/python3.8/dist-packages/numpy/core/fromnumeric.py", line 1216, in argmax
    return _wrapfunc(a, 'argmax', axis=axis, out=out, **kwds)
  File "/usr/local/lib/python3.8/dist-packages/numpy/core/fromnumeric.py", line 57, in _wrapfunc
    return bound(*args, **kwds)
ValueError: attempt to get argmax of an empty sequence

I'm not really sure what this means, but perhaps this is telling us that there is an empty array floating around at these high values of input delay. I will investigate what veoD looks like from the above traceback and see if there is another pattern there along these steps in delay.

lwa-project / elwa Goto Github PK

elwa's Introduction

eLWA

createConfigFile.py

plotConfig.py

superCorrelator.py

superPulsarCorrelator.py

plotUVCoverage.py

plotFringes2.py

plotSniffer.py

fringeSearch.py

buildIDI.py

buildMultiBandIDI.py

vdifCheckTimetags.py

vdifFileCheck.py

vdifSpectra.py

vdifWaterfall.py

elwa's People

Contributors

Stargazers

Watchers

Forkers

elwa's Issues

Recommend Projects

Recommend Topics

Recommend Org