k2sc runs successfully on most of my light curves, but I am finding that for a percent or two of them, it returns an error in the section of gp.py that calls scipy.linalg.cho_factor.

The particular error message is numpy.linalg.LinAlgError: 943-th leading minor of the array is not positive definite
(though the number 943 changes from light curve to light curve), and it causes a fatal crash. I don't understand why this is happening, as the offending light curves do not have any obvious defects.

[suz@glamdring k2sc_sap]$ ~/soft/k2sc/bin/k2plot --show EPIC_247548866_mast.fits
Traceback (most recent call last):
File "/users/suz/soft/k2sc/bin/k2plot", line 102, in
plot_lc(ax2, 1, data)
File "/users/suz/soft/k2sc/bin/k2plot", line 35, in plot_lc
mask = data['mask_ol_%d' % ap].astype(np.bool)
File "/usr/local/shared/python/2.7.6-gcc/lib/python2.7/site-packages/astropy-1.3.3-py2.7-linux-x86_64.egg/astropy/io/fits/fitsrec.py", line 488, in getitem
return self.field(key)
File "/usr/local/shared/python/2.7.6-gcc/lib/python2.7/site-packages/astropy-1.3.3-py2.7-linux-x86_64.egg/astropy/io/fits/fitsrec.py", line 684, in field
column = self.columns[key]
File "/usr/local/shared/python/2.7.6-gcc/lib/python2.7/site-packages/astropy-1.3.3-py2.7-linux-x86_64.egg/astropy/io/fits/column.py", line 1432, in getitem
key = _get_index(self.names, key)
File "/usr/local/shared/python/2.7.6-gcc/lib/python2.7/site-packages/astropy-1.3.3-py2.7-linux-x86_64.egg/astropy/io/fits/column.py", line 1833, in _get_index
raise KeyError("Key '{}' does not exist.".format(key))
KeyError: "Key 'mask_ol_1' does not exist."

In the k2io.py code, there are a few lines at the end which are problematic for 'AMC' format light curve files. The issue is here:
for h in (data.primary_header,hdu.header):
There is no data.primary_header for 'AMC' files, and so the code is unable to add the 'origin', 'program', and 'date' keywords to it, and it crashes. I fixed this by adding a try - except to test whether the data.primary_header exists, but you may wish to fix it some other way.

Hello, I was following the main installation instructions and cloned k2sc off github, and then tried to install with python setup.py install --user

/home/user/anaconda3/lib/python3.5/distutils/dist.py:261: UserWarning: Unknown distribution option: 'install_requires'
  warnings.warn(msg)
running install
running build
running config_cc
unifing config_cc, config, build_clib, build_ext, build commands --compiler options
running config_fc
unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options
running build_py
running build_scripts
error: file '/home/user/K2PL/k2sc/bin/k2plotmast' does not exist

Also, the installation seems to have failed and the k2sc command isn't found.

We've been using k2sc v 1.0 (downloaded from this repository) with george v 0.3.1 to detrend the lightcurves and remove telescope systematics. Unfortunately, we're having trouble replicating the lightcurves provided by your team here:

https://archive.stsci.edu/prepds/k2sc/

I've attached an example -- this is k2 id 210327027:

also known as 2MASS 0355+11, an L3-L6 brown dwarf observed during campaign 4. In the attached plot, the top panel is the raw PDCSAP flux, the middle panel is our attempt to run k2sc on the PDCSAP flux (clearly something off), and the bottom panel is the result from the lightcurves provided by your team (clearly detrended). All lightcurves have been normalised by the median value, and then had 1.0 subtracted from them (so units should be fractional variation from the median).

We installed k2sc following the directions on this github repository, and with the command line:

k2sc ktwo210327027-c04_llc.fits

where ktwo210327027-c04_llc.fits is the lightcurve file downloaded directly from the MAST archive.

I've put both the input and output (EPIC_210327027_mast.fits) files up on Dropbox, along with the script used to plot the lightcurves (plot_lightcurves.py):

https://www.dropbox.com/sh/u3kgy2j7f1jh4i2/AABMSHSti1d2h6k06nNyLf6ka?dl=0

Any insight you might have into what is going on (and what we might have done wrong) would be greatly appreciated! Thanks in advance!

K2sc is looking for two values from sigma_clip, but sigma_clip (in utils.py) now returns three:

This causes the code to completely crash.

Hi folks,

I am trying to use the k2sc code for Kepler K2 data. I've run into a couple of issues for which I hope you can direct me back to the right path.

To get started, I am trying to run the example notebook at:

https://github.com/OxES/k2sc/blob/master/notebooks/lightkurve.ipynb

I've had to make a change for the current version of lightkurve, in that KeplerTargetPixelFile.from.archive is not available, so I switched to search_targetpixelfile, and I specify (cadence='long', mission='K2').

Then I can get a tpf returned and the image looks exactly like the one in the example.

However, the lc.flux plot looks a bit different, which may be ok, but suspicious. Interestingly, when I look at the FITS header in the example, it shows as campaign 6 (correct). The output for the command:

lc.k2sc()

though, prints "Using default splits [2344] for campaign 5. Where did campaign 5 come from, and is that an issue?

Then k2sc iterates for 103 times and eventually completes. When I run the code, lc.k2sc() uses splits for campaign 6 (which seems right), iterates 37 times, and subsequently dies after:

"Starting final outlier detection" in "detrend".

Any suggestions would be great!

George Jacoby
[email protected]

According to the README, I can give K2SC a list of files (I have v0.5). However, when I give it such a list (locations of k2 lightcurves in the form of line-seperated 'ktwo2XXXXXXX-cYY_llc.fits' locations and filenames, it responds with:

Unrecognized input file type for file ...

Some more clarity in what list formats are accepted would be helpful, and/or improved functionality with lists. Cheers!

The way that Python 3 reads in text files is different from Python 2, which means that the header column names get turned into bytes instead of strings, which means that the header check fails. The way I'm writing the ascii files, there is no encoding option. So the way I got this to work again was to change the third line here:

def can_read(cls, fname):
ext_ok = cls.is_extension_valid(fname)
with open(fname, 'r', encoding='utf-8') as f:
header = f.readline().lower().split()
head_ok = all([cn in header for cn in 'dates cadences xpos ypos quality'.split()])
return ext_ok and head_ok

This is running on ubuntu-bionic under WSL with Anaconda3 Python 3.7.3
(base) ljohri@DESKTOP-U671LC6:~/k2sc/bin$ python k2sc 203794605.fits
/home/ljohri/anaconda3/lib/python3.7/importlib/_bootstrap.py:219: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 192 from C header, got 216 from PyObject
return f(*args, **kwds)
/home/ljohri/anaconda3/lib/python3.7/importlib/_bootstrap.py:219: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 192 from C header, got 216 from PyObject
return f(*args, **kwds)
CRITICAL Master: Unrecognized input file type for file 203794605.fits

The documentation says the FLUX column in the output files is the K2SC-detrended flux, but as things stand at the time of creating this issue it is actually a copy of the input flux. We should update the documentation or modify the writer in k2io.py (I vote for the former).

I get the same bug when k2sc tries to define kernels on both Linux and Mac (and with all required packages installed):

Traceback (most recent call last):
  File "./k2sc", line 387, in <module>
    detrend(dataset)
  File "./k2sc", line 119, in detrend
    splits=splits, kernel=BasicKernelEP(),
  File "/home/astro/phrnbe/.local/lib/python2.7/site-packages/k2sc/kernels.py", line 44, in __init__
    self._define_kernel()
  File "/home/astro/phrnbe/.local/lib/python2.7/site-packages/k2sc/kernels.py", line 122, in _define_kernel
    self._k1 = pv[0] * ESK(1/pv[1], ndim=3, dim=0)
TypeError: __init__() got an unexpected keyword argument 'dim'

Hi,

as I can see sigma_clip function has been updated and now if separate_masks is True it returns 3 arrays. However, in standalone.py you only catch two of them, which raises an error. Changing mlow, mhigh = to minf, mlow, mhigh = in line 234 solves the issuse. Additionally minf = isfinite(cflux) in line 232 can be removed as it is calculated by the new sigma_clip.

Cheers,
astrobatty

The way that Python 3 reads in text files is different from Python 2, which means that the header column names get turned into bytes instead of strings, which means that the header check fails. The way I'm writing the ascii files, there is no encoding option. So the way I got this to work again was to change the third line here:

to
with open(fname, 'r', encoding='utf-8') as f:

I've installed k2sc, as well as the supporting packages including george.

I tried running k2sc within /home/k2sc/bin using ./k2sc ../ktwo220186653-c08_llc.fits, but this comes up with the error "ImportError: No module named george", and I'm not sure how to correct this

(george is installed in /home/.local/lib/python3.6/site-packages)

In detrender.py, Detrender, init
self.tr_data = self.data.create_training_set(tr_nrandom, tr_bspan, tr_nblocks)
should be
self.tr_data = self.data.create_training_set(tr_nrandom, tr_nblocks, tr_bspan)

We at Kepler have changed our lightkurve code such that what used to be keplerid is now targetid. That change needs to be propagated through to standalone.py. In particular, self.keplerid should now be self.targetid. Otherwise k2sc crashes. I would change it myself, but I am not sure if there are other places in the code that are similarly affected.

I'm trying to run K2SC (latest version pulled this morning) on SPLOX multi-object LC format. I'm getting the error below. I think it could be something to do with the upgrade to using the astropy version of pyfits that was merged yesterday?

[resselstein@glamdring m67]$ k2sc C05_M67_detrendLC.fits -c 5 --splits 57177 --de-max-time 120 --de-npop 60 --save-dir red_real2 --logfile m67_new
Traceback (most recent call last):
File "/users/resselstein/.local/bin/k2sc", line 41, in
from k2sc.k2io import select_reader, FITSWriter, SPLOXReader
File "/users/resselstein/.local/lib/python2.7/site-packages/k2sc/k2io.py", line 37, in
pf_version = float(re.findall('^([0-9].[0-9]).*', pf.version)[0])
AttributeError: 'module' object has no attribute 'version'

Dear Sir/Mam,

I was using your package for Detrending 1 light curve of K2 campaign 10. but i am continuously getting this error.please help me

Thanks in advance!!!!!!!!!

Regards,
Rishi

THE ERROR IS

INFO Master: Detrending 1 light curves without MPI
INFO Master:
INFO Master: Saving the results to .
INFO Master:
INFO Master: Differential evolution parameters
INFO Master: Population size: 100
INFO Master: Number of iterations: 150
INFO Master: Maximum DE time: 300.00 seconds
INFO Master:
ERROR Worker 0 <229004835>: The campaign not known and no splits given.
Traceback (most recent call last):
File "/usr/local/bin/k2sc", line 417, in
detrend(dataset)
File "/usr/local/bin/k2sc", line 140, in detrend
detrender = Detrender(flux, inputs, mask = mask, splits = splits,
UnboundLocalError: local variable 'splits' referenced before assignment

Would you please give some examples what type of parameters in ASCII file does k2sc support.
I have a '.dat' file that has time, cadence, flux, flux error, xpos, ypos and quality. But I get the errror that this file is not supported. There is no where documented which explains how to use ASCII file how to save the parameters in ASCII file to make ready for running k2sc,

We're trying to include K2SC into a new flare analysis tool AltaiPony and a few more docstrings explaining the input/output parameters, in particular for detrend(dataset, **kwargs) would be help us move forward a lot!

I get the following error message while I am trying to use an ASCII file that is not supported by python3 but supported by python2, and default kernel.

'''''
INFO Worker 0 <248442470>: Starting Lomb-Scargle period search
INFO Worker 0 <248442470>: Using SqrExp position kernel
INFO Worker 0 <248442470>: Found periodicity p = 0.24 (fap 3.9367e-139 < 1e-50), will use a quasiperiodic kernel
INFO Worker 0 <248442470>: Starting global hyperparameter optimisation using DE
Traceback (most recent call last):
File "/anaconda3/envs/py27/bin/k2sc", line 4, in
import('pkg_resources').run_script('K2SC==0.8', 'k2sc')
File "/anaconda3/envs/py27/lib/python2.7/site-packages/pkg_resources/init.py", line 748, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/anaconda3/envs/py27/lib/python2.7/site-packages/pkg_resources/init.py", line 1524, in run_script
exec(script_code, namespace, namespace)
File "/anaconda3/envs/py27/lib/python2.7/site-packages/K2SC-0.8-py2.7.egg/EGG-INFO/scripts/k2sc", line 411, in

File "/anaconda3/envs/py27/lib/python2.7/site-packages/K2SC-0.8-py2.7.egg/EGG-INFO/scripts/k2sc", line 219, in detrend

File "build/bdist.macosx-10.6-x86_64/egg/k2sc/de.py", line 110, in call
File "build/bdist.macosx-10.6-x86_64/egg/k2sc/detrender.py", line 57, in neglnposterior
File "build/bdist.macosx-10.6-x86_64/egg/k2sc/gp.py", line 131, in lnlikelihood
File "build/bdist.macosx-10.6-x86_64/egg/k2sc/gp.py", line 84, in compute
File "/anaconda3/envs/py27/lib/python2.7/site-packages/scipy/linalg/decomp_cholesky.py", line 132, in cho_factor
check_finite=check_finite)
File "/anaconda3/envs/py27/lib/python2.7/site-packages/scipy/linalg/decomp_cholesky.py", line 30, in _cholesky
raise LinAlgError("%d-th leading minor not positive definite" % info)
numpy.linalg.linalg.LinAlgError: 4-th leading minor not positive definite
''''''
I don't get any error if I use basic kernel but this means I am restricting the cases where there are periodic features and quasi-periodic features.

The keyword arguments outlier_sigma and outlier_mwidth are not passed on into detrend()
I think they should go somewhere in here:

Lines 122 and following:

## Iterative sigma-clipping
        ## ------------------------
        print('Starting initial outlier detection')
        omask = mask & sigma_clip(cflux, max_iter=10, max_sigma=5, mexc=mask)
        ofrac = (~omask).sum() / omask.size
        if ofrac < 0.25:
            mask &= omask
            print('  Flagged %i (%4.1f%%) outliers.' % ((~omask).sum(), 100*ofrac))
        else:
            print('  Found %i (%4.1f%%) outliers. Not flagging.' % ((~omask).sum(), 100*ofrac))

Your maintenance work is very much appreciated!