Describe the bug

RandomFourier does not do what is promises. In the implementation (see below), clearly ϕ should not be equal to r - it should be the phases of 𝓕, not the amplitudes. I don't know when this error snuck in, but it is a quick fix.

function (rf::SurrogateGenerator{<:RandomFourier})()
    inverse, m, 𝓕 = getfield.(Ref(rf.init), (:inverse, :m, :𝓕))
    n = length(𝓕)
    r = abs.(𝓕)
    ϕ = abs.(𝓕)
    if rf.method.phases
        randomised_ϕ = rand(rf.rng, Uniform(0, 2π), n)
        new_𝓕 = r .* exp.(randomised_ϕ .* 1im)
    else
        randomised_r = r .* rand(rf.rng, Uniform(0, 2π), n)
        new_𝓕 = randomised_r .* exp.(ϕ .* 1im)
    end
    return inverse*new_𝓕 .+ m
end

Implement the TFTS and STFTS surrogate methods from Small, M., Yu, D., & Harrison, R. G. (2001). Surrogate Test for Pseudoperiodic Time Series Data. Physical Review Letters, 87(18). doi:10.1103/physrevlett.87.188101 .

The implementations should follow the new API (#26) with a surrogate(x, method::TFTS) and surrogate(x, method:STFTS) with an associated types TFTS and STFTS that both store information about the desired frequency domain fϵ.

See discussion in #24 (@Datseris)

Describe the bug

In the documentation for the stable branch, the multidimensional surrogate example plot is empty.

I'm marking this a good first issue.

Most methods boil down to calling rand or randn, both of which accept an RNG argument.

We can introduce a fourth field to the struct SurrogateGenerator, which by default is GLOBAL_RNG. This field can be then propagated in the rand calls in the

function (rf::SurrogateGenerator{<:AAFT})()

like functions.

The review by Lancaster suggests that when using any Fourier-based surrogates the following preprocessing steps should be made:

1. Remove any trends in the data which are not important or relevant to the results, so that they are not incorporated into the surrogates.
2. Correct any mismatch between start and end points and corresponding first derivatives. This is very important and must not be overlooked, and arises from the assumptions made when calculating any of the Fourier transform based surrogates.
3. Subtract the mean to ensure that results are not affected by significantly different mean values. The

They describe an automated process to ensure this:

We should implement this and offer it as a utility function.

Is your feature request related to a problem? Please describe.

Surrogates are not as fast as they could be due to the allocation of a new surrogate array every time a surrogate generator is called.

Describe the solution you'd like

Implement an extra field s in the SurrogateGenerator struct that will hold the surrogate.

Describe alternatives you've considered

Another method surrogate! could be done, but introduces more complexity for the user, which can be better handled internally.

I think since we already know this information it could be helpful to add it to the docstring. So for each Surrogate method in the docstring we add:

1. What property of the timeseries is conserved.
2. What null hypothesis is the method suitable to test.

(1) is mostly done, most methods have it but some miss it. (2) I think we haven't done at all.

If I use iaaft with a NaN in my time series the result is the sorted original time series.
MWE:

ts = rand(100)
ts[1] = NaN
iaaft(ts) 

Is this expected behaviour?

Dropping an idea, so that I don't forget it:

Currently, the cutoff is defined in terms of the ratio of the low-/high-frequency domain to the whole frequency domain (which is what they do in the Nakamura et al. paper). It would be user-friendly if the frequency threshold for the truncated Fourier surrogates family could be specified using an actual frequency or period.

Maybe define a simple

struct FrequencyCutoff
    fϵ::Union{Real, Tuple{Int, Int}}
end

struct PeriodCutoff
    fϵ::Union{Real, Tuple{Int, Int}}
end

Depending on the context, the user can then decide to filter frequencies above/below some FrequencyCutoff, or filter oscillations that are slower/faster than PeriodCutoff. If the cutoffs are a (low, high) tuple, then frequencies/periods within some range are kept, while the rest are randomized.

These types could be applied to any of the truncated Fourier transform-based surrogate methods (until now, truncated version of RandomFourier and AAFT are implemented in #47, but this will be extended to IAAFT and possibly other variants of the AAFT algorithm such as CAAFT).

Any thoughts?

At the moment TimeseriesSurrogates.jl is not reproducible. That is because we don't allow a specified random number generator from the user side.

I propose to allow this and create the syntax sg = surrogenerator(x, method, rng = Random.global_rng()). This rng object then needs to be utilized for every available method.

@JuliaRegistrator register

Implement the random cascade wavelet surrogate from Paluš, Milan. "Bootstrapping multifractals: Surrogate data from random cascades on wavelet dyadic trees." Physical review letters 101.13 (2008): 134101..

This should be relatively straight-forward to do based on the existing WLS implementation. It is just a matter of shuffling the detail coefficients a bit differently.

For the JOSS paper, we need:

Formal stuff

• Instructions on how to contribute
• Instructions on how to report bugs/errors, or seek support.
• Installation instructions.
• Tests for all surrogate methods.
• License file.

Documentation

• Example usage.

For the paper

• Summary for non-specialized audience.
• Statement of need. What does the software solve and what is the target audience?
• State of the field. How does this package compare to other software providing similar functionality?
• References.
• Sources of funding. KAH: ok, GD: missing.

Hi, Tom from Codecov here.

We noticed that you are using Codecov with fairly high frequency, and we’re so excited to see that! However, because you are not using our app, you may have experienced issues with uploading reports or viewing coverage information. This is due to rate-limiting issues from GitHub.

In order to prevent any future outages, we ask that you move over to our GitHub app integration.

The process is extremely simple and shouldn’t require more than a few clicks, and you should not expect any downtime. By moving to our app, you will no longer need an admin or separate account to manage the relationship with GitHub as the team bot.

Let me know if you have any questions, or if I can help at all with this process.

Implement the pseudoperiodic surrogates (PPS) from Small, M., Yu, D., & Harrison, R. G. (2001). Surrogate Test for Pseudoperiodic Time Series Data. Physical Review Letters, 87(18). doi:10.1103/physrevlett.87.188101.

The method should follow the new API once that is done. That is, surrogate(x, method::PPS) with an associated type PPS with fields τ and d controlling the delay and dimension of the reconstructed embedding. Use genembed from DynamicalSystems for the embedding.

See discussion in #24 (@Datseris)

@JuliaRegistrator register

we should also add a CITATION.bib file and a section Citing with the bibtext entry in the docs!

Originally posted by @Datseris in #131 (comment)

(I'm on the docsbranch, see #39 )

For certain time series

using TimeseriesSurrogates
ts = NSAR2() # create a realization of an NSAR2 process
phases = true

# Embedding parameters need to be specified
d, τ, ρ = 3, 2, 0.05
method = PseudoPeriodic(d, τ, ρ)
s = surrogate(ts, method)

surrplot(ts, s)

gives

@Datseris Any idea why the orbit gets stuck in a small neighborhood? I am just naively picking some random value for ρ here, so it might just be because of my ignorance, but I seem to be able to reproduce this for different values of ρ.

If this is an issue inherent to the algorithm, perhaps this should be mentioned in the PseudoPeriodic docstring, or even discussed on the PseudoPeriodic page.

This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

@Datseris Keeping track of your comment in the rng PR:

Noticed something: x_sorted is not used in TFTS. Is this a mistake...?

I don't remember the details anymore, so have to look into it more closely. I'll ping you once I have an answer for why x_sorted isn't used.

Implement the twin surrogates from Thiel, Marco, et al. "Twin surrogates to test for complex synchronisation." EPL (Europhysics Letters) 75.4 (2006): 535

As an alternative to the truncated Fourier transform surrogates for time series with strong trends (#28 ), implement the AAFT_TD and IAAFT_TD methods in https://journals.aps.org/pre/abstract/10.1103/PhysRevE.85.056202, which rely on detrending instead of truncating the spectrum when shuffling Fourier phases.

I wonder, wouldn't it be great if we allowed more than 1 surrogates to be part of the recipe?

Something like surrogateplot(x, args...) with args several surrogates. We go back to 4 subplots, and the last one has all histograms with alpha.

@kahaaga please provide a better name for LS. This is way too short and rather cryptic. Doesn't convey anything. Also notice that in the documentation of LS there is no mentioning to the name "Lomb Scargle", so I am really not sure how I user would make the connection "LS = LombScargle".

@JuliaRegistrator register

Breakspear et al uses Wavelet surrogates with different randomization schemes. We should add this to the docs.

They also introduce a wavelet surrogate where wavelet coefficients are shuffled using block shuffling, wherein each block the detail coefficients are shuffled. This functionality is not present in our methods at the moment, and should be added.

I can move ARFIMA.jl to JuliaDynamics and instead of re-inventing the wheel and defining models here we could just use it in the docs or just cite it in the docs.

https://github.com/Datseris/ARFIMA.jl

Is your feature request related to a problem? Please describe.

It would be nice to have more surrogate methods that operate on multidimensional input data. This becomes necessary in conditional independence testing for multivariate data, for example in the context of conditional mutual information with high-dimensional marginal spaces.

Currently, we only have the ShuffleDimensions multivariate surrogate, but shuffling the dimensions is not the desired behaviour when, for example, one wants to break temporal associations.

Describe the solution you'd like

The RandomShuffle and BlockShuffle surrogate methods can be straight-forwardly extended to multivariate Datasets (from the StateSpaceSets package) Since these methods just permute the indices of the datasets, they can also be used to shuffle the SVectors of a Dataset.

Implementation strategy

This should be pretty easy to implement. It is just a matter of allocating the proper re-useable storage container in the SurrogateGenerator struct. Instead of enforcing surrogenerator(x::AbstractVector, rf::RandomShuffle, args...), the first argument should be allowed to be any iterable surrogenerator(x, rf::RandomShuffle, args...) and similar(x)/copy(x) should be used to allocate the re-usable container.

Use a single surrogate(x, method::Surrogate) function that dispatches on different surrogate types (e.g. Shuffle, AAFT, IAAFT).

See discussion in #24 (@Datseris)

Is your feature request related to a problem? Please describe.

If the input signal length is not a power of 2, then the signal must be extended/padded in order for the discrete wavelet transform (Wavelets.dwt) to work for RandomCascade surrogates.

Currently, the only option is to pad with zeros at the end of the time series.

Describe the solution you'd like

We should add more padding options, like:

• Linearly extrapolate end points
• Constant propagation of end points,
• etc...

Hi, I'd like to do some contributions to the repo. Is it active / maintained?

@JuliaRegistrator register

No need to import UncertainData.jl unless user actually needs it.

@Datseris I just got the following error message when building the docs locally, in a fresh session. It seems to be related to theme compiling. Maybe this is related to why the docs have been taking > 1.5 hours to run in GitHub actions the past few days.

Have you seen this error before?

 include("docs/make.jl")
ERROR: LoadError: UndefVarError: libsass_so not defined
Stacktrace:
 [1] sass_make_file_context(filename::String)
   @ Sass ~/.julia/packages/Sass/K4Tzw/src/context.jl:16
 [2] compile_file(filename::String; input_path::String, source_map_file::Nothing, kwargs::Base.Pairs{Symbol, String, Tuple{Symbol, Symbol}, NamedTuple{(:output_path, :include_path), Tuple{String, String}}})
   @ Sass ~/.julia/packages/Sass/K4Tzw/src/julian_api.jl:53
 [3] #compile_file#2
   @ ~/.julia/packages/Sass/K4Tzw/src/julian_api.jl:85 [inlined]
 [4] compile(src::String, dst::String)
   @ DocumenterTools.Themes ~/.julia/packages/DocumenterTools/m8nQ0/src/Themes.jl:50
 [5] top-level scope
   @ ~/Code/Repos/TimeseriesSurrogates.jl/docs/make.jl:29
 [6] include(fname::String)
   @ Base.MainInclude ./client.jl:451
 [7] top-level scope
   @ REPL[4]:1
in expression starting at [....]/TimeseriesSurrogates.jl/docs/make.jl:29

Is your feature request related to a problem? Please describe.

Moore et al. (2022) introduces an algorithm to generate surrogate data that obeys the same power-law distribution as the original data. The algorithm takes integer-valued time series as input, and works by prime-factor-decomposing each data point, and then shuffling prime factors in some way (I haven't read the paper in detail).

This looks like a method we should offer.

References

https://journals.aps.org/prx/abstract/10.1103/PhysRevX.12.021056

Section 5.2 of the Lancaster review:

The discrete wavelet transform has also been used as the basis of multifractal surrogates, which aim to enable testing of nonlinear interdependence within, with, between or among time series obtained from multifractal processes [113]. Multifractal surrogates preserve the multifractal properties of input data, i.e. interactions among scales and nonlinear dependence structures [113]. This approach was developed further by Keylock [114] and named iterated amplitude adjusted wavelet transform (IAAWT) surrogates. IAAWT surrogates are designed to preserve both, the multifractal characteristics of the time series and also the distribution of values in the time series using the iterative approach used in IAAFT surrogates. Multifractal surrogates have been used to test cross-scale interactions in atmospheric dynamics [115].
5.3.

Introduce block shuffle surrogates. These surrogates preserves short-term correlations in the time series, but breaks any long-term dynamical information.

Time series are divided into n into blocks. If fixedwidth = true, then all blocks will have the same lengths (or close to, will only happen exactly if the time series length N is divisible by n). If fixedwidth = false, then the time series will be divided into n blocks of random width (but with minimum width m).

Needs some checks to ensure that the desired blocks can be constructed from a given time series.

struct BlockShuffle <: Surrogate 
   n::Int
   fixedwidth::Bool
   m::Int
end

Implement the partial Fourier phase randomization method in Ortega, G. J., & Louis, E. (1998). Smoothness implies determinism in time series: A measure based approach. Physical review letters, 81(20), 4345.

Before releasing 1.4 (I do not think we should count the in-place progress breaking change?) here's the remaining to-dos:

• add changelog for all new methods
• add changelog for the new in-place change
• go through the docs once to be sure everything is a-ok
• Actually increment Project.toml to 1.4. EDIT: We'll go for 2.0.

With the new struct-based api we can gain huge performance benefits by planning the fourier transforms. I do something very similar in SignalDecomposition.jl, see e.g. here: https://github.com/JuliaDynamics/SignalDecomposition.jl/blob/master/src/linear/fourier.jl#L36

In my experience this leads to orders of magnitude performance gains.

Links that point to the docs/travis and general registry need to be changed to start with juliadynamics instead of kahaaga

See discussion in #24

@JuliaRegistrator register

To continue the testing discussion from #12.

To provide better tests we can use the hypothesis testing which the surrogates are supposed to do in the testcases.
The idea would be to have for every surrogate Method one time series which will adhere to the null hypothesis and another time series which would reject the null hypothesis. This way, we would have tested, that the surrogates are doing what they should.
To test the null hypothesis which the surrogate supports we could compare the autocorrelation between the original data and an ensemble of surrogates at least for some of the surrogate methods.

In fact testing is an issue we are like you a bit unsure... We do not know exactly how to do "proper testing", since many of the "hypothesis testing" things that are discussed in the papers are rather subjective and not so much something you could do unit tests on.

What do you see as subjective with the hypothesis testing?

In our docs:

@kahaaga I am assigning you because I think (but not sure), this was a method you contributed...?

According to [1] iaaft and aaft are not working for irregular time series, because these methods are based on the Fourier transform.
They propose an alternative to it using the LombScargle periodogram.
Are surrogates for irregular time series in the scope of this package?
Are there algorithms for surrogates in irregular data?

[1] Testing for nonlinearity in unevenly sampled time series
Andreas Schmitz and Thomas Schreiber
https://journals.aps.org/pre/pdf/10.1103/PhysRevE.59.4044

Is your feature request related to a problem? Please describe.

This is part of the issue described in #89.

BlockShuffle should be modified so that blocks can be shuffled internally. This is necessary to replicate the behavior in Breakspear et al. (2003) where wavelet coefficients for the WLS method are shuffled using block randomization.

Just an issue to keep track of what needs to be in the documentation for the 2.0 release (#39):

• Go back to Documenter.jl (no more mkdocs)
• Each surrogate method needs a statement about what properties of the original signal it preserves.
• Plot recipe should accept multiple surrogates (#40). Not necessary. The user can easily add surrogates to the relevant subplot after calling surroplot.
• Fix transparency of histograms (#40). Need to decide on what to include here.
• Make sure all references are properly cited. It seems that Documenter now lists repeated references at the bottom of the page. This can be solved manually by not using the [^ref] syntax and manually putting the references in relevant places. It would be nice to do this automatically, though. This seems to be a documenter issue, and should not stop the 1.0 release.
• A short note in each method docstring stating whether the surrogate method yields constrained surrogate realizations (some sort of shuffling of the original values) or typical surrogate realizations (model-based, directly or indirectly fit some model to the data, then use that data-informed model to generate surrogates).
• Generate all images/gifs with Documenter instead of manually uploading.
• The plot_and_anim.jl page is not necessary. Reproducing these plots is done by the user in very few lines of code. Should be removed.
• All methods should be briefly described on their own page, including a nice plot for a suitable time series. This is a good place to state caveats and potential issues with any methods (see for example #45). We'll keep the docstrings as the main place for info, examples can go elsewhere.

@Datseris Feel free to add things here if I've forgotten something or tick of when done.

Implement the wavelet based surrogates from Keylock (2006).