Memory corruption when using custom grid about kdepy HOT 11 CLOSED

Thanks for your response, this will work for me.

Knowing that the issues lies with data outside the grid alone helps me enough to avoid the problem. :)

Thank you as well for working on a fix!

from kdepy.

Further support that the issues lies within Cython:

If I turn off Cython support by setting:

KDEpy.binning._use_Cython = False

The code above does not crash anymore.

from kdepy.

Hi @NotSpecial , thanks for reporting this! Quick answer since I'm on my phone. This is likely due to data being outside of the grid. E.g. if you use linspace from 2 to 8, and a data point has value 9. Obviously the software should either just warn about the boundary bias, or raise an error - it should not crash. I hope this has not frustrated you too much, and i will fix it when i get the chance.

Could you check if this is the case? Leaviing the issue open until i fix it.

from kdepy.

Hi, and thanks for your response, I finally had time to investigate further now.

You are right, the issue seems to be data points outside the grid. If I select a grid covering the whole range of data (e.g. np.linspace(-705, 705) for the example above), it does not crash anymore.

from kdepy.

@NotSpecial great. I'll leave this open and fix the issue. Currently, I would recommend that you filter out some data prior to running FFTKDE, and then filter the rest afterwards. I haven't run the following code, but I hope it demonstrates what I mean.

# Wish to examine data between 0 and 2.
# First, we use a coarse grid, as this reduces potential boundary bias.
data = np.array(data)
data_filtered = data[(data > -2 ) & ( data < 4)] 

# Prepare the grid
grid = np.linspace(-2, 4, num=2**10)
y = FFTKDE(bw=0.5).fit(data).evaluate()

# Filter the estimate
mask = (grid > 0) & ( grid < 2)
y = y[mask] 
grid = grid[mask] 

plt.plot(grid, y)
``

from kdepy.

Should be fixed now. Closing this Issue. Thanks again @NotSpecial .

from kdepy.

Thank you! :)

from kdepy.

Hi,

I have a little followup question, if you have some time:

Here's where you now check whether the data is inside the grid.
Is there a reason why the grid must be strictly bigger than the data points?
E.g. why is max_grid == max_data not allowed?

I was wondering if a <= check (and >= respectively) would also work or if there is
more to it than I know.

Thank you in advance for your time! :)

from kdepy.

Hi again @NotSpecial,

I don't remember at the top of my head. I can check when I get some time :)

from kdepy.

Hi,

I have a little followup question, if you have some time:

Here's where you now check whether the data is inside the grid. Is there a reason why the grid must be strictly bigger than the data points? E.g. why is max_grid == max_data not allowed?

I was wondering if a <= check (and >= respectively) would also work or if there is more to it than I know.

Thank you in advance for your time! :)

Hi again @NotSpecial,

I don't remember at the top of my head. I can check when I get some time :)

Hi there, would it still not be allowed to have max_grid == max_data (and similar for minima) please?
Because this does not allow me to use the custom grid functionality I think.
I would like to retrieve the KDE-value for each value of the data input.

from KDEpy import FFTKDE
data = np.random.rand(100)
kde = FFTKDE(bw='silverman').fit(data).evaluate(np.sort(data))

File ... 155, in FFTKDE.evaluate(self, grid_points)
    153 max_data = np.max(self.data, axis=0)
    154 if not ((min_grid < min_data).all() and (max_grid > max_data).all()):
  >155     raise ValueError("Every data point must be inside of the grid.")
    156 # Step 1 - Obtaining the grid counts
    157 # TODO: Consider moving this to the fitting phase instead
    158 data = linear_binning(self.data, grid_points=self.grid_points, weights=self.weights)

ValueError: Every data point must be inside of the grid.

from kdepy.

Grid info and data are not the same. Let KDEpy set up its own grid, then inspect the grid - you will see its structure. Also, the grid needs to be equidistant. And the finer the grid, the better the performance (not just on the grid points you are interested in, but over all - data is sampled onto the grid, so a very coarse grid "ruins" the data by sampling it coarsely).

I would like to retrieve the KDE-value for each value of the data input.

I would let KDEPy use a default grid, then sample the results onto your data afterwards. See Fast evaluation on a non-equidistant grid in the examples.

from kdepy.