Hi @kevinhughes27,

I have a dataset that has 2 million-plus samples. Can I do partial_fit and transform in the same iteration loop as (1) or the partial_fit needs to be called separately before transform as (2)?

(1)

for example in examples:
 ccipca.partial_fit(example)
 ccipca.transform(example)

(2)

for example in examples:
 ccipca.partial_fit(example)

for example in examples:
 ccipca.transform(example)

I made a toy example to test your code, but I guess it is somewhat incorrect. The following is the code that I used.

under ipython

from sklearn.decomposition import PCA
from pyIPCA import CCIPCA, Skocaj_IPCA, Hall_IPCA
import numpy as np

make toy data

data = np.random.rand( 10000, 10 ) * 100;

use sklearn pca

ncomp = 2;
pca = PCA( n_components = 2 );
pca.fit( data );
data_pca = pca.transform( data );
pyplot.scatter( data_pca[:,0], data_pca[:,1]),pyplot.title('Sklearn-PCA'), pyplot.show()

use CCIPCA

ipca = CCIPCA( n_components = 2 );
ipca.fit( data );
idata_pca = ipca.transform( data );
pyplot.scatter( idata_pca[:,0], idata_pca[:,1]),pyplot.title('CCIPCA'), pyplot.show()

use Skocaj_IPCA

ipca = Skocaj_IPCA( n_components = 2 );
ipca.fit( data );
idata_pca = ipca.transform( data );
pyplot.scatter( idata_pca[:,0], idata_pca[:,1]),pyplot.title('Skocaj_IPCA'), pyplot.show()

#use Hall_IPCA
ipca = Hall_IPCA( n_components = 2 );
ipca.fit( data );
idata_pca = ipca.transform( data );
pyplot.scatter( idata_pca[:,0], idata_pca[:,1]),pyplot.title('Hall_IPCA'), pyplot.show()

It seems that both CCIPCA and Skocaj_pca does not work properly, because their center after transformation is too far away from the origin (0,0) and their shapes are more like a oval rather than a circle.

By the way the Skocaj_IPCA often invokes the following warning on my machine:
RuntimeWarning: invalid value encountered in divide
explained_variance_.sum())

Many thanks to your contributions in sklearn

Rex

I have a big dataset in hdf5 (~6gb), that means I cannot load all of it to memory at once (one of the reasons to use IPCA). When I used your code for CCIPCA, these lines of function fit():

1: X = array2d(X)
3: X = as_float_array(X, copy=self.copy)

the first one tries to load the whole dataset to memory (from disk), and the second one raises a "Memory Error", saying that the whole matrix WILL be loaded to memory.
Since both are just checks, I commented them out and had no more problems using the code. But I think this should be a common use case, hence the issue. I'm not sure the problem happens with pure numpy arrays, but since h5py datasets have a similar interface (with slicing, etc), the results should be the same.

Configuration:
python (2.7.6)
h5py (2.3.0)
numpy (1.8.1)
scikit-learn (0.14.1)
scipy (0.14.0)