Could you please provide a mini-batch training GMM sample using cpu/gpu? I am interested in your work, and want to train a GMM model on a large data, about 100G. Consider we have a (10m, 128) feature vector now. Thanks !

Hi, thank you for sharing this nice package.

After setting the covariance to 'diag', I am getting the below error (only for K=3) when K is equal to the number of features.

  File "C:\Users\msalr\AppData\Roaming\Python\Python37\site-packages\pycave\bayes\_internal\output.py", line 235, in evaluate
    probabilities = log_normal(data.view(-1, shape[-1]), self.means, self.covars)

  File "C:\Users\msalr\AppData\Roaming\Python\Python37\site-packages\pycave\bayes\_internal\utils.py", line 33, in log_normal
    precisions = precisions.view(1, num_features).expand(

RuntimeError: shape '[1, 3]' is invalid for input of size 9

Could you please check if these lines are correct in bayes/internal/utils.py?

if covars.size(0) == num_features: # shared diagonal covariance
        num_means = means.size(0)
        precisions = precisions.view(1, num_features).expand(
            num_means, num_features
        )

Hi,

Thank you for the library!

In other posts I read that covariances are stored under gmm.gaussian.covars but I cannot find it. The only thing related to the covraiance is gmm.model_.precisions_cholesky but I don't know how to use them. I am attaching the content of fitted GaussianMixture object. Could you please tell me how to find covariances or how to calculate it using precisions_cholesky?

Hi @borchero,
I am using the GPU for clustering or GMM and the initialization operation takes a long time compared to the CPU. After executing the following code segment on the RTX3090, the GPU initialization time is about 4.1 seconds. However, the CPU only takes about 0.17 seconds. Any suggestions to solve this problem?

from pycave.bayes import KMeans
import torch
import time 

input_data = torch.randn(90000, 3)
start_time = time.time()
estimator = KMeans(3, trainer_params = dict(gpus = 1, max_epochs = 10))
estimator.fit(input_data)
end_time = time.time()
print('cost_time: %f seconds', %(end_time - start_time))

I have disabled progress bar using enable_progress_bar=0, but not able to find the correct parameter that will disable the internal model output printing.
Just want to disable those coz I will be running it a lot of times, and don't want the output console to be printed with all the models internal outputs

import torch
import pandas as pd
from pycave.clustering import KMeans
from pycave.bayes import GaussianMixture

X = torch.cat([
    torch.randn(1000, 6) - 5,
    torch.randn(1000, 6),
    torch.randn(1000, 6) + 5,
])

estimator = KMeans(num_clusters = 4, trainer_params=dict(gpus=1, 
                                                         enable_progress_bar=0, 
                                                         max_epochs=100,
                                                         enable_model_summary=False))
labels = estimator.fit_predict(X).numpy()

print(pd.value_counts(labels))

Current Output:

Running initialization...
Fitting K-Means...
{'batch_size': 3000, 'collate_fn': <function collate_tensor at 0x0000015AC3F37C10>}
{'batch_size': 3000, 'collate_fn': <function collate_tensor at 0x0000015AC3F37C10>}
{'batch_size': 1, 'sampler': None, 'batch_sampler': <pytorch_lightning.overrides.distributed.IndexBatchSamplerWrapper object at 0x0000015A92D84040>, 'collate_fn': <function collate_tensor at 0x0000015AC3F37C10>, 'shuffle': False, 'drop_last': False}
1    1000
0    1000
3     509
2     491
dtype: int64

Expected Output:

1    1000
0    1000
3     509
2     491
dtype: int64

Hi,

I want to implement mini-batching training on GMM as discussed in #7 . However, I am little bit confused by the code gmm.reset_parameters(torch.Tensor(fvectors[:500].astype(np.float32))). I am not sure whether it is related to my version of pycave, or maybe my understanding to the code in #7 is wrong. My code doesn't work.

My code are as follows:

from pycave.bayes.gmm import GaussianMixture as GM
from dataloader.gmm_dataset import gmm_dataset

train_gmm_dataset = gmm_dataset(data_path)
train_dataset_loader = torch.utils.data.DataLoader(dataset=train_gmm_dataset,
                                                        batch_size=train_dataloader_config["batch_size"],
                                                        shuffle=train_dataloader_config["shuffle"],
                                                        num_workers=train_dataloader_config["num_workers"])

for i, data in enumerate(train_dataset_loader):  # data:[1, pt, 3]
    data = torch.squeeze(data, 0)
    gmm = GM(num_components=2, covariance_type="diag", init_strategy="kmeans")
    gmm.model_.reset_parameters(data)  
    history = gmm.fit(train_dataset_loader)

And the error is:

`GaussianMixture` has not been fitted yet

Thank you so much!

Best regards,
Daisy

I observed that some of the covariance matrices are not symmetric. Most of them are close but some are considerably off. It turns out that multiplying the precisions_cholesky matrices and then taking an inverse was causing the issue. If you take the inverse and then multiply the resulting inverse precisions_cholesky matrices, then the covariances come out as symmetric. It does take a little longer (about 20%) for some reason but I think this would be a more accurate way to calculate the covariances. Let me know if this makes sense or if I am missing anything important.

Minimal working example:

from pycave.bayes import GaussianMixture
import torch
import numpy as np
import pycave
import random
import time

#Set seed
seed = 0
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)
random.seed(seed)

#Inputs
n = 10000
p = 2000
k = 40

#Make some Gaussian data
X = torch.randn(n,p)

#Fit PyCave GMM
gmm = GaussianMixture(num_components=k,
covariance_type='full',
init_strategy='kmeans',
trainer_params={'gpus':1,'enable_progress_bar':False},
covariance_regularization=1e-3)
gmm = gmm.fit(X)

NOT SYMMETRIC - Show the top corner of the covariance matrix:

t_start = time.time()
covs = gmm.model_.covariances
t_end = time.time()
print('Time to compute covariance =', t_end - t_start)
print(covs[0,0:5,0:5])

SYMMETRIC - Show the top corner of the covariance matrix when calculated differently

t_start = time.time()
P = torch.linalg.inv(gmm.model_.precisions_cholesky)
covs2 = torch.bmm(P.transpose(1, 2), P)
t_end = time.time()
print('Time to compute covariance =', t_end - t_start)
print(covs2[0,0:5,0:5])

Hi,

Like #7 and #19 I am trying to fit a GMM to a large dataset [10^10, 50] and want to (need to) use mini-batching.

However, in contrast to the previous answers, gmm.fit only accpects a TensorLike and won't work with my data which is a torch.utils.data.DataLoader. Even if I input a torch.utils.data.Dataset it only computes a GMM on the first batch.

What is the preferred way to do what I want to do?

Ideally, I would want my code to work like this:

from pycave.bayes import GaussianMixture as GMM
from torch.utils.data import Dataset, DataLoader

data = Data(DATA_PATH).dataloader(batch_size=256)
assert(type(data) == DataLoader)

gmm = GMM(num_components=3, batch_size=256, trainer_params=dict(accelerator='gpu', devices=1))
class_labels = gmm.fit_predict(data)
means, stds = gmm.model_.means, gmm.model_.covariances

Manually changing the code in gmm/estimator.py (among others) from

num_features = len(data[0])
...
loader = DataLoader(
    dataset_from_tensors(data),
    batch_size=self.batch_size or len(data),
    collate_fn=collate_tensor,
)
is_batch_training = self._num_batches_per_epoch(loader) == 1          # Also, shouldn't this be > anyway?

to

num_features = data.dataset[0].shape[1]
...
loader = data
is_batch_training = True

allows the for error-free fitting and prediction but I am not sure if the output is trustworthy.

How can I compute the Bayesian Information Criterion for a GMM using this repo?

Hello:

The title kind of says it all. The gmm.model_.covariances variable is None for some reason for both of these models. I don't think this was the case in the past so maybe one of the last commits had a bug. Unless this is intentional?

Here is a minal working example:

from pycave.bayes import GaussianMixture
import torch
import numpy as np
import pycave
import random
import time

#Set seed
seed = 0
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)
random.seed(seed)

#Inputs
d = 10
k = 1
n = 1000

#Data
x = torch.randn(n,d)

#Fit PyCave GMM
gmm = GaussianMixture(num_components=k,
covariance_type='diag',
init_strategy='kmeans',
trainer_params={'gpus':1,'enable_progress_bar':True, 'logger':True},
covariance_regularization=1e-6)
gmm = gmm.fit(x)

print(type(gmm.model_.covariances))

Hi @borchero,
I want to confirm if I understood correctly the output of the GMM methods. If I compare with sklearn, is this table correct?

Thank you so much

Hi,

I am trying to install pycave using pip3 install pycave , but I get the following error:

ERROR: Could not find a version that satisfies the requirement pycave (from versions: none) ERROR: No matching distribution found for pycave

I've already updated my pip version to the latest version, and I can successfully install other available packages using pip3 install. Is there something wrong with my python environment?

Thanks.
Daisy

Hi, first of all thank you for the amazing repository.

I am trying to do mini-batch training of GMM. After going over #51 #19 and #7, I realized that I need to create my own dataset loader. Here is an sample of my custom dataset loader (Each file of my dataset is stored in .pkl files, so I wrote a pkl_dataset class).

` Dataset Class

class PKL_dataset(Dataset):

def __init__(self, dataset_pth, data_label):
    self.data_dir = dataset_pth
    self.files_ls = os.listdir(dataset_pth)
    self.len = len(self.files_ls)
    self.label = data_label

def __len__(self):
    return self.len

def transform(self, data):

    if data.shape[0] < 2000:

        return np.pad(data, [(0, 2000 - data.shape[0]), (0,0)], 'mean')

    else:

        return data[:2000]

def __getitem__(self, idx):

    file_path = os.path.join(self.data_dir, self.files_ls[idx])
    
    pkl_data = open_pkl(file_path)
    transformed_pkl_data = self.transform(pkl_data)
    
    return transformed_pkl_data`

Now I am calling the Gaussian Mixture class like this,

gmm = GaussianMixture(num_components=ncomp, covariance_type='diag', batch_size=32, covariance_regularization=0.1, init_strategy='kmeans++', trainer_params=dict(accelerator='gpu', devices=1, max_epochs=100))

and passing the my dataset to the fit function like this,

history = gmm.fit(pkl_dataloader)

It gives me the following error,

Traceback (most recent call last): File "asvspoof2021_baseline.py", line 65, in <module> gmm_bona = train_gmm(data_label=data_labels[0], features=features, File "/home/hashim/PhD/Audio_Spoof_Detection/Baseline-CQCC-GMM/python/gmm.py", line 218, in train_gmm history = gmm.fit(pkl_dataloader) File "/home/hashim/PhD/AsvSpoof2021/asvspoof_venv/lib/python3.8/site-packages/pycave/bayes/gmm/estimator.py", line 128, in fit num_features = len(data[0]) TypeError: 'DataLoader' object is not subscriptable

It looks like the fit routine does not accept data as a dataloader.

However, if I do not use a dataloader, the training gets killed because of the memory issues.

Here is a snapshot of my system, My GPU is NVIDIA RTX A2000 12GB

I was wondering if there's an easy way to find the score; as in sklearn, which computes the per-sample average log-likelihood of the given data X?

I get the following error when trying to fit a GMM to some data:
RuntimeError: expected scalar type Double but found Float

The following is the error printout:

I am passing a pyTorch dataloader to the fit function but I have checked their type and it is tensor.double
Have I done something wrong
The code I'm using is almost a direct copy-paste from your quickstart guide:
gmm = GMM(num_components=2, num_features=29, covariance='diag')
history = gmm.fit(train_loader)

Thanks for your help in advance!

I tried to compare the results with sklearn GMM. Unfortunately, results do not accord. Not sure if I'm missing something?
Here's the code, data file (numpy format) is attached 3d_array.zip

`
from pycave.bayes.gmm import GMM
import torch
import numpy as np
from sklearn.mixture import GaussianMixture

X = np.load(' 3d_array.npy')
X = X.astype('float32')
X1 = X[0:3]

n_components=2
print('Sklearn Run')
skl_model = GaussianMixture(n_components=n_components, random_state=0).fit(X)
print('means', skl_model.means_)
labels = skl_model.predict(X1)
print('Prediction labels of X1:', labels)

config = {
'num_components': n_components,
'num_features': X.shape[1],
'covariance': 'spherical', # {'spherical', 'diag-shared'}
}

print('\n Pycave Run')
torch.manual_seed(0)
model = GMM(**config)
model.fit(torch.tensor(X))
print('means', model.gaussian.means.cpu().numpy())
labels = model.predict(torch.tensor(X1)).cpu().numpy()
print('Prediction labels of X1:', np.argmax(labels, axis=1))

`

Hi @borchero,
Is it possible to add three methods or attributes to return the weights, means and covariances?
The covariances would be especially useful to analyse the model when we have "singleton datapoint".

Thank you

Hey Oliver, thank you very much for making PyCave, I think it's going to make my life much easier compared to Sklean.

I think the covariance matrices are not being calculated or stored correctly. For example, I tested a model with 20 components/clusters and 8 features per data point. The covariance matrix in gmm.gaussian.covars output as a 1X20 array when it should be 8X8X20. The component weights and mixture means appear to be correct.

Hi, I've been trying to train a 1000 components GMM using pycave. My data is a numpy array with shape(2300000, 40).
When I try to train on GPU, the system raise an error and the training crush. However, if I train on CPU, I have no problems.

Are there some requirements to train on the GPU?

A snippet of my code

from pycave.bayes import GMM
from torch.utils.data import DataLoader, Dataset

class MyDataset(Dataset):
    def __init__(self, data):
        self.data = data

    def __getitem__(self, index):
        x = torch.from_numpy(self.data[index])
        return x
    
    def __len__(self):
        return len(self.data)

data_set = MyDataset(data) # np array shape (2000000, 40)
loader = DataLoader(data_set, batch_size=10, shuffle=True, num_workers=5)

gmm = GMM(num_components=50, num_features=40, covariance='diag')
history = gmm.fit(loader, gpu=False)

From the snippet above, I can't set gpu=True, the program crush

Hi, I'm just getting started with pycave and would like to use pytorch 2.0. It's torch.compile mechanism could be really powerful.

Would you please allow your package to also allow 2.0 for both pytorch and pytorch-lightning?

Also, would you expected any speedups from using torch.compile?

Thank you! 🙇

Hello,

I'm using pycave for a project where the data is unidimensional of size 8e9. The GPU options works well, and I'm splitting the data in "for loops" to do the predictions. However, as the loops goes on, it takes more and more of the GPU memory, and eventually runs out of memory. To contour this issue, I'm using torch clean cache at each interaction in addition to the garbage collector function in python, as shown in the code below, however this process is slow.

import gc
def clear_gpu_memory():
    torch.cuda.empty_cache()
    gc.collect()
    torch.cuda.empty_cache()

I've tried to use the pycave built-in function of batches as well, but it also runs in the memory issue.

Is there anything I could do to fix this?

Hi, I trained the GaussianMixture model with pycave v3.2.1, however, after training a few epochs nll=nan.0 appears. And this situation happens occasionally, I would like to ask what is the solution. My training code is as follows:

representation = torch.tensor(representation) 
representation = representation.to(self.device)
gmm = GMM(num_components=3, covariance_type="diag", init_strategy="kmeans", batch_size=16, trainer_params=dict(accelerator='gpu', devices=1, max_epochs=50))
gmm.fit(representation)

The error log is as follows:
Epoch 1: 100%|██████████| 15/15 [00:00<00:00, 47.29it/s, nll=-5.33e+3]
Epoch 1: 100%|██████████| 15/15 [00:00<00:00, 47.23it/s, nll=-5.33e+3]
Epoch 1: 100%|██████████| 15/15 [00:00<00:00, 46.75it/s, nll=nan.0]
Epoch 1: 100%|██████████| 15/15 [00:00<00:00, 46.04it/s, nll=nan.0]

Thank you so much!

Best regards,
Zhiwei

The mini-batch part of this repository works great! However, when the batch size is not a factor or the total dataset size, the code throws an error. Is there anyway to make it so that any batch size can be used? Below is a minimal working example of what I am talking about. Essentially, if batch_size = 1000 then everything works fine and the mini-batch procedure seems to work with all 10 batches. However, when batch_size = 999, the last batch (of size 10) causes an error. Thanks!

from pycave.bayes import GaussianMixture
import torch

#Set seed
seed = 0
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)

#Inputs
n = 10000
p = 200
k = 5
batch_size = 999 #1000

#Make some non-Gaussian data
X = torch.randn(n,p)

#Fit PyCave GMM
gmm = GaussianMixture(num_components=k,
covariance_type='full',
init_strategy='kmeans++',
batch_size=batch_size,
trainer_params={'gpus':1,'enable_progress_bar':False},
covariance_regularization=1e-3)
gmm = gmm.fit(X)

I was trying to fit a GMM on data and kept getting the same error (with variying numbers for the batch and order):

_LinAlgError: torch.linalg_cholesky: (Batch element 3): The factorization could not be completed because the input is not positive-definite (the leading minor of order 941 is not positive-definite).

I made a minimal working example to show when this comes up in practice. I compared to sklearn and somehow sklearn is able to avoid this problem. This issue happens both on CPU and GPU. I have PyCave 3.1.3 and sklearn 0.24.2. Do you have any idea what could be the issue?

Minimum working example:

from pycave.bayes import GaussianMixture
import torch
import numpy as np
from sklearn import mixture

#Set seed
seed = 0
np.random.seed(seed)
torch.manual_seed(seed)

#Inputs
n = 5000
p = 2000
k = 10

#Make some non-Gaussian data
X = np.random.randn(n,p)
X = torch.Tensor(X)
X = torch.nn.ReLU()(X-1)

#Fit Sklearn GMM
gmm_sk = mixture.GaussianMixture(n_components=k,
covariance_type='full',
init_params='kmeans')
gmm_sk.fit(X.numpy())

#Fit PyCave GMM
gmm = GaussianMixture(num_components=k,
covariance_type='full',
init_strategy='kmeans')
gmm.fit(X)

Hi @borchero,
I am training a GMM using a 40-dimensional feature vector.
There is no problem when I train the model using the features as is. However, if I standardise the features using sklearn's StandardScaler, I got nan loss after three of four epochs.

Is there any way I can easily debug the model/engine to try to find the source of the issue?

Hi,

I'm wondering if multi-gpu is supported.
It works when using one GPU but when using multiple GPUs I couldn't get either the Kmeans or GMM to fit (i.e. devices = 3). Thanks!

estimator_gmm = GaussianMixture(8, trainer_params=dict(accelerator='gpu', devices=3,
                                                       enable_progress_bar = True),
                                init_means = estimator.model_.centroids)

ProcessRaisedException                    Traceback (most recent call last)
Cell In[29], line 1
----> 1 estimator_gmm.fit(torch.from_numpy(X))

File conda_envs/squidpy/lib/python3.8/site-packages/pycave/bayes/gmm/estimator.py:175, in GaussianMixture.fit(self, data)
    168 else:
    169     module = GaussianMixtureRandomInitLightningModule(
    170         self.model_,
    171         covariance_regularization=self.covariance_regularization,
    172         is_batch_training=is_batch_training,
    173         use_model_means=self.init_means is not None,
    174     )
--> 175     self.trainer(max_epochs=1 + int(is_batch_training)).fit(module, loader)
    177 # Fit model
    178 logger.info("Fitting Gaussian mixture...")

File conda_envs/squidpy/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:700, in Trainer.fit(self, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path)
    681 r"""
    682 Runs the full optimization routine.
    683 
   (...)
    697     datamodule: An instance of :class:`~pytorch_lightning.core.datamodule.LightningDataModule`.
    698 """
    699 self.strategy.model = model
--> 700 self._call_and_handle_interrupt(
    701     self._fit_impl, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path
    702 )

File conda_envs/squidpy/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:652, in Trainer._call_and_handle_interrupt(self, trainer_fn, *args, **kwargs)
    650 try:
    651     if self.strategy.launcher is not None:
--> 652         return self.strategy.launcher.launch(trainer_fn, *args, trainer=self, **kwargs)
    653     else:
    654         return trainer_fn(*args, **kwargs)

File conda_envs/squidpy/lib/python3.8/site-packages/pytorch_lightning/strategies/launchers/multiprocessing.py:103, in _MultiProcessingLauncher.launch(self, function, trainer, *args, **kwargs)
    100 else:
    101     process_args = [trainer, function, args, kwargs, return_queue]
--> 103 mp.start_processes(
    104     self._wrapping_function,
    105     args=process_args,
    106     nprocs=self._strategy.num_processes,
    107     start_method=self._start_method,
    108 )
    109 worker_output = return_queue.get()
    110 if trainer is None:

File conda_envs/squidpy/lib/python3.8/site-packages/torch/multiprocessing/spawn.py:198, in start_processes(fn, args, nprocs, join, daemon, start_method)
    195     return context
    197 # Loop on join until it returns True or raises an exception.
--> 198 while not context.join():
    199     pass

File conda_envs/squidpy/lib/python3.8/site-packages/torch/multiprocessing/spawn.py:160, in ProcessContext.join(self, timeout)
    158 msg = "\n\n-- Process %d terminated with the following error:\n" % error_index
    159 msg += original_trace
--> 160 raise ProcessRaisedException(msg, error_index, failed_process.pid)

ProcessRaisedException: 

-- Process 2 terminated with the following error:
Traceback (most recent call last):
  File "conda_envs/squidpy/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap
    fn(i, *args)
  File "conda_envs/squidpy/lib/python3.8/site-packages/pytorch_lightning/strategies/launchers/multiprocessing.py", line 129, in _wrapping_function
    results = function(*args, **kwargs)
  File "conda_envs/squidpy/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 741, in _fit_impl
    results = self._run(model, ckpt_path=self.ckpt_path)
  File "squidpy/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1101, in _run
    self.strategy.setup_environment()
  File "squidpy/lib/python3.8/site-packages/pytorch_lightning/strategies/strategy.py", line 130, in setup_environment
    self.accelerator.setup_environment(self.root_device)
  File "squidpy/lib/python3.8/site-packages/pytorch_lightning/accelerators/cuda.py", line 43, in setup_environment
    torch.cuda.set_device(root_device)
  File "squidpy/lib/python3.8/site-packages/torch/cuda/__init__.py", line 314, in set_device
    torch._C._cuda_setDevice(device)
  File "squidpy/lib/python3.8/site-packages/torch/cuda/__init__.py", line 207, in _lazy_init
    raise RuntimeError(
RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method