fuy34 / superpixel_fcn Goto Github PK

Can we set the downsize to 32?

What's the difference between 32 * 32 and 16 * 16？

run_demo.py superpixel grid cell, must be same as training setting why?

Can we use 16 * 16 training 32 * 32 to predict?

Thanks for your excellent work, I want if there any code support Python3? Thank you.

Hi, thanks for open-sourcing this awesome work.

If I understand correctly, the number of superpixels are controlled by the size of the input image (Section 3.4 in the paper). In my case, I would like to get relatively large superpixels, say, only 50 superpixels per image. I set downsize=64 in the run_demo.py file and b_enforce_connect=False, but it produces an image like this:

Also in Fig. 5, the model produces more than 1000 superpixels, is this done by inputting an image of size 512*512?

Any help would be appreciated!

Hello, I would like to ask how many epochs you have trained to achieve this effect?

Hello,

thanks first of all for providing the code to a very interesting paper. My question is about superpixel based downsampling. Here the association map and a LABXY_feat_tensor (consisting of property and location vector) is used. Does that downsampling work without using the property vector? Could you please tell me how this was implemented in the application with PSM net?

Hello author, I have two questions to ask.
Firstly, if I still use the BSDS500 dataset, I can set the superpixel size to 20 × 20 or 21 × 21 or any other value?
Secondly, how can I modify the code to achieve this? Directly modifying the downsize as shown in the figure may result in an error. I made some changes to the code myself, but the result was not ideal. I think I overlooked some important parameters. Thank you.

Our team is trying to merge SpixelNet with our task network as well to improve the segmentation boundary accuracy. We do believe this will greatly push the performance of the task network further! However, as we followed your supplement material, we have some confusions:

1. There is a layer called "sImg_1/2" using the kernel named "assoc_1/2" and stride 4 to down sample the association map that output by SpixelNet (the H x W x 9 tensor) to (1/4 H x 1/4 W x 3). This of course makes sense since most of the backbone / task network only take 3 channel input (RGB images). So in order to merge the two models, we need to massage the output of SpixelNet to some 3 channel tensors. So, if I understand this correctly, this "sImg_1/2" is a custom conv2d layer that doesn't belong to SpixelNet nor PSMNet? If so, does it require any special initialization / activation? (Also I'm guessing the "1/2" denotes left/right image?)
2. Furthermore, when you mentioned in the paper: "replace the conventional operations for downsampling (e.g., stride-2 convolutions) and upsampling (e.g., bilinear upsampling) in the task network with a superpixel-based ownsampling/upsampling scheme to effectively preserve object boundaries and fine details." What operations did you use for the "upsampling"? Judging by the supplement material, it seems like a concatenation between the upsampled output from the PSMNet and the association map "assoc_1" (the H x W x 9 tensor? Yet it is marked H x W x D) along dimension one, and maybe followed by some conv2d to make sure the tensor shape satisfies the input shape of the disparity regression module? And why the "assoc_1" has stride 4?

It will be really helpful if we can get some input from you directly! Thanks! : )

I would like to try Superpixel FCN, but I could not construct the environment to execute.
I used nvidia-docker to build the environment of python2.7, Pytorch0.4.1, CUDA9, and Ubuntu16.04. However, I don't know what python packages I should install.
If it's not too much trouble, can you tell me your environment of python2.7 and installed packages?

Shown in the following my Dockerfile.

FROM nvidia/cuda:9.1-cudnn7-devel-ubuntu16.04

USER root

RUN apt-get update && apt-get install -yq --no-install-recommends \
    git \
    python-dev \
    python-pip \
    python-setuptools \
    && apt-get clean && \
    rm -rf /var/lib/apt/lists/*

RUN pip install torch==0.4.1 \
    cython \
    numpy==1.16.6 \
    pillow==4.3.0 \
    PyWavelets==1.0.3 \
    networkx==2.2 \
    matplotlib==2.1.2 \
    scipy==1.2.3 \
    scikit-image==0.14.1

RUN git clone https://github.com/fuy34/superpixel_fcn.git

RUN cd ./superpixel_fcn/third_party/cython/ && python setup.py install --user

When I run run_demo.py, it generates the unique maps as csv file, what does this file represent?

Hello, may I ask how to adjust the number of super pixels?

I have tried to install the requirements via conda, so it will check the requirements' compatibilities.
I get this error, which I searched and I found out, it's due to CUDA error. The new GPUs don't work with CUDA 9, and it's required to either compile torch 0.4.1 from source with CUDA 10, or install higher version of torch, which doesn't work with your code. Also, you didn't mention the torchvision version, but luckily conda would find the proper version(0.2.1). Could you please update your code for python 3+ and recent torch version?:

Traceback (most recent call last):
  File "run_demo.py", line 169, in <module>
    main()
  File "run_demo.py", line 164, in main
    time = test(args, model, tst_lst, save_path, n)
  File "/home/yousef/anaconda3/envs/superpix_py36/lib/python3.6/site-packages/torch/autograd/grad_mode.py", line 46, in decorate_no_grad
    return func(*args, **kwargs)
  File "run_demo.py", line 97, in test
    output = model(img1.cuda().unsqueeze(0))
  File "/home/yousef/anaconda3/envs/superpix_py36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/yousef/Desktop/SpixelFCN/python3-version/superpixel_fcn/models/Spixel_single_layer.py", line 65, in forward
    out1 = self.conv0b(self.conv0a(x)) #5*5
  File "/home/yousef/anaconda3/envs/superpix_py36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/yousef/anaconda3/envs/superpix_py36/lib/python3.6/site-packages/torch/nn/modules/container.py", line 91, in forward
    input = module(input)
  File "/home/yousef/anaconda3/envs/superpix_py36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/yousef/anaconda3/envs/superpix_py36/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 301, in forward
    self.padding, self.dilation, self.groups)
RuntimeError: cuda runtime error (11) : invalid argument at /opt/conda/conda-bld/pytorch_1535491974311/work/aten/src/THC/THCGeneral.cpp:663```

Hi
I find the image maybe is trained in rgb， not for lab？ it's sure?
could you tell me which should i take, rgb or lab, to train?

Can I control all images to output the same number of superpixels？(Those photos have the same resolution.)
According to the traditional algorithm, such as SLIC, I can get a same number of superpixels in some photos.
And, further, can we control a certain number of pixels in a superpixel?
I not sure it we can do it with spixel_fcn?
Thank u very much.

Hello, I have a question here when run demo in my machine, can you help me please?

NameError: name 'enforce_connectivity' is not defined
File "/home/yangfq/PycharmProjects/pythonProject_sfcn/superpixel_fcn/train_util.py", line 243, in get_spixel_image
spix_index_np = enforce_connectivity(spix_index_np[None, :, :], min_size, max_size)[0]
NameError: name 'enforce_connectivity' is not defined

I have already try different version of connectivity library.

How can I solve this problem, Thank you very much.

Hi, Thanks for open-sourcing this awesome work. I noticed that this repo is for the segmentation and depth estimation tasks. Could you also provide the code or hyper-parameters for Eq.(4) in the paper? Thanks!

Hi,

Thank you for sharing your works. I have tried to run your code in (cuda 9.1, pytorch 1.4, conda environment on windows).
When runing the demo I got following error, while I am using cuda device and return TRUE in python. Is there a way to fix it?

Thanks

When I change the Variable "downsize" to 32, it becomes quite hard for the model to training, that is, the loss reduces slowly (have already trained 800 epochs).
Therefore, can I chose my own superpixels? Is there any special hyparameter that should be changed accordingly?

Dear authors,
The paper and the code are impressive. I have a simple question: In file "train_util.py", the definition of the variable "prob" used in functions "poolfeat()" and "upfeat()" is quite confusing. Concretely, for the first dim of "prob" (prob[:, 0, :, :]) in function "poolfeat()", it is used accompanied with the bottom right superpixel. However, in function "upfeat", this dim is used with the top left superpixel. I assume that in the paper, the superpixel and the prob in equation 1 and 2 should be consistent. What is the definition of every dim in the variable "prob" in your code settings?

Thanks!

When I evaluate my predicted results, I got a error as follows,, I don't know the reason. Hoping your response. Thank you very much.

ruby@ruby-MS-7D36:~/code/superpixel-benchmark/examples$ bash bash/my_eval.sh 
54
bash/my_eval.sh: 行 43: /home/ruby/code/AINET/output/test_multiscale_enforce_connect/SPixelNet_nSpixel_54/map_csv: 是一个目录

the settings in the my_eval.sh as follows:

SUPERPIXELS=("54" "96" "150" "216" "294" "384" "486" "600" "726" "864" "1014" "1176" "1350" "1536" "1944") #

#SUPERPIXELS=("293" "1175") #
IMG_PATH=/home/ruby/code/AINET/test
GT_PATH=/home/ruby/code/AINET/test/map_csv

for SUPERPIXEL in "${SUPERPIXELS[@]}"
do
echo $SUPERPIXEL
/home/ruby/code/AINET/output/test_multiscale_enforce_connect/SPixelNet_nSpixel_${SUPERPIXEL}/map_csv $IMG_PATH $GT_PATH

done

the directory of ‘map_csv‘ as follows：
100007_img.csv
100039_img.csv
...

Why are there many isolated regions with identical IDs. The pixels assigned to identical superpixel is not together.

Hi,
I want to know how long you spent in training？

Hi,
Thank you for making your work available. In the paper it was stated that "Convergence is reached at about 300k iterations." under "Implementation details" in section "3.4. Experiments". What is the criteria you used to determine the convergence? Thanks once again.

Amazing work！
Would you like to provide the visualization code or pseudo algorithm on stereo matching?

Could you please include the requirments.txt?
It had some problems with opencv and I'm not sure what are the versions of the libraries you used.
It's also appreciated if you could include python 3 version. 2to3 didn't work for me, and I'm not sure it was due to the requirements version or not.

Hello author, thank you for your contribution. I retrained after modifying the model architecture, but the matrix grid appeared in the results during the test, which seemed to be initialized superpixels. I don’t know if this is normal.

I noticed that in file loss.py, you say ' # empirically we find timing 0.005 tend to better performance'.
Do the results in the paper adopt this strategy? Do the file SpixelNet_bsd_ckpt.tar in pretrain_ckpt folder adopt this strategy?

I run the run_demo.py. Then,

RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.

But my torch.cuda.is_acailable() is True.

What's wrong?

Hi,

My PC gpu is not strong enough to run 300k epoch for training, could you please send me your BSDS train data for 300k epochs?

email: [email protected]

thanks,

Dear authors,
The paper and the code are impressive. Specially, I am very impressive about the idea that modify PSMNet for stereo matching to simultaneously predict superpixels and disparities. But saddly, I can't find the code about the PSMNet. And I can't see the effect of stereo Matching which benifit form superpixel. If you have time, would you mind answer my question.

Thanks!

Hi, thanks for your great work and public code.
Can you provide all the results.csv files of other methods so I can draw the curves as in your paper?

Thanks!

Hi all，
你们的工作非常有趣！
但是请问这个模型可以生成指定数量的superpixel吗？还是只能生成固定数量的呢？
多谢多谢！

Hello, thanks to your nice work.

I am trying to improve your work. When I need to train your model on different dataset, how should the mean and std in the input_transform be adjusted? Or should it remain the same?

The input size must be same as the patch size in training， or the performance will drop?

Hello author, when I tried to evaluate my training in benchmark (when running 'my_eval.sh'), the following error occurred, please help me thank you

Here is the contents of my_eval.sh：

other files:

How can I train the model with my own dataset?

Hi！
In the experiments, u only use the semantic loss and didn't use the slic loss? what u mentioned in other issues 'color loss' is represented the slic loss in the essay?
when I use the custom dataset, the loss_sem were very large(eg:17812.12), is that correctly?

second, Is the code: rgb2lab_torch is universal? Or is for BSD dataset only.

Thank u very much!

Hello, I don't know how to write upfeat and poolfeat pixel formula, can you provide me the pixel formula for upfeat and poolfeat image using Q?

I check your code, maybe label2one_hot_torch have some bug? Because I don't see you to convert labels from N x H x W to N x 1 x H x W .

for i, (input, label) in enumerate(train_loader):

        iteration = i + epoch * epoch_size

        # ========== adjust lr if necessary  ===============
        if (iteration + 1) in args.milestones:
            state_dict = optimizer.state_dict()
            for param_group in state_dict['param_groups']:
                param_group['lr'] = args.lr * ((0.5) ** (args.milestones.index(iteration + 1) + 1))
            optimizer.load_state_dict(state_dict)

        # ========== complete data loading ================
        label_1hot = label2one_hot_torch(label.to(device), C=50) 

def label2one_hot_torch(labels, C=14):
    b, _, h, w = labels.shape
    one_hot = torch.zeros(b, C, h, w, dtype=torch.long).cuda()
    target = one_hot.scatter_(1, labels.type(torch.long).data, 1)  

Hi! Thanks for your share.
When I use my own dataset, I met the problem : "local variable ' slic_loss' referenced before assignment", but when I use BSD500 to train I didn't meet this problem. Can u know this is why?
Thanks for your patiance!

Hy! 
I'm student of research. I'm interested in your code of Superpixel_fcn and I like it strongly.
I want to use this code in own research era.

During installation of cuda 9 , occur errors " graphic driver not compatible with this version of windows ". How I solve it...
In your code which GPu have used for cuda 9.

I have some troubles to install cuda 9. Please help me for solve my problem....

Hello Fengting,

First of all I want to say thank you for for your work and the provided code. Working with it is great but I have the following issue: I'm trying to jointly learn superpixels and an optical flow net.
I use the sintel dataset, cielab color features and the color loss like you did in the stereo application #12 (comment). The problem I have is that I do not learn super pixels. To further narrow down the problem I freeze the optical flow net and start with the provided model (pretrained on the segmentation task). After a few iterations (around 300) I have unlearned the superpixels (they are squares). I experimented with the pos weight parameter but it does not have an effect on the superpixels.

Do you have any idea what the problem is?

When I change the Variable "downsize" to 32, it becomes quite hard for the model to training, that is, the loss reduces slowly (have already trained 800 epochs).
Therefore, can I chose my own superpixels? Is there ant special hyparameter that should be changed accordingly?

Dear authors,

Your work seems very simple and effective (compared to SSN) at grouping the pixels to superpixels and I am wondering whether you have tried it on the Cityscapes semantic segmentation track.
It would be great if you could share with me your conclusion or observation on the semantic segmentation tasks.

Besides, we are also wondering whether you could share the reproduced implementations of the other superpixel methods, e.g., SSN, to ensure the comparison under the fair settings.

Thanks!

Hi！
Thanks for open-sourcing this awesome work.
I have some questions about how to use to train the downstream task?
I noticed that you said in the essay that you develop a downsampling/upsampling scheme for deep networks with the goal of generating high-resolution outputs for dense prediction tasks. Can you share with me about the job? Or can you give me some ideas about how to do?
I'll appreciate it if you replied to me! Thanks!