Hi
@luanfujun

The following code in the file deep-photo-styletransfer/gen_laplacian/gen_laplacian.m is really confusing me.

    disp('Save to disk');
    n = nnz(A);
    [Ai, Aj, Aval] = find(A);
    CSC = [Ai, Aj, Aval];
    %save(['Input_Laplacian_3x3_1e-7_CSC' int2str(i) '.mat'], 'CSC');
    
    [rp ci ai] = sparse_to_csr(A);
    Ai = sort(Ai);
    Aj = ci;
    Aval = ai;
    CSR = [Ai, Aj, Aval];
    save(['Input_Laplacian_3x3_1e-7_CSR' int2str(i) '.mat'], 'CSR');
 

The rp variable, which is the signature of the CSR format, is not used at all. So I think maybe the saved .mat file is not a CSR file.

However, you used mate.load.CSR function to load .mat file in lua file

-- load matting laplacian
  local CSR_fn = 'gen_laplacian/Input_Laplacian_'..tostring(params.patch)..'x'..tostring(params.patch)..'_1e-7_CSR' .. tostring(index) .. '.mat'
  print('loading matting laplacian...', CSR_fn)
  local CSR = matio.load(CSR_fn).CSR:cuda()

could you please explain this conflict?

the structure of torch seems to have changed, THC isn't found at install/include/THC, moved to extra/cutorch/lib/THC.

Including the now found THC.h file results in another issue I already reported here;
torch/cutorch#737

I keep getting every time after, I compile the laplacian with octave and the files were created successfully. Any thoughts on the issue?

Many thanks

loading matting laplacian...	gen_laplacian/Input_Laplacian_3x3_1e-7_CSR1.mat	
Segmentation fault (core dumped)

<optim.lbfgs> 	reached max number of iterations	
gpu, idx = 	0	1	
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:505] Reading dangerously large protocol message.  If the message turns out to be larger than 1073741824 bytes, parsing will be halted for security reasons.  To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:78] The total number of bytes read was 574671192
Successfully loaded models/VGG_ILSVRC_19_layers.caffemodel
conv1_1: 64 3 3 3
conv1_2: 64 64 3 3
conv2_1: 128 64 3 3
conv2_2: 128 128 3 3
conv3_1: 256 128 3 3
conv3_2: 256 256 3 3
conv3_3: 256 256 3 3
conv3_4: 256 256 3 3
conv4_1: 512 256 3 3
conv4_2: 512 512 3 3
conv4_3: 512 512 3 3
conv4_4: 512 512 3 3
conv5_1: 512 512 3 3
conv5_2: 512 512 3 3
conv5_3: 512 512 3 3
conv5_4: 512 512 3 3
fc6: 1 1 25088 4096
fc7: 1 1 4096 4096
fc8: 1 1 4096 1000
loading matting laplacian...	gen_laplacian/Input_Laplacian_3x3_1e-7_CSR1.mat	
Segmentation fault (core dumped)

I got a problem with
[$ make clean && make

find . -type f | xargs -n 5 touch
rm -f libcuda_utils.so
/usr/local/cuda-8.0/bin/nvcc -arch sm_35 -O3 -DNDEBUG --compiler-options '-fPIC' -o libcuda_utils.so --shared cuda_utils.cu -I/home/chloe/torch/install/include/THC -I/home/chloe/torch/install/include/TH -I/home/chloe/torch/install/include -L/home/chloe/torch/install/lib -Xlinker -rpath,/home/chloe/torch/install/lib -lluaT -lTHC -lTH -lpng
cuda_utils.cu(509): error: identifier "luaL_openlib" is undefined

1 error detected in the compilation of "/tmp/tmpxft_00004fc0_00000000-9_cuda_utils.cpp1.ii".
makefile:10: 'libcuda_utils.so' failed operation]

Do you know about this problem to solve? with cuda 8.0

The error obtained is as shown below.

vishi@vishi-workstation:~/NSTYLE/deep-photo-styletransfer$ make /usr/local/cuda-8.0/bin /nvcc -arch sm_35 -O3 -DNDEBUG --compiler-options '-fPIC' -o libcuda_utils.so --shared cuda_utils.cu -I/home/vishi/torch/install /include/THC -I/home/vishi/torch/install /include/TH -I/home/vishi/torch/install /include -L/home/vishi/torch/install /lib -Xlinker -rpath,/home/vishi/torch/install /lib -lluaT -lTHC -lTH -lpng make: execvp: /usr/local/cuda-8.0/bin: Permission denied make: *** [libcuda_utils.so] Error 127

I do have the required permissions for the location mentioned in the error.

The first two lines of the makefile were changed, as needed.

PREFIX=/home/vishi/torch/install NVCC_PREFIX=/usr/local/cuda-8.0/bin

Hello, I'm not very familiar with linux, can someone pleas give me step-by-step instructions on how to install and use it with examples? I understand, that compiler commands will be specific for every machine, but maybe there is simple solution for that?

Sorry for being dumb.

Hi, thanks for the project - it's mind blowing to say the least!

I would like to run this script inside a VPS with no GPU and I was wondering how much of the code is tied to CUDA...

I and some my friends are excited to try to build real ios/android app based on this white paper ideas.
We do not have to limit our scope with it, but we feel impressed by this demo.
If you are interested in participating drop me an email to [email protected] - I add you to our trello/slack.

If possible, please pick a suitable license and place the corresponding file named LICENSE in the root of your repository. I suggest picking ISC, BSD or MIT as the license. Remember to set the year, name and e-mail address of the copyright holder(s) at the top of the license file.

When I try running step 4 (after having generated the intermediate for in1 and tra1 successfully), I get this

th deepmatting_seg.lua -content_image examples/input/in1.png -style_image examples/style/tar1.png -content_seg examples/segmentation/in1.png -style_seg examples/segmentation/tar1.png -init_image serial_example/out1_t_1000.png -serial outputs -f_radius 15 -f_edge 0.01
gpu, idx =      0       1
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:505] Reading dangerously large protocol message.  If the message turns out to be larger than 1073741824 bytes, parsing will be halted for security reasons.  To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:78] The total number of bytes read was 574671192
Successfully loaded models/VGG_ILSVRC_19_layers.caffemodel
conv1_1: 64 3 3 3
conv1_2: 64 64 3 3
conv2_1: 128 64 3 3
conv2_2: 128 128 3 3
conv3_1: 256 128 3 3
conv3_2: 256 256 3 3
conv3_3: 256 256 3 3
conv3_4: 256 256 3 3
conv4_1: 512 256 3 3
conv4_2: 512 512 3 3
conv4_3: 512 512 3 3
conv4_4: 512 512 3 3
conv5_1: 512 512 3 3
conv5_2: 512 512 3 3
conv5_3: 512 512 3 3
conv5_4: 512 512 3 3
fc6: 1 1 25088 4096
fc7: 1 1 4096 4096
fc8: 1 1 4096 1000
loading matting laplacian...    gen_laplacian/Input_Laplacian_3x3_1e-7_CSR1.mat
File could not be opened: gen_laplacian/Input_Laplacian_3x3_1e-7_CSR1.mat
/home/ubuntu/torch/install/bin/luajit: deepmatting_seg.lua:119: attempt to index a nil value
stack traceback:
        deepmatting_seg.lua:119: in function 'main'
        deepmatting_seg.lua:606: in main chunk
        [C]: in function 'dofile'
        ...untu/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:150: in main chunk
        [C]: at 0x00406670

The CSR file is generated with octave 4.0. There's an issue here, that hints at Matlab version compatibility issues. I don't know if this prevents use of Octave.

When I try to compile cuda_utils.cu, I get this message:

sacha@scriabin:~/deep-photo-styletransfer-master$ make clean && make

find . -type f | xargs -n 5 touch
rm -f libcuda_utils.so
/usr/local/cuda-8.0/bin/nvcc -arch sm_35 -O3 -DNDEBUG --compiler-options '-fPIC' -o libcuda_utils.so --shared cuda_utils.cu -I/home/torch/install/include/THC -I/home/torch/install/include/TH -I/home/torch/install/include -L/home/torch/install/lib -Xlinker -rpath,/home/torch/install/lib -lluaT -lTHC -lTH -lpng
cuda_utils.cu:2:18: fatal error: lua.h: No such file or directory
#include "lua.h"
^
compilation terminated.
make: *** [libcuda_utils.so] Error 1

I changed my makefile to the following:

PREFIX=/home/torch/install
NVCC_PREFIX=/usr/local/cuda-8.0/bin

lua.h, lualib.h and lauxlib.h are all in the same folder: /home/torch/install/include.

I already tried to add -lluajit like proposed in #5 and 21 - doesn't work either.

I'm on Ubuntu 14.04.5 LTS, and after a long reading everything before worked just fine (that wasn't the case with my first shot in another computer, now I'm being way more careful about each step). I kept track of all the command lines I used and the files I downloaded if required. The luarocks I used was the one within torch.

Thanks for your help!

first, amazing work!

secondly, just curious why the apple colors have changed even though the two inputs are identical?

unfortunately, my laptop, Dell Inspiron 5000 series & Ubuntu 16, with Intel® Haswell Mobile Graphics, has no nvidia graphics. But i wanna run this project on my laptop. All of stuff is already, no nvidia device founded. Any way to skip CUDA?

i use 1080ti and cuda8.0.And find this problem.

loading matting laplacian... gen_laplacian/Input_Laplacian_3x3_1e-7_CSR1.mat File could not be opened: gen_laplacian/Input_Laplacian_3x3_1e-7_CSR1.mat /home/pushi_dev/torch/install/bin/luajit: deepmatting_seg.lua:119: attempt to index a nil value stack traceback: deepmatting_seg.lua:119: in function 'main' deepmatting_seg.lua:606: in main chunk [C]: in function 'dofile' ..._dev/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:150: in main chunk [C]: at 0x00405d50

Hey there, i was wondering where i can find the λ parameter from the original paper in your code.

The paper states that a value of 10e4 gave the best results, but i woud like to try to experiment a little bit with lower values. I imagine that a lower value could preserve more style detail – for the cost of looking more like the CNNMRF method.

Am i simply overlooking the parameter because it's named differently or is it hidden in the more complex functions?

I also tried playing around withf_radius – reducing the value preserved more detail – transferred more style detail to the final image. Could you explain what f_edge does?

Since you have a lot of landscape examples, but no animals, my test example was an attempt to transfer the style of a tiger to a housecat 😀. Maybe CNNMRF would be better suited for that – but i wanted to take advantage of the photorealistic effect of this method. CNNMRF would make the final image look more like a painting, and i've seen that often enough by now.
The masking of your method offers a great degree of control for finetuning how you want your output to look like.
EDIT: I wrote lion, but of course it's a tiger... Note to self: stripes? = tiger. no stripes, furry mane? = lion. If you f*ck with nature? = liger

Temp result (neural style):

Final results :-) : (f_radius @ 5 – 0.5)

And the masks:

Mask colors

Another quick question about the colors you can use for your mask pngs: In neuralstyle_seg.lua and deepmatting_seg.lua you define that these colors are processed from the segmentation images:

`local color_codes = {'blue', 'green', 'black', 'white', 'red', 'yellow', 'grey', 'lightblue', 'purple'}`

The problem for me (and maybe others users too) was to recognize that you are restricted to these colors when you "paint" your masks. At first i didn't look at the code and just chose random colors – i only realized that something was wrong when the generated output didn't match my selections.

So could you maybe include a info in your Readme how to manually create a mask – which colors you can use (maybe hex colors?). I'm not sure if everyone automatically knows what colors are actually processed.

Thanks!

Running

th neuralstyle_seg.lua -content_image examples/input/in1.png -style_image examples/style/tar1.png -content_seg examples/segmentation/in1.png -style_seg examples/segmentation/tar1.png

yields

gpu, idx =      0       1
Couldn't load models/VGG_ILSVRC_19_layers.caffemodel
/home/ubuntu/torch/install/bin/luajit: neuralstyle_seg.lua:98: attempt to index a nil value
stack traceback:
        neuralstyle_seg.lua:98: in function 'main'
        neuralstyle_seg.lua:546: in main chunk
        [C]: in function 'dofile'
        ...untu/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:150: in main chunk
        [C]: at 0x00406670

loadcaffe andt matio are installed.

Has anyone (successfully) tried running this on Windows?

The Torch framework is not officially compatible with Windows. Is there a way to
somehow use other framework?

Thanks.

Hey thanks for the code, unfortunately i get a compile error when i try to build the cuda_utils.cu:

make clean && make
find . -type f | xargs -n 5 touch
rm -f libcuda_utils.so
/usr/local/cuda/bin/nvcc -arch sm_35 -O3 -DNDEBUG --compiler-options '-fPIC' -o libcuda_utils.so --shared cuda_utils.cu -I/Users/david/torch/install/include/THC -I/Users/david/torch/install/include/TH -I/Users/david/torch/install/include -L/Users/david/torch/install/lib -Xlinker -rpath,/Users/david/torch/install/lib -lluaT -lTHC -lTH -lpng
Undefined symbols for architecture x86_64:
  "_luaL_checknumber", referenced from:
      matting_laplacian(lua_State*) in tmpxft_0001773d_00000000-16_cuda_utils.o
      smooth_local_affine(lua_State*) in tmpxft_0001773d_00000000-16_cuda_utils.o
  "_luaL_error", referenced from:
      checkCudaError(lua_State*) in tmpxft_0001773d_00000000-16_cuda_utils.o
      matting_laplacian(lua_State*) in tmpxft_0001773d_00000000-16_cuda_utils.o
      smooth_local_affine(lua_State*) in tmpxft_0001773d_00000000-16_cuda_utils.o
  "_luaL_openlib", referenced from:
      _luaopen_libcuda_utils in tmpxft_0001773d_00000000-16_cuda_utils.o
  "_lua_call", referenced from:
      getCutorchState(lua_State*) in tmpxft_0001773d_00000000-16_cuda_utils.o
      matting_laplacian(lua_State*) in tmpxft_0001773d_00000000-16_cuda_utils.o
      smooth_local_affine(lua_State*) in tmpxft_0001773d_00000000-16_cuda_utils.o
  "_lua_getfield", referenced from:
      getCutorchState(lua_State*) in tmpxft_0001773d_00000000-16_cuda_utils.o
      matting_laplacian(lua_State*) in tmpxft_0001773d_00000000-16_cuda_utils.o
      smooth_local_affine(lua_State*) in tmpxft_0001773d_00000000-16_cuda_utils.o
  "_lua_settop", referenced from:
      getCutorchState(lua_State*) in tmpxft_0001773d_00000000-16_cuda_utils.o
      matting_laplacian(lua_State*) in tmpxft_0001773d_00000000-16_cuda_utils.o
      smooth_local_affine(lua_State*) in tmpxft_0001773d_00000000-16_cuda_utils.o
  "_lua_touserdata", referenced from:
      getCutorchState(lua_State*) in tmpxft_0001773d_00000000-16_cuda_utils.o
      matting_laplacian(lua_State*) in tmpxft_0001773d_00000000-16_cuda_utils.o
      smooth_local_affine(lua_State*) in tmpxft_0001773d_00000000-16_cuda_utils.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [libcuda_utils.so] Error 1

I've adjusted the links in the makefile and checked my torch install – it should be working. Am i missing something or does OSX need some specific compiler arguments?

Working in MacOS, I've been able to install:

• Torch
• Matlab and Octave
• CUDA
• cudnn

Download VGG-19, but then on make clean && make I get the following error which has been driving me insane:

ld: symbol(s) not found for architecture x86_64 clang: error: linker command failed with exit code 1 (use -v to see invocation) make: *** [libcuda_utils.so] Error 1

Anyone have this problem or know of a fix?

Hi,

how are you doing the segmentation? I tried to use selective search but the results are poors and doesnt work as I want. :(

Any Ideas?

I load my own photos with own segmentations, but in all iterations I have same loss:

Content 1 loss: inf
Style 1 loss: inf
Style 2 loss: inf
Style 3 loss: inf
Style 4 loss: inf
Style 5 loss: nan
Total loss: nan

Do you have any ideas about this problem?

if the input picture is a people's face without making up, and the style is a made up face.
what is the result?

I'd love to experiment with this project, but I'd rather not have to buy Matlab first as it is not free like Torch, Tensorflow, Theano, Caffe, etc...

So is removing the Matlab dependency possible? And if so, how difficult would it be?

how can i run it with nodejs?

I am looking to port the code in PyTorch. I would like to know if it is possible to do so

I'm using input/style images of 700px width. The tmp_result img is relatively high quality, however there's a vertical grey bar in final result:

As the project requires CUDA back-end which is from nvidia, I was wondering if this works with AMD GPU's too?

Can you provide a detailed installation process?

i have generated laplacian file use matlab,but it goes wrong when i'm running gen_all.py,cuda8.0 and cudnn 5.1 have been installed already,so i have no idea about this,and i guess that it should be a nvidia driver configuration problem.

On MacOS after make, I run('gen_laplacian/gen_laplacian.m') using MatLab or Octave in the root folder and get this message.

`Trial>> run('gen_laplacian/gen_laplacian.m')
Warning: Name is nonexistent or not a directory: matting\

In path (line 109)
In addpath (line 88)
In gen_laplacian (line 1)
In run (line 86)
Warning: Name is nonexistent or not a directory: gaimc\
In path (line 109)
In addpath (line 88)
In gen_laplacian (line 2)
In run (line 86)
Working on image index = 1

ans =

437 700 3

Compute Laplacian
Undefined function or variable 'getLaplacian1'.

Error in gen_laplacian (line 20)
A = getLaplacian1(input, zeros(h, w), 1e-7, 1);

Error in run (line 86)
evalin('caller', [script ';']);`

Not sure this path linking is supposed to work as all directories are present.

/home/hckj/torch/install/bin/luajit: /home/hckj/torch/install/share/lua/5.1/trepl/init.lua:389: error loading module 'libcuda_utils' from file './libcuda_utils.so':
libTHC.so.0: cannot open shared object file: No such file or directory
stack traceback:
[C]: in function 'error'
/home/hckj/torch/install/share/lua/5.1/trepl/init.lua:389: in function 'require'
neuralstyle_seg.lua:7: in main chunk
[C]: in function 'dofile'
...hckj/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:150: in main chunk
[C]: at 0x00406670

I got the error on the step2:

WARNING: Skipping content loss	
/torch/install/bin/luajit: deepmatting_seg.lua:291: invalid device function
stack traceback:
	[C]: in function 'matting_laplacian'
	deepmatting_seg.lua:291: in function 'MattingLaplacian'
	deepmatting_seg.lua:253: in function 'opfunc'
	/home/ubuntu/torch/install/share/lua/5.1/optim/lbfgs.lua:66: in function 'lbfgs'
	deepmatting_seg.lua:280: in function 'main'
	deepmatting_seg.lua:606: in main chunk
	[C]: in function 'dofile'
	...untu/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:150: in main chunk
	[C]: at 0x00405d50

Then I installed docker-version of program here, but the error the same:

/root/torch/install/bin/luajit: /root/deep_photo/deepmatting_seg.lua:330: invalid device function
stack traceback:
	[C]: in function 'matting_laplacian'
	/root/deep_photo/deepmatting_seg.lua:330: in function 'MattingLaplacian'
	/root/deep_photo/deepmatting_seg.lua:284: in function 'opfunc'
	/root/torch/install/share/lua/5.1/optim/adam.lua:37: in function 'adam'
	/root/deep_photo/deepmatting_seg.lua:317: in function 'main'
	/root/deep_photo/deepmatting_seg.lua:645: in main chunk
	[C]: in function 'dofile'
	/root/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:150: in main chunk
	[C]: at 0x00406670

It looks like the problem in cuda_utils.cu:

__global__ void matting_laplacian_kernel(
	float *input, float *grad, int h, int w, 
	int *CSR_rowIdx, int *CSR_colIdx, float *CSR_val,
	int N
) { 
	printf(""I'm here now");  // It doesn't output in console
..
 }

int matting_laplacian(lua_State *L) {
...
	printf(""I'm here now");  // It prints

	matting_laplacian_kernel<<<(h*w-1)/TB+1, TB>>>(	
		THCudaTensor_data(state, input),
		THCudaTensor_data(state, grad),
		h, w,
		THCudaIntTensor_data(state, CSR_rowIdx),
		THCudaIntTensor_data(state, CSR_colIdx),
		THCudaTensor_data(state, CSR_val),
		N
	);

	checkCudaError(L); // It outputs error

I'd very appreciate any help!

I came across this very interesting github project. I've been working around with jcjohnson's project a lot, but this seams promising too ;)

The only Problem is that i ran into the "out of memory" - problem...

This is my input:
th neuralstyle_seg.lua -content_image PersonalTest/face_man_eye.png -style_image PersonalTest/style.png -content_seg PersonalTest/black.png -style_seg PersonalTest/black.png -index 1 -num_iterations 100 -save_iter 25 -print_iter 1 -backend cudnn -gpu 0

and then this cames along:

THCudaCheck FAIL file=/home/ubuntu/torch/extra/cutorch/lib/THC/generic/THCStorage.cu line=66 error=2 : out of memory
/home/ubuntu/torch/install/bin/luajit: /home/ubuntu/torch/install/share/lua/5.1/nn/utils.lua:11: cuda runtime error (2) : out of memory at /home/ubuntu/torch/extra/cutorch/lib/THC/generic/THCStorage.cu:66
stack traceback:
	[C]: in function 'resize'
	/home/ubuntu/torch/install/share/lua/5.1/nn/utils.lua:11: in function 'torch_Storage_type'
	/home/ubuntu/torch/install/share/lua/5.1/nn/utils.lua:57: in function 'recursiveType'
	/home/ubuntu/torch/install/share/lua/5.1/nn/Module.lua:160: in function 'type'
	/home/ubuntu/torch/install/share/lua/5.1/nn/utils.lua:45: in function 'recursiveType'
	/home/ubuntu/torch/install/share/lua/5.1/nn/utils.lua:41: in function 'recursiveType'
	/home/ubuntu/torch/install/share/lua/5.1/nn/Module.lua:160: in function 'cuda'
	...untu/torch/install/share/lua/5.1/loadcaffe/loadcaffe.lua:46: in function 'load'
	neuralstyle_seg.lua:98: in function 'main'
	neuralstyle_seg.lua:546: in main chunk
	[C]: in function 'dofile'
	...untu/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:150: in main chunk
	[C]: at 0x00405d50

The image size is by the way 100 x 100 pixels and i've a GTX 660, which should be fine with that.

Thank you in advance for your help.

Hi

I have noticed that there is a huge difference in size and resolution between the content file and style file in the /example/high_res folder. And I have the question that

Is there any difference in the process of style-transferring between common figures in /example/input, /example/style folders and those high-resolution figures?

I am sorry, but I fail to see any comment on this topic in the paper. So could anyone please help me explain that?

Hello, it's me again :-)

Unfortunately the compiled libcuda_utils.so is still not working as intended.

When i import it via torch:

th

  ______             __   |  Torch7
 /_  __/__  ________/ /   |  Scientific computing for Lua.
  / / / _ \/ __/ __/ _ \  |  Type ? for help
 /_/  \___/_/  \__/_//_/  |  https://github.com/torch
                          |  http://torch.ch

th> require 'libcuda_utils'
Segmentation fault: 11

torch abruptly quits with the error Segmentation fault: 11.

When i try to run the first lua script, the same thing happens:

th neuralstyle_seg.lua -content_image examples/input/in1.png -style_image examples/style/tar1.png -content_seg examples/segmentation/in1.png -style_seg examples/segmentation/tar1.png -index 1 -num_iterations 1000 -save_iter 100 -print_iter 1 -gpu 0 -serial examples/tmp_results
Segmentation fault: 11

And the process luajit also crashes.

If it is of any help, i've tried to get some info from my libcuda_utils.so via nm:

Nm displays the name list (symbol table) of each object file in the argument list. If an argument is an archive, a listing for each object file in
the archive will be produced. File can be of the form libx.a(x.o), in which case only symbols from that member of the object file are listed.

I tried the -a flag:

-a Display all symbol table entries, including those inserted for use by debuggers.

This is the output (too long to post it here): http://pastebin.com/J2PKBFvX

Could you please run nm -a [pathTo-libcuda_utils.so] on your file and compare your output to mine?
But just if this is not too tedious for you to check and only if it even helps in this case.

Do you have any ideas how i can check what is causing the Segmentation fault?
 
I guess that OSX needs more specific compiler options... something is still going wrong when compiling libcuda_utils.so, even though i get no errors.

Man, i was so excited to finally test it with my own images. The matlab code was executing fine, the 60 Laplacian-.mat files are ready to test. The only missing thing is getting the lua code to run in torch without crashing...

PS: My first idea to test the code would be using, say 60-120 still images from a 24h-timelapse sequence as style images and then apply them to a similar image that i have taken. Any kind of landscape – doesn't matter, as long as input and style are somewhat similar. Then take the transformed output images, animate them and watch how my image changes to different lighting scenarios from the timelapse. I won't give up till i see the result of this :-).

Someone who successfully runned an algorithm, please share a docker container image?

Is there a direct replacement for getLaplacian1() in python? I'm converting genlaplacian.m to python

Anyone have used deep-photo-style for art neural style?
I hope you can share the result for us.Let's test it together

It should be noted that Octave 4.0 is required, and the Image package must be installed (pkg install <archive> and pkg load image from the Octave prompt) to run the gen_laplacian.m script. Furthermore, the getLaplacian1 function is not found, since the addpath commands seem to have erroneous trailing slashes.

When that is done, the program doesn't seem to terminate:

octave:1> pkg load image
octave:2> run('gen_laplacian.m')
Working on image index = 1
warning: your version of GraphicsMagick limits images to 8 bits per pixel

on abortion, some output is printed, proving that it does compute something (takes hours for 6 images), but the log messages aren't displayed when printed apparently.

Is it normal that this takes so long?

Tested on Ubuntu 14.04 on an AWS P2 instance.

I would like to apply this algorithm to a new image with a new style (so the segmentations don't exist under examples/ folder). Therefore, I am trying to create segmentations that is required by the algorithm.

The Github page notes: "In the main paper we generate all comparison results using automatic scene segmenation algorithm modified from DilatedNet. Manual segmentation enables more diverse tasks hence we provide the masks in examples/segmentation/."

My questions are:

• which library did you exactly used for the segmentation? (I guess it is different from this one: https://github.com/fyu/dilation) since it says 'modified'?
• i am a bit confused by 'manual segmentation'? are the masks under examples/segmentation created manually? (if so, I appreciate if you can suggest some pointers on how to mark these segments manually, e.g. any interface that i can use) or were examples/segmentation masks created using dilanet?

Thanks,
Penny

examples/input/in23.png and examples/style/tar23.png are the same picture.

$ sha256sum in23.png tar23.png
1889c0094c038a11f7cee6afb3bda075b33bbcea090086539f3db81534f85fec  in23.png
1889c0094c038a11f7cee6afb3bda075b33bbcea090086539f3db81534f85fec  tar23.png

Is that intentional?

Instead of calculate matting laplacian in lua codes, you are calculating matting laplacian through cuda_utils.cu.
Is it for CUDA acceleration? I am just curious.
Thanks!

As people requested, here are some automatic or manual tools for creating a segmentation mask for a photo image:

Automatic:

1. MIT Scene Parsing: http://sceneparsing.csail.mit.edu/
2. SuperParsing: http://www.cs.unc.edu/~jtighe/Papers/ECCV10/
3. Nonparametric Scene Parsing: http://people.csail.mit.edu/celiu/LabelTransfer/
4. Berkeley: https://www2.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/resources.html
5. CRF-RNN for Semantic Image Segmentation: https://github.com/torrvision/crfasrnn
6. Selective Search: https://github.com/belltailjp/selective_search_py

Manual:

1. Photoshop Quick Selection Tool: https://helpx.adobe.com/photoshop/using/making-quick-selections.html

I have only one image in the input folder (style has a different size):

$ identify examples/*/*
examples/input/in1.png PNG 1000x750 1000x750+0+0 8-bit sRGB 970KB 0.000u 0:00.009
examples/segmentation/in1.png PNG 1000x750 1000x750+0+0 8-bit sRGB 4.09KB 0.000u 0:00.000
examples/segmentation/tar1.png PNG 700x393 700x393+0+0 8-bit sRGB 938B 0.000u 0:00.000
examples/style/tar1.png PNG 700x393 700x393+0+0 8-bit sRGB 586KB 0.000u 0:00.000

Segmentation masks are completely black, just like for example in9, and I believe I should have enough GPU horsepower for this:

$ nvidia-smi
Wed Mar 29 11:26:09 2017       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 375.26                 Driver Version: 375.26                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  TITAN X (Pascal)    Off  | 0000:02:00.0     Off |                  N/A |
| 23%   24C    P8     9W / 250W |      0MiB / 12189MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  TITAN X (Pascal)    Off  | 0000:03:00.0     Off |                  N/A |
| 23%   30C    P8    10W / 250W |      0MiB / 12189MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  TITAN X (Pascal)    Off  | 0000:82:00.0     Off |                  N/A |
| 23%   23C    P8     8W / 250W |      0MiB / 12189MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  TITAN X (Pascal)    Off  | 0000:83:00.0     Off |                  N/A |
| 23%   27C    P8     9W / 250W |      0MiB / 12189MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

But I get out of memory errors whenever trying to run gen_all.py (with modified variables for 1 image and 1 GPU) or the th script directly:

$ python gen_all.py 
working on image pair index = 1
gpu, idx = 	0	1	
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:604] Reading dangerously large protocol message.  If the message turns out to be larger than 1073741824 bytes, parsing will be halted for security reasons.  To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:81] The total number of bytes read was 574671192
Successfully loaded models/VGG_ILSVRC_19_layers.caffemodel
conv1_1: 64 3 3 3
conv1_2: 64 64 3 3
conv2_1: 128 64 3 3
conv2_2: 128 128 3 3
conv3_1: 256 128 3 3
conv3_2: 256 256 3 3
conv3_3: 256 256 3 3
conv3_4: 256 256 3 3
conv4_1: 512 256 3 3
conv4_2: 512 512 3 3
conv4_3: 512 512 3 3
conv4_4: 512 512 3 3
conv5_1: 512 512 3 3
conv5_2: 512 512 3 3
conv5_3: 512 512 3 3
conv5_4: 512 512 3 3
fc6: 1 1 25088 4096
fc7: 1 1 4096 4096
fc8: 1 1 4096 1000
Exp serial:	examples/tmp_results	
Setting up style layer  	2	:	relu1_1	
Setting up style layer  	7	:	relu2_1	
Setting up style layer  	12	:	relu3_1	
Setting up style layer  	21	:	relu4_1	
Setting up content layer	23	:	relu4_2	
Setting up style layer  	30	:	relu5_1	
WARNING: Skipping content loss	
THCudaCheck FAIL file=/tmp/luarocks_cutorch-scm-1-6378/cutorch/lib/THC/generic/THCStorage.cu line=66 error=2 : out of memory
.../torch/install/bin/luajit: .../torch/install/share/lua/5.1/nn/Container.lua:67: 
In 4 module of nn.Sequential:
cuda runtime error (2) : out of memory at /tmp/luarocks_cutorch-scm-1-6378/cutorch/lib/THC/generic/THCStorage.cu:66
stack traceback:
	[C]: at 0x7f1263b53940
	[C]: in function 'cmul'
	neuralstyle_seg.lua:465: in function <neuralstyle_seg.lua:456>
	[C]: in function 'xpcall'
	.../torch/install/share/lua/5.1/nn/Container.lua:63: in function 'rethrowErrors'
	.../torch/install/share/lua/5.1/nn/Sequential.lua:55: in function 'updateGradInput'
	neuralstyle_seg.lua:232: in function 'opfunc'
	.../torch/install/share/lua/5.1/optim/lbfgs.lua:66: in function 'lbfgs'
	neuralstyle_seg.lua:253: in function 'main'
	neuralstyle_seg.lua:546: in main chunk
	[C]: in function 'dofile'
	.../torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:150: in main chunk
	[C]: at 0x55b757411610

WARNING: If you see a stack trace below, it doesn't point to the place where this error occurred. Please use only the one above.
stack traceback:
	[C]: in function 'error'
	.../torch/install/share/lua/5.1/nn/Container.lua:67: in function 'rethrowErrors'
	.../torch/install/share/lua/5.1/nn/Sequential.lua:55: in function 'updateGradInput'
	neuralstyle_seg.lua:232: in function 'opfunc'
	.../torch/install/share/lua/5.1/optim/lbfgs.lua:66: in function 'lbfgs'
	neuralstyle_seg.lua:253: in function 'main'
	neuralstyle_seg.lua:546: in main chunk
	[C]: in function 'dofile'
	.../torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:150: in main chunk
	[C]: at 0x55b757411610

As far as I know, I followed the procedure in README, so this seems to be either a bug or there are something undocumented that I should do before running gen_all.py.

someone help me!

Undefined symbols for architecture x86_64:
  "_luaL_checknumber", referenced from:
      matting_laplacian(lua_State*) in tmpxft_000037a4_00000000-16_cuda_utils.o
      smooth_local_affine(lua_State*) in tmpxft_000037a4_00000000-16_cuda_utils.o
  "_luaL_error", referenced from:
      checkCudaError(lua_State*) in tmpxft_000037a4_00000000-16_cuda_utils.o
      matting_laplacian(lua_State*) in tmpxft_000037a4_00000000-16_cuda_utils.o
      smooth_local_affine(lua_State*) in tmpxft_000037a4_00000000-16_cuda_utils.o
  "_luaL_openlib", referenced from:
      _luaopen_libcuda_utils in tmpxft_000037a4_00000000-16_cuda_utils.o
  "_lua_call", referenced from:
      getCutorchState(lua_State*) in tmpxft_000037a4_00000000-16_cuda_utils.o
      matting_laplacian(lua_State*) in tmpxft_000037a4_00000000-16_cuda_utils.o
      smooth_local_affine(lua_State*) in tmpxft_000037a4_00000000-16_cuda_utils.o
  "_lua_getfield", referenced from:
      getCutorchState(lua_State*) in tmpxft_000037a4_00000000-16_cuda_utils.o
      matting_laplacian(lua_State*) in tmpxft_000037a4_00000000-16_cuda_utils.o
      smooth_local_affine(lua_State*) in tmpxft_000037a4_00000000-16_cuda_utils.o
  "_lua_settop", referenced from:
      getCutorchState(lua_State*) in tmpxft_000037a4_00000000-16_cuda_utils.o
      matting_laplacian(lua_State*) in tmpxft_000037a4_00000000-16_cuda_utils.o
      smooth_local_affine(lua_State*) in tmpxft_000037a4_00000000-16_cuda_utils.o
  "_lua_touserdata", referenced from:
      getCutorchState(lua_State*) in tmpxft_000037a4_00000000-16_cuda_utils.o
      matting_laplacian(lua_State*) in tmpxft_000037a4_00000000-16_cuda_utils.o
      smooth_local_affine(lua_State*) in tmpxft_000037a4_00000000-16_cuda_utils.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [libcuda_utils.so] Error 1

I was just wondering if you would ever consider making an option to use cltorch for those that don't have an Nvidia gpu, or even removing the need for a gpu so that it would be cpu based.

I've already mentioned this here: #28
But i thought it would be better to add a separate issue for it.

Mask colors

A quick question about the colors you can use for your mask pngs: In neuralstyle_seg.lua and deepmatting_seg.lua you define that these colors are processed from the segmentation images:

`local color_codes = {'blue', 'green', 'black', 'white', 'red', 'yellow', 'grey', 'lightblue', 'purple'}`

The problem for me (and maybe others users too) was to recognize that you are restricted to these colors when you "paint" your masks. At first i didn't look at the code and just selected random colors in Photoshop – i only realized that something was wrong when the generated output didn't match my selections.

So could you maybe include a short info in your Readme about which colors you can use (maybe RGB + hex values?) when you manually create your masks?

I'm not sure if everyone automatically knows what colors are actually processed.

Thanks!

"4 GB of GDDR5 memory" should mean I can support the 3 GB of GPU needed, right? If not, is there a way to decrease the GPU needed?

I think my drivers etc. are up to date, and I already did a lot of processes to fix bugs, like the library path, the version of CUDA... And I used the optional -backend cudnn and -cudnn_autotune.

Also: each time I restart my terminal, I have to write export CUDNN_PATH=/usr/local/cuda/lib64/libcudnn.so.5 , I'm new to Ubuntu, does someone know how to set this path once and for all?

And, unlike this comment suggested, I don't run Lua 5.2 but a previous version (5.1, I think).

Might be double thread, I'm ready to merge it with the thread above if necessary.