Is it possible to merge two different input layers into one? about keras HOT 21 CLOSED

Non-sequential models are not supported at this time. But caption generation can be implemented with a sequential model; for instance see: http://keras.io/examples/#architecture-for-learning-image-captions-with-a-convnet-and-a-gated-recurrent-unit

from keras.

Thanks @fchollet. But I think jointly learning image embedding, word embedding and the neural language model is more natural.

from keras.

I wrote a short code to try that.
Here I want to concatenate two layers' output, lets say A (n by m) and B (n by t by m), A is 2d array and B is 3d array, and I want to combine them together by first reshapeing A to (n by 1 by m) and then concatenating them to get C (n by (t+1) by m)
But there is a problem. Could you please give me some hints on this?

import theano
import theano.tensor as T
import numpy as np

from keras.layers.core import Layer

class MultiIncomingLayer(Layer):
#incomings : a list of :class:Layer instances
def init(self, incoming_layers, axis, **kwargs):
self.incoming_layers = [None if isinstance(incoming_layer, tuple)
else incoming_layer
for incoming_layer in incoming_layers]
self.axis = axis
self.params = [p for p in incoming_layer.params for incoming_layer in incoming_layers]

def _concatenate(self,train):
    layer_inputs = [incoming_layer.output(train=train) for
                    incoming_layer in self.incoming_layers]
    #print layer_inputs[0].ndim, layer_inputs[1].ndim
    layer_inputs[0] = T.reshape(layer_inputs[0], (layer_inputs[0].shape[0], 1, layer_inputs[0].shape[1])) 
    new_input = T.concatenate(layer_inputs, axis=self.axis)
    #print new_input.ndim

def output(self, train):
    output = self._concatenate(train)
    return output

from keras.

Preface: I have not read all of your code through, or tried to replicate it in my instance. But, the following is still applicable.

The Sequential model compiles layers in sequentially, thus the name. Therefore, if you have something along the lines of the below pseudocode:

model = Sequential
model.add(ImageEmbedding)
model.add(WordEmbedding)
model.add(MultiIncomingLayer)

Then the output of the ImageEmbedding layer is input for the WordEmbedding layer, etc. As such, your Concatenate layer would get the wrong dimensionality. Unless you modified the Sequential model (and did not provide that code) or created a new model type, we can't pinpoint the "problem"; not to mention the fact that you did not provide any info as to what the problem was.

I can give you some hits though, although I would not recommend jumping in:

• You'll need a new model (model in terms of a Keras model, of which Sequential is currently the only one)
• You'd have to enforce a Concatenation layer at each "true" layer, such that the compile function treats the layers between consecutive Concatenation layers as those that should be concatenated.
  • Therefore, a simple feed-forward dense layer would need to be prefixed and postfixed by an additional Concatenate layer to explicitly show that the single Dense layer is in fact one "true" layer.
• You'd need to assert dimensionality agreement. All layers feeding into a Concatenate must have exactly the same dimensions (such that all layers coming in must have shape (a,b,c,d,....) )

from keras.

Thanks @patyork Sorry that I forgot to provide the code that how I use this layer.

model = Sequential()
#image embedding
img_emb_layer = Dense(feat_size, 256)
#word embedding
word_emb_layer = Embedding(vocab_size, 256)
#concatenate image embedding and word embeddings
img_sent_input_layer = MultiIncomingLayer([img_emb_layer, word_emb_layer], axis=1)
model.add(img_sent_input_layer)
model.add(Dropout(0.5))
#LSTM
model.add(LSTM(256, 256, return_sequences=True)) #
model.add(Dropout(0.5))
model.add(TimeDistributedDense(256, vocab_size, activation='softmax')) #

Besides, I also change some places in Sequential class, where I use both cnn feature and a sequence of words as two kinds of inputs

self.X = self.layers[0].incoming_layers[0].input # input of model
self.Y = self.layers[0].incoming_layers[1].input

self._train = theano.function([self.X, self.Y], train_loss,
updates=updates, allow_input_downcast=True)

from keras.

The problem is that there is an error with the first Dropout layer,
'an AttributeError: 'NoneType' object has no attribute 'shape' in dropout layer'
I donot know how this error comes, or maybe my idea about this implementation is not correct as you said. (I do not know much about how to debug the code when using a theano-based package)

from keras.

'an AttributeError: 'NoneType' object has no attribute 'shape' in dropout layer'

Unless I missed a line of code, you did not call model.compile

from keras.

@patyork I did call model.compile after adding TimeDistributedDense layer.
model.compile(loss='categorical_crossentropy_3d', optimizer = rmsprop)
where I extend the categorical_crossentropy to 3d case, but I think I can use mean_squared_error loss instead.

from keras.

@patyork I notice that even with only the code
#image embedding
img_emb_layer = Dense(feat_size, 256)
#word embedding
word_emb_layer = Embedding(vocab_size, 256)
#concatenate image embedding and word embeddings
img_sent_input_layer = MultiIncomingLayer([img_emb_layer, word_emb_layer], axis=1)
model.add(img_sent_input_layer)
I can not get what I want. I think that is because just as you said, I have to change the code for layer class and change the way that connect them together.

Sorry that I did not make my problem clear and bring you so much trouble

from keras.

Now possible through the use of the Merge layer; see: http://keras.io/layers/core/#merge

from keras.

@fchollet Great, man. That is very helpful. Thanks!

from keras.

I had met the same problem. I want to merge two different input layers into one,and I defined the way how they merged. the problem is that I can't put the layer I defined to the model (Sequential or Graph) as the first layer. this occasion is really common in practical applications. I found that the only solution is to define your model with Keras functional API. It is rather complicated and I am waiting for some improvements in keras to solve the problem.

from keras.

hello, I am new todeep learning and keras. I don't know if this is the correct place to post my question. I am trying to build an auto encoder with two inputs as defined in https://blog.keras.io/building-autoencoders-in-keras.html.
The idea is build two seprate encoders , merge them and then use the output for the decoder part. But i am not sure how to use the fit function, for multiple inputs and single output. Any suggestions or references would be very helpful.

from keras.

I built a project using (Merge, merge) layer once, and use share layer in another. The problem I can't find any documentation that explains how merge or share layer works like convolution or max-pooling layers.
I will be grateful if anyone can direct me or suggest some paper to me that helps me to understand how these layers work ???

from keras.

from keras.

from keras.

from keras.

from keras.

from keras.

from keras.

I am looking for a recommendation to merge vector information with CNN output into a regression model.

from keras.