nareshps / doi-ml Goto Github PK

Apps I Need

• Note-taking application with time sorted items.
• Ideas sketching.
• Map with route recording.
• Word games to learn new words in a fun way.

Introduce a hidden dimension for cell architecture search.

Types of information capture blocks in seq2seq networks:

1- Essence Blocks
2- Relationship Blocks
3- Generalization Blocks
4- Specialization Blocks

Essence Blocks

They can be composed off:

• Feed-forward layers.
• Specialised information capture layers such as convolution, embedding etc.

Relationship Blocks

They can composed off:

• Attention Layers

Generalization Blocks

They generalize inputs into broad topics. Composition choices: ??

Specialization Blocks

They specialize topics into sub-topics. Composition choices: ??

A strategy for premature deaths of the children with the following goals:

• We want the under-performers and over-achievers to die with a high probability. In a evolutionary environment, they are the misfits.
• We want the average performers to have a higher change of survival.

This idea is more likely to work in a collaborative setting. In an individual setting, better performers have higher chance of survival. That brings us to the next question. How would we model collaboration?

A step higher from point mutation

Networks with layers influencing multiple sections of the computation graph. Questions:

• Does TF allows such an architecture?
• How would the gradient updates be handles?
  • Add all the updates for the same layer?
  • Delta = (Mod(Min)-Mod(Max))/2. Sign(Delta) + Min

• Parent's piece in child

Create two children in each cycle: a rich child and a poor child with an occasional merge.

Split input sequence into blocks. Each blocks is processed by two self-attentions. The blocks themselves are transformed into a second level sequence which use the two self-attentions. One self-attention is shared between the levels.

References

• Transformer-XL (Dai et al., 2019)