The idea is to turn the extrapolation problem into an hybrid interpolation/extrapolation one.

The Relu/Leaky Relu NN has shown some difficulties to extrapolate over long distances away from the pictures edges. In an attempt to "keep the NN alive" for a larger distance, perhaps we could train the same NN on an augmented training set that includes a portion of the inside circle which surrounds the Droste picture - see attached picture for a visual example of what we would train it on.

So we don't force the net to learn the curvature

https://arxiv.org/abs/2006.09661

Earlier extrapolation attempts with Relu, Leaky Relu and SIREN (sinusoidal) activations have performed poorly in regions more than 10% outside of the original picture range.

We suspect this is due to strong gradients (for Relu/Leaky Relu) and oscillations (SIREN) causing the function learnt by the NN to diverge to negative (floored to 0 eventually) and large positve (floored to 255 eventually, or 1 if in floating scale) values.

This results in large patches of white/black outside the painting when we try to extrapolate, which are not very helpful.

One idea is to try an activation function that naturally has vanishing gradients, in an attempt to mitigate this diverging behaviour. Tahn and sigmoids are good first candidates for this.

• Tiles

• Columns

• Some water

Pencilize it before passing it to the GAN

to use smaller sections of the painting itself as an image database. In our example, the extra buildings needed could be generated by looking at the existing buildings on the lithography first, before searching for additional building images on the training database

CNNs seem to have an extrapolation power that SIREN/ Feed FWDs doesn't have

If we can use the CNN as a function that deforms the space and SIREN to give us the content....

The idea is to:

1. Train a NN/ML algo in such a way that the input is a (x, y) coordinate tuple and the output is an (R, G, B) colour tuple

2. The code should take care of the twisting and shrinking - when extrapolating outside of the learnt picture, the spiral effect has to be applied again, and this can be coded in the way (x, y) coordinates are supplied to the NN

3. As we don't have a functional form for the twisting/shrinking, we can try different hypotheses (e.g. the shrinking and rotation increase linearly going from the outer edge of the known painting until the edge of the Droste shrunk/rotated/pasted painting)

The idea is to make a paper explaining everything we have learned about the grid.

This will help us later to save space on our main paper.

It has happened that people are not familiar with the grid as it requires a lot of prior knowledge and it's very easy to get confused. So a long paper dedicated to the grid will help us make the mind of the reader.

The paper can be as long as we want, aimed to the Arxiv only, or maybe sent to an art journal.

It can definitely be part of a website dedicated to the project!

• Use that rotation to straighten the image
• Complete the straighten image

This will ensure us that the final curvature is correct

Some additional thoughts, which may head in a different direction, and which I mention as thoughts on your overall project, rather than on this particular grant application:

•      How will you know if you’ve convincingly completed the Escher?  (I’d love the idea of an analogue of the Turing Test.)
  

•      Could you introduce an Escher style to something along the lines of the Deep Dream generator?  That, if I understand it, uses a reference image as a style source, but I wouldn’t think it seeks to extract or preserve geometric properties in the way that Escher does.
  

•      Related to the above: what if Escher had continued being Escher, rather than dying?  Can you generate new Eschers?  (My thought here might be kicked off by Taylor, Micolich and Jonas’ 1999 Nature article, which dated Pollock paintings by fitting a fractal number to them.  Is that a variable that generally increased through Escher’s work, that you could continue to project out?)
  

1. You could simply do pruning after the pre-training stage

2. You can maybe also try a student-teacher network to reduce ur model parameters ? https://towardsdatascience.com/knowledge-distillation-simplified-dd4973dbc764

Then someone asks this question, which I don't know how we did it:

whats the finetuning strategy? Arey you just training the head of the model or unfreezing everything?
Then this other solution that sounds interesting but I need to think about it:

if you dont want to remove the original feature detection, then create a second decoder (classification/seg/whatever) which accepts the output of the encoder.
you can train that seperately and then combine them at production, or you can freeze all the layers, attach the second decoder (unfrozen), and train from that.
He even made a code for us!
class ComboNetB0(nn.Module):
def init(self, fc1_n_classes, fc2_n_classes, **override_params):
super().init()
# Create a pretrained EfficientNet model from efficientnet_pytorch package
self.network = EfficientNet.from_pretrained("efficientnet-b0", num_classes=fc1_n_classes, **override_params)
# Pretained model will create self.network._fc which is the pretrained final layer, looks something like this:
# self.network._fc = nn.Sequential(nn.Linear(self.network._fc.in_features, 512),
# nn.ReLU(),
# nn.Dropout(0.25),
# nn.Linear(512, 128),
# nn.ReLU(),
# nn.Dropout(0.50),
# nn.Linear(128,num_classes))
# Create a second final layer
self.network._fc2 = nn.Sequential(nn.Linear(self.network._fc.in_features, 512),
nn.ReLU(),
nn.Dropout(0.25),
nn.Linear(512, 128),
nn.ReLU(),
nn.Dropout(0.50),
nn.Linear(128,fc2_n_classes))
def forward(self,x):
features = self.network.extract_features(img)
out1 = self.network._fc(features) # logits of the pretrained classification set
out2 = self.network._fc2(features) # logits of your second classification set
return torch.cat((out1, out2), 0)

The current interpolator does not work

After it is fixed, we need to generate a new correct grid, since the inner squares are difficult to cut.

1. VAE-GANS
2. Coco GANS
3. image GPT