szymonmaszke / torchlayers Goto Github PK

Are there any benchmarks available for the SeparableConv? Specifically, PyTorch's separable convolutions implementation is known be to very slow (see this, this and this). From my experience, reducing the number of parameters in the CNN 3 times with separable convolutions did not reduce the training time at all because of this issue. Does Torchlayers support faster separable convolutions, or is SeparableConv just a convenience layer?

torchlayers is currently not usable on Python 3.6, specifically Python 3.6.9 on Google Colab. Installing from github repo gave me this:

ERROR: Package 'torchlayers' requires a different Python: 3.6.9 not in '>=3.7

Using pip install torchlayers, I got torchlayers 0.1.0 but there are no Conv or Dense or even ReLU in torchlayers imported module.

Btw, thank you for the brilliant library.

is there any possibility for the 'same' padding convolution to work on even values of kernel size, stride, and dilation?

Most Python numerical libraires define an official alias, like np for numpy, because the full name is long and noisy. It's important to discourage the bad practice of import *, and to increase code reusability. In pytorch, we mainly use nn that makes the job.

I propose tl, as it's short, clear, and to my knowledge not used in numerical python.

What do you think about using import torchlayers as tl in all your examples?

Hi,
I have a relatively straightforward situation where I need to validate my input shapes while jit.loading a saved model and I can't seem to find the solution for.

save a traced model and verify inputs shapes:
test_model = torchvision.models.resnet.resnet18(pretrained=True).eval()
with torch.no_grad():
    inp_224 = torch.rand(1, 3, 224, 224, dtype=torch.float)
    script_module_224 = torch.jit.trace(test_model, inp_224)
    graph_inputs = list(script_module_224.graph.inputs())
    graph_inputs = graph_inputs[1:] # ignore self
    print(graph_inputs[0].type().sizes()) # [1, 3, 224, 224]
    script_module_224.save("saved_model_224.pt")

load the same model cannot validate the traced input shapes:

# ... later elsewhere load my saved model
loaded_224 = torch.jit.load("saved_model_224.pt")
# there's nothing preventing me from sending incorrect input shapes
# i.e., traced with 224 but called with 500
inp_500 = torch.rand(1, 3, 500, 500, dtype=torch.float)
loaded_224(inp_500)

I'd like to prevent feeding incorrect shaped inputs after loading models.
In particular, I would like to be able to do something like:

traced_input_shape = loaded_224.get_input_shape()
if (inp_500.shape() != traced_input_shape()):
    print("Error: Trying to run Inference with Incorrect Shaped Inputs!")
    # die

I tried using torchlayers to help with this situation by:

import torchlayers as tl
t.build(loaded_224, inp_224)

This failed (reasonably) with:

PickleError: ScriptModules cannot be deepcopied using copy.deepcopy or saved using torch.save. Mixed serialization of script and non-script modules is not supported. For purely script modules use my_script_module.save(<filename>) instead.

Any recommendations?

Hi

I am trying to build the following:

c = nn.Sequential(
        tl.Conv2d(32, kernel_size=8, stride=4),
        nn.ReLU(),
        tl.Conv2d(64, kernel_size=4, stride=2),
        nn.ReLU(),
        tl.Conv2d(64, kernel_size=3, stride=1),
        nn.ReLU(),
        tl.Linear(512),
        nn.ReLU()
    )

tl.build(c, torch.randn(1, 3, 84, 84))

but I am getting

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/stelios/anaconda3/envs/HM/lib/python3.7/site-packages/torchlayers/__init__.py", line 67, in build
    module(*args, **kwargs)
  File "/home/stelios/anaconda3/envs/HM/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/stelios/anaconda3/envs/HM/lib/python3.7/site-packages/torch/nn/modules/container.py", line 92, in forward
    input = module(input)
  File "/home/stelios/anaconda3/envs/HM/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/stelios/anaconda3/envs/HM/lib/python3.7/site-packages/torchlayers/_dev_utils/infer.py", line 214, in forward
    return infered_module(inputs, *args, **kwargs)
  File "/home/stelios/anaconda3/envs/HM/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/stelios/anaconda3/envs/HM/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 87, in forward
    return F.linear(input, self.weight, self.bias)
  File "/home/stelios/anaconda3/envs/HM/lib/python3.7/site-packages/torch/nn/functional.py", line 1372, in linear
    output = input.matmul(weight.t())
RuntimeError: size mismatch, m1: [448 x 7], m2: [64 x 512] at /tmp/pip-req-build-7mav6f4d/aten/src/TH/generic/THTensorMath.cpp:197

It works just fine with flatten right before the linear.

Just wondering if still maintained...

my torch version is 1.8.1+cpu
but does this issue means my torch version is below 1.3.0?

ERROR: Could not find a version that satisfies the requirement torch>=1.3.0 (from torchlayers) (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)
ERROR: No matching distribution found for torch>=1.3.0

.
.
.
I created new venv and it worked! thanks

Hey!
I have the following issue.

Using Python 3.9.7 and torchlayers 0.1.1.

Trying to do this:

import torchlayers as tl

        # Regularize all parameters of Linear module
        regularized_layer = tl.L1(tl.Linear(30), weight_decay=1e-5

From here.

Unfortunately, I am getting this:

AttributeError: module torchlayers has no attribute L1

If I inspect torchlayers.regularization it does not contain L1. 🤔

Any idea? Thanks in advance.

The current implementation of WeightDecay uses param.grad = self.regularize(param) to calculate the decayed gradient, which has two problems.
First, the decayed gradient is incorrect because the original gradient is replaced instead of having a value added to it. The correct answer should be param.grad += self.regularize(param).
Second, the PyTorch hook API does not allow in-place modification but instead allows the hook to optionally return a new gradient. I think that returning he new gradient value would be correct.

Hi!
I implement the code pip install --user torchlayers in anaconda prompt

Then I get the error as follow:

ERROR: Could not find a version that satisfies the requirement torchlayers (from versions: none) ERROR: No matching distribution found for torchlayers

Would you please help me out?

If you need more detailed information,please tell me!

I appreciate it!

Hi,
Got the error below:

AttributeError Traceback (most recent call last)
in
----> 1 print(Autoencoder())

in init(self)
8 nn.Conv1d(out_channels,in_channels,kernel_size),
9 nn.ReLU(True),
---> 10 torchlayers.Linear(2))

AttributeError: module 'torchlayers' has no attribute 'Linear'

Hi ,

The following model works without the tl layer:

class Autoencoder(nn.Module):
    def __init__(self):
        super(
            Autoencoder, self
        ).__init__()  # This should apply Uniform random values to weights and biases.

        self.encoder = nn.Sequential(
            nn.Conv1d(in_channels, out_channels, kernel_size),
            nn.ReLU(True),
            nn.Conv1d(out_channels, in_channels, kernel_size),
            nn.ReLU(True),
            tl.Linear(2),
        )

        self.decoder = nn.Sequential(
            nn.ConvTranspose1d(in_channels, out_channels, kernel_size),
            nn.ReLU(True),
            nn.ConvTranspose1d(out_channels, in_channels, kernel_size),
        )

    def forward(self, x):
        encoded = self.encoder(x)
        decoded = self.decoder(encoded)
        return decoded, encoded

I have added tl.Linear(2) as the compressed layer between the encoder and decoder.

Autoencoder_tl = tl.build(Autoencoder(), Train_x_t_e) will throw this error:

Train_x_t_e.shape ---> torch.Size([210, 719, 11])

RuntimeErrorTraceback (most recent call last)
in
----> 1 Autoencoder_tl = tl.build(Autoencoder(), Train_x_t_e)

~/.local/lib/python3.7/site-packages/torchlayers/init.py in build(module, *args, **kwargs)
65 with torch.no_grad():
66 module.eval()
---> 67 module(*args, **kwargs)
68 module.train()
69 module = torch_compile(module)

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in call(self, *input, **kwargs)
548 result = self._slow_forward(*input, **kwargs)
549 else:
--> 550 result = self.forward(*input, **kwargs)
551 for hook in self._forward_hooks.values():
552 hook_result = hook(self, input, result)

in forward(self, x)
18
19 def forward(self,x):
---> 20 encoded = self.encoder(x)
21 decoded = self.decoder(encoded)
22 return decoded, encoded

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in call(self, *input, **kwargs)
548 result = self._slow_forward(*input, **kwargs)
549 else:
--> 550 result = self.forward(*input, **kwargs)
551 for hook in self._forward_hooks.values():
552 hook_result = hook(self, input, result)

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/container.py in forward(self, input)
98 def forward(self, input):
99 for module in self:
--> 100 input = module(input)
101 return input
102

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in call(self, *input, **kwargs)
548 result = self._slow_forward(*input, **kwargs)
549 else:
--> 550 result = self.forward(*input, **kwargs)
551 for hook in self._forward_hooks.values():
552 hook_result = hook(self, input, result)

~/.local/lib/python3.7/site-packages/torchlayers/_dev_utils/infer.py in forward(self, inputs, *args, **kwargs)
212 infered_module = getattr(self, module)
213
--> 214 return infered_module(inputs, *args, **kwargs)
215
216 return forward

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in call(self, *input, **kwargs)
548 result = self._slow_forward(*input, **kwargs)
549 else:
--> 550 result = self.forward(*input, **kwargs)
551 for hook in self._forward_hooks.values():
552 hook_result = hook(self, input, result)

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/linear.py in forward(self, input)
85
86 def forward(self, input):
---> 87 return F.linear(input, self.weight, self.bias)
88
89 def extra_repr(self):

/opt/conda/lib/python3.7/site-packages/torch/nn/functional.py in linear(input, weight, bias)
1610 ret = torch.addmm(bias, input, weight.t())
1611 else:
-> 1612 output = input.matmul(weight.t())
1613 if bias is not None:
1614 output += bias

RuntimeError: size mismatch, m1: [150990 x 7], m2: [719 x 2] at /pytorch/aten/src/TH/generic/THTensorMath.cpp:41

I was trying out the basic example given in the repo description in Google Colab and it is throwing an error.

---------------------------------------------------------------------------

AttributeError                            Traceback (most recent call last)

<ipython-input-8-d2b2c25b5f66> in <module>()
      1 model = torch.nn.Sequential(
----> 2     tl.Conv(64),  # specify ONLY out_channels
      3     torch.nn.ReLU(),  # use torch.nn wherever you wish
      4     tl.BatchNorm(),  # BatchNormNd inferred from input
      5     tl.Conv(128),  # Default kernel_size equal to 3

AttributeError: module 'torchlayers' has no attribute 'Conv'

Here's the code,

Sorry to open an issue for such a trivial thing but in the readme, I think there is a small typo where after torchlayers.build() mnist model has conv1d layers and text classification model has conv2d layers