TensorFlow port of PyTorch Image Models (timm) - image models with pretrained weights.
Home Page: https://tfimm.readthedocs.io/en/latest/
License: Apache License 2.0
Hi. First, I want to say that I enjoy this library a lot! Thank you @martinsbruveris for creating it!
I have a question:
I want create a model body using create_model function and add my own classification head. In classification head I want to add another input layer to additional features, call a concatenate layer on last layer of the create_model object and new input layer, and add final dense layer.
Since create_model object is not a Sequential or Functional model object, is there any way I can do that?
I tried using 'model_tfimm.output' or 'model_tfimm.layers[-1].output' calls, because .output call works with Tensorflow models, but it does not seem to work with tfimm models:
dense_1 = tf.keras.layers.Dense(512, activation='relu', name='dense_1')(model_tfimm.layers[-1].output)
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
v:\Git\spellbook\magicClassification.py in <module>
----> 1 dense_1 = tf.keras.layers.Dense(512, activation='relu', name='dense_1')(model_tfimm.layers[-1].output)
~\AppData\Local\Programs\Python\Python37\lib\site-packages\keras\engine\base_layer.py in output(self)
2094 """
2095 if not self._inbound_nodes:
-> 2096 raise AttributeError('Layer ' + self.name + ' has no inbound nodes.')
2097 return self._get_node_attribute_at_index(0, 'output_tensors', 'output')
2098
AttributeError: Layer activation_72 has no inbound nodes.
model_tfimm.output
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
v:\Git\spellbook\magicClassification.py in <module>
----> 1 model_tfimm.output
~\AppData\Local\Programs\Python\Python37\lib\site-packages\keras\engine\base_layer.py in output(self)
2094 """
2095 if not self._inbound_nodes:
-> 2096 raise AttributeError('Layer ' + self.name + ' has no inbound nodes.')
2097 return self._get_node_attribute_at_index(0, 'output_tensors', 'output')
2098
AttributeError: Layer conv_ne_xt_1 has no inbound nodes.
Using Tensorflow Functional API this would like something like this:
model_tfimm = tfimm.create_model(TFIMM_MODEL_NAME, nb_classes=0, pretrained="timm")
feature_extractor = model_tfimm.output
add_input = tf.keras.layers.Input(shape=(NUM_ADD_FEATURES, ), name='input_features_layer')
concat_layer = tf.keras.layers.Concatenate(name='concat_features')([feature_extractor, add_input])
predictions = tf.keras.layers.Dense(NUM_CLASSES, activation=OUTPUT_ACTIVATION)(concat_layer)
model = tf.keras.Model(inputs=[model_tfimm.input, add_input], outputs=predictions)
Any ideas?
Hi, this library is great. I'm wondering, maybe you can tell me this with your experience, what would be your small-list of the "top" essentials of a state-of-art image model these days? Data augmentation of course (mixup?), regularization (layer norm?), EMA, weight decay... I want to get a minimalist competitive ImageNet model working.
Describe the bug
PVT model does not train.
To Reproduce
Steps to reproduce the behaviour:
import tfimm
import tensorflow_datasets as tfds
import tensorflow as tf
def resize_normalize(x, y):
x = tf.image.resize(x, (224, 224)) / 255
return x, y
train_ds = tfds.load('imagenet_v2',
split='test',
as_supervised=True)
train_ds = train_ds.map(resize_normalize).batch(32)
model = tfimm.create_model("pvt_tiny", pretrained=None)
model.compile(optimizer=tf.keras.optimizers.Adam(1e-3), loss="sparse_categorical_crossentropy", metrics=["accuracy"])
model.fit(train_ds)
Epoch 1/5
313/313 [==============================] - 67s 187ms/step - loss: 15.6424 - accuracy: 6.0000e-04
Epoch 2/5
313/313 [==============================] - 60s 191ms/step - loss: 16.2130 - accuracy: 0.0010
Epoch 3/5
313/313 [==============================] - 60s 191ms/step - loss: 16.2144 - accuracy: 0.0010
Epoch 4/5
313/313 [==============================] - 60s 191ms/step - loss: 16.2418 - accuracy: 0.0010
Epoch 5/5
313/313 [==============================] - 60s 191ms/step - loss: 16.2417 - accuracy: 0.0010
Expected behaviour
Convergance of model
Desktop (please complete the following information):
Also note that setting the LR to 1e-4 as the paper does not solve the problem.
When I try to install and import tfimm packages:
!pip install tfimm
import tfimm
This error occurs:
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
/tmp/ipykernel_42/1172421075.py in <module>
3 get_ipython().system('pip install timm')
4 import timm
----> 5 import tfimm
/opt/conda/lib/python3.7/site-packages/tfimm/__init__.py in <module>
----> 1 from . import architectures # noqa: F401
2 from .models.factory import create_model, create_preprocessing # noqa: F401
3 from .models.registry import list_models # noqa: F401
4 from .utils import ( # noqa: F401
5 cached_model_path,
/opt/conda/lib/python3.7/site-packages/tfimm/architectures/__init__.py in <module>
----> 1 from .cait import * # noqa: F401
2 from .convmixer import * # noqa: F401
3 from .convnext import * # noqa: F401
4 from .mlp_mixer import * # noqa: F401
5 from .pit import * # noqa: F401
/opt/conda/lib/python3.7/site-packages/tfimm/architectures/cait.py in <module>
15 from typing import List, Tuple
16
---> 17 import tensorflow as tf
18
19 from tfimm.layers import (
/opt/conda/lib/python3.7/site-packages/tensorflow/__init__.py in <module>
35 import typing as _typing
36
---> 37 from tensorflow.python.tools import module_util as _module_util
38 from tensorflow.python.util.lazy_loader import LazyLoader as _LazyLoader
39
/opt/conda/lib/python3.7/site-packages/tensorflow/python/__init__.py in <module>
35
36 from tensorflow.python import pywrap_tensorflow as _pywrap_tensorflow
---> 37 from tensorflow.python.eager import context
38
39 # pylint: enable=wildcard-import
/opt/conda/lib/python3.7/site-packages/tensorflow/python/eager/context.py in <module>
33 from tensorflow.python import pywrap_tfe
34 from tensorflow.python import tf2
---> 35 from tensorflow.python.client import pywrap_tf_session
36 from tensorflow.python.eager import executor
37 from tensorflow.python.eager import monitoring
/opt/conda/lib/python3.7/site-packages/tensorflow/python/client/pywrap_tf_session.py in <module>
17 # pylint: disable=invalid-import-order,g-bad-import-order, wildcard-import, unused-import
18 from tensorflow.python import pywrap_tensorflow
---> 19 from tensorflow.python.client._pywrap_tf_session import *
20 from tensorflow.python.client._pywrap_tf_session import _TF_SetTarget
21 from tensorflow.python.client._pywrap_tf_session import _TF_SetConfig
ImportError: SystemError: <built-in method __contains__ of dict object at 0x7f9744ee7280> returned a result with an error set
In other repositories I have found that upgrading NumPy solves the problem.
But my NumPy has the latest version.
Can anyone please help me to solve this error?
when i install tfimm, it need tensorflow version <3.0,>=2.5 , please support tf version 2.0-2.4
Right now the preprocessing function does not account for uniform images. This can results in bugs which are hard to track.
I would suggest replacing this function, by the normalisation generally done in tensorflow (Normalisation account for std =0).
I am using this issue to think through the options of adding support for LoRa.
The first fundamental question is: Can we do this without changing the model code itself? I.e., let's assume we are not allowed to touch convnext.py. How would we proceed?
Ultimately our goal would be to replace (all? some?) tf.keras.layers.Conv2D layers with our new LoraConv2D layers. We could try and do this via monkey patching.
Note: All code samples are based on the convnext-tag branch.
import tensorflow as tf
from tfimm.architectures import convnext
class LoraConv2D(tf.keras.layers.Conv2D):
...
def main():
cls, cfg = convnext.convnext_atto()
# Monkey-patching conv layer
old_conv_layer = convnext.tf.keras.layers.Conv2D
convnext.tf.keras.layers.Conv2D = LoraConv2D
model = cls(cfg=cfg)
model(model.dummy_inputs)
# Reversing changes. This would become a context manager of course.
convnext.tf.keras.layers.Conv2D = old_conv_layer
# stem[0] is the first convolutional layer in the stem
print(type(model.stem[0]))
if __name__ == "__main__":
main()
This works, but it has some drawbacks:
convnext.py itself. In fact, most convolutional layers in ConvNeXt are implemented as part of MLP layers in tfimm/layers/transformers.py. This is surmountable, since we can patch Conv2D in all files belonging to tfimm. Implemented as a central context manager the complexity is manageable. It would be a problem, if we were to rely on external code for layers/blocks, but that is not the case at the moment.We would like to achieve layer-wise control, i.e., swap only some layers, but not others. We could do that by specifying layers to be swapped by their names.
When monkey patching we could set convnext.tf.keras.layers.Conv2D = conv_layer_factory with conv_layer_factory being a smart function which returns either a Conv2D layer or a LoraConv2D layer depending on what is needed. Unfortunately, I don't think this function has the necessary context to assemble the full layer name: The full nested name is generated via tf.name_scope when the layer is built in build(), not when it is defined in __init__(). Sample factory code:
def conv_layer_factory(*args, **kwargs):
print(tf.get_current_name_scope(), kwargs["name"])
return LoraConv2D(*args, **kwargs)
One way to achieve that is model surgery. Here is some example code for inserting or swapping layers. But this code creates a new functional model. Ideally, we would modify our model in place.
Is there Conda install for The package?
Right now I am using my own copy of vit_base_patch32_224_in21k. I am trying to initialize the model using tfimm.create_model. I am trying to load the model using the following:
import tfimm
tfimm.create_model(model_name='vit_base_patch32_224_in21k', model_path ='/my/model/path/vit_base_patch32_224_in21k/', input_shape=(224,224))
This results in
File "/etc/pyenv/versions/3.8.6/lib/python3.8/site-packages/tfimm/models/factory.py", line 45, in create_model
loaded_model = tf.keras.models.load_model(model_path)
File "/etc/pyenv/versions/3.8.6/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 67, in error_handler
raise e.with_traceback(filtered_tb) from None
File "/etc/pyenv/versions/3.8.6/lib/python3.8/site-packages/tfimm/models/serialization.py", line 81, in from_config
_cfg = cfg_class(**_cfg)
TypeError: __init__() got an unexpected keyword argument 'in_chans'
Note that I am using tfimm==0.2.1 and everything worked very well with 0.1.5. In addition the following code work:
import tensorflow as tf
tf.keras.models.load_model('/my/model/path/vit_base_patch32_224_in21k/')
Which of the models would be suitable for out of the box person detection?
I recently discovered there's no person/human label in ImageNet dataset so I'm hoping tfimm has models trained on COCO dataset (or some other with person label)
Good day,
I tried to install tfimm via pip but I got the following error.
ERROR: Could not find a version that satisfies the requirement tfimm( from version: None)
ERROR: No matching distribution found for tfimm
Is that a problem because I have not installed another packages?
Hi, I was building a model using ViT by iterating through the layers, but got error Incompatible shapes: [4] vs. [4,196] when I call model.fit.
Any ideas where the mismatch is happening? or it would be grateful if you guide me how to debug it (I am new to tensorflow).
Here is the function for building a ViT model for finetuning.
def get_model(img_size=config.IMAGE_SIZE):
with strategy.scope():
inp = tf.keras.layers.Input(shape = [img_size, img_size, 3], name = 'inp1')
label = tf.keras.layers.Input(shape = (), name = 'inp2')
vit_model = tfimm.create_model("vit_base_patch16_224_miil_in21k", pretrained="timm",nb_classes=0)
x = inp
for layer in vit_model.layers:
x = layer(x)
# Some modification will be made here playing with x
x = tf.keras.layers.Dense(config.N_CLASSES)(x)
output = tf.keras.layers.Softmax(dtype='float32')(x)
model = tf.keras.models.Model(inputs = [inp, label], outputs = [output])
opt = tf.keras.optimizers.Adam(learning_rate = config.LR)
model.compile(
optimizer = opt,
loss = [tf.keras.losses.SparseCategoricalCrossentropy()],
metrics = [tf.keras.metrics.SparseCategoricalAccuracy(),tf.keras.metrics.SparseTopKCategoricalAccuracy(k=5)]
)
return model
Hi!
First of all, thank you very much for your work, I think it's a very useful tool for those of us who are better at dealing with tensorflow :)
I would like to use some pretrained transformers model to extract features before the final layer of classification, so my question is: What should I do to get it? All I have to do is create a pretrained model with nclases=0?
Thank you very much!! 😊
Is there any plan to add efficientnet V1x or V2x models ? I know implementations can be found in the keras module itself but adding these models would make this library trully the equivalent of ross's pytorch-image-models.
Many versions of ConvNeXt are now available pretrained in timm.
To be able to load them in tfimm, the only code to add in:
Would be the following:
@register_model
def convnext_atto():
cfg = ConvNeXtConfig(
name="convnext_atto",
url="[timm]",
embed_dim=(40, 80, 160, 320),
nb_blocks=(2, 2, 6, 2),
)
return ConvNeXt, cfg
@register_model
def convnext_femto():
cfg = ConvNeXtConfig(
name="convnext_femto",
url="[timm]",
embed_dim=(48, 96, 192, 384),
nb_blocks=(2, 2, 6, 2),
)
return ConvNeXt, cfg
@register_model
def convnext_pico():
cfg = ConvNeXtConfig(
name="convnext_pico",
url="[timm]",
embed_dim=(64, 128, 256, 512),
nb_blocks=(2, 2, 6, 2),
)
return ConvNeXt, cfg
@register_model
def convnext_nano():
cfg = ConvNeXtConfig(
name="convnext_nano",
url="[timm]",
embed_dim=(80, 160, 320, 640),
nb_blocks=(2, 2, 8, 2),
)
return ConvNeXt, cfgI've tested it locally and it works perfectly.
Thanks in advance
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.