Hi,

There is a small typo in transformer.py, line 320:

        if attn_mask is not None:
            ann_output_weights += attn_mask

should be

        if attn_mask is not None:
            attn_output_weights += attn_mask

I managed to adapt my dataset in a tf csv object format and perform the finetuning. Now I wanted to see some qualitative results running inference and plotting bboxes. I'm using the same code as the webcam inference example but im getting:

tf.Tensor([], shape=(0, 4), dtype=float32)
tf.Tensor([], shape=(0,), dtype=int64)
tf.Tensor([], shape=(0,), dtype=float32)

as output of the get_model_inference function. Any idea ?

hello, and thanks for this great work.

i'm trrying to fine tune the model exactly like you descried in DETR Tensorflow - Finetuning tutorial.ipynb, and with hard-hat-workers-dataset as a CSV format, but i get an error while load dataset:

from detr_tf.data import load_tfcsv_dataset

# Load the dataset and exclude the person class (for some reason not all person are labeled on the training set)
train_iterator, class_names = load_tfcsv_dataset("train", config.batch_size, config, augmentation=True, exclude=["person"])
valid_iterator, class_names = load_tfcsv_dataset("test", config.batch_size, config, augmentation=False, exclude=["person"])
print("class_names", class_names)

The error is :

TypeError                                 Traceback (most recent call last)
<ipython-input-11-d499c7a684e3> in <module>()
      2 
      3 # Load the dataset and exclude the person class (for some reason not all person are labeled on the training set)
----> 4 train_iterator, class_names = load_tfcsv_dataset("train", config.batch_size, config, augmentation=True, exclude=["person"])
      5 valid_iterator, class_names = load_tfcsv_dataset("test", config.batch_size, config, augmentation=False, exclude=["person"])
      6 print("class_names", class_names)

TypeError: load_tfcsv_dataset() got multiple values for argument 'augmentation'

I have some tribles to understuand this line from detr_tf.data import load_tfcsv_dataset
the load_tfcsv_dataset function is in the file "tfcsv"

Am i missing something ?

Cheers,

elif config is not None and config.normalized_method == "tf_resnet":
image = image + mean
image = image[..., ::-1]
image = image / 255

i tried to initialize it with mean = [103.939, 116.779, 123.68] but all my inferences had a blank picture. i am using tf_backbone, loaded by my weights.

A use my own data in csv to finetune DETR tensorflow model. I cant find any implementation in source code how to save tensorflow model after each epoch. I want to implement saving weights, but there are some errors after this. Maybe you have some examples or recommendations how to save finetuned weights during or after training process.

I have trained the model for around 230 epoch on the coco dataset using your "train_coco.py". I saved the model using detr.save_weights('weights.h5'). When testing on the same image, I got different results when I loading back the saved weights... (and the performance is not good at all) It seems this is an issue with keras （keras-team/keras#4875). Can you provide a solution on how to save your model? Thanks.

When I type python webcam_inference.py I get

2020-12-29 19:08:07.664029: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /root/distro/install/lib:/usr/local/cuda-10.0/lib64/:/root/distro/install/lib
2020-12-29 19:08:07.664062: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2020-12-29 19:08:09.472208: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2020-12-29 19:08:09.472893: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2020-12-29 19:08:09.507898: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-12-29 19:08:09.508338: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce GTX 1050 computeCapability: 6.1
coreClock: 1.493GHz coreCount: 5 deviceMemorySize: 1.95GiB deviceMemoryBandwidth: 104.43GiB/s
2020-12-29 19:08:09.508442: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /root/distro/install/lib:/usr/local/cuda-10.0/lib64/:/root/distro/install/lib
2020-12-29 19:08:09.508514: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcublas.so.11'; dlerror: libcublas.so.11: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /root/distro/install/lib:/usr/local/cuda-10.0/lib64/:/root/distro/install/lib
2020-12-29 19:08:09.508578: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcublasLt.so.11'; dlerror: libcublasLt.so.11: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /root/distro/install/lib:/usr/local/cuda-10.0/lib64/:/root/distro/install/lib
2020-12-29 19:08:09.508649: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcufft.so.10'; dlerror: libcufft.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /root/distro/install/lib:/usr/local/cuda-10.0/lib64/:/root/distro/install/lib
2020-12-29 19:08:09.508724: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcurand.so.10'; dlerror: libcurand.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /root/distro/install/lib:/usr/local/cuda-10.0/lib64/:/root/distro/install/lib
2020-12-29 19:08:09.508805: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcusolver.so.10'; dlerror: libcusolver.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /root/distro/install/lib:/usr/local/cuda-10.0/lib64/:/root/distro/install/lib
2020-12-29 19:08:09.508883: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcusparse.so.11'; dlerror: libcusparse.so.11: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /root/distro/install/lib:/usr/local/cuda-10.0/lib64/:/root/distro/install/lib
2020-12-29 19:08:09.509066: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2020-12-29 19:08:09.509080: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1757] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
Traceback (most recent call last):
File "webcam_inference.py", line 49, in
tf.config.experimental.set_memory_growth(physical_devices[0], True)
IndexError: list index out of range

My cuda is release 10.0, V10.0.130

Which CUDA and CUDNN I should use?

The weights are currently stored at the following locations, seem to be no longer accessible:

WEIGHT_NAME_TO_CKPT = {
"detr": [
"https://storage.googleapis.com/visualbehavior-publicweights/detr/checkpoint",
"https://storage.googleapis.com/visualbehavior-publicweights/detr/detr.ckpt.data-00000-of-00001",
"https://storage.googleapis.com/visualbehavior-publicweights/detr/detr.ckpt.index"
],
"deformable-detr": [
"https://storage.googleapis.com/visualbehavior-publicweights/deformable-detr/checkpoint",
"https://storage.googleapis.com/visualbehavior-publicweights/deformable-detr/deformable-detr.ckpt.data-00000-of-00001",
"https://storage.googleapis.com/visualbehavior-publicweights/deformable-detr/deformable-detr.ckpt.index"
],
"deformable-detr-refine_bbox": [
"https://storage.googleapis.com/visualbehavior-publicweights/deformable-detr-refine_bbox/checkpoint",
"https://storage.googleapis.com/visualbehavior-publicweights/deformable-detr-refine_bbox/deformable-detr-refine_bbox.ckpt.data-00000-of-00001",
"https://storage.googleapis.com/visualbehavior-publicweights/deformable-detr-refine_bbox/deformable-detr-refine_bbox.ckpt.index"
]
}

Do you have them hosted somewhere else?

How to solve this issue? Please anyone help me.
Thanks in advance

I probably miss something, but what is the purpose of tf.sigmoid over the box coordinates?

Hi
I am getting this error when Loading the dataset for fine tuning.

from detr_tf.data import load_tfcsv_dataset
train_iterator, class_names = load_tfcsv_dataset("train", config.batch_size, config, augmentation=True, exclude=["person"])
valid_iterator, class_names = load_tfcsv_dataset("test", config.batch_size, config, augmentation=False, exclude=["person"])
print("class_names", class_names)

I have referred these existing issues and still my issue persists: #48 #40
(tried to copy the code from the weights.txt provided in the issue)

I request you guys to take a look.

CODE:
from detr_tf.networks.detr import get_detr_model
detr = get_detr_model(config, include_top=False, nb_class=10, weights="detr")
detr.summary()

ERROR:
Load weights from weights/detr/detr.ckpt
2024-01-13 12:06:41.142705: W tensorflow/core/util/tensor_slice_reader.cc:98] Could not open weights/detr/detr.ckpt: DATA_LOSS: not an sstable (bad magic number): perhaps your file is in a different file format and you need to use a different restore operator?

STACK TREE:

ValueError Traceback (most recent call last)
Cell In[6], line 6
1 from detr_tf.networks.detr import get_detr_model
----> 6 detr = get_detr_model(config, include_top=False, nb_class=10, weights="detr")
7 detr.summary()

File ~/Documents/Project/detr-tensorflow/notebooks/../detr_tf/networks/detr.py:139, in get_detr_model(config, include_top, nb_class, weights, tf_backbone, num_decoder_layers, num_encoder_layers)
136 detr = DETR(num_decoder_layers=num_decoder_layers, num_encoder_layers=num_encoder_layers)
138 if weights is not None:
--> 139 load_weights(detr, weights)
141 image_input = tf.keras.Input((None, None, 3))
143 # Backbone

File ~/Documents/Project/detr-tensorflow/notebooks/../detr_tf/networks/weights.py:35, in load_weights(model, weights)
33 open(os.path.join(wdir, fname), 'wb').write(r.content)
34 print("Load weights from", os.path.join(wdir, f"{weights}.ckpt"))
---> 35 l = model.load_weights(os.path.join(wdir, f"{weights}.ckpt"))
36 l.expect_partial()
37 else:

File /usr/local/lib/python3.10/dist-packages/keras/src/utils/traceback_utils.py:70, in filter_traceback..error_handler(*args, **kwargs)
67 filtered_tb = _process_traceback_frames(e.traceback)
68 # To get the full stack trace, call:
69 # tf.debugging.disable_traceback_filtering()
---> 70 raise e.with_traceback(filtered_tb) from None
71 finally:
72 del filtered_tb

File /usr/local/lib/python3.10/dist-packages/keras/src/saving/legacy/save.py:480, in load_weights(model, filepath, by_name, skip_mismatch, options)
475 raise ImportError(
476 "load_weights requires h5py package when loading weights "
477 "from HDF5. Try installing h5py."
478 )
479 if not model._is_graph_network and not model.built:
--> 480 raise ValueError(
481 "Unable to load weights saved in HDF5 format into a "
482 "subclassed Model which has not created its variables yet. "
483 "Call the Model first, then load the weights."
484 )
485 model._assert_weights_created()
486 with h5py.File(filepath, "r") as f:

ValueError: Unable to load weights saved in HDF5 format into a subclassed Model which has not created its variables yet. Call the Model first, then load the weights.

These are the checkpoint files that got downloaded:

I even downloaded the detr.ckpt using this link: https://storage.googleapis.com/visualbehavior-publicweights/detr/detr.ckpt

I have all the files as default from the repository except I am using a custom dataset. But obviously it has nothing to do with my problem here. Thank you.

I'm getting the following error
"[[{{node PyFunc}}]] 0 successful operations. 0 derived errors ignored. [Op:__inference_run_val_step_37449]"

on lines

m_outputs, total_loss, log = run_val_step(model, images, t_bbox, t_class, config)
and
m_outputs, total_loss, log, gradient_steps = run_train_step(model, images, t_bbox, t_class, optimizers,

in the training.py file when i try to train my coco dataset and pass it through

training.eval(detr, valid_dt, config, coco_class_names, evaluation_step=200)
training.fit(detr, train_dt, optimzers, config, epoch_nb=0, class_names=coco_class_names)

Does anyone have an idea about this?

Hi, thanks for the great work. The project is highly complete. And there are some different training strategy than official one. I learn some of them.

I am curious how you preprocessing the image.

In the get_coco_from_id, i didnot find the resize part?
It seems it just keep the original size?
But there is no dataset = dataset.padded_batch( in the dataset part.

So I guess you already preprocess the original size to a fix size?
cause in the config img size seems fixed and smaller self.image_size = 376, 672

def get_coco_from_id(coco_id, coco, augmentation, config, img_dir):
    # Load imag
    img = coco.loadImgs([coco_id])[0]
    # Load image
    #data_type = "train2017" if train_val == "train" else "val2017"
    filne_name = img['file_name']
    image_path = os.path.join(img_dir, filne_name) #f"{config.}/{data_type}/{filne_name}"
    image = imageio.imread(image_path)
    # Graycale to RGB if needed
    if len(image.shape) == 2: image = gray2rgb(image)
    # Retrieve the image label
    t_bbox, t_class, is_crowd = get_coco_labels(coco, img['id'], image.shape, augmentation)
    # Apply augmentations
    if len(t_bbox) > 0 and augmentation is not None:
        image, t_bbox, t_class = transformation.detr_transform(image, t_bbox,  t_class, config, augmentation)
    # Normalized images
    image = processing.normalized_images(image, config)
    # Set type for tensorflow        
    image = image.astype(np.float32)
    t_bbox = t_bbox.astype(np.float32)
    t_class = t_class.astype(np.int64)
    return image, t_bbox, t_class, is_crowd

    # Setup the data pipeline
    img_ids = coco.getImgIds()
    shuffle(img_ids)
    dataset = tf.data.Dataset.from_tensor_slices(img_ids)
    # Shuffle the dataset
    dataset = dataset.shuffle(1000)
    # Retrieve img and labels
    outputs_types=(tf.float32, tf.float32, tf.int64, tf.int64)
    dataset = dataset.map(lambda idx: processing.numpy_fc(
        idx, get_coco_from_id, outputs_types=outputs_types, coco=coco, augmentation=augmentation, config=config, img_dir=img_dir)
    , num_parallel_calls=tf.data.experimental.AUTOTUNE)
    dataset = dataset.filter(lambda imgs, tbbox, tclass, iscrowd: tf.shape(tbbox)[0] > 0 and iscrowd != 1)
    dataset = dataset.map(lambda imgs, tbbox, tclass, iscrowd: (imgs, tbbox, tclass), num_parallel_calls=tf.data.experimental.AUTOTUNE)
    
    # Pad bbox and labels
    dataset = dataset.map(processing.pad_labels, num_parallel_calls=tf.data.experimental.AUTOTUNE)

    dataset = dataset.batch(batch_size, drop_remainder=True)
    dataset = dataset.prefetch(32)
    

Hi, i noticed you not using the key_padding_mask is the MultiHeadAttention. Which means the mask are not using in the Transformer?
Mu guess is, from the original author, the mask have the img original size info before padding, so the transformer would know which part is real img and which part is padding from the mask.
But you have fixed size input, transformer no need worried the img padding.
If in that case, why not keep them? Any special reason?

"""
 if key_padding_mask is not None:
     attn_output_weights = tf.reshape(attn_output_weights,
                         [batch_size, self.num_heads, target_len, source_len])
     key_padding_mask = tf.expand_dims(key_padding_mask, 1)
     key_padding_mask = tf.expand_dims(key_padding_mask, 2)
     key_padding_mask = tf.tile(key_padding_mask, [1, self.num_heads, target_len, 1])
     #print("before attn_output_weights", attn_output_weights.shape)
     attn_output_weights = tf.where(key_padding_mask,
                                    tf.zeros_like(attn_output_weights) + float('-inf'),
                                    attn_output_weights)
     attn_output_weights = tf.reshape(attn_output_weights,
                         [batch_size * self.num_heads, target_len, source_len])
 """

the training argument is missing when the TransformerDecoder calls DecoderLayers at

I trained the model on a custom dataset using python finetune_.py script. I see this error when I try to load the finetuned model and use it with webcam.py. Does the code here need to be fixed? Or is there a training knob that I missed during training to have this layer?

detr = get_detr_model(config, include_top=False, nb_class=7, weights=None)
detr.load_weights("model_dir/detr-model.ckpt")

networks/detr.py in get_detr_model(config, include_top, nb_class, weights, tf_backbone, num_decoder_layers, num_encoder_layers)
151 transformer = detr.get_layer("transformer")
152 # Positional embedding of the feature map
--> 153 position_embedding_sine = detr.get_layer("position_embedding_sine")
154 # Used to project the feature map before to fit in into the encoder
155 input_proj = detr.get_layer('input_proj')

Hi,
How to export into saved weights to frozen model?

Implement this paper

https://openreview.net/pdf?id=gZ9hCDWe6ke

DETR has been recently proposed to eliminate the need for many hand-designed
components in object detection while demonstrating good performance. However,
it suffers from slow convergence and limited feature spatial resolution, due to the
limitation of Transformer attention modules in processing image feature maps. To
mitigate these issues, we proposed Deformable DETR, whose attention modules
only attend to a small set of key sampling points around a reference. Deformable
DETR can achieve better performance than DETR (especially on small objects)
with 10× less training epochs. Extensive experiments on the COCO benchmark
demonstrate the effectiveness of our approach. Code shall be released.

I am very confused about the "mask" in the detr.py. Could you explain what is this, please? And, if Intput's image was resized to the same size then we don't need "mask" right?

def downsample_masks(self, masks, x):
        masks = tf.cast(masks, tf.int32)
        masks = tf.expand_dims(masks, -1)
        masks = tf.compat.v1.image.resize_nearest_neighbor(masks, tf.shape(x)[1:3], align_corners=False, half_pixel_centers=False)
        masks = tf.squeeze(masks, -1)
        masks = tf.cast(masks, tf.bool)
        return masks

def call(self, inp, training=False, post_process=False):
    x, masks = inp
    x = self.backbone(x, training=training)

Thanks for your contribution.

Aren't the arguments and parameters reversed for the call to merge here?

I say this because np_tf_linear_sum_assignment seems to expect the axis of the cost_matrix corresponding to the ground truth to be on the rows and corresponding to the predictions to be on the columns as per

whereas, the above call to merge would return tensors when operated upon to produce the a cost matrix where rows correspond to predictions and columns to ground truth.

Hello everyone and thanks in advance to those who decide to help me. I will be really grateful to him. I am working on my master's thesis and I would like to perform a finetuning on the Crowd Human dataset, in order to improve the detection of people.
But I think I'm doing something wrong ... The model I get is bad! The results on the test set are much worse than the original DETR network.

Could anyone help me?

Thank you so much. I would be immensely grateful.

(I could attach the code, although I followed that guide step by step:
Link
modifying only the dataset (train and validation) and the two annotation files.

Please help me. Thanks in advance.

How could I pass from this Error?
OperatorNotAllowedInGraphError: using a tf.Tensor as a Python bool is not allowed: AutoGraph is disabled in this function. Try decorating it directly with @tf.function.

when I train the code through train_coco.py file, this error happend?
is anyone could help me?

It is great that this repository works with both VOC and COCO datasets

However, the instructions only say "python eval.py --datadir /path/to/coco".

What about VOC?

Hello everyone,
I tried to run the finetuning of DETR with the dataset indicated in the tutorial ("hardhat") and it seems to work fine.
Now I have replaced the initial dataset with one of my interest, namely "Crowd Human".
Suddenly, however, I get this error (sometimes it even manages to get to validation step 3 or 4):

`Load weights from weights/detr/detr.ckpt
Model: "detr_finetuning"

Layer (type) Output Shape Param # Connected to

input_2 (InputLayer) [(None, None, None, 0

detr (Functional) (6, None, 100, 256) 41449152 input_2[0][0]

pos_layer (Sequential) (6, None, 100, 4) 132612 detr[0][0]

cls_layer (Dense) (6, None, 100, 2) 514 detr[0][0]

tf_op_layer_strided_slice_6 (Te [(None, 100, 4)] 0 pos_layer[0][0]

tf_op_layer_strided_slice_5 (Te [(None, 100, 2)] 0 cls_layer[0][0]

tf_op_layer_strided_slice_8 (Te [(None, 100, 4)] 0 pos_layer[0][0]

tf_op_layer_strided_slice_7 (Te [(None, 100, 2)] 0 cls_layer[0][0]

tf_op_layer_strided_slice_10 (T [(None, 100, 4)] 0 pos_layer[0][0]

tf_op_layer_strided_slice_9 (Te [(None, 100, 2)] 0 cls_layer[0][0]

tf_op_layer_strided_slice_12 (T [(None, 100, 4)] 0 pos_layer[0][0]

tf_op_layer_strided_slice_11 (T [(None, 100, 2)] 0 cls_layer[0][0]

tf_op_layer_strided_slice_14 (T [(None, 100, 4)] 0 pos_layer[0][0]

tf_op_layer_strided_slice_13 (T [(None, 100, 2)] 0 cls_layer[0][0]

tf_op_layer_strided_slice_4 (Te [(None, 100, 4)] 0 pos_layer[0][0]

tf_op_layer_strided_slice_3 (Te [(None, 100, 2)] 0 cls_layer[0][0]

Total params: 41,582,278
Trainable params: 41,476,038
Non-trainable params: 106,240

/user/spadula/DETECTOR/detr-tensorflow-main/detr_tf/loss/compute_map.py:101: RuntimeWarning: invalid value encountered in true_divide
overlaps = intersections / union
Validation step: [0], ce: [0.78] giou : [1.13] l1 : [0.93] time : [0.00]
Traceback (most recent call last):
File "/user/spadula/.local/lib/python3.6/site-packages/tensorflow/python/eager/context.py", line 2102, in execution_mode
yield
File "/user/spadula/.local/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 758, in _next_internal
output_shapes=self._flat_output_shapes)
File "/user/spadula/.local/lib/python3.6/site-packages/tensorflow/python/ops/gen_dataset_ops.py", line 2610, in iterator_get_next
_ops.raise_from_not_ok_status(e, name)
File "/user/spadula/.local/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 6843, in raise_from_not_ok_status
six.raise_from(core._status_to_exception(e.code, message), None)
File "", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: Paddings must be non-negative: 0 -7
[[{{node Pad_2}}]] [Op:IteratorGetNext]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "finetune_crowd_human.py", line 99, in
run_finetuning(config)
File "finetune_crowd_human.py", line 79, in run_finetuning
training.eval(detr, valid_dt, config, class_names, evaluation_step=100)
File "/user/spadula/DETECTOR/detr-tensorflow-main/detr_tf/training.py", line 72, in eval
for val_step, (images, t_bbox, t_class) in enumerate(valid_dt):
File "/user/spadula/.local/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 736, in next
return self.next()
File "/user/spadula/.local/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 772, in next
return self._next_internal()
File "/user/spadula/.local/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 764, in _next_internal
return structure.from_compatible_tensor_list(self._element_spec, ret)
File "/usr/lib/python3.6/contextlib.py", line 99, in exit
self.gen.throw(type, value, traceback)
File "/user/spadula/.local/lib/python3.6/site-packages/tensorflow/python/eager/context.py", line 2105, in execution_mode
executor_new.wait()
File "/user/spadula/.local/lib/python3.6/site-packages/tensorflow/python/eager/executor.py", line 67, in wait
pywrap_tfe.TFE_ExecutorWaitForAllPendingNodes(self._handle)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Paddings must be non-negative: 0 -7
[[{{node Pad_2}}]]
Traceback (most recent call last):
File "/user/spadula/.local/lib/python3.6/site-packages/tensorflow/python/eager/context.py", line 2102, in execution_mode
yield
File "/user/spadula/.local/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 758, in _next_internal
output_shapes=self._flat_output_shapes)
File "/user/spadula/.local/lib/python3.6/site-packages/tensorflow/python/ops/gen_dataset_ops.py", line 2610, in iterator_get_next
_ops.raise_from_not_ok_status(e, name)
File "/user/spadula/.local/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 6843, in raise_from_not_ok_status
six.raise_from(core._status_to_exception(e.code, message), None)
File "", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: Paddings must be non-negative: 0 -7
[[{{node Pad_2}}]] [Op:IteratorGetNext]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "finetune_crowd_human.py", line 99, in
run_finetuning(config)
File "finetune_crowd_human.py", line 79, in run_finetuning
training.eval(detr, valid_dt, config, class_names, evaluation_step=100)
File "/user/spadula/DETECTOR/detr-tensorflow-main/detr_tf/training.py", line 72, in eval
for val_step, (images, t_bbox, t_class) in enumerate(valid_dt):
File "/user/spadula/.local/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 736, in next
return self.next()
File "/user/spadula/.local/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 772, in next
return self._next_internal()
File "/user/spadula/.local/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 764, in _next_internal
return structure.from_compatible_tensor_list(self._element_spec, ret)
File "/usr/lib/python3.6/contextlib.py", line 99, in exit
self.gen.throw(type, value, traceback)
File "/user/spadula/.local/lib/python3.6/site-packages/tensorflow/python/eager/context.py", line 2105, in execution_mode
executor_new.wait()
File "/user/spadula/.local/lib/python3.6/site-packages/tensorflow/python/eager/executor.py", line 67, in wait
pywrap_tfe.TFE_ExecutorWaitForAllPendingNodes(self._handle)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Paddings must be non-negative: 0 -7
[[{{node Pad_2}}]]
`

What does it mean? and how can I solve the problem?

Thank you so much in advance. Greetings, Simone.

I tried to run your system and darkflow in the same server

Too complicated: one needs to install 2 versions of CUDA and CUDNN and that has to be installed using .sh files instead of .pkg as the latter overwrites the initial system

And one needs to create a virtualenv to run one of those systems

And if the ubuntu is version 20 then it comes with python 3.85 which does not bring up tensorflow 1.14 required by darkflow, as it only brings versions above 2.0, so one needs to also install python 3.7 and hope it comes with pip 3.7

So it'd be nice if your system could identify small objects so one can move away from darkflow as that is just obsolete

Hi,

The file hungarian_matching.py contains unused references to Pytorch and non-existent functions.

Could this code be removed?

I am training a very simple try (just 1 picture) coco model just to try the system and using the command
python train_coco.py --datadir /root/detr-tensorflow/coco --batch_size 8 --target_batch 32

It seems to work, but it is unclear if it actually saves the model? I'd like to later use it in the webcam demo

Any help appreciated, thanks

I keep getting compute_map error. e= unhashable type: 'numpy.ndarray' when training

Dear authors,
You have imported 'imgaug' file in many places but it is missing from the required location. The code gives error on running. Please let me know how to resolve the issue.

@thibo73800 @PhanTask @andife

I'am using your code with Tensorflow version 2.15.0 and Keras version 3.0.4

If i call the lines of get_detr_model(config,...) then an Value Error arises with the following message:
ValueError: A KerasTensor cannot be used as input to a TensorFlow function. A KerasTensor is a symbolic placeholder for a shape and dtype, used when constructing Keras Functional models or Keras Functions. You can only use it as input to a Keras layer or a Keras operation (from the namespaces keras.layersandkeras.operations)

The error seems to be caused by line 172 in detr.py: masks = tf.zeros((tf.shape(x)[0], tf.shape(x)[1], tf.shape(x)[2]), tf.bool).

The code that i use is from your tutorial here at the section Load the detr model.

What am i doing wrong?

Hello,
since detr can also be used for panoptic segmentation I wonder if there are plans to add an example for this as well?
Thank you

Does the dataset only support batch size 1, any plan to fix this?

def retrieve_outputs(augmented_images, augmented_bbox):

outputs_dict = {}
image_shape = None


# We expect only one image here for now
image = augmented_images[0].astype(np.float32)
augmented_bbox = augmented_bbox[0]

bbox, t_class = imgaug_bbox_to_xcyc_wh(augmented_bbox, image.shape[0], image.shape[1])

bbox = np.array(bbox)
t_class = np.array(t_class)

return image, bbox, t_class

I am just wondering if there is a specific procedure for training negatives with xml annotations - voc dataset.

I am loading data from csv but I got this error. I just tried to run the example code again

This example: https://github.com/Visual-Behavior/detr-tensorflow/blob/main/notebooks/How%20to%20load%20a%20dataset.ipynb

I am not sure why this will be produced after I try to load the weights. I have been stuck on this for a month and have no idea how to solve this issue.

Hi, and first of all thank you for your contribution.
To cut it short, I cannot save a trained model using the saved_model format, with the following error:
TypeError: Input 'y' of 'AddV2' Op has type bool that does not match type float32 of argument 'x'

The code I'm using to save a model is:
model.save("./exports", include_optimizer=False)

I need to emphasize that I do not want to save the model via save_weights, as I need to use the saved_model format for inference with tools like TensorflowServing and TritonInferenceServer.

Any help would be much appreciated!

Problem
Cannot save model via model.save(...) or tf.saved_model.save(...)

Minimal steps to reproduce issue

from detr_tf.networks.detr import get_detr_model
from detr_tf.optimizers import setup_optimizers
from detr_tf import training
from detr_tf.inference import get_model_inference, numpy_bbox_to_image

detr = get_detr_model(config, include_top=False, nb_class=2, weights="detr")
detr.save("./exports",  include_optimizer=False)

Reading https://github.com/Visual-Behavior/detr-tensorflow/blob/main/notebooks/How%20to%20load%20a%20dataset.ipynb in order to load a VOC dataset, but still unclear

For example, it keeps saying "ModuleNotFoundError: No module named 'detr_tf'"

It'd be nice if you could produce a python file especifically for voc datasets: just like you have train_coco.py, but train_voc.py

When running finetune_voc.py, an error occurs during the loading of weights. It is as follows:

Traceback (most recent call last):
File "C:\Users\Goh\detr-transformer\detr-tensorflow\finetune_voc.py", line 126, in
run_finetuning(config)
File "C:\Users\Goh\detr-transformer\detr-tensorflow\finetune_voc.py", line 77, in run_finetuning
detr = build_model(config)
File "C:\Users\Goh\detr-transformer\detr-tensorflow\finetune_voc.py", line 50, in build_model
detr = get_detr_model(config, include_top=False, weights="detr", num_decoder_layers=6, num_encoder_layers=6)
File "C:\Users\Goh\detr-transformer\detr-tensorflow\detr_tf\networks\detr.py", line 139, in get_detr_model
load_weights(detr, weights)
File "C:\Users\Goh\detr-transformer\detr-tensorflow\detr_tf\networks\weights.py", line 39, in load_weights
l = model.load_weights(os.path.join(wdir, f"{weights}.ckpt"))
File "C:\Users\Goh\detr-transformer\detr-tensorflow\venv\lib\site-packages\keras\src\utils\traceback_utils.py", line 70, in error_handler
raise e.with_traceback(filtered_tb) from None
File "C:\Users\Goh\detr-transformer\detr-tensorflow\venv\lib\site-packages\tensorflow\python\training\py_checkpoint_reader.py", line 45, in error_translator
raise errors_impl.OpError(None, None, error_message, errors_impl.UNKNOWN)
tensorflow.python.framework.errors_impl.OpError: not an sstable (bad magic number)

I have tried editing the argument for load_weights to os.path.join(wdir, f"{weights}.ckpt.index") and others, including dowloading the hdf5 file, but to no avail. Any help on this would be much appreciated, especially since no one else seems to have had this issue!

Further contexts (in text since can't upload images for some reason):

• checkpoint, ckpt.index and ckpt.data-00000-of-00001 files are 1KB large, are they supposed to be that size?
• no other code was changed
• python version 3.9.0
• the WEIGHT_NAME_TO_CKPT dictionary of the api calls should be the same, but in case it isn't here is mine:
  WEIGHT_NAME_TO_CKPT = {
  "detr": [
  "https://storage.googleapis.com/visualbehavior-publicweights/detr/checkpoint",
  "https://storage.googleapis.com/visualbehavior-publicweights/detr/detr.ckpt.data-00000-of-00001",
  "https://storage.googleapis.com/visualbehavior-publicweights/detr/detr.ckpt.index"
  ]
  }

(this is my first ever issue/post on github sorry if its all messy or uninformative haha)

Getting this error although the file is there and LD_LIBRARY_PATH is correctly set

ImportError: cannot import name 'SegmentationMapsOnImage' from 'imgaug.augmentables.segmaps' (/usr/local/lib/python3.7/dist-packages/imgaug/augmentables/segmaps.py)

python train_coco.py --datadir /root/detr-tensorflow/coco --batch_size 8 --target_batch 32 --log

I get the error:
_**AttributeError: in user code:

/root/detr-tensorflow/training.py:31 run_val_step  *
    total_loss, log = get_losses(m_outputs, t_bbox, t_class, config)
/root/detr-tensorflow/loss/loss.py:30 get_losses  *
    losses = get_detr_losses(m_outputs, t_bbox, t_class, config)
/root/detr-tensorflow/loss/loss.py:129 get_detr_losses  *
    t_bbox = t_bbox.to_tensor()

AttributeError: 'Tensor' object has no attribute 'to_tensor'**_

My setup:
GeForce GTX 1050
Driver Version: 460.27.04
CUDA Version: 11.2
Tensorflow 2.4.0
CUDNN 8_8.0.5.39-1 for cuda11.1
Python 3.8.5

Hi,
When trying to access the transformers internal outputs, in an attempt to visualize attention, I am unable to access these layers on the normal tensorflow way using e.g. "model.get_layer('transformer').get_layer('decoder').get_layer('layer_5').multihead_attn.output". It seems the graph is not fully connected as it should be. I have found a couple of fixes that allowed me to access these layers succesfully:

• define pos_encoding with boolean version of your masks as it is now (masks = tf.zeros((tf.shape(x)[0], tf.shape(x)[1], tf.shape(x)[2]), tf.bool); pos_encoding = self.pos_encoder(masks); but pass the masks to the transformer as tf.float32 instead of tf.bool.
• pass the query to he transformer as self.query_embed(tf.constant(1.0)) instead of self.query_embed(None); in the FixedEmbedding Layer, return tf.multiply(self.w, x) instead of returning self.w; passing 'None' as argument causes the graph to break.
• in the transformer code, make sure to put 'memory' as the first argument in the call to the decoder; put 'target' as the second argument: inputs to call() should always be the first argument; when you put 'memory' as the second argument, the graph breaks apart; in case of multiple inputs, it's even better to put them in a list.
  Applying these changes to the code allowed me to access the multihead attention output using model.get_layer('transformer').get_layer('decoder').get_layer('layer_5').multihead_attn.output

When looking at the hungarian_matching function it shows the outputs are returned in the following order:

return pred_indices, target_indices, pred_selector, target_selector, t_bbox, t_class

However, in the get_detr_losses function the order is:

t_indices, p_indices, t_selector, p_selector, t_bbox, t_class = hungarian_matching(t_bbox, t_class, p_bbox, p_class, slice_preds=True)

This seems it would significantly impact the performance of the model given the wrong predictions and targets will be referenced during the training process. Can you please clarify which order is correct?

@thibo73800 @PhanTask @andife

Hi,

First off, thanks for sharing this code--it is immensely useful and I appreciate it.

I'm working on using this to finetune a dataset, when I follow the instructions to load my own COCO format dataset as described in your tutorial on my personal machine using Tensorflow 2.3 built from source, Python 3.7 and have all the dependencies installed that you are required.

Here is the code (following the tutorial):

`class MyConfig(TrainingConfig):
def init(self):
super().init()
self.data_dir = "path/to/this/dataset/zzz_Weld_COCO"
self.data = DataConfig(data_dir=self.data_dir, img_dir='train_resize', ann_file="annotations\custom_train_0427_v2.json")
self.batch_size = 1

config = MyConfig()
iterator, class_names = load_coco_dataset(config, config.batch_size, augmentation=True)

from pycocotools.coco import COCO
import tensorflow as tf
import numpy as np
import imageio
from skimage.color import gray2rgb
from random import sample, shuffle
import os

from detr_tf.data import transformation
from detr_tf.data import processing
from detr_tf.data import coco as CoCo
import matplotlib.pyplot as plt`

for images, target_bbox, target_class in iterator:
print("image.shape", images.shape)
print("target_bbox.shape", target_bbox.shape)
print("target_class.shape", target_class.shape)`

Each time I go to iterate over the dataset as noted in the load dataset tutorial ( and noted above), I get the following error:

`InvalidArgumentError Traceback (most recent call last)
D:\Python\envs\VS\lib\site-packages\tensorflow\python\eager\context.py in execution_mode(mode)
2101 ctx.executor = executor_new
-> 2102 yield
2103 finally:

D:\Python\envs\VS\lib\site-packages\tensorflow\python\data\ops\iterator_ops.py in _next_internal(self)
757 output_types=self._flat_output_types,
--> 758 output_shapes=self._flat_output_shapes)
759

D:\Python\envs\VS\lib\site-packages\tensorflow\python\ops\gen_dataset_ops.py in iterator_get_next(iterator, output_types, output_shapes, name)
2636 except _core._NotOkStatusException as e:
-> 2637 _ops.raise_from_not_ok_status(e, name)
2638 except _core._FallbackException:

D:\Python\envs\VS\lib\site-packages\tensorflow\python\framework\ops.py in raise_from_not_ok_status(e, name)
6842 # pylint: disable=protected-access
-> 6843 six.raise_from(core._status_to_exception(e.code, message), None)
6844 # pylint: enable=protected-access

D:\Python\envs\VS\lib\site-packages\six.py in raise_from(value, from_value)

InvalidArgumentError: 3-th value returned by pyfunc_0 is int32, but expects int64
[[{{node PyFunc}}]] [Op:IteratorGetNext]

During handling of the above exception, another exception occurred:

InvalidArgumentError Traceback (most recent call last)
in
----> 1 for images, target_bbox, target_class in iterator:
2 print("image.shape", images.shape)
3 print("target_bbox.shape", target_bbox.shape)
4 print("target_class.shape", target_class.shape)

D:\Python\envs\VS\lib\site-packages\tensorflow\python\data\ops\iterator_ops.py in next(self)
734
735 def next(self): # For Python 3 compatibility
--> 736 return self.next()
737
738 def _next_internal(self):

D:\Python\envs\VS\lib\site-packages\tensorflow\python\data\ops\iterator_ops.py in next(self)
770 def next(self):
771 try:
--> 772 return self._next_internal()
773 except errors.OutOfRangeError:
774 raise StopIteration

D:\Python\envs\VS\lib\site-packages\tensorflow\python\data\ops\iterator_ops.py in _next_internal(self)
762 return self._element_spec._from_compatible_tensor_list(ret) # pylint: disable=protected-access
763 except AttributeError:
--> 764 return structure.from_compatible_tensor_list(self._element_spec, ret)
765
766 @Property

D:\Python\envs\VS\lib\contextlib.py in exit(self, type, value, traceback)
128 value = type()
129 try:
--> 130 self.gen.throw(type, value, traceback)
131 except StopIteration as exc:
132 # Suppress StopIteration unless it's the same exception that

D:\Python\envs\VS\lib\site-packages\tensorflow\python\eager\context.py in execution_mode(mode)
2103 finally:
2104 ctx.executor = executor_old
-> 2105 executor_new.wait()
2106
2107

D:\Python\envs\VS\lib\site-packages\tensorflow\python\eager\executor.py in wait(self)
65 def wait(self):
66 """Waits for ops dispatched in this executor to finish."""
---> 67 pywrap_tfe.TFE_ExecutorWaitForAllPendingNodes(self._handle)
68
69 def clear_error(self):

InvalidArgumentError: 3-th value returned by pyfunc_0 is int32, but expects int64
[[{{node PyFunc}}]]`

I'm sorry if this isn't descriptive enough--this is my first time ever posting an issue, so please let me know if there is anything I can do anything to be more helpful.

EDIT: When working on Colab without using a GPU, this is not an issue, but when I switch to using a GPU, the same issue occurs in Colab.

Trying to save the model using tf.keras.model.save_model leads to a TypeError:

File "detr_tf/networks/transformer.py", line 211, in call
key_mem = memory + pos_encoding
TypeError: unsupported operand type(s) for +: 'NoneType' and 'NoneType'