octiapp / keraspersonlab Goto Github PK

Hi @jricheimer

Thank you so much for your implementation, It would be great if we can get access to pretrained model to compare it's accuracy with paper.

Thanks,

Thank you for your reproducing of PersonLab. I used it to train model on COCO train2017 and got a not bad intermediate result. However, when I use the model to detect a 1K(1920x1080) image with about 40 persons, the Inference Time is very long(about 3s do all detections, and 13s do group matching) even with GPUs.
I found it mainly caused by the stage of Group Joints, especially the function compute_heatmaps() with inputs kp_maps, short_offsets which is the refinement of Keypoints Heatmaps. Theoretically, Inference Time of bottom-up method will not grow linearly when persons increase. Is there a problem with the implementation of the function compute_heatmaps()?

From my understanding,

this_skel[kp['id'], :2] = kp['xy']
this_skel[kp['id'], 2] = kp['conf']
path = iterative_bfs(skeleton_graph, kp['id'])[1:]
for edge in path:
      if this_skel[edge[0],2] == 0:
           continue

the order of the path will be uniquely determined by the start keypoint due to the tree-structure of person. So, if some keypoints in the path are not detected, multiple skeletons will be generated for a single body because of this snippet:

if this_skel[edge[0],2] == 0:
           continue

for example, let's assume that the start (the highest score) keypoint is Lwrist and all keypoints except L_elbow are detected for a single person, then this for edge in path: loop will always goto continue and this_skeleton will only contain the Lwrist keypoint location.

I don't know if I understand it correctly?

10993/16028 [===================>..........] - ETA: 59:31 - loss: 152848912.0000Traceback (most recent c
all last):
File "train.py", line 77, in
epochs=config.NUM_EPOCHS, callbacks=callbacks)
File "/root/anaconda3/envs/deep_source/lib/python3.6/site-packages/keras/engine/training.py", line 103
7, in fit
validation_steps=validation_steps)
File "/root/anaconda3/envs/deep_source/lib/python3.6/site-packages/keras/engine/training_arrays.py", l
ine 154, in fit_loop
outs = f(ins)
File "/root/anaconda3/envs/deep_source/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py
", line 2666, in call
return self._call(inputs)
File "/root/anaconda3/envs/deep_source/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 2636, in _call
fetched = self._callable_fn(*array_vals)
File "/root/anaconda3/envs/deep_source/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1382, in call
run_metadata_ptr)
File "/root/anaconda3/envs/deep_source/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 519, in exit
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.UnknownError: AssertionError:
Traceback (most recent call last):

File "/root/anaconda3/envs/deep_source/lib/python3.6/site-packages/tensorflow/python/ops/script_ops.py", line 206, in call
ret = func(*args)

File "/root/networks/KerasPersonLab/tf_data_generator.py", line 71, in transform_data
kp_maps, short_offsets, mid_offsets, long_offsets = get_ground_truth(instance_masks, kp)

File "/root/networks/KerasPersonLab/data_prep.py", line 16, in get_ground_truth
assert(instance_masks.shape[-1] == len(all_keypoints))

AssertionError

The error pretty much explain itself, not sure why shape is mis-matching.

model = get_personlab(train=False, with_preprocess_lambda=True, intermediate_supervision=True, intermediate_layer='res4b12_relu', build_base_func=get_resnet101_base, output_stride=16)

Downloading data from None

TypeError Traceback (most recent call last)
in ()
4 intermediate_layer='res4b12_relu',
5 build_base_func=get_resnet101_base,
----> 6 output_stride=16)
7 print('Loading time: {}'.format(time()-tic))

~/alibaba/repos/KerasPersonLab/model.py in get_personlab(train, input_tensors, with_preprocess_lambda, build_base_func, intermediate_supervision, intermediate_layer, output_stride)
206
207 if with_preprocess_lambda not in [False, 0, None]:
--> 208 base_model = build_base_func(input_tensor=normalized_img, output_stride=output_stride, return_model=True)
209 else:
210 base_model = build_base_func(input_tensor=input_img, output_stride=output_stride, return_model=True)

~/alibaba/repos/KerasPersonLab/resnet101.py in get_resnet101_base(input_tensor, output_stride, return_model)
150
151 model = KM.Model(input_tensor_source, x)
--> 152 weights_path = get_file('resnet101_weights_notop.h5', None, cache_subdir='models')
153 model.load_weights(weights_path)
154

~/virtualenvironment/tensorflow/lib/python3.6/site-packages/keras/utils/data_utils.py in get_file(fname, origin, untar, md5_hash, file_hash, cache_subdir, hash_algorithm, extract, archive_format, cache_dir)
220 try:
221 try:
--> 222 urlretrieve(origin, fpath, dl_progress)
223 except URLError as e:
224 raise Exception(error_msg.format(origin, e.errno, e.reason))

/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py in urlretrieve(url, filename, reporthook, data)
244 data file as well as the resulting HTTPMessage object.
245 """
--> 246 url_type, path = splittype(url)
247
248 with contextlib.closing(urlopen(url, data)) as fp:

/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/parse.py in splittype(url)
936 _typeprog = re.compile('([^/:]+):(.*)', re.DOTALL)
937
--> 938 match = _typeprog.match(url)
939 if match:
940 scheme, data = match.groups()

TypeError: expected string or bytes-like object

Hi!

First, thank you for your hard work!

I am trying to run training and some time after the first epoch begins, I am running into an issue where the following call (in transformer.py) fails:

masks = cv2.warpAffine(masks, M, cv_shape, flags=cv2.INTER_CUBIC, borderMode=cv2.BORDER_CONSTANT, borderValue=0)

Specifically, there is an OpenCV assertion error, stating that "error: (-215) _src.channels() <= 4 || (interpolation != INTER_LANCZOS4 && interpolation != INTER_CUBIC) in function cv::warpAffine". Could you please point me at what might be wrong here?

Thanks a lot!

N.B. Looking at the code, at first glace it would so seem that several Mats are passed into warpAffine(), is this intentional?

when I use python 3, I get the following Error:
TypeError: 'KeysView' object does not support indexing
so when use python 3, I should modify the function data_gen() in tf_data_generator.py in line87:
key = root.keys()
to
key = list(root.keys())
and now key could be iterated.
I think you should tell the environment detail in README so that other people could avoid such mistakes

Great job, thank you for your sharing! Then, could you please report the average precision (AP) of this project on COOC dataset?

Hi，thanks for sharing!
I wonder which version should I use?

When running the following code:

parallel_model.fit(steps_per_epoch=64115//batch_size, 
                   epochs=config.NUM_EPOCHS, 
                   callbacks=callbacks)

Getting this error.

WARNING:tensorflow:From /usr/local/lib/python3.7/site-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
Epoch 1/125
---------------------------------------------------------------------------
UnknownError                              Traceback (most recent call last)
<ipython-input-11-a6811e3578ae> in <module>
      1 parallel_model.fit(steps_per_epoch=64115//batch_size, 
      2                    epochs=config.NUM_EPOCHS,
----> 3                    callbacks=callbacks)

/usr/local/lib/python3.7/site-packages/keras/engine/training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, **kwargs)
   1037                                         initial_epoch=initial_epoch,
   1038                                         steps_per_epoch=steps_per_epoch,
-> 1039                                         validation_steps=validation_steps)
   1040 
   1041     def evaluate(self, x=None, y=None,

/usr/local/lib/python3.7/site-packages/keras/engine/training_arrays.py in fit_loop(model, f, ins, out_labels, batch_size, epochs, verbose, callbacks, val_f, val_ins, shuffle, callback_metrics, initial_epoch, steps_per_epoch, validation_steps)
    152                 batch_logs['size'] = 1
    153                 callbacks.on_batch_begin(step_index, batch_logs)
--> 154                 outs = f(ins)
    155 
    156                 outs = to_list(outs)

/usr/local/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py in __call__(self, inputs)
   2713                 return self._legacy_call(inputs)
   2714 
-> 2715             return self._call(inputs)
   2716         else:
   2717             if py_any(is_tensor(x) for x in inputs):

/usr/local/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py in _call(self, inputs)
   2673             fetched = self._callable_fn(*array_vals, run_metadata=self.run_metadata)
   2674         else:
-> 2675             fetched = self._callable_fn(*array_vals)
   2676         return fetched[:len(self.outputs)]
   2677 

/usr/local/lib/python3.7/site-packages/tensorflow/python/client/session.py in __call__(self, *args, **kwargs)
   1437           ret = tf_session.TF_SessionRunCallable(
   1438               self._session._session, self._handle, args, status,
-> 1439               run_metadata_ptr)
   1440         if run_metadata:
   1441           proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

/usr/local/lib/python3.7/site-packages/tensorflow/python/framework/errors_impl.py in __exit__(self, type_arg, value_arg, traceback_arg)
    526             None, None,
    527             compat.as_text(c_api.TF_Message(self.status.status)),
--> 528             c_api.TF_GetCode(self.status.status))
    529     # Delete the underlying status object from memory otherwise it stays alive
    530     # as there is a reference to status from this from the traceback due to

UnknownError: OSError: Unable to open file (unable to lock file, errno = 35, error message = 'Resource temporarily unavailable')
Traceback (most recent call last):

  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/ops/script_ops.py", line 207, in __call__
    ret = func(*args)

  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 449, in generator_py_func
    values = next(generator_state.get_iterator(iterator_id))

  File "/Users/ghafran/git/KerasPersonLab/tf_data_generator.py", line 85, in data_gen
    h5 = h5py.File(config.H5_DATASET, 'r')

  File "/usr/local/lib/python3.7/site-packages/h5py/_hl/files.py", line 394, in __init__
    swmr=swmr)

  File "/usr/local/lib/python3.7/site-packages/h5py/_hl/files.py", line 170, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)

  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper

  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper

  File "h5py/h5f.pyx", line 85, in h5py.h5f.open

OSError: Unable to open file (unable to lock file, errno = 35, error message = 'Resource temporarily unavailable')


	 [[{{node PyFunc}}]]
	 [[{{node IteratorGetNext}}]]

Hi, thanks a lot for the implementation. there's something that i couldn't figure out after reading the paper and reviewing the code. during the computation of the heatmaps the accumulate_votes function is called and some sort of bilinear interpolation is made. could someone please clarify which function is interpolated (in the code it seems to me that the values of this function are stored in the variable ps) and where do the interpolation weights come from (dx,dy) and what do they mean?
i'm referring to the following part of the code in the post_proc.py:
def accumulate_votes(votes, shape):
xs = votes[:,0]
ys = votes[:,1]
ps = votes[:,2]
tl = [np.floor(ys).astype('int32'), np.floor(xs).astype('int32')]
tr = [np.floor(ys).astype('int32'), np.ceil(xs).astype('int32')]
bl = [np.ceil(ys).astype('int32'), np.floor(xs).astype('int32')]
br = [np.ceil(ys).astype('int32'), np.ceil(xs).astype('int32')]
dx = xs - tl[1]
dy = ys - tl[0]
tl_vals = ps*(1.-dx)(1.-dy)
tr_vals = psdx*(1.-dy)
bl_vals = psdy(1.-dx)
br_vals = psdydx

hi，all
One error happens when training from scratch.

KerasPersonLab-updated_model_def/data_prep.py", line 16, in get_ground_truth

assert(instance_masks.shape[-1] == len(all_keypoints)), '{} != {}'.format(instance_masks.shape[-1], len(all_keypoints))

AssertionError: 7 != 8

Grateful for your implementation! I have some questions about the bilinear sampler.

In this condition,

x00 = tf.gather_nd(x, i00)
x01 = tf.gather_nd(x, i01)
x10 = tf.gather_nd(x, i10)
x11 = tf.gather_nd(x, i11)

will gather values from the destination positions of predicted offsets, shouldn't we gather values from the start positions of offsets?

It seems like that no dilation convolution in origin PersonLab paper, according to classical Resnet .
Did I miss or ignore something important in paper?

I've encountered a couple of errors in my code I was hoping you could help me fix:

1. I used get_resnet50_base and when i tried to load the weights this error popped up
   ValueError: You are trying to load a weight file containing 111 layers into a model with 116 layers.
2. if I put get_resnet50_base as my build base, which intermidiate layer do I use?

I would like to visualize the model. Are you able to generate it for me?

Does someone have managed to get results with this repository?

My loss is near constant, oscillating around 1.50.
I should mention that I have removed the instance segmentation module, but that shouldn't impact the pose estimation module.