octiapp / keraspersonlab Goto Github PK
View Code? Open in Web Editor NEWKeras-tensorflow implementation of PersonLab (https://arxiv.org/abs/1803.08225)
License: MIT License
Keras-tensorflow implementation of PersonLab (https://arxiv.org/abs/1803.08225)
License: MIT License
Hi @jricheimer
Thank you so much for your implementation, It would be great if we can get access to pretrained model to compare it's accuracy with paper.
Thanks,
Thank you for your reproducing of PersonLab. I used it to train model on COCO train2017 and got a not bad intermediate result. However, when I use the model to detect a 1K(1920x1080)
image with about 40 persons, the Inference Time is very long(about 3s do all detections, and 13s do group matching) even with GPUs.
I found it mainly caused by the stage of Group Joints, especially the function compute_heatmaps()
with inputs kp_maps, short_offsets
which is the refinement of Keypoints Heatmaps. Theoretically, Inference Time of bottom-up method will not grow linearly when persons increase. Is there a problem with the implementation of the function compute_heatmaps()
?
Line 97 in 32d44dd
From my understanding,
this_skel[kp['id'], :2] = kp['xy']
this_skel[kp['id'], 2] = kp['conf']
path = iterative_bfs(skeleton_graph, kp['id'])[1:]
for edge in path:
if this_skel[edge[0],2] == 0:
continue
the order of the path
will be uniquely determined by the start keypoint due to the tree-structure of person. So, if some keypoints in the path
are not detected, multiple skeletons will be generated for a single body because of this snippet:
if this_skel[edge[0],2] == 0:
continue
for example, let's assume that the start (the highest score) keypoint is Lwrist
and all keypoints except L_elbow
are detected for a single person, then this for edge in path:
loop will always goto continue
and this_skeleton
will only contain the Lwrist
keypoint location.
I don't know if I understand it correctly?
10993/16028 [===================>..........] - ETA: 59:31 - loss: 152848912.0000Traceback (most recent c
all last):
File "train.py", line 77, in
epochs=config.NUM_EPOCHS, callbacks=callbacks)
File "/root/anaconda3/envs/deep_source/lib/python3.6/site-packages/keras/engine/training.py", line 103
7, in fit
validation_steps=validation_steps)
File "/root/anaconda3/envs/deep_source/lib/python3.6/site-packages/keras/engine/training_arrays.py", l
ine 154, in fit_loop
outs = f(ins)
File "/root/anaconda3/envs/deep_source/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py
", line 2666, in call
return self._call(inputs)
File "/root/anaconda3/envs/deep_source/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 2636, in _call
fetched = self._callable_fn(*array_vals)
File "/root/anaconda3/envs/deep_source/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1382, in call
run_metadata_ptr)
File "/root/anaconda3/envs/deep_source/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 519, in exit
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.UnknownError: AssertionError:
Traceback (most recent call last):
File "/root/anaconda3/envs/deep_source/lib/python3.6/site-packages/tensorflow/python/ops/script_ops.py", line 206, in call
ret = func(*args)
File "/root/networks/KerasPersonLab/tf_data_generator.py", line 71, in transform_data
kp_maps, short_offsets, mid_offsets, long_offsets = get_ground_truth(instance_masks, kp)
File "/root/networks/KerasPersonLab/data_prep.py", line 16, in get_ground_truth
assert(instance_masks.shape[-1] == len(all_keypoints))
AssertionError
The error pretty much explain itself, not sure why shape is mis-matching.
model = get_personlab(train=False, with_preprocess_lambda=True, intermediate_supervision=True, intermediate_layer='res4b12_relu', build_base_func=get_resnet101_base, output_stride=16)
TypeError Traceback (most recent call last)
in ()
4 intermediate_layer='res4b12_relu',
5 build_base_func=get_resnet101_base,
----> 6 output_stride=16)
7 print('Loading time: {}'.format(time()-tic))
~/alibaba/repos/KerasPersonLab/model.py in get_personlab(train, input_tensors, with_preprocess_lambda, build_base_func, intermediate_supervision, intermediate_layer, output_stride)
206
207 if with_preprocess_lambda not in [False, 0, None]:
--> 208 base_model = build_base_func(input_tensor=normalized_img, output_stride=output_stride, return_model=True)
209 else:
210 base_model = build_base_func(input_tensor=input_img, output_stride=output_stride, return_model=True)
~/alibaba/repos/KerasPersonLab/resnet101.py in get_resnet101_base(input_tensor, output_stride, return_model)
150
151 model = KM.Model(input_tensor_source, x)
--> 152 weights_path = get_file('resnet101_weights_notop.h5', None, cache_subdir='models')
153 model.load_weights(weights_path)
154
~/virtualenvironment/tensorflow/lib/python3.6/site-packages/keras/utils/data_utils.py in get_file(fname, origin, untar, md5_hash, file_hash, cache_subdir, hash_algorithm, extract, archive_format, cache_dir)
220 try:
221 try:
--> 222 urlretrieve(origin, fpath, dl_progress)
223 except URLError as e:
224 raise Exception(error_msg.format(origin, e.errno, e.reason))
/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py in urlretrieve(url, filename, reporthook, data)
244 data file as well as the resulting HTTPMessage object.
245 """
--> 246 url_type, path = splittype(url)
247
248 with contextlib.closing(urlopen(url, data)) as fp:
/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/parse.py in splittype(url)
936 _typeprog = re.compile('([^/:]+):(.*)', re.DOTALL)
937
--> 938 match = _typeprog.match(url)
939 if match:
940 scheme, data = match.groups()
TypeError: expected string or bytes-like object
Hi!
First, thank you for your hard work!
I am trying to run training and some time after the first epoch begins, I am running into an issue where the following call (in transformer.py) fails:
masks = cv2.warpAffine(masks, M, cv_shape, flags=cv2.INTER_CUBIC, borderMode=cv2.BORDER_CONSTANT, borderValue=0)
Specifically, there is an OpenCV assertion error, stating that "error: (-215) _src.channels() <= 4 || (interpolation != INTER_LANCZOS4 && interpolation != INTER_CUBIC) in function cv::warpAffine". Could you please point me at what might be wrong here?
Thanks a lot!
N.B. Looking at the code, at first glace it would so seem that several Mats are passed into warpAffine(), is this intentional?
when I use python 3, I get the following Error:
TypeError: 'KeysView' object does not support indexing
so when use python 3, I should modify the function data_gen() in tf_data_generator.py in line87:
key = root.keys()
to
key = list(root.keys())
and now key could be iterated.
I think you should tell the environment detail in README so that other people could avoid such mistakes
Great job, thank you for your sharing! Then, could you please report the average precision (AP) of this project on COOC dataset?
Hi,thanks for sharing!
I wonder which version should I use?
When running the following code:
parallel_model.fit(steps_per_epoch=64115//batch_size,
epochs=config.NUM_EPOCHS,
callbacks=callbacks)
Getting this error.
WARNING:tensorflow:From /usr/local/lib/python3.7/site-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
Epoch 1/125
---------------------------------------------------------------------------
UnknownError Traceback (most recent call last)
<ipython-input-11-a6811e3578ae> in <module>
1 parallel_model.fit(steps_per_epoch=64115//batch_size,
2 epochs=config.NUM_EPOCHS,
----> 3 callbacks=callbacks)
/usr/local/lib/python3.7/site-packages/keras/engine/training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, **kwargs)
1037 initial_epoch=initial_epoch,
1038 steps_per_epoch=steps_per_epoch,
-> 1039 validation_steps=validation_steps)
1040
1041 def evaluate(self, x=None, y=None,
/usr/local/lib/python3.7/site-packages/keras/engine/training_arrays.py in fit_loop(model, f, ins, out_labels, batch_size, epochs, verbose, callbacks, val_f, val_ins, shuffle, callback_metrics, initial_epoch, steps_per_epoch, validation_steps)
152 batch_logs['size'] = 1
153 callbacks.on_batch_begin(step_index, batch_logs)
--> 154 outs = f(ins)
155
156 outs = to_list(outs)
/usr/local/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py in __call__(self, inputs)
2713 return self._legacy_call(inputs)
2714
-> 2715 return self._call(inputs)
2716 else:
2717 if py_any(is_tensor(x) for x in inputs):
/usr/local/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py in _call(self, inputs)
2673 fetched = self._callable_fn(*array_vals, run_metadata=self.run_metadata)
2674 else:
-> 2675 fetched = self._callable_fn(*array_vals)
2676 return fetched[:len(self.outputs)]
2677
/usr/local/lib/python3.7/site-packages/tensorflow/python/client/session.py in __call__(self, *args, **kwargs)
1437 ret = tf_session.TF_SessionRunCallable(
1438 self._session._session, self._handle, args, status,
-> 1439 run_metadata_ptr)
1440 if run_metadata:
1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)
/usr/local/lib/python3.7/site-packages/tensorflow/python/framework/errors_impl.py in __exit__(self, type_arg, value_arg, traceback_arg)
526 None, None,
527 compat.as_text(c_api.TF_Message(self.status.status)),
--> 528 c_api.TF_GetCode(self.status.status))
529 # Delete the underlying status object from memory otherwise it stays alive
530 # as there is a reference to status from this from the traceback due to
UnknownError: OSError: Unable to open file (unable to lock file, errno = 35, error message = 'Resource temporarily unavailable')
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/tensorflow/python/ops/script_ops.py", line 207, in __call__
ret = func(*args)
File "/usr/local/lib/python3.7/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 449, in generator_py_func
values = next(generator_state.get_iterator(iterator_id))
File "/Users/ghafran/git/KerasPersonLab/tf_data_generator.py", line 85, in data_gen
h5 = h5py.File(config.H5_DATASET, 'r')
File "/usr/local/lib/python3.7/site-packages/h5py/_hl/files.py", line 394, in __init__
swmr=swmr)
File "/usr/local/lib/python3.7/site-packages/h5py/_hl/files.py", line 170, in make_fid
fid = h5f.open(name, flags, fapl=fapl)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py/h5f.pyx", line 85, in h5py.h5f.open
OSError: Unable to open file (unable to lock file, errno = 35, error message = 'Resource temporarily unavailable')
[[{{node PyFunc}}]]
[[{{node IteratorGetNext}}]]
Hi, thanks a lot for the implementation. there's something that i couldn't figure out after reading the paper and reviewing the code. during the computation of the heatmaps the accumulate_votes function is called and some sort of bilinear interpolation is made. could someone please clarify which function is interpolated (in the code it seems to me that the values of this function are stored in the variable ps) and where do the interpolation weights come from (dx,dy) and what do they mean?
i'm referring to the following part of the code in the post_proc.py:
def accumulate_votes(votes, shape):
xs = votes[:,0]
ys = votes[:,1]
ps = votes[:,2]
tl = [np.floor(ys).astype('int32'), np.floor(xs).astype('int32')]
tr = [np.floor(ys).astype('int32'), np.ceil(xs).astype('int32')]
bl = [np.ceil(ys).astype('int32'), np.floor(xs).astype('int32')]
br = [np.ceil(ys).astype('int32'), np.ceil(xs).astype('int32')]
dx = xs - tl[1]
dy = ys - tl[0]
tl_vals = ps*(1.-dx)(1.-dy)
tr_vals = psdx*(1.-dy)
bl_vals = psdy(1.-dx)
br_vals = psdydx
hi,all
One error happens when training from scratch.
KerasPersonLab-updated_model_def/data_prep.py", line 16, in get_ground_truth
assert(instance_masks.shape[-1] == len(all_keypoints)), '{} != {}'.format(instance_masks.shape[-1], len(all_keypoints))
AssertionError: 7 != 8
Grateful for your implementation! I have some questions about the bilinear sampler.
Line 49 in 32d44dd
base = base + bilinear_sampler(offsets, base)
instead of base = base + bilinear_sampler(base, offsets)
? Does this mean that we interpolate the offset
according to the base
?
Line 43 in 32d44dd
iy0 = vy0 + h
, then the iy0
will represent the destination position of offsets. Why using these positions when iy0 = tf.where(mask, tf.zeros_like(iy0), iy0)
?
In this condition,
x00 = tf.gather_nd(x, i00)
x01 = tf.gather_nd(x, i01)
x10 = tf.gather_nd(x, i10)
x11 = tf.gather_nd(x, i11)
will gather values from the destination positions of predicted offsets, shouldn't we gather values from the start positions of offsets?
It seems like that no dilation convolution in origin PersonLab paper, according to classical Resnet .
Did I miss or ignore something important in paper?
I've encountered a couple of errors in my code I was hoping you could help me fix:
I would like to visualize the model. Are you able to generate it for me?
Does someone have managed to get results with this repository?
My loss is near constant, oscillating around 1.50.
I should mention that I have removed the instance segmentation module, but that shouldn't impact the pose estimation module.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.