syscv / sam-pt Goto Github PK
View Code? Open in Web Editor NEWSAM-PT: Extending SAM to zero-shot video segmentation with point-based tracking.
Home Page: https://arxiv.org/abs/2307.01197
License: Apache License 2.0
SAM-PT: Extending SAM to zero-shot video segmentation with point-based tracking.
Home Page: https://arxiv.org/abs/2307.01197
License: Apache License 2.0
the latest detectron2 requires pytorch>=1.18 but since the instructions tells us to do conda install pytorch==1.12.0 torchvision==0.13.0 torchaudio==0.12.0 cudatoolkit=11.3 -c pytorch
it always fails.
Installing detectron2 with a torch>=1.8 fails the demo. Instead using python -m pip install detectron2 -f \ https://dl.fbaipublicfiles.com/detectron2/wheels/cu113/torch1.10/index.html
to install detectron2 solves this issue.
I've already change the 'use_point_reinit' to true, but it doesn't change at all,still remain false.
Hi, I need to create model outside sam-pt directory
For now I get the following error
error locating target sam_pt.modelling.sam_pt.SamPt
How do I change _target_
in config in order this to work?
Error executing job with overrides: ["frames_path='D:\computer_vision\sam-pt\data\demo_data\demo'", 'query_points_path=null', 'longest_side_length=1024', 'frame_stride=1', 'max_fr
ames=-1']
Traceback (most recent call last):
File "E:\Anaconda3\envs\sam\lib\runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "E:\Anaconda3\envs\sam\lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "D:\computer_vision\sam-pt\demo\demo.py", line 372, in
main()
File "E:\Anaconda3\envs\sam\lib\site-packages\hydra\main.py", line 94, in decorated_main
_run_hydra(
File "E:\Anaconda3\envs\sam\lib\site-packages\hydra_internal\utils.py", line 394, in _run_hydra
_run_app(
File "E:\Anaconda3\envs\sam\lib\site-packages\hydra_internal\utils.py", line 457, in _run_app
run_and_report(
File "E:\Anaconda3\envs\sam\lib\site-packages\hydra_internal\utils.py", line 223, in run_and_report
raise ex
File "E:\Anaconda3\envs\sam\lib\site-packages\hydra_internal\utils.py", line 220, in run_and_report
return func()
File "E:\Anaconda3\envs\sam\lib\site-packages\hydra_internal\utils.py", line 458, in
lambda: hydra.run(
File "E:\Anaconda3\envs\sam\lib\site-packages\hydra_internal\hydra.py", line 132, in run
_ = ret.return_value
File "E:\Anaconda3\envs\sam\lib\site-packages\hydra\core\utils.py", line 260, in return_value
raise self._return_value
File "E:\Anaconda3\envs\sam\lib\site-packages\hydra\core\utils.py", line 186, in run_job
ret.return_value = task_function(task_cfg)
File "D:\computer_vision\sam-pt\demo\demo.py", line 50, in main
model = load_model(cfg, positive_points_per_mask, negative_points_per_mask)
File "D:\computer_vision\sam-pt\demo\demo.py", line 111, in load_model
return model.to("cuda" if torch.cuda.is_available() else "cpu").eval()
File "E:\Anaconda3\envs\sam\lib\site-packages\torch\nn\modules\module.py", line 927, in to
return self._apply(convert)
File "E:\Anaconda3\envs\sam\lib\site-packages\torch\nn\modules\module.py", line 579, in _apply
module._apply(fn)
File "E:\Anaconda3\envs\sam\lib\site-packages\torch\nn\modules\module.py", line 579, in _apply
module._apply(fn)
File "E:\Anaconda3\envs\sam\lib\site-packages\torch\nn\modules\module.py", line 579, in _apply
module._apply(fn)
[Previous line repeated 3 more times]
File "E:\Anaconda3\envs\sam\lib\site-packages\torch\nn\modules\module.py", line 602, in _apply
param_applied = fn(param)
File "E:\Anaconda3\envs\sam\lib\site-packages\torch\nn\modules\module.py", line 925, in convert
return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 2.00 GiB total capacity; 1.69 GiB already allocated; 0 bytes free; 1.74 GiB reserved in total by PyTorch) If reser
ved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
wandb: Waiting for W&B process to finish... (failed 1). Press Ctrl-C to abort syncing.
wandb: / 0.021 MB of 0.021 MB uploaded (0.000 MB deduped)
wandb: Run summary:
wandb: work_dir D:\computer_vision\s...
wandb:
wandb: View run debug_72_2023.08.12_15.26.10 at: https://wandb.ai/fangyuanguyue38/demo/runs/xodo3pvq
wandb: Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
wandb: Find logs at: .\wandb\run-20230812_152614-xodo3pvq\logs
Exception in thread ChkStopThr:
Traceback (most recent call last):
File "E:\Anaconda3\envs\sam\lib\threading.py", line 932, in _bootstrap_inner
self.run()
File "E:\Anaconda3\envs\sam\lib\threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "E:\Anaconda3\envs\sam\lib\site-packages\wandb\sdk\wandb_run.py", line 278, in check_stop_status
self._loop_check_status(
File "E:\Anaconda3\envs\sam\lib\site-packages\wandb\sdk\wandb_run.py", line 216, in _loop_check_status
local_handle = request()
File "E:\Anaconda3\envs\sam\lib\site-packages\wandb\sdk\interface\interface.py", line 787, in deliver_stop_status
Fatal Python error: could not acquire lock for <_io.BufferedWriter name=''> at interpreter shutdown, possibly due to daemon threads
Python runtime state: finalizing (tstate=000001B9DBB4C980)
Thread 0x0000524c (most recent call first):
File "E:\Anaconda3\envs\sam\lib\threading.py", line 302 in wait
File "E:\Anaconda3\envs\sam\lib\site-packages\sentry_sdk_queue.py", line 240 in get
File "E:\Anaconda3\envs\sam\lib\site-packages\sentry_sdk\worker.py", line 127 in _target
File "E:\Anaconda3\envs\sam\lib\threading.py", line 870 in run
File "E:\Anaconda3\envs\sam\lib\threading.py", line 932 in _bootstrap_inner
File "E:\Anaconda3\envs\sam\lib\threading.py", line 890 in _bootstrap
Thread 0x000053b0 (most recent call first):
File "E:\Anaconda3\envs\sam\lib\site-packages\sentry_sdk\sessions.py", line 117 in _thread
File "E:\Anaconda3\envs\sam\lib\threading.py", line 870 in run
File "E:\Anaconda3\envs\sam\lib\threading.py", line 932 in _bootstrap_inner
File "E:\Anaconda3\envs\sam\lib\threading.py", line 890 in _bootstrap
Thread 0x0000157c (most recent call first):
File "E:\Anaconda3\envs\sam\lib\threading.py", line 306 in wait
File "E:\Anaconda3\envs\sam\lib\threading.py", line 558 in wait
File "E:\Anaconda3\envs\sam\lib\site-packages\wandb\sdk\lib\mailbox.py", line 126 in _wait
File "E:\Anaconda3\envs\sam\lib\site-packages\wandb\sdk\lib\mailbox.py", line 130 in _get_and_clear
File "E:\Anaconda3\envs\sam\lib\site-packages\wandb\sdk\lib\mailbox.py", line 283 in wait
File "E:\Anaconda3\envs\sam\lib\site-packages\wandb\sdk\wandb_run.py", line 224 in _loop_check_status
File "E:\Anaconda3\envs\sam\lib\site-packages\wandb\sdk\wandb_run.py", line 260 in check_network_status
File "E:\Anaconda3\envs\sam\lib\threading.py", line 870 in run
File "E:\Anaconda3\envs\sam\lib\threading.py", line 932 in _bootstrap_inner
File "E:\Anaconda3\envs\sam\lib\threading.py", line 890 in _bootstrap
Thread 0x00003f8c (most recent call first):
File "E:\Anaconda3\envs\sam\lib\site-packages\wandb\sdk\lib\redirect.py", line 640 in write
File "E:\Anaconda3\envs\sam\lib\threading.py", line 1202 in invoke_excepthook
File "E:\Anaconda3\envs\sam\lib\threading.py", line 934 in _bootstrap_inner
File "E:\Anaconda3\envs\sam\lib\threading.py", line 890 in _bootstrap
Current thread 0x00005b04 (most recent call first):
Thank you for your great work! Would you mind sharing the code to get the evaluation results for DAVIS2016, YouTube2018 and MOSE? Thanks a lot!
hello this work is this repository is very nice,
however i got some issue when i try to install the requirements.txt
its report this error
note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for detectron2 Running setup.py clean for detectron2 Failed to build detectron2 ERROR: Could not build wheels for detectron2, which is required to install pyproject.toml-based projects
I think that the problem is on the git+https://github.com/facebookresearch/[email protected]
i try to install manually but its doesn't work either
I'd be grateful for your assistance if you could lend a hand.
(sam) D:\computer_vision\sam-pt>python -m demo.demo frames_path='sam-pt/data/demo_data/demo/' query_points_path=null longest_side_length=1024 frame_stride=1 max_frames=-1
E:\Anaconda3\envs\sam\lib\site-packages\hydra_internal\hydra.py:119: UserWarning: Future Hydra versions will no longer change working directory at job runtime by default.
See https://hydra.cc/docs/1.2/upgrades/1.1_to_1.2/changes_to_job_working_dir/ for more information.
ret = run_job(
Setting seed to 72
model:
point_tracker:
target: sam_pt.point_tracker.pips.PipsPointTracker
checkpoint_path: ${hydra:runtime.cwd}/models/pips_ckpts/reference_model
stride: 4
s: 8
initial_next_frame_visibility_threshold: 0.9
sam_predictor:
sam_model:
image_encoder:
target: segment_anything.modeling.image_encoder.ImageEncoderViT
depth: 32
embed_dim: 1280
img_size: ${ ..image_size }
mlp_ratio: 4
norm_layer:
partial: true
target: torch.nn.LayerNorm
eps: 1.0e-06
num_heads: 16
patch_size: ${ ..vit_patch_size }
qkv_bias: true
use_rel_pos: true
global_attn_indexes:
- 7
- 15
- 23
- 31
window_size: 14
out_chans: ${ ..prompt_embed_dim }
prompt_encoder:
target: segment_anything.modeling.prompt_encoder.PromptEncoder
embed_dim: ${ ..prompt_embed_dim }
image_embedding_size:
- ${ ...image_embedding_size }
- ${ ...image_embedding_size }
input_image_size:
- ${ ...image_size }
- ${ ...image_size }
mask_in_chans: 16
mask_decoder:
target: segment_anything.modeling.mask_decoder.MaskDecoder
num_multimask_outputs: 3
transformer:
target: segment_anything.modeling.transformer.TwoWayTransformer
depth: 2
embedding_dim: ${ ...prompt_embed_dim }
mlp_dim: 2048
num_heads: 8
transformer_dim: ${ ..prompt_embed_dim }
iou_head_depth: 3
iou_head_hidden_dim: 256
target: sam_pt.modeling.sam.SamHydra
checkpoint: ${hydra:runtime.cwd}/models/sam_ckpts/sam_vit_h_4b8939.pth
prompt_embed_dim: 256
image_size: 1024
vit_patch_size: 16
image_embedding_size: 64
pixel_mean:
- 123.675
- 116.28
- 103.53
pixel_std:
- 58.395
- 57.12
- 57.375
target: segment_anything.predictor.SamPredictor
target: sam_pt.modeling.sam_pt.SamPt
sam_iou_threshold: 0.7
iterative_refinement_iterations: 12
positive_point_selection_method: kmedoids
negative_point_selection_method: mixed
positive_points_per_mask: -1
negative_points_per_mask: -1
add_other_objects_positive_points_as_negative_points: true
point_tracker_mask_batch_size: 5
use_patch_matching_filtering: false
patch_size: 3
patch_similarity_threshold: 0.01
use_point_reinit: false
reinit_point_tracker_horizon: 8
reinit_horizon: 8
reinit_variant: reinit-on-horizon-and-sync-masks
logging:
wandb:
entity: null
project: demo
tensorboard: true
log_code_path: ${hydra:runtime.cwd}/sam_pt
debug: false
exp_id: debug_72_2023.08.12_14.24.36
frames_path: sam-pt/data/demo_data/demo/
query_points_path: null
longest_side_length: 1024
frame_stride: 1
max_frames: -1
seed: 72
annot_size: 16
annot_line_width: 6
wandb: W&B API key is configured. Use wandb login --relogin
to force relogin
wandb: Network error (SSLError), entering retry loop.
wandb: Network error (SSLError), entering retry loop.
[2023-08-12 14:25:45,470][urllib3.connectionpool][WARNING] - Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLZero
ReturnError(6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:1131)'))': /api/4504800232407040/store/
[2023-08-12 14:25:46,266][urllib3.connectionpool][WARNING] - Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLZero
ReturnError(6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:1131)'))': /api/4504800232407040/store/
Problem at: D:\computer_vision\sam-pt\demo\demo.py 75 setup_logging
Error executing job with overrides: ["frames_path='sam-pt/data/demo_data/demo/'", 'query_points_path=null', 'longest_side_length=1024', 'frame_stride=1', 'max_frames=-1']
wandb: ERROR Run initialization has timed out after 60.0 sec.
wandb: ERROR Please refer to the documentation for additional information: https://docs.wandb.ai/guides/track/tracking-faq#initstarterror-error-communicating-with-wandb-process-
Traceback (most recent call last):
File "E:\Anaconda3\envs\sam\lib\runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "E:\Anaconda3\envs\sam\lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "D:\computer_vision\sam-pt\demo\demo.py", line 372, in
main()
File "E:\Anaconda3\envs\sam\lib\site-packages\hydra\main.py", line 94, in decorated_main
_run_hydra(
File "E:\Anaconda3\envs\sam\lib\site-packages\hydra_internal\utils.py", line 394, in _run_hydra
_run_app(
File "E:\Anaconda3\envs\sam\lib\site-packages\hydra_internal\utils.py", line 457, in _run_app
run_and_report(
File "E:\Anaconda3\envs\sam\lib\site-packages\hydra_internal\utils.py", line 223, in run_and_report
raise ex
File "E:\Anaconda3\envs\sam\lib\site-packages\hydra_internal\utils.py", line 220, in run_and_report
return func()
File "E:\Anaconda3\envs\sam\lib\site-packages\hydra_internal\utils.py", line 458, in
lambda: hydra.run(
File "E:\Anaconda3\envs\sam\lib\site-packages\hydra_internal\hydra.py", line 132, in run
_ = ret.return_value
File "E:\Anaconda3\envs\sam\lib\site-packages\hydra\core\utils.py", line 260, in return_value
raise self._return_value
File "E:\Anaconda3\envs\sam\lib\site-packages\hydra\core\utils.py", line 186, in run_job
ret.return_value = task_function(task_cfg)
File "D:\computer_vision\sam-pt\demo\demo.py", line 42, in main
setup_logging(cfg)
File "D:\computer_vision\sam-pt\demo\demo.py", line 75, in setup_logging
wandb.init(
File "E:\Anaconda3\envs\sam\lib\site-packages\wandb\sdk\wandb_init.py", line 1169, in init
raise e
File "E:\Anaconda3\envs\sam\lib\site-packages\wandb\sdk\wandb_init.py", line 1150, in init
run = wi.init()
File "E:\Anaconda3\envs\sam\lib\site-packages\wandb\sdk\wandb_init.py", line 769, in init
raise error
wandb.errors.CommError: Run initialization has timed out after 60.0 sec.
Please refer to the documentation for additional information: https://docs.wandb.ai/guides/track/tracking-faq#initstarterror-error-communicating-with-wandb-process-
[2023-08-12 14:25:47,013][urllib3.connectionpool][WARNING] - Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLZero
ReturnError(6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:1131)'))': /api/4504800232407040/store/
[2023-08-12 14:25:48,531][urllib3.connectionpool][WARNING] - Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLZero
ReturnError(6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:1131)'))': /api/4504800232407040/envelope/
[2023-08-12 14:25:49,424][urllib3.connectionpool][WARNING] - Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLZero
ReturnError(6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:1131)'))': /api/4504800232407040/envelope/
[2023-08-12 14:25:50,050][urllib3.connectionpool][WARNING] - Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLZero
ReturnError(6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:1131)'))': /api/4504800232407040/envelope/
Thanks for your great work, but I meet some errors when I running the demo. I tried to change the PYTHONPATH and check the init.py files, but I still have no idea about how to solve it.
Error executing job with overrides: ['frames_path=${hydra:runtime.cwd}/data/demo_data/bees/', 'query_points_path=${hydra:runtime.cwd}/data/demo_data/query_points__bees.txt', 'longest_side_length=1024', 'frame_stride=1', 'max_frames=-1', 'model.use_point_reinit=true']
Traceback (most recent call last):
File "/conda/envs/sampt/lib/python3.8/site-packages/hydra/_internal/utils.py", line 644, in _locate
obj = getattr(obj, part)
AttributeError: module 'sam_pt.modeling' has no attribute 'sam_pt'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/conda/envs/sampt/lib/python3.8/site-packages/hydra/_internal/utils.py", line 650, in _locate
obj = import_module(mod)
File "/conda/envs/sampt/lib/python3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 843, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/sam-pt/sam_pt/modeling/sam_pt.py", line 15, in <module>
from sam_pt.point_tracker import PointTracker, SuperGluePointTracker
File "/sam-pt/sam_pt/point_tracker/__init__.py", line 7, in <module>
from .cotracker import CoTrackerPointTracker
File "/sam-pt/sam_pt/point_tracker/cotracker/__init__.py", line 1, in <module>
from .tracker import CoTrackerPointTracker
File "/sam-pt/sam_pt/point_tracker/cotracker/tracker.py", line 6, in <module>
from cotracker.models.core.cotracker.cotracker import CoTracker
ImportError: cannot import name 'CoTracker' from 'cotracker.models.core.cotracker.cotracker' (/sam-pt/co-tracker/cotracker/models/core/cotracker/cotracker.py)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/conda/envs/sampt/lib/python3.8/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 134, in _resolve_target
target = _locate(target)
File "/conda/envs/sampt/lib/python3.8/site-packages/hydra/_internal/utils.py", line 658, in _locate
raise ImportError(
ImportError: Error loading 'sam_pt.modeling.sam_pt.SamPt':
ImportError("cannot import name 'CoTracker' from 'cotracker.models.core.cotracker.cotracker' (/sam-pt/co-tracker/cotracker/models/core/cotracker/cotracker.py)")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/conda/envs/sampt/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/conda/envs/sampt/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/sam-pt/demo/demo.py", line 371, in <module>
main()
File "/conda/envs/sampt/lib/python3.8/site-packages/hydra/main.py", line 94, in decorated_main
_run_hydra(
File "/conda/envs/sampt/lib/python3.8/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
_run_app(
File "/conda/envs/sampt/lib/python3.8/site-packages/hydra/_internal/utils.py", line 457, in _run_app
run_and_report(
File "/conda/envs/sampt/lib/python3.8/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
raise ex
File "/conda/envs/sampt/lib/python3.8/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
return func()
File "/conda/envs/sampt/lib/python3.8/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
lambda: hydra.run(
File "/conda/envs/sampt/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 132, in run
_ = ret.return_value
File "/conda/envs/sampt/lib/python3.8/site-packages/hydra/core/utils.py", line 260, in return_value
raise self._return_value
File "/conda/envs/sampt/lib/python3.8/site-packages/hydra/core/utils.py", line 186, in run_job
ret.return_value = task_function(task_cfg)
File "/sam-pt/demo/demo.py", line 49, in main
model = load_model(cfg, positive_points_per_mask, negative_points_per_mask)
File "/sam-pt/demo/demo.py", line 109, in load_model
model = instantiate(cfg.model)
File "/conda/envs/sampt/lib/python3.8/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 226, in instantiate
return instantiate_node(
File "/conda/envs/sampt/lib/python3.8/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 333, in instantiate_node
_target_ = _resolve_target(node.get(_Keys.TARGET), full_key)
File "/conda/envs/sampt/lib/python3.8/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 139, in _resolve_target
raise InstantiationException(msg) from e
hydra.errors.InstantiationException: Error locating target 'sam_pt.modeling.sam_pt.SamPt', set env var HYDRA_FULL_ERROR=1 to see chained exception.
full_key: model
Hello, thanks for your work.
I installed the checkpoints and set up the environment but cannot run the demo.
python -m demo.demo frames_path='/home/bbnclxy/ext16t/lyx-star/sam-pt-main/data/demo_data/bees/' query_points_path=null longest_side_length=1024 frame_stride=1 max_frames=-1
it reports error:
Hello, as describe in the dataset prepare part, the second google drive path is no longer available, how can i get the uvo dense and sparse dataset?
"
cd data
pip install gdown
gdown --no-check-certificate https://drive.google.com/uc?id=1AGu4BL-i_vDCMNtwsoSuo5wIyDVd5dRf
unzip UVOv1.0.zip
rm UVOv1.0.zip
gdown --no-check-certificate --folder https://drive.google.com/drive/folders/1fOhEdHqrp_6D_tBsrR9hazDLYV2Sw1XC
unzip UVO_Videos/uvo_videos_dense.zip
unzip UVO_Videos/uvo_videos_sparse.zip
mv uvo_videos_dense/ UVOv1.0/
mv uvo_videos_sparse/ UVOv1.0/
rm -rf UVO_Videos/
rm -rf __MACOSX/"
Can you provide a guideline for training on a custom dataset?
Hi,
Has this model been implemented without any training process?
And is the only required training process the training of HQ-SAM independently?
Alternatively, please provide the GPT/memory usage details for training this model.
Thank you.
First of all thanks for this amazing work..
I am looking for a demo notebook/example to test it out on individual image/s, is there any available?
Hello,
First of all, congrats! Great work.
I run a few test and it has remarkable precision. Except one case: disappearing-reappearing objects, works really poorly (almost every case the tracking was lost) . I'm trying to track different, moving objects with moving camera. I haven't dug into your work in detail yet, but do you have any suggestions for that scenario? (more positive points for example?).
I tested TAM also and it's working really well in this case(because of XMem), but has worse precision with small, blurry objects.
Hi again :)
Your work is truly amazing, although, I wonder If model won't be able to work correctly with, for example, 3-5 frame video?
For now, I have an issue when for short sequences I get zero masks and trajectories, scores are -inf.
Could It be a consequence of no object motion?
Hi, there's a problem when I try to use HQ-SAM-Light-VIT-T model
In sam_pt.yaml I set
- sam@sam_predictor.sam_model: samhq_light_vit_tiny
and when running demo I get
Traceback (most recent call last):
File "C:\Users\Alina\Desktop\Workplace\sam-pt\demo\demo.py", line 55, in main
logits, trajectories, visibilities, scores = run_inference(model, rgbs, query_points, target_hw)
File "C:\Users\Alina\Desktop\Workplace\sam-pt\demo\demo.py", line 131, in run_inference
outputs = model(video)
File "C:\Users\Alina\anaconda3\envs\sam-pt\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "C:\Users\Alina\Desktop\Workplace\sam-pt\sam_pt\modeling\sam_pt.py", line 176, in forward
query_masks = self.extract_query_masks(images, query_points)
File "C:\Users\Alina\Desktop\Workplace\sam-pt\sam_pt\modeling\sam_pt.py", line 324, in extract_query_masks
_, query_masks_logits, _ = self._apply_sam_to_trajectories(
File "C:\Users\Alina\Desktop\Workplace\sam-pt\sam_pt\modeling\sam_pt.py", line 843, in _apply_sam_to_trajectories
mask_frame_logits, iou_prediction_score = predict_mask(visible_point_coords, visible_point_labels)
File "C:\Users\Alina\Desktop\Workplace\sam-pt\sam_pt\modeling\sam_pt.py", line 783, in predict_mask
_, _, low_res_masks = self.sam_predictor.predict_torch(
File "C:\Users\Alina\anaconda3\envs\sam-pt\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "C:\Users\Alina\anaconda3\envs\sam-pt\lib\site-packages\segment_anything\predictor.py", line 229, in predict_torch
low_res_masks, iou_predictions = self.model.mask_decoder(
File "C:\Users\Alina\anaconda3\envs\sam-pt\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
TypeError: forward() missing 2 required positional arguments: 'hq_token_only' and 'interm_embeddings'
Thanks for your work, how much gpu and how much memory are required for training and testing?
我阅读了你们的论文,其中提到了通过提供初始帧掩码的方式进行采样,然后进行跟踪的方法。我想要知道你们提供了相关的测试代码吗?如果有提供就太好了
I read your paper which mentions a method of sampling by providing an initial frame mask and then tracking it. I would like to know if you have provided any test code for this? It would be great if you do.
You are not mentioning any time performance results in your paper. Can we use it for segmenting real time ? Are there any fps rates results of your runs or how much time does it takes per frame with your setup ?
Thanks for your great work,
best wishes,
Hello, thanks for your work.
I installed the checkpoints and setup the environment but I am not able to run the demo.
[...]
$ cd sam-pt/
$ python -m demo.demo frames_path=~/Documents/CoTracker-SAM/samples/horsejump-stick/ query_points_path=null
Error while finding module specification for 'demo.demo' (ModuleNotFoundError: __path__ attribute not found on 'demo' while trying to find 'demo.demo')
Hi! I'm trying to replicate the demo, and I'm running into an issue. I have CUDA 11.8 installed. TensorRT is also installed and can be loaded into python, so I'm not sure what the issue is there.
I tried fresh installs of the environment, CUDA, Tensorflow, Pythorch and TensorRT.
This is the relevant log:
Loading PIPS model from /mnt/sam-pt/models/pips_ckpts/reference_model
reading ckpt from /mnt/sam-pt/models/pips_ckpts/reference_model
Error executing job with overrides: ['frames_path=${hydra:runtime.cwd}/data/demo_data/bees/', 'query_points_path=null', 'longest_side_length=1024', 'frame_stride=1', 'max_frames=-1']
Error in call to target 'sam_pt.point_tracker.pips.tracker.PipsPointTracker':
AssertionError()
full_key: model.point_tracker
Hi, Can I ask if sam-pt can automatically track?
Hi,
Thanks for your work again.
I have two questions about the code.
For the non-interactive demo, passing a different number of positive points for each object is not possible. Is there any way that I can pass the different number of positive points?
The other question is about "mask_n_timestep" which you mentioned here
Can you explain how does it work?
I thought "mask_n_timestep" means the frame number where an object appears in the frame for the first time.
But I experimented on the same sequence and the results were like these:
If we only have three objects in the "Query Points" text file and they all appear in the first frame:
If we have the same objects in the "Query Points" text file + three other points that appear in the next frames for the first time (Like frames 234, 520, 640)] :
If we have the same objects in the "Query Points" text file + only one other point that appears in the next frames for the first time (Like frames 234)] :
I don't why there are some random points appear in the first frame although I mentioned the right frame number for them and I followed your format.
Thanks for your answers in advance.
can I use it for 3d. can you please share some information
Hi, thanks for your efforts and novel ideas.
However, as I am running demo or experiments, I was asked to either enter API or API key not found. (3 options)
Would you kindly provide more detailed tutorial video or instructions?
Thanks again for your time.
Do you plan to release the code?
if so, do you know when?
Thanks, this work looks great!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.