avanetten / cresi Goto Github PK

View Code? Open in Web Editor NEW

173.0 173.0 59.0 326.61 MB

Road network extraction from satellite imagery, with speed and travel time estimates

License: Apache License 2.0

Dockerfile 0.03% Python 2.05% Shell 0.01% Jupyter Notebook 90.11% HTML 7.81%

cresi's People

Contributors

Stargazers

Watchers

cresi's Issues

Several errors when I try to build the image on Win 10 Pro

I have been trying to build the cresi image according to the instructions on GitHub, and rooting the docker located in "gpu folder", but I am encountering several errors stopping the process. I wish it was possible to pull the image automatically which prevents such issues. Anyway, has anyone have any ideas? Any help would be appreciated.

where is road_raster.py

Dear avanetten,

I am trying to reproduce the continuous speed mask extraction algorithm from your codes. There are get_contours_complex() function and get_mask() function which are unreferenced. Could you share with me the link to the road_raster.py?

Kind Regards,
Shicheng Zu

Unable to build the image from Dockerfile

There are 2 errors while I tried to build the image. I walk-around Error1 by comment that line and then I encounter Erro2...

Error1:
ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini /usr/bin/tini
Log:

Step 11/30 : ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini /usr/bin/tini
ADD failed: Get https://github-production-release-asset-2e65be.s3.amazonaws.com/31432573/f039db52-89a3-11e7-8e08-44ac01cb80d5?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20200522%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200522T015128Z&X-Amz-Expires=300&X-Amz-Signature=f9a9f4a90b0c526c0502c9c7aae5497b0b316155b1d651b47e5c8febe8330d69&X-Amz-SignedHeaders=host&actor_id=0&repo_id=31432573&response-content-disposition=attachment%3B%20filename%3Dtini&response-content-type=application%2Foctet-stream: x509: certificate signed by unknown authority

Error2:
RUN conda install -n cresi pytorch torchvision cuda90 -c pytorch && conda clean -p && conda clean -t && conda clean --yes --all
Log

Downloading and Extracting Packages
pytorch-1.4.0        | 432.9 MB  |            |   0%
intel-openmp-2020.1  | 780 KB    | ########## | 100%
ninja-1.10.0         | 1.9 MB    | ########## | 100%
cuda90-1.0           | 3 KB      |            |   0%
mkl-2020.1           | 129.0 MB  | ########## | 100%
torchvision-0.5.0    | 9.1 MB    |            |   0%
cudatoolkit-10.1.243 | 347.4 MB  | ########## | 100%

CondaHTTPError: HTTP 000 CONNECTION FAILED for url <https://conda.anaconda.org/pytorch/linux-64/pytorch-1.4.0-py3.6_cuda10.1.243_cudnn7.6.3_0.tar.bz2>
Elapsed: -

An HTTP error occurred when trying to retrieve this URL.
HTTP errors are often intermittent, and a simple retry will get you on your way.

CondaHTTPError: HTTP 000 CONNECTION FAILED for url <https://conda.anaconda.org/pytorch/linux-64/cuda90-1.0-h6433d27_0.tar.bz2>
Elapsed: -

An HTTP error occurred when trying to retrieve this URL.
HTTP errors are often intermittent, and a simple retry will get you on your way.

CondaHTTPError: HTTP 000 CONNECTION FAILED for url <https://conda.anaconda.org/pytorch/linux-64/torchvision-0.5.0-py36_cu101.tar.bz2>
Elapsed: -

An HTTP error occurred when trying to retrieve this URL.
HTTP errors are often intermittent, and a simple retry will get you on your way.


The command '/bin/bash -c conda install -n cresi pytorch torchvision cuda90 -c pytorch  && conda clean -p       && conda clean -t       && conda clean --yes --all' returned a non-zero code: 1

Unable to generate graph if the image is not from SpaceNet

If I use my own .tif image, it's fine during step 2~4 but failed to generate graph in step 5.
Guess there exists some dependencies about SpaceNet dataset.
Is there any simple way to walk around that?

python 05_wkt_to_G.py /opt/cresi/cresi/configs/sn5_baseline.json

ERROR 1: No PROJ.4 translation for source SRS, coordinate transformation initialization has failed.
ERROR 1: No PROJ.4 translation for source SRS, coordinate transformation initialization has failed.
ERROR 1: No PROJ.4 translation for source SRS, coordinate transformation initialization has failed.
ERROR 1: No PROJ.4 translation for source SRS, coordinate transformation initialization has failed.
ERROR 1: No PROJ.4 translation for source SRS, coordinate transformation initialization has failed.
ERROR 1: No PROJ.4 translation for source SRS, coordinate transformation initialization has failed.
ERROR 1: No PROJ.4 translation for source SRS, coordinate transformation initialization has failed.
ERROR 1: No PROJ.4 translation for source SRS, coordinate transformation initialization has failed.
ERROR 1: No PROJ.4 translation for source SRS, coordinate transformation initialization has failed.
ERROR 1: No PROJ.4 translation for source SRS, coordinate transformation initialization has failed.
ERROR 1: No PROJ.4 translation for source SRS, coordinate transformation initialization has failed.
ERROR 1: No PROJ.4 translation for source SRS, coordinate transformation initialization has failed.
ERROR 1: No PROJ.4 translation for source SRS, coordinate transformation initialization has failed.
ERROR 1: No PROJ.4 translation for source SRS, coordinate transformation initialization has failed.
ERROR 1: No PROJ.4 translation for source SRS, coordinate transformation initialization has failed.
ERROR 1: No PROJ.4 translation for source SRS, coordinate transformation initialization has failed.
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/opt/conda/envs/cresi/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/opt/conda/envs/cresi/lib/python3.6/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "05_wkt_to_G.py", line 354, in pixelToGeoCoord
    geom.Transform(coord_trans)
  File "/opt/conda/envs/cresi/lib/python3.6/site-packages/osgeo/ogr.py", line 7371, in Transform
    return _ogr.Geometry_Transform(self, *args)
TypeError: in method 'Geometry_Transform', argument 2 of type 'OSRCoordinateTransformationShadow *'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "05_wkt_to_G.py", line 1283, in <module>
    main()
  File "05_wkt_to_G.py", line 1274, in main
    wkt_to_G(params[0])
  File "05_wkt_to_G.py", line 887, in wkt_to_G
    verbose=verbose)
  File "05_wkt_to_G.py", line 439, in get_node_geo_coords
    coords_dict_list = pool.map(pixelToGeoCoord, params)
  File "/opt/conda/envs/cresi/lib/python3.6/multiprocessing/pool.py", line 266, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/conda/envs/cresi/lib/python3.6/multiprocessing/pool.py", line 644, in get
    raise self._value
TypeError: in method 'Geometry_Transform', argument 2 of type 'OSRCoordinateTransformationShadow *'

Download failed

I tried to download the datasets according to your notebook but it failed.

$ aws s3 cp --recursive s3://spacenet-dataset/spacenet/SN5_roads/tarballs/ .
download: s3://spacenet-dataset/spacenet/SN5_roads/tarballs/SN5_roads_test_public_AOI_8_Mumbai.tar.gz to ./SN5_roads_test_public_AOI_8_Mumbai.tar.gz
download failed: s3://spacenet-dataset/spacenet/SN5_roads/tarballs/SN5_roads_test_public_AOI_9_San_Juan.tar.gz to ./SN5_roads_test_public_AOI_9_San_Juan.tar.gz [Errno 22] Invalid argument
download failed: s3://spacenet-dataset/spacenet/SN5_roads/tarballs/SN5_roads_test_public_AOI_7_Moscow.tar.gz to ./SN5_roads_test_public_AOI_7_Moscow.tar.gz [Errno 22] Invalid argument
download failed: s3://spacenet-dataset/spacenet/SN5_roads/tarballs/SN5_roads_train_AOI_7_Moscow.tar.gz to ./SN5_roads_train_AOI_7_Moscow.tar.gz [Errno 22] Invalid argument
download failed: s3://spacenet-dataset/spacenet/SN5_roads/tarballs/SN5_roads_train_AOI_8_Mumbai.tar.gz to ./SN5_roads_train_AOI_8_Mumbai.tar.gz [Errno 22] Invalid argument

unexpected keyword argument 'path_data_root'

cresi/cresi/configs/sn5_baseline.json

Line 4 in b996f2d

"path_data_root": "/path/to/data/cresi_data/",

TypeError: __new__() got an unexpected keyword argument 'path_data_root'

found using python 04_skeletonize.py configs/sn5_baseline.json

how to start on Windows machine?

are there any instructions to run on the windows machine without docker?

How to get checkpoint file

I run ./train.sh configs/sn5_baseline.json But I got a error at print (" Missing file:", os.path.join(self.save_path, checkpoint_name))
How to i can create checkpoint file or how to i download checkpoint file?

: wrongly placed

cresi/cresi/configs/sn5_baseline.json

Line 7 in b996f2d

 "speed_conversion_file:" "/path/to/data/cresi_data/cresi_train/SN5_roads_train_speed_conversion_binned.csv" 

should be:

"speed_conversion_file": "/path/to/data/cresi_data/cresi_train/SN5_roads_train_speed_conversion_binned.csv",

Questions about how to match SpaceNet dataset patches up to single large imagery

I'm interested in the routing performance of large road networks in the SpaceNet dataset,
so getting larger imagery or the way to match small patches up to single large imagery is necessary.
In the blog Large Road Networks section 4, that is the same thing that I want to do.
However, I'm unfamiliar with the details.
I notice that patches in SpaceNet road datasets (AOI_2_Vegas, AOI_3_Paris, AOI_4_Shanghai, AOI_5_Khartoum) are 1300x1300,
and they should be cropped from larger imagery like the picture below.
Could you help me with how to figure out the relation between cropped ones and the larger one?
BTW, is it possible to connect roads in different patches, and how?
Thank you very much!

Small patches:

Larger area:

indents are wrong in all file

cresi/cresi/utils/get_paths.py

Line 9 in b996f2d

 weight_save_path = os.path.join(config.results_dir, 'weights', config.save_weights_name) 

key error with Skeleton to graph in 05_wkt_to_G.py (UNABLE TO RUN)

When running %run -i 05_wkt_to_G.py configs/dar_tutorial_cpu.json in dar_tutorial_cpu/cresi_cpu_part2.ipynb, I get the following error (the key 'geometry' is not present within G_projected.edges"

KeyError                               Traceback (most recent call last)
~/Desktop/cresi/cresi/05_wkt_to_G.py in <module>
   1281 ###############################################################################
   1282 if __name__ == "__main__":
-> 1283     main()

~/Desktop/cresi/cresi/05_wkt_to_G.py in main()
   1272         pool.map(wkt_to_G, params)
   1273     else:
-> 1274         wkt_to_G(params[0])
   1275 
   1276     tf = time.time()

~/Desktop/cresi/cresi/05_wkt_to_G.py in wkt_to_G(params)
    918         for i,(u,v,attr_dict) in enumerate(G_projected.edges(data=True)):
    919             print (attr_dict)
--> 920             attr_dict['geometry_wkt'] = attr_dict['geometry'].wkt
    921 
    922         if verbose:

KeyError: 'geometry'

I've also printed the keys/values in G_projected.edges here for example until it threw me that error:

{'start': 10000, 'start_loc_pix': (7350.0, 2.0), 'end': 10001, 'end_loc_pix': (7322.0, 53.0), 'length_pix': 58.180752831155424, 'wkt_pix': 'LINESTRING (7350 2, 7322 53)', 'geometry_pix': 'LINESTRING (7350 2, 7322 53)', 'osmid': 0, 'geometry_latlon_wkt': 'LINESTRING (39.27236399997931 -6.7580028000011, 39.27228839997933 -6.758140500001112)', 'geometry_utm_wkt': 'LINESTRING (530098.2259186266 9252994.531477312, 530089.8629878408 9252979.31387781)', 'length_latlon': 0.0001570880326449568, 'length_utm': 17.36415693108535, 'length': 17.36415693108535, 'utm_zone': 37, 'utm_letter': 'M'}

Error in 04_skeletonize.py: Function "parse_struc" was compiled in object mode without forceobj=True.

Hi,

first of all, thank you for this great project.

I tried to use the 04_skeletonize.py with images that I have created. This is one example image:

The make_skeleton() method runs perfectly and produces an excellent ske image as result.

However, the graph can't be generated, the output is:

len G.nodes(): 1
len G.edges(): 0

What am I doing wrong?

Thanks in advance for help.

Best,

Jan

PS: My configuration (main method). Values are taken from the sn5_baseline.json:

    log_file = '../results/skeleton.log'
    console, logger1 = make_logger.make_logger(log_file, logger_name='log')

    im_dir = "../data/"
    outfile_csv = "../results/out.csv"
    out_ske_dir = "../results/ske/"
    out_gdir = "../results/gdir/"

    debug = False
    add_small = True
    fix_borders = True
    img_shape = ()  # (1300, 1300)
    skel_replicate = 5
    skel_clip = 2
    img_mult = 255
    hole_size = 300
    cv2_kernel_close = 7
    cv2_kernel_open = 7
    # max_out_size=(16000, 8000)  # works fine
    # max_out_size=(8003, 16009)   # works fine
    max_out_size = (2000000, 2000000)
    spacenet_naming_convention = False

    # from config
    min_subgraph_length_pix = 20
    skeleton_band = 0
    num_classes= 1
    im_prefix = ''
    thresh = 0.3

    t0 = time.time()
    df = build_wkt_dir(im_dir, outfile_csv, out_ske_dir, out_gdir, thresh, #threshes={'2': .3, '3': .3, '4': .3, '5': .2},
                debug=debug, add_small=add_small, fix_borders=fix_borders,
                img_shape=img_shape,
                skel_replicate=skel_replicate, skel_clip=skel_clip,
                img_mult=img_mult, hole_size=hole_size,
                min_subgraph_length_pix=min_subgraph_length_pix,
                cv2_kernel_close=cv2_kernel_close, cv2_kernel_open=cv2_kernel_open,
                max_out_size=max_out_size,
                skeleton_band=skeleton_band,
                num_classes=num_classes,
                im_prefix=im_prefix,
                spacenet_naming_convention=spacenet_naming_convention)

    print ("len df:", len(df))
    print ("outfile:", outfile_csv)
    t1 = time.time()
    logger1.info("Total time to run build_wkt_dir: {} seconds".format(t1-t0))

PPS: The complete output:



 1 / 1 : prediction8.png
  img_loc: ../data/prediction8.png
  im_root: prediction8
  out_ske_file: ../results/ske/prediction8.png
Executing make_skeleton...
log     : INFO     1 / 1 : prediction8.png
make_skeleton(), input img_shape: ()
make_skeleton(), img.shape: (256, 256)
make_skeleton(), img.size: 65536
make_skeleton(), img dtype: uint8
run preprocess()...
Run preprocess() with skimage
Time to run preprocess(): 0.002991199493408203 seconds
skeletonize...
Time to run skimage.skeletonize(): 0.0019943714141845703 seconds
fix_borders...
Time fix borders: 0.0 seconds
ske.shape: (256, 256)
Time to run make_skeleton: 21.24360680580139 seconds
Execute sknw...
C:\Users\Jan.noessner\Documents\01_road_detection\resources\centerline\cresi\cresi\utils\sknw.py:20: NumbaWarning: 
Compilation is falling back to object mode WITH looplifting enabled because Function "mark" failed type inference due to: Untyped global name 'neighbors': cannot determine Numba type of <class 'function'>

File "utils\sknw.py", line 22:
def mark(img):  # mark the array use (0, 1, 2)
    nbs = neighbors(img.shape)
    ^

  @jit  # my mark
numba.transforms: INFO     finding looplift candidates
C:\Users\Jan.noessner\Documents\01_road_detection\resources\centerline\cresi\cresi\utils\sknw.py:20: NumbaWarning: 
Compilation is falling back to object mode WITHOUT looplifting enabled because Function "mark" failed type inference due to: Untyped global name 'neighbors': cannot determine Numba type of <class 'function'>

File "utils\sknw.py", line 22:
def mark(img):  # mark the array use (0, 1, 2)
    nbs = neighbors(img.shape)
    ^

  @jit  # my mark
C:\Apps\Anaconda3\envs\ml\lib\site-packages\numba\object_mode_passes.py:178: NumbaWarning: Function "mark" was compiled in object mode without forceobj=True, but has lifted loops.

File "utils\sknw.py", line 22:
def mark(img):  # mark the array use (0, 1, 2)
    nbs = neighbors(img.shape)
    ^

  state.func_ir.loc))
C:\Apps\Anaconda3\envs\ml\lib\site-packages\numba\object_mode_passes.py:188: NumbaDeprecationWarning: 
Fall-back from the nopython compilation path to the object mode compilation path has been detected, this is deprecated behaviour.

For more information visit http://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit

File "utils\sknw.py", line 22:
def mark(img):  # mark the array use (0, 1, 2)
    nbs = neighbors(img.shape)
    ^

  state.func_ir.loc))
C:\Users\Jan.noessner\Documents\01_road_detection\resources\centerline\cresi\cresi\utils\sknw.py:92: NumbaWarning: 
Compilation is falling back to object mode WITH looplifting enabled because Function "parse_struc" failed type inference due to: Untyped global name 'neighbors': cannot determine Numba type of <class 'function'>

File "utils\sknw.py", line 94:
def parse_struc(img):
    nbs = neighbors(img.shape)
    ^

  @jit  # parse the image then get the nodes and edges
numba.transforms: INFO     finding looplift candidates
C:\Users\Jan.noessner\Documents\01_road_detection\resources\centerline\cresi\cresi\utils\sknw.py:92: NumbaWarning: 
Compilation is falling back to object mode WITHOUT looplifting enabled because Function "parse_struc" failed type inference due to: Untyped global name 'neighbors': cannot determine Numba type of <class 'function'>

File "utils\sknw.py", line 94:
def parse_struc(img):
    nbs = neighbors(img.shape)
    ^

  @jit  # parse the image then get the nodes and edges
C:\Apps\Anaconda3\envs\ml\lib\site-packages\numba\object_mode_passes.py:178: NumbaWarning: Function "parse_struc" was compiled in object mode without forceobj=True, but has lifted loops.

File "utils\sknw.py", line 94:
def parse_struc(img):
    nbs = neighbors(img.shape)
    ^

  state.func_ir.loc))
C:\Apps\Anaconda3\envs\ml\lib\site-packages\numba\object_mode_passes.py:188: NumbaDeprecationWarning: 
Fall-back from the nopython compilation path to the object mode compilation path has been detected, this is deprecated behaviour.

For more information visit http://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit

File "utils\sknw.py", line 94:
def parse_struc(img):
    nbs = neighbors(img.shape)
    ^

  state.func_ir.loc))
C:\Users\Jan.noessner\Documents\01_road_detection\resources\centerline\cresi\cresi\utils\sknw.py:92: NumbaWarning: 
Compilation is falling back to object mode WITHOUT looplifting enabled because Function "parse_struc" failed type inference due to: non-precise type pyobject
[1] During: typing of argument at C:\Users\Jan.noessner\Documents\01_road_detection\resources\centerline\cresi\cresi\utils\sknw.py (101)

File "utils\sknw.py", line 101:
def parse_struc(img):
    <source elided>
    nodes = []
    for p in pts:
    ^

  @jit  # parse the image then get the nodes and edges
C:\Apps\Anaconda3\envs\ml\lib\site-packages\numba\object_mode_passes.py:178: NumbaWarning: Function "parse_struc" was compiled in object mode without forceobj=True.

File "utils\sknw.py", line 101:
def parse_struc(img):
    <source elided>
    nodes = []
    for p in pts:
    ^

  state.func_ir.loc))
C:\Apps\Anaconda3\envs\ml\lib\site-packages\numba\object_mode_passes.py:188: NumbaDeprecationWarning: 
Fall-back from the nopython compilation path to the object mode compilation path has been detected, this is deprecated behaviour.

For more information visit http://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit

File "utils\sknw.py", line 101:
def parse_struc(img):
    <source elided>
    nodes = []
    for p in pts:
    ^

  state.func_ir.loc))
C:\Users\Jan.noessner\Documents\01_road_detection\resources\centerline\cresi\cresi\utils\sknw.py:92: NumbaWarning: 
Compilation is falling back to object mode WITHOUT looplifting enabled because Function "parse_struc" failed type inference due to: non-precise type pyobject
[1] During: typing of argument at C:\Users\Jan.noessner\Documents\01_road_detection\resources\centerline\cresi\cresi\utils\sknw.py (108)

File "utils\sknw.py", line 108:
def parse_struc(img):
    <source elided>
    edges = []
    for p in pts:
    ^

  @jit  # parse the image then get the nodes and edges
C:\Apps\Anaconda3\envs\ml\lib\site-packages\numba\object_mode_passes.py:178: NumbaWarning: Function "parse_struc" was compiled in object mode without forceobj=True.

File "utils\sknw.py", line 108:
def parse_struc(img):
    <source elided>
    edges = []
    for p in pts:
    ^

  state.func_ir.loc))
C:\Apps\Anaconda3\envs\ml\lib\site-packages\numba\object_mode_passes.py:188: NumbaDeprecationWarning: 
Fall-back from the nopython compilation path to the object mode compilation path has been detected, this is deprecated behaviour.

For more information visit http://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit

File "utils\sknw.py", line 108:
def parse_struc(img):
    <source elided>
    edges = []
    for p in pts:
    ^

  state.func_ir.loc))
log     : INFO     Time to build graph: 1018.8935737609863 seconds
log     : INFO     Total time to run build_wkt_dir: 1018.8965332508087 seconds
len G.nodes(): 1
len G.edges(): 0
len df: 1
outfile: ../results/out.csv

Process finished with exit code 0

avanetten / cresi Goto Github PK

cresi's People

Contributors

Stargazers

Watchers

Forkers

cresi's Issues

Recommend Projects

Recommend Topics

Recommend Org