When we call API end point /training/publish/trainingId the published training ID is saved into the models table but the status is still =-1 which is DRAFT

We need to change the status to 0 which is published

Improve output from interference

Since installing exact library with gpu support could be tricky hence Plan is to build docker file for development

plan is to integrate osm user with native django user with osm_id as unique value

Currently images are being downloaded also from neighbors area of AOI , Since they are resulting the extra images for the models , we need to find the bug and download only needed images inside AOI

Deny user is another training is already running or submitted for same model

When more than 1 AOI are associated with a dataset, the /dataset/image/build end point fails with the following error

@kshitijrajsharma

• Info included on #47

In the training table we need to show the details for the finished training including graph

Looking at icon only it could be hard to know about what those buttons are for , Creating a supportive label when mouse is hovered over a button would be helpful

Refer to training 101 for dataset 1 model 1 on dev server

While fetching rawdata plan is to handle error when API is down

• This issue is also connected to #50 and #64

Plan is to run backend and frontend tests with github actions

Plan is to manage and merge two code bases of exisiting fAIr and the ouput delivered from omdeena !

Since we have finalized and almost name is widely popular we should rename our repo too 😉 @omranlm

During predictions , threads are not releasing memory so when there are few prediction system fails !
Critical Bug

Right now when creating new model , we need to enter dataset id and it would be more easy if user can select dataset instead of remembering the id of their dataset to create model

Image downloading one by one is taking a lot of time due to which it is resulting more lag on prediction time hence plan is to do downloads using threads

Plan is to store the link of imagery also for the training dataset

In api which returns label for aoi_id , plan is to also send info about total no of labels within aoi

Plan is to create a API functionality and also a frontend functionality to create traceback and visualize it if error occurs

Currently threads are keeping image content on memory , instead of keeping it in memory and saving it to disk after processing is finished we can tell threads to save them to disk directly

Goal is to have a API that will give stats about the dir structure and also allow to download them

Currently it is showing current dataset size to all trainining

This is for exisiting records , plan is to replace if there is record found for same dataset , but it should allow if same label is being used in different dataset

ack_utils.py", line 64, in error_handler
return fn(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/keras/engine/training.py", line 490, in call
return super().call(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/keras/utils/traceback_utils.py", line 64, in error_handler
return fn(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/keras/engine/base_layer.py", line 1014, in call
outputs = call_fn(inputs, *args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/keras/utils/traceback_utils.py", line 92, in error_handler
return fn(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/keras/engine/functional.py", line 458, in call
return self._run_internal_graph(
File "/usr/local/lib/python3.8/dist-packages/keras/engine/functional.py", line 596, in _run_internal_graph
outputs = node.layer(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/keras/utils/traceback_utils.py", line 64, in error_handler
return fn(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/keras/engine/base_layer.py", line 1014, in call
outputs = call_fn(inputs, *args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/keras/utils/traceback_utils.py", line 92, in error_handler
return fn(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/keras/layers/convolutional/depthwise_conv2d.py", line 155, in call
outputs = backend.depthwise_conv2d(
File "/usr/local/lib/python3.8/dist-packages/keras/backend.py", line 5942, in depthwise_conv2d
x = tf.compat.v1.nn.depthwise_conv2d(
Node: 'model/block3b_dwconv/depthwise'
OOM when allocating tensor with shape[24,240,32,32] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node model/block3b_dwconv/depthwise}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.
[Op:__inference_train_function_2852847]

Plan is to create archive option for dataset from the API

Since we are dependent on Ramp , Plan is to copy and maintain the required dependencies under hotosm so that any changes on mother repo won't affect our setup . Since setting up such project will require inch perfect libraries installation it would be necessary to maintain dependencies

Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/celery/app/trace.py", line 451, in trace_task
R = retval = fun(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/celery/app/trace.py", line 734, in protected_call
return self.run(*args, **kwargs)
File "/app/core/tasks.py", line 95, in train_model
raise ex
File "/app/core/tasks.py", line 58, in train_model
final_accuracy, final_model_path = train(
File "/usr/local/lib/python3.8/dist-packages/hot_fair_utilities/training/train.py", line 56, in train
run_main_train_code(cfg)
File "/usr/local/lib/python3.8/dist-packages/hot_fair_utilities/training/run_training.py", line 279, in run_main_train_code
history = the_model.fit(
File "/usr/local/lib/python3.8/dist-packages/keras/utils/traceback_utils.py", line 67, in error_handler
raise e.with_traceback(filtered_tb) from None
File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 54, in quick_execute
tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.ResourceExhaustedError: Graph execution error:

OOM when allocating tensor with shape[16,64,64,64] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node model/decoder_stage2b_relu/Relu-0-1-TransposeNCHWToNHWC-LayoutOptimizer}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.
[Op:__inference_train_function_2701887]

Plan is to create gpx if aoi is supplied so that people can go to id and edit aoi supplied

Plan is to Download the Images from API taking input from API zoom level parameters directly from the POST requests

As in the following, we need to pass the dataset ID as the /label/ with bbox filter is getting labels from other datasets which are created in the same location

Also no need for uniqe key on the OSM id in the labels table

Currently we were using radiantearth's basemodel and due to git-lfs it is going out of quota . Hence Plan is to give user to download basemodel from google drive link and integrate it with github workflow too