muriloxyz / dataset-split Goto Github PK

Functionality will allow the user to undo the split, rebuilding the original dataset with all the original items.

The code flow should be changed. Because of '--copy' implementation it is NEEDED to rethink the code (ugly and not optimized)

The "copy" argument, taken by the app, isn't working at all. Even when it's present, the program doesn't copy any of the files, it just moves them and leave the original dataset empty.

Decide AND code what do when program chases while executing.

Cases to think about:

• If the app crashes while executing: should it leave everything as it is OR leave it how it was when app started?
• Cases where the app should throw an error.

The '--noshuffle' option doesn't let the app shuffle the items. The split will be made to (train, test, validation) in the same order 'ls' listed the items.

It will enhance the usability when the module is imported inside another python script. Allows the user to choose a directory anywhere in the system.

The reshuffle option will allow the user to reshuffle the already splitted dataset.
I realized this would be quite useful during a model evaluation process

• Research and find the best file structure for python structures.
• Modify the app to match it.

Well, I need to understand some crucial stuff before building this app.

Tasks :

• Collect initial path (ipath)
• Find the best way to collect desired ratio for splitting the data
• List all classes (subdirs) in ipath
• Create (train - validation - test) folders
• Create all classes inside train-val-test directories
• Iterate over each original class and:
  • Calculate how many items there's inside
  • Calculate the splits
  • RANDOMLY move the desired amount of items to test-val-test correct subdirs
• Delete original class directories (empty)

• Think of improvements
  • List them here

• Setup CI
• Publish on PyPi

• Finished

What I need/must understand in order to build this app:

• How to manipulate paths the right way
• How to list in a given path
• Create directories
• Move files
• Delete files
• Calculate splits for all amounts of data
• Decent randomness