segmind / segmind-old Goto Github PK

View Code? Open in Web Editor NEW

7.0 7.0 3.0 728 KB

MLOps for end-to-end deep learning lifecycle.

Home Page: https://docs.segmind.com

Shell 0.31% Python 99.69%

deep-learning mlops

segmind-old's Introduction

Installation

pip3 install -r requirements.txt

Code style

Python

We adopt PEP8 as the preferred code style.

We use the following tools for linting and formatting:

flake8: linter
yapf: formatter
isort: sort imports

Style configurations of yapf and isort can be found in setup.cfg.

We use pre-commit hook that checks and formats for flake8, yapf, isort, trailing whitespaces, fixes end-of-files, sorts requirments.txt automatically on every commit. The config for a pre-commit hook is stored in .pre-commit-config.

After you clone the repository, you will need to install initialize pre-commit hook.

pip install -U pre-commit

From the repository folder

pre-commit install

After this on every commit check code linters and formatter will be enforced.

Run locally

cd .

pip install virtualenv

virtualenv venv

source ./venv/bin/activate

pip install -r requirements.txt pip install -e .

segmind-old's People

Contributors

Stargazers

Watchers

Forkers

shreeram0206 saurbhc deepanshunabi

segmind-old's Issues

`segmind upload` - uploads files/folders again when re-ran

New Feature:

Need to check with datastore, if file/folders exist, then skip them.

`!segmind config` has issues on Google Colab Notebook

When using google-colab cell, logging with segmind, like this

!segmind config

internally sets env variable with

import os
os.environ[_ACCESS_TOKEN] = access_token

But, All other segmind API's, doesn't have access to that env variable, and ends up breaking the code.

test issue -- please ignore

`segmind upload` for folder breaks

segmind upload --path <folder-path> --destination_path <folder-path> --datastore_name <datastore-path>

Folder upload functionality is breaking and has unexpected behaviour.

[feature-request] Add API based Authentication

Remove email/password and use Access Token to Login.

segmind config --access_token=<Bearer ...>

`segmind config` Needs persistent tokens

Problem:

tokens set in os.environ vanish after 1st use.

Solution:

Persist the access-token in a secret file.

[feature-request] Create Run with Run-Name & Algo Name

When Starting run explicitly, need an option to add run_name & algo_name

from segmind.tracking import start_run


start_run(
    run_name="My Custom Run Name",  # Optional
    run_algo_name="Linear Regression with Gradient Descent",  # Optional
)

`from segmind.keras import KerasCallback` breaks with old psutil versions

Steps to reproduce this error

pip install --upgrade --pre segmind
from segmind.keras import KerasCallback
keras_cb = KerasCallback()

/usr/local/lib/python3.7/dist-packages/segmind/sys_data/get_systemstats.py in <module>()
  1 import psutil
  ----> 2 ![from psutil._common import bytes2human]()
  3 
  4 
  5 def fan_stats():
ImportError: cannot import name 'bytes2human' from 'psutil._common' 
(/usr/local/lib/python3.7/dist-packages/psutil/_common.py)

`segmind upload` - progress bar breaks when no file found in the folder

To reproduce the issue

mkdir test_folder
segmind upload --path=test_folder --destination_path=test_folder --datastore_name=saurabh01-ds

ZeroDivisionError: float division by zero

`segmind upload` --destination_path breaks with `.` or `/`

Add these checks while uploading

Can't use . as --destination_path
Can't use / as --destination_path

[feature-request] `segmind ls`

List directory/folders in a datastore with these additions:

--human-readable
--summarize
--recursive