alibaba / gpu-scheduler-for-deep-learning Goto Github PK

GPU-scheduler-for-deep-learning

License: MIT License

Starlark 2.54% Batchfile 0.03% Python 34.26% C++ 53.64% C 0.59% Shell 0.48% MLIR 0.46% SWIG 0.12% Jupyter Notebook 1.55% LLVM 0.01% CMake 0.15% Java 0.62% Makefile 0.08% Dockerfile 0.06% HTML 3.42% Objective-C 0.08% Objective-C++ 0.16% Ruby 0.01% Go 1.75% Perl 0.01%

gpu-scheduler-for-deep-learning's Introduction

README

Introduction

This repository contains a re-implementation of our deep learning training infrastructure, described in the paper "AntMan: Dynamic Scaling on GPU Clusters for Deep Learning" (OSDI'20).

Note

The original implementation of our paper is based on FUXI, which is tightly coupled with the internal infrastructure of Alibaba. The goal of this project is to provide a cloud-native solution to demonstrate the feature of the paper and benenfit the community.
This is a WIP project. Please grant us several days to fix the missing components with code cleaning and show the end-to-end demo with some benchmarks. We are working hard to achieve that. More detailed documents are on the way.
The implementation of our kubernetes scheduler is only tested in the ACK cluster of alibaba cloud, based on Kubernetes V1.18. The deployement script we provide may not be able to apply in other kubernetes infrastructures directly.

Modules

The development of this repository is based on some open-source repositories.

k8s-related

KubeDL: an all-in-one operator, responsible to reconcile tfjobs
Scheduler Plugins: a k8s cluster scheduler, responsible to schedule DL GPU pods for both resource-guarantee/opportunistic jobs
k8s-device-plugin: report GPU resources to k8s

TensorFlow

The dynamic scaling mechianism is initially implemented in PAI-TF, a highly-optimized TensorFlow verison used in Alibaba. We port the core implementation to the open-source TensorFlow v1.15.

TensorFlow

gpu-scheduler-for-deep-learning's People

Contributors

Stargazers

Watchers

gpu-scheduler-for-deep-learning's Issues

bazel build error

When I finished configuration in TensorFlow-with-dynamic-scaling, I typed "bazel build --config=cuda [--config=option] //tensorflow/tools/pip_package:build_pip_package".
However, I received "INFO: Options provided by the client:
Inherited 'common' options: --isatty=1 --terminal_columns=120
ERROR: Config value cuda is not defined in any .rc file"at once.
And It will be of great help if someone can share a working example using (TensorFlow-with-dynamic-scaling).

This cycle occurred because of a configuration option when I use bazel build tensorflow in this project.

hi, @WencongXiao @ShiruRen I have an error when i execute this command line:
bazel build tensorflow/tools/pip_package:build_pip_package

The output is:
`ERROR: /usr/src/TensorFlow-with-dynamic-scaling/tensorflow/core/BUILD:3290:1: in cc_library rule //tensorflow/core:core_cpu_lib: cycle in dependency graph:
//tensorflow/tools/pip_package:build_pip_package
//tensorflow/python/distribute:combinations
//tensorflow/python/eager:context
//tensorflow/python/eager:executor
//tensorflow/python:pywrap_tensorflow
//tensorflow/python:pywrap_tensorflow_internal
//tensorflow/python:pywrap_tensorflow_internal.py
//tensorflow/python:pywrap_tensorflow_internal_py_wrap
//tensorflow/python:py_func_lib
//tensorflow/python:safe_ptr
//tensorflow/c/eager:c_api
//tensorflow/core/profiler/lib:profiler_lib
//tensorflow/core/profiler/internal/cpu:host_tracer
.-> //tensorflow/core:core_cpu_lib
| //tensorflow/core:core_cpu_impl
| //tensorflow/contrib/resource_management:gpu_resource_management
| //tensorflow/contrib/resource_management:gpu_usage_adjustment
| //tensorflow/core:gpu_runtime
| //tensorflow/core:gpu_runtime_impl
-- //tensorflow/core:core_cpu_lib

This cycle occurred because of a configuration option
ERROR: Analysis of target '//tensorflow/tools/pip_package:build_pip_package' failed; build aborted
INFO: Elapsed time: 0.612s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded, 0 targets configured)
`

It seems that there are cycle calls when building this code, is there any idea for it?
I hope someone can help me to solve this problem.

Can not pull the image for local-coordinator

I tried to apply the yaml file of local-coordinator in the k8s, but it failed.

Any Progress?

This is a WIP project. Please grant us several days to fix the missing components with code cleaning and show the end-to-end demo with some benchmarks. We are working hard to achieve that. More detailed documents are on the way.

I am currently reading the paper accepted by OSDI' 20, and I wonder if there is any new progress of the code.

Working example Of TensorFlow

It will be of great help if someone can share a working example using (TensorFlow-with-dynamic-scaling). in which more than one DNN training jobs are sharing the same set of resources. Specifically, I have following questions:

How do I make sure AntMan is being used as a resource scheduler for a given DNN training job that I want to execute ? Are there any extra arguments that need to passed to tf.Session while training a given DNN model?
Can I run multiple training jobs separately using the same (TensorFlow-with-dynamic-scaling) source (with different python and bash scripts) and run them on the same set of resources and all of these will automatically be visible to a global AntMan scheduler and will share resources as expected ?
Or I have to execute/register all Jobs that I plan to execute on the same set of resources at once?
How do I specify if a given DNN job is a background/low priority job?

any plan for documentation?

Thanks for the great work!

I'm really interested in it and want to try it out myself. However, there is no instructions/guidelines/documentation on how to build and run it. Is there any plan for documentation in the near future?