Ability to inject IScheduler instance in SchedulerMain about incubator-heron HOT 6 CLOSED

@ashvina - We did a recent checkin where the scheduler architecture is streamlined now. Everything about a cluster is read from a bunch of config files. Try the following

checkout the new master and run

bazel run --config=darwin -- release/packages:heron-client-install.sh --user

This will install the client of heron at your home directory ~/.heron and the executable heron-cli3 in your ~/bin will have a link ~/.heron/bin/heron-cli3.

This installation has already a bunch of configs built in for the local scheduler and they can be found in
~/.heron/conf/local/. Those config files define the classes needed for that cluster. You can parse these config files to get an instance of IScheduler.

This config is available from the client when you launch and the entire config is carried with the topology package and made available in the containers as well. In the container, they will be stored in

./heron-conf/

and several parse routines are available to read those files
https://github.com/twitter/heron/tree/master/heron/spi/src/java/com/twitter/heron/spi/common

SchedulerMain also reads from those files and you can make an instance of the IScheduler class.

from incubator-heron.

With local installation, you can run some topologies right away using the examples using the following command

heron-cli3 submit local ~/.heron/examples/heron-examples.jar com.twitter.heron.examples.ExclamationTopology ExclamationTopology

You won't be able to see the output unless you can get to the logs. The logs are in
~/.herondata/topologies/local//ExclamationTopology/log-files

You can kill the topology using the following command

heron-cli3 kill local ExclamationTopology

Let me know if you run into issues.

from incubator-heron.

We are working on updating the documentation for the new config.

from incubator-heron.

Thanks Karthik. I will look forward to the updated docs.

This seems to be a major change. A number of classes have been removed (for e.g. IConfigLoader). Can I assume that any new scheduler should be added to newscheduler and the old scheduler code is outdated and should be ignored?

from incubator-heron.

Yes. newscheduler is the way forward. It provides a lot of flexibility and clean configuration.

from incubator-heron.

This issue is not applicable anymore.

from incubator-heron.