belltailjp / chainer_slack_report Goto Github PK

In case an exception is raised during the training, SlackReport will eventually kill the whole training process, which is a nightmare for users.
First it to do as much error checking as possible in the initializer to grab errors before the training starts, and once the training starts, exceptions should be reinterpreted as warnings.

How?
ideas:

• New message (:-1:)
• Just truncate
• Change the content to a text attachment
• Thin out old lines (except header)
• Erase the oldest lines (except header)

In addition to Slack Channel, it is useful to keep timeline clean if chainer_slack_report can post it to a slack thread.