Splitting spider logs about crawly HOT 2 CLOSED

Right now my crawler logs are getting swallowed up into the void and mixed with application logs.

from my understanding, Logger can be configured to with additional backends.
There's a file backend package here: https://github.com/onkel-dirtus/logger_file_backend
Log rotation isn't included, though.

It would be interesting to log realtime errors to the web ui like how the new phoenix liveview admin request logging does.

from crawly.

@oltarasenko the simplest method to do this would be to use Logger's metadata to specify spider name and job ids (#102 ) , and allow the user to filter the logs with the logger file backend.

It keeps Crawly's scope limited, while we can bless the logger backend filtering method as recommend way for spider logging.

from crawly.