omrikiei / aws_logging_handlers Goto Github PK

AWS integration for python logging handlers(S3, Kinesis)

Home Page: https://omrikiei.github.io/aws_logging_handlers/

License: MIT License

Python 7.04% CSS 13.06% JavaScript 46.57% HTML 33.33%

aws_logging_handlers's Introduction

AWS Logging Handlers

Please note: this package is useful for standalone applications(like Jupyter notebooks), if you are desinging a scalable and robust solution - I highly recommend on leaving the handling of log records to an external tool and have your application to focus on it's business logic

A python multithreaded logging handler package that streams records to AWS services objects with support for the following AWS services:

S3
Kinesis

Has supports for gzip compression(in S3)

Getting Started

Prerequisites

Asynchronous multipart uploading relies on the ability to use multiple threads

Packages:

boto3

Installing

Installation using pip

pip install aws-logging-handlers

Examples

Stream log records to S3 and Kinesis

import logging
from aws_logging_handlers.S3 import S3Handler
from aws_logging_handlers.Kinesis import KinesisHandler

bucket="test_bucket" # The bucket should already exist

# The log will be rotated to a new object either when an object reaches 5 MB or when 120 seconds pass from the last rotation/initial logging
s3_handler = S3Handler("test_log", bucket, workers=3)
kinesis_handler = KinesisHandler('log_test', 'us-east-1', workers=1)
formatter = logging.Formatter('[%(asctime)s] %(filename)s:%(lineno)d} %(levelname)s - %(message)s')
s3_handler.setFormatter(formatter)
kinesis_handler.setFormatter(formatter)
logger = logging.getLogger('root')
logger.setLevel(logging.INFO)
logger.addHandler(s3_handler)
logger.addHandler(kinesis_handler)

for i in range(0, 100000):
    logger.info("test info message")
    logger.warning("test warning message")
    logger.error("test error message")

logging.shutdown()

To be developed

Support for asyncio
Logging and upload metrics

License

This project is licensed under the MIT License - see the LICENSE.md file for details

aws_logging_handlers's People

Contributors

Stargazers

Watchers

aws_logging_handlers's Issues

Add kinesis streaming handler

How to do AWS Authentication

How can I provide the access_key_id and secret_access_key for the logger?

Add logging to cloudwatch

Possible issue with strftime(%s) etc.

Hello and thanks for the package. I get an error when I run the example code in README.
I am using Python 3.6.7.

import logging
from aws_logging_handlers.S3 import S3Handler

bucket="test_bucket" # The bucket should already exist
s3_handler = S3Handler("test_log", bucket)

I get:
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\username\python_venvs\env_name\lib\site-packages\aws_logging_handlers\S3_init_.py", line 331, in init
compress=compress, **boto_session_kwargs)
File "C:\Users\username\python_venvs\env_name\lib\site-packages\aws_logging_handlers\S3_init_.py", line 98, in init
self.start_time = int(datetime.utcnow().strftime('%s'))
ValueError: Invalid format string

I tried to replace strftime('%s') timestamp() , see:
https://stackoverflow.com/questions/11743019/convert-python-datetime-to-epoch-with-strftime

But then I got another error:
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\username\python_venvs\env_name\lib\site-packages\aws_logging_handlers\S3_init_.py", line 336, in init
signal.signal(signal.SIGQUIT, self._teardown)
AttributeError: module 'signal' has no attribute 'SIGQUIT'

TypeError in S3.StreamObject init

version: 2.0.5

I hit the following error:

self = <aws_logging_handlers.S3.StreamObject object at 0x7f3dc744fdf0>, s3_resource = s3.ServiceResource(), bucket_name = 'batch-detect-bucket', filename = 'sid=source1/module/function-o365-detection_1606755959', buffer_queue = <queue.Queue object at 0x7f3de595e9d0>, encryption = None
    def __init__(self, s3_resource, bucket_name, filename, buffer_queue, encryption):
        self.object = s3_resource.Object(bucket_name, filename)
>       self.uploader = self.object.initiate_multipart_upload(**encryption)
E       TypeError: initiate_multipart_upload() argument after ** must be a mapping, not NoneType
../local/lib/python3.8/site-packages/aws_logging_handlers/S3/__init__.py:34: TypeError

I think I've narrowed down the change that causes this issue.

1ad2c42#diff-915075834d3035bf217f6a9dedfa0afd57882e221e9ca41b6c44b4e88281ddd1

The issue is encryption_options is default initialized as None. Which can't be used with **.

I don't have this issue with version 2.0.3.

KinesisHandler: PartitionKey can be dynamic, not only static

partition= can be specified when creating a KinesisHandler, however it accepts only a string.

The implementation can differ and we may want to generate a random unique number to be sure the logs are spread across multiple shards.

To let the implementation's developer decide, I suggest to change the argument's type to be also a function.

e.g. pseudo code

# my_app.py changes:

k = KinesisHandler(.., partition=uuid4.uuid)


# aws_logging_handlers/KinesisStreamHandler.py changes:

if callable(self.partition):
   ..put_record(.., PartitionKey=self.partition())
else:
   ..put_record(.., PartitionKey=self.partition)

Make signals optional to handle logging in dask workers

I got this error when logging inside a dask worker: signal only works in main thread

Could you make signals optional to allow loggin in dask workers?

Default behavior should not ask for creds+region

Hi,
When building KinesisHandler/S3Handler, it asks for a key_id, a secret and a region.
However, boto3 takes that automatically from .aws/credentials and $AWS_DEFAULT_REGION

I suggest to make these parameters optionals for the simplicity of the users (no need to read the file and read the environment ourselves).

Need to implement only for S3

import logging
from aws_logging_handlers.S3 import S3Handler
import boto3
s3 = boto3.client("s3")
bucket="sing-dev-chumbucket1" # The bucket should already exist

s3_handler = S3Handler("error/output/", bucket, workers=3)

formatter = logging.Formatter('[%(asctime)s] %(filename)s:%(lineno)d} %(levelname)s - %(message)s')
s3_handler.setFormatter(formatter)
logger = logging.getLogger('root')
logger.setLevel(logging.INFO)
logger.addHandler(s3_handler)
for i in range(0, 100):
    logger.info("test info message")
    logger.warning("test warning message")
    logger.error("test error message")

logging.shutdown()

getting an error 
File "C:/Users/Administrator/PycharmProjects/baselineloader/venv/Scripts/testarg.py", line 7, in <module>
    s3_handler = S3Handler("error/output/", bucket, workers=3)
  File "C:\Users\Administrator\PycharmProjects\baselineloader\venv\lib\site-packages\aws_logging_handlers\S3\__init__.py", line 329, in __init__
    self.stream = S3Stream(self.bucket, key, chunk_size=chunk_size, max_file_log_time=time_rotation,
TypeError: ABCMeta object got multiple values for keyword argument 'workers'

kindly help me .

Remove redundant task and task_worker definitions from KinesisHandler

aws_logging_handlers/aws_logging_handlers/KinesisStreamHandler.py

Line 20 in eb045e1

class Task:

Segmentation fault (core dumped) exit code 139

using FROM python:3.8-slim-buster docker and latest aws-logging-handlers>=2.0.3 segfaults with;

 File "/usr/local/lib/python3.8/site-packages/boto3/utils.py", line 52 in import_module
 File "/usr/local/lib/python3.8/site-packages/boto3/utils.py", line 61 in _handler
 File "/usr/local/lib/python3.8/site-packages/botocore/hooks.py", line 211 in _emit
 File "/usr/local/lib/python3.8/site-packages/botocore/hooks.py", line 228 in emit
 File "/usr/local/lib/python3.8/site-packages/botocore/hooks.py", line 356 in emit
 File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 106 in _create_client_class
 File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 79 in create_client
 File "/usr/local/lib/python3.8/site-packages/botocore/session.py", line 831 in create_client
 File "/usr/local/lib/python3.8/site-packages/boto3/session.py", line 258 in client
 File "/usr/local/lib/python3.8/site-packages/boto3/session.py", line 384 in resource
 File "/usr/local/lib/python3.8/site-packages/aws_logging_handlers/S3/__init__.py", line 97 in __init__
 File "/usr/local/lib/python3.8/site-packages/aws_logging_handlers/S3/__init__.py", line 329 in __init__

To see the verbose output run with flags python -d -X dev app.py

script runs fine on the laptop running;

python3 --version
Python 3.8.2

uname -a
Linux 5.4.0-33-generic #37-Ubuntu SMP Thu May 21 12:53:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Logs go to S3 only after handler shutdown

I cannot see my logs before handler shutdown

S3Handler example Error self.uploader = self.object.initiate_multipart_upload(encryption) TypeError: initiate_multipart_upload() argument after must be a mapping, not NoneType

`import logging
from aws_logging_handlers.S3 import S3Handler
from aws_logging_handlers.Kinesis import KinesisHandler

bucket = "services-s3-log"

s3_handler = S3Handler("test_log/log.log", bucket, workers=1)
formatter = logging.Formatter('[%(asctime)s] %(filename)s:%(lineno)d} %(levelname)s - %(message)s')
s3_handler.setFormatter(formatter)
logger = logging.getLogger('root')
logger.setLevel(logging.INFO)
logger.addHandler(s3_handler)

for i in range(0, 100000):
logger.info("test info message")
logger.warning("test warning message")
logger.error("test error message")

logging.shutdown()

Rotation parameters are not configureable

Allow initialization of the relevant parameters:

DEFAULT_CHUNK_SIZE = 5 * 1024 ** 2 # 5 MB
DEFAULT_ROTATION_TIME_SECS = 12 * 60 * 60 # 12 hours
MAX_FILE_SIZE_BYTES = 100 * 1024 ** 2 # 100 MB

Support for dynamic filename (Include directory)

Hi,

I'm dealing with a situation that this library does not cover. I want to structure my logging using Hive partition on S3 to be consumed by other applications. So I need a way to build the S3 file path dynamically like that:

def get_dynamic_path(date):
    # Will return /year=2019/month=06/day=19/20190619.json
    return date.strftime('/year=%Y/month=%m/day=%d/%s')

s3_handler = S3Handler(dynamic_filename=get_dynamic_path)

Would be nice if I could pass my own function to S3Handler to build dynamically the path.

README is wrong!

Please change
from aws_logging_handlers import S3Handler --> from aws_logging_handlers.S3 import S3Handler

from aws_logging_handlers import KinesisHandler --> from aws_logging_handlers.Kinesis import KinesisHandler

Also, even though it uploads logs to S3 bucket, the whole program (in readme) hangs!

Could Buffering be optional?

Writing to a buffer is cool and nice, however sometime we want to write into Kinesis the quickest way as possible.

For instance, if we write crit and those are plugged into a Kinesis stream to be shown in a operator's dashboard, we may don't want to wait.

Any ideas about that ?

Solutions that I have in mind:

disable buffering per handler
disable buffering per severity
have a maximum delay where a buffer should be flushed (e.g. every 200ms the buffer is flushed if no new messages are added)

S3Handler blocks Python from exiting

Using this handler prevents python from exiting - tested on 3.7.3 and 3.6.8

This could be caused by the teardown signal handler for the S3 logging handler, which is wrong. The documentation says that the signal handlers take 2 arguments

The handler is called with two arguments: the signal number and the current stack frame (None or a frame object; for a description of frame objects, see the description in the type hierarchy or see the attribute descriptions in the inspect module).

the _teardown function takes a single arg.

I tried fixing that locally but the exit still hangs.

Closing the log handler explicitly before exit solves the issue but this really shouldn't be necessary.

Hitting ctrl+c once the program is frozen gives the following traceback

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/threading.py", line 1281, in _shutdown
    t.join()
  File "/usr/local/lib/python3.7/threading.py", line 1032, in join
    self._wait_for_tstate_lock()
  File "/usr/local/lib/python3.7/threading.py", line 1048, in _wait_for_tstate_lock
    elif lock.acquire(block, timeout):
KeyboardInterrupt```

Add support for asyncio

Add support to asyncio so that initialization can determine wether to use the event loop or threads

I have this error when I tried to create a new instance of S3Handler.

s3_handler = S3Handler("logs", bucket, workers=3)
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\admin\Documents\crash_bandicoot\api.localgovai\venv\Lib\site-packages\aws_logging_handlers\S3_init_.py", line 329, in init
self.stream = S3Stream(self.bucket, key, chunk_size=chunk_size, max_file_log_time=time_rotation,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\admin\Documents\crash_bandicoot\api.localgovai\venv\Lib\site-packages\aws_logging_handlers\S3_init_.py", line 97, in init
self.start_time = int(datetime.utcnow().strftime('%s'))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: Invalid format string