kfrancischen / pslx Goto Github PK

Python Standard Library eXtension.

The site for detailed documentation: https://kfrancischen.github.io/pslx/.

A standard library for job scheduling, micro services including message queue, RPC, instant messaging and monitoring, and tooling, developed by Kaifeng Chen. The library is written compatible to Python 3.7+, and is integrated upon the distributed file system Galaxy. To use the program, please install the latest protobuf compiler and gRPC tools. This document covers the APIs of PSLX provided, and is organized as follows:

• Running ad-hoc and scheduled jobs with operators and operator containers.

• Data storage using plain text file, proto table, sharded proto table and timestamp-based partitioner.

• General framework of RPC used in PSLX.

• Instant messaging services built for Slack, Rocketchat and Microsoft Teams using the PSLX RPC framework.
• Email service for sending emails through code using the PSLX RPC framework.
• Message Queue for building simple message queue using proto buffers. Part of the implementation also is shared with the PSLX rpc framework.
• Publisher/Subscriber for building simple publisher/subscriber applications.
• PSLX Frontend Flask frontend built to monitor the health of RPC servers, container status, and browser the content of protobufs.

• Tools provided by PSLX for galaxy-based file io, LRC caching, SQL server read/write, Mongodb tool, timestamp key-ed partitioner fetcher and watcher, function registration.

• Utilities provided by PSLX for file operation, protobuf related utilities, async unittesting, timezone related utilities, yaml file io, text coloring, and credential composing, environment variables access and decorators.

• Internal protobuf schemas.

Please also take a look at the example folder for different example implementations.

I do accept BTC/ETH donation. My BTC address is 3LH5VQ51fToFTpb46N4hYn225e6iMDPU7f, and ETH address is 0xd16B5edEc78F9C798d3aD4D94b008ED091722bEC.