Giter Club home page Giter Club logo

pyms's Introduction

Python Microservices Library

PyPI version Build Status Coverage Status Requirements Status Total alerts Language grade: Python Documentation Status Gitter

PyMS, Python MicroService, is a Microservice chassis pattern like Spring Boot (Java) or Gizmo (Golang). PyMS is a collection of libraries, best practices and recommended ways to build microservices with Python which handles cross-cutting concerns:

  • Externalized configuration
  • Logging
  • Health checks
  • Metrics
  • Distributed tracing

PyMS is powered by Flask, Connexion and Opentracing.

Get started with Installation and then get an overview with the Quickstart.

Documentation

To know how to use, install or build a project see the docs.

Installation

pip install py-ms[all]

Quickstart

See our quickstart webpage

Create a project from scaffold

See our Create a project from scaffold webpage

How To contribute

We appreciate opening issues and pull requests to make PyMS even more stable & useful! See This doc for more details.

pyms's People

Contributors

agalin avatar alexppg avatar amandakhol avatar avara1986 avatar dependabot[bot] avatar devendra-nh avatar diptochakrabarty avatar f-arruza avatar fabianfalon avatar hcamino avatar himanshus-dev avatar ibledy avatar javierluna avatar jordivandooren avatar kingdarboja avatar lcd1232 avatar liangjq avatar miguelgrubin avatar ooigavin avatar pyup-bot avatar rapejim avatar vmjelicic avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

pyms's Issues

How do I get the database after the application is ready?

I created an Database connection on the init_libs:

    def init_libs(self) -> None:
        with self.application.test_request_context():
            self.db_client = db.get_db_client()

I see on the debugger that this object is stored on the app.ms.db_client

But now I don't know how to get this object from my API folder, current_app and g give me a "localproxy unbound", and I don't see anyway to get the application context or the app instance itself from elsewhere. Can I have a little bit help on this?

Support Black

  • Check black Formatter
  • Add the config in setup.cfg, see this link
  • Update documentation (CONTRIBUTIND and Docs) with "how to use black"
  • Use pre-commit, hooks... or the method that you want to run automatically Black

Increment type hints and add mypy

Add more Python type hints and add mypy valitations. This validation could be a step in GH actions or tox

You can learn more about type hints in this link

Mypy has a command line checker:

mypy pyms

Ideally, mypy must return 0 errors, but could be a hard work because pyms create dynamic attributes.

Add service discoveries options

Is your feature request related to a problem? Please describe.
Any cool microservices framework has some kind of service discovery. Spring boot has Eureka, but it also has kubernetes.

Why would you need this? To find services to attack programmatically. For example, if k8s service discovery were implemented, you could do canary deployments and configure a service to use the unstable service based on the k8s service labels.

Describe the solution you'd like
A Service Discovery pyms service whith different backends (k8s, eureka, etc).

Describe alternatives you've considered
n/a

Additional context
Spring boot stuff.

GraphQL Support

Is your feature request related to a problem? Please describe.
Create a service to handle GraphQL APIs, by using flask-graphql.

Support more parameters for Connexion

Is your feature request related to a problem? Please describe.
I've recently wanted to use Connexion's feature that validates responses according to API schema. Then found that parameters described in config.yml docs are ALL that can be used and there seem to be absolutely no way of passing any additional value to the API.

Describe the solution you'd like
Either there should be additional settings available in config.yml or at least some kind of a callback that could be overriden to easily pass additional parameters to API constructor. Maybe a mapping CUSTOM_OPTIONS or similar that would just be passed as unpacked kwargs to add_api? Even if not, there aren't that many parameters supporrted by this method so possibly all of them could be moved to config.yml.

Describe alternatives you've considered
Currently end user needs to basically copy whole init_app of the Swagger service - and who knows how much of the Microservice class to support custom service.

Improve README wording and fix trailing whitespaces

Is your feature request related to a problem? Please describe.

I would like to improve the wording of the README, for example:

  • Line 30: how use -> how to use
  • Line 34: started to create microservice -> started creating a microservice
  • Line 45: can`t -> can't
  • Line 98: This initialization inject -> This initialization injects
  • Line 99: like typical -> like a typical
  • Line 103: Initialize Flask instance -> Initialize a Flask instance
  • Line 109: return the flask instance and you can interact with it -> returns the flask instance, which you can interact with
  • Line 116: This command use -> This command uses
  • Line 146: finish to introduce -> finish introducing

I also see some warnings about not following Markdown convention rules, i'm using VSCode with the Markdownlint extension, which displays the following formatting warnings:

2020-10-01_10-57-03

Note: they are convention warnings, sometimes you do want a specific formatting and it's okay to not follow these rules

They were also a lot of MD009/no-trailing-spaces: Trailing spaces [Expected: 0 or 2; Actual: 1], which is a common issue if trailing whitespaces are not trimmed before commiting. In Markdown, two trailing whitespaces make a newline (useful to separate paragraphs). I also would like to fix these.

If you don't want to separate paragraphs with the spaces method, i highly recomment turning on Trim Trailing Whitespace which is available in all of the common text editors/IDEs, (for example the setting is called files.trimTrailingWhitespace in VSCode and trim_trailing_white_space_on_save on Sublime)

Describe the solution you'd like
I would like to sumbit a PR to improve the wording and fix the trailing whitespaces. I would optionally fix some or all of the Markdown convention warnings, if you want to.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Initial Update

The bot created this issue to inform you that pyup.io has been set up on this repo.
Once you have closed it, the bot will open pull requests for updates as soon as they are available.

FastAPI Support

Update init_app to support FastAPI

Possible Issues:

  • Connexion or FlaskTracing will not work with this new framework, add validations to raise an IncompatibilityError
  • Not confirmed: maybe some methods must be declared as async functions

config.yml generator, command

Create a command to ask the user the configuration that he wants and generate a config.yml

Basic Flask config:

  config:
    DEBUG: true
    TESTING: false
    APP_NAME: "Python Microservice"
    APPLICATION_ROOT: ""

Input: do you want to use request? (Y/n) Y

pyms:
  services:
    requests:
      data: ""
  config:
    DEBUG: true
    TESTING: false
    APP_NAME: "Python Microservice"
    APPLICATION_ROOT: ""

Input: do you want to use swagger? (Y/n) Y

pyms:
  services:
    requests:
      data: ""
    swagger:
      path: ""
      file: "swagger.yaml"
  config:
    DEBUG: true
    TESTING: false
    APP_NAME: "Python Microservice"
    APPLICATION_ROOT: ""

Minify dependencies

PyMS install all dependencies of the service: Connexion, requests, prometheus, jaegger... But, if you don´t need any of the service, not install by default, for example

pip install py-ms[requests]

pip install py-ms[jaegger]

Is there a standarized way to validate configuration?

Is your feature request related to a problem? Please describe.
It would be useful that before trying to start a MS, pyms checks that the configuration is using is valid. I understand that in here

def validate_conf(*args, **kwargs):
there's some kind of validation. But it seems on high level, it doesn't validate per service, for example.

Describe the solution you'd like
I'd like that every service states clearly its configuration schema and that this can be validated at when starting the microservice.

Documentation: Introduction to Microservices

PyMS is useful to build microservices with Flask. But, if someone want to contribute to PyMS, the first he needs is know how microservices work and the behavior

  • Create in docs folder a new page with documentation about microservices:
    • Links to videos
    • Links to articles
  • Explain what is a microservices

Could be ok create sections in different languages (english, spanish, etc)

Documentation (CONTRIBUTING.md): How to use Pylint, flake8 and tox

Update CONTRIBUTING.md with information about linters and tools and ways of use these:

  • Pylint: link to official documentation, how can i execute the command, benefits of pylintrc
  • Flake8: link to official documentation, how can i execute the command, benefits of define params in setup.cfg
  • Tox: link to official documentation
  • Mypy: link to official documentation, benefits of define params in setup.cfg (PR open to add this tool)
  • setup.cfg: what's mean this file

Have an config reloader endpoint

Is your feature request related to a problem? Please describe.
Sometimes you have a microservice which you want to change it's configuration. You could make the k8s deployment to recreate the cluster, but that wouldn't be need if the microservice had an endpoint which reloaded the configuration by itself.

Describe the solution you'd like
Implement an endpoint, for example /-/reload that will re-read the configuration.

Publisher Subscriber support

Not all microservices are RESTful service. A common use case is read and/or write information form a queue. A simple solution (and in most case, the best solution) to retrieve data from a queue is a serverless solution like AWS Lambdas or Google Cloud Functions. But, imagine that: you must have a stream of data, and a heavy processing (more than 5-10 minutes) and this process must run in a Kubernetes cluster. In this case, you need a main process with an healthcheck (or status, or metrics) endpoint and other process listening a queue.

Steps:

  • Create a Driver patter like Crypt Driver to support multiple queues systems (Redis; SQS, PubSub, Kafka, AMP, AMQP...)
  • This Queue must run in a different process of main Flask application
  • Update the main class Microservice to search for a Queue Resource
  • Create a blueprint like Healthcheck to get information of the queue (status, pending messages, errors...)

Possible spelling mistake

Describe the bug
Spelling mistake

To Reproduce
Steps to reproduce the behavior:
Check tutorials.md in docs folder

Expected behavior
I think the sentence should be 'This section shows tutorials for advanced use of PyMS. See Examples for simple tutorials'

Screenshots
Selection_080

Can I resolve this small issue

Find a way to standarize the configuration file

Is your feature request related to a problem? Please describe.
The configuration file has some quirks. With the given example:

my-minimal-microservice:
  APP_NAME: "Python Microservice"
  environment: "I'm running in a local machine"
pyms:
  requests:
    data: ""
  metrics: true
  • The APP_NAME and the my-minimal-microservice are redundant
  • APP_NAME is expected to be in capital letters, but others don't.

Describe the solution you'd like
Something like:

config:
  app_name: "Python Microservice"
  environment: "I'm running in a local machine"
pyms:
  requests:
    data: ""
  metrics: true

There should something that converts the app_name to an usable name, changing spaces by slashes or something like that.

Also, if using capital letters for keys of the configuration makes sense, it should be established somewhere which ones are and why.

Describe alternatives you've considered
NaN

Additional context
NaN

Add circuit breaker support

Is your feature request related to a problem? Please describe.
Circuit breaker is a key feature on distributed systems. It would be cool to have some way to support it.
Not sure if it should be a different package or maybe just integrate pyms with existing packages

Describe the solution you'd like
No idea.

Describe alternatives you've considered
There seem to be some packages, but none seem to be well used:

Additional context
Here's an article that explains the pattern.

Support OpenTelemetry for Traces

Is your feature request related to a problem? Please describe.
Would be great if there was option to use OpenTelemetry instead of OpenTracing. It seems that it can quickly become the main tracing protocol considering that it's an official merger of OpenTracing and OpenCensus, a sandbox project of CNCF. Both Microsoft and Google are maintaining the opentelemetry-python repo, Lightstep is one of the founding contributors, Sentry has decided to completely drop experimental OpenTracing support and contributed an official exporter to OpenTelemetry instead and so on.

Describe the solution you'd like
There is an official Flask integration in OpenTelemetry Python SDK repo.

Describe alternatives you've considered
It's possible to stay with just OpenTracing but sooner or later it will be a limiting factor - migration is suggested by OpenTracing maintainers and big projects probably won't add new OpenTracing support anymore (see Sentry example) so it's a dead end.

Centralize all the documentation in one repository

It may make our lives easier to have only one documentation repository. I've seen that in a lot of places there's duplicated information, and it's very easy to forget one of those places when a change is done.
If it makes sense, it can be done after the hacktoberfest, since there's a lot of activity and it might be confusing.

Outdated docstrings

Describe the bug
The docstrings on the conf.py file are outdated. It may happen in other files.

It seems difficult to maintain the scheme of the configuration file, since the structure may change over time. Maybe the structure should be defined in only one way and then be referenced around the project. And this can apply to any code on any docstring.

Add a `enabled` key on configuration of every service

Is your feature request related to a problem? Please describe.
It's uncomfortable to know that a service is loaded. You have to assume that if it exists in the configuration, it's loaded, but you can't really be sure. Also, the only way to deactivate it is to delete it from the configuration.

Describe the solution you'd like
Add a enabled key on every service, that pyms uses to load (or not) the service. I think it's easier to understand and to use any service if you can activate it and deactivated just changing the enabled key.

Change the next configuration:

pyms:
  services:
    requests:
      data: ""
    swagger:
      path: ""
      file: "swagger.yaml"

To:

pyms:
  services:
    requests:
      enabled: true
      data: ""
    swagger:
      enabled: false
      path: ""
      file: "swagger.yaml"

So the requests service is enabled, but the swagger is not.

Support jsonapi

An API returns an marshmallow.Schema object or a list of dictionaries as usually to jsonify parameter like:

from marshmallow import Schema
schema.dump(query.items, many=True)
jsonify(result)

Create a output:

[
  {
    "id": 1,
    "name": "Avengers",
    "pub_date": "2020-01-20"
  },
  {
    "id": 3,
    "name": "Iron Man 2",
    "pub_date": "2010-05-02"
  }
]

But the standard jsonapi say that the response MUST be like:

{
  "links": {
    "self": "http://example.com/articles",
    "next": "http://example.com/articles?page[offset]=2",
    "last": "http://example.com/articles?page[offset]=10"
  },
  "data": [{
    "id": 1,
    "name": "Avengers",
    "pub_date": "2020-01-20"
  },
  {
    "id": 3,
    "name": "Iron Man 2",
    "pub_date": "2010-05-02"
  }
]}
}

Create a wrapper over jsonfy to accept a SQLAlchemy paginated instance and a list of elements and create this JSON object?

An example of lask-rest-api:
https://github.com/Nobatek/flask-rest-api/blob/master/flask_rest_api/pagination.py

Documentation: PyMS with pipenv

PyMS has extra arguments. You can add PyMS in your requirements like:

Flask-SQLAlchemy ==2.4.4
...
py-ms[all]==2.6.1

Explain in the docs how to integrate with pipenv like:

[packages]
Flask-SQLAlchemy = "==2.4.4"
...
py-ms = {extras = ["all"],version = "==2.6.1"}

Add a namespace to all used environment variables

Is your feature request related to a problem? Please describe.
There's some environmental variables that pyms uses, like CONFIGMAP_FILE or KEY_FILE. Those are very generic names, if you see them without context, you might don't know why are they set and what are they purpose.

Describe the solution you'd like
I propose to add a namespace to all variables, so KEY_FILE would be PYMS_KEY_FILE.

Describe alternatives you've considered
None.

Additional context
None.

If you guys agree, I might do this change myself.

Documentation: Pytest

Create documentation about use PyMS with Pytest, in example:

conftest.py

def create_app():
    """Initialize the Flask app, register blueprints and intialize all libraries like Swagger, database, the trace system...
    return the app and the database objects.
    :return:
    """
    ms = MyMicroservice(path=__file__)
    return ms.create_app()

def conf_environment():
    if not os.environ.get("CONFIGMAP_FILE", False):
        os.environ["CONFIGMAP_FILE"] = os.getcwd() + "/config_tests.yml"


@pytest.fixture(scope="session")
def app():
    conf_environment()
    app = create_app()
    return app

This issue is similar to python-microservices/microservices-scaffold#198

Support registration in Consul

Is your feature request related to a problem? Please describe.
Would be nice to add support for registration in Consul. It can then be used as a configuration source for Prometheus for metrics scrapping. It's possible to configure manually now but standardized service would be an useful addition.

Consul can also be used as a service discovery (so would be a possible extension of requests integration as well) and key-value store (flask-consulate actually uses it as a configuration service).

Describe the solution you'd like
Add service that would wrap consulate package.

Describe alternatives you've considered
There is flask-consulate package that aims to provide some of the described functionality but it's barely maintained, provides no wheel binaries (it's messing with build process behind corporate proxy due to direct setup.py call) and some functions overlap with those that should be a part of pyms (it seems to retry service calls).

Additional context
There are other service discovery tools available, like Eureka or cloud-specific providers which means that it should be as provider-agnostic as possible, similar to tracing supporting two collection servers already.

Is there a standarized way to provide default values to service's configuration?

Is your feature request related to a problem? Please describe.
I see that some services have a way to define default values for it the services, but it doesn't seem to be standardized.

Describe the solution you'd like
We could define a way for developers to always follow the same conventions? Maybe not just for this but to add more in the future.

Don't enable services implicitly

Describe the bug
A clear and concise description of what the bug is.
When you don't enable a service, it still may be loaded.
To Reproduce
Steps to reproduce the behavior:
Use the next config:

pyms:
  services:
    metrics: true
  config:
    debug: true
    app_name: alertmanager-actions
    testing: false

When receiving a request, it will return the next error:

Tracer error: 'NoneType' object has no attribute 'tracer'

Expected behavior
It should not return that error.

Additional context
I'm using pyms 2.4.1 version.

When disabling explicitly the service, it does not return the error:

pyms:
  services:
    metrics: true
    tracer:
      enabled: false
  config:
    debug: true
    app_name: alertmanager-actions
    testing: false

Support Vault

Create a service like metrics or requests to support vault Vault https://www.vaultproject.io/

How-to:

Challenge:
Try to pass the configuration from configuration file to VAULT initialization, in example

pyms:
  services:
    vault:
      [host/url]: "[localhost|192.168....]"

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.