bali-framework / bali Goto Github PK

View Code? Open in Web Editor NEW

242.0 4.0 30.0 1.45 MB

Simplify declarative cloud-native development base on FastAPI and gRPC. https://bali-framework.github.io/bali/

License: MIT License

Python 99.57% Dockerfile 0.43%

python fastapi grpc micorservices

bali's People

Contributors

Stargazers

Watchers

bali's Issues

i18n support

ref: https://phrase.com/blog/posts/fastapi-i18n/

Added `events` feature

As Martin Fowler introduced https://www.youtube.com/watch?v=STKCRSUsyP0

Event-Driven architecture is a necessary part of microservices.

fetaure: allows customization of the settings module.

Feature:

like django: DJANGO_SETTINGS_MODULE

Issue:

Mandatory config.py files and main.py files in the same level directory, otherwise the config.py cannot be found.
This is not flexible enough.

Hack Solution:

env.bool("xxx", false)

libs:

pip install environs

Design a ModelAPI for gRPC View

An base class include "Create" "List" "Retrieve" "Update" "Destroy" methods. Subclasses complete business logic through declarative style like "rest_framework" did.

Add a ModelResource

Inherited from Resource, but dedicated to working with Model resources.

Added cache api

from bali.core import cache

# Usage example (API)

# Read cache 
cache.get(key)

# Set cache 
cache.set(key, value, timeout=10)

打开openapi页面时有的sqlalchemy转换报错

python: 3.10.4
bali-core: 3.2.2
app.register注册含有这个的ModelResource类

class Permission(BaseMixin, db.BaseModel):  # type: ignore
    table_comment = "权限表"
    name_doc = "权限名称"
    description_doc = "权限描述"
    module_doc = "所属模块"

    name: str = db.Column(db.String(120), doc=name_doc,
                          comment=name_doc, nullable=False)
    description: str = db.Column(
        db.Text(500), doc=description_doc, comment=description_doc)
    permission_role: 'Permission' = db.relationship(
        'GroupRole', back_populates='permission')
    permission_roles = association_proxy('permission_role', 'role')
    permission_role_ids = association_proxy('permission_role', 'role_id')
    module_id: int = db.Column(db.ForeignKey(
        't_module.id'), doc=module_doc, comment=module_doc)

打开swagger就会报错

INFO:     127.0.0.1:60036 - "GET /openapi.json HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\envs\test_platform\lib\site-packages\uvicorn\protocols\http\httptools_impl.py", line 376, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "C:\ProgramData\Anaconda3\envs\test_platform\lib\site-packages\uvicorn\middleware\proxy_headers.py", line 75, in __call__
    return await self.app(scope, receive, send)
  File "C:\ProgramData\Anaconda3\envs\test_platform\lib\site-packages\bali\application.py", line 71, in __call__
    await self._app.__call__(scope, receive, send)  # pragma: no cover
  File "C:\ProgramData\Anaconda3\envs\test_platform\lib\site-packages\fastapi\applications.py", line 261, in __call__
    await super().__call__(scope, receive, send)
  File "C:\ProgramData\Anaconda3\envs\test_platform\lib\site-packages\starlette\applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "C:\ProgramData\Anaconda3\envs\test_platform\lib\site-packages\starlette\middleware\errors.py", line 181, in __call__
    raise exc
  File "C:\ProgramData\Anaconda3\envs\test_platform\lib\site-packages\starlette\middleware\errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "C:\ProgramData\Anaconda3\envs\test_platform\lib\site-packages\starlette\exceptions.py", line 82, in __call__
    raise exc
  File "C:\ProgramData\Anaconda3\envs\test_platform\lib\site-packages\starlette\exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "C:\ProgramData\Anaconda3\envs\test_platform\lib\site-packages\fastapi\middleware\asyncexitstack.py", line 21, in __call__
    raise e
  File "C:\ProgramData\Anaconda3\envs\test_platform\lib\site-packages\fastapi\middleware\asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "C:\ProgramData\Anaconda3\envs\test_platform\lib\site-packages\starlette\routing.py", line 656, in __call__
    await route.handle(scope, receive, send)
  File "C:\ProgramData\Anaconda3\envs\test_platform\lib\site-packages\starlette\routing.py", line 259, in handle
    await self.app(scope, receive, send)
  File "C:\ProgramData\Anaconda3\envs\test_platform\lib\site-packages\starlette\routing.py", line 61, in app
    response = await func(request)
  File "C:\ProgramData\Anaconda3\envs\test_platform\lib\site-packages\fastapi\applications.py", line 216, in openapi
    return JSONResponse(self.openapi())
  File "C:\Users\Administrator\PycharmProjects\test_platform\backend\main.py", line 56, in custom_openapi
    openapi_schema = get_openapi(
  File "C:\ProgramData\Anaconda3\envs\test_platform\lib\site-packages\fastapi\openapi\utils.py", line 418, in get_openapi
    definitions = get_model_definitions(
  File "C:\ProgramData\Anaconda3\envs\test_platform\lib\site-packages\fastapi\utils.py", line 29, in get_model_definitions
    m_schema, m_definitions, m_nested_models = model_process_schema(
  File "pydantic\schema.py", line 580, in pydantic.schema.model_process_schema
  File "pydantic\schema.py", line 621, in pydantic.schema.model_type_schema
  File "pydantic\schema.py", line 254, in pydantic.schema.field_schema
  File "pydantic\schema.py", line 461, in pydantic.schema.field_type_schema
  File "pydantic\schema.py", line 847, in pydantic.schema.field_singleton_schema
  File "pydantic\schema.py", line 698, in pydantic.schema.field_singleton_sub_fields_schema
  File "pydantic\schema.py", line 526, in pydantic.schema.field_type_schema
  File "pydantic\schema.py", line 921, in pydantic.schema.field_singleton_schema
  File "C:\ProgramData\Anaconda3\envs\test_platform\lib\abc.py", line 123, in __subclasscheck__
    return _abc_subclasscheck(cls, subclass)
TypeError: issubclass() arg 1 must be a class

像如下的model就不会，看上去似乎没有什么区别

class Tag(BaseMixin, db.BaseModel):  # type: ignore
    table_comment = "标签表"
    name_doc = "标签名称"
    color_doc = "标签颜色"
    tag_type_doc = "标签类型"
    description_doc = "标签描述"

    name: str = db.Column(db.String(120), doc=name_doc,
                          comment=name_doc, nullable=False)
    color: str = db.Column(db.String(120), doc=color_doc, comment=color_doc)
    tag_type: int = db.Column(
        db.Integer, doc=tag_type_doc, comment=tag_type_doc, default=get_tag_enum)
    description: str = db.Column(
        db.Text(500), doc=description_doc, comment=description_doc)
    tag_project: 'ProjectInfo' = db.relationship(
        'ProjectTag', back_populates='tag', lazy='selectin', join_depth=2)
    tag_projects = association_proxy('tag_project', 'project')
    tag_project_ids = association_proxy('tag_project', 'project_id')
    tag_testcase_site: 'TestcaseSite' = db.relationship(
        'TestcaseSiteTag', back_populates='tag', lazy='selectin', join_depth=2)
    tag_testcase_sites = association_proxy(
        'tag_testcase_site', 'testcase_site')
    tag_testcase_site_ids = association_proxy(
        'tag_testcase_site', 'testcase_site_id')
    tag_testcase: 'Testcase' = db.relationship(
        'TestcaseTag', back_populates='tag', lazy='selectin', join_depth=2)
    tag_testcases = association_proxy('tag_testcase', 'testcase')
    tag_testcase_ids = association_proxy('tag_testcase', 'testcase_id')

并且from bali.schemas import model_to_schema这个函数转化有关系字段的模型（relationship/association_proxy）也有问题

Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\envs\test_platform\lib\site-packages\sqlalchemy\orm\clsregistry.py", line 393, in _resolve_name
    rval = d[token]
  File "C:\ProgramData\Anaconda3\envs\test_platform\lib\site-packages\sqlalchemy\util\_collections.py", line 746, in __missing__
    self[key] = val = self.creator(key)
  File "C:\ProgramData\Anaconda3\envs\test_platform\lib\site-packages\sqlalchemy\orm\clsregistry.py", line 372, in _access_cls
    return self.fallback[key]
KeyError: 'ProjectMaster'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\Administrator\PycharmProjects\test_platform\backend\main.py", line 13, in <module>
    from app.system.view import (
  File "C:\Users\Administrator\PycharmProjects\test_platform\backend\app\system\view.py", line 11, in <module>
    from app.system.schema import (
  File "C:\Users\Administrator\PycharmProjects\test_platform\backend\app\system\schema.py", line 15, in <module>
    _UserSchema = model_to_schema(User)
  File "C:\ProgramData\Anaconda3\envs\test_platform\lib\site-packages\bali\schemas\__init__.py", line 25, in model_to_schema
    for attr in mapper.attrs:
  File "C:\ProgramData\Anaconda3\envs\test_platform\lib\site-packages\sqlalchemy\util\langhelpers.py", line 1184, in __get__
    obj.__dict__[self.__name__] = result = self.fget(obj)
  File "C:\ProgramData\Anaconda3\envs\test_platform\lib\site-packages\sqlalchemy\orm\mapper.py", line 2488, in attrs
    self._check_configure()
  File "C:\ProgramData\Anaconda3\envs\test_platform\lib\site-packages\sqlalchemy\orm\mapper.py", line 1924, in _check_configure
    _configure_registries({self.registry}, cascade=True)
  File "C:\ProgramData\Anaconda3\envs\test_platform\lib\site-packages\sqlalchemy\orm\mapper.py", line 3483, in _configure_registries
    _do_configure_registries(registries, cascade)
  File "C:\ProgramData\Anaconda3\envs\test_platform\lib\site-packages\sqlalchemy\orm\mapper.py", line 3522, in _do_configure_registries
    mapper._post_configure_properties()
  File "C:\ProgramData\Anaconda3\envs\test_platform\lib\site-packages\sqlalchemy\orm\mapper.py", line 1941, in _post_configure_properties
    prop.init()
  File "C:\ProgramData\Anaconda3\envs\test_platform\lib\site-packages\sqlalchemy\orm\interfaces.py", line 231, in init
    self.do_init()
  File "C:\ProgramData\Anaconda3\envs\test_platform\lib\site-packages\sqlalchemy\orm\relationships.py", line 2145, in do_init
    self._process_dependent_arguments()
  File "C:\ProgramData\Anaconda3\envs\test_platform\lib\site-packages\sqlalchemy\orm\relationships.py", line 2240, in _process_dependent_arguments
    self.target = self.entity.persist_selectable
  File "C:\ProgramData\Anaconda3\envs\test_platform\lib\site-packages\sqlalchemy\util\langhelpers.py", line 1113, in __get__
    obj.__dict__[self.__name__] = result = self.fget(obj)
  File "C:\ProgramData\Anaconda3\envs\test_platform\lib\site-packages\sqlalchemy\orm\relationships.py", line 2107, in entity
    argument = self._clsregistry_resolve_name(self.argument)()
  File "C:\ProgramData\Anaconda3\envs\test_platform\lib\site-packages\sqlalchemy\orm\clsregistry.py", line 397, in _resolve_name
    self._raise_for_name(name, err)
  File "C:\ProgramData\Anaconda3\envs\test_platform\lib\site-packages\sqlalchemy\orm\clsregistry.py", line 375, in _raise_for_name
    util.raise_(
  File "C:\ProgramData\Anaconda3\envs\test_platform\lib\site-packages\sqlalchemy\util\compat.py", line 208, in raise_
    raise exception
sqlalchemy.exc.InvalidRequestError: When initializing mapper mapped class User->t_user, expression 'ProjectMaster' failed to locate a name ('ProjectMaster'). If this is a class name, consider adding this relationship() to the <class 'app.system.model.User'> class after both dependent classes have been defined.

由于无法使用model_to_schema,所以我直接使用的pydantic.BaseModel去定义的schema

class ModelSerializer(BaseModel):
    created_time: datetime | None
    updated_time: datetime | None
    id: int | None

    class Config:
        orm_mode = True
        fields = {'is_active': {'exclude': True}}
        anystr_strip_whitespace = True
        max_anystr_length = 120
        arbitrary_types_allowed = True

    @classmethod
    def from_orm(cls: Type, obj: Any) -> ModelSerializer:
        # 用来主动调用from_orm datetime->str
        original = super().from_orm(obj)
        for k, v in original.__fields__.items():
            if v.type_ == datetime:
                original.__dict__[k] = original.__dict__[k].strftime(settings.DATETIME_FORMAT)
        return original

class PermissionSchema(ModelSerializer):
    name: constr(min_length=2)
    description: constr()
    permission_roles: list['Role'] | None
    permission_role_ids: list[int] | None
    module_id: int


class TagSchema(ModelSerializer):
    name: constr(min_length=2)
    color: constr(min_length=2)
    tag_type: int
    description: constr()

logging out of the box

Now we have a logging configuration file in projects.

It's complex to setting up logging.

asyncio SQLAlchemy

ref: https://docs.sqlalchemy.org/en/20/orm/extensions/asyncio.html

我怎样去修改Resource中自定义action的response.status_code

我似乎并不能像fastapi示例那样直接在函数上加上response对象

@app.put("/get-or-create-task/{task_id}", status_code=200)
def get_or_create_task(task_id: str, response: Response):
    if task_id not in tasks:
        tasks[task_id] = "This didn't exist before"
        response.status_code = status.HTTP_201_CREATED
    return tasks[task_id]

如果加上了response对象参数，发送接口时会提示缺少response关键字参数，这在重写ModelResource的方法时应该也会遇到；而且似乎Resource是无法使用fastapi中的Depends这个十分方便的特性，想问下是不是除了基础的crud，其他的api其实还是需要fastapi原生的方法来写,比如上传下载文件之类的，然后include_router到_app.routes里？
说到这边还有个疑问，Resource/ModelResource怎样和APIRoute配合形成类似flask和sanic中blueprint的效果？

Unify the file structure of different types of services

HTTP / RPC / EVENT has its own package layout.

It's hard to use correctly.

HTTP & RPC support asyncio

Resource generic action `list` support filters

Filters must defined in resource.filters

class GreeterResource(Resource):

    schema = Greeter

    filters = {}

install failed

python version: 3.9.2
pip version: 22.0.4

ERROR: Could not build wheels for ujson, which is required to install pyproject.toml-based projects

Is necessary for 'fastapi[all]' in requirement.txt

Feature:

Add custom parameters that allow separate registration of HTTP Router and RPC Service.

Background:

When you use the router to add a routing prefix and then continue to register the resource with app.register(), duplicate HTTP API registrations occur.
So I think it is necessary to add parameters in the app.register() method, allow only registering rpc_service, skip registering http router.

here:

https://github.com/bali-framework/bali/blob/main/bali/application.py#L246

fix:

add http=True, rpc=True.

@singleton
class Bali:

    ...

    def register(self, resources_cls, http=True, rpc=True):
        if not isinstance(resources_cls, list):
            resources_cls = [resources_cls]

        for resource_cls in resources_cls:
            # Register HTTP service
            if http:
                  self._app.include_router(
                      router=resource_cls.as_router(),
                      prefix=resource_cls._http_endpoint,
                  )

            # Register RPC service
            if rpc and self._rpc_servicer:
                resource_cls.as_servicer(self)

        if http:
            add_pagination(self._app)

usage:

after fix, usage:

from bali.application import Bali

from proto.config import settings
from internal.router import router_v1
from internal.event.event import EventHandler
from internal.resource import TodoResource

app = Bali(
    base_settings=None,
    title=settings.SERVER_NAME,
    routers=[{
        'router': router_v1,  
        # 'prefix': '/v1',
    }],
    backend_cors_origins=['http://127.0.0.1'],  # ["*"]
    # rpc_service=grpc_server_async,
    event_handler=EventHandler
)

# 
# 
# 
app.register(TodoResource, http=false)  # fix here!

if __name__ == "__main__":
    app.start()

FastAPI project with gRPC environments

Freeze common requirements
Initialized project settings files
Docs

希望完善一下tour和examples

Enhanced configuration

1. Store config in the environment

According to 12 factor's config specification. Our configuration still in python code.

We need to enhance it using environment or dotenv.

2. Optimized the stage template code

cache memoize enabled

Need a cache memoize feature according django-cache-memoize https://pypi.org/project/django-cache-memoize/

# Import the cache_memoize from bali core 
from bali.core import cache_memoize

# Attach decorator to cacheable function with a timeout of 100 seconds.
@cache_memoize(100)
def expensive_function(start, end):
    return random.randint(start, end)

Updated PYTHONPATH

I added the following path to PYTHONPATH manually in the Dockerfile

ENV PYTHONPATH="$PYTHONPATH:/finance"
ENV PYTHONPATH="$PYTHONPATH:/finance/clients"
ENV PYTHONPATH="$PYTHONPATH:/finance/clients/auth"
ENV PYTHONPATH="$PYTHONPATH:/finance/clients/user"
ENV PYTHONPATH="$PYTHONPATH:/finance/services/rpc"

Now we need it included in bali initialized period.

How to create a background task in gRPC service

As FastAPI docs metioned, backgroud task could been added using background_tasks.add_task:

https://fastapi.tiangolo.com/tutorial/background-tasks/.

@app.post("/send-notification/{email}")
async def send_notification(email: str, background_tasks: BackgroundTasks):
    background_tasks.add_task(write_notification, email, message="some notification")
    return {"message": "Notification sent in the background"}

But how to create a background task in gRPC service's RPC?

class GreeterResource(Resource):

    schema = Greeter

    permission_class = {
         "get": IsAdmin,
    }

All actions using same permission class

class GreeterResource(Resource):

    schema = Greeter

    permission_class = [IsAdmin,]

The permission class will return a boolean result to determine whether the request is allowed, and will fill in request.user infos

Introduce the protobuf practice which `bali` using

bali-cli(https://github.com/bali-framework/bali/) is based on the practice of protobuf, which comes from the best practice by Google results.

But we never tell the detail of it.

关于 MySQL 连接失效的问题

sqlalchemy 有三个配置可能可以帮助解决问题。具体的设置值还需要更多资料和实践支持：

增加连接池内连接数
:param pool_size=5: the number of connections to keep open
inside the connection pool. This used with
:class:`~sqlalchemy.pool.QueuePool` as
well as :class:`~sqlalchemy.pool.SingletonThreadPool`. With
:class:`~sqlalchemy.pool.QueuePool`, a `pool_size` setting
of 0 indicates no limit; to disable pooling, set `poolclass` to
:class:`~sqlalchemy.pool.NullPool` instead.

定期完全重置连接池
:param pool_recycle=-1: this setting causes the pool to recycle
connections after the given number of seconds has passed. It
defaults to -1, or no timeout. For example, setting to 3600
means connections will be recycled after one hour. Note that
MySQL in particular will disconnect automatically if no
activity is detected on a connection for eight hours (although
this is configurable with the MySQLDB connection itself and the
server configuration as well).

连接前进行 " 预 ping "（在直接使用 pymysql 编程的场景下，它是一种可以工作的策略，尚不清楚对运行中的事务的影响）
:param pool_pre_ping: boolean, if True will enable the connection pool
"pre-ping" feature that tests connections for liveness upon
each checkout.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.

bali-framework / bali Goto Github PK

bali's People

Contributors

Stargazers

Watchers

Forkers

bali's Issues

Feature:

Issue:

Hack Solution:

Feature:

Background:

fix:

usage:

sqlalchemy 有三个 配置 可能可以帮助解决问题。具体的设置值还需要更多资料和实践支持：

Recommend Projects

Recommend Topics

Recommend Org

sqlalchemy 有三个配置可能可以帮助解决问题。具体的设置值还需要更多资料和实践支持：