API Client about datamodel-code-generator HOT 16 OPEN

@dmontagu
Thank you for showing me your thoughts

I decided to delay create a new fastapi code generator by your opinion.
parse_as_type is a very good feature. I'm waiting for published version 1 of pydantic.

OK, I will think about the client.

from datamodel-code-generator.

@dmontagu
I agree.

As a separate note, if we support API client generation, I think we should design the model-generation-templates in such a way that, if your server-side models are all in a self-contained package

How did you think about technical design for other features? (eg. fastapi client)
Now, Paser class receive the model class.

    parser = OpenAPIParser(
        BaseModel,
        CustomRootType,
        filename=input_filename,
        base_class=namespace.base_class,
    )

I think we should change to

    parser = OpenAPIParser(
        BaseModel,
        CustomRootType,
        filename=input_filename,
        base_class=namespace.base_class,
        client=FastAPIClient, # or AsyncClient
        server=FastAPIServer
    )

Also, we may distribute a separate package like this.

$ pip install datamodel-code-generator[fastapi]

I don't know the idea is good. but, I don't want a package heavyweight.
I see it's a balance.

it would be possible to just include your server-side models code directly if desired. (Even if you don't need to generate the models, you still need most of the model-generation logic to generate the api-client operations from the openapi spec.)

I feel we must think about the usability for people at first. next is extensibility.

This way I can retain model methods, which obviously are not supported via OpenAPI schema; it helps make the generated client into something closer to an SDK.

I'm interested. It's a high practicality client 😄

from datamodel-code-generator.

@dmontagu
Thank you for talking me a great plan.
OK, let's think about how to develop features.

I had decided to create another code generator as different projects.
But, I'm thinking datamodel-code-generator should support other features as an extension.
Users don't want to install and run a lot of tools.
I imagine the cli accepts features a sub-command or options
eg.

$ datamodel-codegen --input api.yaml  fastapi-client

Btw, I tell you the base plan on this project.
I think the project should convert data models to python objects.
I expected source models are OpenAPI and JsonSchema, DDL of SQL, GraphQL, smithy and more.
I want to create Pydantic's BaseModel as output.
However, someone may need marshmallow or python's dataclass.
I may implement other output models.

from datamodel-code-generator.

That sounds good to me. I would be on-board with it accepting arbitrary (jinja) templates, but shipping with one or more, including at least pydantic.

I think handling APIs is a natural extension -- it just involves a (related) templated-generation step based on the operations, rather than the schemas.

As a separate note, if we support API client generation, I think we should design the model-generation-templates in such a way that, if your server-side models are all in a self-contained package (possibly with some constraints on the organization of that package), it would be possible to just include your server-side models code directly if desired. (Even if you don't need to generate the models, you still need most of the model-generation logic to generate the api-client operations from the openapi spec.)

This is currently how I use my fastapi_client package -- even though it can generate models, I actually just overwrite the generated models with my server-side models code, and mostly use the generator as a way to generate the APIs (for which there is not a server-side analogy). This way I can retain model methods, which obviously are not supported via OpenAPI schema; it helps make the generated client into something closer to an SDK.

from datamodel-code-generator.

@dmontagu
How did we start the implement?
I want to merge your api-client. but, I think I should develop the pycharm-plugin first.
Would you create a PR? or Are you busy supporting pydantic community for releasing version 1?

from datamodel-code-generator.

@koxudaxi
Here's my tentative breakdown of the steps we'll need to cover (probably in this order):

1. APIs-generation works and generates valid files assuming the models are imported (but requires manually adding import statements).
2. Model generation results in one or more model files that can all be imported together without manual modifications to the generated code
3. Add automated import generation for the APIs from the model code
4. Add the SyncApis and AsyncApis classes (or something equivalent) for use as the primary external interface to the generated client.

I think the first thing to do is to port the generation logic and templates for the apis from openapi-generator format to jinja. I may not be able to get to that today/for a little while, but I'll take responsibility for doing it and making a PR as soon as I can.

Does that make sense to you?

from datamodel-code-generator.

@dmontagu
OK. I seem good the steps.

datamodel-code-generator has the Import and Imports classes for generating import statements dynamically.
We can use the classes to follow the steps when parsing an input file.

Example

source

from datamodel_code_generator.types import Import, Imports

pydantic_import = Import(import_='BaseModel', from_='pydantic')
optional_import = Import(import_='Optional', from_='typing')
list_import = Import(import_='List', from_='typing')
datetime_import = Import(import_='datetime')

imports = Imports()
imports.append(pydantic_import)
imports.append(list_import)
imports.append(optional_import)
imports.append(datetime_import)

print(imports.dump())

output

from pydantic import BaseModel
from typing import List, Optional
import datetime

I may not be able to get to that today/for a little while, but I'll take responsibility for doing it and making a PR as soon as I can.

Thank you. No problem. Would you create the PR when you have time to do?

from datamodel-code-generator.

@koxudaxi that import management is pretty cool!

However, I am concerned about cyclic schema-references (at least with pydantic), which can happen even in cases where there are no cyclic object references. For example, imagine modeling a factory process, where a number of workers are ordered in a sequence, worker types may repeat, and each worker has a reference to the next worker.

This could be modeled in a single file via:

from pydantic import BaseModel
from typing import Union


class WorkerA(BaseModel):
    worker_id: str
    next_worker: Union["WorkerA", "WorkerB", "WorkerC", None]

class WorkerB(BaseModel):
    worker_id: str
    next_worker: Union["WorkerA", "WorkerB", "WorkerC", None]
class WorkerC(BaseModel):
    worker_id: str
    next_worker: Union["WorkerA", "WorkerB", "WorkerC", None]
WorkerA.update_forward_refs()
WorkerB.update_forward_refs()
WorkerC.update_forward_refs()

But note that the "naive approach" where one file is generated per model and there are no forward refs will cause import errors.

I initially thought that this would prevent pydantic from working with an arbitrary openapi schema, but having worked through the above example, I now believe this can be handled in general by putting every BaseModel type-hint in double quotes, and calling update_forward_refs on all models from another file after all models have been imported successfully.

But I think we might need to be careful like this.

from datamodel-code-generator.

Separately, I spent some time reading through the datamodel-code-generator code, and I'm having a little bit of a hard time understanding everything it is doing -- it seems very abstract to me! (But that's not surprising given the goal of supporting arbitrary models...).

In order to support API generation, I think we'd first need something equivalent to the DataModel class, but for the paths field of an OpenAPI spec, rather than the schemas. (I think I'm understanding it correctly, but maybe not quite yet..).

@koxudaxi I'm interested in how you are thinking about the relationship between this project and OpenAPI -- do you want this project to be broader in scope than OpenAPI? If so, I think it might be awkward to include API generation, since that seems very OpenAPI-specific.

I'm interested to hear your thoughts.

from datamodel-code-generator.

@dmontagu
Thank you for your great concern.
I think the way is right to resolve the problem of cyclic schema-references.
However, It may be difficult to detect all invalid situation with cyclic schema-references before implement the code generator.
I seem not unfortunate to implement and run unittest of a few patterns.

from datamodel-code-generator.

@dmontagu
Sorry, the datamodel-code-generator code may be complex because the code separates three layers.
I show you the relation.
parser(eg. openapi parser) --- middle-datamodel --- code-generator(eg. pydantic, fastapi-clinet)

I thought about the scope of the code generator yesterday.

In order to support API generation, I think we'd first need something equivalent to the DataModel class, but for the paths field of an OpenAPI spec, rather than the schemas. (I think I'm understanding it correctly, but maybe not quite yet..).

I think DataModel is for the data class object.
Also, DataModel inherit from TemplateBase.

class DataModel(TemplateBase, ABC):

But, paths have a side of Action in OpenAPI.
We should create Action. class
eg.

class ActionBase(TemplateBase, ABC): # or better name
   ....  # implement to render methods for creating  `function` or `method`   

class OpenAPIAction(ActionBase)
     path: str
     method: str  ## http method
     params: List[Param]
     request_body:  # ...

I'm interested in how you are thinking about the relationship between this project and OpenAPI -- do you want this project to be broader in scope than OpenAPI?

First, I wanted to create another project for OpenAPI generator includes Client/Server which uses datamodel-code-generator.
However, my thinking had changed by your suggestions on the issue.
Imagine if we need another code generator then, Do we create a new one every time?
We and users will confuse.
Also, I do not think to create another model generator project each data type source (eg. JsonSchema, GraphQL). because of the same reason.
btw, swagger-ui project gets distressed to support openapi which may not expect next-generation schema type.

If we can support multi-type sources, it's the best way.
I would solve the complex problem of implementation by architecture.
The answer may be wrong. but, I feel a worthly challenge.

from datamodel-code-generator.

@dmontagu
Have you been interested in the issue yet?

I thought about the issue for a long time.
Also, I use the code-generator for our daily project with co-workers.
I think code-generator is for pre-production ready.
This code-generator supports features enum, allof, anyof...

I have the next plan, which is to create a code-generator for fastapi server.
I said this code generator should have fastapi feature.
But, It's wrong. If I add a feature of server-side into the code-generator then, It's too complex to use. Also, I will be not able to maintain the code. Your thought is correct.

Today, I detect to create a new project of code-generator for fastapi.
I guess the fastapi generator can call this code generator as a module to build pydantic model.

I would set the first scope is to create code-generator for server-side in the next project.
And, a second is to create code-generator for the client.

If you are interested in creating an API Client generator then, Would you build a new project with me or give me your code for API Client?

from datamodel-code-generator.

I have thought some more about it.

You can take a look at the fastapi_client repo. (Feel free to fork or whatever if you want to use the code.)

There are some pydantic features coming down the pipe that make building a typed client easier (e.g. parse_as_type). I think this would be very nice when writing tests of your server (not to mention broader use with other OpenAPI APIs).

I have two main frustrations with the current design of the client: 1) it uses generated models rather than the server side models. It’s nice to use exactly the server side models if possible so that you can reuse logic. 2) It uses openapi-generator, which doesn’t do a great job of handling complex edge cases.

The main benefit I see is getting a (typed) client for your API — currently you can’t get this at all, even if you are willing to copy server code.

I think my priorities may be somewhat orthogonal to yours, given I prefer to reuse the server models directly. But if you have implementation requests related to the client generator stuff I’d be happy to put some in some thought/effort.

from datamodel-code-generator.

Any news on this? At Boloo we are faced with the desire to have an API Client, fully typed. @Bobronium and I thought we could extend your package by having it also generate an API client. We then noticed this thread.

The tool would generate typed endpoints from OpenAPI spec that will interact with a manually-written client (so you could decide how you do actual HTTP requests).

Generating models is a big (if not the biggest) part of it. We are starting the project and currently discovering possibilities. IMO, integrating with datamodel-code-generator would be a great way of implementing it and solve a lot of issues we would discover along the way.

We would love to hear your thoughts on it.

from datamodel-code-generator.

Hello from @Boloo-Group, @koxudaxi! Would love to collaborate with you on the proposal above :)

In our heads, this would be a separate open source project (we've already came up with a sweet name) that would would use this one as its main tool. That will likely require some changes to this project, like providing a stable API for such needs.

Would love to hear your ideas and thoughts on that.

from datamodel-code-generator.

Hi, @guushuizen @Bobronium
Thank you for sharing the idea 👍

I think a typed API client generator is very good for everyone.
But, I don't need the client generator in my current private project.
So, I don't know what we need as a feature 😢

If you have some requests to define a stable API then I help you.
I expect someone uses datamodel-code-generator as a library.
I feel it's a great opportunity to decide the public interface.

from datamodel-code-generator.