Giter Club home page Giter Club logo

koxudaxi / datamodel-code-generator Goto Github PK

View Code? Open in Web Editor NEW
2.3K 25.0 264.0 11.56 MB

Pydantic model and dataclasses.dataclass generator for easy conversion of JSON, OpenAPI, JSON Schema, and YAML data sources.

Home Page: https://koxudaxi.github.io/datamodel-code-generator/

License: MIT License

Python 99.37% Shell 0.02% Batchfile 0.01% Jinja 0.59% Dockerfile 0.01%
openapi code-generator python pydantic generator fastapi json-schema datamodel swagger yaml csv openapi-codegen swagger-codegen dataclass

datamodel-code-generator's Introduction

datamodel-code-generator

This code generator creates pydantic v1 and v2 model, dataclasses.dataclass, typing.TypedDict and msgspec.Struct from an openapi file and others.

PyPI version Conda-forge Downloads PyPI - Python Version codecov license Ruff Pydantic v1 Pydantic v2

Help

See documentation for more details.

Sponsors

JetBrains

Quick Installation

To install datamodel-code-generator:

$ pip install datamodel-code-generator

Simple Usage

You can generate models from a local file.

$ datamodel-codegen --input api.yaml --output model.py
api.yaml
openapi: "3.0.0"
info:
  version: 1.0.0
  title: Swagger Petstore
  license:
    name: MIT
servers:
  - url: http://petstore.swagger.io/v1
paths:
  /pets:
    get:
      summary: List all pets
      operationId: listPets
      tags:
        - pets
      parameters:
        - name: limit
          in: query
          description: How many items to return at one time (max 100)
          required: false
          schema:
            type: integer
            format: int32
      responses:
        '200':
          description: A paged array of pets
          headers:
            x-next:
              description: A link to the next page of responses
              schema:
                type: string
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Pets"
        default:
          description: unexpected error
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Error"
                x-amazon-apigateway-integration:
                  uri:
                    Fn::Sub: arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${PythonVersionFunction.Arn}/invocations
                  passthroughBehavior: when_no_templates
                  httpMethod: POST
                  type: aws_proxy
    post:
      summary: Create a pet
      operationId: createPets
      tags:
        - pets
      responses:
        '201':
          description: Null response
        default:
          description: unexpected error
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Error"
                x-amazon-apigateway-integration:
                  uri:
                    Fn::Sub: arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${PythonVersionFunction.Arn}/invocations
                  passthroughBehavior: when_no_templates
                  httpMethod: POST
                  type: aws_proxy
  /pets/{petId}:
    get:
      summary: Info for a specific pet
      operationId: showPetById
      tags:
        - pets
      parameters:
        - name: petId
          in: path
          required: true
          description: The id of the pet to retrieve
          schema:
            type: string
      responses:
        '200':
          description: Expected response to a valid request
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Pets"
        default:
          description: unexpected error
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Error"
    x-amazon-apigateway-integration:
      uri:
        Fn::Sub: arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${PythonVersionFunction.Arn}/invocations
      passthroughBehavior: when_no_templates
      httpMethod: POST
      type: aws_proxy
components:
  schemas:
    Pet:
      required:
        - id
        - name
      properties:
        id:
          type: integer
          format: int64
        name:
          type: string
        tag:
          type: string
    Pets:
      type: array
      items:
        $ref: "#/components/schemas/Pet"
    Error:
      required:
        - code
        - message
      properties:
        code:
          type: integer
          format: int32
        message:
          type: string
    apis:
      type: array
      items:
        type: object
        properties:
          apiKey:
            type: string
            description: To be used as a dataset parameter value
          apiVersionNumber:
            type: string
            description: To be used as a version parameter value
          apiUrl:
            type: string
            format: uri
            description: "The URL describing the dataset's fields"
          apiDocumentationUrl:
            type: string
            format: uri
            description: A URL to the API console for each API
model.py
# generated by datamodel-codegen:
#   filename:  api.yaml
#   timestamp: 2020-06-02T05:28:24+00:00

from __future__ import annotations

from typing import List, Optional

from pydantic import AnyUrl, BaseModel, Field


class Pet(BaseModel):
    id: int
    name: str
    tag: Optional[str] = None


class Pets(BaseModel):
    __root__: List[Pet]


class Error(BaseModel):
    code: int
    message: str


class Api(BaseModel):
    apiKey: Optional[str] = Field(
        None, description='To be used as a dataset parameter value'
    )
    apiVersionNumber: Optional[str] = Field(
        None, description='To be used as a version parameter value'
    )
    apiUrl: Optional[AnyUrl] = Field(
        None, description="The URL describing the dataset's fields"
    )
    apiDocumentationUrl: Optional[AnyUrl] = Field(
        None, description='A URL to the API console for each API'
    )


class Apis(BaseModel):
    __root__: List[Api]

Projects that use datamodel-code-generator

These OSS projects use datamodel-code-generator to generate many models. See the following linked projects for real world examples and inspiration.

Supported input types

Supported output types

Installation

To install datamodel-code-generator:

$ pip install datamodel-code-generator

http extra option

If you want to resolve $ref for remote files then you should specify http extra option.

$ pip install 'datamodel-code-generator[http]'

graphql extra option

If you want to generate data model from a GraphQL schema then you should specify graphql extra option.

$ pip install 'datamodel-code-generator[graphql]'

Docker Image

The docker image is in Docker Hub

$ docker pull koxudaxi/datamodel-code-generator

Advanced Uses

You can generate models from a URL.

$ datamodel-codegen --url https://<INPUT FILE URL> --output model.py

This method needs the http extra option

All Command Options

The datamodel-codegen command:

usage:
  datamodel-codegen [options]

Generate Python data models from schema definitions or structured data

Options:
  --http-headers HTTP_HEADER [HTTP_HEADER ...]
                        Set headers in HTTP requests to the remote host.
                        (example: "Authorization: Basic dXNlcjpwYXNz")
  --http-ignore-tls     Disable verification of the remote host's TLS
                        certificate
  --http-query-parameters QUERY_PARAMETER [QUERY_PARAMETER ...]
                        Set query parameters in HTTP requests to the remote host.
                        (example: "ref=branch")
  --input INPUT         Input file/directory (default: stdin)
  --input-file-type {auto,openapi,graphql,jsonschema,json,yaml,dict,csv}
                        Input file type (default: auto)
  --output OUTPUT       Output file (default: stdout)
  --output-model-type {pydantic.BaseModel,pydantic_v2.BaseModel,dataclasses.dataclass,typing.TypedDict,msgspec.Struct}
  --url URL             Input file URL. `--input` is ignored when `--url` is
                        used

Typing customization:
  --base-class BASE_CLASS
                        Base Class (default: pydantic.BaseModel)
  --enum-field-as-literal {all,one}
                        Parse enum field as literal. all: all enum field type
                        are Literal. one: field type is Literal when an enum
                        has only one possible value
  --field-constraints   Use field constraints and not con* annotations
  --set-default-enum-member
                        Set enum members as default values for enum field
  --strict-types {str,bytes,int,float,bool} [{str,bytes,int,float,bool} ...]
                        Use strict types
  --use-annotated       Use typing.Annotated for Field(). Also, `--field-
                        constraints` option will be enabled.
  --use-generic-container-types
                        Use generic container types for type hinting
                        (typing.Sequence, typing.Mapping). If `--use-standard-
                        collections` option is set, then import from
                        collections.abc instead of typing
  --use-non-positive-negative-number-constrained-types
                        Use the Non{Positive,Negative}{FloatInt} types instead
                        of the corresponding con* constrained types.
  --use-one-literal-as-default
                        Use one literal as default value for one literal field
  --use-standard-collections
                        Use standard collections for type hinting (list, dict)
  --use-subclass-enum   Define Enum class as subclass with field type when
                        enum has type (int, float, bytes, str)
  --use-union-operator  Use | operator for Union type (PEP 604).
  --use-unique-items-as-set
                        define field type as `set` when the field attribute
                        has `uniqueItems`

Field customization:
  --capitalise-enum-members, --capitalize-enum-members
                        Capitalize field names on enum
  --empty-enum-field-name EMPTY_ENUM_FIELD_NAME
                        Set field name when enum value is empty (default: `_`)
  --field-extra-keys FIELD_EXTRA_KEYS [FIELD_EXTRA_KEYS ...]
                        Add extra keys to field parameters
  --field-extra-keys-without-x-prefix FIELD_EXTRA_KEYS_WITHOUT_X_PREFIX [FIELD_EXTRA_KEYS_WITHOUT_X_PREFIX ...]
                        Add extra keys with `x-` prefix to field parameters.
                        The extra keys are stripped of the `x-` prefix.
  --field-include-all-keys
                        Add all keys to field parameters
  --force-optional      Force optional for required fields
  --original-field-name-delimiter ORIGINAL_FIELD_NAME_DELIMITER
                        Set delimiter to convert to snake case. This option
                        only can be used with --snake-case-field (default: `_`
                        )
  --remove-special-field-name-prefix
                        Remove field name prefix if it has a special meaning
                        e.g. underscores
  --snake-case-field    Change camel-case field name to snake-case
  --special-field-name-prefix SPECIAL_FIELD_NAME_PREFIX
                        Set field name prefix when first character can't be
                        used as Python field name (default: `field`)
  --strip-default-none  Strip default None on fields
  --use-default         Use default value even if a field is required
  --use-default-kwarg   Use `default=` instead of a positional argument for
                        Fields that have default values.
  --use-field-description
                        Use schema description to populate field docstring
  --use-pendulum
                        Use pendulum instead of `datetime` for `date`, 
                        `datetime`, and `time` data types

Model customization:
  --allow-extra-fields  Allow to pass extra fields, if this flag is not
                        passed, extra fields are forbidden.
  --allow-population-by-field-name
                        Allow population by field name
  --class-name CLASS_NAME
                        Set class name of root model
  --collapse-root-models
                        Models generated with a root-type field will be
                        mergedinto the models using that root-type model
  --disable-appending-item-suffix
                        Disable appending `Item` suffix to model name in an
                        array
  --disable-timestamp   Disable timestamp on file headers
  --enable-faux-immutability
                        Enable faux immutability
  --enable-version-header
                        Enable package version on file headers
  --keep-model-order    Keep generated models' order
  --reuse-model         Reuse models on the field when a module has the model
                        with the same content
  --target-python-version {3.6,3.7,3.8,3.9,3.10,3.11}
                        target python version (default: 3.7)
  --use-schema-description
                        Use schema description to populate class docstring
  --use-title-as-name   use titles as class names of models

Template customization:
  --aliases ALIASES     Alias mapping file
  --custom-file-header CUSTOM_FILE_HEADER
                        Custom file header
  --custom-file-header-path CUSTOM_FILE_HEADER_PATH
                        Custom file header file path
  --custom-template-dir CUSTOM_TEMPLATE_DIR
                        Custom template directory
  --encoding ENCODING   The encoding of input and output (default: UTF-8)
  --extra-template-data EXTRA_TEMPLATE_DATA
                        Extra template data
  --use-double-quotes   Model generated with double quotes. Single quotes or
                        your black config skip_string_normalization value will
                        be used without this option.
  --wrap-string-literal
                        Wrap string literal by using black `experimental-
                        string-processing` option (require black 20.8b0 or
                        later)
  --additional-imports  Custom imports for output (delimited list input).
                        For example "datetime.date,datetime.datetime"
  --custom-formatters   List of modules with custom formatter (delimited list input).
  --custom-formatters-kwargs A file with kwargs for custom formatters.

OpenAPI-only options:
  --openapi-scopes {schemas,paths,tags,parameters} [{schemas,paths,tags,parameters} ...]
                        Scopes of OpenAPI model generation (default: schemas)
  --strict-nullable     Treat default field as a non-nullable field (Only
                        OpenAPI)
  --use-operation-id-as-name
                        use operation id of OpenAPI as class names of models
  --validation          Deprecated: Enable validation (Only OpenAPI). this
                        option is deprecated. it will be removed in future
                        releases

General options:
  --debug               show debug message (require "debug". `$ pip install 'datamodel-code-generator[debug]'`)
  --disable-warnings    disable warnings
  --no-color            disable colorized output
  --version             show version
  -h, --help            show this help message and exit

Related projects

fastapi-code-generator

This code generator creates FastAPI app from an openapi file.

https://github.com/koxudaxi/fastapi-code-generator

pydantic-pycharm-plugin

A JetBrains PyCharm plugin for pydantic.

https://github.com/koxudaxi/pydantic-pycharm-plugin

PyPi

https://pypi.org/project/datamodel-code-generator

Contributing

See docs/development-contributing.md for how to get started!

License

datamodel-code-generator is released under the MIT License. http://www.opensource.org/licenses/mit-license

datamodel-code-generator's People

Contributors

aahnik avatar adaamz avatar aedial avatar airwoodix avatar auphofbsf avatar dataway avatar denisart avatar dependabot[bot] avatar dominic-walther avatar duesenfranz avatar florianludwig avatar fokko avatar ghandic avatar goodoldneon avatar i404788 avatar ijrsvt avatar indrat avatar jensheinrich avatar joshbode avatar kevin-lithic avatar kotlinisland avatar koxudaxi avatar kylebebak avatar ldej avatar mmwinther avatar pre-commit-ci[bot] avatar rezen avatar shadchin avatar shnups avatar tcrasset avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

datamodel-code-generator's Issues

Output should import types and BaseModel

The generated output should (optionally?) be a complete file ready for import into other projects, and include imports like:

from pydantic import BaseModel
from types import List, ...

...

Allow custom base class for models

Is your feature request related to a problem? Please describe.
Didn't use this library yet, just looked at code and saw that BaseModel is hardcoded in templates as base class for models:

Describe the solution you'd like
Add ability to choose the name of base class of models (and even create one automatically)
This would be helpful if I wan't to have same config, methods, etc for all subclasses

Describe alternatives you've considered
This can be approached by renaming BaseModel in a result manually, but if it not hard to implement I'd preffered to have this built-in

Additional context
Add any other context or screenshots about the feature request here.

Snake Case Field Option

Is your feature request related to a problem? Please describe.
OpenAPI or JsonSchema may describe key of objects as camelCase.
But, attributes of python should be snake_case.
The generator should provide a feature that converts filed names CamelCase to sanke_case.

Describe the solution you'd like
I would the feature give the generator a command-line option

Support String Formats

Format types

  • date
  • date-time
  • password
  • byte
  • binary
  • email
  • uuid
  • uri
  • hostname
  • ipv4
  • ipv6
  • deciaml

Support decimal

It would be great to support python's decimal.Decimal type.

Describe the solution you'd like
There's some examples, how it could be implemented on the OpenAPI side:

{"type": "string", "format": "decimal"}
{"type": "number", "format": "decimal"}

Default value is double quoted if string

Describe the bug
Default value is double quoted if string

To Reproduce

Example schema:

{
    "definitions": {
        "io.k8s.api.core.v1.ComponentStatus": {
            "description": "ComponentStatus (and ComponentStatusList) holds the cluster validation info.",
            "properties": {
                "apiVersion": {
                    "description": "APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources",
                    "type": "string",
                    "default": "v1"
                },
                "kind": {
                    "description": "Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds",
                    "type": "string",
                    "enum": [
                        "ComponentStatus"
                    ],
                    "default": "ComponentStatus"
                }
            },
            "type": "object",
            "x-kubernetes-group-version-kind": [
                {
                    "group": "",
                    "kind": "ComponentStatus",
                    "version": "v1"
                }
            ]
        }
    }
}

Used commandline:

$ datamodel-codegen --input k8s_1.16.4_patched.json --input-file-type jsonschema --output test_

Expected behavior

class ComponentStatus(BaseModel):
    apiVersion: Optional[str] = Field(
        "v1",
        description="...",
    )

But got:

class ComponentStatus(BaseModel):
    apiVersion: Optional[str] = Field(
        "'v1'",
        description="...",
    )

Version:

  • OS: Linux
  • Python Verison: 3.8
  • datamodel-code-generator git master, 2cc4d0d

Additional context
This is a regression, was working a few commits ago.

Missing import for Field

Describe the bug

To Reproduce

Example schema:
example file https://raw.githubusercontent.com/kubernetes/kubernetes/1.16.4/api/openapi-spec/swagger.json

Generated code in io/k8s/apimachinery/pkg/version.py:

from __future__ import annotations

from pydantic import BaseModel


class Info(BaseModel):
    build_date: str = Field(..., alias="buildDate")
    compiler: str
    git_commit: str = Field(..., alias="gitCommit")
    git_tree_state: str = Field(..., alias="gitTreeState")
    git_version: str = Field(..., alias="gitVersion")
    go_version: str = Field(..., alias="goVersion")
    major: str
    minor: str
    platform: str

Expected behavior

from pydantic import BaseModel, Field

Version:

  • OS: Linux
  • Python Verison: 3.8
  • datamodel-code-generator master

Support for remote $refs

Is your feature request related to a problem? Please describe.

Remote $ref are not resolved

Describe the solution you'd like

Remote $ref to be resolved

Describe alternatives you've considered

Remote $refs are downloaded and converted to local refs by an external component

Additional context

components:
  schemas:
    Problem:
      $ref: "https://opensource.zalando.com/problem/schema.yaml#/Problem"

`oneOf` not working inside `properties`

Describe the bug
I think it is allowed to use oneOf inside properties but it won't work.

To Reproduce
Steps to reproduce the behavior:

small.json

{
    "definitions": {
      "io.k8s.api.admissionregistration.v1.MutatingWebhook": {
        "properties": {
          "timeout": {
            "oneOf": [
                {
                  "type": "string"
                },
                {
                  "type": "integer"
                }
            ]
          }
        },  
        "type": "object"
      }
    }
}
$ datamodel-codegen --input small.json --input-file-type jsonschema 
Traceback (most recent call last):
  [...]
  File "datamodel_code_generator/parser/jsonschema.py", line 158, in get_data_type
    raise ValueError(f"invalid schema object {obj}")
ValueError: invalid schema object items=None uniqueItem=None type=None format=None pattern=None minLength=None maxLength=None minimum=None maximum=None multipleOf=None exclusiveMaximum=None exclusiveMinimum=None additionalProperties=None anyOf=[] allOf=[] enum=[] writeOnly=None properties=None required=[] ref=None nullable=False x_enum_varnames=[] description=None title=None example=None examples=None default=None

Error when creating model with keys starting with a number

Its me again and I have a new crazy JSON here.
Sometimes I have JSON which contains a key starting with a number.

To Reproduce
Example JSON:

{
    "Values": {
        "1 Step": "42",
        "2 Step": "23"
    }
}

Used commandline:

$ datamodel-codegen --input jsons/numbers.json --input-file-type json --output models/numbers.py --target-python-version 3.6

Expected behavior
Model is created. For the keys starting with a number there should be a underscore "_" added at the beginning of the string.

Current behavior
An error is thrown:

Traceback (most recent call last):
  File "/home/flz/.local/share/virtualenvs/modelCreate-nvqMaMoH/lib/python3.6/site-packages/datamodel_code_generator/__main__.py", line 122, in main
    validation=namespace.validation,
  File "/home/flz/.local/share/virtualenvs/modelCreate-nvqMaMoH/lib/python3.6/site-packages/datamodel_code_generator/__init__.py", line 179, in generate
    result = parser.parse()
  File "/home/flz/.local/share/virtualenvs/modelCreate-nvqMaMoH/lib/python3.6/site-packages/datamodel_code_generator/parser/base.py", line 317, in parse
    body = format_code(body, self.target_python_version)
  File "/home/flz/.local/share/virtualenvs/modelCreate-nvqMaMoH/lib/python3.6/site-packages/datamodel_code_generator/format.py", line 26, in format_code
    code = apply_black(code, python_version)
  File "/home/flz/.local/share/virtualenvs/modelCreate-nvqMaMoH/lib/python3.6/site-packages/datamodel_code_generator/format.py", line 46, in apply_black
    string_normalization=not config.get("skip-string-normalization", True),
  File "/home/flz/.local/share/virtualenvs/modelCreate-nvqMaMoH/lib/python3.6/site-packages/black.py", line 725, in format_str
    src_node = lib2to3_parse(src_contents.lstrip(), mode.target_versions)
  File "/home/flz/.local/share/virtualenvs/modelCreate-nvqMaMoH/lib/python3.6/site-packages/black.py", line 836, in lib2to3_parse
    raise exc from None
black.InvalidInput: Cannot parse: 5:5:     1_Step: str = Field(..., alias='1 Step')

Version:

  • OS: Ubuntu 20.04
  • Python Verison: 3.6.11
  • datamodel-code-generator Version 0.5.13

Additional context
I think in Python variable names must not start with a number.

Support pydantic version to v1.0 or later

The code-generator creates models for old-style pydantic (under v1.0) eg: UrlStr
I want to change the supporting version of pydantic to v1.0 or later.

Changes

  • UrlStr to AnyStr for URI
  • use Field()

I considered we should drop the old-pydantic version.
If someone has any idea then, please tell me it.

Related PRs

#100
#99

yaml support is not complete

Describe the bug
Parsing yaml OAS3 files does not work

To Reproduce
See #183 .

Example schema:

components:
  schemas:
    Problem:
      $ref: https://teamdigitale.github.io/openapi/0.0.6/definitions.yaml#/schemas/Problem

Used commandline:

$ datamodel-codegen test.yaml
``

**Expected behavior**
A generated Problem class

**Version:**
 - OS: linux
 - Python Verison: 3.7
 - datamodel-code-generator Version latest

Add JSON Schema Parser

The code generator should handle json schema.
The way is to implement JSON Shema Parser.

Space and special characters in keys

In our project we process a large number of JSON files from external source. We want to use pydantic for data models and also to check the structure of incoming files. The JSON files has spaces and special characters in the keys.

Example:

{
    "Serial Number": "A12345678",
    "Timestamp": "2020-05-26T12:15:25.792741Z",
    "Data": {
        "Length (m)": 12.34,
        "Symmetric deviation (%)": 12.216564148290807,
        "Total running time (s)": 974,
        "Mass (kg)": 42.23,
        "Initial parameters": {
            "V1": 123,
            "V2": 456
        }
    }
}

The keys with space and brackets works perfect. They are generated as Length__m_: float = Field(..., alias='Length (m)') for example.
The generator don't likes the "Data.Initial parameters" key in the example above. It is the name of the sub structure and would result in a class name with space The following error is raised:
black.InvalidInput: Cannot parse: 6:14: class Initial parameters(BaseModel):

It would be cool, if the generator would support such files. Is this possible to add this feature?

API Client

I really like the idea behind this -- jinja is a lot easier to work with than openapi-generator templates and the java-based generators.

Would you have any interest in incorporating api-client-generation capability into this codebase? I would be happy to put a little effort into transforming the API client and templated API generation from https://github.com/dmontagu/fastapi_client into something that would combine better with this code base. That way, it would 1) have fewer generation-time dependencies 2) an easier-to-work-with templating language, and 3) leverage the work you've already put in for model generation.

But I would also understand if you'd rather keep this repository more limited in scope and targeted specifically around model generation.

Recursive elements results in wrong model

I am generating models from JSON Data which comes from an external source. In my JSON file there is a element which contains a other sub element with the same name.

Example JSON

{
    "recursive": {
        "sub": {
            "recursive": {
                "value": 42.23
            }
        }
    }
}

Used commandline:

$ datamodel-codegen --input jsons/recursive.json --input-file-type json --output models/recursive.py --target-python-version 3.6

Expected behavior
The Model should look like this.

class Recursive(BaseModel):
    sub: 'Sub'

class Recursive2(BaseModel):
    sub: float

class Sub(BaseModel):
    recursive: 'Recursive2'

class Model(BaseModel):
    recursive: 'Recursive'

But it looks like this and has an indirect recursion in Recursive->Sub->Recursive... :

class Recursive(BaseModel):
    sub: 'Sub'

class Sub(BaseModel):
    recursive: 'Recursive'

class Model(BaseModel):
    recursive: 'Recursive'

Version:

  • OS: Ubuntu 20.04
  • Python Verison: 3.6
  • datamodel-code-generator Version 0.5.13

two imports with the same import-name are not distinguised

Describe the bug
I am not yet sure how to reproduce which state, as I have two different observation:

Example 1:

Generated code:

from ...apimachinery.pkg.apis.meta import v1
from ..core import v1

(from: https://github.com/FlorianLudwig/pdk8s/blob/e9b5903c4f8cf584fe60a4a84709a42f7e63b771/pdk8s/gen/io/k8s/api/networking/v1beta1.py#L12)

Example 2:

Generated code:

from ...apimachinery.pkg.apis.meta import v1
from ..core import v1 as v1_1

(from: https://github.com/FlorianLudwig/pdk8s/blob/e9b5903c4f8cf584fe60a4a84709a42f7e63b771/pdk8s/gen/io/k8s/api/extensions/v1beta1.py)

In this example, the second import is renamed to avoid the collision but the references are not. Resulting in code trying to access module members that are in v1_1 but refer to v1. (Example)

Expected behavior

The second example is a step in the right direction (or maybe import more of the package name, in this example meta.v1 instead of just v1). But the references must be updated.

Version:

  • datamodel-code-generator: git master

Array of enum not generating code properly

Describe the bug
Hi! It seems that when using an array of enum (string type), the code is not being generated properly. Instead of making a List of enums, it always allows any string because it makes a List[str].

To Reproduce

Example schema:

openapi: 3.0.0
info:
  title: datamodel-code-generator bug example
components:
  schemas:
    Type1:
      type: array
      items:
        type: string
        enum:
          - enumOne
          - enumTwo

    Type2:
      type: string
      enum:
        - enumFour
        - enumFive

Code generated:

from __future__ import annotations

from enum import Enum
from typing import List

from pydantic import BaseModel


class Type1(BaseModel):
    __root__: List[str]


class Type2(Enum):
    enumFour = 'enumFour'
    enumFive = 'enumFive'

Used commandline:

$ datamodel-codegen --input input.yaml --output tmp.py

Expected behavior
Expected behaviour would be that inside of Type1, there should be List[Enum] where Enum is a class that has the correct enum types allowed.

Version:

  • OS: MacOS 10.15
  • Python Verison: 3.7.7
  • datamodel-code-generator Version: 0.5.4

Or is there an alternative way of achieving the above that I am missing?
The goal is to essentially have a class that accepts a List of enum.
Thanks!

JSON key error

Describe the bug
I want to create pydantic models for the Spotify api and tried to parse an example response (see here).

  • datamode-codegen 0.4.4
  • python 3.8

To Reproduce

Download the gist, install datamodel-codegen and run the following command.

The command I am using to generate the the models is

datamodel-codegen --input-file-type jsonschema --input api_response.json --out output.py

Screenshots

image

Support Kubernetes Open API Spec or JSON Schema

I've tried to run this against Kubernetes OpenAPI Spec but ran into some errors.

curl -O https://raw.githubusercontent.com/kubernetes/kubernetes/master/api/openapi-spec/swagger.json
datamodel-codegen --input swagger.json --input-file-type=openapi --output=mode.py
[... error abridged ...]
  File "/proj/.venv/lib/python3.7/site-packages/datamodel_code_generator/parser/base.py", line 164, in parse
    self.parse_raw()
  File "/proj/.venv/lib/python3.7/site-packages/datamodel_code_generator/parser/openapi.py", line 12, in parse_raw
    spec_string=self.text, backend='openapi-spec-validator'
  File "/proj/.venv/lib/python3.7/site-packages/prance/__init__.py", line 106, in __init__
    self.parse()
  File "/proj/.venv/lib/python3.7/site-packages/prance/__init__.py", line 137, in parse
    self._validate()
  File "/proj/.venv/lib/python3.7/site-packages/prance/__init__.py", line 172, in _validate
    validator(parsed)
  File "/proj/.venv/lib/python3.7/site-packages/prance/__init__.py", line 237, in _validate_openapi_spec_validator
    raise_from(ValidationError, type_ex)
  File "/proj/.venv/lib/python3.7/site-packages/prance/util/exceptions.py", line 16, in raise_from
    raise klass(*from_value.args) from from_value
prance.ValidationError: unhashable type: 'dict'

I also tried the JSON Schema version but hit different errors.

pip install openapi2jsonschema
openapi2jsonschema swagger.json --kubernetes
cd schemas
datamodel-codegen --input pod.yaml --input-file-type=jsonschema --output=mode.py
Traceback (most recent call last):
  File "/proj/.venv/bin/datamodel-codegen", line 8, in <module>
    sys.exit(main())
  File "/proj/.venv/lib/python3.7/site-packages/datamodel_code_generator/__main__.py", line 168, in main
    result = parser.parse()
  File "/proj/.venv/lib/python3.7/site-packages/datamodel_code_generator/parser/base.py", line 164, in parse
    self.parse_raw()
  File "/proj/.venv/lib/python3.7/site-packages/datamodel_code_generator/parser/jsonschema.py", line 472, in parse_raw
    self.parse_raw_obj(obj_name, raw_obj)
  File "/proj/.venv/lib/python3.7/site-packages/datamodel_code_generator/parser/jsonschema.py", line 455, in parse_raw_obj
    obj = JsonSchemaObject.parse_obj(raw)
  File "/proj/.venv/lib/python3.7/site-packages/pydantic/main.py", line 402, in parse_obj
    return cls(**obj)
  File "/proj/.venv/lib/python3.7/site-packages/pydantic/main.py", line 283, in __init__
    raise validation_error
pydantic.error_wrappers.ValidationError: 2 validation errors for JsonSchemaObject
properties -> apiVersion -> type
  str type expected (type=type_error.str)
properties -> kind -> type
  str type expected (type=type_error.str)

Can this library support Kubernetes Open API Spec or JSON Schema?

JsonSchemaObject's additionalProperties may be boolean

The OpenAPI v3.0.1 schema allows JsonSchemaObject's additionalProperties to be a boolean or a JsonSchemaObject.

Specifically, the spec (https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.1.md#parameterObject) says:

"additionalProperties - Value can be boolean or object. Inline or referenced schema MUST be of a Schema Object and not a standard JSON Schema."

However the definition at L51 of jsonschema.py has:
additionalProperties: Optional['JsonSchemaObject'].

This fails on schemas with additionalProperties: false.

Suggested fix is to change L51 of jsonschema.py to:
additionalProperties: Union['JsonSchemaObject', bool, None].

Remove models with same set of field.

It would be good if we can identify that a similar model exists and use that model instead of creating a new model, especially when the FIELDS and DATA-TYPE are same.

For example, when my data is as follows,

{
    "Arm Right": {
        "Joint 1": 5,
        "Joint 2": 3, 
        "Joint 3": 66
    }, 
    "Arm Left": {
        "Joint 1": 55,
        "Joint 2": 13, 
        "Joint 3": 6
    }
}

we get the following output,

class ArmRight(BaseModel):
    Joint_1: int = Field(..., alias='Joint 1')
    Joint_2: int = Field(..., alias='Joint 2')
    Joint_3: int = Field(..., alias='Joint 3')

class ArmLeft(BaseModel):
    Joint_1: int = Field(..., alias='Joint 1')
    Joint_2: int = Field(..., alias='Joint 2')
    Joint_3: int = Field(..., alias='Joint 3')

class Model(BaseModel):
    Arm_Right: ArmRight = Field(..., alias='Arm Right')
    Arm_Left: ArmLeft = Field(..., alias='Arm Left')

Describe the solution you'd like
I think it would be good if we can detect that models for ArmLeft and ArmRight are the same, and then to just use one of these models. For example,

class ArmRight (BaseModel):
    Joint_1: int = Field(..., alias='Joint 1')
    Joint_2: int = Field(..., alias='Joint 2')
    Joint_3: int = Field(..., alias='Joint 3')

class Model(BaseModel):
    Arm_Right: ArmRight = Field(..., alias='Arm Right')
    Arm_Left: ArmRight = Field(..., alias='Arm Left')

Describe alternatives you've considered
Currently, I just delete manually. But, we have a big system with many similar structures in JSON and many time we need to manually delete about 1000 models. :(

Thank you very much for this excellent tool.

The underlying validation library is broken/unmaintained.

p1c2u/openapi-spec-validator as great as it is, does not properly validate valid OpenAPI 3 specs (at least not all of them) - since model generation relies on validation this library is also "broken" (I tried to avoid this word but my English is not good enough)

The solution could be:

  1. us taking matters in our own hands
  2. using a proper validation tool like https://www.npmjs.com/package/swagger-parser. I am super uneasy with mixing JS with Python but nowadays running a tool in a pre-made container is a valid option
  3. port the thing to Python
    It's easy to say but I am only capable of doing the second. What you think?

datamodel-code-generator is at this point needed by the Internet - being able to generate FastAPI definitions from OpenAPI 3 will make backend PyDev work much easier and API first is generally a good approach.

Like I said I'm not capable enough of making it on my own - maybe together will be able to figure things out. If you see this "issue" valid of course.

Support for pyproject.toml

datamodel-codegen is gaining a lot of command-line options which is great, but it can make the invocation quite lengthy.

How about adding support for a pyproject.toml section, e.g.

[tool.datamodel-codegen]
field-constraints = true
snake-case-field = true
strip-default-none = false
target-python-version = "3.7"

I'm happy to add this if there is desire for this feature :)

`black.InvalidInput: Cannot parse` error when pattern is explicit in a string property.

Using the JSONSchema way to declare a pattern for a string property fails on datamodel conversion.


Based on the JSONSchema docs, the right way to declare a regex to validate the property is:

{
"rasen":
   "type": "string",
   "pattern": "^(\\([0-9]{3}\\))?[0-9]{3}-[0-9]{4}$"
}

But using this in a property make datamodel-code-generator crash with a InvalidInput error because it cannot parse the arguments to use in the generated Field:

  File "/usr/lib/python3.8/site-packages/black.py", line 725, in format_str
    src_node = lib2to3_parse(src_contents.lstrip(), mode.target_versions)
  File "/usr/lib/python3.8/site-packages/black.py", line 836, in lib2to3_parse
    raise exc from None
black.InvalidInput: Cannot parse: 22:33:     rasen: Optional[constr(regex=^(\([0-9]{3}\))?[0-9]{3}-[0-9]{4}$)] = None

I tried to do the same with a yml file and found that it crash in the same point.

ransen_property:
        type: string
        description: Test
        pattern: "[\/].*$"

The traceback in this case was:

  File "/usr/lib/python3.8/site-packages/black.py", line 725, in format_str
    src_node = lib2to3_parse(src_contents.lstrip(), mode.target_versions)
  File "/usr/lib/python3.8/site-packages/black.py", line 836, in lib2to3_parse
    raise exc from None
black.InvalidInput: Cannot parse: 35:28:     __root__: constr(regex=[/].*$)

I tracked the issue back to the type validator for strings and found that striping the quotes from the regex fires this error. I will make a PR with a suggested workaround to fix this.

add Model Resolver

This code generator can't generate some model in edge-case.
I'm implementing a model resolver to fix the problem.
The model resolver should cover below.

  • manage collected model (include ref )
  • manage unique model names
  • validate valid Python name
  • resolve remote or other local files
  • manage the scoped model name for import statement

Sorry, I'm busy with the main job this month.
But, I implement the feature step by step. I will get the time for this project ๐Ÿ˜„

Related issues

#170
#169
#168
#156
#145

Handle double __ in snake to camel function

Problem

def snake_to_upper_camel(word: str) -> str:
prefix = ''
if word.startswith('_'):
prefix = '_'
word = word[1:]
return prefix + ''.join(x[0].upper() + x[1:] for x in word.split('_'))

It is an assumption that there is only a single _ in field names. This can happen if there are two spaces in a name (https://github.com/koxudaxi/datamodel-code-generator/blob/master/datamodel_code_generator/parser/base.py#L142), or the field itself contains two __.

So I think making an assumption that anything with _ is valid snake case is incorrect.

Proposed solution

Change https://github.com/koxudaxi/datamodel-code-generator/blob/master/datamodel_code_generator/parser/base.py#L45 to

return prefix + ''.join(x[0].upper() + x[1:] for x in word.split('_') if x)

That way if there are multiple _ they're skipped and the camelcase is still correctly created.

Invalid code generated for polymorphic array in Swagger spec

Describe the bug
Invalid code generated for polymorphic array in Swagger spec.

To Reproduce
Using the following Swagger YAML:

openapi: 3.0.0
info:
  title: datamodel-code-generator bug example
components:
  schemas:
    Container:
      allOf:
        - type: object
          required:
            - contents
          properties:
            contents:
              type: array
              items:
                anyOf:
                  - $ref: '#/components/schemas/Type1'
                  - $ref: '#/components/schemas/Type2'
    Type1:
      type: object
      properties:
        prop:
          type: string
    Type2:
      type: object
      properties:
        prop:
          type: string

The generated code:

# generated by datamodel-codegen:
#   filename:  swagger-borked.yaml
#   timestamp: 2020-06-10T22:51:26+00:00

from __future__ import annotations

from typing import List, Optional

from pydantic import BaseModel


class Type1(BaseModel):
    prop: Optional[str] = None


class Type2(BaseModel):
    prop: Optional[str] = None


class Container(BaseModel):
    contents: List[Type1, Type2]

The last line is invalid and causes a runtime error.

TypeError: Too many parameters for typing.List; actual 2, expected 1

Expected behavior
The generated code should be a list containing a union of the types.

class Container(BaseModel):
    contents: List[Union[Type1, Type2]]

This occurs using the latest version of datamodel-code-generator, 0.5.2.

Thanks for the tool! It's very handy. I tried to figure out how to fix this myself but found myself going in circles.

extra_template_data not working

Describe the bug

custom_template_dir seems to be ignored:

datamodel_code_generator.generate(custom_template_dir="some/path")

Version:

  • datamodel-code-generator Version: git master

convert plurals to singular

The PR is to convert plurals to singular for array models.

Detail

code-generator have to define an item from an array.
But, the code-generator don't know the valid singular name.
I think the code-generator can resolve the singular name with this library.
https://pypi.org/project/inflect/

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.