Giter Club home page Giter Club logo

Comments (109)

nonara avatar nonara commented on July 18, 2024 11

Short update

This will be the last update I post here until it's compiling TypeScript to JSON

The project has finally taken its final form. We also have a name, domain, and orgs on npm and GH.

We have opted to make it entirely language agnostic and have created a new universal AST structure which should support every conceivable type in any language. (more detail in that on the thread linked below)

I imagine that most just want to use the tool to transform TS types to JSON Schema, so I won't bore people with the updates. There are many coming at rapid pace as all of the components get migrated in.

However, if anyone here is interested in the development process:

  • I've now made the source base live (parser and compiler aren't migrated in yet, but will be very soon).
  • I've created a thread to share updates as it happens

Invitation

It's been a long six months and has been solo all the way. I'd love it if a few people want to be involved in discussion on the design points for this system. If you or anyone you know would like to be a part of making the final decisions on what I hope to be a widely used platform, I'd really appreciate it. I don't want to force anyone to take an interest, and I'm happy to wrap it up solo, but I'd really like it if anyone wanted to help bounce ideas around. The goal is to to make sure that the final design decisions are well-thought out in order to make sure it scales well and provides a great end-user experience!

If thats you, join the discussion thread, linked below.

If that's not you, that's completely okay! Hold on - I know it's been a long time coming, but the usable tool is just about here.

Links

from ts-json-schema-generator.

nonara avatar nonara commented on July 18, 2024 7

Hi all! I'm nearly finished. TypeAlias preservation works. I'm not going to go into detail on how just now, because I'd rather just get it done and hopefully we can end the fork-mania and many, many implementations trying to solve this problem.

I'd like to get a few opinions from the group, if anyone would care to share. Please have a read of where we're at so far. Questions will follow.

Background

Plugin-based use

The central library is built to run as a plugin (via ts-patch), which incorporates both a Transformer-based plugin and a Language Service plugin in one package. This gives us some powerful new features to make this work and feel more like a native function of TypeScript, itself.

Here's what it can do:

  1. Set your options in a local config file & whenever tsc emits, your schema files emit as well. Standard is to emit [filename].schema.json, along-side your .js file, however, you can configure it to emit a single file, or even change the behaviour (more detail below).

  2. Because of the Language Service component of the plugin, your JSDoc annotations have intellisense, autocompletion, and IDE syntax errors.

Programmatic / CLI

Due to the extensibility described below, programmatic use should not be needed in the majority of cases, but it's still considered critical and the library is built around that concept.

CLI is available as well for those who dont want to use ts-patch or want to separately trigger schema emit.

Extensibility

A lot of bloat seems to exist around different base options. I've seen a lot of people requesting new options for their edge-cases. While I do believe that most are perfectly valid and understandable in their individual cases, it seems more sensible to offer a base-set that have the most common use and beyond that, make a few simple hooks that allow users to change how types are emitted.

Hooks

The following hooks are built in:

onTypeParse

Get information about each base type after its parsed, and optionally modify how, where, and if it's emitted
Signature: (schemaType: SchemaType) => SchemaType['output']
[SchemaType]:

  • name
  • kind (as string, ie. 'type-alias')
  • isExported
  • sourceFile
    • projectName (npm project)
    • projectPath
    • sourceFilePath
  • jsDocTags
  • type (Actual ts.Type object, which includes checker for advanced logic)
  • output - (mutable object which the user can modify and supply as a return value to change the output)
    • name
    • propertyPath (Path within the JSON source file. ie. '/definitions/interfaces')
    • outFile
    • schema (parsed JSON schema)

Note: If a user wants to omit a type from emitting at all, they can return false

beforeEmit

Can modify a full schema file before it's emitted
Signature: (files: SchemaFile[]) => SchemaFile[]
[SchemaFile]: (all fields mutable)

  • fileName
  • schema (JSON Schema Object)

beforeParse

Provides list of filenames before they're parsed. Can be modified to alter what gets parsed / how
Signature: (rootFileNames: string[], compilerOptions: CompilerOptions)

  • rootFileNames is an array of full file paths - This is generated either by supplied ts-config or by array of files or retrieved by a provided glob (if using programmatic / cli)
  • compilerOptions is ts.CompilerOptions - Allows modification before transforming, typechecking, etc.

Options

I'd like to keep base options somewhat sparse, but if over 10% of users need to create hooks to accomplish their goal, then we're probably missing something we should have. Here is what we have so far:

type EmitTarget = 'exported' | 'tagged' | 'internal' | 'all'

export interface TypeSchemaOptions {
  /** When set, all types are emitted to a single file */
  outFile?: string,

  /** Default is 'all' (any inclusion of 'all' in array will behave as 'all' is the sole target) */
  emitTarget?: EmitTarget | EmitTarget[]

  /** Default is tsconfig's `outDir` */
  outDir?: string,

  /** 
    * Default: `<packageName>/~/`
    */
  uriRoot?: string,

  /** (See below) (Default: true) */
  addRelativePathToURI?: boolean,

  /** Whether $id tag is added to schema files (Default: false) */
  noFileIds?: boolean,

  /** Prevent $comment from being added from JSDoc description */
  noDescriptionComment?: string,

  hooks?: { /* described above */ }
}

Default Behaviour

The following are the current design choices for the library

Annotations

Instead of many tags, each on their own line, I propose that we keep it simple to several. (Remember that we have intellisense, completion, and IDE validation, so objects behave like objects, even if they're in JSDoc)

Note: You might want to scroll down and read example section first to see real-use cases, then come back to this

We will support the following tags:

@Schema

Super-impose custom schema onto generated schema

Type: Schema definition
Valid for: Type Declarations & Properties
Output: { ...generatedSchema, ...annotatedSchema }

@schema.base

Serve as base for generated schema

Type: Schema definition
Valid for: Type Declarations & Properties
Output: { ...annotatedSchema, ...generatedSchema }

@schema.lock

Serve as sole schema definition. Generated schema is not used

Type: Schema definition
Valid for: Type Declarations & Properties
Output: { ...generatedSchema }

@schema.emit

If emitTarget includes tagged, this will be emitted

Type: Emit flag
Valid for: Type Declarations

@schema.exclude

Will not be included in emitted schema

Type: Emit flag
Valid for: Type Declarations & Properties

@schema.optional

Mark property optional

Type: Emit flag
Valid for: Properties

@schema.required

Mark property required

Type: Emit flag
Valid for: Properties

@schema.noRef

Will not create a ref element, instead, it will resolve the type.

Type: Emit flag
Valid for: Properties

@schema.noDescriptionComment

Don't use JSDoc description for $comment

Type: Emit flag
Valid for: Declarations & Properties

Examples

/**
 * Username Type
 * @schema { regex: /^[a-zA-Z0-9_]+$/ }
 */
type UserName = string;

/**
 * Will not emit at all
 * @schema.exclude
 */
type HiddenType = { a: number }

type AnotherType = { b: string }

/**
 * @schema.emit
 */
interface Abc {
  /**
   * This property won't be included
   * @schema.exclude
   */
  hiddenProp: number,

  /**
   * Schema will reflect this property as a number (not an object), and it will be optional
   * @schema.optional
   * @schema.lock { type: 'number' }
   */
  someProp: { a: number },

  /**
   * Will be a ref and will be required in output schema
   */
  otherProp?: UserName

  /**
   * Will NOT be a ref, instead it will resolve the schema for HiddenType
   */
  anotherProp: HiddenType

  /**
   * Will NOT be a ref, because of tag, will resolve to schema for AnotherType
   * @schema.noRef
   */
  anotherProp2: AnotherType
}

Output

Some things to keep in mind

  • Before making suggestions on changing this, please make sure you've read this entire document to ensure that it can't be customized how you like.
  • Some of the design choices here may not be a good idea, which is why I'm having this forum first!
  • If you don't like something, please calmly present a case for why.
  • Let's discuss pros & cons respectfully to make sure the default is the best thing for everyone!

Highlights

  • Each file outputs alongside of its emitted js counterpart.
  • uriRoot contains project name
  • propertyPath is separated into where the Type is located (see example)

Example

<SchemaFile>[]

schemaFiles:SchemaFile[] = [
  // Comes from ./main.ts
  {
    'fileName': '/full_path_to/my_pkg/main.schema.json',
    'schema': {
      '$id': '@pkg_scope/my_pkg/~/main.schema.json',
      '$schema': 'http://json-schema.org/draft-07/schema#',
      'exports': {
        'ABCD': {
          'anyOf': [
            { 'ref': '@pkg_scope/my_pkg/~/main.schema.json#/internal/A' },
            { 'ref': '@pkg_scope/my_pkg/~/main.schema.json#/external/B' },
            { 'ref': '@pkg_scope/my_pkg/~/lib/helpers.schema.json#/exports/C' },
            { 'ref': 'sub_package/~/main.schema.json#/external/D' }
          ]
        },
      },
      // Internal is used for all non-exported types
      'internal': {
        'A': { 'type': 'string' }
      },
      // External are references to types which exist outside of the package root files (or those supplied programmatically)
      'external': {
        'B': { 'type': 'string' }
      }
    }
  },
  // Comes from ./lib/helpers.ts
  {
    'fileName': '/full_path_to/my_pkg/lib/helpers.schema.json',
    'schema': {
      '$id': '@pkg_scope/my_pkg/~/lib/helpers.schema.json',
      '$schema': 'http://json-schema.org/draft-07/schema#',
      'exports': {
        'C': { type: 'string' }
      }
    }
  },
  // Comes from ./packaged/sub-package/main.ts
  // Note: This file has its own package.json, therefore the sub-package name is used.
  {
    'fileName': '/full_path_to/my_pkg/packaged/sub-package/main.schema.json',
    'schema': {
      '$id': 'sub_package/~/lib/main.schema.json',
      '$schema': 'http://json-schema.org/draft-07/schema#',
      'exports': {
        'D': { type: 'string' }
      }
    }
  }
];

Questions

First, thank you for participating! The hope is for this to be a forum to decide on the final standard for the library that we can all be happy with moving forward.

I will be finishing this over the next week - possibly a little longer. Let's discuss and figure out what works best for everyone.

Some outstanding questions I have are:

  • What other flags do we absolutely need to have (being conservative)
  • How is the URI root and property path design?
  • Do the flags seem sufficient?
  • Is there anything I'm missing?

from ts-json-schema-generator.

nonara avatar nonara commented on July 18, 2024 7

Hi all. Here's the latest...

Update

As I was working on integrating Functions into the Parser, I came across a fairly big issue.

There was no support for declaration merging!

To my knowledge, the other libraries do not have this either. I considered deferring support, however, for a number of reasons, including that function overloading relies on it and I was already going to be refactoring both the AST and the parser, I realized that it would be best to add it now.

Essentially, it means that we pull the members from Symbol, as opposed to the Type. Formerly, the parser primarily walked the full Type structure, using Symbol as needed in some special cases. The new logic will prefer Symbol where possible. (Some things do not have Symbol such as primitives and anonymous objects).

As of today, this is all fully integrated aside from the changes in the Parser. On the parser side, it's not a tremendous change. Nearly all primary logic remains the same...

What's new?

AST

  • All new AST nodes and integration of sub-nodes are complete.

  • New synthetic node type MixedNode, is created when we have a node whose Symbol has multiple declarations AND multiple base types. So if we merge a class, interface, and function, for example. (Merging with the same base type results in a regular node, ie. ClassNode)

  • A significant re-design has been done on existing AST, including splitting ObjectNode to -> ClassNode, InterfaceNode, and ObjectNode (represents anonymous object type)

  • New function nodes: FunctionNode, FunctionParameterNode, CallSignatureNode, MethodSignatureNode, ConstructSignatureNode, FunctionSignatureNode

  • Bitwise flags have also been updated and can assist in identifying node's contents. (For example, you can identify the base types used in a MixedNode via flags & typeFlags.

json-schema

  • enum added to tsTypes
  • Updates/Added for functions/methods:
    /**
     * Function signature(s)
     * Valid for:
     *   $tsType: 'function'
     */
    $functionSignature?: OneOrMore<FunctionSignature<TDefinition>>

    /**
     * Function call signature(s) (specified in type/interface)
     * Valid for:
     *   $tsType: 'interface' | 'type'
     */
    $callSignature?: OneOrMore<Exclude<FunctionSignature<TDefinition>, 'name'>>

    /**
     * Construct signature(s) (ie. new (): T)
     * Valid for:
     *   $tsType: 'interface' | 'type' | 'class'
     */
    $constructSignature?: OneOrMore<Exclude<FunctionSignature<TDefinition>, 'name'>>

    /**
     *
     * Valid for:
     *   $tsType: 'interface' | 'class'
     *   type: 'object'
     */
    methods?: Record<string, { signature: OneOrMore<RequireSome<FunctionSignature<TDefinition>, 'name'>> }>

Build System

I was still facing some lag issues. I went through a range of different options to mitigate, but overall, I ended up adding a new mode to ts-patch which allows me to 'transform' the actual Program instance that tsc uses.

This allowed me to entirely omit my large options config file which contained all of the heavy mapped types from the Project.

During compilation, the build system plugs into TSC and intelligently determines if a new options source file needs to be generated via watching relevant source files. If it's triggered, it will generate and write the file, then update the Program before TSC emits.

This means:

  1. Generated source is built automatically during compilation
  2. No initial errors from missing generated files
  3. No more lag! 🎉

What's left?

  • Finish symbol-specific parsing
  • Add basic test

Because of the extra, extra, extra delays (#compiler-life) rather than make everyone wait for all of Dominik's tests to pass, I'm going to build a smaller isolated test which will suffice to work out any major kinks in the new code and then take it live.

Not to worry - I will still be implementing all of the standard tests and will not switch my work schedule until they're all passing, but hopefully it will help take it public more quickly!

All in all, I'm still stuck deferring important for-profit work until this goes live, so believe me, I'm motivated 😅

Will post another update next weekend, (fingers crossed) hopefully with news that it's ready!

from ts-json-schema-generator.

domoritz avatar domoritz commented on July 18, 2024 6

From @HoldYourWaffle

I have created issues for most of the problems I encountered in YousefED/typescript-json-schema. I had to switch to that module because conditional types (particularily Omit) are not supported yet.

But there's something else I want to discuss. There are currently 3 modules that accomplish basically the same goal (at least that I know of), and they all have a lot of issues.
vega/ts-json-schema-generator seems to be the best overall solution, featuring clean(er) code and proper type alias support. However, it doesn't support conditional types (#100), which means Omit, Pick, Exclude etc. arent' supported either (#71, #93). This makes this module completely unusable for me because I heavily use these language features. YousefED/typescript-json-schema does support these constructs, but it contains a lot of bugs (I have reported 8 so far). There's also xiag-ag/typescript-to-json-schema, but I have no idea what the difference is between this module and it's 'ancestor' (the README only says that this is an 'extended' version).

I see 3 possible solutions for this forkmania (though there could be more):

Fix YousefED/typescript-json-schema. This can sortof be seen as the 'default' option, as it would leave the current situation mostly untouched. This isn't my preferred solution because the architecture of vega/ts-json-schema-generator looks a lot better and it doesn't really solve the forkmania.

Write a new generator from scratch, using the knowledge and experience from the other three. If we were to do this we'd of course have the amazing power of hindsight and shared knowledge, which would allow us to write a clean, well-designed and future proof generator. However, this would be very time-consuming and since vega/ts-json-schema-generator is already very well designed I don't see much reason to do this.

Merge all the good parts into vega/ts-json-schema-generator and 'opening it up' for general usage. I think this would be the best option because it will actually fix the fork issue without consuming a lot of time and effort. I think this module currently has the best/cleanest code and I don't see a reason for using one of the others if we increased flexibility and supported more use cases than what vega is doing.

If you're interested I can write up a more detailed overview in a couple of hours. No matter what option you/we choose I'd love to help/fix/develop/maintain.

from ts-json-schema-generator.

nonara avatar nonara commented on July 18, 2024 6

Here's a look at the package structure for the new org @type-schema:

  • json-schema (Full typed JSON Schema drafts - intended to serve the type-schema suite as well as any other project which needs up-to-date JSON Drafts and/or Ts-Extras draft)
  • core - Core package for converting types to JSON schema (By itself, can serve for programmatic use)
  • plugin - Combined with core, it allows TSC to emit schema files, and also includes the language service component for autocompletion and validation of tags
  • validation - Runtime type validation addon (uses AJV)
  • cli - Allows CLI use
  • setup - Quick NPX / DLX tool to install the packages needed & setup environment (has CLI prompt / flags)

Validation, CLI, and Setup are at a lower priority, and may not be entirely finished before the rest goes live.

Expect core visibility in the next few weeks

Edit: In case anyone is wondering, the reason for modularity is mainly to keep it light-weight. Helpful as they are, the package requirements for inquirer, etc are bloated, and probably most will never need the CLI, especially if using the plugin.

Meanwhile

If anyone wants to have a look, I'd love feedback or thoughts on the structures in json-schema, which is now public.

https://github.com/type-schema/json-schema

from ts-json-schema-generator.

nonara avatar nonara commented on July 18, 2024 5

Hi folks! Got an update for you, and some code, as promised.

Update

Perhaps surprisingly, the most challenging part of this whole endeavour was the options and tags system. Because this is a full-scale compiler with various usages, the options and tags have a lot of metadata attached to each. To name a few, each option has groups, targets, a category, and flags.
These define the contexts to which they apply, how they're implemented, whether they cascade to child nodes, etc.

Using this metadata, we programmatically filter and create our option sets for each context. We also need to replicate this approach via the Type system to derive Types for various option sets. This is required for some aspects of the compiler to work, such as auto-completion and the AST nodes' option properties.

I had this completed, but as I was wrapping things up, the mapped types for the sets dragged TS down to a crawl both during compile and LS computation integration in the IDE. It was so bad that I could type about 5 keystrokes before a 5 second pause. You can imagine the fun that added!

This unfortunately cost me about a week of doing what I could to refactor in hopes of optimizing or coming up with another strategy. While optimizing helped some, it was still too slow. Ultimately, it meant I had to build a portion of the build system, early, although I'd planned it for after the initial release. That took another 10 days or so.

With all that said - the good news is - all of that is finished and it works great.

What's left?

Very little. At this point, it's mainly just a few last bits in the parser to finish integrating the new TsExtras additions. Everything on the AST and render side is already done for those. I also need to implement rules and readonly.

Overall, it should likely only take a few days to wrap the final bits into the parser.

Finally, we'll sync up the latest set of tests from @domoritz and get everything passing. If there are any discrepancies beyond a simple code bug, I'll shelve those and address after I post the source.

What can I see?

I know everyone is anxious to see it. I'm excited to take it live! But I also didn't come this far to jump the gun, now. Thank you all for your patience! When I started, I didn't realize it would be a compiler! That realization has made me feel a little better about the time taken and abundance of caution put into its structure and naming conventions.

What I'm hoping is that we can collectively work out the rest together before we make an official RC. Discussions need to be had about versioning, etc. But we'll get there soon!

Source Outline

In the mean time, here is an outline package I've put together so you can see some of the key source & definition files to help you get a handle on it before the full monty is released:

StackBlitz: https://stackblitz.com/edit/type-schema-outline
GitHub: https://github.com/type-schema/outline

Notes:

  • This is not a compilable project, so it's sufficient to just check it out on StackBlitz.
  • Unfortunately linebreaks and comment headers are stripped from the AST *.d.t.s files. I like code that reads nicely, but this should be fine for now.

How to look at it

  1. Start in /main.ts and look at generateSchema() to see the steps for compilation along with each hook
  2. Have a look at the SchemaNode structures that can be manipulated during hooks in /ast/nodes and /ast/_schema-node.ts for the base (also see utilities in /ast/utilities)
  3. Have a look at /options/collections/base.collection.ts to see what the main options structure will look like. Venture into the other *.collection.ts to see the option sub-categories
  4. Check out OptionRule in /options/_options.ts for a glimpse at how the rules option is implemented. (Rules will likely be used more often than base hooks)
  5. Take a look at /generated/option-types.ts to see the generated option sets with documentation for all their various contexts

What's next?

Hopefully this will help give a bit of a picture of what's going on! There's a lot more going on under the hood, but I think this is a good start. I don't like that the AST d.ts files aren't nicely formatted, but you get the gist.

I'll post a status update next weekend, at which point I should be wrapping up or done with Dominik's tests.

Beyond this point, I'm hoping the community will help in building out unit tests and in join in discussion regarding structure, public API, options, performance etc. After it's up and going, I'll be transitioning to a 2-3day per week schedule for the compiler, so if folks want to help filling out the tests, I can devote time toward finishing the cool frills like LS extension (autocomplete) more quickly!

Until then -- cheers!

from ts-json-schema-generator.

kayahr avatar kayahr commented on July 18, 2024 3

According to this table mapping types, conditional types or even specific types like Exclude or Omit are not supported in io-ts. So for converting ast to io-ts we still have to do all the complex mapping/condition resolving as we do it now. So nothing gained here in my opinion.

And what about annotations? Currently it is very easy to pass them from typescript to the JSON schema. With io-ts in between this will probably be more difficult.

I think adding io-ts into the chain just adds more complexity and slows down the project.

from ts-json-schema-generator.

nonara avatar nonara commented on July 18, 2024 3

@CaselIT Very close! I feel bad for the poor time estimate. As I continued in, I realized that it would be better served as a more well-built package. Some additional features now built in:

  • Allows choosing json schema (7 and 2019_09 are fully typed and available)
  • Includes meta-schema standard for including additional TS type information (optional)
  • Full, custom AST constructed after parse (Every Type has corresponding SchemaNode, similar to TypeScript Node AST). Allows for easily walking and modifying during hooks
  • Much more robust set of options and tags (Can also re-assign many settings on an individual type-level via jsDoc tag or updating the AST node)

It can parse methods and functions as well. Here's a peak at the ts-extras metaschema:

/**
 * @see core/resources/type-schema-draft-01-example.ts
 */
export abstract class TsExtrasDraft_2020_04<TDraft extends IJsonSchemaDraft> implements ITsExtrasDraft {
  title = 'TS-EXTRAS';
  version = '2020_04';
  URI = ''; // TODO

  tsTypes = [ 'interface', 'class', 'object', 'type', 'method', 'function' ] as const;

  /* ********************************************************* */
  // region: Types
  /* ********************************************************* */

  abstract TsType: this['tsTypes'][number];

  abstract TypeParameter: {
    constraint?: TDraft['JsonDefinition']
    default?: TDraft['JsonDefinition']
    value?: TDraft['JsonDefinition']
  };

  abstract FunctionParameter: {
    name?: string
    type: TDraft['JsonDefinition']
    optional?: boolean
  };

  abstract FunctionSignature: {
    name?: string
    parameters: Array<TsExtrasDraft_2020_04<TDraft>['FunctionParameter']>
    returnType: TDraft['JsonDefinition']
    restParameter?: TsExtrasDraft_2020_04<TDraft>['FunctionParameter']
  };

  abstract Schema: TsExtras_2020_04<TDraft>

  // endregion
}


export interface TsExtras_2020_04<TDraft extends IJsonSchemaDraft> {
  $tsType?: TsExtrasDraft_2020_04<TDraft>['TsType']

  /**
   * $tsType: 'interface' | 'class'
   * Array of references
   */
  $heritageObjects?: Array<this>

  /**
   * $tsType: 'interface' | 'class' | 'object' | 'type' | 'method' | 'function'
   * Values make use of and $extends, default, supplied value is in value root
   */
  $typeParameters?: Record<string, TsExtrasDraft_2020_04<TDraft>['TypeParameter']>

  /**
   * $tsType: 'method' | 'function'
   */
  $functionSignature?: TsExtrasDraft_2020_04<TDraft>['FunctionSignature']

  /**
   * Object keyword
   */
  propertyOrder?: string[]
}

And an example output:

// TODO - Add heritage examples

// Class
export class ABC<G extends string> {
  a!: string;
  b!: G;
  c<T, B extends string = string, C = never>(p1: T, p2: B, p3?: number, ...args: any[]): C {
    return <any>null
  }
  d = function Hello<T>(a: any): T { return <any>null }
}

// Schema
const SchemaResult: JsonSchema = {
  '$tsType': 'class',
  '$typeParameters': {
    'G': {
      'constraint': { type: 'string' },
    }
  },

  'type': 'object',
  'properties': {
    'a': { type: 'string' },

    'b': { $ref: '#/$typeParameters/T' },

    'c': {
      '$tsType': 'method',
      '$typeParameters': {
        'T': {},
        'B': {
          'constraint': { type: 'string' },
          'default': { type: 'string' },
          'value': { '$ref': '#/properties/c/$typeParameters/B/default' }
        },
        'C': {
          'default': false
        }
      },
      '$functionSignature': {
        'parameters': [
          {
            'name': 'p1',
            'type': { $ref: '#/properties/c/$typeParameters/T' }
          },
          {
            'name': 'p2',
            'type': { $ref: '#/properties/c/$typeParameters/B' }
          },
          {
            'name': 'p3',
            'type': { type: 'number' },
            'optional': true
          }
        ],
        'restParameter': {
          'name': 'args',
          'type': true
        },
        'returnType': { $ref: '#/properties/c/$typeParameters/C' }
      },
    },

    'd': {
      '$tsType': 'function',
      '$typeParameters': {
        'T': {}  // Assumes anyAsTrue option set to false
      },
      '$functionSignature': {
        'name': 'Hello',
        'parameters': [
          {
            'name': 'a',
            'type': true
          }
        ],
        'returnType': { $ref: '#/properties/d/$typeParameters/T' }
      }
    }
  }
};

All that said, I'm wrapping up a few last bits now in the build tools to keep the options and tags DRY.

It's a composite project with the plugin (TS compiler hook + LanguageService) separate, and I plan to make the code accessible before I do the plugin aspect, so everyone here can start testing and having a look!

As for when - I don't want to put my foot in my mouth again, but I can confidently say, very soon! I'm still working away. I'll keep everyone updated here!

from ts-json-schema-generator.

cspotcode avatar cspotcode commented on July 18, 2024 3

A few reasons I'm thinking about: motivation, ability to chat with other developers, and avoiding duplicated effort.

I talk to the maintainer and a few other team members pretty regularly on Discord, so it's easier to discuss design decisions, get feedback, we avoid duplicated effort, and it's more motivating to work on typedoc. It's fun to chat and share progress.

Using the compiler API isn't free: it requires understanding the correct way to use it, understanding the quirks, understanding the gotchas to avoid. Typedoc already does that. It uses the typechecker as much as possible except for a few places where it can't.
And the typedoc team is available to explain it.

Typedoc handles exports as you would expect, as opposed to this library which exposes internal types, cannot handle multiple types with the same name, and does not handle alias exports. Typedoc needs to accurately document typescript modules, meaning that the extracted type information is more intuitive: it matches the code one-to-one.

Typedoc has some extra features which might be useful in the future. It can extract type info to a JSON dump, then use that export to render docs multiple times. It might improve performance being able to extract types once and then render multiple schemas.

from ts-json-schema-generator.

domoritz avatar domoritz commented on July 18, 2024 2

I'm super excited to see a simpler ts -> json schema compiler.

from ts-json-schema-generator.

nonara avatar nonara commented on July 18, 2024 2

@domoritz Ah, yeah. I can imagine covering all typeof would require essentially re-creating typescript's parser! Definitely not sustainable!

I need to wrap this up anyway, as I've got to get back to for-profit work next month. I'm pushing to get it done in the next 2-3 weeks. Which should definitely be achievable! There's not too much left to do, and I'll be devoting even more time to ensure it's ready on target.

What I'll do is make it public at the end of the month (at the latest) regardless of where it's at. Though I don't anticipate it not being ready to use. At that point, I may need some help filling out unit testing and improving coverage. If you and any others would like to get involved, we can work out any kinks or options changes before graduating to a full release.

My hope is that it will serve as a foundation for typescript-type / json translation, and that in moving forward, we can collectively improve and maintain it as a community effort!

from ts-json-schema-generator.

nonara avatar nonara commented on July 18, 2024 2

@domoritz Actually, yes! That would certainly be possible by using the AST Nodes.

That brings up an interesting point. Right now each node has a render method which performs the compilation to JSON schema, but it would probably be even better to entirely de-couple from JSON as well. Then parser and render functions could be hosted in separate modules (packages) on a per-language basis.

That would mean the AST was a true language-agnostic abstraction which represented just type-ish parts of other languages. I love that!

from ts-json-schema-generator.

domoritz avatar domoritz commented on July 18, 2024 2

The cross type project isn't ready and there is no timeline for json schema support so I would say send a pull request here.

from ts-json-schema-generator.

cspotcode avatar cspotcode commented on July 18, 2024 2

Is it safe to assume that the cross type project will never be finished and instead refocus on merging the big 3 schema generation libraries, as proposed here? YousefED/typescript-json-schema#295

I think it may be helpful to restart this discussion and decide how best to merge the big 3 and pay off tech debt.

For the sake of the TS ecosystem, I think it is important to point out a fundamental flaw with the approach taken in the cross-type project:

It was not really open-source. What I mean by that is, although the author often asked for help, the code was closed-source for a very long time. I can understand pros and cons to this approach, but it meant that too many potential contributors were encouraged instead to sit back and be very excited in this thread. (you can see this from all the "I can't wait" comments above)

As is so often the case with software development, I believe it's better to iterate on what we already have, and make breaking changes as necessary to eliminate tech debt. We can still reach the same end-state -- having a well-architected, feature-complete generator -- but we'll be better off along the way, because we'll always have something that works as we continue to refine, improve, refactor, and add features.


If this approach sounds acceptable to others, I can start a new issue with a checklist of tasks based on #101 (comment)

from ts-json-schema-generator.

domoritz avatar domoritz commented on July 18, 2024 2

Building and maintaining open source is difficult and it's often not part of our main jobs. I am grateful for the ideas this thread generated and I hope we can integrate what we learned into the next steps.

I'd be happy for someone to combine the best of the different schema generators as long as it supports the test cases in this repo (or make reasonable adjustments). It would be important to set up a sustainable maintenance model as well.

from ts-json-schema-generator.

domoritz avatar domoritz commented on July 18, 2024 2

I don't know but if we can avoid parsing the ast, it will make the code a lot easier and more maintainable. Maybe you can take a stab at a prototype (paying attention to aliases) and see how far you get.

from ts-json-schema-generator.

M-jerez avatar M-jerez commented on July 18, 2024 2

Hi first of all thanks for the good work to all the authors of these libraries.

Secondly, If there is gonna be a new version with breaking changes, the jsdoc annotations could/should be ditch in favour of decorators.

from ts-json-schema-generator.

codler avatar codler commented on July 18, 2024 1

@domoritz Thank you that helped!

from ts-json-schema-generator.

domoritz avatar domoritz commented on July 18, 2024 1

For questions maybe https://discordapp.com/invite/typescript. For a place to put things, I just activated the wiki in this repo so you can use that.

from ts-json-schema-generator.

domoritz avatar domoritz commented on July 18, 2024 1

@nonara I like your design decisions! Making the library extensible and the core opinionated sounds like a good approach for the one library to rule them all.

As others, I would need to play with the actual implementation before committing to a new library. Let us know when you have something to test out. If we all think the new library is the way forward, I am happy to archive ts-json-schema-generator and typescript-json-schema.

from ts-json-schema-generator.

nonara avatar nonara commented on July 18, 2024 1

@domoritz Glad to hear it! Thanks for the reply. I will update everyone again soon on the progress.

from ts-json-schema-generator.

domoritz avatar domoritz commented on July 18, 2024 1

@nonara, I hope your family member is doing well again.

I am really excited about a cleaner implementation of this library and look forward to trying it.

from ts-json-schema-generator.

domoritz avatar domoritz commented on July 18, 2024 1

I'd prefer 1) since it's shorter and the position of the argument is sufficient.

Also, IIRC, someone raised an issue with the < > symbols. Do we need to look at a different format?

It's not a big issue. It's just that references need to be valid URIs so you need to encode the string. Note that only the reference, not the definition name need to be URIs.

from ts-json-schema-generator.

CaselIT avatar CaselIT commented on July 18, 2024 1

@nonara Any update on this? Is the code available?

from ts-json-schema-generator.

nonara avatar nonara commented on July 18, 2024 1

@domoritz Yes. All tests are supported! 🙂

from ts-json-schema-generator.

nonara avatar nonara commented on July 18, 2024 1

I'm currently in the process of starting up a new project which could massively benefit from this, and I'd love to take it for a spin in the next few weeks and help work out any kinks if necessary.

@HoldYourWaffle Great! Will be grateful for the help!

@nonara Sounds good. Do you think it would make sense to put a work in progress version online so that you can get feedback on it?

@domoritz It definitely will. The reason that I haven't yet is because I believe that it would have confused and frustrated people trying to keep up. The whole project structure and each of its parts have been completely rebuilt several times with significant differences. The overall architecture was extremely fluid as I was diving in to the typescript code-base as well as imagining or encountering new crazy scenarios which demanded entirely re-designing the approach, several times over.

That was the reason for this taking so much longer. I don't imagine that we're at the end of discovering new quirks and oddities, but I do believe that we're nearly at a point where I can say it's not going to be shifting around much.

And to be honest, I'm probably looking forward to having more minds involved on it than anyone else! I'm confident in what's been done, but I'm more excited to see where it can go and how it can be improved as the bright minds behind the existing work such as ts-json-schema-generator and the others join in.

Also, would you be interested in having us host the project in the Vega org for added visibility? I could create a repo and give you admin access.

Thanks! NPM-wise, it will need to be released under its own org, because it's modular. With that in mind, I think it'd be good to use the same org-name on GitHub, but I'd be glad to add you as admin, as I know you've been championing this over several projects for some time!

from ts-json-schema-generator.

HoldYourWaffle avatar HoldYourWaffle commented on July 18, 2024 1

I'll be sure to take a look at it once my project is at that stage!

Which ones, in particular, are you talking about?

I honestly have no idea anymore. It's been months since I last looked into this, but I remember starting this whole endeavor because there were like 3 or 4 forks that all seemed to do things just a little differently.

as well as imagining or encountering new crazy scenarios which demanded entirely re-designing the approach, several times over.
That was the reason for this taking so much longer.

I feel like I relate to this way too much.

the bright minds behind the existing work such as ts-json-schema-generator and the others join in

no pressure

@HoldYourWaffle Assuming everyone is happy and we've decided to move forward with the new one, I believe GitHub allows some forms of deprecating / forwarding. Am I right on that?

I think they do, but I've never really looked into it that much. I know you can archive projects, but that's something only the maintainers can do.

Combined with core, it allows TSC to emit schema files, and also includes the language service component for autocompletion and validation of tags

This just made my dayweek.

In case anyone is wondering, the reason for modularity is mainly to keep it light-weight.

I couldn't agree more.

from ts-json-schema-generator.

domoritz avatar domoritz commented on July 18, 2024 1

Btw, I couldn’t wait any longer and added support for object and array literal expressions to this library. There are a few new test cases as well.

from ts-json-schema-generator.

domoritz avatar domoritz commented on July 18, 2024 1

Thanks for the update. It sounds like this won't be ready for a while so I will fix #446 here instead of moving over yet.

from ts-json-schema-generator.

domoritz avatar domoritz commented on July 18, 2024 1

I joined the discord but I prefer keeping decisions and important discussions on github as it's easier to search.

from ts-json-schema-generator.

HoldYourWaffle avatar HoldYourWaffle commented on July 18, 2024

Good to see there's interest in my proposal! Perhaps it would be good to pin this issue so more people will see this and hopefully voice their opinion on the matter?

from ts-json-schema-generator.

domoritz avatar domoritz commented on July 18, 2024

Thank you for the comments. I will use this issue to explain some of the different philosophies behind the different libraries.

The goal of all of these libraries is to convert Typescript to JSON schema. There are different approaches to achieving this and also different interpretations that you can follow. One fundamental issue is that JSON schema and Types are not equivalent. Some things are not expressible in the other.

YousefED/typescript-json-schema

This was the original schema generator that I picked for my main use case, which is Vega-Lite. The philosophy of this library was to be flexible and configurable for different use cases. This means that some configurations work better than others because optimizing all of them is complex.

It worked fairly well once I extended it a bit to deal with some of the more complicated types we use. However, I constantly ran into issues with getting meaningful properties that correspond to aliased types. The fundamental problem is that this library uses the type hierarchy, and not the AST to create the JSON schema. Therefore I went on to look for a different library, which was xiag-ag/typescript-to-json-schema.

I stopped active development but occasionally review PRs and make releases. I consider this library to be in maintenance mode.

xiag-ag/typescript-to-json-schema

This library was mostly written by @mrix and uses the AST to create the schema. It is also much more modular. Rather than having one big file, there are separate modules for parsing and code generation for all different types of AST nodes and types. I had to extend it significantly to make it work for Vega-Lite. That's what vega/ts-json-schema-generator is.

vega/ts-json-schema-generator

This is the extended version of xiag-ag/typescript-to-json-schema. Besides extending the supported AST nodes, I also changed some of the behavior because it was not correct for some of my use cases. Some of these improvements have since been ported back into the original library but not all.

Overall, this library is much more robust than YousefED/typescript-json-schema and produces generally better schemas. It is also much more opinionated. I don't want to have options but instead, make particular design decisions and bake them into the code.

Even though this library still has some missing functionalities, it works in production for Vega-Lite, which is a complex piece of TypeScript. I keep fixing this library if it is necessary for Vega-Lite but otherwise don't have the cycles to do anything else.

I am happy to review PRs if they have sufficient tests, don't introduce options, and provide useful additions (I will reject support for functions because there is no clean mapping to JSON schemas).

I consider this library to be in active development and would be more than thrilled to have people help with it.

Conclusion

I have considered writing a new generator based on the things I have learned. Ideally, it would be a bit leaner than vega/ts-json-schema-generator and not have as many files. However, I don't have the cycles to do it and I'm not convinced that it will be much cleaner. Overall, I think investing resources into vega/ts-json-schema-generator would be the best way forward. I am happy to review PRs and make timely releases for this library. However, my number one priority is to support Vega-Lite and anything that is in conflict with that goal will be rejected (I don't see this as an issue, though).

Let me know what you think.

from ts-json-schema-generator.

HoldYourWaffle avatar HoldYourWaffle commented on July 18, 2024

Since your comment is pretty big I'm just going to respond per section.


One fundamental issue is that JSON schema and Types are not equivalent. Some things are not expressible in the other.

This is of course true. I think the best way to handle this would be to have a section in the README that clearly lists all constructs that are not or only partially supported. It's annoying to discover something you need is unsupported, but discovering it after you've already started using an automated generator setup is way worse (I can unfortunately speak from experience).

Maybe we should also look into a way to manually override generated the generated schema in a fluent way. This way any shortcomings of the library can be manually filled in or corrected. In my own projects I've been using a script that manually changes the generated JSON object but this is very inflexible and error prone. I haven't really thought about how to implement such a mechanism, perhaps a JSdoc annotation with a JSON pointer could be something? I'll think about it.


This means that some configurations work better than others because optimizing all of them is complex.

I'm not sure I understand what you mean by this. Are you trying to say that more options → more complexity → hard to get working correctly? I agree that having more options might increase complexity, but there should always be a sensible default behavior. Having more options to override common sense because it's assumptions doesn't match with your use case is a good thing in my opinion.


It is also much more opinionated. I don't want to have options but instead, make particular design decisions and bake them into the code.

I'm not sure I agree with you on this. Having sensible defaults is always a good thing, and making some assumptions when designing something like this is necessary, but I don't see how this would hinder adding options. Could you give an example of an option you've rejected/don't want to add? I'm probably just misunderstanding what you're saying.


I keep fixing this library if it is necessary for Vega-Lite but otherwise don't have the cycles to do anything else.
I am happy to review PRs if they have sufficient tests, don't introduce options, and provide useful additions (I will reject support for functions because there is no clean mapping to JSON schemas).
I consider this library to be in active development and would be more than thrilled to have people help with it.

I'd love to help you maintain this project if you don't have the time for it! Again I'm not sure why you'd want to reject new options, could you clarify what you mean by this?
And out of curiosity: why do people want to send functions over JSON? There's no function type in the JSON spec, nor can I think of a usecase where one would want to. If you really wanted to put functions in your JSON you can use Function.toString with eval on the other side, but this is pretty unsafe in almost all cases.


Ideally, it would be a bit leaner than vega/ts-json-schema-generator and not have as many files. However, I don't have the cycles to do it and I'm not convinced that it will be much cleaner.

This is the main reason why I think a new generator isn't the best option. Is there a reason why we can't make vega/ts-json-schema-generator leaner without rewriting the whole thing? Also, what's wrong with having more files? It makes it a lot easier to find what you're looking for, as well as leaving less room for weird global state anti-patterns.


Overall, I think investing resources into vega/ts-json-schema-generator would be the best way forward. I am happy to review PRs and make timely releases for this library. However, my number one priority is to support Vega-Lite and anything that is in conflict with that goal will be rejected (I don't see this as an issue, though).

I also think this is be the best way forward. I don't see a reason why there would be conflicts with Vega, since the goal of a general purpose library is to support most (if not all) usecases, Vega included of course.


If we decide to adopt this strategy there's one more issue remaining: uniting the modules/forks to fix the current forkmania. I think YousefED/typescript-json-schema could just be deprecated with a nice forward to this module (as soon as it's ready of course, mainly looking at conditional types).

xiag-ag/typescript-to-json-schema is a different story though. I found this PR by you that aims to merge the 2 repositories together, but as you already know there hasn't been any response in 2 years. It seems like @mrix has disappeared from the community, which of course doesn't help our case.

The main reason why I want the forks to be united is that the current situation is really confusing. Last week I basically went like this:

  • typescript-json-schema, has the most downloads and looks pretty solid
  • typescript-to-json-schema, mentions that it uses a superior approach in the README which apparently fixes type aliases? Seems like this is the best choice then
  • ts-json-schema-generator, is a fork of the previous superior one? What's the difference? There's like 200+ additional commits so I think it's better? Why are all these names so similar?!
    And it only got worse when I discovered that you have contributed to both and recommend both to users of the other.

Removing YousefED/typescript-json-schema from the equation would help a lot. We may never get a response from @mrix, but we can remove this repository's forked status or add a clear explanation in the README on what the differences are and why you should probably use this module instead of the upstream one.


On a completely different note, maybe it's a good idea to create an issue in YousefED/typescript-json-schema referencing this issue and pin it there too. Since that module has more users we're probably going to get more responses then.

from ts-json-schema-generator.

domoritz avatar domoritz commented on July 18, 2024

One fundamental issue is that JSON schema and Types are not equivalent. Some things are not expressible in the other.

This is of course true. I think the best way to handle this would be to have a section in the README that clearly lists all constructs that are not or only partially supported. It's annoying to discover something you need is unsupported, but discovering it after you've already started using an automated generator setup is way worse (I can unfortunately speak from experience).

The issue here really is JSON schema and not Typescript. I have found good ways around missing things in typescript such as maxLength but the other way around it much messier. For example, JSON schema neither supports union types or inheritance properly.

This means that some configurations work better than others because optimizing all of them is complex.

I'm not sure I understand what you mean by this. Are you trying to say that more options → more complexity → hard to get working correctly? I agree that having more options might increase complexity, but there should always be a sensible default behavior. Having more options to override common sense because it's assumptions doesn't match with your use case is a good thing in my opinion.

More options increase the number of paths through the code and make it harder to get right. I am definitely against adding more code paths just to support another use case. Instead, the defaults should be good.

As an example, including or not including aliases or not using a top-level reference have implications on many parts of the code. I can speak from experience that not having the config options avoided a bunch of headaches.

I am not against the ability to configure things that have only implications on localized pieces of the code.

And out of curiosity: why do people want to send functions over JSON? There's no function type in the JSON spec, nor can I think of a usecase where one would want to.

I agree and still there we got PRs and issues for it: https://github.com/YousefED/typescript-json-schema/issues?utf8=%E2%9C%93&q=functions+

Is there a reason why we can't make vega/ts-json-schema-generator leaner without rewriting the whole thing?

No. I think the current design is good enough and I don't see any reason to rewrite.

I think YousefED/typescript-json-schema could just be deprecated with a nice forward to this module

It's already in maintenance mode and I think that's what it should be. A forward link sounds good to me but then I want support with issues that people report ;-)

xiag-ag/typescript-to-json-schema has a few features that we should get working in this fork. See #63

On a completely different note, maybe it's a good idea to create an issue in YousefED/typescript-json-schema referencing this issue and pin it there too.

Go ahead. I will pin it. I already added a note to https://github.com/YousefED/typescript-json-schema#background. Before we can deprecate the other library, we need support for conditionals here.

from ts-json-schema-generator.

HoldYourWaffle avatar HoldYourWaffle commented on July 18, 2024

The issue here really is JSON schema and not Typescript. I have found good ways around missing things in typescript such as maxLength but the other way around it much messier. For example, JSON schema neither supports union types or inheritance properly.

You really did a good job expressing stuff like minLength! It's very fluent and it just makes a lot of sense. It's also self-documenting by definition, which is always a good thing.
I'm not sure why union types would be a problem, isn't that just oneOf? Inheritance is a common issue with JSON schema, but since this is an automated tool it's probably not that bad to have some duplication in the output schema (perhaps adding a description field with the original information would be good for clarity/readability?).

It should also be noted that without additionalProperties: false it's perfectly possible to express inheritance using allOf, so if we really wanted to support a form of inheritance preservance this could be an option, but I think this will get needlessly complex very quickly. Since this is an issue with the JSON schema spec itself and not with this module I think a clear explanation on why this isn't possible in the README would suffice for now.


More options increase the number of paths through the code and make it harder to get right. I am definitely against adding more code paths just to support another use case. Instead, the defaults should be good.
As an example, including or not including aliases or not using a top-level reference have implications on many parts of the code. I can speak from experience that not having the config options avoided a bunch of headaches.
I am not against the ability to configure things that have only implications on localized pieces of the code.

That makes a lot of sense. So something like --strictTuples (is it the default yet?) or --strictNullChecks (however misleading it may be) would be fine? I agree that configuring the overall shape of the schema will get messy very quickly, but as far as I can see there's no reason why options for individual constructs should be rejected.


I think YousefED/typescript-json-schema could just be deprecated with a nice forward to this module

It's already in maintenance mode and I think that's what it should be. A forward link sounds good to me but then I want support with issues that people report ;-)

That makes sense, but is there really a reason why someone would want to use the other module once we include all missing feature here? It's always good to keep providing support for something, but is it really worth the effort if this were to be a (practically) drop-in replacement?


xiag-ag/typescript-to-json-schema has a few features that we should get working in this fork. See #63

I'd love to help, but I can't figure out what new features we're missing since there are so many changes in the vega version. If you can give me some kind of list I'd be more than happy to take a look.


Go ahead. I will pin it.

Done. I also created an issue in @XriM's repository in case there are more lost souls like I was.

from ts-json-schema-generator.

domoritz avatar domoritz commented on July 18, 2024

I'm not sure why union types would be a problem, isn't that just oneOf

I meant intersection types. allOf does not work because of additionalProperties: false as you noted in inheritance as well.

So something like --strictTuples (is it the default yet?) or --strictNullChecks (however misleading it may be) would be fine?

Yep. I think my request to keep the number of code paths low is reasonable and I think you agree.

That makes sense, but is there really a reason why someone would want to use the other module once we include all missing feature here?

In the future, yes.

I'd love to help, but I can't figure out what new features we're missing since there are so many changes in the vega version. If you can give me some kind of list I'd be more than happy to take a look.

See master...xiag-ag:master.

from ts-json-schema-generator.

HoldYourWaffle avatar HoldYourWaffle commented on July 18, 2024

Yep. I think my request to keep the number of code paths low is reasonable and I think you agree.

Of course! I was just wondering where your "line" was on what's too complicated, glad to hear it's in a very reasonable place.

See master...xiag-ag:master.

I tried to look through it, but I fear I'm just not well versed enough in the codebase to know what changed, what hasn't been done here already and what is even applicable to our version. Maybe we could copy over the tests that were added, see which ones fail and go from there? I'd love to try it but I'll have to figure out the test infrastructure first, which is of course going to take some time.

I meant intersection types. allOf does not work because of additionalProperties: false as you noted in inheritance as well.

The more I use JSON schema the more I think "How have they not solved this yet?". I think from a spec perspective there are 2 logical solutions:

  1. Ignore additionalProperties in an allOf clause. I can't think of a use-case where this would be actual useful behavior.
  2. Allow additional keys on a $ref "schema" to supplement/override the reference.

These "ideas" aren't very useful from a generator perspective of course since no validator supports them. The only solution I see is to duplicate & merge the inherited and inheriting schemas but this would probably get really messy really quickly (both in the code and in the output). Maybe adding a description to the frankensteined output would help? I feel like there should be a better solution though...
How is this currently handled?

from ts-json-schema-generator.

domoritz avatar domoritz commented on July 18, 2024

How is this currently handled?

I merge the objects in the allOf into one big object.

from ts-json-schema-generator.

HoldYourWaffle avatar HoldYourWaffle commented on July 18, 2024

Are there any issues with this approach apart from messy output?

from ts-json-schema-generator.

domoritz avatar domoritz commented on July 18, 2024

If you want to generate code from the schema again, the information about intersections and inheritance is lost.

from ts-json-schema-generator.

HoldYourWaffle avatar HoldYourWaffle commented on July 18, 2024

That makes sense. Maybe adding some kind of note to the schema could help with that?

from ts-json-schema-generator.

ForbesLindesay avatar ForbesLindesay commented on July 18, 2024

The problem with deprecating YousefED/typescript-json-schema is that it's the only one that handles conditional types properly. Supporting them requires doing type inference that is on a par with TypeScript itself. Without using getTypeAtLocation, it is incredibly difficult to keep up with the fast pace of TypeScript language improvement.

I think there could be enormous value in refactoring typescript-json-schema to be more modular, and paring down the list of options to reduce the complexity of the many code paths. There also could be merit to using an AST first approach - i.e. do what we can via traversing the AST, where alias refs will work really well, and only fall back to getTypeAtLocation for complex/generic types.

I would be quite interested in taking on this task (I already maintain typescript-json-validator which is a wrapper around typescript-json-schema), but didn't want to before because I don't want to either:

  1. I don't want to contribute to there being "yet another fork".
  2. I don't think the AST based approach is likely to bare a lot of fruit.

Having said that, I have also started work on typeconvert, which aims to do type inference on babel ASTs to convert between TypeScript and Flow (and generate documentation, JSON Schema etc.) Unfortunately it's nowhere near ready for release yet though as I keep realising I've made a mistake and need to fundamentally refactor.

from ts-json-schema-generator.

domoritz avatar domoritz commented on July 18, 2024

Thank you for your message. I agree that a hybrid approach might work well but it's hard to say until we have a working implementation. For me personally, my concern is that I can generate schemas for Vega-Lite. Maybe you can use that as a test case for another implementation of a hybrid schema generator?

from ts-json-schema-generator.

HoldYourWaffle avatar HoldYourWaffle commented on July 18, 2024

I think a hybrid approach could definitely be a good solution. I don't see how this would contribute to the 'yet another fork problem', since I think this can just be integrated into one of the existing ones (preferably this one of course)? I don't know much about the internals of either code base though so I might be completely wrong here.

Supporting them requires doing type inference that is on a par with TypeScript itself

Perhaps it's possible to reuse the logic TypeScript itself is using? Visual Studio Code has (in my experience) near perfect "reflection" on TypeScript code so it should be possible. I think VS Code uses something like this, maybe that's something worth looking into? Again I'm really not qualified to make any well founded argument about this but I try to help as much as I can.

from ts-json-schema-generator.

sparebytes avatar sparebytes commented on July 18, 2024

Maybe we can do something like this: ast -> io-ts -> json schema.
io-ts does a good job representing types at runtime. Looks like their v2.0 milestone includes generating json schema.

Excuse me if this is way off base, I can't read the whole thread right this minute.

from ts-json-schema-generator.

codler avatar codler commented on July 18, 2024

I was about to start try out Typescript to JSON Schema and then I saw this thread. I am now confused which library I should use. Where are we at today and which one do you recommend to use?

from ts-json-schema-generator.

domoritz avatar domoritz commented on July 18, 2024

@codler I added a tldr to the first message. Does that help?

from ts-json-schema-generator.

nonara avatar nonara commented on July 18, 2024

Hi all. Could anyone tell me exactly what does not work with the getTypeAtLocation version?

I've written a working ObjectLiteral parser and have AsExpression in the works, but as I do, it just seems more and more like the wrong route to go in not using the built in TypeChecker functionality, especially as we start dealing with objects, etc. The primary reason is that the language changes and evolves. Take, for example, the addition of 'as const'. This drastically impacts the derived type.

This means as things change or are added, we're constantly recreating TypeScript's work, which is both inefficient and bound to encounter bugs, as we have to try to determine and replicate all of the nuance each time, which means digging through source code.

I completely understand @domoritz 's point of not having the time to rewrite things. With that said, my situation is currently the opposite. I'm full-time writing a library (TS extension via ts-patch) which allows type validation during runtime via schema generation and ajv. The intent isn't to be just yet another fork, but rather, as some have suggested, a hybridized version which combines all of what we know.

To that end, I've already slimmed down and optimized the majority of this library and am close to completing it. What would be very helpful for me is to have the exact cases in which the former library which uses getTypeAtLocation fails to work. Also, if anyone would like to join me, please let me know. I'd be glad to collaborate with you on what I believe will be an incredibly useful TS extension!

Thanks, again, to all who've worked on this and committed their time so far!

from ts-json-schema-generator.

domoritz avatar domoritz commented on July 18, 2024

Hi @nonara. Thank you for your comment. Your assessment of the situation is accurate. It's a lot of work to maintain this library and keep up with the development of TypeScript. Moreover, we have bugs because I haven't kept up with the development or misinterpreted the AST.

For a bit of context, I have worked on both https://github.com/YousefED/typescript-json-schema and https://github.com/vega/ts-json-schema-generator. While the former was a lot simpler, I struggled to get aliases right and put effort into the latter instead.

My use case is to generate a JSON schema for Vega-Lite. It is super important that the schema is readable and that means that type names from TypeScript appear in the JSON schema as well. We use the names of TS types both for our documentation and also to generate Altair. ts-json-schema-generator works quite well for my use case but I have run into bugs from time to time (as well as bad aliases).

All of this is to say that I am not dogmatic about not using getTypeAtLocation. I personally just haven't figure out a way to make it work well with aliases. However, the last time I really looked into this was a few years ago and I didn't know much about the TypeScript compiler then. I would be super thrilled to use a simpler approach that uses type inference provided by the compiler rather than walking over the AST (which is difficult and error prone).

from ts-json-schema-generator.

cspotcode avatar cspotcode commented on July 18, 2024

Is it possible for a maintainer to post a minimal example of the issue with aliases? I posted two examples below; are either or both of those cases accurate?

I've worked with the compiler APIs before, so I'm not a beginner, and I don't mind if the explanation dives into compiler internals.


Example 1

export interface SomeInterface {
    foo: string;
    bar: number;
}
type SomeAliasIWantToAppearInSchema = SomeInterface;

getTypeAtLocation cannot see SomeAliasIWantToAppearInSchema; it can only see SomeInterface. The schema is incomplete.


Example 2

export interface SomeInterfaceA {
    foo: AliasedType;
}
export interface SomeInterfaceB {
    bar: AliasedType;
}
type AliasedType = {baz: 'biff'};

getTypeAtLocation says that bar is of type {baz: 'biff'}. It does not give any indication that there's an alias, so the aliased type is inlined in the schema twice. This causes undesirable duplication in the schema.

from ts-json-schema-generator.

domoritz avatar domoritz commented on July 18, 2024

If you want to see more test cases, I have accumulated a bunch of them over the years at https://github.com/vega/ts-json-schema-generator/tree/master/test/valid-data.

Your examples are good and they show that ts-json-schema-generator works well here. I would be happy about any solution involving getTypeAtLocation that can replicate this behavior.

Here is example 1 in ts-json-schema-generator

export interface SomeInterface {
    foo: string;
    bar: number;
}
export type SomeAliasIWantToAppearInSchema = SomeInterface;
$ ts-node ts-json-schema-generator.ts -p test.ts -t SomeAliasIWantToAppearInSchema
{
  "$ref": "#/definitions/SomeAliasIWantToAppearInSchema",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "definitions": {
    "SomeAliasIWantToAppearInSchema": {
      "$ref": "#/definitions/SomeInterface"
    },
    "SomeInterface": {
      "additionalProperties": false,
      "properties": {
        "bar": {
          "type": "number"
        },
        "foo": {
          "type": "string"
        }
      },
      "required": [
        "foo",
        "bar"
      ],
      "type": "object"
    }
  }
}

And example 2 with AliasedType exported (which is probably what you want and it avoids the issue @cspotcode pointed at).

export interface SomeInterfaceA {
    foo: AliasedType;
}
export interface SomeInterfaceB {
    bar: AliasedType;
}
export type AliasedType = { baz: "biff" };
$ ts-node ts-json-schema-generator.ts -p test.ts
{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "definitions": {
    "AliasedType": {
      "additionalProperties": false,
      "properties": {
        "baz": {
          "enum": [
            "biff"
          ],
          "type": "string"
        }
      },
      "required": [
        "baz"
      ],
      "type": "object"
    },
    "SomeInterfaceA": {
      "additionalProperties": false,
      "properties": {
        "foo": {
          "$ref": "#/definitions/AliasedType"
        }
      },
      "required": [
        "foo"
      ],
      "type": "object"
    },
    "SomeInterfaceB": {
      "additionalProperties": false,
      "properties": {
        "bar": {
          "$ref": "#/definitions/AliasedType"
        }
      },
      "required": [
        "bar"
      ],
      "type": "object"
    }
  }
}

and with AliasedType hidden (not super important to support IMHO)

export interface SomeInterfaceA {
    foo: AliasedType;
}
export interface SomeInterfaceB {
    bar: AliasedType;
}
type AliasedType = { baz: "biff" };

you get

$ ts-node ts-json-schema-generator.ts -p test.ts
{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "definitions": {
    "SomeInterfaceA": {
      "additionalProperties": false,
      "properties": {
        "foo": {
          "additionalProperties": false,
          "properties": {
            "baz": {
              "enum": [
                "biff"
              ],
              "type": "string"
            }
          },
          "required": [
            "baz"
          ],
          "type": "object"
        }
      },
      "required": [
        "foo"
      ],
      "type": "object"
    },
    "SomeInterfaceB": {
      "additionalProperties": false,
      "properties": {
        "bar": {
          "additionalProperties": false,
          "properties": {
            "baz": {
              "enum": [
                "biff"
              ],
              "type": "string"
            }
          },
          "required": [
            "baz"
          ],
          "type": "object"
        }
      },
      "required": [
        "bar"
      ],
      "type": "object"
    }
  }
}

from ts-json-schema-generator.

nonara avatar nonara commented on July 18, 2024

Thank you both for your responses! @domoritz Is it safe to say, if we can get passes on everything in valid-types, it should work?

from ts-json-schema-generator.

domoritz avatar domoritz commented on July 18, 2024

I think it's a pretty comprehensive test set. I'm also open to changing some of the outputs (e.g. always output aliases even if they are not exported).

from ts-json-schema-generator.

cspotcode avatar cspotcode commented on July 18, 2024

I played around with the compiler API last night. I tried extracting types solely from the type system, avoiding the AST entirely.

I created a program, then grabbed the exports for a given source file:

sourceFileNode = program.getSourceFileByPath()
sourceFileSymbol = typeChecker.getSymbolAtLocation(sourceFileNode)
exportsSymbols = typeChecker.getExportsOfModule(moduleSymbol)

Then I use getDeclaredTypeOfSymbol() whenever possible. For property declarations on an interface, getTypeOfSymbolAtLocation(symbol, sourceFileNode) is necessary since property declarations are a "value" symbol.

interface Foo {prop: MyAlias} worked correctly and gave me a reference to the MyAlias type / symbol.

But interface Foo {prop: Omit<Whatever, never>} did not work as expected. It gave me a Pick type, skipping over Omit. I guess if an alias points at another alias, the first is skipped over.


What if a schema generator was built as follows:

a) Extract type info by navigating symbols / types exclusively, without looking at the AST.
b) Have a list of special-cases where the AST is consulted to see if it's a ts.isTypeReferenceNode. (alias) If so, convert the TypeReferenceNode to a "$ref". If not, fallback to (a).

If (b) ever fails, the fallback is always (a). This means (b) only needs to deal with TypeReferenceNodes, because they might be aliases. Everything else is handled by the inference-driven (a). (b) can initially be a no-op, then built up over time.

If new TypeScript features are released, schemas will still be generated correctly. Updates to (b) might be required to achieve ideal alias behavior, but these updates are not necessarily blocking users from upgrading their TypeScript compiler.

from ts-json-schema-generator.

domoritz avatar domoritz commented on July 18, 2024

Thank you for describing your approach. This sounds a lot more reliable and not too complicated. Let me know when you take a stab at it. I’m happy to even retire or replace this library with your code when it works for Vega-Lite.

from ts-json-schema-generator.

HoldYourWaffle avatar HoldYourWaffle commented on July 18, 2024

Glad to see this library getting some love again, even though I don't fully understand the discussion anymore 🙃

from ts-json-schema-generator.

cspotcode avatar cspotcode commented on July 18, 2024

What is a good place to ask about nitty gritty compiler internals? Is there a subreddit, a Slack community, or something where people like us congregate?

It's tough to ask questions about compiler internals because most people are users of typescript, and they never get into the compiler APIs.

For example, my question right now is if the public API lets me differentiate between true and false boolean literal types.

I'd also like to share what I've learned but don't have a great place to put it.

from ts-json-schema-generator.

nonara avatar nonara commented on July 18, 2024

Just discovered discord a few days ago. Compiler chat is great. Actual MSFT employees on the TS team. The other is https://gitter.im/Microsoft/TypeScript - but mentioning compiler usually gets crickets.

@cspotcode - Shoot me a PM on Discord when you have a second to chat. I believe I've found a much simpler approach. I'd love to chat a bit and see if you think it's viable and get an idea at what your use-case is.

from ts-json-schema-generator.

cspotcode avatar cspotcode commented on July 18, 2024

Is there anything I'm missing?

Nice! Where's the code? It's tough to assess without throwing it at my team's codebase and seeing what comes out.

Flags

Is there an example of where we set flags in a config file? Do they go in our tsconfig, inside the plugin object?

from ts-json-schema-generator.

nonara avatar nonara commented on July 18, 2024

Nice! Where's the code? It's tough to assess without throwing it at my team's codebase and seeing what comes out.

Not live yet. The idea was to open the floor up on the proposed standard, first. I will post the source shortly, when it's ready to use. My main goal is to make sure that the wider audience is good with the extensibility and proposed default behaviours.

I'm hoping that we can end the fork-mania, so I'm choosing to wait until it's complete to post the source.

Flags

I meant to re-label those as options.

After some thought, and because we're allowing for hooks (which will be in the form of functions), I propose that we load the options from a file in package root, schema.config.ts. By using TS, we can also allow for intellisense completion.

An example file might look like this:

export = <SchemaOptions> {
  noFileIds: true,
  hooks: {
    onTypeParse (schemaType: SchemaType) => {
      // Route interfaces to a special path
      if (schemaType.kind === 'interface') return { ...schemaType.output, propertyPath: 'Interfaces' }
      // Don't include Types that start with 'Secret'
      if (/^Secret/.test(schemaType.name)) return false;
    }
  }
}

from ts-json-schema-generator.

maneetgoyal avatar maneetgoyal commented on July 18, 2024

Thanks @nonara for the nice and detailed background on the upcoming tool. Looking forward.

Is there anything I'm missing?

In my project, we are using a monorepo setup, so it will be great if the extends property in tsconfig is respected. It may be the case already but just wanted to be sure. There was one issue reported too (YousefED/typescript-json-schema#326) by @cspotcode.

from ts-json-schema-generator.

nonara avatar nonara commented on July 18, 2024

@maneetgoyal Thanks for that suggestion! Normal plugin use would use what was loaded by tsc, but programmatic was lacking in that respect.

Looks like I can use getParsedCommandLineOfConfigFile in the TS compiler to make sure I cover those bases.

from ts-json-schema-generator.

nonara avatar nonara commented on July 18, 2024

While I'm here, here's a quick update on the status:

The holiday season slowed me down quite a bit. I've also been caring for a family member after a surgery which meant loss of her dominant arm for a few months. <right hand man pun here />

I've also slowed down intentionally a bit, just to make sure I'm really thinking through any possible eventualities that haven't been covered in previous implementations as well as using the compiler API to the fullest. Basically, that's meant a lot of time studying TS compiler source and stepping through code to understand the intricacies of Type, Symbols, flags, etc.

The good news is, I'm into the final stretch. I'm just now wrapping up the last bits of the generator. It should be passing all of the valid-types tests and more in the next couple of days.

I'll be out for the week of the 12 - 18th, but with the holidays over, I should be able to dedicate the majority of my time to wrapping up. Barring any complication, I hope to have it up for everyone to test with their code-base around the end of the month.

Happy holidays!

from ts-json-schema-generator.

domoritz avatar domoritz commented on July 18, 2024

@nonara Is there a preview version we can play with? I'd love to take your schema generator for a spin and provide feedback.

from ts-json-schema-generator.

nonara avatar nonara commented on July 18, 2024

@domoritz Thanks! It's just about ready to pre-release for review. Just wrapping up a few last bits, now. I've been able to be back on it full-time, and it's very close.

Looking forward to the input!

While I have you, I had a question regarding generics. I recall seeing an issue or two raised on the naming convention. Was there a consensus on naming?

I'm looking at either:

  1. MyGeneric<MyType, true>
  2. MyGeneric<T=MyType, TIsUnique=true>

Given type MyGeneric<T, TIsUnique extends Boolean> = ...

I think having the var names might have some value and am inclined to go with option 2. Any thoughts/objections?

Also, IIRC, someone raised an issue with the < > symbols. Do we need to look at a different format?

from ts-json-schema-generator.

domoritz avatar domoritz commented on July 18, 2024

That's exciting @nonara. Do you support all test cases in https://github.com/vega/ts-json-schema-generator/tree/master/test? If not, what are the differences?

from ts-json-schema-generator.

domoritz avatar domoritz commented on July 18, 2024

Awesome. I am excited to take your tool for a spin.

from ts-json-schema-generator.

domoritz avatar domoritz commented on July 18, 2024

@nonara Do you have a timeline for when I can take your tool for a spin? I have a case this generator does not support right now and I don't want to spend more time on it if there will be an alternative soon.

export const Foo = {
    x: "x",
    y: "y",
};

export type MyType = keyof typeof Foo;

from ts-json-schema-generator.

HoldYourWaffle avatar HoldYourWaffle commented on July 18, 2024

That's so great to hear!

I'm currently in the process of starting up a new project which could massively benefit from this, and I'd love to take it for a spin in the next few weeks and help work out any kinks if necessary.

from ts-json-schema-generator.

domoritz avatar domoritz commented on July 18, 2024

@nonara Sounds good. Do you think it would make sense to put a work in progress version online so that you can get feedback on it?

Also, would you be interested in having us host the project in the Vega org for added visibility? I could create a repo and give you admin access. If you want to do that, what is a good name for the repo?

from ts-json-schema-generator.

HoldYourWaffle avatar HoldYourWaffle commented on July 18, 2024

Speaking of visibility, is there a way to point people that stumble upon the "legacy" modules to the "new" one?

from ts-json-schema-generator.

domoritz avatar domoritz commented on July 18, 2024

Which ones, in particular, are you talking about? I put a pinned issue in the one I have been working on.

Screen Shot 2020-04-06 at 13 40 34

from ts-json-schema-generator.

nonara avatar nonara commented on July 18, 2024

@HoldYourWaffle Assuming everyone is happy and we've decided to move forward with the new one, I believe GitHub allows some forms of deprecating / forwarding. Am I right on that?

Worst case, we can always put notices up.

from ts-json-schema-generator.

timini avatar timini commented on July 18, 2024

Watching with baited breath..

Have you guys see https://github.com/google/intermock also?

from ts-json-schema-generator.

timini avatar timini commented on July 18, 2024

Great news, keep up the good work!

from ts-json-schema-generator.

nonara avatar nonara commented on July 18, 2024

Hi folks! Sorry it's been awhile since the last update. I had to spend some time doing maintenance and updates on another library.

That said, there are updates on this front also.

Declaration merging is integrated!

New nodes are created and everything is integrated into the parser. Modifying for symbol parsing brought up some interesting scenarios, but all is complete now.

Extras

  • readonly modifier and ReadonlyArray support is added

Updates

One thing that's bothered me for awhile has been the AST Nodes. They weren't necessarily true AST nodes, in the sense that some represented TypeScript and some were more toward pure JSON. This and the nodes' flags bitwise enums were not well thought out. (redundant in some cases, etc)

In light of finishing all possible Nodes for TS, I've been able to re-think this a bit.

This sort of mashup between the two was a bit confusing and also limiting, in a sense. The typeFlags was overcrowded and redundant, replicating information already easily able to be discerned by the node Kind or origin.

Ultimately, I've arrived at the realization that there is more value in separation between the AST layer and the TypeScript parser. In other words, the AST should be able to represent type information in a language-agnostic manner whenever possible in terms of Nodes, and the typeFlags should be represent an entirely language-agnostic way of identifying the type it represents.

The end result is the start of a scalable, universal AST for Types that compile to JSON. This opens up the possibility to add more 'parsers' for more languages as we move forward.

Updated Structure (de-coupled AST)

  • A SchemaNode for every type (some will be universal like FunctionNode, some will be specialized per-language like TypeAliasNode)
  • typeFlags bitwise enum represents language-agnostic underlying type (ie. primitive, class, interface, etc) and possibly modifiers (ie. paramerized, mixed)

Note: Any non-exposed TS specific type info (like whether the origin for TrueSchemaNode is any or unknown) is still accessible through the origin property, which is the originating ts.Symbol.

Left TODO

The changes aren't major from a code perspective. Mostly a few semantic changes in enums.

  1. Update flags enum
    • Primitives and Literals get PrimitiveKind and LiteralKind enums (frees up much needed space in typeFlags
  2. De-couple SchemaNode from direct links to anything TS-parser specific (mainly only the TypeWalkerContext, which has little impact)
  3. Initial test to ensure all possible nodes are created and render properly
  4. Finish implementing the logic for ParseOptions (deferring most to after releasing src)
  5. Implement Rules (deferring to after releasing src)

from ts-json-schema-generator.

domoritz avatar domoritz commented on July 18, 2024

Nice. Do these changes also mean that I can use typescript to generate something other than json schema? For example, we generate Altair from the generated Vega-Lite json schema right now but I wonder whether we can go directly from ts to python interfaces.

from ts-json-schema-generator.

domoritz avatar domoritz commented on July 18, 2024

@nonara when do you plan to upload the code to support JSON schema generation? I would love to take it for a spin.

from ts-json-schema-generator.

nonara avatar nonara commented on July 18, 2024

@nonara when do you plan to upload the code to support JSON schema generation? I would love to take it for a spin.

New AST + tests are done. I'm migrating the TS type parser in the next work session. That will give you a look at the generated AST for all TS types. After that, I'll port the JSON compiler, then you'll be able to see the full transition!

To allow people to start playing with it sooner, I'll be migrating all of the parse and compile options (tags, etc) after.

from ts-json-schema-generator.

cspotcode avatar cspotcode commented on July 18, 2024

I hope this is ok to ask here. We're using schema generation and it doesn't support the unknown type. I assume schemas should treat it identically to any.

Should I send a PR here to add support? Or to the crosstype project?

from ts-json-schema-generator.

cspotcode avatar cspotcode commented on July 18, 2024

If anyone would like to discuss this realtime, I am usually hanging out on the TypeScript Discord. We can create a dedicated channel or thread for JSON schema generation.

https://discord.gg/typescript

from ts-json-schema-generator.

cspotcode avatar cspotcode commented on July 18, 2024

Agreed, and realtime chat tends to have a lot of repetition, because newcomers cannot easily see what's already been discussed.

from ts-json-schema-generator.

cspotcode avatar cspotcode commented on July 18, 2024

I'm reading through the chatlog, attempting to make a short-list of issues.

There was debate between 2x approaches for getting types: using the typechecker API, or parsing TS ASTs. The typechecker API made it difficult to preserve type aliases.

Does anyone know if this has been addressed by recent changes in TS 4.2 which preserve type aliases in more situations? https://www.typescriptlang.org/docs/handbook/release-notes/typescript-4-2.html#smarter-type-alias-preservation

from ts-json-schema-generator.

cspotcode avatar cspotcode commented on July 18, 2024

Is there a spec for the property names used in the "definitions" object? For example, supposing someone wants to create a schema that includes every type exported from all-schemas.ts, are there some guarantees that the schema for named export Foo will be found at "definitions" "Foo"?

from ts-json-schema-generator.

cspotcode avatar cspotcode commented on July 18, 2024

Thoughts on using typedoc's type extraction engine to power schema generation? Seems like typedoc has more active maintenance and has better kept pace with typescript's type system, using the typechecker for more of the extraction work than this library. We are fairly active on Discord.

from ts-json-schema-generator.

domoritz avatar domoritz commented on July 18, 2024

I don't care what we use under the hood as long as we can generate a good schema. I love the idea of using something well maintained as the basis.

What's the benefit over using the typescript compiler?

from ts-json-schema-generator.

domoritz avatar domoritz commented on July 18, 2024

That all sounds fantastic. Do you want to try making a POC?

from ts-json-schema-generator.

cspotcode avatar cspotcode commented on July 18, 2024

I want to ask about this specifically, since it may be a breaking change if we go with typedoc:

Typedoc handles exports as you would expect, as opposed to this library which exposes internal types, cannot handle multiple types with the same name, and does not handle alias exports.

This library exposes non-exported types in the schema, using their internal, non-exported names. But I'm not sure why that is necessary, and I'm not sure it is compatible with typedoc's extracted reflections. Do you know if the requirement to expose internals is documented anywhere? Do you know if the identifiers assigned to "definitions" is specced anywhere?

I'd like to propose a simpler way to tell the schema extractor which schemas to extract:
The user writes a single file, schemas.ts, which exports one or more types using named exports. The schema generator is given this file as an entrypoint and will emit a single schema containing a "definition" for every type in schemas.ts, with a name matching the named export. All other "definition"s will have verbose, fully-qualified names that include their sourcefile's path to avoid name collisions. If the user wants a top ref, they add a default export.

This pushes configuration into the TypeScript language. schemas.ts can re-export types from elsewhere in a codebase, so it gives you full control over the included schemas. There is an intuitive one-to-one between identifiers in TS and "definition" names in the schema. And name collisions are handled by the TS language.

from ts-json-schema-generator.

domoritz avatar domoritz commented on July 18, 2024

I think it can be nice to use internal aliases to name definitions in the schema if we need them (e.g. when we have a recursive data structure). I don't think we usually expose internal aliases otherwise, do we?

I'd like to propose a simpler way to tell the schema extractor which schemas to extract:
The user writes a single file, schemas.ts, which exports one or more types using named exports. The schema generator is given this file as an entrypoint and will emit a single schema containing a "definition" for every type in schemas.ts, with a name matching the named export. All other "definition"s will have verbose, fully-qualified names that include their sourcefile's path to avoid name collisions. If the user wants a top ref, they add a default export.

Where would they add a default export? In the source file or in schemas.ts?

I do like that we would avoid the challenge of duplicate types and make it very explicit what types get exported. However, it's not how typedoc works and some users may get confused why they don't see some types. Maybe it's okay if people already have the relevant types exported in their index.ts but I worry that some types might get very deep.

My only use case for this library right now is https://github.com/vega/vega-lite and it would be a lot of work to export all the relevant types again. Do you have a suggestion for how I could reduce the necessary work?

from ts-json-schema-generator.

cspotcode avatar cspotcode commented on July 18, 2024

I don't think we usually expose internal aliases otherwise, do we?

I'm thinking of this flag:

-e, --expose <all|none|export>
    all: Create shared $ref definitions for all types.
    none: Do not create shared $ref definitions.
    export (default): Create shared $ref definitions only for exported types (not tagged as `@internal`).

It's tough to tell how that flag is meant to behave, but does it fail if 2x non-exported types in different files have the same identifier? I'm fine with creating shared ref definitions, but the tool should ensure they do not have naming conflicts, and the names do not need to follow any particular rules. They can be made human-readable, but there do not need to be any guarantees about their name. If the user wants a type to have a guaranteed "definition" name, they can achieve that by exporting it from schemas.ts

Where would they add a default export? In the source file or in schemas.ts?

In schemas.ts

from ts-json-schema-generator.

domoritz avatar domoritz commented on July 18, 2024

Where would they add a default export? In the source file or in schemas.ts?

In schemas.ts

I think you meant a normal export then, not a default expert (as there can be only one). Right?

from ts-json-schema-generator.

cspotcode avatar cspotcode commented on July 18, 2024

The idea is that the root $ref is determined by the default export, if it exists. Any other named exports will be guaranteed to reside at their name in "definitions". For example:

export {Foo as default};
export {Bar, Baz, Biff};

In the emitted JSON schema:

The root "$ref" is guaranteed to refer to Foo. You can use the emitted JSON schema as-is to validate that an object matches Foo.

"definitions": {"Bar" is guaranteed to be the definition for Bar. There will likely be many other definitions in the emitted JSON, but their names are generated by the tool and may be more verbose, may include filenames encoded in some form, and naming collisions will be avoided.

This means that you can trivially modify the root $ref to use a single schema to validate against Bar, Baz, or Biff because their definitions are guaranteed to reside at those names in "definitions"

from ts-json-schema-generator.

domoritz avatar domoritz commented on July 18, 2024

There will likely be many other definitions in the emitted JSON, but their names are generated by the tool and may be more verbose, may include filenames encoded in some form, and naming collisions will be avoided.

Oh, I missed the part about other definitions being included potentially. I see. So definitions that are exported will have a predictable name and any other aliases or types could have arbitrary names (e.g. to include the path). That's a good idea. I think it makes sense overall.

from ts-json-schema-generator.

maneetgoyal avatar maneetgoyal commented on July 18, 2024

Hi all, I have been using Zod for input validation in some of my projects. Since, the discussion here seems to be related to producing JSON schema from TS definitions, thought of sharing ts-to-zod and zod-to-json-schema libraries. Combining both, I think we can do TS --> Zod --> JSON Schema. Would love to know what the developers on this forum think about the limitations of these tools.

from ts-json-schema-generator.

domoritz avatar domoritz commented on July 18, 2024

That's interesting. I wonder how flexible Zod is, though. Does any information get lost in the intermediate representation?

from ts-json-schema-generator.

cspotcode avatar cspotcode commented on July 18, 2024

My immediately thought: does it support customization of the schema with JSDoc @tags? Currently we can customize the JSON schema:

/** @minimum 10 */ foo: number; to set a JSON Schema "minimum"
/** @TJS-default {10} */ foo: number; to set JSON schema default
/** @TJS-type {integer} */ foo: number; to override the schema type to integer instead of number

from ts-json-schema-generator.

maneetgoyal avatar maneetgoyal commented on July 18, 2024

Does any information get lost in the intermediate representation?

Running some tests currently. So far, it seems like it can't handle circular dependencies in type definitions. Getting the following warning while running npx ts-to-zod some-src.ts some-dest.ts:

›   Warning: Some schemas can't be generated due to circular dependencies:

from ts-json-schema-generator.

maneetgoyal avatar maneetgoyal commented on July 18, 2024

does it support customization of the schema with JSDoc tags?

As per their docs, they support only 6 of the JSDOC keywords.

From their docs:

// source.ts
export interface HeroContact {
  /**
   * The email of the hero.
   *
   * @format email
   */
  email: string;

  /**
   * The name of the hero.
   *
   * @minLength 2
   * @maxLength 50
   */
  name: string;

  /**
   * The phone number of the hero.
   *
   * @pattern ^([+]?d{1,2}[-s]?|)d{3}[-s]?d{3}[-s]?d{4}$
   */
  phoneNumber: string;

  /**
   * Does the hero has super power?
   *
   * @default true
   */
  hasSuperPower?: boolean;

  /**
   * The age of the hero
   *
   * @minimum 0
   * @maximum 500
   */
  age: number;
}

// output.ts
export const heroContactSchema = z.object({
  /**
   * The email of the hero.
   *
   * @format email
   */
  email: z.string().email(),

  /**
   * The name of the hero.
   *
   * @minLength 2
   * @maxLength 50
   */
  name: z.string().min(2).max(50),

  /**
   * The phone number of the hero.
   *
   * @pattern ^([+]?d{1,2}[-s]?|)d{3}[-s]?d{3}[-s]?d{4}$
   */
  phoneNumber: z.string().regex(/^([+]?d{1,2}[-s]?|)d{3}[-s]?d{3}[-s]?d{4}$/),

  /**
   * Does the hero has super power?
   *
   * @default true
   */
  hasSuperPower: z.boolean().default(true),

  /**
   * The age of the hero
   *
   * @minimum 0
   * @maximum 500
   */
  age: z.number().min(0).max(500),
});

from ts-json-schema-generator.

domoritz avatar domoritz commented on July 18, 2024

Can you explain why?

from ts-json-schema-generator.

M-jerez avatar M-jerez commented on July 18, 2024

@domoritz
Decorators are the official way to add metadata to typescript and es6. jsDoc annotations are a workaround used before decorators existed, jsDoc annotations should be used only/mostly for documentation.

Also most of the libraries related to data mapping , orm etc, are using decorators now.
Working with decorators might also be easier more flexible and extensible than jsDoc annotations.

from ts-json-schema-generator.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.