Concerns regarding the "scenario" of the approach outlined in the wiki about lingbm HOT 5 CLOSED

Thanks a lot for documenting your concerns. We will look into them and come back with a detailed response as soon as possible.

/cc @chengsijin0817

from lingbm.

Again, thanks for documenting your concern with the initial version of our proposal!

I understand that your concern is about the scenario captured by the benchmark. Before getting into the details, let me mention that one of the major use cases we have in mind for this scenario are public GraphQL APIs such as the Github API. Hereafter, I refer to such APIs as public query APIs because they provide a public (open data) interface to query the underlying dataset in a fairly flexible manner. Such flexible querying is in contrast to the more limited/restricted functionality offered by GraphQL APIs that are built to be used internally by a particular Web or mobile application; since the latter class of APIs is merely about data retrieval, I would call them app-specific data retrieval APIs. Perhaps a third class are private query APIs that provide a flexible query functionality similar to the public ones but within the security of an organization's intranet. (Please let me know if you find this classification inaccurate or if you think some important class is missing.)

Now to the problems that you mention:

1. Your approach here instead will end up testing the schema delegation layer and not the GraphQL server? Isn't this a flawed approach to testing the efficiency of a GraphQL server's implementation?

First of all, what we mean by GraphQL server implementation is the combination of everything that needs to be set up on the server side to support the scenario (i.e., to enable clients to query a legacy database in terms of a publisher-designed GraphQL schema). Therefore, if the server is built by using a system that provides an "out-of-the-box" generated schema (such as Hasura, Postgraphile, and Prisma), then there needs to be an additional schema delegation component, and what will be benchmarked is the combination of both (the "out-of-the-box" system and the delegation component). I don't see this as a flawed approach because to support the use cases captured by the benchmark scenario I would need the schema delegation component if I want to use an "out-of-the-box" system.

2. If I was writing a user-defined GraphQL schema, why would that schema expose complex subquerying, filtering, traversal, limit, offset? I would expose exactly the GraphQL schema my apps need. What is the rationale behind putting a schema delegation layer in front of an "out-of-the-box" GraphQL server?

You may want to expose query features like the ones you mention if your are defining the GraphQL schema of a public query API because you may want to give the users of that API some flexibility in how they can query (parts of) your database. You do not know what apps they are going to build on top of your data. Moreover, even an app-specific data retrieval API may enable the app to do filtering (e.g., based on user input), paging, shallow traversals.

Regarding the rationale for putting a schema delegation layer in front of a GraphQL server that provides an "out-of-the-box" generated schema, I do not think that this is such a strange idea. If I were to create a GraphQL API, I would try to reduce the amount of hand-written code and manual development effort as much as possible. In this sense, it seems to be a reasonable option to use an "out-of-the-box" system together with a schema delegation component for which I only have to specify how the different parts of my public schema map to the schema generated by the "out-of-the-box" system.

I find this approach [...] heavily biased towards a Prisma style of implementation [...]. Are there other GraphQL vendor or products that are GraphQL middleware/ORMs for authors of APIs other than Prisma and Dreamfactory, that you intend to benchmark with this suite?

First and just to be clear about it, it is of course not our intention for the benchmark to be biased towards any style of implementation.

To your question, we are still considering the options of what kinds of GraphQL server implementations we are going to use when we apply the benchmark as part of our research work on developing the benchmark. There are not only implementations on top of relational databases but also, for instance, on top of graph databases and these try to directly rewrite GraphQL queries (expressed in terms of a user-specified schema) into graph database queries. Similarly, a complete GraphQL query for a user-specified schema may be rewritten into an SQL query over an underlying relational database, which to the best of my knowledge is not supported by any system at the moment (I am happy to be proven otherwise!). In this context I should also mention that I do not consider a benchmark as something that can be used only to evaluate the state of the art but also as a means to advance the state of the art and to have a well-understood basis for playing with different ideas (e.g., along the lines of what you mention in the context of your second recommendation).

Now to your two recommendations. These are really useful! I think with only a small extension of our initial goal, we can create the benchmark to be useful for both of these aspects. To this end, we may consider a second scenario along your recommendation 1 (i.e., the scenario in which data from a legacy database has to be exposed as a read-only GraphQL API for which the GraphQL schema can be auto-generated). Then, for selected systems we may rewrite our scenario-1 queries (expressed over the GraphQL schema of the benchmark) into queries that are expressed in terms of the system-generated schemas and that are guaranteed to be semantically equivalent (i.e., produce the same result for the same version of the benchmark database). In that way, the benchmark may even be used to exactly pinpoint the overhead of schema delegation and to advance the state of the art of approaches to do schema delegation (here I can also foresee approaches that rewrite complete GraphQL queries into one another to translate from one schema into another).

Please let us know whether these points address your concerns.

from lingbm.

@coco98 do the points in my earlier comment address your concerns?

from lingbm.

@hartig It was great chatting with you :) I got a more clear understanding of the motivation.

As mentioned, I'm just jotting down the key points from our conversation pertinent to this issue:

1. For tools like hasura and postgraphile, the best way to reshape a database-schema into a user-defined schema is by using postgres views/functions
2. To make this more of a real-world test, it would be nice to include an element of authorization and access control

If you need any help with the setup, especially for hasura, or want to brainstorm please do feel free to reach out!

from lingbm.

@coco98

@hartig It was great chatting with you :)

Ditto! :-)

1. For tools like hasura and postgraphile, the best way to reshape a database-schema into a user-defined schema is by using postgres views/functions

That was news to me, but it totally makes sense. Thanks for the clarification.

2. To make this more of a real-world test, it would be nice to include an element of authorization and access control

Excellent idea! However, I would first like to have an end-to-end version of the benchmark with an initial, limited set of features. Thereafter, the benchmark can be extended with additional features, and authorization + access control is certainly very interesting to add then.

If you need any help with the setup, especially for hasura, or want to brainstorm please do feel free to reach out!

Thanks for the offer. I will keep it in mind and we will most likely come back to it.

from lingbm.