restler grammar should support fuzzable elements for format: date-span and uri about restler-fuzzer HOT 8 OPEN

I will open a separate bug for these:

One is datetime from schema to produce correct ISO format in dict.json and the second is to have this format create a correct datetime in grammar.py.

from restler-fuzzer.

Hello @larsivi, thanks for bringing this up. We have added 'date' recently, it's included in 8.1.0 (yep, adding it is a bit more involved at this point than just adding the case to the code... we're looking into to improve the implementation to avoid having to extend the engine for every type of format, see issue #369. If you read the description in that issue, let us know if you have feedback.)

Thanks!

Marina

from restler-fuzzer.

Is there any property to use for DateTime strings such that their name from/to that is from the swagger translated to proper date-time string format?

For example, OpenAPI YAML:

        - in: query
          name: to
          description: You can find things that were raised before this time.
          docs-review: true
          schema:
            type: string
            format: datetime

instead of being translated to fuzzstring;

  "restler_fuzzable_string": [
    "fuzzstring"
  ],

Could it be translated to:

  "restler_fuzzable_datetime_string": [
    "2021-03-09T16:01:13Z"
  ],

And what is the short-term solution to bypass 400 errors b/c of the above problem?

from restler-fuzzer.

Hello @mbuki,

restler_fuzzable_datetime can be used in the dictionary. Are you saying that in the grammar.py these are restler_fuzzable_string? If yes, this looks like a bug - could you please copy/paste the grammar.py snippet you get for the above schema?

As a workaround, you can use restler_custom_payload, but values need to be specified separately for each of the parameters. So, in the above:

restler_custom_payload {
  "to": ["2021-03-09T16:01:13Z", "2021-03-10T16:01:13Z"],
  "from": ["2021-03-08T16:01:13Z", "2021-03-11T16:01:13Z"]
}

Another less precise/verbose workaround would be to enter these values into the list of restler_fuzzable_string values. This will cause more parameter combinations to be tried, but eventually RESTler should plug in those values into the query parameter.

from restler-fuzzer.

Yes, in the grammar.py these are restler_fuzzable_string.

primitives.restler_static_string("from="),
primitives.restler_fuzzable_string("fuzzstring", quoted=False),
primitives.restler_static_string("&"),
primitives.restler_static_string("to="),
primitives.restler_fuzzable_string("fuzzstring", quoted=False),
primitives.restler_static_string("&"),

There are some additional notes here, so I am explaining from the beginning.
When I run the following command from the root directory of my project where the YAML file is:

Restler compile --api_spec openapi.yaml

I get a new folder called `Compile' with all the default grammar, dict, etc. files.

Looking into that generated dict.json, I also see that the datetime property is not in the right format, i.e.,

"restler_fuzzable_datetime": [
   "6/25/2019 12:00:00 AM"
 ]

This is while the server only accepts ISO timestamps. e.g. "2021-02-09T16:54:13Z".

This is my openapi:

        - in: query
          name: from
          description: runtime after this time.
          schema:
            type: string
            format: datetime
        - in: query
          name: to
          description: runtime before this time.
          schema:
            type: string
            format: datetime

So I think issues are two here. One is datetime from schema to produce correct ISO format in dict.json and the second is to have this format create a correct datetime in grammar.py.

As a workaround, you can use restler_custom_payload, but values need to be specified separately for each of the parameters. So, in the above:

Even if I use:

restler_custom_payload {
  "to": ["2021-03-09T16:01:13Z", "2021-03-10T16:01:13Z"],
  "from": ["2021-03-08T16:01:13Z", "2021-03-11T16:01:13Z"]
}

The former issue persists, i.e., not ISO compliant.

Above all, note that our server does not need quotes around strings, which I see the quoted flag in grammar.py is set to False. So this should be fine as long as the problem from the above matters could be resolved, thanks.

from restler-fuzzer.

Thanks for these details, this is very helpful.

Could you please clarify, when you change the dictionary (e.g. to add a restler_custom_payload section), are you re-compiling with

Restler compile config.json

where the config.json points to your new, updated dictionary?

Whenever you add new properties in the dictionary, you need to recompile because RESTler may need to update the grammar. Specifically, when you use restler_custom_payload, RESTler will replaced the fuzzable string grammar elements with restler_custom_payload("to") and use the values in the restler_custom_payload part of the dictionary.

We will update the docs to make this more clear.

from restler-fuzzer.

Hi @marina-p,

Yes, that's correct. When I change dict.json in Compile folder I do the following. I go to the Compile folder and run:
Restler compile config.json
That generates a new grammar.py based on changes in dict.json. My config.json looks like this:

{
  "SwaggerSpecFilePath": [
    "/root/openapi.yaml"
  ],
  "GrammarOutputDirectoryPath": "/root/Compile",
  "GrammarInputFilePath": "/root/Compile/grammar.json",
  "CustomDictionaryFilePath": "/root/Compile/dict.json",
  "EngineSettingsFilePath":  "/root/Compile/restler_user_settings.json",
  "IncludeOptionalParameters": true,
  "UseBodyExamples": true,
  "UseQueryExamples": true,
  "DiscoverExamples": true,
  "ExamplesDirectory": "/root/Compile/Example",
  "DataFuzzing": true,
  "ReadOnlyFuzz": false,
  "ResolveQueryDependencies": true,
  "ResolveBodyDependencies": true,
  "UseRefreshableToken": true,
  "AllowGetProducers": false
}

My dict.json looks like this:

{
  "restler_fuzzable_string": [
    "fuzzstring"
  ],
  "restler_fuzzable_string_unquoted": [],
  "restler_fuzzable_datetime": [
    "6/25/2019 12:00:00 AM"
  ],
  "restler_fuzzable_datetime_unquoted": [],
  "restler_fuzzable_uuid4": [
    "903bcc44-30cf-4ea7-968a-d9d0da7c072f"
  ],
  "restler_fuzzable_uuid4_unquoted": [],
  "restler_fuzzable_int": [
    "0",
    "1"
  ],
  "restler_fuzzable_number": [
    "0.1",
    "1.2"
  ],
  "restler_fuzzable_bool": [
    "true"
  ],
  "restler_fuzzable_object": [
    "{}"
  ],
  "restler_custom_payload": {
  "to": ["2021-03-09T16:01:13Z", "2021-03-10T16:01:13Z"],
  "from": ["2021-03-08T16:01:13Z", "2021-03-11T16:01:13Z"]
  },
  "restler_custom_payload_unquoted": {},
  "restler_custom_payload_uuid4_suffix": {}
}

And so when I run the compile command stated above with the settings described, I still get the following:

    ....
    primitives.restler_static_string("&"),
    primitives.restler_static_string("from="),
    primitives.restler_fuzzable_string("fuzzstring", quoted=False),
    primitives.restler_static_string("&"),
    primitives.restler_static_string("to="),
    primitives.restler_fuzzable_string("fuzzstring", quoted=False),
    primitives.restler_static_string("&"),
    primitives.restler_static_string("sort="),
    primitives.restler_static_string("["),
    primitives.restler_fuzzable_string("fuzzstring", quoted=True),
    primitives.restler_static_string("]"),
    primitives.restler_static_string("&"),
    ....

I see 4 issues here:

• The compiler does not create ISO format for datetime properties seen in the openapi. see the "restler_fuzzable_datetime" field above.
• The compiler does not take "restler_fuzzable_datetime" values for constructing datetime formats in the grammar.py.
• The compiler does not pick the custom payload for datetime and keeps seeing datetime by its type which is a string and so places fuzzstrings in grammar.py for it. This results in 400 error responses by the server.
• Wherever we have

       style: form
       explode: false

in the open api for object properties, we have to create a list without brackets, and quotations. See Query Parameters table, second row, in https://swagger.io/docs/specification/serialization/.

from restler-fuzzer.

I found this today, and it seems the compiler only supports the date-time format for strings, of the ones stated in the swagger specification. For me, this caused issues with 'date'. I'm on 7.4, but from the code this seems to still be a problem in 8. I just added 'date' to the 'date-time' case in the swaggervisitor in my local source tree, but presumably this is not the fully correct solution (and mainly works because I know how the API I'm testing works).

from restler-fuzzer.