every property needs to know for sure what type it is, even if you use oneOf to add additional validation. For example, the following currently would be evaluted to an object type, when it should be a string (a few examples did this, like Kinesis Stream).

{
    "type": "object",
    "properties": {
        "Test": {
            "anyOf": [
                { "$ref": "#/definitions/SomeStringWithRegex"},
                { "$ref": "#/definitions/SomeOtherStringWithRegex"}
            ]
        }
    }
}

The following should work, though we've apparently not been able to get it working:

"anyOf": [
   { "required": ["type"]},
   { "required": ["$ref"] }
]

similar to #106 -- the tests are too large

The init command should generate a simple sample schema, so that the whole CLI can work end to end without doing anything. This will also jump start development.

The command could potentially take the resourceType so that it could auto populate that in the schema

When playing with the init wizard, I noticed that if I re-run init in the same repo and change my resource type name, the generated sources from a prior execution are left in place (not overwritten or removed) which can lead to broken build.

To repro;

uluru-cli init
a::b::c
mvn package

Builds OK.

uluru-cli init
My::Actual::Resource
mvn package

Fails, as the src from a/b/c is left in the repo. Not a biggie, just a usability enhancement.

identifiers was introduced in #47 , but I think our example schemas are using it incorrectly, since specifying multiple values is equivalent to a compound identifier.

• AWS::SQS::Queue example says: "identifiers": ["#/properties/Arn", "#/properties/QueueName", "#/properties/URL"], however a combination of these doesn't make sense, since either Arn or URL (maybe even QueueName) is enough to uniquely identify the resource (L106-L110)
• AWS::EC2::Instance doesn't have it at all
• AWS::S3::Bucket is similar to Queue, BucketName is already globally unique (L954-L958)

will make it simpler

At the moment, examples aren't validated. The result is invalid examples, e.g. this ARN (rouge dot at the end):
https://github.com/awslabs/aws-cloudformation-rpdk/blob/2f77d71eb377187dce4fcd7f48a4683efc875dc9/examples/schema/resource/aws.sqs.queue.v1.json#L9

It's a shame JSON schema doesn't do this, but we can add this.

Add logging to the event listener that listens for progress events from handlers. Also add better handling of unexpected progress event type i.e. if a posted event is not a json object. Currently we are sending an error response back, whereas it may be better to record the error on the server side

Automatically run generate when the user builds the Java project to avoid desync between schema and code. Ideally, call generate as part of the maven build step. If that isn't possible, add a pre-build step to the IntelliJ project.

The current resource generator in #104 handles simpler cases of resource generation. The below items are necessary to bring the generator to a more operable state that can handle most if not all valid resource definition schemas.

• Refs are not handled. This is fine, but needs some kind of step prior to make sure it works
• Type keywords. Keywords that can be specified in jsonschema draft 07 for primitive types should be accounted for in resource generation. #104 handles some of this but not all (e.g. minLength and maxLength is not supported for arrays).
• Assuming type is a string. We assume the schema type is a string and cannot be an array. Arrays are valid values for type keys and we need to generate values based off all of the types listed.
• String formats. jsonschema has a "format" keyword that allows for specific string formats like json pointers, domain names, email addresses, etc. that should be able to be generated by the resource generator. The logic to do this is in place, but we need to construct regular expression strategies that we can map to format names.
• Options to generate more than the required properties. Currently the generator only generates those that are required. Having an option to generate a maximal set of properties would be beneficial for testing the robustness of handlers.
• Combiners. Currently allOf and anyOf are not supported by the generator. anyOf should be a simple addition because it is handled in the same way as oneOf. allOf will require to merge the child schemas.
• Const and enum. Constant values and enum schemas are not supported currently.
• Needs to only generate a resource blob (i.e. the stuff under properties)
• Refactor generation functions for readability.
• #287

Also, some research needs to be done to see if the flattener and resource generator can be combined in some way. They are both walking the schema in a similar way, so it would be nice to remove duplicate implementations of this to lessen the amount of code to be maintained.

After a successful provision, it might be good to apply termination protection (https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-protect-stacks.html) to the stacks. This will prevent accidental deletion of the handler stacks.

This could be done by the CLI with an API call to CFN after package succeeds.

I'm not necessarily sure we should do this as this may be a valid use case; in fact, AWS::EMR::Cluster does this. However, the topic was up for debate, so I will raise an issue about it.

A SAM template is needed for the package command in the rpdk. This should be generated during either the init or generate stages, and it should contain a local file path to the jar location specified in the pom by the artifact id and output directory. This should also include sufficient roles for cloudformation to execute the lambda function

See @rjlohan's commits on PR #4. Issue to add the schema(s) and example definition(s) to data.

http://sparkjava.com/

This provides an alternative to SAM CLI for running local java compute

Both have walk functions which take in the ref path and the subschema, but the functions take the arguments in reverse order from each other. It would be good for code quality to make this consistent

additionalItems is very weird property, makes little sense, and doesn't need to be used for a resource schema.
details are here https://json-schema.org/understanding-json-schema/reference/array.html

This goes hand in hand with removing multiple schemas for the items array. An ordered array of different schema objects (and then the rest of the objects being specified using the additionalItems schema) is very confusing for both developers and customers. The best way to model this would be with key:value pairs

An additional semantic (like readOnly and writeOnly) should be expressed in the schema to describe those properties which can be mutated post-create for a resource.

Alternately, the semantic can be described as createOnly to define those properties of a resource which are write-once (or "immutable" in CloudFormation's current resource specification parlance).

NOTE: This semantic describes those properties of a resource which are documented as "Update requires: Replacement".

Right now, various errors are dumped to the console as a Python stack trace. Needs cleaning up for users. Errors should be clear, actionable and link out to detailed documentation where possible.

Each subobject should be its own definition

Currently the handler request interface calls for resourceProperties when Creating/Updating a file. For the test command, the user passes a json object of properties with no reference to the resource type into the command. E.g a for a Queue, the test command expects {"QueueName": "Something"} Changing this input to reflect the resource schema that CloudFormation accepts would be less confusing to users, so the SQS Queue input would become { "Type" : "AWS::SQS::Queue", "Properties" : { "QueueName": Something } }

To simplify resource definition authoring, we could restrict the use of sub-properties and require that sub-property objects are defined in the definitions block of the document. This would encourage re-use, avoid typos/errors on repeat sub-properties, and generally reduce nesting - making for simpler authoring experience and a consistent schema to read.

How will schemas be versioned, and how will this fit in with the package and generate steps?

When specifying "-v" on the test subcommand, the cli says that the verbosity argument is not recognized. The verbosity argument parser is being added at the wrong level and is being ignored. Need to add the argument at the right level so that this argument works for all test subcommands

The project-settings option is currently broken.

Traceback (most recent call last):
  File "/usr/local/bin/uluru-cli", line 11, in <module>
    load_entry_point('aws-cloudformation-rpdk', 'console_scripts', 'uluru-cli')()
  File "aws-cloudformation-rpdk/src/rpdk/cli.py", line 59, in main
    args.command(args)
  File "aws-cloudformation-rpdk/src/rpdk/project_settings.py", line 13, in project_settings
    settings = f.read().decode("utf-8")
AttributeError: 'str' object has no attribute 'decode'

We need to test if handlers are idempotent. For Create, Read, and Delete operations after requests are verified as successful, send an identical request with the same token and ensures it returns the same result.

• Tests should delete resources even if an exception occurred (context manager or try...finally)
• These deletes shouldn't ever fail, but can log if we couldn't delete the resource
• Generally err on the side of deleting

the flattener currently assumes that items must be a dict, though it could be an array of schemas, and it does not consider "contains"

Right now, -v/--verbose must be specified before the subcommand, which is unusual.

By using argparse's parents feature, it's possible to make shared parameters that can appear anywhere (like -h/--help).

#16 needs a pre-release version of jsonschema. For now, we're using pip to install that. Once a stable version is released, we need to undo changes to the README and buildspec.yml, as well as pin the jsonschema version in setup.py so people will always have a version that supports draft-6.

Currently, a developer needs to provide a resource example when running contract tests. A more convenient user experience would be to generate valid examples based off the schema that they create/provide.

"normalizer" is pretty confusing and flattener is a more accurate description of what its doing

As discussed in #45, it would be nice to provide validation above and beyond what's expressible in JSON Schema, e.g. the mutually exclusiveness of readOnly and writeOnly properties.

I realised after posting #60, that the init command is what sets up the workspace (#protip: think first, PR second).

However that got me thinking; now that I'm going through this workflow, I'm realising there's some deficiencies we should address. Currently, init takes language and output-directory args. As goes generate. Perhaps it would be neater if we had a single init command which setup a workspace with language of your choice, and moved generate to an implicit build action for the initialised projects.

I think 2 steps is one too many, and there's too many ways to misalign your CLI args. And since I was one of the designers of the current workflow and I can't get it right in usage, that's not a good sign. 😆

Things like Arn, AvailabilityZone, CidrBlocks, Tags, etc. can all be extracted out as reusable types for service teams. Add unit tests for these validations.

The RefInliner change of #56 exposed some errors in the example schemas. Why were these not caught using the normal validation method? Did we set it up wrong, so it doesn't do recursive validation?

In contract tests, whenever resources are returned by handlers, verify that the returned resource is compliant with the resource schema for said resource.

the current schema defines a $title property, which was a mistake and should just be "title"

Generate doco and cleanup some info from the README into the doco

the unit tests for the flattener should be improved to test smaller units of the code. Currently, most of the tests cover way too large portion of the code, if not the entire thing.

For the contract tests, it would be ideal if we could generate properties from the schema. For most properties, this isn't hard (e.g. generating a random integer with min and max constraints). The benefit is we could use hypothesis for this, and maybe test e.g. negative integers to see if they are valid.

However, e.g. a string with regex validation would make this difficult. One thing we could do is try and generate e.g. 100 random strings. If less than 80 pass validation, then we could require e.g. 3 examples. Then we could auto-generate create requests + update requests and have one spare.

Read/List handlers should simply return an object(s). They shouldn't follow the async ProgressEvent callback semantic of other handlers. Discuss.

a property should not be able to have multiple types. For example, we currently would resolve:

{
    "properties": {
        "Test": {"type": "object"}
    },
    "anyOf": [
        {
            "properties": {
                "Test": {"type": "integer"}
            }
        }
    ]
}

to:

{
    "properties": {
        "Test": {"type": "integer"}
    }
}

but we should throw a validation error

On commands that require interaction with AWS (package and submit), we should fast fail and report some error to the user if their credentials are invalid. Currently the user would have to look in the logs to see this result. A function that checks credentials could be added before these command execute their main portions, and this function could inform the user of no credentials or bad credentials.

At the moment, if the SAM CLI isn't running, the contract tests still plow through and fail all. This isn't super useful if you've e.g. forgotten to run the SAM CLI.

Missing branch coverage in src/rpdk/jsonutils/flattener.py, introduced by #92 , #116 .

The identifiers semantic is a top-level metadata item supplied as a list of json-pointers to the properties used to uniquely identify the resource.

Note that multiple properties may serve as identifiers. For example, AWS::ECS::Service is uniquely identified by “ServiceName” and “Cluster”, because a ServiceName is only unique within a given Cluster.

PR #10 adds a plugin system that loads plugins on initialize. This could be changed to lazy loading.

It doesn't make sense for a SubObject to have some properties defined, and others undefined. If some of the properties are undefined, it's an additional arbitrary map of properties, which itself should be a property.

Increase unittest coverage to 95%