The old scripts use oasis query and depend on an oasis directory layout.
These need to be ported to deal with the new dune layout and be fully automated.
Our options are:

1. Use topkg, atd repo has some instructions https://github.com/mjambon/atd/blob/be57549f4035d4d52a928b29542f5df9563acbe2/internal-doc/README.md
2. Use dune-release
3. Use opam publish

Initial investigation of dune-release suggests it doesn't handle multiple packages from a single repository very well. Most of the commands seem to make this assumption.

The service descriptions found in the input/ subdirectory contain documentation strings that can be inserted into generated code.

I tried to install the library using opam which somehow succeed but leaves me with only two modules Aws and Aws_lwt. How can I arrange so that material found in libraries/ is also compiled and installed?

Also, I am a bit confused by the whole library/ directory. If I understand correctly, its content is generated by the aws_gen program. But why add this to the SCM system?

Thanks!

Setup circleci or travis with the correct AWS permissions to run integration tests again AWS.

I have some circleci setup for other OCaml projects that might be useful.

Bonus would be to have a cleanup stage post CI so we don't leave any hanging resources around that cost me $$$$.

Version 2.0.0 of yojson removed biniou, which causes the compiler error below. There is a lower bound on yojson in the opam files, but no upper bound. Pinning yojson locally to 1.7.0 allows the build to work for me.

File "src/endpoints_j.ml", line 40, characters 11-13:
  40 |   write__2 ob x;
Error: This expression has type Bi_outbuf.t
       but an expression was expected of type Buffer.t

The module "name" has a "dash:"

ocaml# #require "aws-ec2";;
ocaml# open Aws_ec2;;
Error: Unbound module Aws_ec2

 $ ls `ocamlfind query aws-ec2`/*.cmi
/home/smondet/.opam/4.03.0/lib/aws-ec2/aws-ec2.cmi

 $ ocamlobjinfo `ocamlfind query aws-ec2`/*.cma
File /home/smondet/.opam/4.03.0/lib/aws-ec2/aws-ec2.cma
Force custom: no
Extra C object files:
Extra C options:
Extra dynamically-loaded libraries:
Unit name: Aws-ec2
Interfaces imported:
	711e5c0d799e5bc2329ebf7485a79aec	Aws-ec2
	8511e1de84e74c17dacc88c35f5be478	Xmlm
	f17596e00890cdbfa772cbc596b3ec17	Uri
	331e41ad213579d006efd84f19005bd5	Unix
	c9608ce7bf745054a217110dcaac126e	Sys
	c28a3ca42a30c6ffa0bbab4a05329226	String
	fc4cd9c73967ba1a8bc6360b21cf5f3a	Str
	0b2cb4ad5159673670474028f5fac7a6	Sexplib
	f665d1463ac108184ac6ad6d8bac076e	Random
	0145d8daf24f9afd43ffac3c376fa3c4	Printf
	999b28e3b7638771c87eebf5a8325e42	Pervasives
	ecf089d7539fcff97623be26ecebf193	Nativeint
	ac5f6095cc0a546330ada0df0986a497	List
	330cec55af19abb867bbde7885b2588f	Lexing
	b2e565a5cdbd351dc15bc9061d30c458	Lazy
	8deb8d60f2294e2f3dc6971e7f2efba2	Int64
	d0de3bfd44513ff6fd70cbf476e9fa55	Int32
	049026f8240c3b61bd91f719b0f55877	Hashtbl
	60c2e7663dd57d13b5920931742e1c10	Format
	701e80504b28ac51442bbd6cb8925d28	Ezxmlm
	def386d89a071f0355f7bda7e3a074fd	Complex
	4a517d12500bc3789093103f6fee94bf	Char
	9642e3ed163e46770985ca668738ed5f	CamlinternalFormatBasics
	d2b09586c51f5e3c7bc0527bd55f0b38	CalendarLib
	6dc691300ced97c0e319cbcc0a715044	Bytes
	3bd1af04573ce2da7fc3dc04403e852e	Buffer
	7cb259b8efbf4550648d25651a3342ee	Bigarray
	0d57ddf07f596b30b0d744c01f7d0bd7	Aws
Uses unsafe features: no
Force link: no

 $ opam show aws-ec2
             package: aws-ec2
             version: 0.1.0
          repository: default
        upstream-url: https://github.com/inhabitedtype/ocaml-aws/releases/download/aws-1.0.2/aws-ec2-0.1.0.tar.gz
       upstream-kind: http
   upstream-checksum: 22eee043b576df98b095f508b440a846
            homepage: https://github.com/inhabitedtype/ocaml-aws
         bug-reports: https://github.com/inhabitedtype/ocaml-aws/issues
            dev-repo: https://github.com/inhabitedtype/ocaml-aws.git
              author: Spiros Eliopoulos <[email protected]>, Daniel Patterson <[email protected]>
             license: BSD-3-clause
             depends: aws >= 0.1.0 & ocamlfind
   installed-version: 0.1.0 [4.03.0]
   available-version: 0.1.0
         description: Amazon Elastic Compute Cloud

See https://docs.aws.amazon.com/general/latest/gr/signing_aws_api_requests.html where Simple DB (SDB) is the only service that doesn't support the newer V4 signing process.

Need to implement the V2 signing process and only apply it to SDB service calls.

Minimal test showing a failing SDB request here https://github.com/tmcgilchrist/ocaml-aws/tree/topic/sdb_test

I am trying to use the S3 API. I tried both ListObjects and GetObjects.

For both, the result is the same - the answer from the server is not the one expected:

<?xml version="1.0" encoding="UTF-8"?>
<ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
...
</ListAllMyBucketsResult>

Vs

<?xml version="1.0" encoding="UTF-8"?>
<GetObjectOutput>
...
</GetObjectOutput>

Here is the code I use:

let downloadFile = (filename) => {
  let settings = Config.settings;

  let run_request =
    Aws_lwt.Runtime.run_request(
      ~access_key=settings.accessKey,
      ~secret_key=settings.secretKey,
      ~region=settings.region,
      (module Aws_s3.GetObject),
      Aws_s3.Types.GetObjectRequest.make(
        ~bucket=settings.bucket,
        ~key=filename,
        (),
      ),
    );

  Lwt_main.run(run_request);
};

Where settings.bucket = "bucket-name" and settings.region = "us-east-1".

When reading the documentation, I am under the impression that the host endpoint to target should be:
bucket-name.s3.amazonaws.com, but from what I see, the library targets s3.amazonaws.com, which might explain issue.

The original service definitions were copied across a long time ago.
Look at the latest versions from botocore, regenerate services and test to determine the impact of upgrading.

See
https://github.com/boto/botocore/tree/develop/botocore/data

• Bump lower bound to 4.08 to include Uri 4.1 fixes for escaping #104
• Generate AWS Lambda #108
• Migrate CI to github actions #104
• Need AWS Lambda tests and example usage
• Align ENV variable usage with official AWS client usage #111
• Include AWS documentation for services in generated code
• Bump ppx_tools_versioned lower bound to 4.08, Better option to remove the dependency #126
• Support for signing requests with assume_role tokens #117
• Add KMS support #118

In the interests of visibility these are the pieces I'm working on for the next minor release of ocaml-aws.

EDIT
Removing the aws-lambda support from this release, it requires a larger re-write than I expected.

I've been working on adding SQS support to this library and I ran into a some code in generate.ml that seems to assume all service url's are of the format:

(app1 "Uri.of_string" (str ("https://" ^ service ^ ".amazonaws.com")))

There needs to be a bit more logic to this, as the endpoints aren't really consistent, documented here: https://docs.aws.amazon.com/general/latest/gr/rande.html

I see the _endpoints.json file in the input directory and I'm thinking that by grabbing an updated version from boto, it could be used to generate an Endpoint module which would allow the appropriate url creation logic to be applied per service/region.

I have been mucking around with fixing some issues in the enum generator and ran into this. If wanted, I can probably pop open a wip PR for both items in a day or so.

Should be simple enough to fix if someone wants to dig into this:

Error (warning 3): deprecated: Yojson.Basic.json
json types are being renamed and will be removed in the next Yojson major version. Use type t instead
File "src/reading.mli", line 36, characters 18-35:
Error (warning 3): deprecated: Yojson.Basic.json
json types are being renamed and will be removed in the next Yojson major version. Use type t instead
File "src/reading.mli", line 38, characters 22-39:
Error (warning 3): deprecated: Yojson.Basic.json
json types are being renamed and will be removed in the next Yojson major version. Use type t instead```

Are there any plans to support working with S3 directly? Or does that exist already in the library/libraries somewhere?

Simple code for describeimagesresult

  | `Ok r ->
    let (b : Yojson.Basic.json) =
      Aws_ec2.Types.DescribeImagesResult.to_json r
    in
    Yojson.Basic.pretty_to_string b
    |> Lwt_io.printl
  | `Error oops ->
    Aws.Error.format Aws_ec2.Errors.to_string oops |> Lwt_io.printl

I need something similar for Aws_ec2.CreateVolume, currently have:

let do_run =
  let (access_key, secret_key) = keys () in
  Aws_lwt.Runtime.run_request
    (* Need to pick the appropriate region, won't work with a wrong
         region *)
    ~region:"us-east-1"
    ~access_key
    ~secret_key

let do_create_volume () =
  do_run
    (module Aws_ec2.CreateVolume)
    (Aws_ec2.Types.CreateVolumeRequest.make
       ~dry_run:true
       ~availability_zone:"us-east-1"
       ()) >>= function
  | `Ok r ->
.... But Aws_ec2.Types.CreateVolumeResult does not exist.
  | `Error oops ->
    Lwt_io.printl "Done"

So how can I get a result? I understand the code is auto generated but does that mean that the ec2 bindings are incomplete at the moment?

Next major release

• Rewritten code generation based of ppxlib directly.

When trying to use the ocaml-aws library, I couldn't figure out a way to supply temporary credentials using the API as it is. What I needed to support temporary credentials, was to add a "x-amz-security-token" header with a supplied token. I think it might be as easy as adding an optional parameter to sign_request that will add the appropriate header, but I think that would then need to be surfaced in places that is being called, like Aws_async.Runtime.run_request.

More info on temporary credentials:
-Making Requests
-REST authentication

This will make it possible to reuse a single source distribution for all the different libraries.

In the opam repository of this package (https://opam.ocaml.org/packages/aws/) there is an upper version limit (implemented as a conflict) on the cohttp package:

conflicts: [
  "cohttp" {< "0.17.0"}
  "cohttp" {>="0.99"}
]

Is there any good reason for this? Everything compiles just fine for me with cohttp.2.5.4.

I was upgrading one of my projects that uses this project and attempted to update it to use the new ec2 bindings that were generated in 4c69f5e that were generated with the files from but there appears to be an issue with them.

It looks like a difference in the the generated Types.*. to_query function. In the older version , the initial character of the keys were lowercase but the new version 4c69f5e has it as uppercase. This causes keys to not be recognized as valid by AWS and the request to be rejected. I quickly checked the schema for 2015-04-15 vs 2016-11-15 and it doesn't look like a difference in there.

I think the next steps are to look at the code that was changed in the code generation in between these two generations but I haven't had time to do so and I wanted to document this error so other people could be aware. The old bindings still work well so I've just pinned those for the moment. When I get sometime in the next couple weeks, I'll attempt to spend more time on this.

The current code generation in src/* uses ppx_tools_versioned . We should probably use ppxlib instead.

See https://github.com/inhabitedtype/ocaml-aws/blob/master/libraries/ec2/opam/opam#L14

remove: [
  ["ocamlfind" "remove" "aws-autoscaling"]
]

• Add S3 generation and tests back into master (Revert #92)
• Fix deserialisation. The current working implementations from Haskell and python, skip the root element and look for it's children. While the OCaml looks for the root element to be named as outlined in the service definition.

Thanks for this! I had a similar idea a year ago but wasn't good enough at that time to make it work.

Installed a fresh copy directly via pinning, using lwt, and I'm not getting the Aws_ec2 module. At the top level I only get Aws, Aws_lwt and Aws_gen.

It looks to me that this library only supports a couple AWS services.

1. Am I wrong and things like S3 are supported already?

2. Are there plans to add more services as the library grows?

3. Unrelated question: can I install from opam with the lwt runtime support?

I'm attempting to send an SQS message, where the payload is JSON data and it's failing. The specific error "AWS query cannot have multiple values for same key" (which appears to be from here).

With an invocation like

Aws_async.Runtime.run_request ~region ~access_key ~secret_key
    (module SendMessage)
   (Types.SendMessageRequest.make ~queue_url:(Uri.to_string queue_url) ~message_body:message ())

Where message is a JSON document in a string. To reproduce the issue, any JSON map with more than one keypair would work, so something like:

{"a": 1, "b": 2}

Doing some debugging, it looks like this code is where the request is getting broken:

(Uri.query_of_encoded
               (Query.render (SendMessageRequest.to_query req)))

It is using the Uri library to encode the message into a query string, which has some issues when encoding things for AWS (see mirage/ocaml-uri#65). What it looks like is happening here is that it's treating the commas as the query key having multiple values. That causes the "AWS query cannot have multiple values for the same key" error and also causes the JSON document to become invalid.

I'm considering trying to spend my own time to get this library to a more polished state for our own company's use. It seems like a few PR's have languished, and I'm wondering about LOE to get it more feature-complete. Is there anything specific blocking work here or is it just a lot of details?

There are a few tests for ec2 but it would be great to have more tests across the rest of the libraries.

Current state of basic runnable tests:

• Autoscaling
• SQS - In progress #73
• S3
• EC2
• cloudformation
• cloudwatch
• elasticloadbalancing
• RDS
• STS
• cloudtrail
• dynamodb
• elasticache
• elasticmapreduce
• route53
• SDB - Blocked by V2 request signing #74
• SSM

I can't seem to install the package from opam, am I missing something?

$ opam pin add aws .
aws is now path-pinned to /Users/fredlhsu/code/ocaml

[aws] /Users/fredlhsu/code/ocaml/ synchronized

aws needs to be installed.
The following actions will be performed:
  ∗  install calendar      2.03.2             [required by aws]
  ∗  install mirage-no-xen 1                  [required by nocrypto]
  ∗  install conf-gmp      1                  [required by zarith]
  ∗  install xmlm          1.2.0              [required by ezxmlm]
  ∗  install zarith        1.4.1              [required by nocrypto]
  ∗  install ezxmlm        1.0.1              [required by aws]
  ∗  install nocrypto      0.5.2              [required by aws]
  ∗  install aws           1.0.0*
===== ∗  8 =====
Do you want to continue ? [Y/n] y

=-=- Gathering sources =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=  🐫
[aws.1.0.0] /Users/fredlhsu/code/ocaml/ already up-to-date
[default] https://opam.ocaml.org/archives/ezxmlm.1.0.1+opam.tar.gz downloaded
[default] https://opam.ocaml.org/archives/calendar.2.03.2+opam.tar.gz downloaded
[default] https://opam.ocaml.org/archives/xmlm.1.2.0+opam.tar.gz downloaded
[default] https://opam.ocaml.org/archives/zarith.1.4.1+opam.tar.gz downloaded
[default] https://opam.ocaml.org/archives/nocrypto.0.5.2+opam.tar.gz downloaded

=-=- Processing actions -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=  🐫
∗  installed mirage-no-xen.1
∗  installed conf-gmp.1
∗  installed xmlm.1.2.0
∗  installed calendar.2.03.2
∗  installed zarith.1.4.1
∗  installed ezxmlm.1.0.1
∗  installed nocrypto.0.5.2
[ERROR] The compilation of aws failed at "ocaml setup.ml -configure --prefix
        /Users/fredlhsu/.opam/4.01.0 --enable-lwt --disable-async".
Processing  8/8: [aws: ocamlfind remove]
#=== ERROR while installing aws.1.0.0 =========================================#
# opam-version 1.2.2
# os           darwin
# command      ocaml setup.ml -configure --prefix /Users/fredlhsu/.opam/4.01.0 --enable-lwt --disable-async
# path         /Users/fredlhsu/.opam/4.01.0/build/aws.1.0.0
# compiler     4.01.0
# exit-code    2
# env-file     /Users/fredlhsu/.opam/4.01.0/build/aws.1.0.0/aws-53720-a7c577.env
# stdout-file  /Users/fredlhsu/.opam/4.01.0/build/aws.1.0.0/aws-53720-a7c577.out
# stderr-file  /Users/fredlhsu/.opam/4.01.0/build/aws.1.0.0/aws-53720-a7c577.err
### stderr ###
# Cannot find file setup.ml.



=-=- Error report -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=  🐫
The following actions failed
  ∗  install aws 1.0.0
The following changes have been performed
  ∗  install calendar      2.03.2
  ∗  install conf-gmp      1
  ∗  install ezxmlm        1.0.1
  ∗  install mirage-no-xen 1
  ∗  install nocrypto      0.5.2
  ∗  install xmlm          1.2.0
  ∗  install zarith        1.4.1
[NOTE] Pinning command successful, but your installed packages may be out of
       sync.

The former state can be restored with:
    opam switch import "~/.opam/4.01.0/backup/state-20170418172309.export"```

Pinned to master, for opam install aws-rds I get:

+ /home/smondet/.opam/4.03.0/bin/ocamlfind ocamlc -c -g -annot -bin-annot -package aws -I lib -o lib/describeAccountAttributes.cmi lib/describeAccountAttributes.mli
File "lib/describeAccountAttributes.mli", line 2, characters 13-47:
Error: Unbound module DescribeAccountAttributesMessage
Command exited with code 2.

The sort of a request / response body is based on protocol in a service description file.
For example, the field in DynamoDB description file is "json" and each body of requests / response is JSON, whereas the field in EC2 is "ec2" and each of bodies is XML.
Then, Content-Type header follows the request body (already described #60 (comment)).
Especially the version of x-amz-json is corresponding to jsonVersion field in a service description file.

The current implementation encodes query to URL parameter, doesn't send body, and parses response as XML.
It is not valid in json-protocol services.

With this:

let region = "us-east-1" in
let access_key =  env_exn "AWS_KEY_ID" in
let secret_key =  env_exn "AWS_SECRET_KEY" in
Aws_lwt.Runtime.run_request
  ~region ~access_key ~secret_key
  (module Aws_sdb.CreateDomain)
  (Aws_sdb.Types.CreateDomainRequest.make
     ~domain_name:"smondet-test" ())

I get

HttpError(401 - AwsError): [AuthFailure: AWS was not able to authenticate the request: access credentials are missing]

but the exact same credentials work well with the Aws_ec2 module.

And with AWS-CLI: aws sdb create-domain --domain-name smondet-test0 also works.

Aws.sign_request
seems to be using “Version 4” of the protocol.

And issue aws/aws-sdk-cpp#372
says that SDB is using “Version 2”?

And this doc
http://docs.aws.amazon.com/AmazonSimpleDB/latest/DeveloperGuide/HMACAuth.html
seems to confirm:

This section describes how to create the signature. The method described in
the following procedure is known as signature version 2, and uses the
HMAC-SHA256 signing method.

I'm opening this issue to gather early feedback on some work I've been doing.

I've started working on generating code to support all the AWS services on this branch: https://github.com/inhabitedtype/ocaml-aws/compare/master...anmonteiro:more-apis?expand=1

Let me know if this is something you'd be welcome to accept.