DynamoDB Item API support about aws HOT 42 CLOSED

@wuzzeb seems to be on hiatus with his work on DynamoDB. (But then I haven't asked him personally, so I don't really know more than you.)

from aws.

Ok, I've started adding a subset of the API. Once I know it's working I'll submit a pull request.

from aws.

@manny-fp How far along is this? @MichaelXavier and I were hoping to move this forward.

from aws.

@bitemyapp I'm afraid my work on this has stalled (project priorities changed).

from aws.

I was hoping for this as well..

Just wanted to point out that I have a fork from ~1 year ago that had Get, Put and UpdateItem working at the time. I had the framework up so that each additional command would be just a couple hours of tedious work.

See:

https://github.com/ozataman/aws
https://github.com/ozataman/aws/tree/master/Aws/DynamoDb/Commands

Not sure if we can benefit from this work or have to start over, but wanted to mention in any case. I'm re-evaluating DynamoDb for use at work and may have to quickly get this part up and running for some testing.

from aws.

@ozataman it might require parting out/copy-pasta rather than a straight-up rebase, but this is still helpful, thank you!

from aws.

I just tried a straight merge and it seems it may not be that hard to actually get it working.

from aws.

Pull request from you incoming, Ozgun?:)
Am 23.07.2014 21:18 schrieb "Ozgun Ataman" [email protected]:

I just tried a straight merge and it seems it may not be that hard to
actually get it working.

—
Reply to this email directly or view it on GitHub
#104 (comment).

from aws.

@aristidb @ozataman should I hold off then? I was about to see about integrating the commands.

Edit: I don't mind waiting, I'm at LambdaJam right now. This can happen when I get home :)

from aws.

:)

I will take a quick stab and see if it's possible without too much work. I'm afraid I wouldn't have a ton of time to sink into this if I fail... I'll report back soon.

from aws.

I am happy to merge it and take it forward if it proves too time-consuming, just keep me posted.

Thank you!

On Jul 23, 2014, at 2:52 PM, Ozgun Ataman [email protected] wrote:

:)

I will take a quick stab and see if it's possible without too much work. I'm afraid I wouldn't have a ton of time to sink into this if I fail... I'll report back soon.

—
Reply to this email directly or view it on GitHub.

from aws.

Gentlemen, I'm happy to say that I have the merge working. I also brought in some enhancements to the DynamoDb.Core module. You can see on my fork (posted above).

@bitemyapp I hope you don't mind, but this merge changed some of the naming conventions you had, mainly in the configuration object. It shouldn't be a big deal either way and we can change names if anyone feels strongly about them. It did make sense, however, to bring in a bunch of my prior work as it had a bit more structure and strong typing than your current version, particularly around the error and metadata types. I did away with my query signing though and used yours instead.

@aristidb I'll hold off on sending a pull request, as I'd like to play around with the API a little bit and implement a couple more things missing from the current Get/Put/Update commands. Once I'm done, it will hopefully be a good example of how to do the remainder of the commands.

Any comments or feedback most welcome.

from aws.

Alright guys, I think I'm reaching a good point with the API for a pull request. Please take a look at my fork and let me know if you have any comments. I will send a pull request within the next 1-2 days.

It is a pretty feature-complete implementation of the AWS API for Get, Put and Update. I have also built in a minor amount of API conveniences while staying fairly low level. Further API magic could be added as a separate layer.

We have an Examples/DynamoDb.hs now that compiles into an executable with -fexamples, which is probably a good starting point. Haddocks also have a decent amount of documentation in them, which should hopefully help.

Scan/Query commands should probably be next in priority order.

from aws.

Looks nice, just one small niggle: Can't you make the attrAs use Proxy or a regular ADT? I don't like those "text = undefined" definitions.

from aws.

Yeah, those were kludges from last year and aare n eyesore for me also. Should be possible to make proxy variants without much trouble.

On Jul 26, 2014, at 4:06 AM, Aristid Breitkreuz [email protected] wrote:

Looks nice, just one small niggle: Can't you make the attrAs use Proxy or a regular ADT? I don't like those "text = undefined" definitions.

—
Reply to this email directly or view it on GitHub.

from aws.

I tried to build @ozataman version, have I missed something?

[60 of 68] Compiling Aws.DynamoDb.Core ( Aws/DynamoDb/Core.hs, dist/dist-sandbox-41afd995/build/Aws/DynamoDb/Core.o )

Aws/DynamoDb/Core.hs:302:8:
    Could not deduce (DVal b0)
      arising from the ambiguity check for ‘hrk’
    from the context (DVal a, DVal b)
      bound by the type signature for
                 hrk :: (DVal a, DVal b) =>
                        T.Text -> DValue -> T.Text -> DValue -> PrimaryKey
      at Aws/DynamoDb/Core.hs:(302,8)-(307,17)
    The type variable ‘b0’ is ambiguous
    In the ambiguity check for:
      forall a b.
      (DVal a, DVal b) =>
      T.Text -> DValue -> T.Text -> DValue -> PrimaryKey
    To defer the ambiguity check to use sites, enable AllowAmbiguousTypes
    In the type signature for ‘hrk’:
      hrk :: (DVal a, DVal b) =>
             T.Text -> DValue -> T.Text -> DValue -> PrimaryKey
cabal: Error: some packages failed to install:
aws-0.9.1 failed during the building phase. The exception was:
ExitFailure 1

What am I missing?

from aws.

I am on 7.6.3, which is probably why I hadn't noticed this (it builds fine for me). Fixed.

from aws.

Also, I'm still playing with the DVal convenience interface, so a few things may change this weekend.

from aws.

should hrk :: (DVal a, DVal b) => T.Text -> DValue -> T.Text -> DValue -> PrimaryKey
be something like
hrk :: (DVal a, DVal b) =>T.Text -> a -> T.Text -> b -> PrimaryKey ?

from aws.

(because those type class variables are all missing from the right hand side?)

from aws.

@ozataman I hate nitpicking but can't dive in very deeply yet. DynVal nomenclature threw me for a loop, could something more explicit be used?
DynamoValue? Something like that?

from aws.

I prefer shorter names to longer ones, especially when the word is going to
be used very frequently. DynVal is clear enough in my opinion but I am
happy to discuss further. I personally don't see how any of your
suggestions make any difference :-)

Also note that due to the type tricks there, we have multiple types
mentioning dyn. You may wonder why they are this way and the answer is
mostly to minimize code duplication for the end user and overall
API convenience.

I am certainly open to alternatives - this is all experimentally seeking
the ideal design.

Also, I'll be in Boston this weekend for the meetup and should be on IRC
most of the time. We can also discuss in real time if you're going to be
around.

On Thursday, July 31, 2014, Chris Allen [email protected] wrote:

@ozataman https://github.com/ozataman I hate nitpicking but can't dive
in very deeply yet. DynVal nomenclature threw me for a loop, could
something more explicit be used?
DynamoValue, DynoValue, DynoVal? Something like that?

—
Reply to this email directly or view it on GitHub
#104 (comment).

Ozgun Ataman

CEO
Soostone Inc.
25 Broadway, 9th Floor
New York, NY, 10004

+1 917 725 0849 (Office)
+1 734 262 0676 (Mobile)
+1 212 320 0251 (Fax)
[email protected]

from aws.

@ozataman Not in Boston, I was at LambdaJam.

DynVal is fine.

For @aristidb as well: one question I had is about tests. There is a local DynamoDB instance you can fire up and run commands against - where are the tests? I can't find any in the cabal file.

If none exist and we would like them to exist, I can add a tests module.

from aws.

No Dynamo tests yet, sorry
Am 02.08.2014 22:30 schrieb "Chris Allen" [email protected]:

@ozataman https://github.com/ozataman Not in Boston, I was at LambdaJam.

DynVal is fine.

For @aristidb https://github.com/aristidb as well: one question I had
is about tests. There is a local DynamoDB instance you can fire up and run
commands against - where are the tests? I can't find any in the cabal file.

—
Reply to this email directly or view it on GitHub
#104 (comment).

from aws.

@aristidb that's fine, but I'm asking if it's okay if I add some starting with the DynamoDB modules. Second question, do you want me to bundle in the dynamodb local test server?

from aws.

Tests are fine. Add a link to the test server, no copy please.
Am 02.08.2014 22:37 schrieb "Chris Allen" [email protected]:

@aristidb https://github.com/aristidb that's fine, but I'm asking if
it's okay if I add some starting with the DynamoDB modules. Second
question, do you want me to bundle in the dynamodb local test server?

—
Reply to this email directly or view it on GitHub
#104 (comment).

from aws.

@aristidb understood, thank you.

from aws.

@bitemyapp The reason I brought Boston up was to say that I would be on IRC to chat if needed.

I've since then added Query and Scan, including convenient pagination via conduit Producer. I'm also incrementally using my fork in our experiment for work. Hoping to wire 2M+ ops/day through this by the end of this weekend.

from aws.

Had a look at IteratedTransaction and ListTransaction? Maybe you can use
them
Am 02.08.2014 22:48 schrieb "Ozgun Ataman" [email protected]:

@bitemyapp https://github.com/bitemyapp The reason I brought Boston up
was to say that I would be on IRC to chat if needed.

I've since then added Query and Scan, including convenient pagination via
conduit Producer. I'm also incrementally using my fork in our experiment
for work. Hoping to wire 2M+ ops/day through this by the end of this
weekend.

—
Reply to this email directly or view it on GitHub
#104 (comment).

from aws.

@ozataman How's it lookin'?:)

from aws.

It's looking good, but I work on this sort of thing in iterations and might move the API around until I am happy with it. So far so good though.

We can move it over to a branch on your repo if you want to give me access, or I'll let it simmer for a bit here and send you a pull request after the dust has settled. Either way, we should probably wait a while before releasing on Hackage. I'll be deploying this to prod soon, which should help clear out any lurking issues.

from aws.

I don't think it makes much of a difference whether it's in a branch on
mine or your repo. So let's just wait until you got it in production and
ironed out any problems you discover that way? And then a pull request
please!

2014-08-03 20:35 GMT+02:00 Ozgun Ataman [email protected]:

It's looking good, but I work on this sort of thing in iterations and
might move the API around until I am happy with it. So far so good though.

We can move it over to a branch on your repo if you want to give me
access, or I'll let it simmer for a bit here and send you a pull request
after the dust has settled. Either way, we should probably wait a while
before releasing on Hackage. I'll be deploying this to prod soon, which
should help clear out any lurking issues.

—
Reply to this email directly or view it on GitHub
#104 (comment).

from aws.

@ozataman By the way, even something imperfect would be far better than what we have in AWS right now...

from aws.

Agreed. The main reason I've been putting it off was to avoid locking into a specific API and major bugs. If you don't mind potentially breaking the API in a future release (if needed), I'm happy to send a pull request. It is still missing a few commands, but is vastly more than what we had before.

from aws.

I think we should just put it in and mark it as experimental. People are never entirely happy with breaking changes, but if we update version numbers according to the PVP they can protect against that, and I think we're better off with something imperfect that needs breaking changes later than nothing.

from aws.

@ozataman Please change your Query and Scan pagination stuff to use the standard aws pagination machinery: IteratedTransaction.

from aws.

@ozataman I've actually done that myself now. :) I think I'll merge it to master like this.

from aws.

Thanks Aristid. I havent had a chance to look at your change but I would definitely like to keep the conduit layer I had in there. I dont care so much for the underlying machinery I hand-rolled. I'll take a look when I get the chance.

On Aug 16, 2014, at 6:13 AM, Aristid Breitkreuz [email protected] wrote:

@ozataman I've actually done that myself now. :) I think I'll merge it to master like this.

—
Reply to this email directly or view it on GitHub.

from aws.

awsIteratedSource gives you a conduit as well.:)
Am 16.08.2014 16:10 schrieb "Ozgun Ataman" [email protected]:

Thanks Aristid. I havent had a chance to look at your change but I would
definitely like to keep the conduit layer I had in there. I dont care so
much for the underlying machinery I hand-rolled. I'll take a look when I
get the chance.

On Aug 16, 2014, at 6:13 AM, Aristid Breitkreuz <
[email protected]> wrote:

@ozataman I've actually done that myself now. :) I think I'll merge it
to master like this.

—
Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHub
#104 (comment).

from aws.

@aristidb I have a need for a slightly more flexible interface for the conduit converters. It was easy to implement, however. I haven't tested it yet, but here it is:

Soostone@13eb694

Will circle back when I've confirmed these work as intended.

from aws.

Here where? oO

from aws.

Ah. I suggest you implement the old less generic functions in terms of the generic ones.

from aws.