S3 requests take path - which is NOT the S3 key, - Leading '/' problem about asi-http-request HOT 6 CLOSED

Hi Tom

Thanks for your report!

As you'll have seen, there isn't a lot of documentation for the S3 stuff, but all the examples at http://allseeing-i.com/ASIHTTPRequest/S3 use the forward slash at the beginning.

Looking at it now, I can see that S3 queries return keys without the forward slash, so I agree it would be better if you didn't need to use it when creating a request.

To me, adding the slash automatically if it isn't there seems like a good solution - can you explain more about why you think this would be a problem? I could change the API, but this would break any existing apps using it, and I think it would be confusing to add another method that does pretty much the same thing.

Best

Ben

from asi-http-request.

Ben,

I think that adding a forward slash if one is missing will work fine. I can't think of why this would be a problem - and I have been working with ASIRequest for a few days now, and I still can't see the problem.

OK - I thought of some problems....

The max length of a key is 1024 UTF8 bytes. If the passed in key is exactly 1024 bytes long, then adding a / will put us over the top. Unlikely - but I guess will happen. You can measure the length via lengthOfBytesUsingEncoding.

I changed the code as follows in my copy of ASIRequest

I found another issue which is this: The key needs to be encoded correctly in the URL.

Here is requestWithBucket as I have it now. It worked on the few keys I tried that had spaces, etc...

// NSURL's stringByAddingPercentEscapesUsingEncoding: does not escape
// some characters that should be escaped in URL parameters, like ? and some others.
// We'll use CFURL to force the encoding of those
// NOTE: Amazon S3 handles the / char fine in the 'path' portion of a url, so it is left alone 
//
// Reference: http://www.ietf.org/rfc/rfc3986.txt

//const CFStringRef kCharsToForceEscape = CFSTR("!*'();:@&=+$,/?%#[]");

const CFStringRef kCharsToForceEscape = CFSTR("!*'();:@&=+$,?%#[]"); // I took '/' out since it makes the URLs much easier to read..
+ (NSString *)stringByURLEncodingForS3Path:(NSString *)str; 
{

  NSString *resultStr = str;

  CFStringRef originalString = (CFStringRef) str;
  CFStringRef leaveUnescaped = NULL;

  CFStringRef escapedStr;
  escapedStr = CFURLCreateStringByAddingPercentEscapes(kCFAllocatorDefault,
                                                       originalString,
                                                       leaveUnescaped,
                                                       kCharsToForceEscape,
                                                       kCFStringEncodingUTF8);
  if (escapedStr) {
    resultStr = [(id)CFMakeCollectable(escapedStr) autorelease];
  }
  return resultStr;
}


+ (id)requestWithBucket:(NSString *)bucket path:(NSString *)path
{
    if (![path hasPrefix:@"/"])
        path = [@"/" stringByAppendingString:path];

    // S3 has a restriction on key lengths of 1024 bytes. The strings are utf8 strings. 
    const NSInteger kMaxS3CloudKeyLengthUTF8Bytes = 1024;
    NSInteger length = [path lengthOfBytesUsingEncoding:NSUTF8StringEncoding];
    if (length > kMaxS3CloudKeyLengthUTF8Bytes)
        return nil; // this will usually cause an exception later, it seems...

    NSString* escapedPath = [self stringByURLEncodingForS3Path:path];
    ASIS3Request* request = [[[self alloc] initWithURL:[NSURL URLWithString:[NSString stringWithFormat:@"http://%@.s3.amazonaws.com%@",bucket, (NSString*)escapedPath]]] autorelease];
    [request setBucket:bucket];
    [request setPath:(NSString*)escapedPath]; // this needs to be the same escaped string as we set in the URL so signing will work. 

    return request;
}

The other way of looking at this is to leave it to the caller to properly escape the path before sending it in. This may be what you want to do, and is fine with me. In fact would adding code like this cause some paths to break? - EG ones using the current API that are passing in already escaped paths into requestWithBucket.
--Tom

from asi-http-request.

Looking at the examples in ASIRequest it seems that requestWithBucket assumes that the passed path is already escaped, so it is up to the caller to add something like stringByURLEncodingForS3Path before submitting a path to any of the calls.

Since a URL encoded string's length can be much greater than the actual utf8 byte length of the string, you can't easily check the length of the key in request with bucket. So its back to the just adding a leading / if necc.

Thanks for looking at this.

--Tom

from asi-http-request.

Hi Tom

I agree, I think it's best to leave encoding the URL up to the user, I'm guessing problems with URLs that aren't properly encoded won't be unique to S3.

BTW, ASIFormDataRequest has a encodeURL: method that might work for this.

I'll make the change to add the forward slash today.

Best

Ben

from asi-http-request.

Hi again

Ok, I've changed my mind... :)

I'm doing some major work on the S3 stuff today (to support bucket requests), and I've decided the API will need to change anyhow. So, I think I'll update the object API to take real S3 keys as per your original suggestion, rather than forcing the user to encode them in advance. I'll get rid of the path property, and have a key property instead (which should always be a real S3 key), only url-encoding it when it needs to go into a url.

Thanks so much for all your suggestions, I'll update here when there's something new to experiment with.

Ben

from asi-http-request.

Hi Tom

The S3 API has had a big overhaul in the newest version in Github, and it shouldn't be necessary to encode keys manually anymore. I've added a variation of your code to encode the keys when adding them to a url.

Thanks again

Ben

from asi-http-request.