S3Drive - S3Drive on Mac: Cannot get past S3 read test.

GET reports 404 but S3Drive fails with invalid XML - 404 error does not return XML Created .s3drive_bucket_read_test to avoid XML error but clearly the content is wrong. Can the read test be bypassed or what content should be in .s3drive_bucket_read_test?

Hi, What S3 provider/implementation are you using? S3Drive expects 404 NoSuchKey when reading that. The valid format is XML: https://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html#RESTErrorResponses You can possibly get past this check, by importing credentials: https://docs.s3drive.app/setup/import/

Tom

Hi, What S3 provider/implementation are you using? S3Drive expects 404 NoSuchKey when reading that. The valid format is XML: https://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html#RESTErrorResponses You can possibly get past this check, by importing credentials: https://docs.s3drive.app/setup/import/

@Tom Thank you for the response. S3 implementation: Hammerspace Interesting that S3Drive requires the Error XML but returning that XML confuses the aws client. In any case, that got me past that problem. Thanks When uploading a file, I see that the HEAD request uses an Authorization header whereas PUT uses QueryParams. Why use different authentication approaches on a per request basis?

hammerspace

@Tom Thank you for the response. S3 implementation: Hammerspace Interesting that S3Drive requires the Error XML but returning that XML confuses the aws client. In any case, that got me past that problem. Thanks When uploading a file, I see that the HEAD request uses an Authorization header whereas PUT uses QueryParams. Why use different authentication approaches on a per request basis?

Interesting that S3Drive requires the Error XML but returning that XML confuses the aws client.

What do you mean? Are you finding any out-of-spec behavior for S3Drive? In principle once valid XML with Not Found error code is returned, that should satisfy this check. Not all S3 providers are compliant and we've implemented workarounds, but in its essence this check is important for us to understand that we talk to valid S3 endpoint.

Why use different authentication approaches on a per request basis?

Presigned URLs offers better flexibility when we e.g. pass the download URL to a different module which isn't aware of protocol (S3) specifics, but must simply do the job of downloading a file. Same applies for file upload. https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-presigned-url.html Presigned URLs are structured differently and as you pointed they use QueryParams. Remaining requests e.g. HEAD don't need to or can't be presigned, so we use standard S3 V4 authorization. (edited)

Tom

Interesting that S3Drive requires the Error XML but returning that XML confuses the aws client.

What do you mean? Are you finding any out-of-spec behavior for S3Drive? In principle once valid XML with Not Found error code is returned, that should satisfy this check. Not all S3 providers are compliant and we've implemented workarounds, but in its essence this check is important for us to understand that we talk to valid S3 endpoint.

Why use different authentication approaches on a per request basis?

Presigned URLs offers better flexibility when we e.g. pass the download URL to a different module which isn't aware of protocol (S3) specifics, but must simply do the job of downloading a file. Same applies for file upload. https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-presigned-url.html Presigned URLs are structured differently and as you pointed they use QueryParams. Remaining requests e.g. HEAD don't need to or can't be presigned, so we use standard S3 V4 authorization. (edited)

Presigned URL: Okay, I was just curious. Thanks. XML Error content: Let's say the error is 404.

When I DO NOT return the Error XML content to the aws client, i.e. just return the HTTP header, I get: An error occurred (404) when calling the GetObject operation: Not Found
When I return the Error XML content to the aws client, the aws client doesn't recognize the 404 status: An error occurred () when calling the GetObject operation I'm referring to the aws cli: https://awscli.amazonaws.com/v2/documentation/api/latest/reference/index.html#cli-aws

hammerspace

Presigned URL: Okay, I was just curious. Thanks. XML Error content: Let's say the error is 404.

When I DO NOT return the Error XML content to the aws client, i.e. just return the HTTP header, I get: An error occurred (404) when calling the GetObject operation: Not Found
When I return the Error XML content to the aws client, the aws client doesn't recognize the 404 status: An error occurred () when calling the GetObject operation I'm referring to the aws cli: https://awscli.amazonaws.com/v2/documentation/api/latest/reference/index.html#cli-aws

Based on AWS S3 documentation of GetObject action: https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html#API_GetObject_Errors

NoSuchKey

    The specified key does not exist.

    HTTP Status Code: 404

It requires both the XML and the 404 HTTP status code. I am not sure if this is what you're replying with or whether you refer to anything else? We are happy to help out with the compliance though. (edited)

Tom

Based on AWS S3 documentation of GetObject action: https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html#API_GetObject_Errors

NoSuchKey

    The specified key does not exist.

    HTTP Status Code: 404

It requires both the XML and the 404 HTTP status code. I am not sure if this is what you're replying with or whether you refer to anything else? We are happy to help out with the compliance though. (edited)

@Tom, It's not causing me a problem. I just thought it was curious that the aws cli can't handle the XML when reporting an error. (edited)