is there any support for CORS? I'm trying to use s3ninja from another webapp that I'm using (just for testing), and I'm getting a CORS error in the console

Currently the homepage only offers an unversioned download url, which makes packaging hard. Would it be possible to have s3ninja-BUILD.zip in addition to s3ninja.zip (which would be the latest build).

I'm trying to create a homebrew formula and without pointing to a specific build the checksum would be out of date soon.

Hi Andy

I was wondering if we could start the s3 localhost from our java junit code. Currently,I downloaded the entire zip to the project directory and kicked off the start.sh from the project location. But, when i try to pull the s3ninja project through maven, how do i kick off the server from my code. The s3 ninja jar doesn't contain a startup script. Is there a way to perform this dynamically from my java code to start and stop the s3 local server.

Please help

Thanks
Koushik

Is there a way to run without the UI elements?

Also how one should block the noAuth=true option for private buckets (making them really private)?

http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectHEAD.html

A HEAD request has the same options as a GET operation on an object. The response is identical to the GET response except that there is no response body.

Via

We sign some S3 URLs using query params, since we return them as 301 redirects. When upgrading to the newest fog, we noticed that s3ninja is now rejecting these signed URLs, because fog is now signing them with v4 and s3ninja appears to only support v4 in headers.

Working around it for now by telling fog to use v2 signatures.

1. I run S3ninja over linux.
2. I upload a file, where the file name starts with a "." character.
   E.g.: "._metadata.crc" is a typical file in Parquet containers
3. Upload was successful, I can see the uploaded file in the UI page in browser
4. But I cannot download or open the file

My first guess is that those files are treated as hidded files in linux as so they are not accessible.

When trying to PUT a file with a subdirectory (i.e. PUT /s3/my_bucket/en/image.png), 404 error is responded with the message No dispatcher found for: /s3/my_bucket/en/image.png, even if the bucket my_bucket exists. That behaviour is inconsistent with the real S3 API.

Hi,

I want to have a way of starting and stopping s3ninja. I'm using it in clojure:

(ns s3-emulation.core
  (:import sirius.kernel.Sirius
           sirius.kernel.Setup
           sirius.kernel.Setup$Mode
           clojure.lang.DynamicClassLoader))

(def ^:dynamic *running* false)

(defn s3-start []
  (let [setup (Setup. Setup$Mode/TEST
                       (DynamicClassLoader/getSystemClassLoader))]
    (Sirius/start setup)
    (alter-var-root #'*running* (fn [_] true))))

(defn s3-stop []
  (when *running*
    (Sirius/stop)
    (alter-var-root #'*running* (fn [_] false))))

When I call (s3-start) it starts s3ninja, and I can also stop it, but when I call s3-start again, it does not start up. Is there a way of being able to start and stop it repeatedly?

I want to use this awesome repo as a mock for tests running in a CI environment. This is for node-s3-client.

How do I get the AWS_KEY and AWS_SECRET in this situation? Is it randomly generated each time s3ninja is started or static?

Is there a way to embed s3ninja so that it could, for example, be brougth up by a JUnit @BeforeClass method and shutdown by a @afterclass method?

If not, what about providing guidance or code for that matter?

Aws4HashCalculator.buildCanonicalRequest uses the client x-amz-content-sha256 header instead of reading the request payload and calculating the hash. This allows the client spoof the payload.

So far I've run into two separate issues trying to run version 2.7 on Mac:

1. when attempting to start via './sirius.sh start', an error related to readlink occurs. I resolved this by overriding with a version from 'brew install coreutils'.
2. attempting to run with Java 7 fails.

Please support running on Mac or at least mention the above in your documentation to save others time.

replace('/', '_') (https://github.com/scireum/s3ninja/blob/master/src/main/java/ninja/S3Controller.java#L295) should be applied to name.startsWith(prefix) (https://github.com/scireum/s3ninja/blob/master/src/main/java/ninja/Bucket.java#L168) in order to properly match keys with / characters.

I am getting com.amazonaws.AmazonClientException: Unable to execute HTTP request
for every API call I make.

From AmazonS3Client.java - the convertToVirtualHostEndpoint method changes the URI
to http://.localhost:9333/s3 (instead of http://localhost:9333/s3/)

What can be done?

I'm using AWS Java SDK 1.9.11

Hello -

I'm trying to get Fog running with S3 Ninja, and I run into an issue:
`No dispatcher found for: /fog-demo-1399303796/"

Command output here:
https://gist.github.com/metadave/5e01865f276ce29b4758

This error seems to happen for any directory that I pass in. Is it possible that the /s3 path is interfering with s3 commands? Would running the GUI and the S3 endpoint on separate ports allow S3 to listen at /?

I believe this issues is related: #3

Cheers -
Dave

I'm using AWS Java SDK 1.9.11.

I've set up s3ninja correctly. From the UI everything works.
From Java if I call s3client.putObject everything is working.
From Java if I call s3clind.getObject I get the following exception:

Failed getting object: chen/chen-test (5 retries exhausted): Unauthorized (Service: Amazon S3; Status Code: 401; Error Code: 401 Unauthorized; Request ID: null) {S3Manager} [main]

From the exception I've extracted the following:

Unable to parse error response. Error XML Not in proper format.

Unauthorized

Authentication required

Diagnostic:

• node: Chens-MacBook-Pro-2.local
• duration: 1 Millisecond
• flow: Chens-MacBook-Pro-2.local/650
• uri: /s3/chen/chen-test
• system: web-mvc::S3Controller::/s3/chen/chen-test
• route: /s3/:1/**
• scope: default
• userId: (public)
• username: (public)
• template: file:/Users/chenharel/Sparktale/Apps/s3ninja/app/default/view/errors/default.html

At this time there`s no support for pre assigned urls? right?
like this http://docs.aws.amazon.com/AmazonS3/latest/dev/PresignedUrlUploadObject.html

[]`s

Does s3ninja support GeneratePresignedUrlRequest (aws-java-sdk)?

I tried running s3ninja on java 7, but it did not run. Had to upgrade to java 8.
Does the code have java 8 specific syntax?
If not then can it be recompiled in java 7 and run on an environment using java 7?
I am trying to use s3ninja for our project and migrating the whole project to java 8 will be difficult.

Hello there,
I tried to use the S3 ninja and it seems to be perfect tool with perfect interface, but whatever S3 API I call, I can see rejected in the logs. Am I doing something wrong ? Does anyone else noticed this issue ?

I am passing the same credentials generated by the S3 ninja UI , but anyway I think that the credentials are not validated in the proper way. So really have no idea why everything is rejected.

Thanks for any toughts

When using evaporatejs for multipart signing and uploading, s3ninja appears not to expose the Etag header on the pre-flight OPTIONS request, resulting in this error (Chrome): Refused to get unsafe header "ETag" (i.e. at https://github.com/TTLabs/EvaporateJS/blob/master/evaporate.js#L325)

I believe the solution is to return Access-Control-Expose-Headers: Etag but I'm not sure how to set that in s3ninja.

Please include instructions for starting the S3 mock server, in terms of:

• Command line instructions (for quick bash runs)
• Java calls (for pure Java integration tests)

I would like to import/export my file from S3 to s3ninja.

Is there a way to do this?

S3Ninja 2.2.2 version in maven requires lib Sirius-ipl-1.1-SNAPSHOT, which is not in maven

S3Controller.delete() will run into an infinite recursion when a non-directory file can not be deleted.

Of course this can happen on Windows when the file was not closed.

The reason /why/ the file wasn't closed apparently is the use of memory mapped I/O; see http://bugs.java.com/view_bug.do?bug_id=4724038.

The following change:

private static void delete(File file) {
    if (file.isDirectory()) {
        if (file.list().length == 0) {
            if (!file.delete()) {
                System.out.println("Directory " + file + " not deleted.");
                System.gc();
            }
        } else {
            String[] files = file.list();

            for (String temp : files) {
                delete(new File(file, temp));
            }
            delete(file);
        }
    } else {
        if (!file.delete()) {
            System.out.println("File " + file + " not deleted.");
            System.gc();
        }
    }
}

makes the tests pass for me; but of course I'm not going to suggest to adopt this verbatim :-).

A saner approach might be to have a code path for Memory-mapped-IO-challenged platforms that uses simple File I/O.

Hi trying to set my client up. But receiving the error.

the http://localhost:9444/ is responding to requests from browser.

Help is much appreciated, Thanks

com.amazonaws.AmazonClientException: Unable to unmarshall response (Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListAllMyBucketsHandler). Response Code: 200, Response Text: OK

` ClientConfiguration clientConfig = new ClientConfiguration();
clientConfig.setProtocol(Protocol.HTTP);

    AmazonS3Client mS3client = new AmazonS3Client(credentials,clientConfig);
    mS3client.setS3ClientOptions(new S3ClientOptions().withPathStyleAccess(true));

    mS3client.setEndpoint("localhost:9444/s3");`

I am curious; why do you not have a single jar file
that contains s3ninja and all its dependencies.
Then one could just do 'java -jar s3ninja.jar'
to start it up.

aws s3 returns a NoSuchKey error code if an object is not found.
http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html

s3ninja just returns a 404 with a html body.
when using the java sdk, the error code is "404 Not Found" iso "NoSuchKey"

I followed the installation instructions, and I get this error when going to localhost:9444:

An unexpected exception occurred: Error load template instance for view.errors.default.html (java.lang.RuntimeException)

I'm running window 7.
Any ideas?

This is the log:

2014-05-20 17:44:21,637 INFO [booting/0|main] sirius - ---------------------------------------------------------
2014-05-20 17:44:21,637 INFO [booting/0|main] sirius - Booting the SIRIUS Framework...
2014-05-20 17:44:21,637 INFO [booting/0|main] sirius - ---------------------------------------------------------
2014-05-20 17:44:21,637 INFO [booting/0|main] sirius - Updated log4j config! To block this, set the system property 'sirius.manual-logging' to true! [-Dsirius.manual-logging=true]
2014-05-20 17:44:21,637 INFO [booting/0|main] sirius - Setting DNS-Cache to 10 seconds...
2014-05-20 17:44:21,639 INFO [booting/0|main] sirius - Setting UTF-8 as default encoding (file.encoding)
2014-05-20 17:44:21,639 INFO [booting/0|main] sirius - Setting ISO-8859-1 as default mime encoding (mail.mime.charset)
2014-05-20 17:44:21,639 INFO [booting/0|main] sirius - ---------------------------------------------------------
2014-05-20 17:44:21,639 INFO [booting/0|main] sirius - Loading config...
2014-05-20 17:44:21,639 INFO [booting/0|main] sirius - ---------------------------------------------------------
2014-05-20 17:44:21,654 INFO [booting/0|main] sirius - using application.conf from classpath...
2014-05-20 17:44:21,679 INFO [booting/0|main] sirius - instance.conf not present work directory
2014-05-20 17:44:21,679 INFO [booting/0|main] sirius - ---------------------------------------------------------
2014-05-20 17:44:21,679 INFO [booting/0|main] sirius - Starting the system...
2014-05-20 17:44:21,679 INFO [booting/0|main] sirius - ---------------------------------------------------------
2014-05-20 17:44:21,697 INFO [booting/0|main] sirius - Initializing the log system:
2014-05-20 17:44:21,699 INFO [booting/0|main] sirius - * Setting ignored to: OFF
2014-05-20 17:44:21,717 INFO [booting/0|main] di - Initializing the MicroKernel....
2014-05-20 17:44:21,717 INFO [booting/0|main] di - Stage 1: Scanning .class files...
2014-05-20 17:44:22,021 INFO [booting/0|main] di - Stage 2: Applying 1 class load actions on 289 classes...
2014-05-20 17:44:22,051 INFO [booting/0|main] di - Stage 3: Enhancing context
2014-05-20 17:44:22,051 INFO [booting/0|main] di - Stage 4: Initializing static parts-references...
2014-05-20 17:44:22,075 INFO [booting/0|main] di - Stage 5: Initializing parts...
2014-05-20 17:44:22,084 INFO [booting/0|default-1] sirius - Starting: timer (System Timer Services)
2014-05-20 17:44:22,084 INFO [booting/0|default-2] sirius - Starting: web (netty HTTP Server)
2014-05-20 17:44:22,084 INFO [booting/0|default-3] sirius - Starting: templates (Rythm)
2014-05-20 17:44:22,084 INFO [booting/0|default-2] web - Initializing netty at port 9444
2014-05-20 17:44:22,089 INFO [booting/0|default-0] sirius - Starting: async (Async Execution Engine)
2014-05-20 17:44:22,189 INFO [booting/0|main] sirius - ---------------------------------------------------------
2014-05-20 17:44:22,190 INFO [booting/0|main] sirius - System is UP and RUNNING - 598 Milliseconds
2014-05-20 17:44:22,190 INFO [booting/0|main] sirius - ---------------------------------------------------------
2014-05-20 17:44:22,190 INFO [booting/0|main] sirius - Opening port 9191 as shutdown listener
2014-05-20 17:46:35,902 ERROR [Kiss-KSP-PC/29|web-mvc-0] errors - An unexpected exception occurred: Error load template instance for view.main.html (java.lang.RuntimeException)
sirius.kernel.health.HandledException: An unexpected exception occurred: Error load template instance for view.main.html (java.lang.RuntimeException)
at sirius.kernel.health.Exceptions$ErrorHandler.handle(Exceptions.java:221)
at sirius.kernel.health.Exceptions.handle(Exceptions.java:319)
at sirius.web.templates.RythmConfig$1.tryLoadTemplate(RythmConfig.java:128)
at org.rythmengine.resource.TemplateResourceManager.tryLoadTemplate(TemplateResourceManager.java:66)
at org.rythmengine.RythmEngine.testTemplate(RythmEngine.java:1394)
at org.rythmengine.internal.CodeBuilder.setExtended(CodeBuilder.java:452)
at org.rythmengine.internal.parser.build_in.ExtendsParser$1$1.call(ExtendsParser.java:82)
at org.rythmengine.internal.CodeBuilder.invokeDirectives(CodeBuilder.java:732)
at org.rythmengine.internal.CodeBuilder.build(CodeBuilder.java:699)
at org.rythmengine.internal.compiler.TemplateClass.buildSourceCode(TemplateClass.java:462)
at org.rythmengine.internal.compiler.TemplateClass.refresh(TemplateClass.java:589)
at org.rythmengine.internal.compiler.TemplateClass.refresh(TemplateClass.java:435)
at org.rythmengine.internal.compiler.TemplateClass.(TemplateClass.java:332)
at org.rythmengine.internal.compiler.TemplateClass.(TemplateClass.java:315)
at org.rythmengine.internal.compiler.TemplateClass.(TemplateClass.java:307)
at org.rythmengine.RythmEngine.getTemplate(RythmEngine.java:836)
at org.rythmengine.RythmEngine.getTemplate(RythmEngine.java:871)
at org.rythmengine.RythmEngine.render(RythmEngine.java:946)
at org.rythmengine.Rythm.render(Rythm.java:265)
at sirius.web.http.Response.template(Response.java:912)
at ninja.NinjaController.onError(NinjaController.java:62)
at ninja.NinjaController.index(NinjaController.java:87)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at sirius.web.controller.ControllerDispatcher$2.run(ControllerDispatcher.java:96)
at sirius.kernel.async.ExecutionBuilder$TaskWrapper.run(ExecutionBuilder.java:68)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.RuntimeException: Error load template instance for view.main.html
at org.rythmengine.internal.compiler.TemplateClass.templateInstance_(TemplateClass.java:379)
at org.rythmengine.internal.compiler.TemplateClass.asTemplate(TemplateClass.java:396)
at org.rythmengine.internal.compiler.TemplateClass.asTemplate(TemplateClass.java:402)
at sirius.web.templates.RythmConfig$1.tryLoadTemplate(RythmConfig.java:126)
... 28 more
Caused by: java.lang.RuntimeException: org.eclipse.jdt.internal.compiler.classfmt.ClassFormatException
at org.rythmengine.internal.compiler.TemplateCompiler$1.findType(TemplateCompiler.java:232)
at org.rythmengine.internal.compiler.TemplateCompiler$1.findType(TemplateCompiler.java:177)
at org.eclipse.jdt.internal.compiler.lookup.LookupEnvironment.askForType(LookupEnvironment.java:113)
at org.eclipse.jdt.internal.compiler.lookup.UnresolvedReferenceBinding.resolve(UnresolvedReferenceBinding.java:49)
at org.eclipse.jdt.internal.compiler.lookup.BinaryTypeBinding.resolveType(BinaryTypeBinding.java:122)
at org.eclipse.jdt.internal.compiler.lookup.LookupEnvironment.getTypeFromTypeSignature(LookupEnvironment.java:1335)
at org.eclipse.jdt.internal.compiler.lookup.LookupEnvironment.getTypeFromVariantTypeSignature(LookupEnvironment.java:1391)
at org.eclipse.jdt.internal.compiler.lookup.LookupEnvironment.getTypeArgumentsFromSignature(LookupEnvironment.java:1172)
at org.eclipse.jdt.internal.compiler.lookup.LookupEnvironment.getTypeFromTypeSignature(LookupEnvironment.java:1340)
at org.eclipse.jdt.internal.compiler.lookup.BinaryTypeBinding.createMethod(BinaryTypeBinding.java:550)
at org.eclipse.jdt.internal.compiler.lookup.BinaryTypeBinding.createMethods(BinaryTypeBinding.java:630)
at org.eclipse.jdt.internal.compiler.lookup.BinaryTypeBinding.cachePartsFrom(BinaryTypeBinding.java:365)
at org.eclipse.jdt.internal.compiler.lookup.LookupEnvironment.createBinaryTypeFrom(LookupEnvironment.java:688)
at org.eclipse.jdt.internal.compiler.lookup.LookupEnvironment.createBinaryTypeFrom(LookupEnvironment.java:667)
at org.eclipse.jdt.internal.compiler.Compiler.accept(Compiler.java:296)
at org.eclipse.jdt.internal.compiler.lookup.LookupEnvironment.askForType(LookupEnvironment.java:118)
at org.eclipse.jdt.internal.compiler.lookup.UnresolvedReferenceBinding.resolve(UnresolvedReferenceBinding.java:49)
at org.eclipse.jdt.internal.compiler.lookup.BinaryTypeBinding.resolveType(BinaryTypeBinding.java:122)
at org.eclipse.jdt.internal.compiler.lookup.PackageBinding.getTypeOrPackage(PackageBinding.java:168)
at org.eclipse.jdt.internal.compiler.lookup.Scope.getPackage(Scope.java:2379)
at org.eclipse.jdt.internal.compiler.ast.ParameterizedQualifiedTypeReference.internalResolveLeafType(ParameterizedQualifiedTypeReference.java:138)
at org.eclipse.jdt.internal.compiler.ast.ParameterizedQualifiedTypeReference.internalResolveType(ParameterizedQualifiedTypeReference.java:132)
at org.eclipse.jdt.internal.compiler.ast.ParameterizedQualifiedTypeReference.resolveType(ParameterizedQualifiedTypeReference.java:351)
at org.eclipse.jdt.internal.compiler.lookup.SourceTypeBinding.resolveTypesFor(SourceTypeBinding.java:1538)
at org.eclipse.jdt.internal.compiler.lookup.SourceTypeBinding.methods(SourceTypeBinding.java:1175)
at org.eclipse.jdt.internal.compiler.lookup.SourceTypeBinding.faultInTypesForFieldsAndMethods(SourceTypeBinding.java:678)
at org.eclipse.jdt.internal.compiler.lookup.CompilationUnitScope.faultInTypes(CompilationUnitScope.java:469)
at org.eclipse.jdt.internal.compiler.Compiler.process(Compiler.java:756)
at org.eclipse.jdt.internal.compiler.Compiler.compile(Compiler.java:468)
at org.rythmengine.internal.compiler.TemplateCompiler.compile(TemplateCompiler.java:333)
at org.rythmengine.internal.compiler.TemplateClass.compile(TemplateClass.java:645)
at org.rythmengine.internal.compiler.TemplateClassLoader.loadTemplateClass(TemplateClassLoader.java:306)
at org.rythmengine.internal.compiler.TemplateClassLoader.loadClass(TemplateClassLoader.java:252)
at org.rythmengine.internal.compiler.TemplateClass.getJavaClass(TemplateClass.java:354)
at org.rythmengine.internal.compiler.TemplateClass.templateInstance_(TemplateClass.java:370)
... 31 more
Caused by: org.eclipse.jdt.internal.compiler.classfmt.ClassFormatException
at org.eclipse.jdt.internal.compiler.classfmt.ClassFileReader.(ClassFileReader.java:372)
at org.rythmengine.internal.compiler.TemplateCompiler$1.findStandType(TemplateCompiler.java:197)
at org.rythmengine.internal.compiler.TemplateCompiler$1.findType(TemplateCompiler.java:211)
... 65 more

Hi,
Worked like a Ninja for a couple of months, but suddenly it doesn't anymore (I might have changed something, I'm not sure)...
What happens is that all GetObject requests get "rejected" (401) whne calling via java client. Requests via clicking on same object in web view work fine.
Bucket is public. I see nothing special in the logs.

My calling code is as follows (it hasn't changed recently, but I did "play" with the project settings):

ClientConfiguration clientConfiguration = new ClientConfiguration();
client = new AmazonS3Client(clientConfiguration.withSocketTimeout(0));
client.setEndpoint("http://localhost:9444/s3/");
client.setS3ClientOptions(new S3ClientOptions().withPathStyleAccess(true));
...
client.getObject(new GetObjectRequest(sourceBucketName, sourceKey), tmpfile);

Can you please advise?

This is preventing maven resolution from working. Is it possible to to change the dependency to a non-snapshot?

Hi, our client app (java) is forcing HTTPS to S3 so in order to mock it I must provide HTTPS support.
I can go with local http server that proxies the requests to localhost:9444 but thought maybe you have another option to do that?

Should Sirius.stop() set the initialized boolean to false?

My use case is a set of automated integration tests run in junit. As part of my @Before setup, I initialize an instance of S3Ninja, and I shut it down in my @After cleanup.

However, it looks like after I call Sirius.stop() in my cleanup, the next test that runs that tries to spin up S3Ninja again, skips the init() method because I think the static volatile booleaninitialized value is still set to true.

Am I using S3Ninja incorrectly? I'd like to be able to bring up an entirely clean context for the integration tests to run in.

Thanks.

It would be nice to be able to launch/stop the server from the test by using some Java class. There shouldn't be any external dependency other than a jar (that could be available in maven).

Just started playing with s3ninja.

I'm basically doing a PUT to a private bucket, followed by a GET, using the default s3ninja access key and secret key.

I'm using the AWS Java SDK S3 client, version 1.9.21.

I had to enable path-style access to get the PUT to work.

When I try the GET, the AmazonS3Client logs a HTTP 401 error.

I see "Rejected" in the s3ninja logs, but not much explanation.

Looking the s3ninja code, I'm guessing something's gone wrong with the auth hash calc.

Help!

p.s. From the browser, I was able to do the GET by adding "noAuth" to the query params...but that's not much help with the Java S3 client.

took the unit tests from the sources, and converted it to scala.

tried to run the code against the latest s3ninja version (2.70).

the following exception is thrown:

Exception in thread "main" com.amazonaws.services.s3.model.AmazonS3Exception: Unauthorized (Service: Amazon S3; Status Code: 401; Error Code: 401 Unauthorized; Request ID: null), S3 Extended Request ID: null
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1305)
at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:852)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:630)
at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:405)
at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:367)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:318)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3785)
at com.amazonaws.services.s3.AmazonS3Client.initiateMultipartUpload(AmazonS3Client.java:2702)
at com.amazonaws.services.s3.transfer.internal.UploadCallable.initiateMultipartUpload(UploadCallable.java:350)
at com.amazonaws.services.s3.transfer.internal.UploadCallable.uploadInParts(UploadCallable.java:178)
at com.amazonaws.services.s3.transfer.internal.UploadCallable.call(UploadCallable.java:121)
at com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:139)
at com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:47)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

I am trying to execute the following code:

import java.io.ByteArrayInputStream

import com.amazonaws.ClientConfiguration
import com.amazonaws.auth.AWSCredentials
import com.amazonaws.auth.BasicAWSCredentials
import com.amazonaws.services.s3.AmazonS3Client
import com.amazonaws.services.s3.S3ClientOptions
import com.amazonaws.services.s3.model.AmazonS3Exception
import com.amazonaws.services.s3.model.ObjectMetadata
import com.amazonaws.services.s3.transfer.TransferManager
import com.amazonaws.services.s3.transfer.TransferManagerConfiguration
import com.google.common.base.Charsets
import com.google.common.io.ByteStreams
//import sirius.kernel.BaseSpecification


/**
  * Created by eran.mizrahi on 23/05/2016.
  */
object Test1 {
  def getClient():AmazonS3Client = {
    val credentials = new BasicAWSCredentials("AKIAIOSFODNN7EXAMPLE", "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY");
    val newClient = new AmazonS3Client(credentials,
      new ClientConfiguration());
    newClient.setS3ClientOptions(S3ClientOptions.builder().setPathStyleAccess(true).build())// withPathStyleAccess(true));
    newClient.setEndpoint("http://localhost:9444/s3");

    return newClient;
  }

  def main(args: Array[String]) {
    val client = getClient();

    //"PUT and then HEAD bucket as expected with AWS4 signer"
    if (client.doesBucketExist("test")) {
      client.deleteBucket("test");
    }
    client.createBucket("test");
    client.doesBucketExist("test")


    //"PUT and then DELETE bucket as expected with AWS4 signer"
    {
      if (!client.doesBucketExist("test")) {
        client.createBucket("test");
      }
      client.deleteBucket("test");
      !client.doesBucketExist("test")
    }

    //"PUT and then GET work as expected with AWS4 signer"
    {
      client.putObject("test", "test", new ByteArrayInputStream("Test".getBytes(Charsets.UTF_8)), new ObjectMetadata());
      val content = new String(ByteStreams.toByteArray(client.getObject("test", "test").getObjectContent()), Charsets.UTF_8);
      content == "Test"
    }

    //"MultipartUpload and then GET work as expected with AWS4 signer"
    {
      val transfer = new TransferManager(client);
      val config = new TransferManagerConfiguration();
      val meta = new ObjectMetadata();
      val message = "Test".getBytes(Charsets.UTF_8);
      config.setMultipartUploadThreshold(1l);
      config.setMinimumUploadPartSize(1);
      transfer.setConfiguration(config);
      meta.setContentLength(message.length);
      val upload = transfer.upload("test", "test", new ByteArrayInputStream("Test".getBytes(Charsets.UTF_8)), meta);
      upload.waitForUploadResult();
      val content = new String(ByteStreams.toByteArray(client.getObject("test", "test").getObjectContent()), Charsets.UTF_8);
      content == "Test"
    }
  }
}

I wonder if S3 ninja could collaborate with S3Proxy? S3Proxy offers a high degree of S3 compatibility which would address several of the issues reported. This could be as invasive as the S3 ninja UI wrapping S3Proxy (both are written in Java) or more cooperative as using S3Proxy and s3-tests as a compatibility yardstick.

As s3ninja is often used in integration tests, it would be helpful to accept any S3 credential values, to simplify test setup code and configuration.

Background

The project https://github.com/CodeStory/fluent-http provides a great API for testing:

public abstract class ConfigMatrixDownloadAbstractTest  {
  /**
   * Start the webserver and retrieve the random port number.
   */
  private int serverPort = new WebServer().startOnRandomPort().port();

• Creates a random port number during tests, allowing Parallel tests to be executed
• Serve static contents from src/test/resources/app
• Serves dynamic content bound by design
• It's a simple, yet powerful, way to implement integration tests

Proposal

• Implement an embedded class S3NinjaWebServer that uses the S3Ninja backend
  • See Routes https://github.com/CodeStory/fluent-http#more-routes
• Create a S3NinjaTest class that spawns the Embedded server in the background during test
• Mock the AmazonS3 Client, setting the http://localhost:(random-port) computed from the S3NinjaTest class https://github.com/MeteoGroup/s3mock/blob/master/test/src/test/java/org/meteogroup/s3mock/S3MockTest.java#L54

For integration test it will be really useful to have it published to maven central.
Ideally will be to have a maven plugin for it.

Thanks in advance

Getting this when initializing the system. Not sure how to correct.

10:45:28.601 INFO [booting/0|main] sirius - ---------------------------------------------------------
10:45:28.601 INFO [booting/0|main] sirius - System is STARTING...
10:45:28.601 INFO [booting/0|main] sirius - ---------------------------------------------------------
10:45:28.601 INFO [booting/0|main] sirius - Loading config...
10:45:28.601 INFO [booting/0|main] sirius - ---------------------------------------------------------
10:45:28.709 INFO [booting/0|main] sirius - using application.conf from classpath...
10:45:28.848 INFO [booting/0|main] sirius - test.conf not present in classpath
10:45:28.848 INFO [booting/0|main] sirius - ---------------------------------------------------------
10:45:28.848 INFO [booting/0|main] sirius - Starting the system...
10:45:28.848 INFO [booting/0|main] sirius - ---------------------------------------------------------
10:45:28.988 INFO [booting/0|main] sirius - Initializing the log system:
10:45:29.004 INFO [booting/0|main] sirius - * Setting velocity to: WARN
10:45:29.004 INFO [booting/0|main] sirius - * Setting ignored to: OFF
10:45:29.004 INFO [booting/0|main] sirius - Enabled 2 of 2 frameworks...
10:45:29.004 INFO [booting/0|main] sirius - Active Customizations: []
10:45:29.065 INFO [booting/0|main] di - Initializing the MicroKernel....
10:45:29.065 INFO [booting/0|main] di - ~ Scanning .class files...
10:45:29.096 ERROR [booting/0|main] di - A system error occurred: Failed to load dependent class: ninja.NinjaExtension
sirius.kernel.health.HandledException: A system error occurred: Failed to load dependent class: ninja.NinjaExtension
at sirius.kernel.health.Exceptions$ErrorHandler.handle(Exceptions.java:218)
at sirius.kernel.di.Injector.lambda$init$15(Injector.java:99)
at sirius.kernel.di.Injector$$Lambda$16/275266973.accept(Unknown Source)
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:270)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:512)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:502)
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
at sirius.kernel.di.Injector.init(Injector.java:75)
at sirius.kernel.Sirius.init(Sirius.java:217)
at sirius.kernel.Sirius.start(Sirius.java:302)
at test.java.com.lmco.sgp.filemover.main.UnitTest.setUP(UnitTest.java:35)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
Caused by: java.lang.NoClassDefFoundError: sirius/web/templates/rythm/RythmExtension
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:455)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:367)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:344)
at sirius.kernel.di.Injector.lambda$init$15(Injector.java:80)
... 53 more
Caused by: java.lang.ClassNotFoundException: sirius.web.templates.rythm.RythmExtension
at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 68 more

Today it seems the files are persisted to disk as a /

Some filesystems have a harder problem with many many files per folder so usually a file is broken into several subfolders per his md5 bytes.

Have you considered something like that for S3Ninja?

I'm getting failures during a HEAD request while using the AWS SDK. The version is aws-sdk-core-2.1.4.

Net::HTTPBadResponse:wrong status line: "<html>"
/home/ericu/.rvm/rubies/jruby-1.7.19/lib/ruby/1.9/net/http.rb:2573:in `read_status_line'
/home/ericu/.rvm/rubies/jruby-1.7.19/lib/ruby/1.9/net/http.rb:2560:in `read_new'
/home/ericu/.rvm/rubies/jruby-1.7.19/lib/ruby/1.9/net/http.rb:2072:in `wait_for_continue'
/home/ericu/.rvm/rubies/jruby-1.7.19/lib/ruby/1.9/net/http.rb:1956:in `send_request_with_body_stream'
/home/ericu/.rvm/rubies/jruby-1.7.19/lib/ruby/1.9/net/http.rb:1930:in `exec'
/home/ericu/.rvm/gems/jruby-1.7.19@community/gems/aws-sdk-core-2.1.24/lib/seahorse/client/net_http/patches.rb:78:in `new_transport_request'
org/jruby/RubyKernel.java:1270:in `catch'
/home/ericu/.rvm/gems/jruby-1.7.19@community/gems/aws-sdk-core-2.1.24/lib/seahorse/client/net_http/patches.rb:77:in `new_transport_request'
/home/ericu/.rvm/rubies/jruby-1.7.19/lib/ruby/1.9/net/http.rb:1302:in `request'
/home/ericu/.rvm/gems/jruby-1.7.19@community/gems/newrelic_rpm-3.9.8.273/lib/new_relic/agent/instrumentation/net.rb:27:in `request_with_newrelic_trace'
/home/ericu/.rvm/gems/jruby-1.7.19@community/gems/newrelic_rpm-3.9.8.273/lib/new_relic/agent.rb:453:in `disable_all_tracing'
/home/ericu/.rvm/gems/jruby-1.7.19@community/gems/newrelic_rpm-3.9.8.273/lib/new_relic/agent/instrumentation/net.rb:26:in `request_with_newrelic_trace'
/home/ericu/.rvm/gems/jruby-1.7.19@community/gems/newrelic_rpm-3.9.8.273/lib/new_relic/agent/cross_app_tracing.rb:48:in `tl_trace_http_request'
/home/ericu/.rvm/gems/jruby-1.7.19@community/gems/newrelic_rpm-3.9.8.273/lib/new_relic/agent/instrumentation/net.rb:23:in `request_with_newrelic_trace'
/home/ericu/.rvm/gems/jruby-1.7.19@community/gems/aws-sdk-core-2.1.24/lib/seahorse/client/net_http/connection_pool.rb:337:in `request'
/home/ericu/.rvm/gems/jruby-1.7.19@community/gems/aws-sdk-core-2.1.24/lib/seahorse/client/net_http/handler.rb:61:in `transmit'
/home/ericu/.rvm/gems/jruby-1.7.19@community/gems/aws-sdk-core-2.1.24/lib/seahorse/client/net_http/handler.rb:110:in `session'
/home/ericu/.rvm/gems/jruby-1.7.19@community/gems/aws-sdk-core-2.1.24/lib/seahorse/client/net_http/connection_pool.rb:108:in `session_for'
/home/ericu/.rvm/gems/jruby-1.7.19@community/gems/aws-sdk-core-2.1.24/lib/seahorse/client/net_http/handler.rb:108:in `session'
/home/ericu/.rvm/gems/jruby-1.7.19@community/gems/aws-sdk-core-2.1.24/lib/seahorse/client/net_http/handler.rb:60:in `transmit'
/home/ericu/.rvm/gems/jruby-1.7.19@community/gems/aws-sdk-core-2.1.24/lib/seahorse/client/net_http/handler.rb:34:in `call'
/home/ericu/.rvm/gems/jruby-1.7.19@community/gems/aws-sdk-core-2.1.24/lib/seahorse/client/plugins/content_length.rb:12:in `call'
/home/ericu/.rvm/gems/jruby-1.7.19@community/gems/aws-sdk-core-2.1.24/lib/aws-sdk-core/plugins/s3_request_signer.rb:149:in `call'
/home/ericu/.rvm/gems/jruby-1.7.19@community/gems/aws-sdk-core-2.1.24/lib/aws-sdk-core/plugins/s3_request_signer.rb:33:in `call'
/home/ericu/.rvm/gems/jruby-1.7.19@community/gems/aws-sdk-core-2.1.24/lib/aws-sdk-core/xml/error_handler.rb:8:in `call'
/home/ericu/.rvm/gems/jruby-1.7.19@community/gems/aws-sdk-core-2.1.24/lib/aws-sdk-core/plugins/s3_request_signer.rb:124:in `call'
/home/ericu/.rvm/gems/jruby-1.7.19@community/gems/aws-sdk-core-2.1.24/lib/aws-sdk-core/plugins/s3_redirects.rb:15:in `call'
/home/ericu/.rvm/gems/jruby-1.7.19@community/gems/aws-sdk-core-2.1.24/lib/aws-sdk-core/plugins/retry_errors.rb:87:in `call'
/home/ericu/.rvm/gems/jruby-1.7.19@community/gems/aws-sdk-core-2.1.24/lib/aws-sdk-core/plugins/s3_md5s.rb:33:in `call'
/home/ericu/.rvm/gems/jruby-1.7.19@community/gems/aws-sdk-core-2.1.24/lib/aws-sdk-core/plugins/s3_expect_100_continue.rb:21:in `call'
/home/ericu/.rvm/gems/jruby-1.7.19@community/gems/aws-sdk-core-2.1.24/lib/aws-sdk-core/rest/handler.rb:7:in `call'
/home/ericu/.rvm/gems/jruby-1.7.19@community/gems/aws-sdk-core-2.1.24/lib/aws-sdk-core/plugins/user_agent.rb:12:in `call'
/home/ericu/.rvm/gems/jruby-1.7.19@community/gems/aws-sdk-core-2.1.24/lib/seahorse/client/plugins/endpoint.rb:34:in `call'
/home/ericu/.rvm/gems/jruby-1.7.19@community/gems/aws-sdk-core-2.1.24/lib/aws-sdk-core/plugins/param_validator.rb:21:in `call'
/home/ericu/.rvm/gems/jruby-1.7.19@community/gems/aws-sdk-core-2.1.24/lib/seahorse/client/plugins/raise_response_errors.rb:14:in `call'
/home/ericu/.rvm/gems/jruby-1.7.19@community/gems/aws-sdk-core-2.1.24/lib/aws-sdk-core/plugins/s3_sse_cpk.rb:18:in `call'
/home/ericu/.rvm/gems/jruby-1.7.19@community/gems/aws-sdk-core-2.1.24/lib/aws-sdk-core/plugins/param_converter.rb:21:in `call'
/home/ericu/.rvm/gems/jruby-1.7.19@community/gems/aws-sdk-core-2.1.24/lib/seahorse/client/plugins/response_target.rb:21:in `call'
/home/ericu/.rvm/gems/jruby-1.7.19@community/gems/aws-sdk-core-2.1.24/lib/seahorse/client/request.rb:70:in `send_request'
/home/ericu/.rvm/gems/jruby-1.7.19@community/gems/aws-sdk-core-2.1.24/lib/seahorse/client/base.rb:207:in `put_object'
/home/ericu/.rvm/gems/jruby-1.7.19@community/gems/aws-sdk-resources-2.1.24/lib/aws-sdk-resources/request.rb:24:in `call'
/home/ericu/.rvm/gems/jruby-1.7.19@community/gems/aws-sdk-resources-2.1.24/lib/aws-sdk-resources/operations.rb:41:in `call'
/home/ericu/.rvm/gems/jruby-1.7.19@community/gems/aws-sdk-resources-2.1.24/lib/aws-sdk-resources/operation_methods.rb:19:in `put'
/home/ericu/resourceful-attributes/lib/resourceful_attributes/s3.rb:39:in `write'

The code at the bottom of the stack trace is boring and looks like this

def write(k, v, options = nil)
      bucket.object(k).put({ :body => v.read })
      self
end

It blows up in the call to put from what I can tell. This same code works w/o issue with the fakes3 gem.

Let me know if you need additional info to debug this.

Ensure listObjects() works as expected

Is there anyway to make the endpoint something like "http://localhost:9444"? i.e. not including the /s3 at the end?

It have taken me some time to figure out that I need to install Java 8 in addition of the versions I had before.
The last version is compiled with Java 8!

Hello,

your project looks awesome. I would like to build it on my own. But when I look at the project, the landing page (http://s3ninja.net/) or the github wiki, I was not able to find a docu how to build the project.

Where can I find this guide?

Thanks!

Looks like sending k/e/y breaks it into k_e_y.

This make the following keys the same, which could create problems:

a/b
a_b

S3 treats them different. S3Ninja, the same.

Sorry to bother, but I am having trouble connection to my Local S3 Ninja.I can go to
http://localhost:9444/ just fine, but when I set do
java s3.setEndpoint(http://localhost:9444/s3) all calls fail with a 404. I've also tried setting the Client configuration object with

ClientConfiguration config = new ClientConfiguration()
                .withMaxConnections(50)
                .withProxyHost("localhost")
                .withProxyPort(9444)
                .withProtocol(Protocol.HTTP);

But no Joy. The website also shows no API calls being made. To be clear I have never gotten this working before, so I may be missing something.

Any tips?