Documentation on submitting multiple POST requests about parallec HOT 16 CLOSED

@prasenjithaty you can use your customized async http client with the config to set retry for requests. We have not supported other low level retry such as the interceptors but we can possibly enhance it to make it work like that by enhancing the task worker and the requesting worker

Another way to set general retry is to only retry on those failed requests with another parallel task only targeting. (we can start the 2nd parallel task in another thread by polling a hashmap storing only those failed the 1st attempt)

https://asynchttpclient.github.io/async-http-client/apidocs/com/ning/http/client/AsyncHttpClientConfig.Builder.html#setMaxRequestRetry(int)

from parallec.

@prasenjithaty can you see if this helps? http://www.parallec.io/docs/submit-task/#apis-on-variable-replacement-for-heterogeneous-requests

from parallec.

Hi @jeffpeiyt,

The example you shared is for heterogeneous requests to different hosts. How about doing the same with a single host? I'm not really able to figure out a way of doing this. Also javadoc on ParallelClient#setReplace* methods would be pretty useful for users.

Thanks.

from parallec.

can you try https://github.com/eBay/parallec-samples/blob/master/sample-apps/src/main/java/io/parallec/sample/app/http/HttpDiffRequestsSameServerApp.java

but put the $VAR_NAME inside of the string in setHttpEntityBody

if you scroll down a bit you can see:

Different requests to the same target host

Here is the example of hitting 2 different APIs to the same target host. $ZIP is the variable being replaced. setReplaceVarMapToSingleTargetSingleVar(String variable, List replaceList, String uniformTargetHost) is the API to use. There are more complex replacements APIs available in javadoc. Complete sample code is here.

http://www.parallec.io/userdata/sample_weather_48824.txt
http://www.parallec.io/userdata/sample_weather_95037.txt
pc.prepareHttpGet("/userdata/sample_weather_$ZIP.txt")
.setReplaceVarMapToSingleTargetSingleVar("ZIP",
Arrays.asList("95037","48824"), "www.parallec.io")
.setResponseContext(responseContext)
.execute(new ParallecResponseHandler() {...}...

from parallec.

but put the $VAR_NAME inside of the string in setHttpEntityBody

^^ was the right pointer.

Thanks. Got it working now. Now, how can I collect all the responses?

from parallec.

Great! Can you check examples from https://github.com/eBay/parallec-samples/tree/master/sample-apps/src/main/java/io/parallec/sample/app/http ?

http://www.parallec.io/docs/track-status/

from parallec.

I already checked those. But, I want to set the key in the map dynamically. Maybe fileURL from my first post in the request body

from parallec.

@prasenjithaty is your question on aggregating/collecting responses or setting the variables in different requests?

from parallec.

My question is to collect responses with the key being a field from my request body.

from parallec.

I thought I figured out with the below code, but it breaks during unmarshalling

.execute(new ParallecResponseHandler() {
    public void onCompleted(ResponseOnSingleTask res,
            Map<String, Object> responseContext) {
        String responseContent = res
                .getResponseContent();
        TaskRequest taskRequest = res.getRequest();
        LOGGER.info("post data = {}", taskRequest.getPostData());
        LOGGER.info("post body = {}", taskRequest.getRequestContent());
        MetadataRequest metadataRequest = JAXB.unmarshal(taskRequest.getRequestContent(), MetadataRequest.class);
        LOGGER.info("!!Response: {}, TargetHost: ", responseContent, res.getHost());
        responseContext.put(metadataRequest.getFileUrl(), responseContent);
    }
});

However if, I write responseContext.put(UUID.randomUUID().toString(), responseContent); I'm able to collect the responses.

from parallec.

Will take a look later today

from parallec.

@prasenjithaty Can you log/print the response content rather unmarshal the request? https://github.com/eBay/parallec-samples/blob/master/sample-apps/src/main/java/io/parallec/sample/app/http/HttpBasicMinimumApp.java#L43

For this question My question is to collect responses with the key being a field from my request body.

I suggest you pass the response context into the handler with a hashmap. In the response handler to update the hashmap with key as the field, value is the response ( or the part you need from the response).

Then when you finish ( outside of the handler) to check on this hashmap from the context again to parse the aggregated data

here is an example: https://github.com/eBay/parallec/blob/master/src/test/java/io/parallec/core/main/http/ParallelClientHttpBasicTest.java#L105

from parallec.

@jeffpeiyt log/printing the response content prints the responses perfectly fine. It breaks only when I try to unmarshal the request.

And I've passed response context as a hashmap into the handler as given in the example. if you see my first post on Ln.6 that is what I've done.

from parallec.

Figured out the issue. Problem was while unmarshalling. Got it working finally. Thanks for your help @jeffpeiyt

from parallec.

@prasenjithaty great! Thanks for trying parallec.io!

from parallec.

@jeffpeiyt does the library support an easy way of retrying requests? Like the interceptors in OkHttp? https://github.com/square/okhttp/wiki/Interceptors

from parallec.