Comments (12)
brunoribeiro
commented on May 27, 2024
Is there any workaround to get the headers from the ResponseOnSingleTask?
from parallec.
jeffpeiyt
commented on May 27, 2024
@brunoribeiro thanks so much for trying parallec.
The code changes should be straightforward, but I do not see a workaround directly.
How soon will you need this feature? Let me plan the dev work accordingly.
from parallec.
brunoribeiro
commented on May 27, 2024
@jeffpeiyt Thank you for the quick reply. I would love to have it before November, as i'm evaluating parallec for a new feature to be release in December.
Can you clarify me the needed code changes to make it work immediately?
from parallec.
jeffpeiyt
commented on May 27, 2024
EDITED.
Sure. Roughly the following. I will see if I can get them by the end of this week.
- Add the similar code as follows into http worker. Then try to save the needed part of the original header map
FluentCaseInsensitiveStringsMapinto the ResponseOnSingeRequest . probably usingMap<String, List<String>>to save the header map. - Add a this header map of responseHeaders into ResponseOnSingeRequest and ResponseOnSingleTask
- in Task builder : add option .saveResponseHeader(Set keys, boolean getAll) ; default as to not to save the header
- Corresponding tests to maintain the coverage etc.
etc.
from parallec.
jeffpeiyt
commented on May 27, 2024
@brunoribeiro One concern is that saving all the headers will take quite some spaces in memory. Are you looking for a specific pair in the header or want to check all of them?
from parallec.
brunoribeiro
commented on May 27, 2024
@jeffpeiyt the one I am looking is specific to my server implementation, maybe we could pass a list of header keys to be returned if present, but this will surely make it slower as it need to check all them to find the matches.
from parallec.
jeffpeiyt
commented on May 27, 2024
@brunoribeiro yes. I am thinking the same way. As long as the needed keys are provided, it is O(K) to get it out where K is the number of needed keys. Most time we may be just interested in 1 or 2 keys in the headers. (K is very small) So I wound not worry much. Also by default we do not fetch this; and this is done in parallel (http worker).
from parallec.
jeffpeiyt
commented on May 27, 2024
Done features and basic test .saveResponseHeaders(new ResponseHeaderMeta(null, true))
@Test
public void hitWebsitesMinSyncWithAllResponses() {
Map<String, Object> responseContext = new HashMap<String, Object>();
pc
.prepareHttpGet("/validateInternals.html")
.setConcurrency(1700)
.handleInWorker()
.saveResponseHeaders(new ResponseHeaderMeta(null, true))
.setTargetHostsFromString(
"www.parallec.io www.jeffpei.com www.restcommander.com")
.execute(new ParallecResponseHandler() {
@Override
public void onCompleted(ResponseOnSingleTask res,
Map<String, Object> responseContext) {
Map<String, List<String>> responseHeaders = res.getResponseHeaders();
for(Entry<String, List<String>> entry: responseHeaders.entrySet()){
logger.info("response header: {} - {}", entry.getKey(), entry.getValue());
}
responseContext.put(res.getHost(), responseHeaders.size());
logger.debug(res.toString());
}
});
for (Object o : responseContext.values()) {
int headerKeySize = Integer.parseInt((String) o);
Asserts.check(headerKeySize > 0,
" Fail to extract http header");
}
//logger.info("Task Pretty Print: \n{}", task.prettyPrintInfo());
}
14:01:33.525 [main] INFO i.p.c.ParallelTaskBuilder - Executing task PT_3_20160929140133524_44c749f0-058 in SYNC mode...
14:01:33.525 [Thread-1] INFO i.p.c.t.ParallelTaskManager - Added task PT_3_20160929140133524_44c749f0-058 to the running inprogress map...
14:01:33.527 [Thread-1] INFO i.p.c.t.ParallelTaskManager - !!STARTED sendAgentCommandToManager : PT_3_20160929140133524_44c749f0-058 at 2016-09-29 14:01:33.527-0700
14:01:33.531 [ParallecActorSystem-akka.actor.default-dispatcher-3] INFO i.p.c.a.ExecutionManager - parallec task state : IN_PROGRESS
14:01:33.531 [ParallecActorSystem-akka.actor.default-dispatcher-3] INFO i.p.c.a.ExecutionManager - Before Safety Check: total entry count: 3
14:01:33.531 [ParallecActorSystem-akka.actor.default-dispatcher-3] INFO i.p.c.a.ExecutionManager - After Safety Check: total entry count in nodeDataMapValidSafe: 3
14:01:33.531 [ParallecActorSystem-akka.actor.default-dispatcher-3] INFO i.p.c.a.ExecutionManager - !Obtain command request for target host meta id THM_3_20160929140133523_d3583d40-9dd with count: 3
14:01:33.544 [ParallecActorSystem-akka.actor.default-dispatcher-4] INFO i.p.c.a.AssistantExecutionManager - Now finished sending all needed messages. Done job of ASST Manager at 2016.09.29.14.01.33.543-0700
14:01:33.714 [ParallecActorSystem-akka.actor.default-dispatcher-2] INFO i.p.c.TestBase - response header: x-cache - [HIT]
14:01:33.714 [ParallecActorSystem-akka.actor.default-dispatcher-2] INFO i.p.c.TestBase - response header: server - [GitHub.com]
14:01:33.714 [ParallecActorSystem-akka.actor.default-dispatcher-2] INFO i.p.c.TestBase - response header: access-control-allow-origin - [*]
14:01:33.715 [ParallecActorSystem-akka.actor.default-dispatcher-2] INFO i.p.c.TestBase - response header: connection - [keep-alive]
14:01:33.715 [ParallecActorSystem-akka.actor.default-dispatcher-2] INFO i.p.c.TestBase - response header: last-modified - [Fri, 23 Sep 2016 23:56:04 GMT]
14:01:33.715 [ParallecActorSystem-akka.actor.default-dispatcher-2] INFO i.p.c.TestBase - response header: date - [Thu, 29 Sep 2016 21:01:33 GMT]
14:01:33.715 [ParallecActorSystem-akka.actor.default-dispatcher-2] INFO i.p.c.TestBase - response header: via - [1.1 varnish]
14:01:33.715 [ParallecActorSystem-akka.actor.default-dispatcher-2] INFO i.p.c.TestBase - response header: accept-ranges - [bytes]
14:01:33.715 [ParallecActorSystem-akka.actor.default-dispatcher-2] INFO i.p.c.TestBase - response header: cache-control - [max-age=600]
14:01:33.715 [ParallecActorSystem-akka.actor.default-dispatcher-2] INFO i.p.c.TestBase - response header: x-served-by - [cache-den6024-DEN]
14:01:33.715 [ParallecActorSystem-akka.actor.default-dispatcher-2] INFO i.p.c.TestBase - response header: vary - [Accept-Encoding]
14:01:33.715 [ParallecActorSystem-akka.actor.default-dispatcher-2] INFO i.p.c.TestBase - response header: expires - [Thu, 29 Sep 2016 20:52:11 GMT]
14:01:33.715 [ParallecActorSystem-akka.actor.default-dispatcher-2] INFO i.p.c.TestBase - response header: content-length - [620]
14:01:33.715 [ParallecActorSystem-akka.actor.default-dispatcher-2] INFO i.p.c.TestBase - response header: x-cache-hits - [1]
14:01:33.715 [ParallecActorSystem-akka.actor.default-dispatcher-2] INFO i.p.c.TestBase - response header: x-fastly-request-id - [489951baad8349dd30ea8707d4eaa5e10992bec1]
14:01:33.715 [ParallecActorSystem-akka.actor.default-dispatcher-2] INFO i.p.c.TestBase - response header: age - [559]
14:01:33.715 [ParallecActorSystem-akka.actor.default-dispatcher-2] INFO i.p.c.TestBase - response header: x-github-request-id - [C71B4E17:332B:9A3F9A0:57ED7CA2]
14:01:33.715 [ParallecActorSystem-akka.actor.default-dispatcher-2] INFO i.p.c.TestBase - response header: content-type - [text/html; charset=utf-8]
14:01:33.718 [ParallecActorSystem-akka.actor.default-dispatcher-6] INFO i.p.c.a.ExecutionManager -
[1]__RESP_RECV_IN_MGR 1 (+2) / 3 (33.333%) AFT 0.187 S @ www.parallec.io @ 2016.09.29.14.01.33.717-0700 , TaskID : 44c749f0-058 , CODE: 200 OK, RESP_BRIEF: <!DOCTYPE html>
<html><head><met
14:01:33.830 [ParallecActorSystem-akka.actor.default-dispatcher-6] INFO i.p.c.TestBase - response header: server - [AmazonS3]
14:01:33.830 [ParallecActorSystem-akka.actor.default-dispatcher-6] INFO i.p.c.TestBase - response header: etag - ["7cc07a9153ea2e01e915fcfee0c921ba"]
14:01:33.830 [ParallecActorSystem-akka.actor.default-dispatcher-6] INFO i.p.c.TestBase - response header: last-modified - [Wed, 09 Apr 2014 06:34:54 GMT]
14:01:33.830 [ParallecActorSystem-akka.actor.default-dispatcher-6] INFO i.p.c.TestBase - response header: x-amz-request-id - [873E27207EE5B946]
14:01:33.830 [ParallecActorSystem-akka.actor.default-dispatcher-6] INFO i.p.c.TestBase - response header: content-length - [594]
14:01:33.831 [ParallecActorSystem-akka.actor.default-dispatcher-6] INFO i.p.c.TestBase - response header: x-amz-id-2 - [lYm0wcoroScpmYgnbl3+85knVVigMkHhzCzcCeVXabw4NJPJhBcGnLNUBlPzstGSMzGeqbopVkg=]
14:01:33.831 [ParallecActorSystem-akka.actor.default-dispatcher-6] INFO i.p.c.TestBase - response header: date - [Thu, 29 Sep 2016 21:01:34 GMT]
14:01:33.831 [ParallecActorSystem-akka.actor.default-dispatcher-6] INFO i.p.c.TestBase - response header: content-type - [text/html]
14:01:33.832 [ParallecActorSystem-akka.actor.default-dispatcher-2] INFO i.p.c.a.ExecutionManager -
[2]__RESP_RECV_IN_MGR 2 (+1) / 3 (66.667%) AFT 0.301 S @ www.restcommander.com @ 2016.09.29.14.01.33.831-0700 , TaskID : 44c749f0-058 , CODE: 200 OK, RESP_BRIEF: <!DOCTYPE html>
<html>
<body>
14:01:34.891 [ParallecActorSystem-akka.actor.default-dispatcher-2] INFO i.p.c.TestBase - response header: server - [AmazonS3]
14:01:34.891 [ParallecActorSystem-akka.actor.default-dispatcher-2] INFO i.p.c.TestBase - response header: etag - ["538a4857516afb35dd416dacea1a4b1d"]
14:01:34.891 [ParallecActorSystem-akka.actor.default-dispatcher-2] INFO i.p.c.TestBase - response header: last-modified - [Wed, 09 Apr 2014 06:37:52 GMT]
14:01:34.891 [ParallecActorSystem-akka.actor.default-dispatcher-2] INFO i.p.c.TestBase - response header: x-amz-request-id - [6384E719DE34F682]
14:01:34.891 [ParallecActorSystem-akka.actor.default-dispatcher-2] INFO i.p.c.TestBase - response header: content-length - [725]
14:01:34.891 [ParallecActorSystem-akka.actor.default-dispatcher-2] INFO i.p.c.TestBase - response header: x-amz-id-2 - [KdWONL7fZ8u1yHtx10d47sO3VendT5RSy9XrlslEGcsyESlZDUcL29+N32u1fKgHVyfYTS2qdaA=]
14:01:34.892 [ParallecActorSystem-akka.actor.default-dispatcher-2] INFO i.p.c.TestBase - response header: date - [Thu, 29 Sep 2016 21:01:34 GMT]
14:01:34.892 [ParallecActorSystem-akka.actor.default-dispatcher-2] INFO i.p.c.TestBase - response header: content-type - [text/html]
14:01:34.892 [ParallecActorSystem-akka.actor.default-dispatcher-7] INFO i.p.c.a.ExecutionManager -
[3]__RESP_RECV_IN_MGR 3 (+0) / 3 (100.00%) AFT 1.362 S @ www.jeffpei.com @ 2016.09.29.14.01.34.892-0700 , TaskID : 44c749f0-058 , CODE: 200 OK, RESP_BRIEF: <!DOCTYPE html>
<!-- saved from
14:01:34.893 [ParallecActorSystem-akka.actor.default-dispatcher-7] INFO i.p.c.a.ExecutionManager - task.state : COMPLETED_WITHOUT_ERROR
14:01:34.893 [ParallecActorSystem-akka.actor.default-dispatcher-7] INFO i.p.c.a.ExecutionManager - task.totalJobNumActual : 3 InitCount: 3
14:01:34.893 [ParallecActorSystem-akka.actor.default-dispatcher-7] INFO i.p.c.a.ExecutionManager - task.response received Num 3
14:01:34.893 [ParallecActorSystem-akka.actor.default-dispatcher-7] INFO i.p.c.a.ExecutionManager - SUCCESSFUL GOT ON ALL RESPONSES: Received all the expected messages. Count matches: 3 at time: 2016.09.29.14.01.34.893-0700
14:01:34.895 [ParallecActorSystem-akka.actor.default-dispatcher-7] INFO i.p.c.a.ExecutionManager -
Time taken to get all responses back : 1.365 secs
from parallec.
jeffpeiyt
commented on May 27, 2024
@brunoribeiro ok, a very simple way for the case insensitiveness is to save with all with lower cases. The logs have been updated as above. Are you good with this?
responseHeaders.put(key.toLowerCase(Locale.ROOT),
response.getHeaders().get(key));from parallec.
brunoribeiro
commented on May 27, 2024
perfect, thank you so much.
On Thu, Sep 29, 2016 at 10:03 PM, Yuanteng (Jeff) Pei <
[email protected]> wrote:
@brunoribeiro https://github.com/brunoribeiro ok, a very simple way for
the case insensitiveness is to save with all with lower cases. The logs
have been updated as above. Are you good with this?responseHeaders.put(key.toLowerCase(Locale.ROOT),
response.getHeaders().get(key));—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#24 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ADAwe0xDBmbBgzQNtNp9XTDKA828eirOks5qvCesgaJpZM4G2sa-
.
Bruno Ribeiro
from parallec.
jeffpeiyt
commented on May 27, 2024
@brunoribeiro thank you for the confirmation. Will add more tests cases / docs and update here after a release
from parallec.
jeffpeiyt
commented on May 27, 2024
@brunoribeiro released in version 0.10.1-beta; please let me know for any more questions.
from parallec.
Related Issues (20)
- Tasks keep running HOT 5
- Https Issues HOT 4
- How to create task with m2m request HOT 2
- Check url protocol in setting up target host HOT 3
- Creating API getter HOT 2
- Which parameter is to control the NettyAsyncHttpProvider? HOT 3
- NPE in ChunkedWriteHandler HOT 3
- Documentation on submitting multiple POST requests HOT 16
- Error when build Jar file w/ this library HOT 4
- May I know how to call an api and get json response using java HOT 1
- Time out error HOT 2
- Ping response time HOT 2
- Tcpclient can't flush the command HOT 1
- Allow sending parallel gRPC requests HOT 1
- Adding a Scala example HOT 1
- Multiple Global Configs HOT 1
- Bulk job of POST requests HOT 1
- Getting time out after 14 seconds HOT 1
- multiple hosts execute different ssh command concurrently
- ParallelTaskBuilder.execute hangs forever waiting for a task completion in sync mode after resources released HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from parallec.