flightstats / hub Goto Github PK
View Code? Open in Web Editor NEWfault tolerant, highly available service for data storage and distribution
Home Page: http://www.flightstats.com
License: MIT License
fault tolerant, highly available service for data storage and distribution
Home Page: http://www.flightstats.com
License: MIT License
As /previous goes backwards in time, if the TTL is greater than 21 years, links are not returned.
Aashish says they aren't using them, and aren't required now.
requested by @crozierm
This only seems applicable to legacyScraperRaw
Mostly, would love some docs on how to set up the back end, etc.
Thanks!
Lucas was having an issue with this:
http://hub-v2.svc.staging/channel/kratzen_lgw_last/2015/04/16/23/49/13/164/TrNHM6/previous/7
would hang
but
http://hub-v2.svc.staging/channel/kratzen_lgw_last/2015/04/16/23/49/13/164/TrNHM6/previous/6
would return quickly
This will rely on the NAS for redundancy and compression.
Create a new ChannelService implementation which uses a single volume, and does not use S3 or Dynamo for items, channel metadata or group metadata.
This should be able to reuse most of FileSpokeStore for actual storage on disk.
Start with a simple daily period when an alert should not be sent.
@aashishpatel I recreated my group callback starting a the first item and noticed that the errors are still associated with the new group(i used the same name).
https://enc-v2-prod.hub.flightstats.com/group/aav2_tdmObserverProfileToLegacyBridge
This is only for query apis, not for group callback or websocket
When querying the forward end of a channel:
hub/channel/{name}/{key path}/next/10
If the hub returns zero items, it should still have a previous link.
The reverse is true for the back end of the channel.
For example, if you pass in 99
for the month you'll get an HTTP 500 with a JodaTime error message.
Request:
curl http://hub-v2.svc.dev/channel/zanderSandbox/2015/99/01/12/30/00/000/a
Response:
HTTP ERROR: 500
Problem accessing /channel/zanderSandbox/2015/99/01/12/30/00/000/a. Reason:
org.joda.time.IllegalFieldValueException: Value 99 for monthOfYear must be in the range [1,12]
Powered by Jetty://
I would expect a 400 for this kind of error.
The hub currently support channel alerting when data stop being written for a given period.
It would be awesome if we could do something similar for group callbacks to make sure they are keeping up with the latest item on a channel.
The second one has the same data as the first
http://hub-v2.svc.prod/channel/escalationAlerts/2015/05/14/14/53/40/610/V8uaYx
http://hub-v2.svc.prod/channel/escalationAlerts/2015/05/14/15/38/56/066/2N4FbT
In a replicated channel, @chriskessel has found the following
1 - call /next/10 near the head of the channel
2 - record the last item
3 - call /latest
4 - latest returns an item previous to step 2
This happens infrequently, ~once in 50 attempts.
Request from @aashishpatel
Allow creation of group callbacks which go backwards from a starting point to an optional end point
After a complete traversal, the group callback should delete itself
Advantages over client implementing:
Load can be more easily distributed
Client does not need to track cursor location
/spoke/latest/FlexFlightHistoryReplicatedHubV1Staging/2015/03/17/17/36/49/000/ZZZZZ
returns
FlexFlightHistoryReplicatedHubV1Staging/2015/03/17/17/31/43/691/2905200
/spoke/latest/FlexFlightHistoryReplicatedHubV1Staging/2015/03/17/17/37/15/000/ZZZZZ
returns
FlexFlightHistoryReplicatedHubV1Staging/2015/03/17/17/31/13/686/2905180
https://flightstats.pagerduty.com/incidents/PRVG1R5
"http://hub-v2.svc.prod/channel/aav2_obfuscatedItineraries/2015/05/14/13/28?stable=true",
"http://hub-v2.svc.prod/channel/aav2_obfuscatedItineraries/2015/05/14/13/26?stable=true",
"http://hub-v2.svc.prod/channel/aav2_obfuscatedItineraries/2015/05/14/13/27?stable=true",
"http://hub-v2.svc.prod/channel/aav2_obfuscatedItineraries/2015/05/14/13/29?stable=true",
"http://hub-v2.svc.prod/channel/aav2_obfuscatedItineraries/2015/05/14/13/28?stable=true",
"http://hub-v2.svc.prod/channel/aav2_obfuscatedItineraries/2015/05/14/13/30?stable=true",
"http://hub-v2.svc.prod/channel/aav2_obfuscatedItineraries/2015/05/14/13/29?stable=true",
"http://hub-v2.svc.prod/channel/aav2_obfuscatedItineraries/2015/05/14/13/31?stable=true",
"http://hub-v2.svc.prod/channel/aav2_obfuscatedItineraries/2015/05/14/13/30?stable=true",
"http://hub-v2.svc.prod/channel/aav2_obfuscatedItineraries/2015/05/14/13/32?stable=true",
"http://hub-v2.svc.prod/channel/aav2_obfuscatedItineraries/2015/05/14/13/31?stable=true",
"http://hub-v2.svc.prod/channel/aav2_obfuscatedItineraries/2015/05/14/13/33?stable=true",
"http://hub-v2.svc.prod/channel/aav2_obfuscatedItineraries/2015/05/14/13/34?stable=true",
"http://hub-v2.svc.prod/channel/aav2_obfuscatedItineraries/2015/05/14/13/32?stable=true",
"http://hub-v2.svc.prod/channel/aav2_obfuscatedItineraries/2015/05/14/13/33?stable=true",
when resizing an instance, the cluster's delta time can swing dramatically.
add a 50% margin to the adjustment, which will have minimal impact during normal operations, and a wider buffer during time synchronization.
Aashish wants to modify parallel calls to tune performance w/out having to recreate the group callback.
/status & /time both return results for missing channels
It is possible to incorrectly return an empty lastCompleted
WARN 2015-03-09 17:59:40,054 [pool-6-thread-1] com.flightstats.hub.spoke.RemoteSpokeStore [line 111] - unable to get content aav2_pfTdmItineraries/2015/03/09/17/58/59/991/a2sLK8
com.sun.jersey.api.client.ClientHandlerException: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No subject alternative DNS name matching encrypted-hub-v2-03.cloud-east.staging found.
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:155) ~[jersey-client-1.19.jar:1.19]
at com.sun.jersey.api.client.Client.handle(Client.java:652) ~[jersey-client-1.19.jar:1.19]
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682) ~[jersey-client-1.19.jar:1.19]
at com.sun.jersey.api.client.WebResource.get(WebResource.java:193) ~[jersey-client-1.19.jar:1.19]
at com.flightstats.hub.spoke.RemoteSpokeStore.read(RemoteSpokeStore.java:102) ~[hub-2015-02-17.29.jar:na]
at com.flightstats.hub.spoke.SpokeContentDao.read(SpokeContentDao.java:68) [hub-2015-02-17.29.jar:na]
at com.flightstats.hub.dao.s3.S3WriteQueue.writeContent(S3WriteQueue.java:75) [hub-2015-02-17.29.jar:na]
at com.flightstats.hub.dao.s3.S3WriteQueue.access$100(S3WriteQueue.java:23) [hub-2015-02-17.29.jar:na]
at com.flightstats.hub.dao.s3.S3WriteQueue$2.call(S3WriteQueue.java:62) [hub-2015-02-17.29.jar:na]
at com.flightstats.hub.dao.s3.S3WriteQueue$2.call(S3WriteQueue.java:59) [hub-2015-02-17.29.jar:na]
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) [guava-retrying-1.0.6.jar:na]
at com.github.rholder.retry.Retryer.call(Retryer.java:110) [guava-retrying-1.0.6.jar:na]
at com.flightstats.hub.dao.s3.S3WriteQueue.write(S3WriteQueue.java:59) [hub-2015-02-17.29.jar:na]
at com.flightstats.hub.dao.s3.S3WriteQueue.access$000(S3WriteQueue.java:23) [hub-2015-02-17.29.jar:na]
at com.flightstats.hub.dao.s3.S3WriteQueue$1.call(S3WriteQueue.java:50) [hub-2015-02-17.29.jar:na]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_31]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_31]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_31]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_31]
Caused by: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No subject alternative DNS name matching encrypted-hub-v2-03.cloud-east.staging found.
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[na:1.8.0_31]
at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1937) ~[na:1.8.0_31]
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302) ~[na:1.8.0_31]
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296) ~[na:1.8.0_31]
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1478) ~[na:1.8.0_31]
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:212) ~[na:1.8.0_31]
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:957) ~[na:1.8.0_31]
at sun.security.ssl.Handshaker.process_record(Handshaker.java:892) ~[na:1.8.0_31]
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1050) ~[na:1.8.0_31]
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1363) ~[na:1.8.0_31]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1391) ~[na:1.8.0_31]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1375) ~[na:1.8.0_31]
at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:563) ~[na:1.8.0_31]
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185) ~[na:1.8.0_31]
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1512) ~[na:1.8.0_31]
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440) ~[na:1.8.0_31]
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) ~[na:1.8.0_31]
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338) ~[na:1.8.0_31]
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:253) ~[jersey-client-1.19.jar:1.19]
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:153) ~[jersey-client-1.19.jar:1.19]
... 18 common frames omitted
Caused by: java.security.cert.CertificateException: No subject alternative DNS name matching encrypted-hub-v2-03.cloud-east.staging found.
4 items are missing from dev:
http://ops-jenkins01.cloud-east.dev/job/hub-v2-dev-replication-verifier/1170/console
'http://hub/channel/syslog_web_legacy_pdx/2015/05/07/11/32/03/288/NECZjE',
'http://hub/channel/syslog_web_legacy_pdx/2015/05/07/11/32/05/387/HtPn3U',
'http://hub/channel/syslog_web_legacy_pdx/2015/05/07/11/32/07/657/sTm5nV',
'http://hub/channel/syslog_web_legacy_pdx/2015/05/07/11/32/09/911/SybIsp',
This appears possible when leadership changes before alert status is changed.
http://ops-jenkins01.cloud-east.dev/job/hub-v2-dev-alert-verifier/25/console
For two different groups in the encrypted hub, the same patten:
INFO 2015-06-23 19:10:31,730 [LeaderRotatorService RUNNING] com.flightstats.hub.cluster.CuratorLeader [line 88] - abdicating leadership for /GroupLeader/aav2_tdmItineraryToLegacyBridge
did not follow the expected pattern:
INFO 2015-06-23 19:10:31,730 [LeaderRotatorService RUNNING] com.flightstats.hub.cluster.CuratorLeader [line 88] - abdicating leadership for /GroupLeader/zomboEscalationAlerts
INFO 2015-06-23 19:10:35,242 [Curator-LeaderSelector-8] com.flightstats.hub.group.GroupCaller [line 117] - stopping last completed at 2015/05/13/23/56/55/254/EbKKmf zomboEscalationAlerts
INFO 2015-06-23 19:10:35,243 [Curator-LeaderSelector-8] com.flightstats.hub.cluster.CuratorLeader [line 70] - lost leadership /GroupLeader/zomboEscalationAlerts
If a CallbackQueue has an error, it does not cause the GroupCaller to abdicate leadership.
This should allow users to easily support longer cache times and arbitrary values
When sending a POST request to a channel that has a replication source, the response "cannot modified while replicating" was sent as plain text when JSON was requested and expected.
This issue seems nitpicky but our hub client's HTTP library was very unhappy.
When a group callback has many parallel processes, a spike in data can cause a large spike in CPU usage. This can adversely impact other traffic to the hub.
Mitigation:
(non - coding solutions)
(dev solutions)
Callback only Hub node - in addition to the traditional 3 - so callbacks won't affect other traffic
Spin up node as needed (might be SciFi)
Improve performance of hub processes
dev integration tests are showing that earliest/3 sometimes returns only two items
This appears to have a race condition between when the underlying group callback is created, and when the first item is posted.
Channel is replicated from prod into staging, and the group callback is from staging.
I would love to use the hub to store data permanently. One missing feature is the ability to delete content from the hub.
From Lucas:
$ nc hub-v2.svc.staging 80
GET /channel/kratzen_ams/2015/03/19/03/47/55/655/ujs8Nb HTTP/1.1
User-Agent: Java/1.7.0_71
Host: hub-v2.svc.staging
Accept: text/html, image/gif, image/jpeg, ; q=.2, */; q=.2
Stack trace from logs:
java.lang.IllegalArgumentException: java.text.ParseException: Expected separator '/' instead of ';'
at com.sun.jersey.core.header.MediaTypes.createMediaTypes(MediaTypes.java:234) ~[jersey-bundle-1.19.jar:1.19]
at com.flightstats.hub.channel.ChannelContentResource.contentTypeIsNotCompatible(ChannelContentResource.java:324) ~[hub-2015-03-18.7.jar:na]
Caused by: java.text.ParseException: Expected separator '/' instead of ';'
at com.sun.jersey.core.header.reader.HttpHeaderReader.nextSeparator(HttpHeaderReader.java:121) ~[jersey-bundle-1.19.jar:1.19]
at com.sun.jersey.core.impl.provider.header.MediaTypeProvider.valueOf(MediaTypeProvider.java:90) ~[jersey-bundle-1.19.jar:1.19]
at com.sun.jersey.core.header.reader.HttpHeaderReader$2.create(HttpHeaderReader.java:320) ~[jersey-bundle-1.19.jar:1.19]
at com.sun.jersey.core.header.reader.HttpHeaderReader$2.create(HttpHeaderReader.java:318) ~[jersey-bundle-1.19.jar:1.19]
at com.sun.jersey.core.header.reader.HttpHeaderReader.readList(HttpHeaderReader.java:481) ~[jersey-bundle-1.19.jar:1.19]
at com.sun.jersey.core.header.reader.HttpHeaderReader.readMediaTypes(HttpHeaderReader.java:325) ~[jersey-bundle-1.19.jar:1.19]
at com.sun.jersey.core.header.MediaTypes.createMediaTypes(MediaTypes.java:228) ~[jersey-bundle-1.19.jar:1.19]
Trey would like to have a self contained version of the hub which does not use S3 or Dynamo to run inside a Docker container for testing.
Create a channel with Body works as expected but:
curl -i -X PUT http://hub/channel/testBen2
returns
HTTP/1.1 500 Server Error
Date: Tue, 24 Feb 2015 20:28:35 GMT
Cache-Control: must-revalidate,no-cache,no-store
Content-Type: text/html; charset=ISO-8859-1
Content-Length: 315
Server: Jetty(9.2.6.v20141205)
Problem accessing /channel/testBen2. Reason:
java.lang.NullPointerException
currently this returns a 500 with an NPE
INFO 2015-03-10 21:00:04,082 [qtp1969238242-80136] com.flightstats.hub.spoke.SpokeResource [line 132] - NPE - unable to get latest Station 2015/03/10/20/59/59/000/ZZZZZ
should return a 404
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.