asfadmin / thin-egress-app Goto Github PK
View Code? Open in Web Editor NEWAPI Gateway & Lambda AWS Thin Egress App
License: Other
API Gateway & Lambda AWS Thin Egress App
License: Other
36 is safe. Is 37?
thin-egress-app/terraform/main.tf
Line 52 in 15794d8
If any inbound request comes with a x-origin-request-id
header, record that in the logs. Also, add that header to all response payloads (this is similar to x-request-id
now) as well as any outbound requests to EOSDIS services (ie EDL).
Lines 239 to 243 in eb85b14
Its not a HUGE deal and if you have a CloudFront error configured you see that. But it does mask the error a bit. You see this:
instead of the actual error.
URSredirectNoDomainURI is superfluous. Instead, map it to the values based on the DomainNameIsSet
condition.
ExternalEndpoint should also respect DomainNameIsSet
PRIVATE_BUCKETS:
nsidc-cumulus-uat-protected:
- ICESat-II Cloud Early Access UAT
- Staff UAT
nsidc-cumulus-uat-protected/ATLAS:
- ICESat-II Cloud Early Access UAT
nsidc-cumulus-uat-protected/ATLAS/ATL06:
- Staff UAT
- ICESat-II Cloud Early Access UAT
nsidc-cumulus-uat-protected/ATLAS/ATL03:
- Staff UAT
Right now, ONLY the first object prefix is evaluated. We should create a better methodology that allows arbitrary depth prefixes.
HyP3 would like to consume the EDL user's Email for sending notification. The call to /api/users/{userid}
to return the users profile returns email_address
. We should add this into the JWT payload.
Any time we make any outbound request, even to AWS services like S3, STS, etc, drop a log message with a parsable log payload like:
{
"service": "S3",
"endpoint": "s3://bucket/object",
"request_type": "HEAD",
"duration": "105"
"unit": "milleseconds"
}
This is deployed to us-west-2. It is part of a deployment of https://github.com/nasa/cumulus using TEA version 102.
We have not deployed this to us-west-2, but it worked when deploying to us-east-1 in April. Deployment to us-east-1 no longer works because of #386
The CloudFormation stack fails creation with these relevant events (in chronological order) for the TriggerInRegionCIDRUpdate component:
(1)
CloudFormation did not receive a response from your Custom Resource. Please check your logs for requestId [05d47ac7-1b99-45b0-8353-246973729264]. If you are using the Python cfn-response module, you may need to update your Lambda function code so that CloudFormation can attach the updated version.
(2)
The following resource(s) failed to create: [TriggerInRegionCIDRUpdate]. Rollback requested by user.
(3)
CloudFormation did not receive a response from your Custom Resource. Please check your logs for requestId [8b32a2cb-57a4-4f0c-a192-72c298050321]. If you are using the Python cfn-response module, you may need to update your Lambda function code so that CloudFormation can attach the updated version.
(4)
The following resource(s) failed to delete: [TriggerInRegionCIDRUpdate].
In the logs for the TriggerInRegionCIDRUpdate lambda, this appears to be the problem:
START ...
Current reigon in us-west-2
END ...
Task timed out after 6.01 seconds
There are no other log messages around this.
In the README.md, we're currently advising users who want to package the data to zip up ONLY the top level .py
files, which miss the common/*.py modules
.
https://github.com/asfadmin/thin-egress-app/blame/devel/README.MD#L52
Provide RequestID as a template parameter for error pages and add it to the error page somewhere discrete.
Currently there is no easy way to view the invalid range requested by the user.
Line 297 in 07631ae
Can we create a method by which we can define URI path by bucket & object prefix allowing greater flexibility for partitioning access to buckets? I imagine it would look something like this scenario:
In the example below:
https://<tea>/pathA/prefix1/object_1.ext
=> s3://bucket-name1/prefix1/object_1.ext
would require a user to be in URS group access_group_a
https://<tea>/pathB/prefix2/object_2.ext
=>s3://bucket-name1/prefix2/object_2.ext
would require a user to be in URS group access_group_b
https://<tea>/pathC/some_file.ext
=> s3://bucket-name2/some_file.ext
would require only simple URS authhttps://<tea>/pathC/bowse/image.jpg
=> s3://bucket-name2/browse/image.jpg
would be publicly accessibleQuestions to be resolvled:
s3://bucket-name1/prefix1/object_1.ext
, which path makes more sense?
https://<tea>/pathA/prefix1/object_1.ext
https://<tea>/pathA/object_1.ext
s3://bucket-name1/hidden.obj
?MAP:
pathA: bucket-name1/prefix1
pathB: bucket-name1/prefix2
pathC: bucket-name2
PUBLIC_BUCKETS
bucket-name2/browse: "Browse Imagery
PRIVATE_BUCKETS:
bucket-name1/prefix1:
- access_group_a
bucket-name1/prefix2:
- access_group_b
Add an optional CF Stack Parameter that allows for customizing the log group that TEA Lambda writes logs to.
The default should be the current default or just not specifying a log group, which would use the default.
Line 184 in df7a296
If TEA is redirected to DURING CORS, the CORS origin:
header is "null"
instead of the apparent origin header.
We need to account for the null
case as it happens when datapool.asf.alaska.edu
redirects to sentinel1.asf.alaska.edu
etc.
That solution looks something line changing L184 to:
origin_header = app.current_request.headers['origin']
if cors_origin and ( origin_header.endswith(cors_origin) or origin_header.lower() == 'null' ):
We shouldn't have any params that reference anything specific to ASF. These should be empty strings or obviously fake dummy examples.
Also https://github.com/asfadmin/thin-egress-app/blob/devel/cloudformation/thin-egress-app.yaml#L80
thin-egress-app/terraform/main.tf
Line 30 in 6aec7d8
Should be added TF main & vars
seems obvious in retrospect, but would be useful for capturing the final result.
Alternative solution:
logger
call...{"message": "<VALUE>"}
be something to the affect of "cumulus log response payload"Line 140 in 4111b52
IPv4/IPv6 manipulation library
Library home page: https://files.pythonhosted.org/packages/c2/f8/49697181b1651d8347d24c095ce46c7346c37335ddc7d255833e7cde674d/ipaddress-1.0.23-py2.py3-none-any.whl
Path to dependency file: thin-egress-app/lambda/requirements.txt
Path to vulnerable library: thin-egress-app/lambda/requirements.txt
Dependency Hierarchy:
Found in HEAD commit: 92290bb78c314696d35697edd726df993fbeb93f
Found in base branch: devel
Lib/ipaddress.py in Python through 3.8.3 improperly computes hash values in the IPv4Interface and IPv6Interface classes, which might allow a remote attacker to cause a denial of service if an application is affected by the performance of a dictionary containing IPv4Interface or IPv6Interface objects, and this attacker can cause many dictionary entries to be created. This is fixed in: v3.5.10, v3.5.10rc1; v3.6.12; v3.7.9; v3.8.4, v3.8.4rc1, v3.8.5, v3.8.6, v3.8.6rc1; v3.9.0, v3.9.0b4, v3.9.0b5, v3.9.0rc1, v3.9.0rc2.
Publish Date: 2020-06-18
URL: CVE-2020-14422
Base Score Metrics:
Type: Upgrade version
Origin: https://security-tracker.debian.org/tracker/CVE-2020-14422
Release Date: 2020-06-18
Fix Resolution: 3.5.3-1+deb9u2, 3.7.3-2+deb10u2, 3.8.4~rc1-1
Step up your Open Source Security Game with WhiteSource here
Need a mechanism to pull fresh config TEA on demand. This should be able to function independent of an application re-deployments.
Potentially utilize a hot-updatable Lambda env variable to trigger a soft-flush of environment.
How do we want to trigger the flush? AWS API level Invocation of a VPC-restricted? Auth'd ๐ฌ API Endpoint?
Line 622 in 7cf716d
If the profile has been mutated, its likely the groups the user belongs to have changed. We should update the JWT cookie payload!
Error: Error in function call
on .terraform/modules/thin_egress_app/main.tf line 34, in resource "aws_s3_bucket_object" "lambda_code_dependency_archive":
34: key = "${filemd5(local.dependency_layer_filename)}.zip"
|----------------
| local.dependency_layer_filename is ".terraform/modules/thin_egress_app/dependencylayer.zip"
Call to function "filemd5" failed: no file exists at
.terraform/modules/thin_egress_app/dependencylayer.zip.
It should just echo the build.#
When TEA is behind CloudFront, the value @ app.current_request.context['identity']['sourceIp']
represents the CloudFront service, not the user IP:
Line 246 in 458635f
When checking in vs out of region requests, we should first check if there is a x-forwarded-for
header present in the app.current_request.headers
object. If that object is present, the string value should be split, and the 0th element should be utilized for checking the user region.
x-forwarded-for': '137.229.86.134, 130.176.100.154'
Build on asfadmin/rain-api-core#51
The PyPA recommended tool for installing Python packages.
Library home page: https://files.pythonhosted.org/packages/54/0c/d01aa759fdc501a58f431eb594a17495f15b88da142ce14b5845662c13f3/pip-20.0.2-py2.py3-none-any.whl
Path to dependency file: /tmp/ws-scm/thin-egress-app/lambda/requirements.txt
Path to vulnerable library: /tmp/ws-scm/thin-egress-app/lambda/requirements.txt
Dependency Hierarchy:
Found in HEAD commit: 18e84670c8f0bc1fa0d5934e39a695d712e94c0c
An issue was discovered in pip (all versions) because it installs the version with the highest version number, even if the user had intended to obtain a private package from a private index. This only affects use of the --extra-index-url option, and exploitation requires that the package does not already exist in the public index (and thus the attacker can put the package there with an arbitrary version number).
Publish Date: 2020-05-08
URL: CVE-2018-20225
Base Score Metrics:
Step up your Open Source Security Game with WhiteSource here
YAML parser and emitter for Python
Library home page: https://files.pythonhosted.org/packages/64/c2/b80047c7ac2478f9501676c988a5411ed5572f35d1beff9cae07d321512c/PyYAML-5.3.1.tar.gz
Path to dependency file: thin-egress-app/lambda/requirements.txt
Path to vulnerable library: thin-egress-app/lambda/requirements.txt
Dependency Hierarchy:
Found in HEAD commit: 70cd8b34eb3c9d86e1b193cb19b57fa4a193ae34
A vulnerability was discovered in the PyYAML library in all versions, where it is susceptible to arbitrary code execution when it processes untrusted YAML files through the full_load method or with the FullLoader loader. .load() defaults to using FullLoader and FullLoader is still vulnerable to RCE when run on untrusted input. Applications that use the library to process untrusted input may be vulnerable to this flaw. An attacker could use this flaw to execute arbitrary code on the system by abusing the python/object/new constructor.
The fix for CVE-2020-1747 was not enough to fix this issue.
Publish Date: 2020-07-21
URL: CVE-2020-14343
Base Score Metrics:
Step up your Open Source Security Game with WhiteSource here
Patrick Quinn has requested we add Cache-Control: private, max-age=600
header.
Rather than specially coding that one header, TEA should provide a configuration process that allows app admin to specify custom headers.
Configuration could be a data redirect (global) config, or potentially a per-bucket setting similar to defining PUBLIC/PRIVATE buckets.
Instead of referencing AWS::StackName
directly, figure out a way to use only the first ~35 characters to avoid name length issues (specifically with UpdatePolicyLambdaIamRole
).
It appears that if a JWT cookies is created for a user BEFORE they are in a private app group, and the user is them subsequently added to the group, the cookie is not getting refreshed and user cannot get the data until clearing cookies.
We seen testing at scale that normal usage is closer to 400MB, but spikes to near 700MB. However, at 1792 MB, we can allocate an entire vCPU to help mitigate backgrounding.
1792MB seems a little on the excessive side when normal usage is around 400 and spikes only go up to 700, however Lambda only charges you for RAM used, not RAM allocated. Getting that full proportional vCPU should offset any concerns of wasted RAM.
We should add a new Parameter to the CF & TF to support dynamic setting of the Lambda Memory allocation, and have it default to 1792.
How might we inject a semantic version id into the build process? Trigger release from a v#.#.#
tag (like our cirrus builds) and inject the tag name into the code instead of build.#
value? Alternatively, accept a build version id as a jenkins build param?
Do we rely on build.#
syntax in the release code anywhere?
Every @app.route()
should generate a unique id (perhaps a UUID or part of a UUID. That ID would be appended to ALL log entries and would allow easier filtering in logs.
This could be done by adding a global variable "REQ_ID" and adding that the log handler in the way we stamp build id
Unfortunately, this line doesn't actually work:
The solution is far more in-depth:
DownloadRoleInRegion
(this may require tweeking policy names)ip_prefix
for .region=="us-west-2" and .service=="AMAZON"
"Condition": { "IpAddress":{"aws:SourceIp":["52.95.255.112/28","99.77.253.0/24",....]
arn:aws:sns:us-east-1:806199016981:AmazonIpSpaceChanged
This will keep the IP CIDR's in the policy fresh. Policy updates should be Near Realtime.
Line 66 in 888eb82
After URS Auth, the return URL is from https://d24c78m3es2lu9.cloudfront.net/GRD/S5/SIRC1_16_GRD_006_010_010_19940409T185535_19940409T185550.zip is https://d24c78m3es2lu9.cloudfront.net/API/ GRD/S5/SIRC1_16_GRD_006_010_010_19940409T185535_19940409T185550.zip
This behavior is correct (because its new!) in shared token handling:
https://github.com/asfadmin/thin-egress-app/blob/devel/lambda/app.py#L536-L544
For normal EDL auth, we aren't currently getting the JSON payload.
When a user attempts to log in to an EDL App and reject/avoids accepting the EULA, TEA does not receive the code value necessary to proceed through the rest of the process. Instead of the user being 302 redirected to https://<redirect_uri>?code=<code>
, the user is sent to https://<redirect_uri>?error=access_denied
.
In that case that originating https://uat.urs.earthdata.nasa.gov/oauth/authorize
request has the app_type=401
parameter, 302 redirect is a little more complex: https://<redirect_uri>?error=access_denied&error_msg=App%20EULA%20has%20been%20updated,%20please%20reauthorize%20the%20application%20via%20the%20Earthdata%20Login%20GUI
However, at no point in the process is there a JSON payload, nor is there a resolution_url value.
I'd propose that instead of:
<redirect_uri>?error=access_denied&error_msg=App%20EULA%20has%20been%20updated,%20please%20reauthorize%20the%20application%20via%20the%20Earthdata%20Login%20GUI
the 302 is to:
<redirect_uri>?error=access_denied&reason=EULA%20Acceptance%20Failure&app_type=401&resolution_url=https://urs.earthdata.nasa.gov/approve_app?app_uid=<APP-UID>
... then, downstream apps (like TEA) can reformat that response to a JSON payload.
In terms of TEA though, this is not a tractionable ticket until EDL behavior is changed.
This would allow external entities to validate a JWT token.
https://<tea>/pubkey
would return something like the below payload, where rsa_pub_key
is the value from the JWT Secret Manager key.
{
"rsa_pub_key": "c3NoLXJ....5lZHUK"
}
This feature would facilitate easy validation of secrets across deployments, and allow non-EDL enable applications to securely validate JWT's across cookies domain space.
This is an idea I had earlier today. It's not specifically needed by ASF, but it may be feature that would help another user.
TEA currently checks if an object exists in S3 before creating a signed URL. This takes a little time, so we provide the option to turn off that check. This is a binary choice, either all or none are checked, depending on that setting.
https://github.com/asfadmin/thin-egress-app/blob/devel/lambda/app.py#L294-L301
Would it be useful to do checks for some types of files and not others?
Right now we can change the JWT secret, but that would invalidate ALL sessions.
It would be nice if we could kill ONE specific user session.
I'm not sure how we could pull it off, but it'd be a useful feature.
When TEA talks to ELD, make sure to include the x-request-id
as a request header.
Request by Matt Savoie.
Cumulus' Elasticsearch / Kibana needs stable, robust log messages for successful and failed download attempts.
Previously, success was determined by searching for to_url
, probably this log message:
Line 78 in 2496a6e
We need to provide a log message with a distinct code at time of success.
Failure is determined by searching for Could not download
, probably this log message:
Line 156 in 2496a6e
We need to provide a log message with a useful code in cases of failure.
These should be logged at INFO
level or higher
A very fast and expressive template engine.
Library home page: https://files.pythonhosted.org/packages/7e/c2/1eece8c95ddbc9b1aeb64f5783a9e07a286de42191b7204d67b7496ddf35/Jinja2-2.11.3-py2.py3-none-any.whl
Path to dependency file: thin-egress-app/lambda/requirements.txt
Path to vulnerable library: thin-egress-app/lambda/requirements.txt
Dependency Hierarchy:
Found in HEAD commit: 818c03e35772114b7983cf8c97624e604232a200
This affects the package jinja2 from 0.0.0 and before 2.11.3. The ReDOS vulnerability of the regex is mainly due to the sub-pattern [a-zA-Z0-9.-]+.[a-zA-Z0-9.-]+ This issue can be mitigated by Markdown to format user content instead of the urlize filter, or by implementing request timeouts and limiting process memory.
Publish Date: 2021-02-01
URL: CVE-2020-28493
Base Score Metrics:
Type: Upgrade version
Origin: https://cve.mitre.org/cgi-bin/cvename.cgi?name=2020-28493
Release Date: 2021-02-01
Fix Resolution: 2.11.3
Step up your Open Source Security Game with WhiteSource here
Traceback (most recent call last):
File "/opt/python/chalice/app.py", line 1104, in _get_view_function_response
response = view_function(**function_args)
File "/var/task/app.py", line 212, in logout
user_id = cookievars['urs-user-id']
KeyError: 'urs-user-id'
Line 212 in 5e8920a
This value should be wrapped in a if 'urs-user-id' in cookievars:
block.
thin-egress-app/terraform/main.tf
Line 86 in d29f639
MAP:
nsidc-cumulus-uat-protected: nsidc-cumulus-uat-protected
nsidc-cumulus-uat-public: nsidc-cumulus-uat-public
PUBLIC_BUCKETS:
nsidc-cumulus-uat-public: ""
PRIVATE_BUCKETS:
nsidc-cumulus-uat-protected:
- Staff UAT
nsidc-cumulus-uat-protected/ATLAS:
- ICESat-II Cloud Early Access UAT
- Staff UAT
In the above use case, check_private_bucket()
matches nsidc-cumulus-uat-protected
and does not properly evaluate nsidc-cumulus-uat-protected/ATLAS
causing ICESat-II Cloud Early Access UAT
to be denied access to /ATLAS
.
This can be resolved by running through PRIVATE_BUCKETS:
blocks that contain /
BEFORE blocks that DON'T.
Short term fix for above was to re-order /ATLAS
above the parent bucket, but that is NOT a long-term solution.
Pure-Python RSA implementation
Library home page: https://files.pythonhosted.org/packages/02/e5/38518af393f7c214357079ce67a317307936896e961e35450b70fad2a9cf/rsa-4.0-py2.py3-none-any.whl
Path to dependency file: /tmp/ws-scm/thin-egress-app/lambda/requirements.txt
Path to vulnerable library: /tmp/ws-scm/thin-egress-app/lambda/requirements.txt
Dependency Hierarchy:
Found in HEAD commit: 7a7150830e3c44c9d03bac1e685c163285f1c218
Python-RSA 4.0 ignores leading '\0' bytes during decryption of ciphertext. This could conceivably have a security-relevant impact, e.g., by helping an attacker to infer that an application uses Python-RSA, or if the length of accepted ciphertext affects application behavior (such as by causing excessive memory allocation).
Publish Date: 2020-06-01
URL: CVE-2020-13757
Base Score Metrics:
Step up your Open Source Security Game with WhiteSource here
I apologize in advance for what may be a confusing or incorrect bug report, as I'm pretty new to working with all of these software components, but I think my analysis of this is correct.
The problem we're seeing is this error when the lambda that updates the role policy runs:
There was a problem updating policy my-app-thin-egress-app-IamPolicyDownload for Role my-app-thin-egress-app-DownloadRoleInRegion in region us-east-1: An error occurred (LimitExceeded) when calling the PutRolePolicy operation: Maximum policy size of 10240 bytes exceeded for role my-app-thin-egress-app-DownloadRoleInRegion
This is part of a deployment of https://github.com/nasa/cumulus
I believe the issue is that the policy condition to allow all AWS IPs is too large in us-east-1. Looking at the number of ip ranges for each of these regions (there are 6 lines in this output for each CIDR block), us-east-1 has nearly 1000, while the others have less than half as many. I think this results in a policy that's more than 10k characters and exceeds the AWS limit.
$ jq '.prefixes[] | select(.region=="us-east-1")' < Downloads/ip-ranges.json | wc
5262 8770 110088
~ via ๏ข v14.17.5 on :a:
$ jq '.prefixes[] | select(.region=="us-west-2")' < Downloads/ip-ranges.json | wc
2292 3820 48248
~ via ๏ข v14.17.5 on :a:
$ jq '.prefixes[] | select(.region=="us-east-2")' < Downloads/ip-ranges.json | wc
1890 3150 39043
In normal operation, there is always at least one header that arrives with the lambda event payload.
However, when you run a API Gateway Method Execution Test, unless you explicitly supply a header key pair, no headers actually end up in the event payload and event['headers']
is None
. That causes THIS line:
Line 46 in 1868602
... to fail with this error:
File "/var/task/app.py", line 46, in __call__
userid = get_jwt_field(get_cookie_vars(event['headers']), 'urs-user-id')
File "/var/task/rain_api_core/view_util.py", line 101, in get_cookie_vars
cooks = get_cookies(headers)
File "/var/task/rain_api_core/view_util.py", line 129, in get_cookies
c = hdrs.get('cookie', hdrs.get('Cookie', hdrs.get('COOKIE', None)))
...and then the result in the Test harness is:
Request: /version
Status: 502
Latency: 1314 ms
Response Body:
{
"message": "Internal server error"
}
Response Headers
{"x-amzn-ErrorType":"InternalServerErrorException"}
I think fixings this requires that we find all references to event['headers']
and replace with something like event['headers'] or {}
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.