$ aws lambda publish-layer-version \
  --layer-name prefix-CMA-layer \
  --zip-file fileb://path/to/cumulus-message-adapter.zip

from https://nasa.github.io/cumulus/docs/deployment/deployment-readme#deploy-the-cumulus-instance

It will be better to use 3 slashes for files:///path/to/ as well.

I think example/ should be example/cumulus-tf/ directory.

#### Configure the Cumulus deployment

In the `example` directory, copy `terraform.tfvars.example` to
`terraform.tfvars` and update all of the parameters using values appropriate for
your deployment.

All new granules are failing to ingest into the LP DAAC Cumulus instance. The logs do not indicate what is causing the problem. The operator should be able to tell from the logs what is causing the problem. We also need to get ingest working again. Our operators should have a way to remedy this problem as well.

Can you provide any guidance on the size and number of nodes required to run a production Cumulus Elasticsearch cluster?

This is minor, since this information is available on the github releases page (https://github.com/nasa/cumulus/releases), but the "Cumulus Versions" section of the documentation is missing instructions for upgrading to 1.1.12 and 1.1.13:

https://nasa.github.io/cumulus/docs/next/cumulus-docs-readme

The information in the docs for Lambda versioning is incorrect:
https://github.com/nasa/cumulus/blob/master/docs/lambda_versioning.md

In that there is no supported field uniqueIdentifier under s2Source for Lambdas, and instead an error will be thrown if provided.

I'm not sure if that means the rest of the document is still relevant, does the versioning still happen like described here? Does it use the s3Source key field instead as an identifier?

I saw an event where for some time we were unable to complete the OAuth token handeshake due to a timeout somewhere in the system.

You can see that it timed out ("Task timed out after 20.02 seconds"), but there is no stack trace or indication of what what it was doing that timed out (waiting for some URL?), which makes this difficult to triage.

Here are example logs:

19:13:55
[06/Jun/2019:19:13:55 +0000] "GET /token?state=https%3A%2F%2Fd3sxx7zkzigde1.cloudfront.net%2F%23%2Fauth HTTP/1.1"307 11 "https://d3sxx7zkzigde1.cloudfront.net/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.157 Safari/537.36" -

19:13:55
END RequestId: 86e097d4-90ac-4fa6-9c40-92586125f011

19:13:55
REPORT RequestId: 86e097d4-90ac-4fa6-9c40-92586125f011 Duration: 95.12 ms Billed Duration: 100 ms Memory Size: 512 MB Max Memory Used: 512 MB

19:13:59
START RequestId: 864f7a3d-0f2c-4c08-83aa-b41684e8cea7 Version: $LATEST

19:14:19
END RequestId: 864f7a3d-0f2c-4c08-83aa-b41684e8cea7

19:14:19
REPORT RequestId: 864f7a3d-0f2c-4c08-83aa-b41684e8cea7 Duration: 20020.32 ms Billed Duration: 20000 ms Memory Size: 512 MB Max Memory Used: 512 MB

19:14:19
2019-06-06T19:14:19.346Z 864f7a3d-0f2c-4c08-83aa-b41684e8cea7 Task timed out after 20.02 seconds
2019-06-06T19:14:19.346Z 864f7a3d-0f2c-4c08-83aa-b41684e8cea7 Task timed out after 20.02 seconds

Copy the terraform.tf.example to terraform.tf file, substituting the appropriate values for bucket, dynamodb_table, and <stack>. This tells Terraform where to store its remote state.

from https://nasa.github.io/cumulus/docs/deployment/deployment-readme

When I deploy Cumulus v1.13.0, I'm getting an error that one of the lambdas is too large unzipped:

InRegionS3PolicyLambdaFunction | Unzipped size must be smaller than 262144000 bytes (Service: AWSLambdaInternal; Status Code: 400; Error Code: InvalidParameterValueException; Request ID: 50d92946-814b-11e9-bbaf-9f7f6002a11b)

S3 shows the file as 54.9 MB. When I extract it, it's 319 mb. Within the zip file, there are the following folders:

49M app
25M bootstrap
12K bulkDelete
14M cleanExecutions
16M createReconciliationReport
25M dbIndexer
27M distribution
5.7M emsDistributionReport
26M emsReport
25M executeMigrations
25M indexer
25M jobs
14M messageConsumer
4.1M payloadLogger
14M sfScheduler
4.2M sfSemaphoreDown
1.6M sfSnsBroadcast
4.2M sfStarter

It looks like the space is consumed by unminified output of webpack. All these files appear to come from the NPM distribution, so I'm curious how this works for other people.

Is this a known issue? I'll update this ticket once I find out more information.

This is moderately useful for debugging performance / timeout issues.

The word "exposure" is misspelled (exporsure) on the architecture page.

The FTP connection between Cumulus and MODAPS fails way too frequently. Here is the error message reported on the Cumulus Dashboard.

{"message":"FTP list failed for modpdr01.nascom.nasa.gov: read ECONNRESET","details":{"message":"read ECONNRESET"}}

What is causing this problem? Is this an issue at MODAPS? Would using a different protocol (switching away from FTP) be beneficial? How do operators at the DAACs trouble-shoot/dig deeper into issues like this as opposed to just looking at the logs that are displayed on the Dashboard?

I'm getting a timeout in the bootstrap lambda. This is most likely an environment issue on our side, that I'm debugging.

The log message we get does not have the full stack trace for the primary issue. I.e. you can't tell which of the bootstrap operations is failing, though it is likely connecting to one of DynamoDB or Elasticsearch.

All of these functions except the dynamodb one are use async/await, so it may be that if that function was async you'd get more - once I confirm that this is the case I'll file a PR.

    bootstrapUsers(get(users, 'table'), get(users, 'records')),
    bootstrapCmrProvider(get(cmr, 'Password')),
    bootstrapDynamoDbTables(dynamos)

Here are the logs:

{
    "error": {
        "name": "Error",
        "message": "Request Timeout after 50000ms",
        "stack": [
            "Error: Request Timeout after 50000ms",
            "    at /var/task/index.js:288620:15",
            "    at Timeout.<anonymous> (/var/task/index.js:288649:7)",
            "    at ontimeout (timers.js:482:11)",
            "    at tryOnTimeout (timers.js:317:5)",
            "    at Timer.listOnTimeout (timers.js:277:5)"
        ]
    },
    "level": "error",
    "message": "Request Timeout after 50000ms",
    "sender": "unknown",
    "timestamp": "2019-06-26T18:36:30.965Z"
}

{
    "level": "info",
    "message": "RESPONSE BODY:\n {\"Status\":\"SUCCESS\",\"PhysicalResourceId\":\"cumulus-bootstraping-daac-ops-api-deployment\",\"StackId\":\"arn:aws:cloudformation:us-west-2:---:stack/pipe-sit/f43602a0-9840-11e9-ad97-06972e890616\",\"RequestId\":\"a0f360cb-17e9-4d39-a0a5-f96deb39642d\",\"LogicalResourceId\":\"CumulusCustomResource\",\"Data\":null}",
    "sender": "unknown",
    "timestamp": "2019-06-26T18:36:45.829Z"
}```

@indiejames I noticed that we will have two discover-pdrs under tasks because wee have also called the discovery portion of PDR process discover-pdrs. I'm fine renaming ours. Any suggestions?

Hi, Couple minor things I saw while doing my first deployment.

https://github.com/nasa/cumulus/blob/master/docs/deployment/README.md

says to clone:

git clone https://github.com/nasa/ template-deploy

actual repo name is:

https://github.com/nasa/ cumulus-template-deploy

Since not all deployments are us-east-1, users will probably need to add a couple more parameters to the S3 bucket creation mentioned here:

https://nasa.github.io/cumulus/docs/deployment/create_bucket

aws s3api create-bucket \
  --bucket foobar-internal \
  --region your-zone \
  --create-bucket-configuration LocationConstraint=your-zone

Probably wouldn't hurt to have those parameters spelled out.

We have been using yarn in cumulus-api deployments. Lerna supports yarn and obeys yarn.lock but it doesn't support npm5 and the built-in package-lock.json.

I suggest we keep using yarn for installation and package management. Thoughts?

@bilts @jasongilman @indiejames

There are a few small changes that should be included for the shared-template.

Granule ingest duration and the average processing times are negative on the Cumulus Dashboard. It looks like (from the 3rd screen print) the Created, Ingest started, and processing started times are incorrect. If the Created, Ingest started, and processing started times are corrected, it may fix the ingest duration and average processing time issues.

When clicking to review schema examples, the links are incorrect and return 404.

e.g.:
https://cumulus-nasa.github.io/schemas/schemas/example-data/example-message-envelope.json

should be:
https://cumulus-nasa.github.io/schemas/example-data/example-message-envelope.json

We'll need to build this in CI and push to NGAP, rather than the current mechanism of copying from a manual S3 deployment.

In example/README.md, region is missing for the following command.

$ aws dynamodb create-table \
    --table-name my-tf-locks \
    --attribute-definitions AttributeName=LockID,AttributeType=S \
    --key-schema AttributeName=LockID,KeyType=HASH \
    --billing-mode PAY_PER_REQUEST

https://github.com/nasa/cumulus/blob/3b07b1c0fba29be8d860041959f5d5deeb03f454/docs/system-documentation/system-documentation.md

The section at the bottom: Error: Unable to import module 'index': Error

And the solution: "In order to resolve this issue, update the lambda source (in lambdas.yml, to point to the parent directory of the index.js file."

This is not correct. Add any syntax error inside your index.js file, and CloudWatch Logs will report this error. The error means that it is unable to import the module [because of some error].

It is not an error referencing the file.

When I expand the section in AWS Step Functions, it reveals the real error with my index.js file:

"cause": {
    "errorMessage": "Cannot find module 'node_modules/@cumulus/cumulus-message-adapter-js'",

But clicking the button in the Step Function console that says "CloudWatch Logs" just shows me the generic:

Unable to import module 'index': Error

For example, the following paragraph in [1] bothers me:

Also note, if you dont specify the amiid, it will try to use a default, which may or may not exist. The default AMI is an NGAP-approved AMI. The most recent NGAP AMI can be found using these instructions.

I wish CUMULUS is for non-NGAP users by default.

[1] https://nasa.github.io/cumulus/docs/deployment/config_descriptions

These two pages refer you to the Earthdata wiki:

https://nasa.github.io/cumulus/docs/system-documentation/system-documentation
https://nasa.github.io/cumulus/docs/operator-docs/lifecycle-policies

Would it be possible for someone with appropriate access to copy over the relevant information from that wiki?

Thanks.

It should be possible to add Lambda layers to a Lambda without having to provide your own cloudformation.template.yml file.

/cumulus/packages/deployment/app/cloudformation.template.yml:642

{{#each lambdas}}
  {{@key}}LambdaFunction:
    Type: AWS::Lambda::Function
    Properties:
      Code:
        S3Bucket: {{this.bucket}}
        S3Key: {{this.remote}}
      FunctionName: {{../stackName}}-{{@key}}
    {{#if this.layers}}
      Layers:
      {{#each this.layers}}
        - {{this}}
      {{/each}}
    {{/if}}
      Environment:
        Variables:

The addition is from {{#if this.layers}} to it's ending {{/if}}

Then layers can be specified in lambdas.yml such as:

MyLambda:
  runtime: python3.6
  handler: lambda_function.lambda_handler
  timeout: 300
  memory: 128
  s3Source:
    bucket: '{{buckets.internal.name}}'
    key: deploy/mylambda/0.1.0.zip
  layers:
    - arn:aws:lambda:us-west-2:552188055668:layer:geolambda:1

I could make a PR for this, but I'm not sure that the contributing guidelines are still valid given that issues aren't tracked within GitHub.

CMA deployment reports two things.

    "LayerArn": "arn:aws:lambda:us-east-1:12345:layer:tf-CMA-layer",
    "LayerVersionArn": "arn:aws:lambda:us-east-1:12345:tf-CMA-layer:1"

The following TF variable name seems to match LayerArn but example matches LayerVersionArn.

# Replace 12345 with your actual AWS account ID
cumulus_message_adapter_lambda_layer_arn = "arn:aws:lambda:us-east-1:12345:layer:Cumulus_Message_Adapter:4"

I'm not an NGAP user.

Error: error getting Lambda Function (gsfc-ngap-sh-s3-sts-get-keys): ResourceNotFoundException: Function not found: arn:aws:lambda:us-east-1:xxxxx:function:gsfc-ngap-sh-s3-sts-get-keys
    status code: 404, request id: 348ca34c-bdcf-4651-9dfd-013f71570325

The above error happens during

module.cumulus.module.archive.data.aws_dynamodb_table.granules: Refreshing state...

There appears to be a mismatch between the tags for at least versions 1.11.2 and 1.11.3 and the same versions published to npm.

Cloned, checked out 1.11.2, 1.11.3, 1.12.0, and 1.12.1, then installed from npm and used weld to compare the directories.

The differences in the 1.12.x versions were as expected - no test directories in the npm package, and other minor differences likely due to packaging (I didn't look in depth across all).

The differences in the 1.11.x tag versions and their npm package counterparts however were substantial.

Just showing an example screenshot of the API for 1.11.2

Blue shows file differences, and when examined these are significant differences in the code. There's also new files that have been added. All the sub-packages I looked at had similar differences.

This came up as I was using 1.11.3 and wanted to use a custom cloudformation.template.yml file and ran into problems until I realized there was a difference between the version I was using (the 1.11.3 tagged version in the repo) and the original version I was using in my deployment that was from npm.

NPM's server response with ENOAUDIT has been frustrating for the audit-ci project, random failures have unnecessarily halted CI environments.

The recent v2.0.1 release (Diff) should fix the retry mechanism for NPM, addressing failures in your pipeline (ex: https://travis-ci.org/nasa/cumulus/jobs/531849256).

v2.0.0 is a major release (CHANGELOG) with soft breaking changes (only deprecation warnings). Your report will look slightly different since it will show "important" info rather than just the "summary" (significant improvement IMO).

I have investigated your audit-ci.json file and there are no actions required other than updating to v2.0.1 and uncommenting the job in Travis.

As a follow up to #113, references to npm commands should be replaced with yarn commands.

Running npm run bootstrap as detailed in the README's setup instructions fails with an error like:

MacBook-Pro:cumulus aimeebarciauskas$ npm run bootstrap

> @1.0.0 bootstrap /Users/aimeebarciauskas/DevSeed/cumulus
> lerna bootstrap

...
lerna ERR! execute callback with error
lerna ERR! Error: Command failed: npm install
lerna ERR! npm WARN package.json @ No description
lerna ERR! npm WARN package.json @ No repository field.
lerna ERR! npm WARN package.json @ No README data
lerna ERR! npm WARN package.json @ No license field.
lerna ERR! npm ERR! addLocal Could not install /Users/aimeebarciauskas/DevSeed/packages/test-data
lerna ERR! npm ERR! Darwin 17.2.0
lerna ERR! npm ERR! argv "/Users/aimeebarciauskas/.nvm/versions/node/v4.3.2/bin/node" "/Users/aimeebarciauskas/.nvm/versions/node/v4.3.2/bin/npm" "install"
lerna ERR! npm ERR! node v4.3.2
lerna ERR! npm ERR! npm  v2.14.12
lerna ERR! npm ERR! path /Users/aimeebarciauskas/DevSeed/packages/test-data

Since yarn is a preferred node package manager because of how it locks dependencies, we should migrate cumulus to use yarn.

The application seems to be throwing an error once the access token expires, as there's no refresh token except after the first authorization.

To get around this for login purposes, you can revoke the access in the Google Account security management.

As far as reproducing this:

• Log in via Google Oauth
• Hit log out
• Immediately try to log back in with same account
• You should hit an error, which may be different depending on version.

1.11.2 looks like:

{
statusCode: 401,
error: "Unauthorized",
message: "The record has validation errors"
}

I'm following up this issue with a PR

I updated the LPDAAC_HTTP_MODIS provider on the Cumulus Dashboard today and the Last Update field didn't change. The Last Update for that provider stayed at "11:13:39 04/13/17". I would have expected that provider to have a new date and time stamp.

Check out the Last Update column in the screen print.

The following command

cfn-lint --template app/cloudformation.yml --region us-east-1 --ignore-checks W
generates some errors due to type:

E3012 Property Resources/es5Domain/Properties/ElasticsearchVersion should be of type String
app/cloudformation.yml:492:7

E3012 Property Resources/es5Domain/Properties/ElasticsearchClusterConfig/InstanceCount should be of type Integer
app/cloudformation.yml:494:9

E3012 Property Resources/es5Domain/Properties/SnapshotOptions/AutomatedSnapshotStartHour should be of type Integer
app/cloudformation.yml:497:9

E3012 Property Resources/cleanExecutionsLambdaFunction/Properties/Environment/Variables/nonCompleteExecutionPayloadTimeout should be of type String
app/cloudformation.yml:1322:11

E3012 Property Resources/cleanExecutionsLambdaFunction/Properties/Environment/Variables/completeExecutionPayloadTimeout should be of type String
app/cloudformation.yml:1323:11

E3012 Property Resources/cleanExecutionsLambdaFunction/Properties/Environment/Variables/nonCompleteExecutionPayloadTimeoutDisable should be of type String
app/cloudformation.yml:1324:11

E3012 Property Resources/cleanExecutionsLambdaFunction/Properties/Environment/Variables/completeExecutionPayloadTimeoutDisable should be of type String
app/cloudformation.yml:1325:11

This line attempts to remove the ElasticSearch hostname so that it does not appear in the logs.

https://github.com/cumulus-nasa/cumulus/blob/master/daac-ops-api/lib/bootstrap.js#L128

The key it's removing is "es", but the actual key is "ElasticSearch". As a result, the ES hostname is being shown in the logs.

@jasongilman I'm working on making sure lerna run test works for the whole repository. This command runs the test script on every package including gibs-dashboard and gibs-ops-api. It fails on these two repos because of linting error. I can fix the issues or I can wait for you to fix them. Whichever works better for you.

Let me know.

I see the following comment, and experienced the behavior as described by it:

**Imoprtant Note:** Configuring point-in-time recovery is not supported by the CloudFormation (as of June 2018). We enable this feature deployment using AWS API. However, due to a limitation of AWS API, the feature fails to be enabled if it is running against newly created tables.

This is no longer true - you can do the following in Cloudformation:

PointInTimeRecoverySpecification:
PointInTimeRecoveryEnabled: true

• update readme
• create root level entry point for using the library
• rename gitc to cumulus
• create authors/contributors file
• update package.json across modules
• prepare modules for publishing on npm

cc @bilts

The ECS settings support adding additional EBS storage, but never mount it to the EC2 images.

Sensitive information is removed from the request before it is logged:

https://github.com/cumulus-nasa/cumulus/blob/master/daac-ops-api/lib/bootstrap.js#L123

The usernames and PW hashes are still being displayed. That seems like something we wouldn't want to be logging.

Add \ after PAY_PER_REQUEST.

$ aws dynamodb create-table \
--table-name my-tf-locks \
--attribute-definitions AttributeName=LockID,AttributeType=S \
--key-schema AttributeName=LockID,KeyType=HASH \
--billing-mode PAY_PER_REQUEST
--region us-east-1

https://github.com/cumulus-nasa/cumulus/tree/master/cumulus/tasks/discover-granules function throws the following error when using a provider with http/https protocol
{
"level": 50,
"time": 1513367217626,
"msg": "Protocol http is not supported.",
"pid": 1,
"hostname": "ip----",
"name": "cumulus",
"file": "discover-granules/index.js",
"type": "Error",
"stack": "Error: Protocol http is not supported.\n at Object.selector (/var/task/discover-granules/index.js:21486:16)\n at handler (/var/task/discover-granules/index.js:8900:31)",
"v": 1
}

Ref: #28 (comment)

Under the section describing where Cumulus packages are located, there's a link to a search on the npm website for @cumulus, but doesn't look like npm's search handles that correctly.

Perhaps using cumulus as a keyword would be more appropriate if everything is definitely tagged with it? https://www.npmjs.com/search?q=keywords:CUMULUS

https://nasa.github.io/cumulus/docs/deployment/deployment-readme#option-1-get-packages-from-npm

• CUMULUS-398 - Fix not able to filter executions bu workflow

I think it should be "but."