aws-samples / data-lake-as-code Goto Github PK

Hi, I am no longer able to access the CHEMBL29/27/25 datasets from S3. ie: s3://aws-roda-hcls-datalake/chembl_29/

Error: All access to this object has been disabled (Status Code: 403; Error Code: AllAccessDisabled)]

Can someone please confirm that the access is still available?

Code version:

• branch: mainline
• commit: 57766358d414da5f427f0786fffe85dce48c2244
• bug location: bin/aws.ts line 23

Steps to reproduce:

• git clone the project
• npm install
• uncomment bin/aws.ts lines 22-26
• cdk synth CoreDataLake ExampleS3DataSet

Expected output:

App at '' should be created in the scope of a Stack, but no Stack found
Subprocess exited with error 1

As of AWS CDK 1.70.0, s3.Bucket.fromBucketName expects the type of its first argument (scope) to be Construct, not App.

Proposed solution:

Initialise the S3 bucket in lib/ExampleS3DataSet-stack.ts line 26 instead of passing it as a prop:

sourceBucket: s3.Bucket.fromBucketName(this, 'exampleS3DataSetSourceBucket', '--- YOUR EXISTING BUCKET NAME GOES HERE ---'),

I'm consistently getting the following error when trying to create the datalake as code from the Cloud Formation Template. I don't seem to have access to the template. Is it not public anymore?

S3 error: The bucket you are attempting to access must be addressed using the specified endpoint.

I'm trying to demo Athena using the pre-built RODA Clinvar dataset. I deployed using the stack here: https://github.com/aws-samples/data-lake-as-code/tree/roda#readme.

The stack deploys without any issues noted, and after that I'm able to query most of the tables from Athena. However, when I run the following command in Athena:

SELECT * FROM "clinvar_summary_variants_dl-awsroda"."submission_summary";

The count of records returned is as-expected, but the values in every column are empty.

Notably, there appears to be a disconnect between the column names in the glue data catalog and the column names in the corresponding parquet metadata: in the data catalog columns are assigned distinct names (e.g. variationid, clinicalsignificance,...) in accordance with the official clinvar file definition README, but in the parquet file for this table each column appears to be named 'col' + autoincremented integer (e.g. col0, col1,...).

Hello,

Was accessing the data up until a yesterday, but now it appears to be missing from the s3://aws-roda-hcls-datalake/clinvar_summary_variants/

Will this data re-appear sometime?

Hi,

I am trying to launch this Athena query on "opentargets_latest_dl_awsroda" database:

SELECT *
FROM targets
LIMIT 10;

The result is empty. Indeed, following the AWS Glue Data Catalog, it appears that the S3 "s3://aws-roda-hcls-datalake/opentargets_latest/targets/" path does not contains data.

Is this intentionally?

Build scripts call aws sts get-caller-identity and then generate role arn by user arn. Looks very strange - why role name should be related with current user name? Could you please add explanation about this? For now I got error: starterAdminPermission Invalid principal, arn: arn:aws:iam::111111111111:role/user.name (Service: AWSLakeFormation; Status Code: 400; Error Code: InvalidInputException; Request ID: 32d6e4c9-a0e3-4af9-a938-521737333d7a; Proxy: null)

Could you please post iam role definition to create it manually?

Right now only IAM users and roles are supported.

Hi,

I used OpenTargets CloudFormation to create an AWS Glue catalog and queried data using Athena. However, I recently noticed that some data in the tables is missing, such as in the searchdisease, searchdrug, searchtarget, and molecule tables.

I'm certain that the data was previously there, but I'm not sure why it disappeared, or if it was just my case. I checked and it's possible that the data was removed from the source (https://platform.opentargets.org/downloads/data), and as a result, was also removed from the associated Glue tables.

Could you please check the issue? Thank you!

Best,
Yuki

Hi Guys,

I'm form OpenTargets. One of our users reported that OT data fetched from S3 has some problem: the data seems to have unexplainable duplication. We believe the problem might due to how the data is synced from EBI ftp. The datasets our pipelines generated via spark are partitioned into smaller chunks with filenames containing a release specific hash. As the hash is different from release to release, the line below probably will not overwrite the content of the S3 buckets, instead, these chunks keep accumulating.

For more details, please see the issue in our tracker.

Hi!

I'm trying to access y8m dataset, but getting the AccesDenied:

$ aws s3 ls --no-sign-request s3://aws-roda-ml-datalake/yt8m_ods/

An error occurred (AllAccessDisabled) when calling the ListObjectsV2 operation: All access to this object has been disabled

Couple of weeks ago the same command worked fine. Could you please suggest how to get the data with s3 client?