raju009f Goto Github PK

canvaas-coding-test

data-engineering-101

Introduction to Data Engineering workshop, learn to build a data pipeline with Luigi!

flask-rest-ful-app

flask-restful-api-app

gcbdemo

GCP CI CD Pipeline using Cloud Build and Cloud Run etc ...

gcpdlp

GCPDLP

hadoop-spark

hello-world

simple repository

luigi-tfidf-similarity

machinelearning

my-work

myprofile

MyProfile

os-sample-python

Sample Python Flask application for testing OpenShift 3 deployment using OpenShift default Python S2I builder and gunicorn.

placement-stuff

Placement coding problem practise

professional-services

Common solutions and tools developed by Google Cloud's Professional Services team

retail_db

retail_db

retail_db_json

retail_db_json

tfidf-similarity

The Luigi pipeline should contain the following tasks Parse/Cleanup - This step should parse the documents.txt file and remove any punctuation Compute TF - This step should compute the term-frequency for each term in each document. Compute IDF - This step should compute the inverse document frequency for each term Compute TF-IDF - This step will compute the TF-IDF weight of each term in each document Compute Similarity - This step should determine the similarity between all documents by calculating the Euclidian Distance between each TF-IDF vector. Expected output The final output of the pipeline should be a csv with columns corresponding to: Document 1 ID Document 2 ID Similarity Document id is in the index of the document according to the order defined by the input file. Output is ordered from most similar to least. Similarity computed for each pair of documents in the input where document_id_1 < document_id_2