Giter Club home page Giter Club logo

crawlexir's Introduction

Crawlexir

Introduction

Web application extracting large amounts of data from the Google search results page.

Project Setup

  • Install dependencies with mix deps.get
  • Create and migrate the database with mix ecto.setup
  • Install Node.js dependencies with cd assets && npm install
  • Start Phoenix endpoint with mix phx.server

Now you can visit localhost:4000 from your browser.

Tests

Execute all test with mix test

crawlexir's People

Contributors

olivierobert avatar

Stargazers

avatar Tolsee avatar Alyson avatar

Watchers

avatar James Cloos avatar Rossukhon Leagmongkol avatar

crawlexir's Issues

As a user, I can sign up

Implement the signup page and backend logic to sign up a users

Acceptance Criteria

  • Implement sign up form with two fields: email and password
  • Validate that email must be unique and case insensitive
  • Store the user details into a table

As a user, each of my keywords trigger a search on Google

Schedule an async job/task to fetch results from Google search page.

Acceptance Criteria

  • Compare pros and cons to use a full-fledged queue system backed by Redis vs. using Supervisors/Tasks
  • Do NOT implement search result page scarping as this will be done in another task

As a user, I can see the list of my searched keywords

List all user keywords with status of fetching results.

Acceptance Criteria

  • Add new method to Search context to list user keywords
  • Add flag to the keyword to know if the keyword has been processed already or not

As a user, I can upload a CSV file

Implement the CSV upload form for keywords.

Acceptance Criteria

  • Implement upload form with single file input field
  • Show errors if CSV file is empty or of incorrect format
  • Show success message if the file upload was successful
  • Do NOT parse the CSV file yet as this will be done in another task

As a user, I can sign in

Implement the sign-in page and backend logic to log in a users

Acceptance Criteria

  • Implement sign-in form with two fields: email and password
  • Show errors if the credentials are incorrect
  • Store the logged in users in the session
  • Redirect anonymous users back to the login page

As a user, my keywords are parsed from my CSV file

Process the uploaded CSV file from the upload form.

Acceptance Criteria

  • Process the CSV file and store each keyword in a new table for keywords
  • Show errors if CSV file is empty or of incorrect format
  • Show success message if the file upload was successful

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.