carryall / go-google-scraper-challenge Goto Github PK

Why

As there're are a lot of make request functions; authenticated and not authenticated, form and JSON request and it's getting messy now so we better refactor it

Acceptance Criteria

• All tests should pass

• connect to Postgres database
• ORM
• CRUD works

Why

Currently we didn't run any migration before running the test, and while running orm.RunSyncdb Beego ORM will sync the database tables according to the struct on models, which might produce a table structure that differs from the migration.
So we should either run the migration before the test or generate the schema and load that schema before running the test.

• run migration before test
• don't use orm.RunSyncdb

Issue

When displaying 2 result tables and one of them have less result the table row stretch out

Expected

The table row height should remain the same regardless of the number of results displaying

Steps to reproduce

1. Go to the production site
2. Login as
   email: [email protected]
   password: 1234567890
3. See the result table on the main page

Why

As a user, I can search the URL result by Regex

Acceptance Criteria

• query result URLs by Regex

Sample question this query should be able to answer
How many keywords have URLs in stored reports with 2 or more “/” or 1 or more “>”.

Why

Users should be able to search the result by keyword

Acceptance Criteria

• the /reports API should accept a keyword to search the result's keyword
• response with a list of search results that match the search keyword

• use the scraper lib https://github.com/gocolly/colly
• scrap Google search result on the keyword
• workaround to handle search API rate limit
• gather the following info
• Number of AdWords advertisers in the top position.
• Total number of AdWords advertisers on the page.
• URLs of the AdWords advertisers in the top position.
• Number of the non-AdWords results on the page.
• URLs of the non-AdWords results on the page.
• Total number of links (all of them) on the page
  HTML code of the page/cache of the page.

Why

Store the error and success message in separate so it's easier to maintain when we want to change the message

Acceptance Criteria

• store messages as constant in a file

Why

The mobile application clients ofter use a library to parse the JSON response with JSON API format so that the response for all API endpoints should follow that format

Acceptance Criteria

• Update the /oauth/clients API response to follow JSON API
• Update the /register API response to follow JSON API
• Update the /login API response to follow JSON API

Resources

JSON API

• provide /login Oauth 2 JSON API

Why

As a user, I can see a list of the report regarding the keywords from files I uploaded

Acceptance Criteria

• display result list on the main page after user logged in
• display result keyword and status

• replace log with Beego log

Why

Users should be able to see list of their search result reports

Acceptance Criteria

• add /reports API
• response with all user reports from all keywords
• response with pagination (optional)

• add test
• setup CI on Github Action

Why

As a user, I can see a detail of the search result so that I know

• Number of AdWords advertisers in the top position.
• Total number of AdWords advertisers on the page.
• URLs of the AdWords advertisers in the top position.
• Number of the non-AdWords results on the page.
• URLs of the non-AdWords results on the page.
• Total number of links (all of them) on the page
• HTML code of the page/cache of the page.

Acceptance Criteria

• Add an API results/:id

JSON Response should include the following fields

• id
• keyword
• array of non-ad links
• array of ad links object
  • position
  • type
• page cache

Why

Users need to be logged in before uploading the keyword file via API

Acceptance Criteria

• Response with 401 status when the request does not contain a valid authentication token
• Otherwise respond with 200 status

• setup OAuth server
• provide a page to generate new OAuth Client

• Go (1.15 or above)
• Beego
• Tailwind
• SCSS

• add results table in DB
• store the result to DB

The model structure:

type Result struct {
	Keyword string
	Adwords []Adword
	NonAdLinks []string
}

type Adword struct {
	Link string
	AdType string
}

const (
	TOP_IMG_AD = "TOP_IMG_AD"
	TOP_LINK_AD = "TOP_LINK_AD"
	SIDE_IMSGE_AD = "SIDE_IMSGE_AD"
	BOTTOM_AD = "BOTTOM_AD"
)

Issue

Some style on Staging is missing

Expected

Expect the style on the staging site to be the same as on the development

Staging	Development

Steps to reproduce

1. run the project locally and visit http://localhost:8080/signin
2. visit https://google-scraper-staging.herokuapp.com/signin and compare the design

• using Ginkgo

Why

Currently in testing static string used on fabrication

Acceptance Criteria

• add Faker
• replace static string with Faker dummy data on fabrication

Why

Beego stores the session in memory by default so when restarting the server the session gone. So we should keep the session somewhere else.

• use Redis/Postgres/file instead of memory
  https://beego.me/docs/mvc/controller/session.md#example-usage

Issue

The deployment pipeline is currently broken with this error

So that after the PRs are merged it didn't deploy to the staging site

Expected

The merged PRs should be deployed to the staging site

Steps to reproduce

1. Merge the PR
2. The change is not deployed on the staging site

Why

As a user, I can upload a CSV file of keywords

• the file can be of any size from 1 to 1,000 keywords

Acceptance Criteria

• add upload form
• extract keywords from file
• upload form validation
• display error message when user try to upload a file with > 1000 keywords
• style for upload form

• provide /logout API

Why

As after I revamp the project from Beego to Gin but the UIs are not migrated yet so to populate the user account a registration API is needed

Acceptance Criteria

• Add /register API
• Provide a way to generate client ID and secret

Why

After pausing this IC for a while, Nimble has adopted the Gin framework so we better update this IC to align with the Gin template which would be used to bootstrap client projects.
The existing UI features would be deprioritized in favor of API stories.

Acceptance Criteria

• Replace the codebase with a fresh new application bootstrapped from the Gin template
• Move any useful files along to the new codebase (so we can use them later)

Resources

• Gin template
• Gin framework

Why

The test fail because of the dependency conflict

Acceptance Criteria

• Bump the version of Tailwind packages

• rename user table to users

• layout for register screen
• style for registration form
• SVG icons
• alert template
• set default controller attributes

Why

Display a lot of reports on the first page is a bad UX, so adding pagination to the result page is something we should do

Acceptance Criteria

• display user's latest 20 results on the first page
• add pagination

Why

As a user, I can see a detail of the search result so that I know

• Number of AdWords advertisers in the top position.
• Total number of AdWords advertisers on the page.
• URLs of the AdWords advertisers in the top position.
• Number of the non-AdWords results on the page.
• URLs of the non-AdWords results on the page.
• Total number of links (all of them) on the page
• HTML code of the page/cache of the page.

Acceptance Criteria

• Add a link from the result link on the main page to the result detail page

Display

• Links on the page separate by Adwords and non-Adwords with the total number of them
• Adwords links by position with the total number of them
• Non-Adwords links with the total number of them
• A link to display the page cache

• login page
• login form
• reuse form style from sign up
• set current user to session when successfully logged in
• redirect to main page when successfully logged in
• display errors if failed to login

Why

Users should be able to upload a CSV file of keyword via API

Acceptance Criteria

• Add a API endpoint with POST method which receive a CSV file
• get keywords from file and validate keyword size from 1 to 1,000
• response with error if validation fail
• save keywords to DB if validation pass

Note: the scraping part would be handled on another backend story

Why

To make the code cleaner, we should centralize the error title, detail and code in one place

Acceptance Criteria

• Refactor the error response rendering

Issue

When the file contains more than one keyword, the response contains only one result

Expected

The number of results in the response should be as same as the number of keywords in the file

Steps to reproduce

1. Call the file upload API POST /results with a file containing more than one keyword
2. Observe the response

Why

As a user, I can log in via API with a valid client id and secret so that I can use keywords to search

Acceptance Criteria

• Add a API /login
• Which should return access token and refresh token

Why

As a developer, we should be able to create an OAuth client so that we can use its ID and secret on authentication requests

Acceptance Criteria

• Add POST /oauth/clients API

Why

Auto log user in after they sign up makes it easier for new users

Acceptance Criteria

• after signup the user supposed to stay logged in and redirect to the main page

• deploy to staging and production
• setup the CD on Github Action

• provide registration screen on /signup
• registration form
• database migration for users
• user model

Issue

There's an error on staging and production because of a missing database relation

Expected

The main page should show when visiting the root path

Steps to reproduce

1. Go to production site or staging site
2. The page show error

Why

Add Github issue template so it's easier to create an issue

Who Benefits?

Developer

Why

Currently, a function in base controller is used to ensure the valid authenticated header from the request, but we'll need to call that function on every private controller action and also need to include a pair of test cases to check the valid and invalid request header

Some example can be found in varun-api and recruitmate-web

Acceptance Criteria

• Use the middleware instead of the guard function
• Adjust the routes into route groups

• add logout link
• redirect to login page after logged out

Why

Refactor form validations regarding these constrain

• current app show only one form error
• while setting the error to validation there's an error return after set, should we handle and check whether it set or not?

// SetError Set error message for one field in ValidationError
func (v *Validation) SetError(fieldName string, errMsg string) *Error {
	err := &Error{Key: fieldName, Field: fieldName, Tmpl: errMsg, Message: errMsg}
	v.setError(err)
	return err
}

• prefer separate validation into functions instead of expose them in Valid function
• prefer setting the error within each validation function

Acceptance Criteria

• all test should pass

Why

After revamping the application and replacing Beego with Gin so the previous Beego task which handles the search result scraping would no longer work so we need to replace it with a background job

Acceptance Criteria

• Setup cron job or worker e.g.: gocraft/work, go-co-op/gocron
• Handle the scraping after keywords are stored in DB

Why

The current scraper is using a normal in-memory queue and not yet running in the background which would end up getting blocked by Google

Acceptance Criteria

• run the scraper in the background using either work or task package
• might find the way to fake proxy

Why

As a user, I can search the result keyword by keyword

Acceptance Criteria

• Add search form
• When submit a search form filter the search result with the given keyword
• Display number of result