Node.js
http
url
path
fs
readline
npm run start:dev
or
node server.js
http://localhost:8081/log/view
startDate Start date in YYYY-MM-DD format, e.g. 2021-01-12
endDate End date in YYYY-MM-DD format e.g. 2021-01-12
startTS Start datetime (unix time, UTC time zone) in millis, e.g. start=1369728000
endTS End datetime (unix time, UTC time zone) in millis, e.g. start=1369728000
At least one of
startDateorstartTSis required
The application takes a log file which maybe several 1000s of MB, and which is provided (for the time being) in the data folder. It reads the contents of the file and provides the necessary response to the user. The codebase is built using only the in-built modules of Node.js.
For reading the file, a readStream is created which doesn't load the entire file to memory and instead reads in chunks of data. A readline interface is used for reading line by line from the readStream.
The controller.log-view.js file takes care of the following 4 conditions
The date string is obtained from each line using split(). This is compared with the given datetimes. If the datetime is on or after startTS and on or before the endTS, we send the logs for the date range to our client
http://localhost:8081/log/view?startTS=1577844025909&endTS=1577844133845
{
"statusCode": 200,
"message": "Log file read successfully",
"data": {
"count": 7,
"logs": [
"2020-01-01T02:00:25.909Z Querying table projects",
"2020-01-01T02:00:47.057Z Finished reading projects",
"2020-01-01T02:01:00.812Z Querying table users",
"2020-01-01T02:01:18.265Z Response 200 sent to 142.220.133.184 for /example",
"2020-01-01T02:01:22.875Z Finished reading users",
"2020-01-01T02:01:50.798Z Querying table posts",
"2020-01-01T02:02:13.845Z Querying table projects"
]
}
}
Similar logic as the above Condition 1 could have been applied, but here 2 passes of reading the file is done; first we obtain the start and end index of the start and end date by substring search of the line, and on the second pass we push to logs array the lines between the start and end index
http://localhost:8081/log/view?startDate=2020-01-01&endDate=2020-01-03
{
"statusCode": 200,
"message": "Log file read successfully",
"data": {
"count": 11523,
"logs": [
"2020-01-01T00:00:40.528Z Finished reading users",
"2020-01-01T00:01:05.502Z Querying table users",
.
.
.
]
}
}
Simple substring search to see if line includes the requested date
http://localhost:8081/log/view?startDate=2020-01-01
{
"statusCode": 200,
"message": "Log file read successfully",
"data": {
"count": 5788,
"logs": [
"2020-01-01T00:00:11.172Z Request received from 211.254.246.209 for /about",
"2020-01-01T00:00:40.528Z Finished reading users",
"2020-01-01T00:01:05.502Z Querying table users",
.
.
.
]
}
}
Similar to Condition 3, handled with a substring search to see if the ISO date string is present on the line
http://localhost:8081/log/view?startTS=1577844025909
{
"statusCode": 200,
"message": "Log file read successfully",
"data": {
"count": 1,
"logs": [
"2020-01-01T02:00:25.909Z Querying table projects"
]
}
}
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent