sarthakmakhija / goselect Goto Github PK

Provide support for "isArchive" scalar function. isArchive must take 1 parameter that is the mimeType. User can pass mimeType attribute or any literal that represents a mimeType like text/plain

Example usage: isArchive(mimeType)

Sample queries for isArchive

select * from . where isArchive(mime)
select * from . where isArchive(mimeType)

Constraints

1. isArchive MUST accept 1 parameter, throw an error if sufficient parameters are not provided
2. isArchive MUST be registered as a where clause function
3. isArchive MUST return true if the file has an archive mime type. Look at the library mime for supported mime types. goselect uses mime for determining the mime type of a file
4. Tests must include all the mime types that are considered as archive. For example, application/x-7z-compressed, application/zip etc and also the mime types that are not considered as archive

Necessary details

1. New functions are onboarded in Functions.go
2. Scalar function definition goes in ScalarFunctionBlock.go

Reference

1. Other scalar functions in ScalarFunctionBlock.go
2. Other scalar functions that work on mimeType like isText, isVideo etc

goselect should run on Windows OS. At this point, goselect runs on MacOS and linux with ARM64, AMD64 and i386 processor.

One of the main challenges in ensuring that it runs on windows is the way golang exposes

• CreatedTime, ModifiedTime, AccessTime from file API
• The way user and group information needs to be found in windows
• The way block information needs to be found in windows
• The way hidden file are found in windows (hidden_windows.go already exists)

In order to provide support for windows we must make the following changes:

Changes

1. Provides CreatedTime, ModifiedTime, AccessTime for windows by providing a go file built for windows
2. Provide user and group information for windows
3. Provide blocks information for windows
4. Ensure all the existing tests run
5. Create a build pipeline for windows and all the tests should run

Reference

1. Look at the platform package inside context

goselect supports regular expressions in a scalar function like. These regular expressions do not support group that means a function like like(name, ([0-9]?)([a-z]{1})) does not work. This does not work because the goselect parser does not support parentheses inside a function.

goselect plans to use to delimit a group inside a regular expression. This means the above like function would be like(name, <g>[0-9]?</g><g>[a-z]{1}</g>)

Scope

1. Adding a test inside tokenizer to confirm the type of tokens that would be generated given a string of the form like(name, [0-9]?[a-z]{1})
2. Like function is already available in ScalarFunctionBlock.go. The first change must be replacing and inside like function with ( and ) respectively.
3. After and have been replaced in the regular expression string, it must be cached. Like function already leverages FunctionExecutionCache
4. Like function must also match every with its counter part and throw an error if the groups are not well formed
5. Add tests to ensure valid and invalid groups
6. An integration test to ensure that like function with group works even if the user enters group delimiters as:

<G></g>
<G></G>
< g ><g />
< g>< g/>
< g>< g / >

Necessary details

1. New functions are onboarded in Functions.go
2. Scalar function definition goes in ScalarFunctionBlock.go

Reference

1. Like functions in ScalarFunctionBlock.go
2. Tokenizer is defined in Tokenizer.go

Provide support for "between" scalar function. Between must take 3 parameters, the one first is the attribute that needs to be compared against the second and the third parameter values.

Example usage: between(size, 100, 600)
indicates that the size must be between 100 and 600 bytes (both inclusive) for between to return true.

Sample queries for between

select * from . where between(size, 100, 600)
select * from . where between(size, parseSize(1 Mib), parseSize(5 Mib))

Constraints

1. between MUST have bet as an alias
2. between MUST accept 3 parameters, throw an error if sufficient parameters are not provided
3. between MUST throw an error if the value of the third parameter is less than the value of the second parameter. Eg; between(size, 100, 10) MUST throw an error
4. between MUST be registered as a where clause function
5. between MUST throw an error if the parameter values are not comparable. For example, between(size, 10, hello) MUST throw an error
6. One of the tests MUST highlight the use of between with timestamp type values, with string, with numeric values and with values that can't be compared

Necessary details

1. New functions are onboarded in Functions.go
2. Scalar function definition goes in ScalarFunctionBlock.go

Reference

1. Other scalar functions in ScalarFunctionBlock.go

Provide support for "contents" attribute. This attribute represents the contents of a file.

Sample queries using contents attribute

select * from . where like(contents, LengthFunctionBlock)
select * from . where like(contents, TrimFunctionBlock)

Constraints

1. contents MUST be a lazily evaluated attribute. Refer AttributeLazyEvaluationBlock.go to understand lazy evaluation
2. Once contents is evaluated it must be set in FileAttributes object
3. contents MUST be evaluated only for text files
4. contents MUST be evaluated only for text files if the size is less than 20 Mib, else set an empty string in FileAttributes object
5. Add tests to cover evaluation of contents at least for text files, hidden files that are text files, an image file, a doc and a pdf file

Necessary details

1. New attributes are onboarded in Attributes.go
2. FileAttributes.go acts as a wrapper over fileInfo

Reference

1. Refer mimeType attribute for lazy evaluation in Attributes.go and FileAttributes.go