A search engine for sequence data
License: Apache License 2.0
Multiple parts of our webapp will have requirements on their performance, such as the FastaProcessor. We need to investigate possible solutions for this.
FASTA a might not be a complex format, still it imposes a lot of challenges on asynchronous parsers. Come up with a prototype of an asynchronous parsers that uses Akka streams
We have to support multiple indices to allow users to query different data sets.
/index/1While playing with the FileUpload feature it became clear that the FastaProcessor is not stable enough to be used in production (it is just a prototype). Some edge cases that lead to trouble are:
Create REST endpoint to index FASTA files.
It should be very helpful to have an impression about the current of the index (e.g. How many sequences are indexed?)
The webapp should be almost entirely controllable via REST. As of now, there are only a few controllers but as this number increases we have to document them systematically.
The current FastaFlow expects the incoming bytestrings to be lines. This is doomed to be violated some day.
Write about:
We now have a relatively clear picture of how our matching algorithm should work. To really think it through it may be best to document it.
This should serve as a mock for a potential upload of a (FASTA) file that is uploaded so the user can search our index against the contents of his file. Because there is no index as of now, it suffices to show the content of the file to check if our FASTA processor really works/performs.
When the file is uploaded, all its contents are shown on the page.
Our search function works but we are not 100% sure how. For example fuzzy matching works to a certain extent but we have to be sure to which extent.
The application starts to have some parameters so we should investigate how to do this is properly in Scala. A good start might be: https://github.com/lightbend/config
To really make us of our super cool FileUpload pipeline, we have to create a sink to pipe all the incoming stuff into, which in the far end is Lucene.
That is why we have to come up with a custom sink for this.
At some point in time we have to care about styling. This issue should prototype a very basic style to look how we can integrate it into our build process.
I (personally) doom runtime dependency injection. I want the compiler to check if everything is wired correctly. This ticket should migrate the application from the play-default (runtime) to read-monads (compile-time).
This may not be super urgent but it is easier to switch now than when the application is large.
We need a way of specifying parameters of our project. E.g our configuration can host production and test databases but the database that is actually used at runtime still has to be specified.
After spending a month on really digging deep into the Polay framework and Akka, I can hardly see an advantage using Play instead of just using Akka Http. A lot of things should get simpler when we switch to Akka Http entirely, such as compile-time dependency.
Further down the road I want this project to have a UI. This should be done with a webapp. To clearly separate the logic of the webapp from other parts of the project we have to create a subproject.
sbt webapp/runThe Lucene indexing logic (it's tokenizers and so on) are independent of akka streams and our domain models. Therefore it should be separated into a subproject (named index).
In issue #25 we investigated the possibility to also index the reverse complement. It turned out, that this is the wrong approach. Instead, we should additionally reverse complement the search query and search with both queries against the same index.
It is to easy to loose track of your test coverage. That's why we have to setup a pipeline for this.
The user needs to submit his search queries via UI. We have to build a component for this.
It is becoming tedious to write the same scaffold over and over again. GitHub supports issue-templates so we should use it.
Code formatters are a safety net for the programmer in case he (or his IDE) missed something. Also, it presents a common standard for everyone who is contributing.
As an initial hoster we can use Heroku. It is not only free but there are also hooks into our CI.
Additionally, Heroku is something nice to learn about.
It is very inefficient to return the matched positions in lucene. That's why we have to shrink the size of our documents in the database.
The ultimate goal of the project is to provide a (near) real-time search experience against large sequence data sets such as NCBI. To accomplish this, our indexing process must be a lot smarter. Fortunately, Lucene is extremely customizable.
We should write our own tokenizer which follows best practices of algorithmic/biological pattern matching:
As always in this day and age of this project, we don't have to be perfect here. It suffices to make it work without extremely high latencies.
Focusing on Scalameter was a premature decision. It turned out to be not a mature benchmark framework. I small investigation into JMH turned out to be very succusful. Additionally, JMH seems to be the industry standard.
We want to show the world that we care about stability. A visual proof that our tests are green is a great thing to have.
As a first visual step towards a search engine, a search bar is needed. This issue should also serve a blueprint on now to integrate the Javascript ecosystem into the project.
Create REST endpoint to search for sequences in the index.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.