Currently there is no way to search the DB for any websites from a consumer facing perspective.

Due to how BullMQ handles unique keys, it has serious scalibitly issues, especially since ram usage exponentially increases. It has been advised that a change to RabiitMQ or Kafka would be most beneficial.

There are concerns that mongo won't be able to scale to the proper searchability of a general Internet search engine. Some current contenders are RavenDB, and Weaviate.

Just need to make a simple privacy policy at some point to cover the legal basics

Need to improve documentation within and around the codebase.

To do list:

• improve type docs
  • explain purpose of everything
  • explain choices
• contributing guide