Giter Club home page Giter Club logo

ipfs-search-frontend's Introduction

pipeline status Maintainability Test Coverage Documentation Status Go Reference Backers on Open Collective Sponsors on Open Collective

Search engine for the Interplanetary Filesystem. Sniffs the DHT gossip and indexes file and directory hashes.

Metadata and contents are extracted using ipfs-tika, searching is done using OpenSearch, queueing is done using RabbitMQ. The crawler is implemented in Go, the API and frontend are built using Node.js.

The ipfs-search command consists of two components: the crawler and the sniffer. The sniffer extracts hashes from the gossip between nodes. The crawler extracts data from the hashes and indexes them.

Docs

Documentation is hosted on Read the Docs, based on files contained in the docs folder. In addition, there's extensive Go docs for the internal API as well as SwaggerHub OpenAPI documentation for the REST API.

Contact

Please find us on our Freenode/Riot/Matrix channel #ipfs-search:matrix.org.

Snapshots

ipfs-search provides the daily snapshot for all of the indexed data using snapshots. To learn more about downloading and restoring snapshots please refer to the relevant section in our documentation.

Related repo's

Contributors wanted

Building a search engine like this takes a considerable amount of resources (money and TLC). If you are able to help out with either of them, do reach out (see the contact section in this file).

Please read the Contributing.md file before contributing.

Roadmap

For discussing and suggesting features, look at the issues.

External dependencies

  • Go 1.19
  • OpenSearch 2.3.x
  • RabbitMQ / AMQP server
  • NodeJS 9.x
  • IPFS 0.7
  • Redis

Internal dependencies

Building

$ go get ./...
$ make

Running

Docker

The most convenient way to run the crawler is through Docker. Simply run:

docker-compose up

This will start the crawler, the sniffer and all its dependencies. Hashes can also be queued for crawling manually by running ipfs-search a <hash> from within the running container. For example:

docker-compose exec ipfs-crawler ipfs-search add QmS4ustL54uo8FzR9455qaxZwuMiUhyvMcX9Ba8nUH4uVv

Ansible deployment

Automated deployment can be done on any (virtual) Ubuntu 16.04 machine. The full production stack is automated and can be found in it's own repository.

Contributors

This project exists thanks to all the people who contribute.

Backers

Thank you to all our backers! 🙏 [Become a backer]

Sponsors


ipfs-search is supported by NLNet through the EU's Next Generation Internet (NGI0) programme.


RedPencil is supporting the hosting of ipfs-search.com.

Support this project by becoming a sponsor. Your logo will show up here with a link to your website. [Become a sponsor]

ipfs-search-frontend's People

Contributors

dependabot[bot] avatar dokterbob avatar ember-tomster avatar madnificent avatar mehran-naghizadeh avatar vandrongelen avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

ipfs-search-frontend's Issues

Use search result for discovering file type, rather than search query

The API has been extended (a while ago) allowing us to discover the file type without fetching the type or relying on the information from the search. Discovering the file type makes the previews work more often and would make the overview nicer.

The filetype is currently supplied to the hint card. We can remove it, calculate it where needed, cleaning up the code in one go.

Keep filename in URL

As described here: https://github.com/ipfs/go-ipfs/blob/master/docs/gateway.md#filenames

All direct URLs should have an extra parameter filename with the file name in order to let the browser find the file type and display the content correctly

Example:
https://ipfs.io/ipfs/<hash>?filename=<filename>.<extension>
Or even with extension only
https://ipfs.io/ipfs/<hash>?filename=.<extension>

This seems easy to implement because we always have the filename or the mime type provided by the metadata

User documentation / "Help" menu

On the main page: http://ipfs-search.com/ there are API, Code and Contact hyperlinks. I propose that you add and "Help" one. The suggested content can be the following:

  1. What is this engine about
  2. What kind of search language is supported
  3. What is the format of the results
  4. Examples

Best regards.

Show parent directories

When i search for file in ipfs-search i see that it have long path in the name.
Example:
\distributed_lab\Новичок\extra\habr\SPHINCS — Википедия_files\444px-Binary_hash_tree.png

I want to browse folder that contain that file but have only file hash. Can you additional show directory hash in file properties? It can be hash of root directory (\) and directory that contains that file (SPHINCS — Википедия_files).

Build is broken

The current build instructions result in a failing build (#2), which makes it impossible to update the production version and it inhibits outside contribution.

Please, provide updated build instructions (Docker-based or otherwise) or any other means of reliably creating production and development builds.

New release :)

@vandrongelen @dokterbob An issue for keeping up with drafting the next release as mentioned in #36. Helps us keep it separate from specific issues. Feel free to comment or add to this issue ^_^

  • Merge changes of #36
  • Approve or clean #7 #32
  • Ship it!

Humanize byte size formatting

As in the current go-ipfs implementation, the current search engine reports the byte sizes of the objects. While this is good in some cases, it is not convenient to be read by humans at all. Hence, my proposal is to introduce more friendly like formatting. For example the effect should be as the common POSIX switch -h is used as in ls -lh, du -h.

Depending on the choice 1000 base or 1024 can be used. IMO stick to https://en.wikipedia.org/wiki/Orders_of_magnitude_(data) conventions.

if less than 1000 bytes →XYZ B
if less than 1000^3 bytes →XYZ KB (if X KB →X.Y KB)
if less than 1000^6 bytes →XYZ MB (if X MB →X.Y MB)
....

Best regards.

Pagination not working at all

Currently, clicking to pages > 0 are not working well. If I then reload the browser, it kind of seems to work again.

Related: #8?

Title should not be IpfsSearch

The title of the page should be IPFS Search or ipfs-search.com.

Ideally/eventually it would be: <query> · ipfs-search.com and ipfs-search.com when there's no query.

900 KB JS file

The website currently includes a whopping 900 KB JS file. Could we reduce this to something reasonable, like 150-300 KB?

Design problems

  1. Preview images width bigger than container width.
  2. Can't see current page
  3. Can't see what last previwed link

Need to set style rule for img:

img{
    max-width: 100%;
}

Before
before

After
after

Need to show current page

.page-link.current-page {
    background-color: lightgray;
}

Before
pages-before

After
pages-after

Show last previwed result

.selected{
    background-color: lightgray;
}

Before
previwed-before

After
previwed-after

ipfs: video streaming is not allowed

When try to preview some video files clowdflare-ipfs return 403 error with this message:

ipfs: video streaming is not allowed

Need to add alternative sources for video by using source tag.

<video controls preload="metadata">
 <source src="https://clowdflare-ipfs.com/ipfs/...">
 <source src="https://gateway.ipfs.io/ipfs/..." > 
</video>

If CF return 403 than video switch to gateway.ipfs.io.

Make metadata button clickable and unselectable

The 'Show metadata' button should explicitly be clickable and not selectable so it behaves as expected. Because it looks like a button, we should probably convert it into a button tag. If that would make it hard on the styling, we could revert to CSS in order to make it behave as we'd want it to.

Discussion: disable package-lock.json

NPM has a package-lock.json to indicate specific dependencies which tends to be rather large. Assuming dependencies use semantic versioning themselves, the lockfile provides little value.

There is value to be gotten from this lockfile when dependencies change incorrectly. The value of this safety-net may be negated by the change of file format across NPM versions. We have disabled the use of this lockfile globally and across all of our projects for now.

Should we disable the package-lock.json explicitly for this repository so people that clone it don't encounter the file and accidentally end up with a bunch of changes?

Serve or build results in page without working components

Having installed the app according to the instructions and then running ember serve or ember build results in a site where typing in the search bar or clicking on the filter does not yield characters typed or the filter control opening and gives the following JS error:

Uncaught TypeError: Cannot set property placement of [object Object] which has only a getter
    at defineProperty (ember-metal.js:886)
    at EmberPopper.CoreObject (core_object.js:129)
    at EmberPopper.EmberObject (object.js:28)
    at EmberPopper.FrameworkObject (object.js:72)
    at EmberPopper.Class (core_object.js:207)
    at EmberPopper.Class (core_object.js:207)
    at new Class (core_object.js:207)
    at new EmberPopperBase (ember-popper-base.js:60)
    at new EmberPopper (ember-popper.js:8)
    at Function.create (core_object.js:220)

It seems that on page load, I'm also getting the following error:

jquery.js:3818 jQuery.Deferred exception: Cannot set property placement of [object Object] which has only a getter TypeError: Cannot set property placement of [object Object] which has only a getter
    at defineProperty (http://localhost:4200/assets/vendor.js:44215:30)
    at EmberPopper.CoreObject (http://localhost:4200/assets/vendor.js:55473:46)
    at EmberPopper.EmberObject (http://localhost:4200/assets/vendor.js:55872:75)
    at EmberPopper.FrameworkObject (http://localhost:4200/assets/vendor.js:55916:78)
    at EmberPopper.Class (http://localhost:4200/assets/vendor.js:55551:72)
    at EmberPopper.Class (http://localhost:4200/assets/vendor.js:55551:72)
    at new Class (http://localhost:4200/assets/vendor.js:55551:72)
    at new EmberPopperBase (http://localhost:4200/assets/vendor.js:89894:22)
    at new EmberPopper (http://localhost:4200/assets/vendor.js:90245:21)
    at Function.create (http://localhost:4200/assets/vendor.js:55564:16) undefined

Tested in Chrome, running NodeJS 10.13.0, ember-cli 3.5.1

What will filters look like?

Pretty soon, we could introduce filters for:

  • first_seen
  • last_seen
  • duration (audio/video)
  • megapixels (images)
  • file_size

What might this look like and/or function?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.