this would be an enhancement to the existing project

We don't have any tests at the moment and would love to test our scrapers.

Let's discuss the choice of tester.

• Unittest https://docs.python.org/2/library/unittest.html

Currently, a file contains all the queried strings and the results of each query are stored in separate files.

Add a database to store the queries and strings.

Image and video results don't have any endpoint in API of query server currently, which specifically takes file-type as input. Please implement it to allow us to include a query service for images and videos as well.

Right now we have the following:

• Server in Node.js
• Scrapper in python

and the server executes the python as a process. Which is not efficient at all. Imagine once this is integrated into the Open Event project. There might be many API calls coming to the query-server and each of these API calls will be spawning a new python process. We'll be having so many process spawns (one for each API call).

This will also make adding unit tests etc easier since all is in a single platform.

Instead, we are looking to implement the following

• Server also in python (A micro-framework or a basic server would do. Flask or werkzeug is recommended). The scrapper will be a python module in the server. (No more process spawns for API calls. Response would be quicker too)
• Dockerfile needs to be adjust accordingly

One goal of Query Server should provide search results to the Yacy Grid server. The results can be used as way to verify results or add an additional layer of search results.

Please set up Yacy Grid (https://github.com/yacy/yacy_grid_mcp) and check possibilities to run Query Server as a component.

Please choose an open-source license for the project.

We have added a starter file for travis. Please help to make travis work and add other tests.

Currently search results are limited to 10 results. Please increase the output to 100 results per search.

There currently doesn't seem to be a travis yml file for this repository, hence the pull requests run against some ruby tests (?). Lets add proper tests.

Referring to Issue : #59

I suggest If there could be a drop-down for the number of results.
As it could be user's choice to get a particular number of data as results.

Thanks!

when an active format button is clicked, it makes it inactive and sets format=None in the request parameters. this is leading to an error 400 error.
before pressing the button

after pressing the button

Currently the teaser text is being shown only for BING on query-server, but it isn't being shown for other search engines. Please improve the scraper to provide teaser text for Google and other search engines as well.

In query_list.txt same query strings are stored multiple times. For example, searching the following :
"fossasia", "openspource","opensource is fun" one after the other stores the following in query_list.txt :

fossasia

fossasia
opensource

opensource
fossasia
opensource is fun

(I deleted the existing contents of query_list.txt before executing the above)

I think this is happening because the entire query list is being appended to the query_list.txt file each time we make a new search

I am not sure whether it is bug or it has been done on purpose.
If its a bug I would love to send a PR :)

Button for uploading to docker cloud

No XML output yet, but it should.

@enigmaeth

Let's move all the app code into an src directory.
Let the root have all the meta files such as .travis.yml, package.json, requirements.txt etc

• Incorrect search engine - return 404 - Not Found
• Zero responses - return 404 - Not Found
• if format is something other than xml or json - return 400 - Bad Request
• if query is empty or missing - return 400 - Bad Request

Along with the proper HTTP Status code header, the body should also contain the error message in either json or xml (depending on what was asked by the user)

fossasia/badgeyay#99
@niccokunzmann

Yaydoc is FOSSASIA's own automatic documentation generation and deployment project. At the crux of it, Yaydoc generates a documentation website using the markdown present in a project's repository and keeps the website in sync with the changes made in project's documentation.

The project has been under development since the start of GSoC 2017 and we believe that it can be brought into production. For that, we'd like to start with the Query-server project. The project requires a .yaydoc.yml configuration file at the project's root directory. Adding necessary information in the configuration file and registering the project at https://yaydoc.herokuapp.com/ will be enough to generate the documentation website.

As per the discussion on gitter channel by @mariobehling different scrapers for each domain, similar to the loklak server be implemented.

As shown in image even if the value of num is set to 10 we are getting 30 results.
URL: http://0.0.0.0:7001/api/v1/search/duckduckgo?query=fossasia&format=json&num=10
Issue with Duckduckgo search engine only. For rest it's working fine.

If its alright, I would like to add bing as an option for the search engines

The server is a command line tool currently. Add a GUI so that when deploying this server as a service, we can have a landing/home page similar to the one at https://loklak.org/ or http://api.asksusi.com/ .

I have mentioned this problem in searss in fossasia/searss#13 and also have done a PR with its fix fossasia/searss#14

If the PR there is accepted, if allowed I would also like to update the rss-generator.py here.

Given compatibility issues, all scrapers and the app in general should be ported to Python3.

• Update all functions and modules to Python 3.
• Adhere to PEP 8 styling standards.

to be done after #45

README.md currently lists the project dependencies as:
Dependencies

• Python 2.x or Python 3.x
• Node.js
• Pip
• Flask
• BeautifulSoup4

But requirements.txt currently list many more dependencies.

This repo only contains two Python scripts and a quick read of them says that they use:

• server.py: dicttoxml, flask, and pymongo
• scraper.py: requests, BeautifulSoup4

Could the following dependencies be safely removed from requirements.txt?

coverage>=4.3.4  # could be pip installed in .travis.yml but not shipped in production
coveralls>=1.1  # could be pip installed in .travis.yml but not shipped in production
feedgen>=0.5.1
futures>=3.0.5
html5lib>=0.9999999
Jinja2>=2.9.5
librabbitmq>=1.5.1
mechanize>=0.2.5
pytest>=3.0.6  # there are no tests in the repo
pytest-cov>=2.4.0  # no pytest means no pytest-cov?
webencodings>=0.5

Current Behaviour:
As of now, there is no/less responsiveness as in mobile mode searchbox disappears, and other buttons behaves abnormally.

Expected Behaviour:
Website should adapt to various screen sizes

I am interseted in working on this issue

There are several reasons to support duckduckgo instead of google for your query server:

• As of now, there is no python 3 version of Mechanize library.
• Dependency of mechanize library is removed if we try to scrape the duckduckgo search results.
• Your IP is more prone to getting blocked by google as compared to duckduckgo if you exceed the no. of permissible requests.

So, should I make a pull request with a modified version of query search using duckduckgo?

Need of migration:

• New 5 tier grid system for smaller screens
  Bootstrap has a sophisticated responsive grid system that allows developers to target devices with different viewports. Bootstrap 3 currently has 4 grid classes for columns, .col-xs-XX for mobile phones, .col-sm-XX for tablets, .col-md-XX for desktops, and .col-lg-XX for larger desktops. Bootstrap 4 will enhance the grid system with a fifth one that will facilitate developers to target smaller devices under 480px viewport width.

• Relative CSS units
  Instead of pixels, the new major release will use REMs and EMs that make it possible to implement responsive typography on Bootstrap sites. This will also increase readability, and make sites more accessible for disabled users.

• Bootstrap cards
  Introduced new UI components called cards. Cards will replace the former wells, thumbnails and panels, and will provide users with a more streamlined workflow.

• Flexbox support
  Bootstrap 4 makes it possible to take leverage of CSS3’s Flexbox Layout and makes use of the float and display CSS properties to implement a fluid layout.

• All the JavaScript plugins have been rewritten to support ES6. This means you have better plugin support in Bootstrap 4 and can take advantage of the new ES6 features

Benefit: Overall UI improvements + Responsiveness

I want to work on this issue

@niranjan94 I am currently working on creating an api for query server. Can you please specify the parameters to be passed to query server by external user to get the response.

The readme contains html5lib and mechanize as dependencies. Now that we're using requests, we don't require html5lib and mechanize, hence the readme should be revamped.

@mariobehling May I change the readme and also add workflow gifs,etc just like in searss?

@enigmaeth @mariobehling
this is in regards to issue fossasia/open-event-server#2675
so what all are left to get the query server hosted?
If it is deployable. can I work on deploying the query-server

Right now, for both json and xml the return content type is text/html. Ensure proper content type and no-cache headers are set.

The version of librabbitmq that this project is using contains at least one Python 3 syntax error.

See: celery/librabbitmq#99

https://travis-ci.org/fossasia/query-server/jobs/275757559#L479

    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-eiupkwho/librabbitmq/setup.py", line 191
        except Exception, exc:
                        ^
    SyntaxError: invalid syntax

Yahoo search is not working.

More info here: #40 (comment)

The "things" being searched are not added to the database..........

the condition inside this if is wrong.

in the above screen shot the databases are not actually made since the mongoDB doesn't create a database until and unless something is inserted to it...........

i would like to update the code so that it starts adding the searched queries to database

Current Behaviour:
All buttons looks same

Expected Behaviour:
Buttons should have logo of the search engine

I am interested in working on this issue

I used this query: https://query-server.herokuapp.com/api/v1/search/google?query=anshul&format=json&num=10

And this is what I got :

Please set up the query server on Heroku from the master branch and add the info to the Readme.md.

Some search results have a cryptic addition at the end, e.g. as follows
<item> <title>FOSSASIA Summit 2017 | Asia's Open Technology Event about ...</title> <link>http://2017.fossasia.org//RK=1/RS=KKHGnF8gw2hmJFCYS8.5sUEeQ6o-</link> </item> <item>
Please find the reason for this and take it out.

This will be an enhancement to the existing project

README.md shows that :

Dependencies

• Python 2.x or Python 3.x
• Node.js
• Pip
• Flask
• BeautifulSoup4

But dependencies using pip could not be installed for python3.x .

Quoting the error message :

File "/tmp/pip-build-s4ac1_v3/librabbitmq/setup.py", line 191
except Exception, exc:

This is because librabbitmq is not supported in Python3.x version.
Please change the README.md .

This script raises a NameError when run in Python 3.

Add a tab to show 10 results that are common to all the search engines for the stated query.

The page becomes unusable in smaller screen sizes

The README file contains instructions pertaining to the old command line based usage. Update it according to the current state.

• update readme.md with the latest parameters for the parameters, i.e. including the number of results to be searched for.
• Handling TypeError for the same , whenever a 500 request is sent from the server.

I would like to work on this issue.
Thanks :D