I couldn't find elasticsearch in the repository.
If this software isn't used elasticsearch, it should be removed from tags for avoiding misleading.

What do you think?
Thanks.

I am facing problems running Docker in my machine, like

• once the docker is started, it never stops, even after deleting those corresponding images
• could not able to start gRPC server

It would be better if you could give the poem example without docker and gRPC
like the example you gave for image search (in a simple notebook manner).

Problem and Question

Hi, I have take a look the poem project. And want to use another data for indexing. How to use labeled csv to do supervised learning with text data?

my data sample.tsv with thai text:

intent	question
ClassA	FAQ 1? ចូលទៅប្រើប្រាស់កម្មវិធីនេះ?
ClassA	Another FAQ Question similar to FAQ 1?
ClassB	TestQuestion with thai text ចូលទៅប្រើប្រាស់កម្មវិធីនេះ?
ClassB	Another data sample

What I have trial

1. I try to pass Pandas Series and it raise GRPC Error,
2. I try to pass tuple with (intent, question) and raise GRPC error
3. I try to use the question only to index it and convert the str into bytes. This is successfully build without GRPC error, but it raise

W:EncoderService:[enc:emb: 42]:document (doc_id=20) contains no chunks!
W:IndexerService:[ind:_ha: 57]:document (doc_id=10) contains no chunks!
W:EncoderService:[enc:emb: 42]:document (doc_id=22) contains no chunks!
W:IndexerService:[ind:_ha: 57]:document (doc_id=12) contains no chunks!
W:EncoderService:[enc:emb: 42]:document (doc_id=24) contains no chunks!
W:IndexerService:[ind:_ha: 57]:document (doc_id=16) contains no chunks!
E:EncoderService:[enc:emb: 67]:can't convert CUDA tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.
W:EncoderService:[enc:emb: 68]:encoder service throws an exception, the sequel pipeline may not work properly

Question

How to train the labeled csv data?

hi, all

have you think about to using some static language like c++ or golang to programming the core modules for this project?

i have go throughed this project and just do though it is not effective to build such large project by using some language like python. even though it is easily for demostration on purposes but not good eough for mature and high-scaleble architecture for search engine. so from my perspective .the project may be should devide to two pieces:

for ML training modules : it is good to be programming by python like it is Depth calculation model which maybe updating dynamicly

for service modules: which should be running effectively and performance-sensitive with highly scalible. i would like suggest to programming by using C++ or golang .

here is some project for example like:
https://github.com/alibaba/euler

While installing gnes using pip install gnes:

I get the following file missing error with the latest release. What could be the posiible reasons?

building 'gnes.indexer.chunk.bindexer.cython' extension
creating build/temp.linux-x86_64-3.5
creating build/temp.linux-x86_64-3.5/gnes
creating build/temp.linux-x86_64-3.5/gnes/indexer
creating build/temp.linux-x86_64-3.5/gnes/indexer/chunk
creating build/temp.linux-x86_64-3.5/gnes/indexer/chunk/bindexer
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.5m -c gnes/indexer/chunk/bindexer/bindexer.c -o build/temp.linux-x86_64-3.5/gnes/indexer/chunk/bindexer/bindexer.o -O3 -g0
gnes/indexer/chunk/bindexer/bindexer.c:4:20: fatal error: Python.h: No such file or directory
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

----------------------------------------

Command "/usr/bin/python3 -u -c "import setuptools, tokenize;file='/tmp/pip-install-75yi68kt/gnes/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-record-2wvrek1d/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-install-75yi68kt/gnes/

According to the marketing page, there are storage integrations with tencent cloud, but they don't appear to exist in the source code.

1. How do you plan on persisting indexes between pod restarts?
2. Can there be more than one index per cluster?
3. Can indexes be written and read from in an online manner?

Hi Community,

Sept. 13-15 is Mid-Autumn festival for Chinese, there won't be any update on GNES from the core-dev team.

Next week on Sept 17 and Sept. 20, GNES will start the roadshow in Shanghai and Vienna. I will upload slides, photos and demos to https://github.com/gnes-ai/roadshow-2019/

Stay tuned!

When I run 'python index.py', the error occurs:

the yaml file of the encoder:

index.py:

I am trying to run GNES locally on my Linux machine. However, when I run bash run.sh, I received the following errors.

E:EncoderService:[bas:run:305]:could not determine a constructor for the tag '!GPT2Encoder'
in "gpt2.yml", line 3, column 5

Namely, there is something wrong with gpt2.yml. I replaced the value of model_path with a real path to a downloaded GPT2 model, but it still did not work.

I installed gnes and also was running the docker service. Is there anything else we need to do to get it working? Thanks!

Dear Community,

Oct. 1-7 is national day for China, there won't be any update on GNES from the core-dev team.

Stay tuned!

I really like the ambition and the direction this project set out. However since the last commit is from october, I am a bit concerned that the project died :(. Can you give any comments on that?

🙇Hello there, I'd like to thank you for your interests in GNES, greatly appreciated!

After reading through the README, you may realize that GNES has an ambitious goal and the workload to enable any-to-any semantic search is quite something. Right now we are rapidly iterating GNES in terms of its usability, stability. This includes:

• adding new APIs;
• refactoring old APIs;
• restructuring the project structure;
• writing tutorial;
• building a healthy GNES eco-system;
• making demos

⏰A new release is scheduled every Friday's evening (GMT+8), the release procedure can be found here.

Please be patient if you encounter some issues, and please understand that we may not respond in time regarding your issues. Thanks for your understanding. 🙏

Meanwhile, if you'd like to understand the design principles behind GNES, feel free to read this blog post: Generic Neural Elastic Search: From bert-as-service and Go Way Beyond

For other questions, please contact our team via [email protected]

Amazing project! Been using it for a week now and I love the architecture of the entire system.

If it's not too drastic of a change, it would be nice to have the HTTP client optionally added to the flow so that it is possible to send json requests through the client to the frontend. I would like to be able to deploy the client image in the swarm yaml. This would have the benefit of fully containerizing gnes and minimizing dependencies for the end user. Also, I've seen a few people have problems with leveraging the bytes generator directly.

Could potentially add to the service map then to the node graph?

I'm running the demo-poems-ir and it seems to be stuck at

I:MyClient:[bas:__i:124]:setting up grpc insecure channel...
I:MyClient:[bas:__i:133]:waiting channel to be ready...

Probably something with my setup. Anyone else seen this.
Thanks

Explanation

Hi. I has been implement the demo-poems-ir. And it is working properly. Right now I want to use the sample for my custom class.
here the class:

class FaqClient(CLIClient):

    @property
    def bytes_generator(self):
        num_rows = 0

        for rr in csv.DictReader(self.args.txt_file, delimiter='\t'):
            yield json.dumps(rr['answer']).encode()
            num_rows += 1
            if num_rows > self.args.num_poems:
                return

    def query_callback(self, req, resp):
        print(colored(req.search.query, 'green'))
        for k in resp.search.topk_results:
            print(colored(k.doc.doc_id, 'magenta'))
            print(colored(k.doc.raw_text, 'yellow'))
            print(colored(k.score.value, 'blue'))
            pprint.pprint(json.loads(k.score.explained))
            input('press any key to continue...')

        if self.args.prompt:
            input('on to next query, press any key to continue...')

Problem

When I run make client_query or make client_index it will stuck in:

I:FaqClient:[bas:__i:136]:create new stub...

Here the screenshot

Questions

What's exactly happened when the Client info create new stub...? I read the source it only init GnesRPCStub with the channel. It stuck there and can not passing the

Kindly advise

Hi. I found an issue when use Flow to indexing text as type str it will encounter horrible message:

E:CLIClient:[cli:sta: 53]:<_Rendezvous of RPC that terminated with:
	status = StatusCode.UNKNOWN
	details = "Exception iterating requests!"
	debug_error_string = "None"
>

This caused by passing the string not bytes.

so to solve this you need to convert/encode your str type to bytes.
I will make PR for this soon

I have 3 questions:

1. How will I combine Text and Image instead of text to text, image to image...? Example: Consider there is a product which has images as well as textual information, how will I use GNES to get the most similar matching product from the index?

2. How scalable is this solution? Can I index 100 million images or text?

3. What to do if I want to save the indexes to database and load them during run time?

really like this projects. how ever sometimes the data is too large can we want to do sharding and merge the result.

Team: Great work!

I am able to run Docker Swarm Stack with all the mentioned services running successfully. However, I am not familiar with gPRC, how do I make use of this frontend to pass the documents for ingestion and further query ??

I am familiar with Python, fyi. Also, I see that you have HTTP end-to-end in works, if there is a beta ready, I would love to try and provide early feedback for you!!

Thank you!

Hi.
Let's think about we have 100.000 sentence data or images.
And want to remove 5 sentences or images from searching scope.
How it is possible? because remove all of them and recompute need lot of time and resources

Hi,
This is a VERY GREAT looking project. Thank you for your effort.

But from the documentation is not clear what kind of uses are possible with this project.
For example, in a text search, I would imagine that gnes would enable me to find similar documents. Is that right?

And how it compared with Vespa.ai (https://github.com/vespa-engine) ?

Some examples of what is possible with Gnes would be nice!

HI, i really appreciate and see potential in this gnes project. I have two questions:

1. I have been able to successfully run the flowers demo, along with other images. However i am unsure how to run text objects through. can this be further elaborated.

2. Also i have tried the semantics poem demo but when i run make client_index d=10 it seems to get output " GNES client ready" but gets stuck at the first indexing with 0.0 batch/s speed.

I would greatly appreciate any insight into the above. Thanks in advance