fullscale / elastic.js Goto Github PK

There is currently no way to “update” a mapping in ES … we have to simply rebuild the index. Scan/Scroll allows us to quickly perform a query and iterate over the results. So using scan/scroll over an old index and inserting into a new index is a neat feature.

Here's a link to an example of a Perl client for ES that does this:
https://github.com/clintongormley/ElasticSearch.pm#examples

It would be pretty cool to have something like this in elastic.js for node ... what do you think?

Using curl I can get statistics about the heap e.g.

$ curl -s -XGET 'http://localhost:9200/_cluster/nodes/stats?jvm=true&pretty=true' | grep heap
          "heap_used" : "4.4gb",
          "heap_used_in_bytes" : 4777105152,
          "heap_committed" : "6.9gb",
          "heap_committed_in_bytes" : 7446462464,
          "non_heap_used" : "41.7mb",
          "non_heap_used_in_bytes" : 43812696,
          "non_heap_committed" : "63mb",
          "non_heap_committed_in_bytes" : 66142208,

I'd like to know how I can get the same stats using Elastic.js. I've tried using NodeStats module and setting .jvm(true) and also .all(true) however the result does not contain any stats about the heap.

Hi guys,

I appreciate all the hard work that you put into elastic.js, but I'm a bit lost on how this could ever be used for a public facing application. I have ES running on an Amazon EC2 instance and it's currently sitting behind a nginx proxy. From what I can tell, if I run a search (using request.doSearch()) on ES using Elastic.js, it tries to POST the search request to ES. I'm curious why a post is being executed in order to retrieve data from the ES instance as opposed to a GET. If I was to use your service in a public application, I would have to allow anyone using it to be able to POST to my server, leaving me open to potential security issues.

Can anyone explain this thought process to me? I've been spending the past day or two trying to wrap my head around all this, and getting my proxy to the place where I want it so that I can securely interact with ES over AJAX.

It also might be nice to include some examples in the future on how to interact with a public instance using elastic.js

Thanks!

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations.html

A very useful feature and sometime soon aggs will supersede facets.

Using the _search endpoint with query param search_type=count allows facets to be used with count queries, and is just as performant as the _count endpoint. Currently, elastic.js doesn't provide an option to utilize this execution path.

To enable this, the doCount method should change:

var queryData = JSON.stringify(query.query);

to:

var queryData = JSON.stringify(query);

...and should set:

params.search_type="count"

(and other query types should clear that params entry)

I added this to ejs.TermsFacet and it seems to work:

    shard_size: function (shardSize) {
        if (shardSize == null) {
            return facet[name].terms.shard_size;
        }

        facet[name].terms.shard_size = shardSize;
        return this;
    },

When I install the module with npm (from https://registry.npmjs.org/elastic.js/-/elastic.js-1.1.1.tgz) the module is missing FunctionScoreQuery, although the GitHub code contains the code for this and is at version 1.1.1 as well. Could you please double-check the npm deployment process and maybe re-publish the module? Thanks!

Hello,

I tried the term suggester and I always get the "term" option before the text in the suggest request, which generates an error in Elastic Search.

e.g. For

ejs.TermSuggester("my_suggestion")
            .text("geographoc")
            .field("title");

The suggest request will be:

"suggest":{"my_suggestion":{"term":{"field":"title"}, "text":"geographoc"}} 

instead of:

"suggest":{"my_suggestion":{"text":"geographoc", "term":{"field":"title"}}} 

Thanks,
Iulia

First of all thanks for this great library!

I'm wiling to use it with an AngularJS APP + Elastiseach as SaaS, but, I'm a little worried about making my API key public.

How would you implement it in a "production environment"? Should I create a Proxy using Node.JS (my app server) or something like that? I know I can use the library on server-side but I wanted to let queries be made by the browser.

Sorry if this is a silly question, but I could find no answer.

(The solution may be to have a "public" and a "private" key, but my Elasticsearch provider - SearchBox - provides only one "master key".)

I haven't found any mention in the guide or documentation of how to connect to a cluster. In fact finding how to connect is generally lacking. It should be one of the first mentioned steps in your guide.

Maybe I'm missing it, but this is one of the few calls I couldn't find. I have a number of instances where this saves me some time, not sure how difficult it would be to implement.

http://www.elasticsearch.org/guide/reference/api/multi-search/

Not sure if it will be useful, but I ran into some issues when implementing it myself via Angular.

http://stackoverflow.com/questions/15481445/post-new-line-in-json-for-msearch-in-elasticsearch

This is the line throwing error :(

nc = require('elastic/elastic-node-client');

The current version of elastic.js hosted on bower.io is not the most recent. It does not have any of the aggregation functionality, or any of the .toJSON() methods.

Hi,

I am using ES for more than a year now. This library draws my interest and looks very promising. I have never used node.js but I do want to use this library for its jquery client.

Can I use it without node.js?
I have tried running examples which you have given but its throwing "Request header field Content-Type is not allowed by Access-Control-Allow-Headers." error on chrome.

A pure jquery based example will be really helpful.

Regards
Jagdeep

hi .........
i want to know that is there is any thing possible in elastic search that while searching in an index on different fields we can give preference or u can say weightage to a particular field while searching...................
if yes then plz tell me how to implement it as i was using elastic search on windows

Hello,
it would be nice to support bower so that we can handle the depenancy.

The Apache Lucene doco lists the following special characters that need to be escaped with \ in query strings:

+ - && || ! ( ) { } [ ] ^ " ~ * ? : \

I've also found that / needs escaping.

The problem I have is knowing when to escape. For example, an unescaped ~ on it's own produces an 'internal server error' from ES but when appended to a word e.g. blah~ I get a result.

Is this something that Elastic.js can handle for me, or do I need to create an ellaborate regex?

1. A lot of Elasticsearch as a service providers now use es-jetty or nginx to provide basic authN ... so it would be pretty cool if we could control specify username password for basic authN in the node.js for elastic.js client.
   http://stackoverflow.com/questions/16137919/how-to-specify-credentials-username-password-for-elasticjs-client

2. I don't see a way to specify https either, so I'm guessing that the ability to specify that will have to be exposed as well?

I can't figure out how to make a has_child filter with a filter as described in ES documentation (http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-has-child-filter.html).

The HasChildFilter here is expecting a query so the following declaration throws an exception. Is there another way to do it ?

ejs.HasChildFilter(
    ejs.BoolFilter().must([
        ejs.TermsFilter('brand', 'b'), 
        ejs.TermsFilter('model', 'm')
    ])
)

Have the authors of elastic.js ever run into the scenario where a 411 status code is returned while posting data with this client?

<html>
<head><title>411 Length Required</title></head>
<body bgcolor="white">
<center><h1>411 Length Required</h1></center>
<hr><center>nginx/1.1.19</center>
</body>
</html>

Its understandable that the client isn't ready for this and the JSON.parse() method in elastic-node-client.js:208:25 gets confused and the system throws a SyntaxError:

undefined:1
<html>
^
SyntaxError: Unexpected token <
    at Object.parse (native)
    at IncomingMessage.ejs.NodeClient.put (./node_modules/elastic.js/elastic-node-client.js:208:25)

But I'm more curious as to why the client might run into such a situation in the 1st place.

First, thanks for a very useful module :) It makes building queries for ES a lot more structured!

However, the current version in the npm registry, v1.1.1, seems to be quite different from the version in the Github repository (also v1.1.1), and it doesn't seem to be very usable (I can't get any of the examples documented in the README.md to work, for instance, whereas they work fine for the GH version).

Perhaps it's worth documenting this in the README, and to explain how to install the version from GH using the npm CLI tool:

$ npm install fullscale/elastic.js

Ideally, the latest version would be posted to the npm repository (npm can tag a publish as being a beta, so it won't be pushed as latest if it isn't yet ready to publish as a proper update).

The node client only supports http. Add support for https.

Hi There! Hoping to get some expert eyes to help me work through this config issue.

I've successfully run elasic.js vs a local ElasticSearch server. No problems or hangups to speak of--wacky fun! However, when I try to connect to Bonsai on Heroku, I start to see a barrage of these errors:

failed to index user/bruce: Error: getaddrinfo EFAULT
failed to index user/chan: Error: getaddrinfo EFAULT
failed to index user/chuck: Error: getaddrinfo EFAULT

I'm fairly well convinced this has something to do with the URL structure, which contains auth creds and does not seem to have a port number. It looks very close to this:

https://d38xxxxkb:[email protected]

I can CURL against that URL with great success:

> curl -X POST https://d38xxxxkb:[email protected]
{"ok":true,"acknowledged":true}

Additionally, however I set up the port number in elastic.js, I get the exact same error:

var URL = 'https://d38xxxxkb:[email protected]';
ejs.client = nc.NodeClient(URL,  9200);  // same error
ejs.client = nc.NodeClient(URL,  80);     // same error
ejs.client = nc.NodeClient(URL,  null);   // same error
ejs.client = nc.NodeClient(URL,  '');       // same error
ejs.client = nc.NodeClient(URL,  false); // same error
ejs.client = nc.NodeClient(URL,  'foo');  // same error

I see a similar error running this from the jQuery client (it removes the auth creds and reports a 404):

ejs.client = ejs.jQueryClient('https://d47xxxxx:[email protected]');
ejs.Document('index', 'user', '1').source({
     user: 'mrweber',
     message: 'Elastic.js - a Javascript implementation of the ElasticSearch Query DSL and Core API'})
     .doIndex(function() {});

What horrible sin of noobness have I fallen victim to?

Cheers,

Hi @mattweber

Do you plan to implement bulk apis? I'm preparing a guide for Searchly and we would like to encourage people to use bulk to reduce indexing cost etc.

Ferhat

right now, npm install elastic.js doesn't work because this hasn't been published yet.

please publish it

Hi,

Is there a way to extend elasticjs lib dynamically to expose custom facets.
something like:

ejs.CustomHistogramFacet

The problem I am facing is that util.js functions are not exposed, while they are needed within the custom facet.

Thanks

ExtJS is a very popular and widely used library it would be nice to have client (driver) for it. ExtJS has all async communications nicely encapsulated - will be easy to work with

http://docs.sencha.com/ext-js/4-1/#!/api

In the new versions (1.3) of elasticsearch there is a new aggregation's type. It could be great to add this feature to elastic.js

i need show the most highlighted field, have this option?
or one way to make this?

The Request object has a method indices about which the doco says:

Allows you to set the specified indices on this request object. This is the set of indices that will be used when the search is executed.

Attempting to use this results in a POST URL with each index separated by a comma e.g.
http://<host>:<port>/<index1>,<index2>/_search

for me, that results in a HTTP 500 Internal Server Error. Is this a bug, or am I just misunderstanding how this should be used?

I have found that the current API's does not provide multiple field aggregation. Basically i wanted to achieve this:

  "aggs": {
    "Daily": {"date_histogram": {"field": "log_date","interval": "1d","format": "E"},
      "aggs": {
        "Hourly": {"date_histogram": {"field": "log_date","interval": "1h"},
          "aggs": {
            "HourlyExamCount": {"cardinality": {"field": "ExamRowKey"}},
            "HourlyExamDuration": {"sum": {"field": "ExamDuration"}},
            "HourlyCOTDuration": {"sum": {"field": "COTDuration"}},
            "HourlyPreparationDuration": {"sum": {"field": "PreparationDuration"}},
            "HourlyISDDuration": {"sum": {"field": "ISDDuration"}} 

          }
        },
        "DailyExamCount": {"cardinality": {"field": "ExamRowKey"}},
        "DailyExamDuration": {"sum": {"field": "ExamDuration"}},
        "DailyCOTDuration": {"sum": {"field": "COTDuration"}},
        "DailyPreparationDuration": {"sum": {"field": "PreparationDuration"}},
        "DailyISDDuration": {"sum": {"field": "ISDDuration"}} 
      }
    }
  }

Let me know if it supports how to achieve it.

Hi,
Thanks for elastic.js. I have a question/request for a useful feature that could make elastic.js a better tool. I want to display my json results in a table using https://datatables.net/ that adds advanced interaction controls to any HTML table. Some help with how to accomplish this with elastic.js and datatables data sources and server side processing is greatly appreciated. Thanks

Need to add support for rescoring, which can be very helpful for in my case, wildcard queries to limit down by rescoring with a phrase query, to bubble up better matches.

http://www.elasticsearch.org/guide/reference/api/search/rescore/

It looks like the Function Score Query is supposed to replace a handful of custom scoring queries in Elasticsearch.

It would be nice with support for that in elastic.js.

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-function-score-query.html

Hi,

If the ES server is not responding, then the following exception appears in the javscript console log : "Failed to load resource"
Is there a way to get this info from ejs.Request() ?

Regards,

Charles

The minified version currently in the dist folder does not match the full source in the same folder. It seems to be generated from an earlier version. For example, the source function for the Request object is not available in the minified version but is in elastic.js.

It appears that there are number of breaking changes since you wrote your guides. I've been unable to find complete examples for your new API.

How to specify multiple ip address for ejs.client = nc.NodeClient object
example:
ejs.client = nc.NodeClient("server1, server2,server3")

So for my version, I have CORS enabled and for communicating with my API, I need to set some custom headers for authentication, so I do so with the common headers in Angular, but that conflicts with the calls to Elastic as it is not configured to accept those headers, so I need to remove them. Locally when I call it, I just pass a transformRequest, but with elastic.js, there's no way to do that. It appears to be a minor change to allow that, here is my proposed change in elastic-angular-client.js, let me know what you think.

https://gist.github.com/dariusriggins/5255188

There seems to be no correspondent for the bulk API call.

I need this in a node client so I can populate a test ES install for benchmarking. I've successfuly used Document.doIndex, but it's too slow to be sending documents one by one. I notice this is present in https://github.com/phillro/node-elasticsearch-client so I'll switch to that for now.

So on a project I just did bower install elastic.js, and got most of the repo copied, with the exception of the dist folder, which in this case is the only one I actually care about. This is on bower 0.9.2, and it got v 1.1.1 of elastic.js.

In the elastic.js, you serialize the query using JSON.stringify, but Angular does the same thing when passing the data through it, so for example when I try to do a simple query like this.

The query gets sent over as {"size": 15}, which elastic doesn't understand. I can fix it by adding JSON.parse on the elastic-engular-client file, but the double serialization seems unnecessary.

Client implementations, excluding node, allow error callbacks but that is no exposed though index and search requests. Allow a user to pass in an error handling callback in methods such as doSearch and doIndex and update the node client.

The documentation for the library states that, by default, timestamps are created for each new document indexed. In my simple tests, this doesn't seem to be the case. The documents mention that you should add a mapping, I don't see how to do this in ElasticJS

Hi,

It would be useful if you had some information about compatibility with angular versions.

I just spent a while trying to work out why my code wouldn't work, the response of a query was missing the data in the response object. It seemed that I had too new a version of Angular as I was using the latest 1.2.3. I downgraded to 1.0.8 and it worked fine.

Just thought that might help some others.

Loving your work though :)

Chris

Please add support for Sourcing Filtering introduced recently in ES 1.0.0.

Thanks.

Both the Angular and Jquery examples link to localhost.

When using the elastic.js angular client with ng-boilerplate grunt reports this warning

I solved it by simply adding /*jslint node: true */ before 'use strict';

I used the NumericRange Filter for date fields. In the past it worked fine, but with ElasticSearch 1.0/1.1 it behaves wrong. I assume from/to must be replaced with "gte" / "lte" or NumericRangeFilter must be replaced with RangeFilter.

http://www.elasticsearch.org/guide/en/elasticsearch/reference/0.90/query-dsl-numeric-range-filter.html

Hey guys,

i just want to ask a short, but important question.
Is there any chance to get an extendend completion suggester functionality in the sence of elastic search at this great api?

For better understanding what i am talking about follow the link i supposed below:
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-suggesters-completion.html

I look forward to get an answer on this issue. ;)

Have grate day,
Mr. Love