CXX hyperdex_binary_test-binary-test.o
binary-test.cc: In function ‘int main(int, char*)’:
binary-test.cc:141:35: error: call of overloaded ‘pack64le(uint32_t&, uint8_t [8])’ is ambiguous
binary-test.cc:141:35: note: candidates are:
/opt/hyperdex/include/e/endian.h:103:1: note: uint8_t e::pack64le(uint64_t, uint8_t_)
/opt/hyperdex/include/e/endian.h:286:1: note: uint8_t_ e::pack64le(int64_t, uint8_t_)
make[1]: *_* [hyperdex_binary_test-binary-test.o] Error 1
make[1]: Leaving directory `/home/nick/proj/HyperDex'
make: *** [all] Error 2

While trying to compile Libe I get the following error:

Jacobs-MacBook-Air-2:libe-0.2.5 jacob$ make
make  all-recursive
Making all in benchmarks
  CXX    lockfree_hash_map.o
lockfree_hash_map.cc: In function ‘void worker_thread(e::lockfree_hash_map<long long unsigned int, long long unsigned int, id>*)’:
lockfree_hash_map.cc:143: error: ‘drand48_data’ was not declared in this scope
lockfree_hash_map.cc:143: error: expected `;' before ‘buf’
lockfree_hash_map.cc:147: error: invalid conversion from ‘_opaque_pthread_t*’ to ‘short unsigned int’
lockfree_hash_map.cc:148: error: ‘buf’ was not declared in this scope
lockfree_hash_map.cc:148: error: ‘seed48_r’ was not declared in this scope
lockfree_hash_map.cc:154: error: ‘lrand48_r’ was not declared in this scope
lockfree_hash_map.cc:163: error: ISO C++ forbids comparison between pointer and integer
lockfree_hash_map.cc:175: error: invalid conversion from ‘_opaque_pthread_t*’ to ‘long long unsigned int’
lockfree_hash_map.cc:175: error:   initializing argument 2 of ‘bool e::lockfree_hash_map<K, V, H>::insert(const K&, const V&) [with K = long long unsigned int, V = long long unsigned int, uint64_t (* H)(const K&) = id]’
make[2]: *** [lockfree_hash_map.o] Error 1
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2
Jacobs-MacBook-Air-2:libe-0.2.5 jacob$ 

I have no idea to make of it, but any help would be appreciated (if it's a problem with my system) or hopefully the bug can be found (if it's a problem with the codebase)

thanks!

foobar baz

Traceback (most recent call last):
File "scripts/client.py", line 78, in
t = hc.benchmarkGet()
File "scripts/client.py", line 59, in benchmarkGet
self.get(self.data[x]['name'])
File "scripts/client.py", line 26, in get
d = self.c.async_get(self.s, f_name)
File "hyperclient.pyx", line 955, in hyperclient.Client.async_get (hyperclient/python/hyperclient.c:16262)
File "hyperclient.pyx", line 509, in hyperclient.DeferredGet.cinit (hyperclient/python/hyperclient.c:7265)
hyperclient.HyperClientException: Unknown Error (file a bug)

When vals is a list of numbers, the following is possible:

>>> c.list_rpush('keyvalues','1234',{'vals': 54.2355})
>>> False

What is expected is:

>>> c.list_rpush('keyvalues','1234',{'vals': 54.2355})
Exception:  TypeError

54.2355 is not an integer, so a TypeError should be thrown.

The latest e ( commit 614ad1452d ) doesn't seem to have the file ... include/e/endian.h that the latest Hyperdex ( commit 29b8a02) tries to include.

CXX hyperclient/libhyperclient_la-hyperclient.lo
hyperclient/hyperclient.cc:37:22: fatal error: e/endian.h: No such file or directory
compilation terminated.
make[1]: *** [hyperclient/libhyperclient_la-hyperclient.lo] Error 1
make[1]: Leaving directory `/home/nick/proj/HyperDex'
make: *** [all] Error 2

Hi!
I tried to build hyperdex from source, but didn't find vis.h. On ubuntu, you need to install libbsd-dev to have this file. Imo, you should include it as a configure dependency.

I am testing a sorted_search() that requests the results be sorted by an integer attribute with limit=1 and maximize=true.

The first call to loop() returns with status SUCCESS, and I'm able to read the attributes for the expected record.

The next call to loop(), which I expect to return status SEARCHDONE, instead returns SUCCESS. The attributes remain unchanged, pointing to the first expected record.

A third call to loop() returns -1, with status NONEPENDING.

Test case:

#include <cassert>
#include <iostream>

#include <hyperclient.h>


namespace {

/* space test
 * dimensions name, value (int64)
 * key name auto 1 3 */

const char *SPACE_TEST = "test";
const char *ATTR_NAME = "name";
const char *ATTR_VALUE = "value";

struct test_record {
  std::string name;
  int64_t value;
};

std::ostream& operator<<(std::ostream &stream, const test_record &entry)
{
  return stream << "{ name: \"" << entry.name
      << "\", value: " << entry.value << " }";
}


int insert(hyperclient *client, const test_record &entry)
{
  enum hyperclient_returncode status = HYPERCLIENT_SUCCESS;
  const struct hyperclient_attribute attrs[] = {
    {
      ATTR_VALUE,
      (const char*)&entry.value,
      sizeof(entry.value),
      HYPERDATATYPE_INT64
    }
  };
  const size_t attrs_sz = sizeof(attrs)/sizeof(*attrs);
  const int64_t operation = client->put(SPACE_TEST,
                                        entry.name.c_str(),
                                        entry.name.size(),
                                        attrs, attrs_sz,
                                        &status);
  std::cout << "client.put(" << entry.name << "=" << entry.value <<
      ") returned " << operation << " with status " << status << std::endl;
  if (operation < 0)
    return 0;

  const int64_t loop = client->loop(10, &status);
  std::cout << "client.loop() returned " << loop
      << " with status " << status << std::endl;

  return loop == operation && status == HYPERCLIENT_SUCCESS;
}

int search(hyperclient *client, test_record *result)
{
  enum hyperclient_returncode status = HYPERCLIENT_SUCCESS;
  struct hyperclient_attribute *attrs;
  size_t attrs_sz;
  const int64_t operation = client->sorted_search(SPACE_TEST,
                                                  NULL, 0,
                                                  NULL, 0,
                                                  ATTR_VALUE, 1, true,
                                                  &status,
                                                  &attrs, &attrs_sz);
  std::cout << "client.sorted_search() returned "
      << operation << " with status " << status << std::endl;
  if (operation < 0)
    return 0;

  // see if there are any results
  int64_t loop = client->loop(10, &status);
  std::cout << "client.loop(1) returned " << loop
      << " with status " << status << std::endl;

  if (loop != operation)
    return 0;

  // if there were no results, don't call _destroy_attrs()
  if (status == HYPERCLIENT_SEARCHDONE)
    return 0;

  // read the attributes
  for (size_t i = 0; i < attrs_sz; i++)
  {
    if (strcmp(attrs[i].attr, ATTR_NAME) == 0)
      result->name.assign(attrs[i].value,
                          attrs[i].value + attrs[i].value_sz);
    else if (strcmp(attrs[i].attr, ATTR_VALUE) == 0)
      result->value = *(int64_t*)(attrs[i].value);
  }
  std::cout << "result 1: " << *result << std::endl;

  // loop again to get SEARCHDONE
  loop = client->loop(10, &status);
  std::cout << "client.loop(2) returned " << loop
      << " with status " << status << std::endl;

  if (status == HYPERCLIENT_SUCCESS)
  {
    // expected SEARCHDONE.. read the attributes again
    test_record result2;
    for (size_t i = 0; i < attrs_sz; i++)
    {
      if (strcmp(attrs[i].attr, ATTR_NAME) == 0)
        result2.name.assign(attrs[i].value,
                            attrs[i].value + attrs[i].value_sz);
      else if (strcmp(attrs[i].attr, ATTR_VALUE) == 0)
        result2.value = *(int64_t*)(attrs[i].value);
    }
    std::cout << "result 2: " << result2 << std::endl;

    // call loop again for status
    loop = client->loop(10, &status);
    std::cout << "client.loop(3) returned " << loop
        << " with status " << status << std::endl;
  }

  // free attribute results
  hyperclient_destroy_attrs(attrs, attrs_sz);

  return loop == operation && status == HYPERCLIENT_SEARCHDONE;
}

} // namespace


int main(int argc, char *argv[])
{
  const test_record a = { "a", 0 };
  const test_record b = { "b", 1 };
  const test_record c = { "c", 2 };
  const test_record d = { "d", 3 };
  test_record result;
  hyperclient *client;

  assert(client = hyperclient_create("127.0.0.1", 1234));

  assert(insert(client, a));
  assert(insert(client, b));
  assert(insert(client, c));
  assert(insert(client, d));

  assert(search(client, &result));
  assert(result.name == "d");
  return 0;
}

Produces the output:

client.put(a=0) returned 1 with status HYPERCLIENT_SUCCESS
client.loop() returned 1 with status HYPERCLIENT_SUCCESS
client.put(b=1) returned 2 with status HYPERCLIENT_SUCCESS
client.loop() returned 2 with status HYPERCLIENT_SUCCESS
client.put(c=2) returned 3 with status HYPERCLIENT_SUCCESS
client.loop() returned 3 with status HYPERCLIENT_SUCCESS
client.put(d=3) returned 4 with status HYPERCLIENT_SUCCESS
client.loop() returned 4 with status HYPERCLIENT_SUCCESS
client.sorted_search() returned 5 with status HYPERCLIENT_SUCCESS
client.loop(1) returned 5 with status HYPERCLIENT_SUCCESS
result 1: { name: "d", value: 3 }
client.loop(2) returned 5 with status HYPERCLIENT_SUCCESS
result 2: { name: "d", value: 3 }
client.loop(3) returned -1 with status HYPERCLIENT_NONEPENDING
sorted_search: test_sorted_search.cc:147: int main(int, char**): Assertion `search(client, &result)' failed.

Add methods that remove and return items from a list with list_lpop and list_rpop.

There is currently only minimal documentation about cluster elasticity (or it is not explicitly marked as such.) A good start would be answers to the following:

1. Does hyperdex allow additional data-nodes to be added to a cluster at runtime?
2. How is keyspace migration handled for newly added nodes?
3. How is keyspace migration handled during new/old node failure?
4. How does hyperdex deal with network partitioning and re-join of nodes?

Per tutorial, got the following errors to run "autoreconf" to prep for build.

autoreconf -i -v

autoreconf: Entering directory .' autoreconf: configure.ac: not using Gettext autoreconf: running: aclocal -I m4 ${ACLOCAL_FLAGS} autoreconf: configure.ac: tracing autoreconf: configure.ac: not using Libtool autoreconf: running: /usr/bin/autoconf autoreconf: running: /usr/bin/autoheader autoreconf: running: automake --add-missing --copy --no-force Makefile.am:38: Libtool library used butLIBTOOL' is undefined
Makefile.am:38: The usual way to define LIBTOOL' is to addAC_PROG_LIBTOOL'
Makefile.am:38: to configure.ac' and runaclocal' and autoconf' again. Makefile.am:38: IfAC_PROG_LIBTOOL' is in `configure.ac', make sure
Makefile.am:38: its definition is in aclocal's search path.
autoreconf: automake failed with exit status: 1

Please advise what can be missing.

By the way, build tools autoconf, automake, and libtool have the following versions on debian.

autoconf, 2.67-2, automatic configure script builder
automake, 1:1.11.1-1, A tool for generating GNU Standards-compliant Makefiles
libltdl7, 2.2.6b-2, A system independent dlopen wrapper for GNU libtool

It looks like that on my debian, I have libltdl7 instead of "libtool". Does it have to be exact?

One section in tutorial on wiki does not work. The section is concerning creating a new table. Upon entering nc cmd and subsequent lines. No 'SUCCESS' response is received by nc from HyperDex.

Please advices how to proceed or how to diagnose the problem.

$ autoreconf -i -v;
autoreconf2.50: Entering directory .' autoreconf2.50: configure.ac: not using Gettext autoreconf2.50: running: aclocal -I m4 ${ACLOCAL_FLAGS} autoreconf2.50: configure.ac: tracing autoreconf2.50: running: libtoolize --copy libtoolize: putting auxiliary files in.'.
libtoolize: copying file ./ltmain.sh' libtoolize: putting macros in AC_CONFIG_MACRO_DIR,m4'.
libtoolize: copying file m4/libtool.m4' libtoolize: copying filem4/ltoptions.m4'
libtoolize: copying file m4/ltsugar.m4' libtoolize: copying filem4/ltversion.m4'
libtoolize: copying file `m4/lt~obsolete.m4'
autoreconf2.50: running: /usr/bin/autoconf
configure.ac:31: error: possibly undefined macro: AC_PROG_JAR
If this token and others are legitimate, please use m4_pattern_allow.
See the Autoconf documentation.
configure.ac:32: error: possibly undefined macro: AC_PROG_JAVAC
configure.ac:33: error: possibly undefined macro: AC_JNI_INCLUDE_DIR
configure.ac:151: error: possibly undefined macro: AC_PYTHON_DEVEL
autoreconf2.50: /usr/bin/autoconf failed with exit status: 1

Hi,

When running the configuration script with "./configure --enable-python-bindings PKG_CONFIG_PATH=/usr/local/lib/pkgconfig PYTHON_VERSION=2.7", I get:
"
configure: error:
Could not link test program to Python. Maybe the main Python library has been
installed in some non-standard library path. If so, pass it to configure,
via the LDFLAGS environment variable.
Example: ./configure LDFLAGS="-L/usr/non-standard-path/python/lib"

ERROR!
You probably have to install the development version of the Python package
for your distribution. The exact name of this package varies among them.

"

I'm running Ubuntu 12.04. Could you please let me know how to fix this?

Thanks!

Lin

As hyperdex can now store signed int64_t types, shouldn't range predicate types for upper and lower members also be int64_t instead of uint64_t?

1. Start fresh coord
   ploj@hive:~/github/HyperDex$ hyperdex-coordinator --control-port 6970 --host-port 1234 --logging debug
   INFO:root:Fresh cluster starting.
2. Add space via coord-client - please note no int64 type use (the other bug that would fail config):
   ploj@hive:~/github/HyperDex$ echo -e "space phonebook3\ndimensions username, first, last, phone\nkey username auto 1 3\nsubspace first, last, phone auto 2 3\n" | hyperdex-coordinator-control --host 127.0.0.1 --port 6970 add-space
3. Start daemon:
   ploj@hive:~$ hyperdex-daemon --host 127.0.0.1 --port 1234 --bind-to 127.0.0.2 --data /home/ploj/daemon10
   W0318 17:55:59.907731 31014 datalayer.cc:367] Started data-flush thread.
   W0318 17:55:59.907739 31015 datalayer.cc:367] Started data-flush thread.
   W0318 17:55:59.907915 31016 datalayer.cc:367] Started data-flush thread.
   W0318 17:55:59.907696 31012 datalayer.cc:266] Started optimistic-I/O thread.
   W0318 17:55:59.908464 31013 datalayer.cc:367] Started data-flush thread.
   W0318 17:55:59.913175 31017 ongoing_state_transfers.cc:557] State transfer "cron" thread started.
   I0318 17:55:59.914693 31011 daemon.cc:134] Connecting to the coordinator.
   W0318 17:55:59.914693 31018 replication_manager.cc:1366] Replication "cron" thread started.
   I0318 17:55:59.917939 31011 daemon.cc:143] Starting network workers.
   I0318 17:55:59.918061 31011 daemon.cc:155] Network workers started.
   I0318 17:55:59.923060 31011 daemon.cc:162] Installing new configuration version 0
   E0318 17:55:59.924705 31011 daemon.cc:171] We've been configured to a dummy node.
   I0318 17:55:59.932317 31011 daemon.cc:187] Pausing communication for reconfiguration.
   I0318 17:55:59.980757 31011 daemon.cc:202] Reconfiguration complete; unpausing communication.
4. Daemon rejects config:
   INFO:root:new host uses ID Unidentified(127.0.0.1, 42735)
   INFO:root:Unidentified(127.0.0.1, 42735) identified by Instance(addr='127.0.0.2', inport=38747, inver=1, outport=58378, outver=1)
   DEBUG:root:Instance(addr='127.0.0.2', inport=38747, inver=1, outport=58378, outver=1) acked config 0

Please note that if sequence is different: daemon -> node -> add space, all is fine:
ploj@hive:~/github/HyperDex$ hyperdex-coordinator --control-port 6970 --host-port 1234 --logging debug
INFO:root:Fresh cluster starting.
INFO:root:new host uses ID Unidentified(127.0.0.1, 42736)
INFO:root:Unidentified(127.0.0.1, 42736) identified by Instance(addr='127.0.0.2', inport=60824, inver=1, outport=34956, outver=1)
DEBUG:root:Instance(addr='127.0.0.2', inport=60824, inver=1, outport=34956, outver=1) acked config 0
INFO:root:created new space "phonebook3"
DEBUG:root:Instance(addr='127.0.0.2', inport=60824, inver=1, outport=34956, outver=1) acked config 1

[callen@sorna HyperDex]$ /usr/local/Cellar/autoconf/2.68/bin/autoreconf -i
configure.ac:10: warning: AC_LANG_CONFTEST: no AC_LANG_SOURCE call detected in body
../../lib/autoconf/lang.m4:198: AC_LANG_CONFTEST is expanded from...
../../lib/autoconf/general.m4:2671: _AC_LINK_IFELSE is expanded from...
../../lib/autoconf/general.m4:2681: AC_LINK_IFELSE is expanded from...
/usr/bin/../share/aclocal/libtool.m4:1100: _LT_SYS_MODULE_PATH_AIX is expanded from...
/usr/bin/../share/aclocal/libtool.m4:5331: _LT_LINKER_SHLIBS is expanded from...
/usr/bin/../share/aclocal/libtool.m4:5416: _LT_LANG_C_CONFIG is expanded from...
/usr/bin/../share/aclocal/libtool.m4:240: _LT_SETUP is expanded from...
/usr/bin/../share/aclocal/libtool.m4:104: LT_INIT is expanded from...
configure.ac:10: the top level
configure.ac:10: warning: AC_LANG_CONFTEST: no AC_LANG_SOURCE call detected in body
../../lib/autoconf/lang.m4:198: AC_LANG_CONFTEST is expanded from...
../../lib/autoconf/general.m4:2671: _AC_LINK_IFELSE is expanded from...
../../lib/autoconf/general.m4:2681: AC_LINK_IFELSE is expanded from...
/usr/bin/../share/aclocal/libtool.m4:5331: _LT_LINKER_SHLIBS is expanded from...
/usr/bin/../share/aclocal/libtool.m4:5416: _LT_LANG_C_CONFIG is expanded from...
/usr/bin/../share/aclocal/libtool.m4:240: _LT_SETUP is expanded from...
/usr/bin/../share/aclocal/libtool.m4:104: LT_INIT is expanded from...
configure.ac:10: the top level
configure.ac:16: warning: AC_LANG_CONFTEST: no AC_LANG_SOURCE call detected in body
../../lib/autoconf/lang.m4:198: AC_LANG_CONFTEST is expanded from...
../../lib/autoconf/general.m4:2671: _AC_LINK_IFELSE is expanded from...
../../lib/autoconf/general.m4:2681: AC_LINK_IFELSE is expanded from...
/usr/bin/../share/aclocal/libtool.m4:1100: _LT_SYS_MODULE_PATH_AIX is expanded from...
/usr/bin/../share/aclocal/libtool.m4:6443: _LT_LANG_CXX_CONFIG is expanded from...
/usr/bin/../share/aclocal/libtool.m4:822: _LT_LANG is expanded from...
/usr/bin/../share/aclocal/libtool.m4:811: LT_LANG is expanded from...
configure.ac:16: the top level
configure.ac:10: warning: AC_LANG_CONFTEST: no AC_LANG_SOURCE call detected in body
../../lib/autoconf/lang.m4:198: AC_LANG_CONFTEST is expanded from...
../../lib/autoconf/general.m4:2671: _AC_LINK_IFELSE is expanded from...
../../lib/autoconf/general.m4:2681: AC_LINK_IFELSE is expanded from...
aclocal.m4:1112: _LT_SYS_MODULE_PATH_AIX is expanded from...
aclocal.m4:5343: _LT_LINKER_SHLIBS is expanded from...
aclocal.m4:5424: _LT_LANG_C_CONFIG is expanded from...
aclocal.m4:258: _LT_SETUP is expanded from...
aclocal.m4:122: LT_INIT is expanded from...
configure.ac:10: the top level
configure.ac:10: warning: AC_LANG_CONFTEST: no AC_LANG_SOURCE call detected in body
../../lib/autoconf/lang.m4:198: AC_LANG_CONFTEST is expanded from...
../../lib/autoconf/general.m4:2671: _AC_LINK_IFELSE is expanded from...
../../lib/autoconf/general.m4:2681: AC_LINK_IFELSE is expanded from...
aclocal.m4:5343: _LT_LINKER_SHLIBS is expanded from...
aclocal.m4:5424: _LT_LANG_C_CONFIG is expanded from...
aclocal.m4:258: _LT_SETUP is expanded from...
aclocal.m4:122: LT_INIT is expanded from...
configure.ac:10: the top level
configure.ac:16: warning: AC_LANG_CONFTEST: no AC_LANG_SOURCE call detected in body
../../lib/autoconf/lang.m4:198: AC_LANG_CONFTEST is expanded from...
../../lib/autoconf/general.m4:2671: _AC_LINK_IFELSE is expanded from...
../../lib/autoconf/general.m4:2681: AC_LINK_IFELSE is expanded from...
aclocal.m4:1112: _LT_SYS_MODULE_PATH_AIX is expanded from...
aclocal.m4:6451: _LT_LANG_CXX_CONFIG is expanded from...
aclocal.m4:834: _LT_LANG is expanded from...
aclocal.m4:823: LT_LANG is expanded from...
configure.ac:16: the top level
glibtoolize: putting auxiliary files in .'. glibtoolize: copying file./ltmain.sh'
glibtoolize: putting macros in AC_CONFIG_MACRO_DIR, m4'. glibtoolize: copying filem4/libtool.m4'
glibtoolize: copying file m4/ltoptions.m4' glibtoolize: copying filem4/ltsugar.m4'
glibtoolize: copying file m4/ltversion.m4' glibtoolize: copying filem4/lt~obsolete.m4'
configure.ac:10: warning: AC_LANG_CONFTEST: no AC_LANG_SOURCE call detected in body
../../lib/autoconf/lang.m4:198: AC_LANG_CONFTEST is expanded from...
../../lib/autoconf/general.m4:2671: _AC_LINK_IFELSE is expanded from...
../../lib/autoconf/general.m4:2681: AC_LINK_IFELSE is expanded from...
m4/libtool.m4:1100: _LT_SYS_MODULE_PATH_AIX is expanded from...
m4/libtool.m4:5331: _LT_LINKER_SHLIBS is expanded from...
m4/libtool.m4:5416: _LT_LANG_C_CONFIG is expanded from...
m4/libtool.m4:240: _LT_SETUP is expanded from...
m4/libtool.m4:104: LT_INIT is expanded from...
configure.ac:10: the top level
configure.ac:10: warning: AC_LANG_CONFTEST: no AC_LANG_SOURCE call detected in body
../../lib/autoconf/lang.m4:198: AC_LANG_CONFTEST is expanded from...
../../lib/autoconf/general.m4:2671: _AC_LINK_IFELSE is expanded from...
../../lib/autoconf/general.m4:2681: AC_LINK_IFELSE is expanded from...
m4/libtool.m4:5331: _LT_LINKER_SHLIBS is expanded from...
m4/libtool.m4:5416: _LT_LANG_C_CONFIG is expanded from...
m4/libtool.m4:240: _LT_SETUP is expanded from...
m4/libtool.m4:104: LT_INIT is expanded from...
configure.ac:10: the top level
configure.ac:16: warning: AC_LANG_CONFTEST: no AC_LANG_SOURCE call detected in body
../../lib/autoconf/lang.m4:198: AC_LANG_CONFTEST is expanded from...
../../lib/autoconf/general.m4:2671: _AC_LINK_IFELSE is expanded from...
../../lib/autoconf/general.m4:2681: AC_LINK_IFELSE is expanded from...
m4/libtool.m4:1100: _LT_SYS_MODULE_PATH_AIX is expanded from...
m4/libtool.m4:6443: _LT_LANG_CXX_CONFIG is expanded from...
m4/libtool.m4:822: _LT_LANG is expanded from...
m4/libtool.m4:811: LT_LANG is expanded from...
configure.ac:16: the top level
configure.ac:10: warning: AC_LANG_CONFTEST: no AC_LANG_SOURCE call detected in body
../../lib/autoconf/lang.m4:198: AC_LANG_CONFTEST is expanded from...
../../lib/autoconf/general.m4:2671: _AC_LINK_IFELSE is expanded from...
../../lib/autoconf/general.m4:2681: AC_LINK_IFELSE is expanded from...
m4/libtool.m4:1100: _LT_SYS_MODULE_PATH_AIX is expanded from...
m4/libtool.m4:5331: _LT_LINKER_SHLIBS is expanded from...
m4/libtool.m4:5416: _LT_LANG_C_CONFIG is expanded from...
m4/libtool.m4:240: _LT_SETUP is expanded from...
m4/libtool.m4:104: LT_INIT is expanded from...
configure.ac:10: the top level
configure.ac:10: warning: AC_LANG_CONFTEST: no AC_LANG_SOURCE call detected in body
../../lib/autoconf/lang.m4:198: AC_LANG_CONFTEST is expanded from...
../../lib/autoconf/general.m4:2671: _AC_LINK_IFELSE is expanded from...
../../lib/autoconf/general.m4:2681: AC_LINK_IFELSE is expanded from...
m4/libtool.m4:5331: _LT_LINKER_SHLIBS is expanded from...
m4/libtool.m4:5416: _LT_LANG_C_CONFIG is expanded from...
m4/libtool.m4:240: _LT_SETUP is expanded from...
m4/libtool.m4:104: LT_INIT is expanded from...
configure.ac:10: the top level
configure.ac:16: warning: AC_LANG_CONFTEST: no AC_LANG_SOURCE call detected in body
../../lib/autoconf/lang.m4:198: AC_LANG_CONFTEST is expanded from...
../../lib/autoconf/general.m4:2671: _AC_LINK_IFELSE is expanded from...
../../lib/autoconf/general.m4:2681: AC_LINK_IFELSE is expanded from...
m4/libtool.m4:1100: _LT_SYS_MODULE_PATH_AIX is expanded from...
m4/libtool.m4:6443: _LT_LANG_CXX_CONFIG is expanded from...
m4/libtool.m4:822: _LT_LANG is expanded from...
m4/libtool.m4:811: LT_LANG is expanded from...
configure.ac:16: the top level
configure.ac:29: error: possibly undefined macro: AC_PROG_JAR
If this token and others are legitimate, please use m4_pattern_allow.
See the Autoconf documentation.
configure.ac:30: error: possibly undefined macro: AC_PROG_JAVAC
configure.ac:31: error: possibly undefined macro: AC_JNI_INCLUDE_DIR
configure.ac:150: error: possibly undefined macro: AC_PYTHON_DEVEL
autoreconf: /usr/local/Cellar/autoconf/2.68/bin/autoconf failed with exit status: 1

Hi,

I built this space:

$ hyperdex-coordinator-control --host 127.0.0.1 --port 6970 add-space << EOF
space test
dimensions id (int64), name
key id auto 1 3
EOF

When trying to insert a row I get:

c.put ('test',1, {'name': 'abc' })
Traceback (most recent call last):
File "", line 1, in
File "hyperclient.pyx", line 827, in hyperclient.Client.put (hyperclient/python/hyperclient.c:11838)
TypeError: Argument 'key' has incorrect type (expected str, got int)

It looks like a key cannot be an integer?

Cheers
Peter

Immediate consistency makes operations easy to reason about, however without the ability to conditionally modify, some of it's advantage is lost. Would it be possible to extend the public API to return, and accept for comparison, some hash value of a current dataset entry such that conditional update can be performed?

In each case the hash would be a small, base64 encoded, constant length hash of the entries attributes. The following should be valid for all modifiers, i.e. insertion and deletion.

• A null hash field would indicate to perform the modification without condition, as per the current implementation.
• A hash with any value should only perform it's intended operation if the hash given to the command is identical to the hash contained within the current entry.
• An empty hash (empty string) matches with the intentional non-existence of an entry (i.e. insert would fail if an entry for that key currently existed with the space.)

An example interaction may be:

Insert entry, now returns special _hash value.

>>> import hyperclient
>>> c = hyperclient.Client('127.0.0.1', 1234)
>>> c.insert('phonebook', 'jsmith1', {'first': 'John', 'last': 'Smith', 'phone': 6075551024})
True
>>> c.lookup('phonebook', 'jsmith1')
phonebook(first='John', last='Smith', phone='0\x8d!j\x01', username='jsmith1', _hash: 'a6n9bf')

Attempt to modify with incorrect hash fails:

>>> c.insert('phonebook', 'jsmith1', {'phone': 6075552048}, 'zzz9bf')
False
>>> c.lookup('phonebook', 'jsmith1')
phonebook(first='John', last='Smith', phone='0\x8d!j\x01', _hash: 'a6n9bf')

Attempt to modify with correct hash succeeds:

>>> c.insert('phonebook', 'jsmith1', {'phone': 6075552048}, 'a6n9bf')
True
>>> c.lookup('phonebook', 'jsmith1')
phonebook(first='John', last='Smith', phone='0\x91!j\x01', _hash: 'q8bbo0')

Attempt to modify with null hash succeeds:

>>> c.insert('phonebook', 'jsmith1', {'phone': 6075551024})
True
>>> c.lookup('phonebook', 'jsmith1')
phonebook(first='John', last='Smith', phone='0\x8d!j\x01', _hash: 'q8bbo0')

Attempt to modify with empty hash fails as key already in use:

>>> c.insert('phonebook', 'jsmith1', {'first': 'Johnny', 'last': 'Smithton', 'phone': 923478383}, "")
False
>>> c.lookup('phonebook', 'jsmith1')
phonebook(first='John', last='Smith', phone='0\x8d!j\x01', _hash: 'q8bbo0')

Attempt to modify with null hash succeeds, overwriting the current entry:

>>> c.insert('phonebook', 'jsmith1', {'first': 'Johnny', 'last': 'Smithton', 'phone': 6075551024})
False
>>> c.lookup('phonebook', 'jsmith1')
phonebook(first='Johnny', last='Smithton', phone='0\x6f!j\x01', _hash: '93jj7d')

I think that this adds a huge use-case for the store.

The tests are binary-test, count-test, and replication-stress-test. I tried to run them based on my understanding, e.g. adding spaces with some specific descriptions, and invoking the command line. But nothing happened.
It'd be good if there are instructions on how to run them so once we changed something we have a way to detect regression (of course, more tests are wanted)

I'd like to see setrange and getrange added to the API (functionally equivalent to the respective redis operations).
These operations are needed for a k/v-store to support segmented mmap()-style usage.

There are several things wrong when trying to install the binary package on plain Ubuntu 12.04 following the instructions at http://hyperdex.org/download :

A) Ubuntu does not support a root user by default so the correct commands for installing the package would be something like:

wget -O - http://ubuntu.hyperdex.org/hyperdex.gpg.key | sudo apt-key add -
sudo wget -O /etc/apt/sources.list.d/hyperdex.list http://ubuntu.hyperdex.org/hyperdex.list
sudo aptitude update

B) aptitude is not installed by default, so you would need:

sudo apt-get install aptitude

C) The package libgoogle-glog0 is not supported anymore and would require adding untrusted PPAs to install it.

When working with hyperdex, due to it's speed, one of the time costs is requests. Ideally there would be a mechanism to send multiple operations to hyperdex in a single request to minimise this networking cost. Clients may be intelligent enough to split up a multiple request if various indexes are best sent to different nodes within a cluster.

While OSX supports native pthreads, a few missing functions keep libe and libpo6 from building (and sadly keeps HyperDex from compiling on OSX).

Specifically, OSX lacks:
pthread_barriers
pthread_spinlock

Perhaps you could use the more robust primitives from:
www.concurrencykit.org

Also, you are using:
REALTIME_CLOCK which is not supported on OSX.

Use mach_absolute time instead:
http://developer.apple.com/library/mac/#qa/qa1398/_index.html

....

Compile errors occur in benchmarks subfolder:

make[2]: Entering directory .../libe/0.2.7/benchmarks' CXX lockfree_fifo.o CXX lockfree_hash_map.o CXX lockfree_hash_set.o CXX locking_iterable_fifo.o CXX nonblocking_bounded_fifo.o In file included from locking_iterable_fifo.cc:42: ...libpo6/0.2.3/include/po6/threads/barrier.h:57: error: ‘pthread_barrier_t’ does not name a type ...libpo6/0.2.3/include/po6/threads/barrier.h: In constructor ‘po6::threads::barrier::barrier(size_t)’: ...libpo6/0.2.3/include/po6/threads/barrier.h:62: error: class ‘po6::threads::barrier’ does not have any field named ‘m_barrier’ ...libpo6/0.2.3/include/po6/threads/barrier.h:64: error: ‘m_barrier’ was not declared in this scope ...libpo6/0.2.3/include/po6/threads/barrier.h:64: error: ‘pthread_barrier_init’ was not declared in this scope ...libpo6/0.2.3/include/po6/threads/barrier.h: In destructor ‘po6::threads::barrier::~barrier()’: ...libpo6/0.2.3/include/po6/threads/barrier.h:75: error: ‘m_barrier’ was not declared in this scope ...libpo6/0.2.3/include/po6/threads/barrier.h:75: error: ‘pthread_barrier_destroy’ was not declared in this scope ...libpo6/0.2.3/include/po6/threads/barrier.h: In member function ‘bool po6::threads::barrier::wait()’: ...libpo6/0.2.3/include/po6/threads/barrier.h:88: error: ‘m_barrier’ was not declared in this scope ...libpo6/0.2.3/include/po6/threads/barrier.h:88: error: ‘pthread_barrier_wait’ was not declared in this scope ...libpo6/0.2.3/include/po6/threads/barrier.h:90: error: ‘PTHREAD_BARRIER_SERIAL_THREAD’ was not declared in this scope ...libpo6/0.2.3/include/po6/threads/barrier.h:95: error: ‘PTHREAD_BARRIER_SERIAL_THREAD’ was not declared in this scope In file included from locking_iterable_fifo.cc:47: .../libe/0.2.7/include/e/timer.h: In function ‘uint64_t e::time()’: .../libe/0.2.7/include/e/timer.h:50: error: ‘CLOCK_REALTIME’ was not declared in this scope .../libe/0.2.7/include/e/timer.h:50: error: ‘clock_gettime’ was not declared in this scope .../libe/0.2.7/include/e/timer.h: In member function ‘void e::stopwatch::reset()’: .../libe/0.2.7/include/e/timer.h:127: error: ‘CLOCK_REALTIME’ was not declared in this scope .../libe/0.2.7/include/e/timer.h:127: error: ‘clock_gettime’ was not declared in this scope .../libe/0.2.7/include/e/timer.h: In member function ‘uint64_t e::stopwatch::resolution()’: .../libe/0.2.7/include/e/timer.h:137: error: ‘CLOCK_REALTIME’ was not declared in this scope .../libe/0.2.7/include/e/timer.h:137: error: ‘clock_getres’ was not declared in this scope .../libe/0.2.7/include/e/timer.h: In member function ‘uint64_t e::stopwatch::peek()’: .../libe/0.2.7/include/e/timer.h:149: error: ‘CLOCK_REALTIME’ was not declared in this scope .../libe/0.2.7/include/e/timer.h:149: error: ‘clock_gettime’ was not declared in this scope In file included from .../libe/0.2.7/include/e/locking_iterable_fifo.h:40, from locking_iterable_fifo.cc:48: ...libpo6/0.2.3/include/po6/threads/spinlock.h: At global scope: ...libpo6/0.2.3/include/po6/threads/spinlock.h:62: error: ‘pthread_spinlock_t’ does not name a type ...libpo6/0.2.3/include/po6/threads/spinlock.h: In constructor ‘po6::threads::spinlock::spinlock()’: ...libpo6/0.2.3/include/po6/threads/spinlock.h:80: error: class ‘po6::threads::spinlock’ does not have any field named ‘m_spin’ ...libpo6/0.2.3/include/po6/threads/spinlock.h:82: error: ‘m_spin’ was not declared in this scope ...libpo6/0.2.3/include/po6/threads/spinlock.h:82: error: ‘pthread_spin_init’ was not declared in this scope ...libpo6/0.2.3/include/po6/threads/spinlock.h: In destructor ‘po6::threads::spinlock::~spinlock()’: ...libpo6/0.2.3/include/po6/threads/spinlock.h:93: error: ‘m_spin’ was not declared in this scope ...libpo6/0.2.3/include/po6/threads/spinlock.h:93: error: ‘pthread_spin_destroy’ was not declared in this scope ...libpo6/0.2.3/include/po6/threads/spinlock.h: In member function ‘void po6::threads::spinlock::lock()’: ...libpo6/0.2.3/include/po6/threads/spinlock.h:106: error: ‘m_spin’ was not declared in this scope ...libpo6/0.2.3/include/po6/threads/spinlock.h:106: error: ‘pthread_spin_lock’ was not declared in this scope ...libpo6/0.2.3/include/po6/threads/spinlock.h: In member function ‘bool po6::threads::spinlock::trylock()’: ...libpo6/0.2.3/include/po6/threads/spinlock.h:117: error: ‘m_spin’ was not declared in this scope ...libpo6/0.2.3/include/po6/threads/spinlock.h:117: error: ‘pthread_spin_trylock’ was not declared in this scope ...libpo6/0.2.3/include/po6/threads/spinlock.h: In member function ‘void po6::threads::spinlock::unlock()’: ...libpo6/0.2.3/include/po6/threads/spinlock.h:136: error: ‘m_spin’ was not declared in this scope ...libpo6/0.2.3/include/po6/threads/spinlock.h:136: error: ‘pthread_spin_unlock’ was not declared in this scope lockfree_hash_map.cc: In function ‘void worker_thread(e::lockfree_hash_map<long long unsigned int, long long unsigned int, id>*)’: lockfree_hash_map.cc:143: error: ‘drand48_data’ was not declared in this scope lockfree_hash_map.cc:143: error: expected;' before ‘buf’
lockfree_hash_map.cc:147: error: invalid conversion from ‘opaque_pthread_t’ to ‘short unsigned int’
lockfree_hash_map.cc:148: error: ‘buf’ was not declared in this scope
lockfree_hash_map.cc:148: error: ‘seed48_r’ was not declared in this scope
lockfree_hash_map.cc:154: error: ‘lrand48_r’ was not declared in this scope
lockfree_hash_map.cc:163: error: ISO C++ forbids comparison between pointer and integer
lockfree_hash_map.cc:175: error: invalid conversion from ‘opaque_pthread_t’ to ‘long long unsigned int’
lockfree_hash_map.cc:175: error: initializing argument 2 of ‘bool e::lockfree_hash_map<K, V, H>::insert(const K&, const V&) [with K = long long unsigned int, V = long long unsigned int, uint64_t (* H)(const K&) = id]’
lockfree_hash_set.cc: In function ‘void worker_thread(e::lockfree_hash_set<long long unsigned int, id>_)’:
lockfree_hash_set.cc:142: error: ‘drand48_data’ was not declared in this scope
lockfree_hash_set.cc:142: error: expected ;' before ‘buf’ lockfree_hash_set.cc:146: error: invalid conversion from ‘_opaque_pthread_t_’ to ‘short unsigned int’ lockfree_hash_set.cc:147: error: ‘buf’ was not declared in this scope lockfree_hash_set.cc:147: error: ‘seed48_r’ was not declared in this scope lockfree_hash_set.cc:152: error: ‘lrand48_r’ was not declared in this scope make[2]: **\* [locking_iterable_fifo.o] Error 1 make[2]: **\* Waiting for unfinished jobs.... make[2]: **\* [lockfree_hash_set.o] Error 1 make[2]: **\* [lockfree_hash_map.o] Error 1 make[2]: Leaving directory .../libe/0.2.7/benchmarks'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory`.../libe/0.2.7'
make: *** [all] Error 2

Hey,

I can't get the latest repo to compile because it's looking for a (seemingly non-existent) include in the po6 library.

/hyperdex/hyperdex/coordinatorlink.h:39:30: fatal error: po6/net/hostname.h: No such file or directory

It looks like this was changed recently and my guess is that I will be able to pull behind that commit and get past this issue, but I figured I would let you guys know. I looked at the po6 source as well (here on github), but I can't see any reference to this file there.

Cheers,
Mike

Currently the search command allows the predicate of an attribute to be a range of integers whereas the condput command only allows an equality match. There are scenarios where it would be useful to test if the value of an attribute is in a particular interval before updating the object.

Every op that exists for int64 should exist for float as well.

I tried to start with ./hyperdex-coordinator -?, but I get the aforementioned error message. I also tried the command in the tutorial: hyperdex-coordinator --control-port 6970 --host-port 1234 --logging debug
As far as I understand, this should work? I installed all the prerequisites (they got installed into /usr/local/lib64), although I'm a little confused about the fact that the link to libpopt here http://hyperdex.org/download/ leads to rpm5. I have libpopt0 installed though, I'm on opensuse 12.1.
I installed hyperdex from the tarball into a folder in my home directory.

also more, I tried to start the daemon anyway: Where does hyperdex look for the libraries? ./hyperdex-daemon: error while loading shared libraries: libcityhash.so.0: cannot open shared object file: No such file or directory
It's installed into /usr/local/lib64

CXX hyperdaemon/hyperdex_daemon-searches.o
hyperdaemon/searches.cc: In member function ‘void hyperdaemon::searches::group_keyop(const hyperdex::entityid&, const hyperdex::entityid&, uint64_t, const hyperspacehashing::search&, hyperdex::network_msgtype, const e::slice&)’:
hyperdaemon/searches.cc:266:19: error: ‘class e::buffer::packer’ has no member named ‘copy’
In file included from /opt/hyperdex/include/e/lockfree_hash_map.h:36:0,
from ./hyperdisk/hyperdisk/disk.h:43,
from hyperdaemon/searches.cc:32:

When trying to put() any complex type -- list, set or map -- that contain int64 or float elements/key/value primitives I get a Server Error

How to reproduce:

Space definition:

hyperdex-coordinator-control --host 127.0.0.1 --port 6970 add-space << EOF
space phonebook
dimensions username, first, last, pets (map (string, string)), phone (map (string, int64)), address (map (int64, string)), pant (map (int64, int64))
key username auto 1 3
subspace first, last auto 2 3
EOF

Python session where the put() fails:

Python 2.7.3 (default, May 29 2012, 14:54:22)
[GCC 4.6.3 20120306 (Red Hat 4.6.3-2)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import hyperclient
c = hyperclient.Client('127.0.0.1', 1234)
c.put('phonebook', 'jsmith1', {'first': 'John', 'last': 'Smith', 'pets': {'padthai': 'dog', 'pebbles': 'bird', 'felix': 'cat'}, 'phone': {'home': 4165551024, 'work': 4165551025, 'mobile': 4165551026}, 'address': {15: 'Islington Ave', 34: 'West Hoyven St.', 5123: 'Cladwell Cresent'}, 'pant': { 34: 32, 36: 34, 40: -5}})
Traceback (most recent call last):
File "", line 1, in
File "hyperclient.pyx", line 1126, in hyperclient.Client.put (hyperclient/python/hyperclient.c:17487)
File "hyperclient.pyx", line 774, in hyperclient.DeferredFromAttrs.wait (hyperclient/python/hyperclient.c:12213)
hyperclient.HyperClientException: HyperClient(HYPERCLIENT_SERVERERROR, Server Error)

Python session where the put() succeeds: ie., don't put() any complex attrs with int64 or float types

Python 2.7.3 (default, May 29 2012, 14:54:22)
[GCC 4.6.3 20120306 (Red Hat 4.6.3-2)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import hyperclient
c = hyperclient.Client('127.0.0.1', 1234)
c.put('phonebook', 'jsmith1', {'first': 'John', 'last': 'Smith', 'pets': {'padthai': 'dog', 'pebbles': 'bird', 'felix': 'cat'}})
True

I get the exact same exception in java:

Exception in thread "main" hyperclient.HyperClientException: Server Error
at hyperclient.DeferredFromAttrs.waitFor(DeferredFromAttrs.java:57)
at hyperclient.HyperClient.put(HyperClient.java:734)
at HyperPut.main(HyperPut.java:74)

With the code:

import hyperclient.*;
import java.math.*;
import java.util.*;

public class HyperPut
{
        public static void main(String[] args) throws Exception
        {
                String key = args[0];

                String first = args[1];
                String last = args[2];
                //String phone = args[3];

        HashMap<String,Object> attributes = new HashMap<String,Object>();

        Vector<Object> list = new Vector<Object>();
        //list.add("dog");
        //list.add("cat");
        //list.add("tree");
        //list.add("shrub");
        //list.add(null);
        //list.add(new Long(10));
        //list.add(new Long(5));

        HashSet<Object> set = new HashSet<Object>();
        set.add("tree");
        set.add("shrub");
        //set.add("bat");
        //set.add("cat");
        //set.add("dog");
        //set.add(null);
        //set.add(new Long(10));
        //set.add(new Long(5));

        HashMap<Object,Object> pets = new HashMap<Object,Object>();
        pets.put("padthi","dog");
        pets.put("felix","cat");
        pets.put("pebbles","bird");

        HashMap<Object,Long> phone = new HashMap<Object,Long>();
        /*
        phone.put("home",new Long(4165551024L));
        phone.put("work",new Long(4165551025L));
        phone.put("mobile",new Long(4165551026L));
        */
        phone.put("home",4165551034L);
        phone.put("work",4165551035L);
        phone.put("mobile",4165551036L);

        HashMap<Object,Object> address = new HashMap<Object,Object>();
        address.put(new Long(34),"Wayward Dr.");
        address.put(new Long(1024),"Silicon Rd.");
        address.put(new Long(25),"Green Strings Crt.");

        HashMap<Object,Object> pant = new HashMap<Object,Object>();
        pant.put(new Long(34),new Long(32));
        pant.put(new Long(36),new Long(30));
        pant.put(new Long(40),new Long(52));

        attributes.put("first",first);
        attributes.put("last",last);
        //attributes.put("bogus","bogus");
        //attributes.put("phone",Long.parseLong(phone));
        //attributes.put("phone",list);
        //attributes.put("phone",set);
        attributes.put("pets",pets);
        attributes.put("phone",phone);
        attributes.put("address",address);
        attributes.put("pant",pant);

                HyperClient c = new HyperClient("127.0.0.1",1234);

                System.out.println(c.put("phonebook",key,attributes));

        attributes.put("first","George");
                System.out.println(c.put("phonebook","gtolomic",attributes));

        //Deferred d = c.async_put("phonebook",key,attributes);

        //System.out.println(d.waitFor());
        }
}

This used to completely work in python and java.

Here's the trivial patch as written by me for gentoo. Feel free to use or improve.

diff -Naur libe-0.2.7/benchmarks/lockfree_hash_map.cc libe-0.2.7-new/benchmarks/lockfree_hash_map.cc
--- libe-0.2.7/benchmarks/lockfree_hash_map.cc 2011-10-12 04:02:24.000000000 +0200
+++ libe-0.2.7-new/benchmarks/lockfree_hash_map.cc 2012-08-20 14:59:39.135789923 +0200
@@ -30,6 +30,7 @@
// C includes
#include
#include <stdint.h>
+#include <unistd.h>

// C++
#include
diff -Naur libe-0.2.7/benchmarks/lockfree_hash_set.cc libe-0.2.7-new/benchmarks/lockfree_hash_set.cc
--- libe-0.2.7/benchmarks/lockfree_hash_set.cc 2011-08-26 16:24:54.000000000 +0200
+++ libe-0.2.7-new/benchmarks/lockfree_hash_set.cc 2012-08-20 15:00:37.877793998 +0200
@@ -30,6 +30,7 @@
// C includes
#include
#include <stdint.h>
+#include <unistd.h>

// C++
#include

I think I have already success installed the HyperDex.
When I run order "hyperdex-coordinator --control-port 6970 --host-port 1234" as tutorial wrote. It ok.

but When I run the order :"hyperdex-daemon --host 127.0.0.1 --port 1234 --bind-to 127.0.0.2 ", an error happened, it said

"
hyperdex-daemon: error while loading shared libraries: libhyperdisk.so.0: cannot open shared object file: No such file or directory

"

Need Help. Thank You.

Apologies if some of my questions are answered somewhere, I couldn't find it.

• Are values in HyperDex binary safe (garbage in/garbage out)?
• Is there a limit on the value-size of individual keys, what is the recommended maximum size?

It would be great if the narrative documentation could be supplemented with a structured data-type/command-reference (similar to the redis-reference) so that questions like the above can be easily looked up.

Per Robert Escriva's email:

Currently, the system is assumed to be always on. Shutting down the
system without violating the consistency guarantees in the process
requires global coordination to flush outstanding operations. This will
be part of a near-future release of HyperDex as it's an important
feature to have.

$ autoreconf -i -v;

autoreconf2.50: Entering directory .' autoreconf2.50: configure.ac: not using Gettext autoreconf2.50: running: aclocal -I m4 ${ACLOCAL_FLAGS} autoreconf2.50: configure.ac: tracing autoreconf2.50: running: libtoolize --copy libtoolize: putting auxiliary files in.'.
libtoolize: copying file ./ltmain.sh' libtoolize: putting macros in AC_CONFIG_MACRO_DIR,m4'.
libtoolize: copying file m4/libtool.m4' libtoolize: copying filem4/ltoptions.m4'
libtoolize: copying file m4/ltsugar.m4' libtoolize: copying filem4/ltversion.m4'
libtoolize: copying file `m4/lt~obsolete.m4'
autoreconf2.50: running: /usr/bin/autoconf
configure.ac:31: error: possibly undefined macro: AC_PROG_JAR
If this token and others are legitimate, please use m4_pattern_allow.
See the Autoconf documentation.
configure.ac:32: error: possibly undefined macro: AC_PROG_JAVAC
configure.ac:33: error: possibly undefined macro: AC_JNI_INCLUDE_DIR
configure.ac:151: error: possibly undefined macro: AC_PYTHON_DEVEL
autoreconf2.50: /usr/bin/autoconf failed with exit status: 1

I success configuare the hyperdex-0.2b8 without error . But When I 'make',the error happen.

"
make all-am
make[1]: Entering directory /usr/home/yanglong1/hyperdex/hyperdex-0.2b8' CXX hyperspacehashing/libhyperspacehashing_la-cfloat.lo hyperspacehashing/cfloat.cc: In function ‘uint64_t hyperspacehashing::cfloat(uint64_t, unsigned int)’: hyperspacehashing/cfloat.cc:61: warning: converting to ‘uint64_t’ from ‘double’ CXX hyperspacehashing/libhyperspacehashing_la-hashes.lo hyperspacehashing/hashes.cc: In function ‘uint64_t hyperspacehashing::lendian(const e::slice&)’: hyperspacehashing/hashes.cc:56: error: ‘le64toh’ was not declared in this scope make[1]: *** [hyperspacehashing/libhyperspacehashing_la-hashes.lo] Error 1 make[1]: Leaving directory/usr/home/yanglong1/hyperdex/hyperdex-0.2b8'
make: *** [all] Error 2
"

Need Help.
I use Centos 5.4 x64.
Thank You.

Hi,

Can I limitting memory usage of a hyperdex node? Can't find how to do this anywhere.
Is it possible to limit it size into 30MB (like in hypertable)?

Hi, I'm using Lubuntu 12.04... I compiled hyperdex 0.2b9 and all dependencies from source, making sure to use ./configure --enable-python-bindings. But when I try import hyperclient in Python, I get the following message: ImportError: libhyperclient.so.0: cannot open shared object file: No such file or directory.

autoreconf will generate a non-functional configure script because of 'm4/anal_warnings_cxx.m4' -- running the resulting configure script will fail with a syntax error. I had to delete 'm4/anal_warnings_cxx.m4' and run autoreconf again to get a working configure script.

In file included from replication_manager.cc:70:
In file included from ../hyperdaemon/runtimeconfig.h:35:
/usr/local/include/e/envconfig.h:54:49: error: default arguments cannot be added to an out-of-line definition of a member of a class template
envconfig<T> :: envconfig(const char* envvar, T def = T())
                                                ^     ~~~

The clang developers seem to be of the opinion that this is not a clang bug, but c++ that should be illegal:
http://clang-developers.42468.n3.nabble.com/default-arguments-cannot-be-added-to-an-out-of-line-definition-of-a-member-of-a-class-template-tp3182781p3182879.html

I download the libpo6-0.2.2.tar.gz from http://hyperdex.org/src/, and then my code are:

tar zxvf libpo6-0.2.2.tar.gz
cd libpo6-0.2.2
./configure
make
make install

then I get the following:

Making install in include
make[1]: Entering directory /root/hyperdex/libpo6-0.2.2/include' make[2]: Entering directory/root/hyperdex/libpo6-0.2.2/include'
make[2]: Nothing to be done for install-exec-am'. test -z "/usr/local/include" || /bin/mkdir -p "/usr/local/include" /bin/mkdir -p '/usr/local/include/po6/io' /usr/bin/install -c -m 644 po6/io/fd.h '/usr/local/include/po6/io' /bin/mkdir -p '/usr/local/include/po6/threads' /usr/bin/install -c -m 644 po6/threads/rwlock.h po6/threads/cond.h po6/threads/spinlock.h po6/threads/thread.h po6/threads/mutex.h po6/threads/barrier.h '/usr/local/include/po6/threads' /bin/mkdir -p '/usr/local/include/po6' /usr/bin/install -c -m 644 po6/error.h po6/noncopyable.h po6/pathname.h '/usr/local/include/po6' /bin/mkdir -p '/usr/local/include/po6/net' /usr/bin/install -c -m 644 po6/net/ipaddr.h po6/net/location.h po6/net/socket.h '/usr/local/include/po6/net' make[2]: Leaving directory/root/hyperdex/libpo6-0.2.2/include'
make[1]: Leaving directory /root/hyperdex/libpo6-0.2.2/include' Making install in test make[1]: Entering directory/root/hyperdex/libpo6-0.2.2/test'
make[2]: Entering directory /root/hyperdex/libpo6-0.2.2/test' make[2]: Nothing to be done forinstall-exec-am'.
make[2]: Nothing to be done for install-data-am'. make[2]: Leaving directory/root/hyperdex/libpo6-0.2.2/test'
make[1]: Leaving directory /root/hyperdex/libpo6-0.2.2/test' make[1]: Entering directory/root/hyperdex/libpo6-0.2.2'
make[2]: Entering directory /root/hyperdex/libpo6-0.2.2' make[2]: Nothing to be done forinstall-exec-am'.
test -z "/usr/local/lib/pkgconfig" || /bin/mkdir -p "/usr/local/lib/pkgconfig"
/usr/bin/install -c -m 644 libpo6.pc '/usr/local/lib/pkgconfig'
make[2]: Leaving directory /root/hyperdex/libpo6-0.2.2' make[1]: Leaving directory/root/hyperdex/libpo6-0.2.2'

But when I configure the "libe-0.2.5" , it shows that :
"
checking for PO6... no
configure: error: Package requirements (libpo6 >= 0.2) were not met:

No package 'libpo6' found
"

Why??? I use RedHat x86_64.
Need Help~~
Thank You.

I had to make the following same modification to each file below to make them build:

- .. include:: shards/linearizable.rst

• .. include::shards/linearizable.rst

I don't know the syntax of *.rst files, but the space after the '::' struck me as being odd, so I removed it and then everything built

doc/api/shards/atomic_add.rst
doc/api/shards/atomic_and.rst
doc/api/shards/atomic_div.rst
doc/api/shards/atomic_mod.rst
doc/api/shards/atomic_mul.rst
doc/api/shards/atomic_or.rst
doc/api/shards/atomic_sub.rst
doc/api/shards/atomic_xor.rst
doc/api/shards/del.rst
doc/api/shards/get.rst
doc/api/shards/list_lpush.rst
doc/api/shards/list_rpush.rst
doc/api/shards/put.rst
doc/api/shards/set_add.rst
doc/api/shards/set_intersect.rst
doc/api/shards/set_remove.rst
doc/api/shards/set_union.rst
doc/api/shards/string_append.rst
doc/api/shards/string_prepend.rst

Hi,

the tutorial on how to create a new space is this example:

$ hyperdex-coordinator-control --host 127.0.0.1 --port 6970 add-space << EOF
space phonebook
dimensions username, first, last, phone (uint64)
key username auto 1 3
subspace first, last, phone auto 2 3
EOF

however I get this error message:

Expected "key" (at char 56), (line:2, col:41)

Creating the space like this:

$ hyperdex-coordinator-control --host 127.0.0.1 --port 6970 add-space << EOF
space phonebook
dimensions username, first, last, phone (int64)
key username auto 1 3
subspace first, last, phone auto 2 3
EOF

works as expected (giving the type as int64 instead of uint64).

OS: OpenSuSE 12.1 x86_64
hyperdex-0.2b9

Cheers
Peter

Hello.

I am trying to put together a simple demo application using hyperdex C api. As a guidance I am using code from python hyperclient. You can see my application at https://github.com/agiz/hyperdex-c-test/blob/master/test_hyperdex.c. What it does is initialize a hyperclient instance and tries to create attributes of an object.

Before running this applications I followed instructions on how to create space described in http://hyperdex.org/doc/tutorial/. I do not know if that is important or not but when creating a space using hyperdex-coordinator-control --host 127.0.0.1 --port 6970 add-space << EOF and after typing EOF the program does not exits and just stalls. After an hour or two I have interrupted this program and received following output:

^CTraceback (most recent call last):
  File "/usr/lib64/python2.6/runpy.py", line 122, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib64/python2.6/runpy.py", line 34, in _run_code
    exec code in run_globals
  File "/usr/lib/python2.6/site-packages/hypercoordinator/client.py", line 163, in <module>
    sys.exit(main(sys.argv[1:]))
  File "/usr/lib/python2.6/site-packages/hypercoordinator/client.py", line 158, in main
    return args.func(args)
  File "/usr/lib/python2.6/site-packages/hypercoordinator/client.py", line 83, in add_space
    return send_msg(args.host, args.port, 'add-space', data)
  File "/usr/lib/python2.6/site-packages/hypercoordinator/client.py", line 46, in send_msg
    d = s.recv(4096)
KeyboardInterrupt

I do however think that space does get created nonetheless.
This is reported by hyperdex-coordinator when creating the space:

INFO:root:Coordinator started
INFO:root:new host uses ID Unidentified(127.0.0.1, 35361)
INFO:root:Unidentified(127.0.0.1, 35361) identified by Instance(addr='127.0.0.2', inport=43296, inver=1, outport=35622, outver=1)
DEBUG:root:Instance(addr='127.0.0.2', inport=43296, inver=1, outport=35622, outver=1) acked config 0
INFO:root:created new space "phonebook"
DEBUG:root:Instance(addr='127.0.0.2', inport=43296, inver=1, outport=35622, outver=1) acked config 1

When I run my test application, test_hyperdex , status reports 8520 -- HYPERCLIENT_UNKNOWNATTR.

This is reported by hyperdex-coordinator when I run my program.

INFO:root:new host uses ID Unidentified(127.0.0.1, 35467)
INFO:root:Unidentified(127.0.0.1, 35467) identified by Client(127.0.0.1, 35467)

I am using CentOS 6.2, kernel 2.6.32-220.13.1.el6.x86_64. I have built libe, libpo6 and HyperDex using github sources with gcc version 4.4.6 20110731 (Red Hat 4.4.6-3) (GCC). Before running autoreconf -i I had to set export LC_ALL=C and change AC_PREREQ([2.66]) to AC_PREREQ([2.63]). When running autoreconf -i in HyperDex for the first time I received:

configure.ac:27: error: possibly undefined macro: AC_PROG_JAR
      If this token and others are legitimate, please use m4_pattern_allow.
      See the Autoconf documentation.
configure.ac:28: error: possibly undefined macro: AC_PROG_JAVAC
configure.ac:29: error: possibly undefined macro: AC_JNI_INCLUDE_DIR
autoreconf: /usr/bin/autoconf failed with exit status: 1

but running it again reported no errors. I hope these errors are meaningless since I did not use any java bindings.

If there is anything more you need to replicate/solve this issue please let me know.

Regards, Ziga

Hi,Robert:
I've just successfully compiled hyperdex-0.2b8 in my virtualbox of ubuntu11.10.
then i started the Coordinator, Daemon and Created a phonebook space as the tutorial said, that all goes fine.
but when i did the client task:"import hyperclient", it failed with:
"Traceback (most recent call last):
File "", line 1, in
ImportError: No module named hyperclient "

i'm not quite familiar with python, could you please tell me how can i solve this problem?

thanks.

Here are some informations:
echo $PYTHONPATH
/home/tiger/Share/hyperdex/hyperdex-install/hyperdex/lib/python2.7/site-packages:/home/tiger/Share/hyperdex/hyperdex-install/hyperdex/lib/python2.7/site-packages/hypercoordinator:/home/tiger/Share/hyperdex/hyperdex-install/hyperdex/lib

ls -l /home/tiger/Share/hyperdex/hyperdex-install/hyperdex/lib/python2.7/site-packages/hypercoordinator
total 174
-rwxrwxrwx 1 root root 3992 2012-03-17 22:00 client.py*
-rwxrwxrwx 1 root root 3494 2012-03-17 22:00 client.pyc*
-rwxrwxrwx 1 root root 3494 2012-03-17 22:00 client.pyo*
-rwxrwxrwx 1 root root 27438 2012-03-17 22:00 coordinator.py*
-rwxrwxrwx 1 root root 27509 2012-03-17 22:00 coordinator.pyc*
-rwxrwxrwx 1 root root 27509 2012-03-17 22:00 coordinator.pyo*
-rwxrwxrwx 1 root root 7415 2012-03-17 22:00 hdtypes.py*
-rwxrwxrwx 1 root root 12828 2012-03-17 22:00 hdtypes.pyc*
-rwxrwxrwx 1 root root 12728 2012-03-17 22:00 hdtypes.pyo*
-rwxrwxrwx 1 root root 1560 2012-03-17 22:00 init.py*
-rwxrwxrwx 1 root root 196 2012-03-17 22:00 init.pyc*
-rwxrwxrwx 1 root root 196 2012-03-17 22:00 init.pyo*
-rwxrwxrwx 1 root root 14677 2012-03-17 22:00 parser.py*
-rwxrwxrwx 1 root root 15554 2012-03-17 22:00 parser.pyc*
-rwxrwxrwx 1 root root 15419 2012-03-17 22:00 parser.pyo*

Compile Error in hyperdex-0.2b9.

This list of errors:

yperdaemon/physical.cc: In member function 'hyperdaemon::physical::returncode hyperdaemon::physical::send(const po6::net::location&, std::auto_ptr<e::buffer>)':
hyperdaemon/physical.cc:223:9: error: 'or_32_nobarrier' is not a member of 'e::atomic'
hyperdaemon/physical.cc:231:9: error: 'and_32_nobarrier' is not a member of 'e::atomic'
hyperdaemon/physical.cc: In member function 'hyperdaemon::physical::returncode hyperdaemon::physical::recv(po6::net::location_, std::auto_ptr<e::buffer>)':
hyperdaemon/physical.cc:340:13: error: 'or_32_nobarrier' is not a member of 'e::atomic'
hyperdaemon/physical.cc:348:9: error: 'and_32_nobarrier' is not a member of 'e::atomic'
hyperdaemon/physical.cc: In member function 'hyperdaemon::physical::returncode hyperdaemon::physical::get_channel(const hazard_ptr&, po6::net::socket, hyperdaemon::physical::channel__)':
hyperdaemon/physical.cc:524:9: error: 'or_32_nobarrier' is not a member of 'e::atomic'
hyperdaemon/physical.cc: In member function 'bool hyperdaemon::physical::work_read(const hazard_ptr&, hyperdaemon::physical::channel_, po6::net::location_, std::auto_ptr<e::buffer>, hyperdaemon::physical::returncode)':
hyperdaemon/physical.cc:633:9: error: 'or_32_nobarrier' is not a member of 'e::atomic'
hyperdaemon/physical.cc: In member function 'bool hyperdaemon::physical::work_write(const hazard_ptr&, hyperdaemon::physical::channel_, hyperdaemon::physical::returncode_)':
hyperdaemon/physical.cc:741:13: error: 'or_32_nobarrier' is not a member of 'e::atomic'
make[1]: *_* [hyperdaemon/libhyperdaemon_la-physical.lo] Error 1
make[1]: Leaving directory `/home/insion/Downloads/hyperdex/hyperdex-0.2b9'
make: *** [all] Error 2

There is currently no documentation about the persistence mechanism used for hyperdex. Answers to the following would be a good start:

0. Does hyperdex persist data to disk?
1. Does hyperdex persist spaces greater than memory onto disk?
2. How does hyperdex handle requests for items currently stored on disk?
3. What format does hyperdex persist data to disk?
4. Is data persisted to disk synchronously or asynchronously flushed?
5. Is there a mechanism to re-start a cluster, without data-loss, from a catastrophic full-cluster failure?
6. Is there any possibility for the disk storage to become corrupted? Are there mechanisms in place to recover from corrupted storage?

Same for every atomic_* function.

make all-am
make[1]: Entering directory /chroot/local/portage/dev-db/hyperdex-0.4.0/work/hyperdex-0.4.0' CXX hyperdex_binary_test-binary-test.o CXX hyperdex_daemon-daemon.o x86_64-pc-linux-gnu-g++ -DHAVE_CONFIG_H -I. -O2 -pipe -march=native -c -o hyperdex_binary_test-binary-test.otest -f 'binary-test.cc' || echo './'binary-test.cc x86_64-pc-linux-gnu-g++ -DHAVE_CONFIG_H -I. -I/chroot/local/portage/dev-db/hyperdex-0.4.0/work/hyperdex-0.4.0/hyperspacehashing -I/chroot/local/portage/dev-db/hyperdex-0.4.0/work/hyperdex-0.4.0/hyperdisk -O2 -pipe -march=native -c -o hyperdex_daemon-daemon.otest -f 'daemon.cc' || echo './'daemon.cc CXX hyperdex/hyperdex_daemon-configuration.o x86_64-pc-linux-gnu-g++ -DHAVE_CONFIG_H -I. -I/chroot/local/portage/dev-db/hyperdex-0.4.0/work/hyperdex-0.4.0/hyperspacehashing -I/chroot/local/portage/dev-db/hyperdex-0.4.0/work/hyperdex-0.4.0/hyperdisk -O2 -pipe -march=native -c -o hyperdex/hyperdex_daemon-configuration.otest -f 'hyperdex/configuration.cc' || echo './'hyperdex/configuration.cc CXX hyperdex/hyperdex_daemon-configuration_parser.o x86_64-pc-linux-gnu-g++ -DHAVE_CONFIG_H -I. -I/chroot/local/portage/dev-db/hyperdex-0.4.0/work/hyperdex-0.4.0/hyperspacehashing -I/chroot/local/portage/dev-db/hyperdex-0.4.0/work/hyperdex-0.4.0/hyperdisk -O2 -pipe -march=native -c -o hyperdex/hyperdex_daemon-configuration_parser.otest -f 'hyperdex/configuration_parser.cc' || echo './'hyperdex/configuration_parser.cc CXX hyperdex/hyperdex_daemon-coordinatorlink.o x86_64-pc-linux-gnu-g++ -DHAVE_CONFIG_H -I. -I/chroot/local/portage/dev-db/hyperdex-0.4.0/work/hyperdex-0.4.0/hyperspacehashing -I/chroot/local/portage/dev-db/hyperdex-0.4.0/work/hyperdex-0.4.0/hyperdisk -O2 -pipe -march=native -c -o hyperdex/hyperdex_daemon-coordinatorlink.otest -f 'hyperdex/coordinatorlink.cc' || echo './'hyperdex/coordinatorlink.cc In file included from /usr/include/e/bufferio.h:32:0, from hyperdex/coordinatorlink.cc:35: /usr/include/po6/io/fd.h: In member function 'void po6::io::fd::close()': /usr/include/po6/io/fd.h:109:9: error: '::close' has not been declared /usr/include/po6/io/fd.h: In member function 'ssize_t po6::io::fd::read(void*, size_t)': /usr/include/po6/io/fd.h:118:12: error: '::read' has not been declared /usr/include/po6/io/fd.h: In member function 'ssize_t po6::io::fd::write(const void*, size_t)': /usr/include/po6/io/fd.h:155:12: error: '::write' has not been declared In file included from hyperclient/hyperclient.h:53:0, from binary-test.cc:42: /usr/include/po6/io/fd.h: In member function 'void po6::io::fd::close()': /usr/include/po6/io/fd.h:109:9: error: '::close' has not been declared /usr/include/po6/io/fd.h: In member function 'ssize_t po6::io::fd::read(void*, size_t)': /usr/include/po6/io/fd.h:118:12: error: '::read' has not been declared /usr/include/po6/io/fd.h: In member function 'ssize_t po6::io::fd::write(const void*, size_t)': /usr/include/po6/io/fd.h:155:12: error: '::write' has not been declared make[1]: *** [hyperdex_binary_test-binary-test.o] Error 1 make[1]: *** Waiting for unfinished jobs.... make[1]: *** [hyperdex/hyperdex_daemon-coordinatorlink.o] Error 1 make[1]: Leaving directory/chroot/local/portage/dev-db/hyperdex-0.4.0/work/hyperdex-0.4.0'
make: *** [all] Error 2

gcc version 4.7.1 (Gentoo 4.7.1 p1.2, pie-0.5.3)

Show me the river to follow? http://is.gd/Py6OcG http://youtu.be/iZFI7ISEuWw

aptitude install hyperdex

Reading package lists... Done
Building dependency tree
Reading state information... Done
Reading extended state information
Initializing package states... Done
The following packages are BROKEN:
hyperdex-daemon python-hyperclient python-hypercoordinator
The following NEW packages will be installed:
hyperdex libcityhash0{a} libe0{a} libhyperclient0{a} libhyperdisk0{a} libhyperspacehashing0{a} python-pyparsing{a}
0 packages upgraded, 10 newly installed, 0 to remove and 201 not upgraded.
Need to get 1,342kB of archives. After unpacking 6,132kB will be used.
The following packages have unmet dependencies:
hyperdex-daemon: Depends: libgoogle-glog0 which is a virtual package.
Depends: libpopt0 (>= 1.16) but 1.15-1 is installed.
Depends: libstdc++6 (>= 4.6) but 4.4.3-4ubuntu5 is installed and it is kept back.
python-hyperclient: Depends: python (>= 2.7) but 2.6.5-0ubuntu1 is installed.
python-hypercoordinator: Depends: python (>= 2.7) but 2.6.5-0ubuntu1 is installed.
The following actions will resolve these dependencies:

Keep the following packages at their current version:
hyperdex [Not Installed]
hyperdex-daemon [Not Installed]
python-hyperclient [Not Installed]
python-hypercoordinator [Not Installed]

Score is -9846

Accept this solution? [Y/n/q/?] q
Abandoning all efforts to resolve these dependencies.
Abort.

root@srv1 /home/kunthar # uname -a
Linux srv1.sintez.com.tr 2.6.32-23-server #37-Ubuntu SMP Fri Jun 11 09:11:11 UTC 2010 x86_64 GNU/Linux

root@srv1 /home/kunthar # cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=10.04
DISTRIB_CODENAME=lucid
DISTRIB_DESCRIPTION="Ubuntu 10.04.1 LTS"