hypertable / hypertable Goto Github PK

This project forked from nuggetwheat/hypertable

A flexible database focused on performance and scalability

License: GNU General Public License v3.0

CMake 2.06% Shell 4.04% Perl 8.43% C++ 62.35% Python 1.08% Ruby 0.95% HTML 0.17% JavaScript 0.44% CSS 0.19% PHP 9.09% Java 8.05% C 2.77% Thrift 0.36% Batchfile 0.01%

hypertable's Introduction

HOW TO INSTALL

You can either download an appropriate binary package for your platform or build from source. Binary packages can be obtained from here.

See this wiki page for getting started with hypertable binary packages.

HOW TO BUILD FROM SOURCE

Download the source:

You can either download a release source tar ball from the download page and unpack it in your source directory say ~/src:
```
cd ~/src
tar zxvf <path_to>/hypertable-<version>-src.tar.gz
```
or from our git repository:
```
cd ~/src
git clone git://github.com/hypertable/hypertable.git
```
From now on, we assume that your hypertable source tree is ~/src/hypertable
Install the development environment:

Run the following script to setup up the dev environment:
```
~/src/hypertable/bin/src-utils/htbuild --install dev_env
```
If it did not work for your platform, check out the HowToBuild wiki for various tips on building on various platforms.

Patches for htbuild to support your platforms are welcome :)
Configure the build:

Assuming you want your build tree to be ~/build/hypertable
```
mkdir -p ~/build/hypertable
cd ~/build/hypertable
cmake ~/src/hypertable
```
By default, hypertable gets installed in /opt/hypertable. To install into your own install directory, say $prefix, you can use:
```
cmake -DCMAKE_INSTALL_PREFIX=$prefix ~/src/hypertable
```
By default the build is configured for debug. To make a release build for production/performance test/benchmark:
```
cmake -DCMAKE_BUILD_TYPE=Release ~/src/hypertable
```
If you would like to install the build in a directory that contains a version suffix (e.g. 0.9.3.0.1d45f8d), you can configure as follows:
```
cmake -DCMAKE_BUILD_TYPE=Release -DVERSION_ADD_COMMIT_SUFFIX=1 ~/src/hypertable
```
If you would like to build OS specific packages in a directory that contains the specific OS and version (e.g. hypertable-0.9.5.0-suse_11.4-i386), you can configure as follows:
```
cmake -DCMAKE_BUILD_TYPE=Release -DPACKAGE_OS_SPECIFIC=1 ~/src/hypertable
```
Note, you can also use:
```
ccmake ~/src/hypertable
```
to change build parameters interactively.

To build shared libraries, e.g., for scripting language extensions:
```
cmake -DBUILD_SHARED_LIBS=ON ~/src/hypertable
```
Since PHP has no builtin package system, its thrift installation needs to be manually specified for ThriftBroker support: cmake -DTHRIFT_SOURCE_DIR=/home/user/src/thrift
Build Hypertable binaries.
```
make (or make -jnumber_of_cpu_or_cores_plus_1 for faster compile)
make install
```
Note, if it is a shared library install, you might need to do:
```
echo $prefix/$version/lib | \
    sudo tee /etc/ld.so.conf.d/hypertable
sudo /sbin/ldconfig
```
Or, you can use the usual LD_LIBRARY_PATH (most Unix like OS) and DYLD_LIBRARY_PATH (Mac OS X) to specify non-standard shared library directories.

HOW TO MAKE BINARY PACKAGES

The available package generators for cmake are a little different on different platforms. It is usually best to generate binary packages on their native platforms.

cd ~/build/hypertable
~/src/hypertable/bin/src-utils/htpkg generator...

where generator can be DEB (for debian packages), RPM (for redhat/fedora/centos packages), PackageMaker (for Mac OS X packages) and STGZ (for self extracting gzipped tar files.) etc. See cpack(1) for more options.

For every generator listed on the command line, the above command will generate two packages: one for the entire hypertable distribution and one for thriftbroker only.

You can also use htbuild to generate binary packages on remote machines that are not setup for development, even a base Amazon EC2 image:

~/src/hypertable/bin/src-utils/htbuild --ami ami-5946a730 DEB RPM TBZ2

At the moment, I recommend an Ubuntu/Debian machine for generating binary packages as it can generate both RPM and DEB correctly faster without having to disable prelinking and selinux like on Fedora/Redhat.

HOW TO RUN REGRESSION TESTS

Make sure software is built and installed according to "HOW TO BUILD FROM SOURCE"
Run the regression tests
```
cd ~/build/hypertable
make alltests
```

Note: there are two other high-level test targets: coretests (for core tests) and moretests (for more longer running tests, which is included in alltests.) These high-level test targets will (re)start test servers automatically when new servers are installed. There is also a low-level "test" target in root of the build tree and src (and src/lang/Components) and tests/integration directories that does NOT auto restart test servers.

HOW TO BUILD SOURCE CODE DOCUMENTATION TREE

Install the following tools:
- doxygen
- graphviz
Note: if you ran htbuild --install dev_env, these should already be installed
If you have doxygen installed on your system, then CMake should detect this and add a "doc" target to the make file. Building the source code documentation tree is just a matter of running the following commands:
```
cd ~/build/hypertable
make doc
```

To view the docs, load the following file into a web browser:

~/build/hypertable/doc/html/index.html for source code documentation and,
~/build/hypertable/gen-html/index.html for Thrift API reference.
~/build/hypertable/hqldoc/index.html for HQL reference.

The Thrift API doc can also be generated independently (without doxygen installed) by running the follow command in the build tree:

make thriftdoc

Similarly HQL doc can be generated independently (without doxygen etc.):

make hqldoc

hypertable's People

Contributors

Stargazers

Watchers

hypertable's Issues

RangeServer-row-overflow-1 failed on SSD

Ran tests on AWS instance with SSDs and the RangeServer-row-overflow-1 failed. Here's the output:

148/280 Testing: RangeServer-row-overflow-1
148/280 Test: RangeServer-row-overflow-1
Command: "/bin/env" "INSTALL_DIR=/opt/hypertable/0.9.8.5" "PYTHONPATH=/root/src/hypertable/src/py/ThriftClient:/root/src/hypertable/src/py/ThriftClient/gen-p
y" "/root/src/hypertable/tests/integration/row-overflow/run.sh"
Directory: /root/build/hypertable/release/tests/integration/row-overflow
"RangeServer-row-overflow-1" start time: Feb 27 03:43 EST
Output:
----------------------------------------------------------
======================================
Row overflow WRITE test (TableMutator)
======================================
WARNING Low file descriptor limit (1024)
FsBroker.local appears to be running (14501):
root 14501 14500 0 03:43 pts/0 00:00:00 /opt/hypertable/0.9.8.5/bin/htFsBrokerLocal --pidfile /opt/hypertable/0.9.8.5/run/FsBroker.local.pid --verbose --Hype
rtable.RangeServer.Range.SplitSize=2500K --Hypertable.RangeServer.AccessGroup.MaxMemory=400K --Hypertable.RangeServer.MaintenanceThreads=8 --Hypertable.Range
Server.Maintenance.Interval=10k --Hypertable.RangeServer.AccessGroup.CellCache.PageSize=5k
Killing RangeServer.pid 14848
Killing Master.pid 14667
Killing Hyperspace.pid 14567
Shutdown thrift broker complete
Shutdown hypertable master complete
Shutdown range server complete
Shutdown hyperspace complete
Removed /hypertable/servers in FS
Removed /hypertable/tables in FS
Cleared hyperspace
Shutdown FS Broker complete
WARNING Low file descriptor limit (1024)
FS Broker (local) Started
Hyperspace Started
Master Started
RangeServer Started

Welcome to the hypertable command interpreter.
For information about Hypertable, visit http://hypertable.com

Type 'help' for a list of commands, or 'help shell' for a
list of shell meta commands.


  Elapsed time:  0.00 s

  Elapsed time:  0.03 s
0
(1) 'row overflow' not found, exiting

Test time =  16.84 sec
----------------------------------------------------------
Test Failed.
"RangeServer-row-overflow-1" end time: Feb 27 03:43 EST
"RangeServer-row-overflow-1" time elapsed: 00:00:16
----------------------------------------------------------

Clang compile problem on FreeBSD

Hello,
When I try to compile on FreeBSD (10.x ) with clang compiler I got the following error

I have found the following after some search:
http://stackoverflow.com/questions/20390806/c-huge-errors-using-clang-to-compile-a-map-with-iterators-as-values

Hypertable version is the latest ( from git ). How can I fix it?

PS. I am trying to port current hypertable on Latest FBSD.

Thanks

Error:

/home/vampie/hypertable/src/cc/Hypertable/RangeServer/CellStoreBlockIndexArray.h:256:37: error: no viable
conversion from '__map_iterator<__tree_iterator<union std::__1::map<const char *, long, struct LtCstr,
struct Hypertable::StlAllocator<struct std::__1::pair<const char *, long>, struct
Hypertable::PageArenaAllocator<struct std::__1::pair<const char *, long>, class
Hypertable::PageArena<unsigned char, struct Hypertable::DefaultPageAllocator> > > >::__value_type,
class std::__1::__tree_node<union std::__1::map<const char *, long, struct LtCstr, struct
Hypertable::StlAllocator<struct std::__1::pair<const char *, long>, struct
Hypertable::PageArenaAllocator<struct std::__1::pair<const char *, long>, class
Hypertable::PageArena<unsigned char, struct Hypertable::DefaultPageAllocator> > > >::__value_type, void
*> *, [...]>>' to '__map_iterator<__tree_iterator<union std::__1::map<const char *, long, struct
LtCstr, class std::__1::allocator<struct std::__1::pair<const char *const, long> > >::__value_type,
class std::__1::__tree_node<union std::__1::map<const char *, long, struct LtCstr, class
std::__1::allocator<struct std::__1::pair<const char *const, long> > >::__value_type, void *> *,
[...]>>'
CstrToInt64MapT::iterator iter = split_row_data.find(last_row);
^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/c++/v1/map:499:24: note: candidate constructor (the implicit copy constructor) not viable: no
known conversion from 'iterator' (aka '__map_iterator') to 'const
std::__1::__map_iterator<std::__1::__tree_iterator<std::__1::map<const char *, long, LtCstr,
std::__1::allocator<std::__1::pair<const char *const, long> > >::__value_type,
std::__1::__tree_node<std::__1::map<const char *, long, LtCstr, std::__1::allocator<std::__1::pair<const
char *const, long> > >::__value_type, void *> *, long> > &' for 1st argument
class _LIBCPP_TYPE_VIS __map_iterator
^
/usr/include/c++/v1/map:499:24: note: candidate constructor (the implicit move constructor) not viable: no
known conversion from 'iterator' (aka '__map_iterator') to
'std::__1::__map_iterator<std::__1::__tree_iterator<std::__1::map<const char *, long, LtCstr,
std::__1::allocator<std::__1::pair<const char *const, long> > >::__value_type,
std::__1::__tree_node<std::__1::map<const char *, long, LtCstr, std::__1::allocator<std::__1::pair<const
char *const, long> > >::__value_type, void *> *, long> > &&' for 1st argument
class _LIBCPP_TYPE_VIS __map_iterator
^
/usr/include/c++/v1/map:523:5: note: candidate constructor not viable: no known conversion from 'iterator'
(aka '__map_iterator') to 'std::__1::__tree_iterator<std::__1::map<const char
*, long, LtCstr, std::__1::allocator<std::__1::pair<const char *const, long> > >::__value_type,
std::__1::__tree_node<std::__1::map<const char *, long, LtCstr, std::__1::allocator<std::__1::pair<const
char *const, long> > >::__value_type, void *> *, long>' for 1st argument
__map_iterator(_TreeIterator __i) _NOEXCEPT : _i(__i) {}
^
In file included from /home/vampie/hypertable/src/cc/Hypertable/RangeServer/AccessGroup.cc:34:
In file included from /home/vampie/hypertable/src/cc/Hypertable/RangeServer/CellStoreV7.h:38:
/usr/include/c++/v1/ext/hash_set:202:2: warning: Use of the header <ext/hash_set> is deprecated. Migrate to
<unordered_set> [-W#warnings]

warning Use of the header <ext/hash_set> is deprecated. Migrate to <unordered_set>

^
3 warnings and 1 error generated.
src/cc/Hypertable/RangeServer/CMakeFiles/HyperRanger.dir/build.make:54: recipe for target 'src/cc/Hypertable/RangeServer/CMakeFiles/HyperRanger.dir/AccessGroup.cc.o' failed
gmake[2]: *** [src/cc/Hypertable/RangeServer/CMakeFiles/HyperRanger.dir/AccessGroup.cc.o] Error 1
CMakeFiles/Makefile2:3500: recipe for target 'src/cc/Hypertable/RangeServer/CMakeFiles/HyperRanger.dir/all' failed
gmake[1]: *** [src/cc/Hypertable/RangeServer/CMakeFiles/HyperRanger.dir/all] Error 2
Makefile:146: recipe for target 'all' failed
gmake: *** [all] Error 2

No thrift package file

/bin/src-utils/htbuild trying to download non-existing file http://hypertable.org/pub/thrift-0.2.0-incubating.tar.gz

RangeServer crashing on shutdown

I've seen this happen a couple of times during the 50_load_ascending_keys system test. Here's the stack trace:

Thread 13 (Thread 0x7f87713ce700 (LWP 2169)):
#0  0x00000032dd80ef3d in nanosleep () from /lib64/libpthread.so.0
#1  0x00000000006ccbd8 in Hypertable::RangeServer::Request::Handler::Shutdown::run() ()
#2  0x000000000057a43c in Hypertable::ApplicationQueue::Worker::operator()() ()
#3  0x00007f879c29c082 in thread_proxy () from /opt/hypertable/doug/current/lib/libboost_thread.so.1.54.0
#4  0x00000032dd8079d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00000032dd4e89dd in clone () from /lib64/libc.so.6
...
Thread 1 (Thread 0x7f876bd9b700 (LWP 2328)):
#0  0x00000032dd432625 in raise () from /lib64/libc.so.6
#1  0x00000032dd433e05 in abort () from /lib64/libc.so.6
#2  0x000000373fa6007d in __gnu_cxx::__verbose_terminate_handler () at /usr/src/gcc-4.9.2/libstdc++-v3/libsupc++/vterminate.cc:95
#3  0x000000373fa5e0e6 in __cxxabiv1::__terminate (handler=) at /usr/src/gcc-4.9.2/libstdc++-v3/libsupc++/eh_terminate.cc:47
#4  0x000000373fa5e131 in std::terminate () at /usr/src/gcc-4.9.2/libstdc++-v3/libsupc++/eh_terminate.cc:57
#5  0x000000373fa5e348 in __cxxabiv1::__cxa_throw (obj=, tinfo=, dest=) at /usr/src/gcc-4.9.2/libstdc++-v3/libsupc++/eh_th\
row.cc:87
#6  0x000000373fab9cdf in std::__throw_system_error (__i=) at /usr/src/gcc-4.9.2/libstdc++-v3/src/c++11/functexcept.cc:127
#7  0x000000000074f220 in Hypertable::MetaLog::Writer::record_state(std::shared_ptr) ()
#8  0x00000000006a648a in Hypertable::Range::split_install_log() ()
#9  0x00000000006b3417 in Hypertable::Range::split() ()
#10 0x00000000005c5717 in Hypertable::MaintenanceQueue::Worker::operator()() ()
#11 0x00007f879c29c082 in thread_proxy () from /opt/hypertable/doug/current/lib/libboost_thread.so.1.54.0
#12 0x00000032dd8079d1 in start_thread () from /lib64/libpthread.so.0
#13 0x00000032dd4e89dd in clone () from /lib64/libc.so.6

hypertable C client compile error with glibc

compiling Client.thrift thrift gen works well yet with thrift 0.9.2 the c client code of generated thrift implementation have errors:

if ((ret = thrift_protocol_read_string (protocol, &_elem22, error)) < 0)
                      return 0;
                    xfer += ret;
                    g_ptr_array_add (*_elem21, _elem22);
                  }
                  if ((ret = thrift_protocol_read_list_end (protocol, error)) < 0)
                    return 0;
                  xfer += ret;

[OUTPUT]

In function ‘client_service_client_recv_scanner_get_cells_as_arrays’:
/home/ubuntu/Downloads/ht_thrf/gen-c_glib/client_service.c:8488:38: warning: passing argument 1 of ‘g_ptr_array_add’ from incompatible pointer type
g_ptr_array_add (_elem21, _elem22);

/usr/include/glib-2.0/glib/garray.h:168:12: note: expected ‘struct GPtrArray ’ but argument is of type ‘struct GPtrArray *’
void g_ptr_array_add (GPtrArray _array,
/home/ubuntu/Downloads/ht_thrf/gen-c_glib/client_service.c: In function ‘client_service_client_recv_next_cells_as_arrays’:
/home/ubuntu/Downloads/ht_thrf/gen-c_glib/client_service.c:8733:38: warning: passing argument 1 of ‘g_ptr_array_add’ from incompatible pointer type
g_ptr_array_add (_elem23, elem24);
^
In file included from /usr/include/glib-2.0/glib.h:31:0,
from /usr/include/glib-2.0/gobject/gbinding.h:28,
from /usr/include/glib-2.0/glib-object.h:23,
from /opt/hypertable/current/include/thrift/c_glib/transport/thrift_socket.h:23,
from :1:
/usr/include/glib-2.0/glib/garray.h:168:12: note: expected ‘struct GPtrArray *’ but argument is of type ‘struct GPtrArray *’
void g_ptr_array_add (GPtrArray *array,

as described Thrift Jira noted till now as bug
does Hypertable will make any amended to the thrift file ?

i report the bug to Thrift JIRA

Get rid of Common/Compat.h

We should verify that it is not needed and if so, get rid of it. It's non-standard and adds more typing work and things to remember when creating a .cc file. Also, if you forget to add it, the file will fail to compile which creates additional debugging work.

Consider having ht-start-all-servers.sh consult cluster.def

Encountered a situation where a sysadmin ran the ht-start-all-servers.sh script on all of the machines thinking that it would start the appropriate servers. Once cluster.def becomes standard, we should consider having ht-start-all-servers.sh read that file to figure out which services need to be started on the machine and only start those.

76 - SystemState (OTHER_FAULT) test failure

On Centos 32 release build:

1423790931 FATAL system_state_test : (/root/src/hypertable/src/cc/Hypertable/Master/tests/system_state_test.cc:233) No notification generated after auto_set(SystemVariable::READONLY, true)

FsBroker should be restartable

We should be able to restart the FsBroker while the entire Hypertable system is running.

issue783 test failure

issue783 test hung and eventually timed out:

        Start 189: Defects-issue783-offload1
189/278 Test #189: Defects-issue783-offload1 ................***Failed  1843.66 sec

The log shows this:

189/278 Testing: Defects-issue783-offload1
189/278 Test: Defects-issue783-offload1
Command: "/bin/env" "INSTALL_DIR=/opt/hypertable/doug/0.9.8.4" "/home/doug/src/hypertable/tests/integration/defects/issue783/run1.sh"
Directory: /home/doug/build/hypertable/debug/tests/integration/defects/issue783
"Defects-issue783-offload1" start time: Jan 16 18:56 PST
Output:
----------------------------------------------------------
=======================
Defect #783 - OFFLOAD 1
=======================
FsBroker OK
Shutdown thrift broker complete
Shutdown hyperspace complete
Shutdown range server complete
Shutdown hypertable master complete
Removed /hypertable/servers in FS
Removed /hypertable/tables in FS
Cleared hyperspace
Shutdown FS Broker complete
FsBroker OK
Hyperspace OK
Started Hypertable.Master

Welcome to the hypertable command interpreter.
For information about Hypertable, visit http://hypertable.com

Type 'help' for a list of commands, or 'help shell' for a
list of shell meta commands.


  Elapsed time:  0.00 s

  Elapsed time:  0.00 s

  Elapsed time:  2.83 s

  Elapsed time:  2.08 s

0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
***************************************************


        Elapsed time: 5.56 s
Total cells inserted: 100000
Throughput (cells/s): 17978.48
Total bytes inserted: 30700000
Throughput (bytes/s): 5519391.85

wait_for_file /opt/hypertable/doug/0.9.8.4/run/monitoring/rangeservers/rs1_stats_v0.rrd
-rw-rw-r-- 1 doug doug 3843960 Jan 16 18:57 /opt/hypertable/doug/0.9.8.4/run/monitoring/rangeservers/rs1_stats_v0.rrd
wait_for_file /opt/hypertable/doug/0.9.8.4/run/monitoring/rangeservers/rs2_stats_v0.rrd
-rw-rw-r-- 1 doug doug 3843960 Jan 16 18:57 /opt/hypertable/doug/0.9.8.4/run/monitoring/rangeservers/rs2_stats_v0.rrd
wait_for_file /opt/hypertable/doug/0.9.8.4/run/monitoring/rangeservers/rs3_stats_v0.rrd
-rw-rw-r-- 1 doug doug 3843960 Jan 16 18:57 /opt/hypertable/doug/0.9.8.4/run/monitoring/rangeservers/rs3_stats_v0.rrd

  Elapsed time:  0.00 s

  Elapsed time:  0.00 s

  Elapsed time:  0.00 s

  Elapsed time:  2.07 s
1421463424 WARN Hypertable.Master : (/home/doug/src/hypertable/src/cc/Hypertable/Master/Context.cc:345) RangeServer rs2: disk use 100% exceeds t
hreshold; will not assign ranges

  Elapsed time:  0.00 s

  Elapsed time:  0.00 s
1421465228 WARN hypertable : (/home/doug/src/hypertable/src/cc/AsyncComm/IOHandlerData.cc:599) Received response for non-pending event (id=17762
34,version=1,total_len=695)
1421465228 ERROR hypertable : dump_error_history (/home/doug/src/hypertable/src/cc/Hypertable/Lib/RangeLocator.h:154): Hypertable::Exception: Pr
oblem creating scanner on second-level METADATA (start row = 2:0002436) - HYPERTABLE request timeout
    at int Hypertable::RangeLocator::find(const Hypertable::TableIdentifier*, const char*, Hypertable::RangeLocationInfo*, Hypertable::Timer
&, bool) (/home/doug/src/hypertable/src/cc/Hypertable/Lib/RangeLocator.cc:444)
    at void Hypertable::Lib::RangeServer::Client::do_create_scanner(const Hypertable::CommAddress&, const Hypertable::TableIdentifier&, cons
t Hypertable::RangeSpec&, const Hypertable::Lib::ScanSpec&, Hypertable::ScanBlock&, int32_t) (/home/doug/src/hypertable/src/cc/Hypertable/Lib/Ra
ngeServer/Client.cc:282): RangeServer create_scanner() failure : Event: type=ERROR "HYPERTABLE request timeout" from=192.168.17.11:15871
1421465228 ERROR hypertable : retry_or_abort (/home/doug/src/hypertable/src/cc/Hypertable/Lib/IntervalScannerAsync.cc:405): Hypertable::Exceptio
n: Locating range for table 2 row='0002436' - HYPERTABLE request timeout
    at void Hypertable::IntervalScannerAsync::find_range_and_start_scan(const char*, bool) (/home/doug/src/hypertable/src/cc/Hypertable/Lib/
IntervalScannerAsync.cc:251)
    at void Hypertable::RangeLocator::find_loop(const Hypertable::TableIdentifier*, const char*, Hypertable::RangeLocationInfo*, Hypertable:
:Timer&, bool) (/home/doug/src/hypertable/src/cc/Hypertable/Lib/RangeLocator.cc:249): Locating range for table 2 row='0002436'
1421465228 ERROR hypertable : handle_error (/home/doug/src/hypertable/src/cc/Hypertable/Lib/TableScannerAsync.cc:496): Hypertable::Exception: Ev
ent: type=MESSAGE version=1 total_len=68 header_len=38 header_checksum=1581830720 flags=8 id=1776232 gid=0 timeout_ms=0 payload_checksum=0 comma
nd=2 from=192.168.17.11:15870 - RANGE SERVER range not found
Error: Event: type=MESSAGE version=1 total_len=68 header_len=38 header_checksum=1581830720 flags=8 id=1776232 gid=0 timeout_ms=0 payload_checksu
m=0 command=2 from=192.168.17.11:15870 - RANGE SERVER range not found

Notice the time difference between the WARN log line and the log line before it (1421465228-1421463424=1804). My guess is this is some sort of deadlock.

ht_ssh segfault on Amazon instance

It appears to get stuck in a loop, printing out the following:

[ht-new-infomart-master] ssh_connect: Socket connecting, now waiting for the callbacks to work
[ht-new-infomart-master] ssh_connect: Socket connecting, now waiting for the callbacks to work
[ht-new-infomart-master] ssh_connect: Socket connecting, now waiting for the callbacks to work
[ht-new-infomart-master] ssh_connect: Socket connecting, now waiting for the callbacks to work
[ht-new-infomart-master] ssh_connect: Socket connecting, now waiting for the callbacks to work
[ht-new-infomart-master] ssh_connect: Socket connecting, now waiting for the callbacks to work
[ht-new-infomart-master] ssh_connect: Socket connecting, now waiting for the callbacks to work

Then it terminates with a core dump here:

(gdb) where
#0  0x00007f0163f8217c in ssh_legacy_log_callback () from /opt/hypertable/0.9.8.6/lib/libssh.so.4
#1  0x00007f0163f908dc in ssh_log_function () from /opt/hypertable/0.9.8.6/lib/libssh.so.4
#2  0x00007f0163f90abe in _ssh_log () from /opt/hypertable/0.9.8.6/lib/libssh.so.4
#3  0x00007f0163f88443 in ssh_connect () from /opt/hypertable/0.9.8.6/lib/libssh.so.4
#4  0x0000000000435284 in Hypertable::SshSocketHandler::handle(int, int) ()
#5  0x0000000000488d9a in Hypertable::IOHandlerRaw::handle_event(epoll_event*, long) ()
#6  0x0000000000498546 in Hypertable::ReactorRunner::operator()() ()
#7  0x00007f0162bc4082 in thread_proxy () from /opt/hypertable/0.9.8.6/lib/libboost_thread.so.1.54.0
#8  0x00007f0161b2bf18 in start_thread () from /lib64/libpthread.so.0
#9  0x00007f0160984b9d in clone () from /lib64/libc.so.6

Implement Range File locality on HDFS

Support for specifying a set of favored nodes was recently added to HDFS. See https://issues.apache.org/jira/browse/HDFS-2576.

MasterOperation-DropTable test failure

Looks like it is executing the parent and subop simultaneously. Here's the output:

1427919555 INFO op_test_driver : (/root/src/hypertable/src/cc/Hypertable/Master/tests/op_test_driver.cc:225) Adding operation DropTable /drop_table_test id=12
 state=SCAN_METADATA
1427919555 INFO op_test_driver : (/root/src/hypertable/src/cc/Hypertable/Master/tests/op_test_driver.cc:225) Adding operation DropTable /^^drop_table_test id=
20 state=INITIAL
1427919555 INFO op_test_driver : (/root/src/hypertable/src/cc/Hypertable/Master/OperationDropTable.cc:95) Entering 0x8f24000 DropTable-20(/^^drop_table_test, 
if_exists=false, parts=PRIMARY) state=INITIAL
1427919555 INFO op_test_driver : (/root/src/hypertable/src/cc/Hypertable/Master/OperationDropTable.cc:114) DropTable(/^^drop_table_test) dependencies: 
1427919555 INFO op_test_driver : (/root/src/hypertable/src/cc/Hypertable/Master/OperationDropTable.cc:115) DropTable(/^^drop_table_test) obstructions: "Operat
ionDropTable subop OperationDropTable 20" 
1427919555 INFO op_test_driver : (/root/src/hypertable/src/cc/Hypertable/Master/OperationDropTable.cc:269) Leaving DropTable-20(/^^drop_table_test) state=SCAN
_METADATA
1427919555 INFO op_test_driver : (/root/src/hypertable/src/cc/Hypertable/Master/OperationDropTable.cc:95) Entering 0x8f24000 DropTable-20(/^^drop_table_test, 
if_exists=false, parts=PRIMARY) state=SCAN_METADATA
1427919555 INFO op_test_driver : (/root/src/hypertable/src/cc/Hypertable/Master/OperationDropTable.cc:114) DropTable(/^^drop_table_test) dependencies: "9 move
 range" "METADATA" 
1427919555 INFO op_test_driver : (/root/src/hypertable/src/cc/Hypertable/Master/OperationDropTable.cc:115) DropTable(/^^drop_table_test) obstructions: "Operat
ionDropTable subop OperationDropTable 20" 
1427919555 INFO op_test_driver : (/root/src/hypertable/src/cc/Hypertable/Master/tests/op_test_driver.cc:225) Adding operation DropTable /drop_table_test id=12
 state=SCAN_METADATA
1427919555 INFO op_test_driver : (/root/src/hypertable/src/cc/Hypertable/Master/tests/op_test_driver.cc:225) Adding operation DropTable /^^drop_table_test id=
20 state=ISSUE_REQUESTS
1427919555 INFO op_test_driver : (/root/src/hypertable/src/cc/Hypertable/Master/OperationDropTable.cc:95) Entering 0x8f24300 DropTable-20(/^^drop_table_test, 
if_exists=false, parts=PRIMARY) state=ISSUE_REQUESTS
1427919555 INFO op_test_driver : (/root/src/hypertable/src/cc/Hypertable/Master/OperationDropTable.cc:114) DropTable(/^^drop_table_test) dependencies: "9 move
 range" "METADATA" "rs1" "rs2" 
1427919555 INFO op_test_driver : (/root/src/hypertable/src/cc/Hypertable/Master/OperationDropTable.cc:115) DropTable(/^^drop_table_test) obstructions: "Operat
ionDropTable subop OperationDropTable 20" 
1427919555 INFO op_test_driver : (/root/src/hypertable/src/cc/Hypertable/Master/OperationDropTable.cc:95) Entering 0x8f24300 DropTable-12(/drop_table_test, if
_exists=true, parts=ALL) state=SCAN_METADATA
1427919555 INFO op_test_driver : (/root/src/hypertable/src/cc/Hypertable/Master/OperationDropTable.cc:114) DropTable(/drop_table_test) dependencies: "7 move r
ange" "METADATA" "OperationDropTable subop OperationDropTable 20" 
1427919555 INFO op_test_driver : (/root/src/hypertable/src/cc/Hypertable/Master/OperationDropTable.cc:115) DropTable(/drop_table_test) obstructions: 
digraph G {
0[label="DropTable /drop_table_test"];
1[label="DropTable /^^drop_table_test"];
0->1 ;
}

1427919555 FATAL op_test_driver : (/root/src/hypertable/src/cc/Hypertable/Master/Operation.h:500) failed expectation: m_state == OperationState::COMPLETE

The above output is instrumented with the OperationProcessor address after "Entering". Also, the OperationProcessor dependency graph is displayed.

Hang in RangeServer-failover-master-23 test

The Master failure wasn't induced. The following yielded no output.

$ fgrep CRASH master.output.23

It looks like rs1 never came up:

[doug@test06 archive-Fri-Feb-27-00:10:09-PST-2015]$ cat rangeserver.rs1.output.23 
1425023937 INFO htRangeServer : (/home/doug/src/hypertable/src/cc/Common/Config.cc:632) Initializing htRangeServer (Hypertable 0.9.8.5 (v0.9.8.5-6-gf7fa3fe))...
CPU cores count=12
FsBroker.DisableFileRemoval=true
FsBroker.Host=localhost
FsBroker.Local.Root=fs/local
HdfsBroker.Workers=20
HdfsBroker.fs.default.name=hdfs://localhost:9000
Hyperspace.Replica.Dir=hyperspace
Hyperspace.Replica.Host=[localhost]
Hyperspace.Replica.Workers=20
Hypertable.CommitLog.Compressor=none
Hypertable.Failover.GracePeriod=30000
Hypertable.Failover.Quorum.Percentage=40
Hypertable.Failover.Timeout=10000
Hypertable.LoadBalancer.BalanceDelay.NewServer=10000
Hypertable.Master.Host=localhost
Hypertable.Master.Workers=20
Hypertable.Monitoring.Interval=5000
Hypertable.RangeServer.CommitLog.Compressor=none
Hypertable.RangeServer.Maintenance.Interval=20
Hypertable.RangeServer.Port=15870
Hypertable.RangeServer.ProxyName=rs1
Hypertable.RangeServer.Range.SplitSize=1000000
Hypertable.RangeServer.Reactors=12
Hypertable.RangeServer.Timer.Interval=20
Hypertable.RangeServer.Workers=20
Hypertable.Request.Timeout=45000
Hypertable.Verbose=true
config=/home/doug/src/hypertable/tests/integration/rangeserver-failover-master/test.cfg
fs-host=localhost
hs-host=[localhost]
master-host=localhost
pidfile=/opt/hypertable/doug/0.9.8.5/run/RangeServer.rs1.pid
port=15870
reactors=12
timeout=45000
verbose=true
workers=20
1425023940 INFO htRangeServer : (/home/doug/src/hypertable/src/cc/Hypertable/RangeServer/HyperspaceSessionHandler.cc:32) Hyperspace session state change:  SAFE
1425023940 ERROR htRangeServer : main (/home/doug/src/hypertable/src/cc/Hypertable/RangeServer/main.cc:120): Hypertable::Exception: Hyperspace 'open' error, name=/hypertable/master flags=0x3 events=0x0 - HYPERSPACE file not found
    at uint64_t Hyperspace::Session::open(Hyperspace::ClientHandleStatePtr&, Hypertable::CommBufPtr&, Hypertable::Timer*) (/home/doug/src/hypertable/src/cc/Hyperspace/Session.cc:182)
1425023942 WARN htRangeServer : (/home/doug/src/hypertable/src/cc/Hyperspace/ClientConnectionHandler.cc:73) Event: type=DISCONNECT from=127.0.0.1:15861

Bug in ht cluster install_package

When run, it generates this error:

# ht cluster install_package
1425327558 ERROR ht_cluster : (/root/src/hypertable/src/cc/Common/FileUtils.cc:291) open("/opt/hypertable/0.9.8.6.20522d3/conf/core.tasks") failure - No such file or directory

Looks like it references core.tasks from the installation that has not yet been installed. The fix is to change the following line in cluster.def:

include: "$INSTALL_PREFIX/$HYPERTABLE_VERSION/conf/core.tasks"

include: "core.tasks"

Replace foreach_ht with range for loop

We should get rid of the foreach_ht macro and replace it with the C++11 range for loop.

HostSpecification problem

$ ht expand_hostspec "test[00-09] + test[0-9]"
test00 test01 test02 test03 test04 test05 test06 test07 test08 test09

This should be fixed in the Java HostSpecification as well.

Should be able to drop empty access group

Master operations should not clear dependencies after creating sub-ops

When a sub-operation is created, a dependency is created: parent -> subop. Maybe we should create a method Operation::clear_dependencies() which removes all dependencies except the parent -> subop ones. Same goes for obstructions.

metalog_dump should have a --no-timestamps option

This would allow us to use it in tests without having to pipe the output through a complicated perl expression to strip the timestamps.

Add "Creating a thrift client" section to Perl documentation

Looks like we missed this one for Perl.

Deadlock in Defects-issue783-offload1 test

The scan is stuck because it can't find the range:

RANGE SERVER range not found - (a) 2[0098861..??]

This range is in the process of being loaded by rs3, but it can't update the METADATA table. Here's the stack trace:

Thread 77 (Thread 0x7f726c809700 (LWP 1474)):
#0  0x000000399d4df1b3 in poll () from /lib64/libc.so.6
#1  0x00000000007a5485 in Hypertable::TableMutatorAsyncScatterBuffer::create_redo_buffer(unsigned int) ()
#2  0x000000000079ab3f in Hypertable::TableMutatorAsync::buffer_finish(unsigned int, int, bool) ()
#3  0x00000000007a2bbd in Hypertable::TableMutatorAsyncScatterBuffer::finish() ()
#4  0x00000000008337b5 in Hypertable::TableMutatorAsyncHandler::run() ()
#5  0x0000000000794b36 in Hypertable::TableMutator::wait_for_flush_completion(Hypertable::TableMutatorAsync*) ()
#6  0x0000000000795843 in Hypertable::TableMutator::flush() ()
#7  0x0000000000592565 in Hypertable::Apps::RangeServer::load_range(Hypertable::ResponseCallback*, Hypertable::TableIdentifier const&, Hypertable::RangeSpec const&, Hypertable::RangeState const&, bool) ()
#8  0x00000000006a9d9b in Hypertable::RangeServer::Request::Handler::LoadRange::run() ()
#9  0x0000000000566c35 in Hypertable::ApplicationQueue::Worker::operator()() ()
#10 0x00007f72819dc082 in thread_proxy () from /opt/hypertable/doug/0.9.8.5/lib/libboost_thread.so.1.54.0
#11 0x000000399d8079d1 in start_thread () from /lib64/libpthread.so.0
#12 0x000000399d4e89dd in clone () from /lib64/libc.so.6

Over in rs1, it appears that the METADATA range 0/0[0/0:??..??] is stuck at the end of run_compaction blocked on the LiveFileTracker mutex. Here's a snippet of the rangeserver dump:

RANGE 0/0[0/0:??..??]
...
state=3

State 3 is RELINQUISH_LOG_INSTALLED and here's the stack trace from rs1:

Thread 34 (Thread 0x7f1964493700 (LWP 1575)):
#0  0x000000399d4df1b3 in poll () from /lib64/libc.so.6
#1  0x00000000007a5485 in Hypertable::TableMutatorAsyncScatterBuffer::create_redo_buffer(unsigned int) ()
#2  0x000000000079ab3f in Hypertable::TableMutatorAsync::buffer_finish(unsigned int, int, bool) ()
#3  0x00000000007a2bbd in Hypertable::TableMutatorAsyncScatterBuffer::finish() ()
#4  0x00000000008337b5 in Hypertable::TableMutatorAsyncHandler::run() ()
#5  0x0000000000794b36 in Hypertable::TableMutator::wait_for_flush_completion(Hypertable::TableMutatorAsync*) ()
#6  0x0000000000795843 in Hypertable::TableMutator::flush() ()
#7  0x000000000067cbdc in Hypertable::MetadataNormal::write_files(std::basic_string, std::allocator > const&, std::basic_string, std::allocator > const&, long, unsigned int) ()
#8  0x0000000000656ecd in Hypertable::LiveFileTracker::update_files_column() ()
#9  0x00000000005dcf60 in Hypertable::AccessGroup::run_compaction(int, Hypertable::AccessGroup::Hints*) ()
#10 0x00000000006964c5 in Hypertable::Range::relinquish_compact() ()
#11 0x0000000000696e18 in Hypertable::Range::relinquish() ()
#12 0x00000000005b01df in Hypertable::MaintenanceQueue::Worker::operator()() ()
#13 0x00007f19950ac082 in thread_proxy () from /opt/hypertable/doug/0.9.8.5/lib/libboost_thread.so.1.54.0
#14 0x000000399d8079d1 in start_thread () from /lib64/libpthread.so.0
#15 0x000000399d4e89dd in clone () from /lib64/libc.so.6

I've also seen it here:

Thread 34 (Thread 0x7f1964493700 (LWP 1575)):
#0  0x000000399d80e264 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x000000399d809508 in _L_lock_854 () from /lib64/libpthread.so.0
#2  0x000000399d8093d7 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x0000000000566028 in boost::mutex::lock() ()
#4  0x0000000000656cc4 in Hypertable::LiveFileTracker::update_files_column() ()
#5  0x00000000005dcf60 in Hypertable::AccessGroup::run_compaction(int, Hypertable::AccessGroup::Hints*) ()
#6  0x00000000006964c5 in Hypertable::Range::relinquish_compact() ()
#7  0x0000000000696e18 in Hypertable::Range::relinquish() ()
#8  0x00000000005b01df in Hypertable::MaintenanceQueue::Worker::operator()() ()
#9  0x00007f19950ac082 in thread_proxy () from /opt/hypertable/doug/0.9.8.5/lib/libboost_thread.so.1.54.0
#10 0x000000399d8079d1 in start_thread () from /lib64/libpthread.so.0
#11 0x000000399d4e89dd in clone () from /lib64/libc.so.6

I also see a number of threads (in rs1) stuck here:

Thread 87 (Thread 0x7f19862e3700 (LWP 1418)):
#0  0x000000399d80e264 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x000000399d809508 in _L_lock_854 () from /lib64/libpthread.so.0
#2  0x000000399d8093d7 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00000000005660b8 in boost::unique_lock::lock() ()
#4  0x000000000065669f in Hypertable::LiveFileTracker::remove_references(std::vector, std::allocator >, 
std::allocator, std::allocator > > > const&) ()
#5  0x00000000005d6c40 in Hypertable::AccessGroup::release_files(std::vector, std::allocator >, std::all
ocator, std::allocator > > > const&) ()
#6  0x000000000066c057 in Hypertable::MergeScannerAccessGroup::~MergeScannerAccessGroup() ()
#7  0x000000000066c3b9 in Hypertable::MergeScannerAccessGroup::~MergeScannerAccessGroup() ()
#8  0x0000000000671556 in Hypertable::MergeScannerRange::~MergeScannerRange() ()
#9  0x0000000000563325 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() ()
#10 0x000000000058f898 in Hypertable::Apps::RangeServer::create_scanner(Hypertable::RangeServer::Response::Callback::CreateScanner*, Hypertable::TableIdentif
ier const&, Hypertable::RangeSpec const&, Hypertable::Lib::ScanSpec const&, Hypertable::QueryCache::Key*) ()
#11 0x00000000006a0bc5 in Hypertable::RangeServer::Request::Handler::CreateScanner::run() ()
#12 0x0000000000566c35 in Hypertable::ApplicationQueue::Worker::operator()() ()
#13 0x00007f19950ac082 in thread_proxy () from /opt/hypertable/doug/0.9.8.5/lib/libboost_thread.so.1.54.0
#14 0x000000399d8079d1 in start_thread () from /lib64/libpthread.so.0
#15 0x000000399d4e89dd in clone () from /lib64/libc.so.6

and here ...

Thread 70 (Thread 0x7f197b8d2700 (LWP 1456)):
#0  0x000000399d80e264 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x000000399d809508 in _L_lock_854 () from /lib64/libpthread.so.0
#2  0x000000399d8093d7 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00000000005660b8 in boost::unique_lock::lock() ()
#4  0x0000000000656a36 in Hypertable::LiveFileTracker::add_references(std::vector, std::allocator >, std
::allocator, std::allocator > > > const&) ()
#5  0x00000000005db064 in Hypertable::AccessGroup::create_scanner(boost::intrusive_ptr&) ()
#6  0x000000000068b8ef in Hypertable::Range::create_scanner(boost::intrusive_ptr&, std::shared_ptr&) 
()
#7  0x000000000058f713 in Hypertable::Apps::RangeServer::create_scanner(Hypertable::RangeServer::Response::Callback::CreateScanner*, Hypertable::TableIdentif
ier const&, Hypertable::RangeSpec const&, Hypertable::Lib::ScanSpec const&, Hypertable::QueryCache::Key*) ()
#8  0x00000000006a0bc5 in Hypertable::RangeServer::Request::Handler::CreateScanner::run() ()
#9  0x0000000000566c35 in Hypertable::ApplicationQueue::Worker::operator()() ()
#10 0x00007f19950ac082 in thread_proxy () from /opt/hypertable/doug/0.9.8.5/lib/libboost_thread.so.1.54.0
#11 0x000000399d8079d1 in start_thread () from /lib64/libpthread.so.0
#12 0x000000399d4e89dd in clone () from /lib64/libc.so.6

split-failover-1 test failure

Load generator is hanging here:

Thread 1 (Thread 0x7f9ec27ad7e0 (LWP 1352)):
#0  0x000000399d80b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000000000d2b301 in void boost::condition_variable_any::wait >(boost::unique_lock&) ()
#2  0x0000000000e92ac6 in Hypertable::TableMutatorQueue::wait_for_buffer(boost::unique_lock&, Hypertable::ApplicationHandler**) ()
#3  0x0000000000e90c0b in Hypertable::TableMutator::wait_for_flush_completion(Hypertable::TableMutatorAsync*) ()
#4  0x0000000000e90843 in Hypertable::TableMutator::auto_flush() ()
#5  0x0000000000e90287 in Hypertable::TableMutator::set_cells(__gnu_cxx::__normal_iterator > > >, __gnu_cxx::__normal_iterator > > >) ()
#6  0x0000000000e92efd in Hypertable::TableMutator::set_cells(std::vector > > const&) ()
#7  0x0000000000d1fc3f in LoadClient::set_cells(std::vector > > const&) ()
#8  0x0000000000cf9285 in generate_update_load(boost::intrusive_ptr&, std::basic_string, std::allocator >&, bool, bool, unsigned long, unsigned long, bool, std::basic_string, std::allocator >&, int, bool) ()
#9  0x0000000000cf68a0 in main ()

Compaction-exception-relinquish-2 test failure

Saw this test fail on test01 with the following output:

215/280 Testing: Compaction-exception-relinquish-2
215/280 Test: Compaction-exception-relinquish-2
Command: "/bin/env" "TEST=6" "INSTALL_DIR=/opt/hypertable/doug/0.9.8.6" "PYTHONPATH=/home/doug/src/hypertable/src/py/ThriftClient:/home/doug/src/hypertable/src/py/ThriftClient/gen-py" "bash" "-x" "/home/doug/src/hypertable/tests/integration/compaction/run.sh"
Directory: /home/doug/build/hypertable/debug/tests/integration/compaction
"Compaction-exception-relinquish-2" start time: Mar 10 01:02 PDT
Output:
----------------------------------------------------------
+ HT_HOME=/opt/hypertable/doug/0.9.8.6
+ HYPERTABLE_HOME=/opt/hypertable/doug/0.9.8.6
+ HT_SHELL='/opt/hypertable/doug/0.9.8.6/bin/ht shell'
++ dirname /home/doug/src/hypertable/tests/integration/compaction/run.sh
+ SCRIPT_DIR=/home/doug/src/hypertable/tests/integration/compaction
+ MAX_KEYS=500000
+ RS1_PIDFILE=/opt/hypertable/doug/0.9.8.6/run/RangeServer.rs1.pid
+ RS2_PIDFILE=/opt/hypertable/doug/0.9.8.6/run/RangeServer.rs2.pid
+ RS1_PORT=26001
+ RS2_PORT=26002
++ pwd
+ RUN_DIR=/home/doug/build/hypertable/debug/tests/integration/compaction
+ . /opt/hypertable/doug/0.9.8.6/bin/ht-env.sh
++ ulimit -c unlimited
++ ulimit -v unlimited
++ '[' e == e ']'
++ RUNTIME_ROOT=/opt/hypertable/doug/0.9.8.6
++ '[' /opt/hypertable/doug/0.9.8.6 ']'
++ '[' -d /opt/hypertable/doug/0.9.8.6/run ']'
++ '[' -d /opt/hypertable/doug/0.9.8.6/log ']'
++ export LD_LIBRARY_PATH=/opt/hypertable/doug/0.9.8.6/lib:/opt/hypertable/doug/0.9.8.6/lib:
++ LD_LIBRARY_PATH=/opt/hypertable/doug/0.9.8.6/lib:/opt/hypertable/doug/0.9.8.6/lib:
++ export DYLD_LIBRARY_PATH=/opt/hypertable/doug/0.9.8.6/lib:/opt/hypertable/doug/0.9.8.6/lib::/opt/hypertable/doug/0.9.8.6/lib::
++ DYLD_LIBRARY_PATH=/opt/hypertable/doug/0.9.8.6/lib:/opt/hypertable/doug/0.9.8.6/lib::/opt/hypertable/doug/0.9.8.6/lib::
+ . /home/doug/src/hypertable/tests/integration/compaction/utilities.sh
++ DIGEST='openssl dgst -md5'
++ MASTER_LOG=/opt/hypertable/doug/0.9.8.6/log/Master.log
++ MASTER_PIDFILE=/opt/hypertable/doug/0.9.8.6/run/Master.pid
++ ROW_SEED=1
+ '[' 6 == 1 ']'
+ rm -f report.txt
+ gen_test_data
+ '[' '!' -s golden_dump.500000.md5 ']'
+ let j=1
+ '[' 6 == 1 ']'
+ let j+=1
+ '[' 6 == 2 ']'
+ let j+=1
+ '[' 6 == 3 ']'
+ let j+=1
+ '[' 6 == 4 ']'
+ let j+=1
+ '[' 6 == 5 ']'
+ let j+=1
+ '[' 6 == 6 ']'
+ run_test compact-relinquish-2:throw:0 ''
+ local INDUCER1_ARG=
+ local INDUCER2_ARG=
+ local MANUAL=false
+ local RELINQUISH=false
+ local MANUAL_COMPACT_SERVER_PORT
+ test -n compact-relinquish-2:throw:0
+ INDUCER1_ARG=--induce-failure=compact-relinquish-2:throw:0
+ [[ compact-relinquish-2:throw:0 == *manual* ]]
+ [[ compact-relinquish-2:throw:0 == *relinquish* ]]
+ RELINQUISH=true
+ MANUAL_COMPACT_SERVER_PORT=26001
+ shift
+ test -n ''
+ shift
+ /opt/hypertable/doug/0.9.8.6/bin/ht-start-test-servers.sh --no-rangeserver --no-master --no-thriftbroker --clear --FsBroker.DisableFileRemoval=true
FS Broker (local) Started
Shutdown thrift broker complete
Shutdown hypertable master complete
Shutdown range server complete
Shutdown hyperspace complete
Removed /hypertable/servers in FS
Removed /hypertable/tables in FS
Cleared hyperspace
Shutdown FS Broker complete
FS Broker (local) Started
Hyperspace Started
+ rm -f rangeserver.rs1.compaction-exception-6.output
+ rm -f rangeserver.rs2.compaction-exception-6.output
+ CONFIG='--Hypertable.RangeServer.Maintenance.Interval=100            --Hypertable.Master.Split.SoftLimitEnabled=false            --Hypertable.Failover.Quorum.Percentage=30            --Hypertable.RangeServer.Range.SplitSize=1M'
+ start_master_and_rangeservers --induce-failure=compact-relinquish-2:throw:0 '' '--Hypertable.RangeServer.Maintenance.Interval=100            --Hypertable.Master.Split.SoftLimitEnabled=false            --Hypertable.Failover.Quorum.Percentage=30            --Hypertable.RangeServer.Range.SplitSize=1M'
+ local INDUCER1_ARG=--induce-failure=compact-relinquish-2:throw:0
+ shift
+ local INDUCER2_ARG=
+ shift
+ local 'CONFIG=--Hypertable.RangeServer.Maintenance.Interval=100            --Hypertable.Master.Split.SoftLimitEnabled=false            --Hypertable.Failover.Quorum.Percentage=30            --Hypertable.RangeServer.Range.SplitSize=1M'
+ shift
+ /opt/hypertable/doug/0.9.8.6/bin/ht-start-master.sh --Hypertable.RangeServer.Maintenance.Interval=100 --Hypertable.Master.Split.SoftLimitEnabled=false --Hypertable.Failover.Quorum.Percentage=30 --Hypertable.RangeServer.Range.SplitSize=1M
Master Started
+ '[' true == true ']'
+ /opt/hypertable/doug/0.9.8.6/bin/ht-start-thriftbroker.sh
+ /opt/hypertable/doug/0.9.8.6/bin/ht RangeServer --verbose --pidfile=/opt/hypertable/doug/0.9.8.6/run/RangeServer.rs1.pid --Hypertable.RangeServer.ProxyName=rs1 --Hypertable.RangeServer.Port=26001 --induce-failure=compact-relinquish-2:throw:0 --Hypertable.RangeServer.Maintenance.Interval=100 --Hypertable.Master.Split.SoftLimitEnabled=false --Hypertable.Failover.Quorum.Percentage=30 --Hypertable.RangeServer.Range.SplitSize=1M
+ /opt/hypertable/doug/0.9.8.6/bin/ht RangeServer --verbose --pidfile=/opt/hypertable/doug/0.9.8.6/run/RangeServer.rs2.pid --Hypertable.RangeServer.ProxyName=rs2 --Hypertable.RangeServer.Port=26002 --Hypertable.RangeServer.Maintenance.Interval=100 --Hypertable.Master.Split.SoftLimitEnabled=false --Hypertable.Failover.Quorum.Percentage=30 --Hypertable.RangeServer.Range.SplitSize=1M
ThriftBroker Started
+ /opt/hypertable/doug/0.9.8.6/bin/ht shell --no-prompt

Welcome to the hypertable command interpreter.
For information about Hypertable, visit http://hypertable.com

Type 'help' for a list of commands, or 'help shell' for a
list of shell meta commands.


  Elapsed time:  4.17 s

  Elapsed time:  1.89 s
+ /opt/hypertable/doug/0.9.8.6/bin/ht load_generator --spec-file=/home/doug/src/hypertable/tests/integration/compaction/data.spec --max-keys=500000 --row-seed=1 --table=LoadTest --Hypertable.Mutator.ScatterBuffer.FlushLimit.PerServer=2M --Hypertable.Mutator.FlushDelay=500 update

0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
***************************************************


        Elapsed time: 82.23 s
Total cells inserted: 500000
Throughput (cells/s): 6080.25
Total bytes inserted: 110000000
Throughput (bytes/s): 1337655.96

+ '[' 0 '!=' 0 ']'
+ '[' true == true ']'
+ let j=3
+ '[' 3 -gt 0 ']'
++ /home/doug/src/hypertable/tests/integration/compaction/generate_range_move.py rs1
+ HQL_COMMAND='balance ("2"["00000000004733620960".."00000000004808000718"], "rs1", "rs2") duration=5;'
+ echo balance '("2"["00000000004733620960".."00000000004808000718"],' '"rs1",' '"rs2")' 'duration=5;'
balance ("2"["00000000004733620960".."00000000004808000718"], "rs1", "rs2") duration=5;
+ /opt/hypertable/doug/0.9.8.6/bin/ht shell --batch --Hypertable.Request.Timeout=20000
+ echo balance '("2"["00000000004733620960".."00000000004808000718"],' '"rs1",' '"rs2")' 'duration=5;'
Hypertable CRITICAL - MASTER operation in progress (Client operation balance failed)
+ '[' 2 -ne 0 ']'
+ stop_master_and_rangeservers
+ echo 'shutdown;quit;'
+ /opt/hypertable/doug/0.9.8.6/bin/ht master --batch
+ wait_for_server_shutdown master master
+ wait_for_server shutdown master master
+ become=shutdown
+ shift
+ server=master
+ shift
+ server_desc=master
+ shift
+ max_retries=40
+ report_interval=5
++ /opt/hypertable/doug/0.9.8.6/bin/ht serverup --display-address=true master
+ address=localhost:15864
+ '[' localhost:15864 ']'
+ address=' (localhost:15864)'
+ check_server master
+ /opt/hypertable/doug/0.9.8.6/bin/ht serverup --silent master
+ ret=1
+ retries=0
+ should_wait 1 shutdown
+ ret=1
+ become=shutdown
+ case $become in
+ return 1
+ should_wait 1 shutdown
+ ret=1
+ become=shutdown
+ case $become in
+ return 1
+ show_success master shutdown
+ server_desc=master
+ become=shutdown
+ case $become in
+ echo 'Shutdown master complete'
Shutdown master complete
+ return 0
+ '[' 0 '!=' 0 ']'
+ echo 'shutdown; quit;'
+ /opt/hypertable/doug/0.9.8.6/bin/ht rangeserver localhost:26001

Welcome to the rangeserver command interpreter.
For information about Hypertable, visit http://hypertable.com

Type 'help' for a list of commands, or 'help shell' for a
list of shell meta commands.

++ cat /opt/hypertable/doug/0.9.8.6/run/RangeServer.rs1.pid
+ kill -9 27230
/home/doug/src/hypertable/tests/integration/compaction/run.sh: line 50: kill: (27230) - No such process
+ rm -f /opt/hypertable/doug/0.9.8.6/run/RangeServer.rs1.pid
+ echo 'shutdown; quit;'
+ /opt/hypertable/doug/0.9.8.6/bin/ht rangeserver localhost:26002

Welcome to the rangeserver command interpreter.
For information about Hypertable, visit http://hypertable.com

Type 'help' for a list of commands, or 'help shell' for a
list of shell meta commands.

++ cat /opt/hypertable/doug/0.9.8.6/run/RangeServer.rs2.pid
+ kill -9 27231
+ rm -f /opt/hypertable/doug/0.9.8.6/run/RangeServer.rs2.pid
+ /opt/hypertable/doug/0.9.8.6/bin/ht-stop-servers.sh
Shutdown ThriftBroker complete
Shutdown Master complete
Shutdown RangeServer complete
Shutdown FS Broker complete
Killing Hyperspace.pid 26955
Shutdown Hyperspace complete
Shutdown thrift broker complete
Shutdown hyperspace complete
Shutdown hypertable master complete
+ exit 1

Test time = 125.54 sec
----------------------------------------------------------
Test Failed.
"Compaction-exception-relinquish-2" end time: Mar 10 01:04 PDT
"Compaction-exception-relinquish-2" time elapsed: 00:02:05
----------------------------------------------------------

Row keys with non UTF-8 characters cause MapReduce to fail

Hi Doug. Could you please look at the error we got from our mapreduce task? What's wrong with the ScanSpec here:
ScanSpec(row_intervals:[RowInterval(start_row:hmh:Ldm3AQ:\xd0\x91\xd1\x83\xd0\xbb\xd0\xb0\xd0\xbd\xd0\xbe\xd0\xb2 \xd0\x9c\xd0\xb0\xd0\xba\xd1\x81\xd0\xb8\xd0\xbc :14072706, start_inclusive:false, end_row:hmh:Ldm3AQ:\xd0\x93\xd0\xbe\xd1\x80\xd0\xb1\xd0\xbe\xd0\xbb\xd1\x8c\xd0\xbd\xd0\xb8\xd1\x86\xd0\xb0 \xe2\x84\x963 \xd0\xb3. \xd0\x97\xd0\xbb\xd0\xb0\xd1\x82\xd0\xbe\xd1\x83\xd1\x81\xd1\x82 \xd0\xa2\xd0\xb8\xd1\x85\xd0\xbe\xd0\xbd\xd1\x8e\xd0\xba \xd0\xa2.\xd0\x90. :14040912, end_inclusive:true)])
ScanSpec()
ScanSpec(row_intervals:[RowInterval(start_row:hmh:Ldm3AQ:пїЅпїЅпїЅпїЅпїЅпїЅпїЅ пїЅпїЅпїЅпїЅ пїЅпїЅпїЅпїЅпїЅпїЅпїЅпїЅпїЅпїЅпїЅ:13080714, start_inclusive:false, end_row:hmh:Ldm3AQ:пїЅпїЅпїЅпїЅпїЅпїЅ пїЅпїЅпїЅпїЅпїЅпїЅпїЅ.пїЅпїЅ:13072119, end_inclusive:true)])
ClientException(code:30, message:HYPERTABLE bad scan specification - scanner=109001632)
at org.hypertable.thriftgen.ClientService$scanner_get_cells_result$scanner_get_cells_resultStandardScheme.read(ClientService.java:49542)

Defects-issue783-load2 test failure

Here's the test output:

196/280 Testing: Defects-issue783-load2
196/280 Test: Defects-issue783-load2
Command: "/bin/env" "INSTALL_DIR=/opt/hypertable/doug/0.9.8.5" "/home/doug/src/hypertable/tests/integration/defects/issue783/run6.sh"
Directory: /home/doug/build/hypertable/release/tests/integration/defects/issue783
"Defects-issue783-load2" start time: Feb 26 22:14 PST
Output:
----------------------------------------------------------
=======================
Defect #783 - LOAD 2
=======================
FsBroker.local appears to be running (12769):
doug 12769 12768 4 22:12 pts/0 00:00:03 /opt/hypertable/doug/0.9.8.5/bin/htFsBrokerLocal --pidfile /opt/hypertable/doug/0.9.8.5/run/FsBroker.local.pid --verb
ose --Hypertable.Monitoring.Interval=3000
Killing Master.pid 12960
Killing Hyperspace.pid 12849
Shutdown thrift broker complete
Shutdown range server complete
Shutdown hyperspace complete
Shutdown hypertable master complete
Removed /hypertable/servers in FS
Removed /hypertable/tables in FS
Cleared hyperspace
Shutdown FS Broker complete
FS Broker (local) Started
Hyperspace Started
Master Started

Welcome to the hypertable command interpreter.
For information about Hypertable, visit http://hypertable.com

Type 'help' for a list of commands, or 'help shell' for a
list of shell meta commands.


  Elapsed time:  3.25 s

  Elapsed time:  2.85 s

0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
***************************************************


        Elapsed time: 4.69 s
Total cells inserted: 100000
Throughput (cells/s): 21306.96
Total bytes inserted: 30700000
Throughput (bytes/s): 6541238.25

wait_for_file /opt/hypertable/doug/0.9.8.5/run/monitoring/rangeservers/rs1_stats_v0.rrd
-rw-rw-r-- 1 doug doug 3843960 Feb 26 22:14 /opt/hypertable/doug/0.9.8.5/run/monitoring/rangeservers/rs1_stats_v0.rrd
wait_for_file /opt/hypertable/doug/0.9.8.5/run/monitoring/rangeservers/rs2_stats_v0.rrd
-rw-rw-r-- 1 doug doug 3843960 Feb 26 22:14 /opt/hypertable/doug/0.9.8.5/run/monitoring/rangeservers/rs2_stats_v0.rrd
wait_for_file /opt/hypertable/doug/0.9.8.5/run/monitoring/rangeservers/rs3_stats_v0.rrd
-rw-rw-r-- 1 doug doug 3843960 Feb 26 22:14 /opt/hypertable/doug/0.9.8.5/run/monitoring/rangeservers/rs3_stats_v0.rrd
RangeServer CRITICAL - connect error

  Elapsed time:  0.11 s
did not find 'RangeServer rs2: disk use 100% exceeds threshold' in /opt/hypertable/doug/0.9.8.5/log/Master.log but it should be there

Test time =  93.03 sec
----------------------------------------------------------
Test Failed.
"Defects-issue783-load2" end time: Feb 26 22:15 PST
"Defects-issue783-load2" time elapsed: 00:01:33
----------------------------------------------------------

Bad-RSML-Master-recover failure

Happened on test05, release build.

Support for C client.

Hi, is there any support for C, as far I have been working on hypertable through the thrift client which is depend on GLIB as bridge between Thirft and C. any C API with hypertable ?

METADATA-split-recovery-7 test failure

This happened on a centos64 build machine with a release build.

+ echo 'USE '\''/'\''; LOAD DATA INFILE '\''dump.tsv'\'' INTO TABLE LoadTest;'
+ /opt/hypertable/0.9.8.7.f0d7b70/bin/ht shell --batch

Loading 693,292 bytes of input data...

0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
***************************************************
Load complete.

  Elapsed time:  0.03 s
Avg value size:  150.00 bytes
  Avg key size:  30.00 bytes
    Throughput:  21734594.07 bytes/s (25116545.30 bytes/s)
   Total cells:  3333
    Throughput:  120747.74 cells/s
       Resends:  0
+ echo 'USE '\''/'\''; SELECT * FROM LoadTest INTO FILE '\''select-b.7'\'';'
+ /opt/hypertable/0.9.8.7.f0d7b70/bin/ht shell --batch
1428054233 ERROR ht_hypertable : dump_error_history (/root/src/hypertable/src/cc/Hypertable/Lib/RangeLocator.h:157): Hypertable::Exception: Problem creating s
canner for start row '0/0:3:000000000004580155557679878442' on METADATA[..??] - HYPERTABLE request timeout
    at int Hypertable::RangeLocator::find(const Hypertable::TableIdentifier*, const char*, Hypertable::RangeLocationInfo*, Hypertable::Timer&, bool) (/roo
t/src/hypertable/src/cc/Hypertable/Lib/RangeLocator.cc:372)
    at void Hypertable::Lib::RangeServer::Client::do_create_scanner(const Hypertable::CommAddress&, const Hypertable::TableIdentifier&, const Hypertable::
RangeSpec&, const Hypertable::Lib::ScanSpec&, Hypertable::ScanBlock&, int32_t) (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServer/Client.cc:282): RangeSe
rver create_scanner() failure : Event: type=ERROR "HYPERTABLE request timeout" from=10.31.152.249:15865
1428054233 ERROR ht_hypertable : retry_or_abort (/root/src/hypertable/src/cc/Hypertable/Lib/IntervalScannerAsync.cc:397): Hypertable::Exception: Locating rang
e for table 3 row='000000000004580155557679878442' - HYPERTABLE request timeout
    at void Hypertable::IntervalScannerAsync::find_range_and_start_scan(const char*, bool) (/root/src/hypertable/src/cc/Hypertable/Lib/IntervalScannerAsyn
c.cc:248)
    at void Hypertable::RangeLocator::find_loop(const Hypertable::TableIdentifier*, const char*, Hypertable::RangeLocationInfo*, Hypertable::Timer&, bool)
 (/root/src/hypertable/src/cc/Hypertable/Lib/RangeLocator.cc:249): Locating range for table 3 row='000000000004580155557679878442'
1428054233 ERROR ht_hypertable : handle_error (/root/src/hypertable/src/cc/Hypertable/Lib/TableScannerAsync.cc:493): Hypertable::Exception: Event: type=MESSAG
E version=1 total_len=86 header_len=38 header_checksum=-1925115986 flags=8 id=34734 gid=0 timeout_ms=0 payload_checksum=0 command=2 from=10.31.152.249:15865 -
 RANGE SERVER range not found
Hypertable CRITICAL - RANGE SERVER range not found (Event: type=MESSAGE version=1 total_len=86 header_len=38 header_checksum=-1925115986 flags=8 id=34734 gid=
0 timeout_ms=0 payload_checksum=0 command=2 from=10.31.152.249:15865)

Hypertable compile

hi , as recently i tried to re build hypertable from source it appear the new version of boost had an issue with python lib, also the hypertable ht_build didn't recognize ceph new version as weel and i had fixed it by changing the bash script yet i still didn't slove the python problems as the libboost_python is already installed but the cmake is showing error for library

[CMAKE OUTPUT]

node@ubuntu:~/hypertable/build/hypertable$ cmake ../..
-- Use thread library: -lpthread
-- Node.js Version: v0.10.25
-- Python Shell Version: Python 2.7.9
-- Found Python-devel: PYTHON_LIBRARY-NOTFOUND
-- Looking for required boost libraries...
-- GCC version: 4.9.2
-- Boost version: 1_58
-- Boost thread lib: /usr/local/lib/libboost_thread.so
-- Boost program options lib: /usr/local/lib/libboost_program_options.so
-- Boost filesystem lib: /usr/local/lib/libboost_filesystem.so
-- Boost iostreams lib: /usr/local/lib/libboost_iostreams.so
-- Boost python lib: /usr/local/lib/libboost_python.so
-- Boost chrono lib: /usr/local/lib/libboost_chrono.so
-- Boost system lib: /usr/local/lib/libboost_system.so
-- Boost lib dir: /usr/local/lib
-- Boost libs: /usr/local/lib/libboost_iostreams.so;/usr/local/lib/libboost_program_options.so;/usr/local/lib/libboost_filesystem.so;/usr/local/lib/libboost_thread.so;/usr/local/lib/libboost_system.so;/usr/local/lib/libboost_chrono.so;/usr/local/lib/libboost_python.so;PYTHON_LIBRARY-NOTFOUND
-- Found BerkeleyDB: /usr/local/BerkeleyDB.4.8/lib/libdb_cxx.so
-- Berkeley DB version: 4.8.26
-- cb=TRUE c= val= foo=Change Dir: /home/node/hypertable/build/hypertable/CMakeFiles/CMakeTmp/CMakeFiles/CMakeTmp

Run Build Command:"/usr/bin/make" "cmTryCompileExec3992150298/fast"
/usr/bin/make -f CMakeFiles/cmTryCompileExec3992150298.dir/build.make CMakeFiles/cmTryCompileExec3992150298.dir/build
make[1]: Entering directory '/home/node/hypertable/build/hypertable/CMakeFiles/CMakeTmp/CMakeFiles/CMakeTmp'
/usr/bin/cmake -E cmake_progress_report /home/node/hypertable/build/hypertable/CMakeFiles/CMakeTmp/CMakeFiles/CMakeTmp/CMakeFiles 1
Building CXX object CMakeFiles/cmTryCompileExec3992150298.dir/SystemVersion.cc.o
/usr/bin/c++ -Wno-long-long -Wall -pedantic -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_REENTRANT -fPIC -std=c++11 -I/usr/local/include -o CMakeFiles/cmTryCompileExec3992150298.dir/SystemVersion.cc.o -c /home/node/hypertable/cmake/SystemVersion.cc
Linking CXX executable cmTryCompileExec3992150298
/usr/bin/cmake -E cmake_link_script CMakeFiles/cmTryCompileExec3992150298.dir/link.txt --verbose=1
/usr/bin/c++ -Wno-long-long -Wall -pedantic -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_REENTRANT -fPIC -std=c++11 CMakeFiles/cmTryCompileExec3992150298.dir/SystemVersion.cc.o -o cmTryCompileExec3992150298 -rdynamic -ldl /usr/local/lib/libsigar-amd64-linux.so -ldl
make[1]: Leaving directory '/home/node/hypertable/build/hypertable/CMakeFiles/CMakeTmp/CMakeFiles/CMakeTmp'

-- Found SIGAR: /usr/local/lib/libsigar-amd64-linux.so;dl
-- Operating System: Ubuntu_15.04
-- Found RE2: /usr/local/lib/libre2.so
-- version: 0.0.0
-- Found Snappy: /usr/lib/libsnappy.so
-- version: 0.0.0
-- Found rrdtool in /usr/bin
-- Found cronolog in /usr/bin
-- Found Tcmalloc: /usr/lib/libtcmalloc.so
version: 2.2.1
-- Found Jemalloc: /usr/local/lib/libjemalloc.so
-- version: unknown -- make sure it's 1.1+
-- Not Found Hoard: Hoard_LIBRARY-NOTFOUND
-- Did not find ceph libraries
-- Found Maven: Picked up JAVA_TOOL_OPTIONS: -javaagent:/usr/share/java/jayatanaag.jar
-- Javac: Picked up JAVA_TOOL_OPTIONS: -javaagent:/usr/share/java/jayatanaag.jar
javac 1.8.0_45
-- Found libevent: /usr/lib/x86_64-linux-gnu/libevent.so
-- Found Libssh: /usr/lib/x86_64-linux-gnu/libssh.so;/usr/local/ssl/lib/libssl.a;/usr/local/ssl/lib/libcrypto.a
-- Libssh include: /usr/include
version: 0.6.3
-- Libssl include: /usr/local/ssl/include
-- Libssl libraries: /usr/local/ssl/lib/libssl.a /usr/local/ssl/lib/libcrypto.a
-- Found Libssl: /usr/local/ssl/lib/libssl.a
version: foo
-- Found thrift: /usr/local/lib/libthrift.so;/usr/local/lib/libthriftnb.so
-- compiler: Thrift version 0.9.2
-- Thrift for ruby not found. ThriftBroker support for ruby will be disabled
-- PHP Thrift files not found. ThriftBroker support for php will be disabled
-- Thrift for perl not found. ThriftBroker support for perl will be disabled
-- Found thrift for python
-- Java headers found at: /usr/lib/jvm/jdk1.8.0_45/include
-- Please set HADOOP_INCLUDE_PATH variable for Legacy Hadoop support
-- Please set HADOOP_LIB_PATH variable for Legacy Hadoop support
-- Not building shared libraries. Legacy Hadoop support will be disabled
-- Problem installing PYTHON_LIBRARY-NOTFOUND soname=libboost_python.so.1.58.0 to lib
-- Problem installing /usr/local/ssl/lib/libssl.a soname=libssh.so.4 to lib
-- Problem installing /usr/local/ssl/lib/libcrypto.a soname=libssh.so.4 to lib
-- Problem installing /usr/local/ssl/lib/libssl.a soname=libssh.so.4 to lib
-- Problem installing /usr/local/ssl/lib/libcrypto.a soname=libssh.so.4 to lib
CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
EDITLINE_INCLUDE_DIR (ADVANCED)
used as include directory in directory /home/node/hypertable
used as include directory in directory /home/node/hypertable
used as include directory in directory /home/node/hypertable
used as include directory in directory /home/node/hypertable
used as include directory in directory /home/node/hypertable
used as include directory in directory /home/node/hypertable
used as include directory in directory /home/node/hypertable
used as include directory in directory /home/node/hypertable
used as include directory in directory /home/node/hypertable
used as include directory in directory /home/node/hypertable
used as include directory in directory /home/node/hypertable
used as include directory in directory /home/node/hypertable
used as include directory in directory /home/node/hypertable
used as include directory in directory /home/node/hypertable
used as include directory in directory /home/node/hypertable/src/cc/Common
used as include directory in directory /home/node/hypertable/src/cc/Common
used as include directory in directory /home/node/hypertable/src/cc/Common
used as include directory in directory /home/node/hypertable/src/cc/Common
used as include directory in directory /home/node/hypertable/src/cc/Common
used as include directory in directory /home/node/hypertable/src/cc/Common
used as include directory in directory /home/node/hypertable/src/cc/Common
used as include directory in directory /home/node/hypertable/src/cc/Common
used as include directory in directory /home/node/hypertable/src/cc/Common
used as include directory in directory /home/node/hypertable/src/cc/Common
used as include directory in directory /home/node/hypertable/src/cc/Common
used as include directory in directory /home/node/hypertable/src/cc/Common
used as include directory in directory /home/node/hypertable/src/cc/Common
used as include directory in directory /home/node/hypertable/src/cc/Common
used as include directory in directory /home/node/hypertable/src/cc/Common
used as include directory in directory /home/node/hypertable/src/cc/Common
used as include directory in directory /home/node/hypertable/src/cc/Common
used as include directory in directory /home/node/hypertable/src/cc/Common
used as include directory in directory /home/node/hypertable/src/cc/Common
used as include directory in directory /home/node/hypertable/src/cc/Common
used as include directory in directory /home/node/hypertable/src/cc/Common
used as include directory in directory /home/node/hypertable/src/cc/Common
used as include directory in directory /home/node/hypertable/src/cc/Common
used as include directory in directory /home/node/hypertable/src/cc/Common
used as include directory in directory /home/node/hypertable/src/cc/Common
used as include directory in directory /home/node/hypertable/src/cc/Common
used as include directory in directory /home/node/hypertable/src/cc/Common
used as include directory in directory /home/node/hypertable/src/cc/Common
used as include directory in directory /home/node/hypertable/src/cc/Common
used as include directory in directory /home/node/hypertable/src/cc/HyperAppHelper
used as include directory in directory /home/node/hypertable/src/cc/HyperAppHelper
used as include directory in directory /home/node/hypertable/src/cc/HyperAppHelper
used as include directory in directory /home/node/hypertable/src/cc/ThirdParty
used as include directory in directory /home/node/hypertable/src/cc/AsyncComm
used as include directory in directory /home/node/hypertable/src/cc/AsyncComm
used as include directory in directory /home/node/hypertable/src/cc/AsyncComm
used as include directory in directory /home/node/hypertable/src/cc/AsyncComm
used as include directory in directory /home/node/hypertable/src/cc/AsyncComm
used as include directory in directory /home/node/hypertable/src/cc/AsyncComm
used as include directory in directory /home/node/hypertable/src/cc/AsyncComm
used as include directory in directory /home/node/hypertable/src/cc/AsyncComm
used as include directory in directory /home/node/hypertable/src/cc/Hyperspace
used as include directory in directory /home/node/hypertable/src/cc/Hyperspace
used as include directory in directory /home/node/hypertable/src/cc/Hyperspace
used as include directory in directory /home/node/hypertable/src/cc/Hypertable/Lib
used as include directory in directory /home/node/hypertable/src/cc/Hypertable/Lib
used as include directory in directory /home/node/hypertable/src/cc/Hypertable/Lib
used as include directory in directory /home/node/hypertable/src/cc/Hypertable/Lib
used as include directory in directory /home/node/hypertable/src/cc/Hypertable/Lib
used as include directory in directory /home/node/hypertable/src/cc/Hypertable/Lib
used as include directory in directory /home/node/hypertable/src/cc/Hypertable/Lib
used as include directory in directory /home/node/hypertable/src/cc/Hypertable/Lib
used as include directory in directory /home/node/hypertable/src/cc/Hypertable/Lib
used as include directory in directory /home/node/hypertable/src/cc/Hypertable/Lib
used as include directory in directory /home/node/hypertable/src/cc/Hypertable/Lib
used as include directory in directory /home/node/hypertable/src/cc/Hypertable/Lib
used as include directory in directory /home/node/hypertable/src/cc/Hypertable/Lib
used as include directory in directory /home/node/hypertable/src/cc/Hypertable/Lib
used as include directory in directory /home/node/hypertable/src/cc/Hypertable/Lib
used as include directory in directory /home/node/hypertable/src/cc/Hypertable/Lib
used as include directory in directory /home/node/hypertable/src/cc/Hypertable/Lib
used as include directory in directory /home/node/hypertable/src/cc/Hypertable/Lib
used as include directory in directory /home/node/hypertable/src/cc/Hypertable/Lib
used as include directory in directory /home/node/hypertable/src/cc/Hypertable/Lib
used as include directory in directory /home/node/hypertable/src/cc/Hypertable/Lib
used as include directory in directory /home/node/hypertable/src/cc/Hypertable/Lib
used as include directory in directory /home/node/hypertable/src/cc/Hypertable/Lib
used as include directory in directory /home/node/hypertable/src/cc/Hypertable/Lib
used as include directory in directory /home/node/hypertable/src/cc/Hypertable/Lib
used as include directory in directory /home/node/hypertable/src/cc/Hypertable/Lib
used as include directory in directory /home/node/hypertable/src/cc/Hypertable/Lib
used as include directory in directory /home/node/hypertable/src/cc/Hypertable/RangeServer
used as include directory in directory /home/node/hypertable/src/cc/Hypertable/RangeServer
used as include directory in directory /home/node/hypertable/src/cc/Hypertable/RangeServer
used as include directory in directory /home/node/hypertable/src/cc/Hypertable/RangeServer
used as include directory in directory /home/node/hypertable/src/cc/Hypertable/RangeServer
used as include directory in directory /home/node/hypertable/src/cc/Hypertable/RangeServer/tests
used as include directory in directory /home/node/hypertable/src/cc/Hypertable/RangeServer/tests
used as include directory in directory /home/node/hypertable/src/cc/Hypertable/RangeServer/tests
used as include directory in directory /home/node/hypertable/src/cc/Hypertable/RangeServer/tests
used as include directory in directory /home/node/hypertable/src/cc/Hypertable/RangeServer/tests
used as include directory in directory /home/node/hypertable/src/cc/Hypertable/Master
used as include directory in directory /home/node/hypertable/src/cc/Hypertable/Master
used as include directory in directory /home/node/hypertable/src/cc/Hypertable/Master
used as include directory in directory /home/node/hypertable/src/cc/Hypertable/Master
used as include directory in directory /home/node/hypertable/src/cc/Hypertable/Master
used as include directory in directory /home/node/hypertable/src/cc/Hypertable/Master
used as include directory in directory /home/node/hypertable/src/cc/Tools/get_property
used as include directory in directory /home/node/hypertable/src/cc/Tools/prepend_md5
used as include directory in directory /home/node/hypertable/src/cc/Tools/balance_plan_generator
used as include directory in directory /home/node/hypertable/src/cc/Tools/expand_hostspec
used as include directory in directory /home/node/hypertable/src/cc/Tools/cluster
used as include directory in directory /home/node/hypertable/src/cc/Tools/cluster/ClusterDefinition
used as include directory in directory /home/node/hypertable/src/cc/Tools/cluster/ClusterDefinition
used as include directory in directory /home/node/hypertable/src/cc/Tools/Lib
used as include directory in directory /home/node/hypertable/src/cc/Tools/metalog_dump
used as include directory in directory /home/node/hypertable/src/cc/Tools/serverup
used as include directory in directory /home/node/hypertable/src/cc/Tools/dumplog
used as include directory in directory /home/node/hypertable/src/cc/Tools/ssh
used as include directory in directory /home/node/hypertable/src/cc/Tools/prune_tsv
used as include directory in directory /home/node/hypertable/src/cc/Tools/client/hypertable
used as include directory in directory /home/node/hypertable/src/cc/Tools/client/hypertable
used as include directory in directory /home/node/hypertable/src/cc/Tools/client/hypertable
used as include directory in directory /home/node/hypertable/src/cc/Tools/client/rangeserver
used as include directory in directory /home/node/hypertable/src/cc/Tools/client/thriftbroker
used as include directory in directory /home/node/hypertable/src/cc/Tools/client/master
used as include directory in directory /home/node/hypertable/src/cc/Tools/client/hyperspace
used as include directory in directory /home/node/hypertable/src/cc/Tools/client/hyperspace
used as include directory in directory /home/node/hypertable/src/cc/Tools/client/fsbroker
used as include directory in directory /home/node/hypertable/src/cc/Tools/client/fsbroker
used as include directory in directory /home/node/hypertable/src/cc/Tools/generate_error_code_doc
used as include directory in directory /home/node/hypertable/src/cc/Tools/check_integrity
used as include directory in directory /home/node/hypertable/src/cc/Tools/load_generator
used as include directory in directory /home/node/hypertable/src/cc/FsBroker/Lib
used as include directory in directory /home/node/hypertable/src/cc/FsBroker/local
used as include directory in directory /home/node/hypertable/src/cc/ThriftBroker
used as include directory in directory /home/node/hypertable/src/cc/ThriftBroker
used as include directory in directory /home/node/hypertable/src/cc/ThriftBroker
used as include directory in directory /home/node/hypertable/src/cc/ThriftBroker
used as include directory in directory /home/node/hypertable/src/cc/ThriftBroker
used as include directory in directory /home/node/hypertable/src/js
used as include directory in directory /home/node/hypertable/src/rb/ThriftClient
used as include directory in directory /home/node/hypertable/src/perl/ThriftClient
used as include directory in directory /home/node/hypertable/src/py/ThriftClient
used as include directory in directory /home/node/hypertable/src/php/ThriftClient
used as include directory in directory /home/node/hypertable/src/cc/HyperPython
used as include directory in directory /home/node/hypertable/examples/apache_log
used as include directory in directory /home/node/hypertable/examples/apache_log
used as include directory in directory /home/node/hypertable/examples/freebase
used as include directory in directory /home/node/hypertable/tests/integration/defects/issue827
PYTHON_LIBRARY (ADVANCED)
linked by target "HyperCommon" in directory /home/node/hypertable/src/cc/Common
linked by target "HyperAppHelper" in directory /home/node/hypertable/src/cc/HyperAppHelper
linked by target "HyperPython" in directory /home/node/hypertable/src/cc/HyperPython

-- Configuring incomplete, errors occurred!
See also "/home/node/hypertable/build/hypertable/CMakeFiles/CMakeOutput.log".
See also "/home/node/hypertable/build/hypertable/CMakeFiles/CMakeError.log".

Improve ht shell --help text

Currently it just shows options.

Exception should be thrown if timestamp is supplied with COUNTER column update

This is causing some confusion. It would be better if the API threw an Exception.

LIMIT problem

hypertable> select title from url where row =^ "org.wikipedia.ru" limit 1;

Elapsed time:  14 ms
Cells scanned:  2272
Cells returned:  0
Bytes scanned:  175164
Bytes returned:  0
Disk read:  549690
Scan blocks:  1
Sub scanners:  1
Servers:  rs1

hypertable> select title from url where row =^ "org.wikipedia.ru" limit 2;
org.wikipedia.ru/wiki/(2405) Уэлч   title   (2405) Уэлч
org.wikipedia.ru/wiki/(2405) Уэлш   title   (2405) Уэлш — Википедия
org.wikipedia.ru/wiki/(2405) Уэлш   title   (2405) Уэлш

Elapsed time:  39 ms
Cells scanned:  2276
Cells returned:  3
Bytes scanned:  175498
Bytes returned:  224
Disk read:  825484
Scan blocks:  2
Sub scanners:  2
Servers:  rs1

See this thread: Strange limit behaviour in HQL and Thrift. Limit hides results.

Index queries returning inconsistent results

I have a problem when querying Hypertable. Given this two queries, Hypertable gives back different results:

SELECT * FROM similarity WHERE source = 'finnish-municipalities.g' and target = 'medicare.g' and distanceType = 'basicSynonymDistance' and threshold = '0.5';

(No results)

SELECT * FROM similarity WHERE source = 'finnish-municipalities.g' and target = 'medicare.g' and threshold = '0.5' and distanceType = 'basicSynonymDistance';

dc76c7f9-5860-45f8-a624-14f60ff819f2 distanceType basicSynonymDistance
dc76c7f9-5860-45f8-a624-14f60ff819f2 source finnish-municipalities.g
dc76c7f9-5860-45f8-a624-14f60ff819f2 target medicare.g
dc76c7f9-5860-45f8-a624-14f60ff819f2 threshold 0.5
dc76c7f9-5860-45f8-a624-14f60ff819f2 val 0.0

The table has been created this way:

CREATE TABLE similarity (
distanceType, INDEX distanceType,
source, INDEX source,
target, INDEX target,
threshold, INDEX threshold,
val,
ACCESS GROUP default (distanceType, source, target, threshold, val)
);

See Different results when changing the order of column value predicates from a select.

Deleted indexed column cells appearing with MAX_VERSIONS 1

See Retrieved Updated Value in column family on user mailing list.

We should make sure queries like SELECT a WHERE b="1"; queries work with deleted cells. These queries should be querying the index for both a and b and only return results if b hasn't been deleted.

Remove use of boost intrusive_ptr

Should be replaced with C++11 shared_ptr and make_shared.

Improve BAD_KEY exception message

Hi, I have a code which inserts cells with mutator, it works well but somtimes I have some errors
how can I get the cell responsible for th error ( every time i insert 10000 cells with mutator)
I am using thrift client in ruby
the is my code:

mutator = client.open_mutator(ns, table, 0, 0)
client.mutator_set_cells(mutator, cells)
client.close_mutator(mutator)

An exception is thrown, this is the message:

HYPERTABLE bad key - mutator=34407360 cell.size=149755
/opt/hypertable/current/lib/rb/hypertable/gen-rb/client_service.rb:1238:in `recv_mutator_set_cells'

See cell responsible of exception

Client-no-log-sync test failure

Here's the test output:

115/281 Testing: Client-no-log-sync
115/281 Test: Client-no-log-sync
Command: "/bin/env" "TEST_BIN_DIR=/home/doug/build/hypertable/debug/src/cc/Hypertable/Lib/" "/home/doug/src/hypertable/tests/integration/mutator-no-log-sync/run.sh" "/opt/hypertable/doug/0.9.8.6/bin"
Directory: /home/doug/build/hypertable/debug/tests/integration/mutator-no-log-sync
"Client-no-log-sync" start time: Mar 11 10:45 PDT
Output:
----------------------------------------------------------

TEST_BIN=./MutatorNoLogSyncTest
cd ${TEST_BIN_DIR};

${TEST_BIN} $@
FsBroker.local appears to be running (17218):
doug 17218 17216 0 10:43 ? 00:00:00 /opt/hypertable/doug/0.9.8.6/bin/htFsBrokerLocal --pidfile /opt/hypertable/doug/0.9.8.6/run/FsBroker.local.pid --verbose
Killing ThriftBroker.pid 17796
Killing RangeServer.pid 17599
Killing Master.pid 17409
Killing Hyperspace.pid 17297
Shutdown thrift broker complete
Shutdown range server complete
Shutdown hyperspace complete
Shutdown hypertable master complete
Removed /hypertable/servers in FS
Removed /hypertable/tables in FS
Cleared hyperspace
Shutdown FS Broker complete
FS Broker (local) Started
Hyperspace Started
Master Started
*** Load 1
Killing './htRangeServer' pid=18726
Killing './htRangeServer' pid=18827
Killing './htRangeServer' pid=18908
1426095977 INFO MutatorNoLogSyncTest : (/home/doug/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:378) Received event Event: type=DISCONNECT from=192.168.17.12:15872
1426095977 INFO MutatorNoLogSyncTest : (/home/doug/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:442) Disconnected: Problem connecting to RangeServer, will retry in 3000 milliseconds...
1426095977 INFO MutatorNoLogSyncTest : (/home/doug/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:378) Received event Event: type=DISCONNECT from=192.168.17.12:15870
1426095977 INFO MutatorNoLogSyncTest : (/home/doug/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:442) Disconnected: Problem connecting to Root RangeServer, will retry in 3000 milliseconds...
1426095980 INFO MutatorNoLogSyncTest : (/home/doug/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:378) Received event Event: type=DISCONNECT "COMM connect error" from=192.168.17.12:15872
1426095980 INFO MutatorNoLogSyncTest : (/home/doug/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:442) Disconnected: Problem connecting to RangeServer, will retry in 3000 milliseconds...
1426095980 INFO MutatorNoLogSyncTest : (/home/doug/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:378) Received event Event: type=DISCONNECT "COMM connect error" from=192.168.17.12:15870
1426095980 INFO MutatorNoLogSyncTest : (/home/doug/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:442) Disconnected: Problem connecting to Root RangeServer, will retry in 3000 milliseconds...
1426095983 INFO MutatorNoLogSyncTest : (/home/doug/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:378) Received event Event: type=DISCONNECT "COMM connect error" from=192.168.17.12:15872
1426095983 INFO MutatorNoLogSyncTest : (/home/doug/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:442) Disconnected: Problem connecting to RangeServer, will retry in 3000 milliseconds...
*** Load 2
Killing './htRangeServer' pid=19148
Killing './htRangeServer' pid=19229
Killing './htRangeServer' pid=19310
1426095989 INFO MutatorNoLogSyncTest : (/home/doug/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:378) Received event Event: type=DISCONNECT from=192.168.17.12:15870
1426095989 INFO MutatorNoLogSyncTest : (/home/doug/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:442) Disconnected: Problem connecting to Root RangeServer, will retry in 3000 milliseconds...
1426095989 INFO MutatorNoLogSyncTest : (/home/doug/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:378) Received event Event: type=DISCONNECT from=192.168.17.12:15872
1426095989 INFO MutatorNoLogSyncTest : (/home/doug/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:442) Disconnected: Problem connecting to RangeServer, will retry in 3000 milliseconds...
1426095992 INFO MutatorNoLogSyncTest : (/home/doug/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:378) Received event Event: type=DISCONNECT "COMM connect error" from=192.168.17.12:15870
1426095992 INFO MutatorNoLogSyncTest : (/home/doug/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:442) Disconnected: Problem connecting to Root RangeServer, will retry in 3000 milliseconds...
1426095992 INFO MutatorNoLogSyncTest : (/home/doug/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:378) Received event Event: type=DISCONNECT "COMM connect error" from=192.168.17.12:15872
1426095992 INFO MutatorNoLogSyncTest : (/home/doug/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:442) Disconnected: Problem connecting to RangeServer, will retry in 3000 milliseconds...
1426095995 INFO MutatorNoLogSyncTest : (/home/doug/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:378) Received event Event: type=DISCONNECT "COMM connect error" from=192.168.17.12:15872
1426095995 INFO MutatorNoLogSyncTest : (/home/doug/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:442) Disconnected: Problem connecting to RangeServer, will retry in 3000 milliseconds...
*** Load 3
Killing './htRangeServer' pid=19436
Killing './htRangeServer' pid=19517
Killing './htRangeServer' pid=19601
1426095999 INFO MutatorNoLogSyncTest : (/home/doug/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:378) Received event Event: type=DISCONNECT from=192.168.17.12:15870
1426095999 INFO MutatorNoLogSyncTest : (/home/doug/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:442) Disconnected: Problem connecting to Root RangeServer, will retry in 3000 milliseconds...
1426095999 INFO MutatorNoLogSyncTest : (/home/doug/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:378) Received event Event: type=DISCONNECT from=192.168.17.12:15872
1426095999 INFO MutatorNoLogSyncTest : (/home/doug/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:442) Disconnected: Problem connecting to RangeServer, will retry in 3000 milliseconds...
1426096002 INFO MutatorNoLogSyncTest : (/home/doug/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:378) Received event Event: type=DISCONNECT "COMM connect error" from=192.168.17.12:15870
1426096002 INFO MutatorNoLogSyncTest : (/home/doug/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:442) Disconnected: Problem connecting to Root RangeServer, will retry in 3000 milliseconds...
1426096002 INFO MutatorNoLogSyncTest : (/home/doug/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:378) Received event Event: type=DISCONNECT "COMM connect error" from=192.168.17.12:15872
1426096002 INFO MutatorNoLogSyncTest : (/home/doug/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:442) Disconnected: Problem connecting to RangeServer, will retry in 3000 milliseconds...
1426096005 INFO MutatorNoLogSyncTest : (/home/doug/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:378) Received event Event: type=DISCONNECT "COMM connect error" from=192.168.17.12:15872
1426096005 INFO MutatorNoLogSyncTest : (/home/doug/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:442) Disconnected: Problem connecting to RangeServer, will retry in 3000 milliseconds...
1426096010 ERROR MutatorNoLogSyncTest : (/home/doug/src/hypertable/src/cc/Hypertable/Lib/tests/MutatorNoLogSyncTest.cc:422) RangeServer on port 15871 did not come up

Test time =  94.17 sec
----------------------------------------------------------
Test Failed.
"Client-no-log-sync" end time: Mar 11 10:46 PDT
"Client-no-log-sync" time elapsed: 00:01:34
----------------------------------------------------------

Authentication problem with ht_ssh

See this thread: ht cluster fails with hostnames containing '-'.

System-status test failure

Hypertable CRITICAL - FsBroker COMM not connected - FS send_request to 127.0.0.1:15863 failed
4c4
< Hypertable OK

---
> Hypertable WARNING - RangeServer rs1 () Range initialization not yet complete

Add support for rolling restart

With the 0.9.8.5 release, the serialized data structures are forward compatible in addition to backwards compatible. Adding support for rolling restart should be fairly straightforward. We should add a system test for it as well.

Spurious ERROR message in RS log after table schema change

This error message:

1421501986 ERROR Hypertable.RangeServer : create_scanner (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1546): Hypertable::Exception: RangeServer Schema generation for table '2/7' is 1421501985124379000 but supplied is 1421501971123058000 - RANGE SERVER generation mismatch
at void Hypertable::RangeServer::create_scanner(Hypertable::ResponseCallbackCreateScanner*, const Hypertable::TableIdentifier*, const Hypertable::RangeSpec*, const Hypertable::ScanSpec*, Hypertable::QueryCache::Key*) (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1416)

should be changed to an INFO message since it is normal.

Compilation 0.9.8.8 error

is this an issue with Berkley DB 4.8 or Allocator ?

[OUTPUT]

linux:~/hypertable-0.9.8.8/build/hypertable # make
[ 6%] Built target HyperCommon
[ 6%] Built target Base64_test
[ 6%] Built target Crypto_test
[ 6%] Built target HostSpecification_test
[ 6%] Built target Serializable_test
[ 6%] Built target TimeWindowTest
[ 6%] Built target bloom_filter_test
[ 6%] Built target config_test
[ 6%] Built target container_test
[ 7%] Built target crontab_test
[ 7%] Built target escaper_test
[ 7%] Built target exception_test
[ 7%] Built target failure_inducer_test
[ 7%] Built target filesystem_test
[ 7%] Built target hash_test
[ 8%] Built target ht_system_info
[ 8%] Built target ht_wordstream
[ 8%] Built target inetaddr_test
[ 8%] Built target init_test
[ 8%] Built target logging_test
[ 8%] Built target md5_base64_test
[ 8%] Built target mutex_test
[ 8%] Built target pagearena_test
[ 8%] Built target properties_test
[ 8%] Built target scope_guard_test
[ 9%] Built target sertest
[ 9%] Built target stats_serialize_test
[ 9%] Built target string_compressor_test
[ 9%] Built target timeinline_test
[ 12%] Built target HyperComm
[ 21%] Built target HyperFsBroker
[ 21%] Built target HyperThirdParty
[ 21%] Built target HyperTools
[ 23%] Built target Hyperspace
[ 23%] Building CXX object src/cc/Hypertable/Lib/CMakeFiles/Hypertable.dir/Client.cc.o
In file included from /root/hypertable-0.9.8.8/src/cc/Hypertable/Lib/TableMutator.h:25:0,
from /root/hypertable-0.9.8.8/src/cc/Hypertable/Lib/Namespace.h:31,
from /root/hypertable-0.9.8.8/src/cc/Hypertable/Lib/NamespaceCache.h:28,
from /root/hypertable-0.9.8.8/src/cc/Hypertable/Lib/Client.h:35,
from /root/hypertable-0.9.8.8/src/cc/Hypertable/Lib/Client.cc:24:
/root/hypertable-0.9.8.8/src/cc/Hypertable/Lib/Cells.h: In constructor 'Hypertable::CellsBuilder::CellsBuilder(size_t)':
/root/hypertable-0.9.8.8/src/cc/Hypertable/Lib/Cells.h:46:48: error: no matching function for call to 'std::set<const char*, LtCstr, Hypertable::PageArenaAllocator<const char*> >::set(Hypertable::CellsBuilder::CstrSetAlloc)'
m_str_set = CstrSet(CstrSetAlloc(m_arena));
^
/root/hypertable-0.9.8.8/src/cc/Hypertable/Lib/Cells.h:46:48: note: candidates are:
In file included from /usr/include/c++/4.8/set:61:0,
from /root/hypertable-0.9.8.8/src/cc/AsyncComm/CommAddress.h:36,
from /root/hypertable-0.9.8.8/src/cc/AsyncComm/HandlerMap.h:34,
from /root/hypertable-0.9.8.8/src/cc/AsyncComm/ReactorRunner.h:31,
from /root/hypertable-0.9.8.8/src/cc/AsyncComm/ApplicationHandler.h:40,
from /root/hypertable-0.9.8.8/src/cc/AsyncComm/ApplicationQueueInterface.h:31,
from /root/hypertable-0.9.8.8/src/cc/Hypertable/Lib/Client.h:29,
from /root/hypertable-0.9.8.8/src/cc/Hypertable/Lib/Client.cc:24:
/usr/include/c++/4.8/bits/stl_set.h:218:7: note: std::set<_Key, _Compare, _Alloc>::set(std::initializer_list<_Tp>, const _Compare&, const allocator_type&) [with Key = const char; _Compare = LtCstr; _Alloc = Hypertable::PageArenaAllocator; std::set<_Key, _Compare, _Alloc>::allocator_type = Hypertable::PageArenaAllocator<const char*>]
set(initializer_list<value_type> __l,
^
/usr/include/c++/4.8/bits/stl_set.h:218:7: note: no known conversion for argument 1 from 'Hypertable::CellsBuilder::CstrSetAlloc {aka Hypertable::PageArenaAllocator<const char*>}' to 'std::initializer_list<const char*>'
/usr/include/c++/4.8/bits/stl_set.h:204:7: note: std::set<_Key, _Compare, _Alloc>::set(std::set<_Key, _Compare, _Alloc>&&) [with Key = const char; _Compare = LtCstr; _Alloc = Hypertable::PageArenaAllocator]
set(set&& __x)
^
/usr/include/c++/4.8/bits/stl_set.h:204:7: note: no known conversion for argument 1 from 'Hypertable::CellsBuilder::CstrSetAlloc {aka Hypertable::PageArenaAllocator<const char*>}' to 'std::set<const char*, LtCstr, Hypertable::PageArenaAllocator<const char*> >&&'
/usr/include/c++/4.8/bits/stl_set.h:193:7: note: std::set<_Key, _Compare, _Alloc>::set(const std::set<_Key, _Compare, _Alloc>&) [with Key = const char; _Compare = LtCstr; _Alloc = Hypertable::PageArenaAllocator]
set(const set& __x)
^
/usr/include/c++/4.8/bits/stl_set.h:193:7: note: no known conversion for argument 1 from 'Hypertable::CellsBuilder::CstrSetAlloc {aka Hypertable::PageArenaAllocator<const char*>}' to 'const std::set<const char*, LtCstr, Hypertable::PageArenaAllocator<const char*> >&'
/usr/include/c++/4.8/bits/stl_set.h:180:2: note: template std::set<_Key, _Compare, _Alloc>::set(_InputIterator, _InputIterator, const _Compare&, const allocator_type&)
set(_InputIterator __first, _InputIterator __last,
^
/usr/include/c++/4.8/bits/stl_set.h:180:2: note: template argument deduction/substitution failed:
In file included from /root/hypertable-0.9.8.8/src/cc/Hypertable/Lib/TableMutator.h:25:0,
from /root/hypertable-0.9.8.8/src/cc/Hypertable/Lib/Namespace.h:31,
from /root/hypertable-0.9.8.8/src/cc/Hypertable/Lib/NamespaceCache.h:28,
from /root/hypertable-0.9.8.8/src/cc/Hypertable/Lib/Client.h:35,
from /root/hypertable-0.9.8.8/src/cc/Hypertable/Lib/Client.cc:24:
/root/hypertable-0.9.8.8/src/cc/Hypertable/Lib/Cells.h:46:48: note: candidate expects 4 arguments, 1 provided
m_str_set = CstrSet(CstrSetAlloc(m_arena));
^
In file included from /usr/include/c++/4.8/set:61:0,
from /root/hypertable-0.9.8.8/src/cc/AsyncComm/CommAddress.h:36,
from /root/hypertable-0.9.8.8/src/cc/AsyncComm/HandlerMap.h:34,
from /root/hypertable-0.9.8.8/src/cc/AsyncComm/ReactorRunner.h:31,
from /root/hypertable-0.9.8.8/src/cc/AsyncComm/ApplicationHandler.h:40,
from /root/hypertable-0.9.8.8/src/cc/AsyncComm/ApplicationQueueInterface.h:31,
from /root/hypertable-0.9.8.8/src/cc/Hypertable/Lib/Client.h:29,
from /root/hypertable-0.9.8.8/src/cc/Hypertable/Lib/Client.cc:24:
/usr/include/c++/4.8/bits/stl_set.h:163:2: note: template std::set<_Key, _Compare, _Alloc>::set(_InputIterator, _InputIterator)
set(_InputIterator __first, _InputIterator __last)
^
/usr/include/c++/4.8/bits/stl_set.h:163:2: note: template argument deduction/substitution failed:
In file included from /root/hypertable-0.9.8.8/src/cc/Hypertable/Lib/TableMutator.h:25:0,
from /root/hypertable-0.9.8.8/src/cc/Hypertable/Lib/Namespace.h:31,
from /root/hypertable-0.9.8.8/src/cc/Hypertable/Lib/NamespaceCache.h:28,
from /root/hypertable-0.9.8.8/src/cc/Hypertable/Lib/Client.h:35,
from /root/hypertable-0.9.8.8/src/cc/Hypertable/Lib/Client.cc:24:
/root/hypertable-0.9.8.8/src/cc/Hypertable/Lib/Cells.h:46:48: note: candidate expects 2 arguments, 1 provided
m_str_set = CstrSet(CstrSetAlloc(m_arena));
^
In file included from /usr/include/c++/4.8/set:61:0,
from /root/hypertable-0.9.8.8/src/cc/AsyncComm/CommAddress.h:36,
from /root/hypertable-0.9.8.8/src/cc/AsyncComm/HandlerMap.h:34,
from /root/hypertable-0.9.8.8/src/cc/AsyncComm/ReactorRunner.h:31,
from /root/hypertable-0.9.8.8/src/cc/AsyncComm/ApplicationHandler.h:40,
from /root/hypertable-0.9.8.8/src/cc/AsyncComm/ApplicationQueueInterface.h:31,
from /root/hypertable-0.9.8.8/src/cc/Hypertable/Lib/Client.h:29,
from /root/hypertable-0.9.8.8/src/cc/Hypertable/Lib/Client.cc:24:
/usr/include/c++/4.8/bits/stl_set.h:148:7: note: std::set<_Key, _Compare, _Alloc>::set(const _Compare&, const allocator_type&) [with Key = const char; _Compare = LtCstr; _Alloc = Hypertable::PageArenaAllocator; std::set<_Key, _Compare, _Alloc>::allocator_type = Hypertable::PageArenaAllocator<const char*>]
set(const _Compare& __comp,
^
/usr/include/c++/4.8/bits/stl_set.h:148:7: note: no known conversion for argument 1 from 'Hypertable::CellsBuilder::CstrSetAlloc {aka Hypertable::PageArenaAllocator<const char*>}' to 'const LtCstr&'
/usr/include/c++/4.8/bits/stl_set.h:139:7: note: std::set<_Key, _Compare, _Alloc>::set() [with Key = const char; _Compare = LtCstr; _Alloc = Hypertable::PageArenaAllocator]
set()
^
/usr/include/c++/4.8/bits/stl_set.h:139:7: note: candidate expects 0 arguments, 1 provided
src/cc/Hypertable/Lib/CMakeFiles/Hypertable.dir/build.make:422: recipe for target 'src/cc/Hypertable/Lib/CMakeFiles/Hypertable.dir/Client.cc.o' failed
make[2]: *** [src/cc/Hypertable/Lib/CMakeFiles/Hypertable.dir/Client.cc.o] Error 1
CMakeFiles/Makefile2:2212: recipe for target 'src/cc/Hypertable/Lib/CMakeFiles/Hypertable.dir/all' failed
make[1]: *** [src/cc/Hypertable/Lib/CMakeFiles/Hypertable.dir/all] Error 2
Makefile:147: recipe for target 'all' failed
make: *** [all] Error 2

OperationInitialize being run when in COMPLETE state

Noticed a core file for the Master when running the system-status test. Here's the stack trace:

(gdb) where
#0  0x00000039d340f5db in raise () from /lib64/libpthread.so.0
#1  0x0000000000feeef3 in Hypertable::OperationInitialize::execute (this=0x1c24a20) at /home/doug/src/hypertable/src/cc/Hypertable/Master/OperationInitializ\
e.cc:212
#2  0x0000000000ffaba3 in Hypertable::OperationProcessor::Worker::operator() (this=0x1c191b8) at /home/doug/src/hypertable/src/cc/Hypertable/Master/Operatio\
nProcessor.cc:321
#3  0x000000000102e7f2 in boost::detail::thread_data::run (this=0x1c19000) at /usr/local/include/boost/thread/detail\
/thread.hpp:117
#4  0x00007f3ef327e082 in thread_proxy () from /opt/hypertable/doug/0.9.8.4/lib/libboost_thread.so.1.54.0
#5  0x00000039d34079d1 in start_thread () from /lib64/libpthread.so.0
#6  0x00000039d30e89dd in clone () from /lib64/libc.so.6

Here's the line in the source code:

  default:
=>  HT_FATALF("Unrecognized state %d", state);
  }

(gdb) p state
$3 = 1

State 1 is the COMPLETE state.

Change hflush() to hsync() for HDFS (at least for METADATA)

See Support hsync in HDFS

Add unit tests to Javascript classes

See nodeunit and node-int64 test.js

Broken connection from RangeServer to Master not handled properly

Ran into a regression failure in test issue783-offload1 where the connection from rs1 to the master was broken. The Master started logging the following over-and-over again:

1424828380 INFO htMaster : (/home/doug/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:65) Entering GatherStatistics-7704 state=INITIAL
1424828380 WARN htMaster : (/home/doug/src/hypertable/src/cc/Hypertable/Master/DispatchHandlerOperation.cc:121) Couldn't locate connection object for 192.168.17.13:15870
1424828380 ERROR htMaster : result_callback (/home/doug/src/hypertable/src/cc/Hypertable/Master/DispatchHandlerOperationGetStatistics.cc:93): Received 'get_statistics' response from unexpected connection 192.168.17.13:15870
1424828380 INFO htMaster : (/home/doug/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:132) Aggregate disk usage = 52.86
1424828380 INFO htMaster : (/home/doug/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:163) Leaving GatherStatistics-7704

and the OperationProcessor state was this:

Num vertices = 6
Busy count = 2
Active set size = 2
Need order recompute = false
Shutdown = false
Paused = false

0: RegisterServerBlocker(rs1)
  active: false
  live: true
  exclusive: false
  perpetual: false
  blocked: true
  state: STARTED
  dependencies: ()
  obstructions: (RegisterServerBlocker rs1)
  exclusivities: ()

0: RegisterServerBlocker(rs2)
  active: false
  live: true
  exclusive: false
  perpetual: false
  blocked: true
  state: STARTED
  dependencies: ()
  obstructions: (RegisterServerBlocker rs2)
  exclusivities: ()

0: RegisterServerBlocker(rs3)
  active: false
  live: true
  exclusive: false
  perpetual: false
  blocked: true
  state: STARTED
  dependencies: ()
  obstructions: (RegisterServerBlocker rs3)
  exclusivities: ()

0: MoveRange rs3 2[0098861..??] -> rs3
  active: true
  live: true
  exclusive: false
  perpetual: false
  blocked: false
  state: LOAD_RANGE
  dependencies: (2666844875433519387,INIT,METADATA,ROOT,SERVERS,SYSTEM)
  obstructions: (2 move range,OperationMove 2[0098861..??])
  exclusivities: (-3858079322379480769)

0: RegisterServer rs1
  active: false
  live: true
  exclusive: false
  perpetual: false
  blocked: false
  state: INITIAL
  dependencies: (RegisterServerBlocker rs1)
  obstructions: ()
  exclusivities: (RegisterServer rs1)

0: CollectGarbage
  active: true
  live: true
  exclusive: true
  perpetual: false
  blocked: false
  state: INITIAL
  dependencies: (INIT,METADATA)
  obstructions: ()
  exclusivities: ()

Current:
MoveRange rs3 2[0098861..??] -> rs3
CollectGarbage
RegisterServerBlocker(rs1)
RegisterServerBlocker(rs2)
RegisterServerBlocker(rs3)
[retired]
*

Execution order:
MoveRange rs3 2[0098861..??] -> rs3 (time=0)
CollectGarbage (time=0)
RegisterServerBlocker(rs1) (time=0)
RegisterServerBlocker(rs2) (time=0)
RegisterServerBlocker(rs3) (time=0)
[retired]
*RegisterServer rs1 (time=1)

Graphviz:
digraph G {
0[label="RegisterServerBlocker(rs1)"];
1[label="RegisterServerBlocker(rs2)"];
2[label="RegisterServerBlocker(rs3)"];
3[label="MoveRange 2\n0098861\nEND_ROW_MARKER"];
4[label="RegisterServer rs1"];
5[label=CollectGarbage];
4->0 ;
}

All scripts should have descriptive help text

We should make sure all of the scripts have good help text and support the Nagios help options (-h and --help).

Compaction-exception-split-2 test failure

Version 0.9.8.5 release build. The load_generator is hung here:

Thread 1 (Thread 0x7f40aa6447e0 (LWP 5402)):
#0  0x00000039d340b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000005cdb2c in void boost::condition_variable_any::wait >(boost::unique_lock&) ()
#2  0x00000000006c8ab4 in Hypertable::TableMutatorQueue::wait_for_buffer(boost::unique_lock&, Hypertable::ApplicationHandler**) ()
#3  0x00000000006c696f in Hypertable::TableMutator::wait_for_flush_completion(Hypertable::TableMutatorAsync*) ()
#4  0x00000000006c6c1f in Hypertable::TableMutator::auto_flush() ()
#5  0x00000000006c6ed7 in Hypertable::TableMutator::set_cells(__gnu_cxx::__normal_iterator > > >, __gnu_cxx::__normal_iterator > > >) ()
#6  0x00000000005c8f09 in LoadClient::set_cells(std::vector > > const&) ()
#7  0x00000000005b39c5 in generate_update_load(boost::intrusive_ptr&, std::basic_string, std::allocator >&, bool, bool, unsigned long, unsigned long, bool, std::basic_string, std::allocator >&, int, bool) ()
#8  0x00000000005c64b6 in main ()

Defects-issue777 test failure

This happened on Debian 32-bit release build. It appears that the servers came up, but the Master wasn't fully up before it was shutdown. We should build a test that verifies that the Master can come back up even when it crashes at various points during Initialization. In this particular situation, when the Master came up after being stopped during initialization, it logged these messages:

1425073278 INFO htMaster : (/root/src/hypertable/src/cc/AsyncComm/IOHandler.h:170) Event: type=CONNECTION_ESTABLISHED from=10.9.65.81:52346
1425073278 INFO htMaster : (/root/src/hypertable/src/cc/Hypertable/Master/OperationSystemUpgrade.cc:81) Entering SystemUpgrade-14
1425073278 INFO htMaster : (/root/src/hypertable/src/cc/Hypertable/Master/OperationSystemUpgrade.cc:103) Leaving SystemUpgrade-14
1425073278 ERROR htMaster : main (/root/src/hypertable/src/cc/Hypertable/Master/main.cc:313): Hypertable::Exception: sys/RS_METRICS - HYPERTABLE table does not exist
    at void Hypertable::Table::initialize() (/root/src/hypertable/src/cc/Hypertable/Lib/Table.cc:108)