wavii / darner Goto Github PK
View Code? Open in Web Editor NEWsimple, lightweight message queue
License: Other
simple, lightweight message queue
License: Other
I've tried kestrel-client (build on top of memcached gem), the memcached gem and the dalli gem, and all fail to perform a get from darner.
For example, I start up darner:
$ darner -d /var/spool/darner/
[INFO] 2012-Sep-07 19:41:46.008201: darner: queue server
[INFO] 2012-Sep-07 19:41:46.008847: build: Sep 7 2012 (18:54:17) v0.1.4 (c) Wavii, Inc.
[INFO] 2012-Sep-07 19:41:46.008932: listening on port: 22133
[INFO] 2012-Sep-07 19:41:46.008978: data dir: /var/spool/darner/
[INFO] 2012-Sep-07 19:41:46.009047: starting up
Then start up irb, with dalli:
$ irb -r rubygems -r dalli
1.9.3p0 :001 > client = Dalli::Client.new 'localhost:22133'
=> #<Dalli::Client:0x007fae532e3f20 @servers="localhost:22133", @options={}, @ring=nil>
1.9.3p0 :002 > client.get 'foo'
I, [2012-09-07T19:44:05.305352 #77838] INFO -- : localhost:22133 failed (count: 0)
Dalli::RingError: No server available
from /Users/jnunemaker/.rvm/gems/ruby-1.9.3-p0/gems/dalli-2.1.0/lib/dalli/ring.rb:45:in `server_for_key'
from /Users/jnunemaker/.rvm/gems/ruby-1.9.3-p0/gems/dalli-2.1.0/lib/dalli/client.rb:250:in `perform'
from /Users/jnunemaker/.rvm/gems/ruby-1.9.3-p0/gems/dalli-2.1.0/lib/dalli/client.rb:50:in `get'
from (irb):2
from /Users/jnunemaker/.rvm/rubies/ruby-1.9.3-p0/bin/irb:16:in `<main>'
Same thing from the memcached gem:
$ irb -r rubygems -r memcached
1.9.3p0 :001 > client = Memcached.new 'localhost:22133'
=> #<Memcached:0x007fe0430bc838 @struct=#<Rlibmemcached::MemcachedSt:0x007fe0430bc7e8>, @options={:hash=>:fnv1_32, :no_block=>false, :noreply=>false, :distribution=>:consistent_ketama, :ketama_weighted=>true, :buffer_requests=>false, :cache_lookups=>true, :support_cas=>false, :tcp_nodelay=>false, :show_backtraces=>false, :retry_timeout=>30, :timeout=>0.25, :rcv_timeout=>0.25, :poll_timeout=>0.25, :connect_timeout=>4, :prefix_key=>"", :prefix_delimiter=>"", :hash_with_prefix_key=>true, :default_ttl=>604800, :default_weight=>8, :sort_hosts=>false, :auto_eject_hosts=>true, :server_failure_limit=>2, :verify_key=>true, :use_udp=>false, :binary_protocol=>false, :credentials=>nil, :experimental_features=>false, :exception_retry_limit=>5, :exceptions_to_retry=>[Memcached::ServerIsMarkedDead, Memcached::ATimeoutOccurred, Memcached::ConnectionBindFailure, Memcached::ConnectionFailure, Memcached::ConnectionSocketCreateFailure, Memcached::Failure, Memcached::MemoryAllocationFailure, Memcached::ReadFailure, Memcached::ServerEnd, Memcached::ServerError, Memcached::SystemError, Memcached::UnknownReadFailure, Memcached::WriteFailure, Memcached::SomeErrorsWereReported]}, @default_ttl=604800, @not_found=#<Memcached::NotFound: Memcached::NotFound>, @not_stored=#<Memcached::NotStored: Memcached::NotStored>>
1.9.3p0 :002 > client.get 'foo'
return else
"foo"
8
Memcached::ProtocolError: Key {"foo"=>"localhost:22133:8"}
from /Users/jnunemaker/.rvm/gems/ruby-1.9.3-p0/gems/memcached-1.4.3/lib/memcached/memcached.rb:623:in `reraise'
from /Users/jnunemaker/.rvm/gems/ruby-1.9.3-p0/gems/memcached-1.4.3/lib/memcached/memcached.rb:600:in `check_return_code'
from /Users/jnunemaker/.rvm/gems/ruby-1.9.3-p0/gems/memcached-1.4.3/lib/memcached/memcached.rb:517:in `get'
from (irb):2
from /Users/jnunemaker/.rvm/rubies/ruby-1.9.3-p0/bin/irb:16:in `<main>'
set
seems to work for the memcached gem, but dalli always reports darner as dead. I also tried with the raw flag true, to avoid marshaling, but it still failed.
I was able to use telnet just fine, so it seems to be something specific that the ruby gems need in the memcache protocol, but I'm not familiar enough. I can try to dig in, but I thought I would see if it is something obvious or easy that I am missing first.
I currently use kestrel with the kestrel-client gem no problem. I'd love to test out darner. Any ideas?
Mainly user error, but exceptions (when starting up) are a bit un-helpful. Here is an example of a invalid data directory on OSX.
[INFO] 2012-Sep-22 17:24:47.901668: darner: queue server
[INFO] 2012-Sep-22 17:24:47.902440: build: Aug 30 2012 (17:15:58) v0.1.4 (c) Wavii, Inc.
[INFO] 2012-Sep-22 17:24:47.902481: listening on port: 22133
[INFO] 2012-Sep-22 17:24:47.902515: data dir: /tmp/
[INFO] 2012-Sep-22 17:24:47.902544: debug logging is turned ON
[INFO] 2012-Sep-22 17:24:47.902574: starting up
libc++abi.dylib: terminate called throwing an exception
Abort trap: 6
Could you please provide documentation or atleast a sample config that shows how we can setup a multi-server darner queue cluster.
I was trying to /close
a message but not get the next one yet.
stats
STAT queue_job_queue_items 2
STAT queue_job_queue_waiters 0
STAT queue_job_queue_open_transactions 0
END
get job_queue/open
VALUE job_queue 0 32
{"url": "http://www.google.com"}
END
stats
STAT queue_job_queue_items 1
STAT queue_job_queue_waiters 0
STAT queue_job_queue_open_transactions 1
END
get job_queue/close
VALUE job_queue 0 32
{"url": "http://www.google.com"}
END
stats
STAT queue_job_queue_items 0
STAT queue_job_queue_waiters 0
STAT queue_job_queue_open_transactions 0
END
It looks like /close
is returning a message, as well as closing the transaction (and not opening a new transaction like /close/open
would).
Is this expected behavior?
make
Scanning dependencies of target darner
Scanning dependencies of target db
Scanning dependencies of target test
[ 6%] [ 18%] [ 25%] [ 25%] [ 31%] [ 37%] [ 43%] [ 50%] Building CXX object CMakeFiles/db.dir/bench/db.o
Building CXX object CMakeFiles/test.dir/src/net/request.o
Building CXX object CMakeFiles/darner.dir/src/net/handler.o
Building CXX object CMakeFiles/darner.dir/src/net/request.o
Building CXX object CMakeFiles/darner.dir/src/queue/iqstream.o
Building CXX object CMakeFiles/test.dir/src/queue/iqstream.o
Building CXX object CMakeFiles/darner.dir/src/util/log.o
Building CXX object CMakeFiles/test.dir/src/queue/oqstream.o
[ 56%] [ 62%] Building CXX object CMakeFiles/test.dir/src/queue/queue.o
Building CXX object CMakeFiles/darner.dir/src/queue/oqstream.o
[ 68%] Building CXX object CMakeFiles/darner.dir/src/queue/queue.o
[ 75%] Building CXX object CMakeFiles/test.dir/src/util/log.o
In file included from /tmp/homebrew-darner-0.1.0-67cD/wavii-darner-7f087e5/src/net/handler.cpp:1:
/tmp/homebrew-darner-0.1.0-67cD/wavii-darner-7f087e5/include/darner/net/handler.h:83:19: error: no matching function for call to 'buffer'
socket_, boost::asio::buffer(msg), boost::bind(&handler::read_request, shared_from_this(), _1, _2));
^~~~~~~~~~~~~~~~~~~
/usr/local/include/boost/asio/buffer.hpp:698:26: note: candidate function not viable: no known conversion from 'const char *' to 'const boost::asio::mutable_buffer' for 1st argument;
inline mutable_buffers_1 buffer(const mutable_buffer& b)
^
/usr/local/include/boost/asio/buffer.hpp:727:24: note: candidate function not viable: no known conversion from 'const char *' to 'const boost::asio::const_buffer' for 1st argument;
inline const_buffers_1 buffer(const const_buffer& b)
^
/usr/local/include/boost/asio/buffer.hpp:779:26: note: candidate template ignored: failed template argument deduction
inline mutable_buffers_1 buffer(PodType (&data)[N])
^
/usr/local/include/boost/asio/buffer.hpp:809:24: note: candidate template ignored: failed template argument deduction
inline const_buffers_1 buffer(const PodType (&data)[N])
^
/usr/local/include/boost/asio/buffer.hpp:910:26: note: candidate template ignored: failed template argument deduction
inline mutable_buffers_1 buffer(boost::array<PodType, N>& data)
^
/usr/local/include/boost/asio/buffer.hpp:941:24: note: candidate template ignored: failed template argument deduction
inline const_buffers_1 buffer(boost::array<const PodType, N>& data)
^
/usr/local/include/boost/asio/buffer.hpp:975:24: note: candidate template ignored: failed template argument deduction
inline const_buffers_1 buffer(const boost::array<PodType, N>& data)
^
/usr/local/include/boost/asio/buffer.hpp:1106:26: note: candidate template ignored: failed template argument deduction
inline mutable_buffers_1 buffer(std::vector<PodType, Allocator>& data)
^
/usr/local/include/boost/asio/buffer.hpp:1155:24: note: candidate template ignored: failed template argument deduction
inline const_buffers_1 buffer(
^
/usr/local/include/boost/asio/buffer.hpp:1202:24: note: candidate template ignored: failed template argument deduction
inline const_buffers_1 buffer(
^
/usr/local/include/boost/asio/buffer.hpp:792:26: note: candidate function template not viable: requires 2 arguments, but 1 was provided
inline mutable_buffers_1 buffer(PodType (&data)[N],
^
/usr/local/include/boost/asio/buffer.hpp:822:24: note: candidate function template not viable: requires 2 arguments, but 1 was provided
inline const_buffers_1 buffer(const PodType (&data)[N],
^
/usr/local/include/boost/asio/buffer.hpp:924:26: note: candidate function template not viable: requires 2 arguments, but 1 was provided
inline mutable_buffers_1 buffer(boost::array<PodType, N>& data,
^
/usr/local/include/boost/asio/buffer.hpp:955:24: note: candidate function template not viable: requires 2 arguments, but 1 was provided
inline const_buffers_1 buffer(boost::array<const PodType, N>& data,
^
/usr/local/include/boost/asio/buffer.hpp:989:24: note: candidate function template not viable: requires 2 arguments, but 1 was provided
inline const_buffers_1 buffer(const boost::array<PodType, N>& data,
^
/usr/local/include/boost/asio/buffer.hpp:1129:26: note: candidate function template not viable: requires 2 arguments, but 1 was provided
inline mutable_buffers_1 buffer(std::vector<PodType, Allocator>& data,
^
/usr/local/include/boost/asio/buffer.hpp:1179:24: note: candidate function template not viable: requires 2 arguments, but 1 was provided
inline const_buffers_1 buffer(
^
/usr/local/include/boost/asio/buffer.hpp:1225:24: note: candidate function template not viable: requires 2 arguments, but 1 was provided
inline const_buffers_1 buffer(
^
/usr/local/include/boost/asio/buffer.hpp:765:24: note: candidate function not viable: requires 2 arguments, but 1 was provided
inline const_buffers_1 buffer(const void* data,
^
/usr/local/include/boost/asio/buffer.hpp:756:26: note: candidate function not viable: requires 2 arguments, but 1 was provided
inline mutable_buffers_1 buffer(void* data, std::size_t size_in_bytes)
^
/usr/local/include/boost/asio/buffer.hpp:739:24: note: candidate function not viable: requires 2 arguments, but 1 was provided
inline const_buffers_1 buffer(const const_buffer& b,
^
/usr/local/include/boost/asio/buffer.hpp:710:26: note: candidate function not viable: requires 2 arguments, but 1 was provided
inline mutable_buffers_1 buffer(const mutable_buffer& b,
^
1 error generated.
make[2]: *** [CMakeFiles/darner.dir/src/net/handler.o] Error 1
make[2]: *** Waiting for unfinished jobs....
[ 81%] Building CXX object CMakeFiles/test.dir/tests/queue.o
[ 87%] Building CXX object CMakeFiles/test.dir/tests/request.o
[ 93%] Building CXX object CMakeFiles/test.dir/tests/main.o
Linking CXX executable db
[ 93%] Built target db
make[1]: *** [CMakeFiles/darner.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
Linking CXX executable test
[ 93%] Built target test
make: *** [all] Error 2
Error: #<BuildError: Failed executing: make >
$ darner -V
terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::program_options::unknown_option> >'
what(): unrecognised option '-V'
Aborted (core dumped)
darner version is 0.2.0.
boost version is 1.50.0
I am getting some odd behavior with doing an /open, performing work, and a subsequent /close.
If I use a single worker client, it performs as expected. When I use more than one connection (python with greenlets) I sometimes get the expected behavior (one client "/open"s the message, appears to get a lock on it, then "/close"s it), and sometimes each client gets the job multiple times, and it seems the close it ignored (or perhaps the message isn't being locked).
Darner is in debug log mode, but no logging is occurring, so it is hard to tell what is doing on inside the server. Dumping the memcache protocol shows that client side source ports are not changing (implies that connections are indeed persistent and not churning).
Any advice on what to look for?
Thanks.
Notes:
When trying to build on centos 6.3, I get this error:
$ make
[ 6%] Building CXX object CMakeFiles/darner.dir/src/net/handler.o
In file included from /build/darner/include/darner/queue/iqstream.h:6,
from /build/darner/include/darner/net/handler.h:14,
from /build/darner/src/net/handler.cpp:1:
/build/darner/include/darner/queue/queue.h: In member function ‘void darner::queue::put(const darner::queue::key_type&, const std::string&)’:
/build/darner/include/darner/queue/queue.h:211: error: no match for call to ‘(const boost::system::error_category) ()’
/build/darner/include/darner/queue/queue.h: In member function ‘void darner::queue::get(const darner::queue::key_type&, std::string&)’:
/build/darner/include/darner/queue/queue.h:217: error: no match for call to ‘(const boost::system::error_category) ()’
/build/darner/include/darner/queue/queue.h: In member function ‘void darner::queue::write(leveldb::WriteBatch&)’:
/build/darner/include/darner/queue/queue.h:223: error: no match for call to ‘(const boost::system::error_category) ()’
In file included from /build/darner/include/darner/net/handler.h:16,
from /build/darner/src/net/handler.cpp:1:
/build/darner/include/darner/util/queue_map.hpp: In constructor ‘darner::queue_map::queue_map(boost::asio::io_service&, const std::string&)’:
/build/darner/include/darner/util/queue_map.hpp:33: error: ‘struct std::basic_string<char, std::char_traits<char>, std::allocator<char> >’ has no member named ‘string’
make[2]: *** [CMakeFiles/darner.dir/src/net/handler.o] Error 1
make[1]: *** [CMakeFiles/darner.dir/all] Error 2
make: *** [all] Error 2
Boost version is 1.41.0 (version in centos repos).
Is this version too old?
Hi,
Is the project alive? Can't build it on Mavericks with Xcode nor gcc-4.2.
Hi,
Can't build it on windows platform? Can you provide any guidance for building this project on windows?Do you have any plans to support it in future?
if Homebrew/legacy-homebrew#14203 gets merged it would be nice to have instructions for homebrew.
Kestrel have a simple webinterface called stats.json which output some statistics. Would be cool if darner had this aswell.
Darner ought to support the memcached binary protocol:
I think I have some queue corruption. We upgraded 0.1.4 to 0.2.0 via put binary in place, stop; start; of the same dir.
After restart, the clients of one queue started getting:
SERVER_ERROR Input/output error
I copied off the spool dir and ran darner on it in a separate environment to see what the state looked like. I got a few of these in the darner output:
[ERROR] 2013-Jul-11 00:25:54.528810: handler<0x7fe7d00047d0>::get: Input/output error
Also odd, one queue in stats shows almost 42 million items in it, but before the upgrade it had very few. During the upgrade the open transactions stat also showed what appeared to be a wrapped 64bit int (it was huge, sadly I didn't record it).
I am poking at things a bit more.
Anything in particular to look for?
The darner instance was under pretty heavy load when it was restarted, I am wondering if the older version didn't exit gracefully and may have corrupted the leveldb transaction log or something.
If I connect to darner via telnet and quit
, I'll get an error:
$ telnet localhost 22133
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
quit
ERROR
Connection closed by foreign host.
"Erik Frey", "Homebrew", and two "Kestrel" links are broken.
Also, under "Resident Memory", there is one errant line that says "Kestrel", which, I believe should be "Darner". It doesn't look good:
kestrel 262024 requests: 465148 kB
kestrel 524048 requests: 520476 kB
kestrel 1048576 requests: 611612 kB
darner 0 requests: 2220 kB
darner 1024 requests: 3492 kB
darner 2048 requests: 5872 kB
darner 4096 requests: 8136 kB
darner 8192 requests: 15520 kB
darner 16384 requests: 25656 kB
darner 32768 requests: 27412 kB
darner 65536 requests: 24324 kB
darner 131072 requests: 28440 kB
darner 262024 requests: 28524 kB
darner 524048 requests: 32104 kB
kestrel 1048576 requests: 33848 kB
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.