apple / foundationdb Goto Github PK
View Code? Open in Web Editor NEWFoundationDB - the open source, distributed, transactional key-value store
Home Page: https://apple.github.io/foundationdb/
License: Apache License 2.0
FoundationDB - the open source, distributed, transactional key-value store
Home Page: https://apple.github.io/foundationdb/
License: Apache License 2.0
Hello,
I believe it's technically possible to redirect all SQLite's read and write operations to Foundationdb, using Sqlite's so-call Virtual Table mechanism (http://www.sqlite.org/vtab.html).
But the question is how feasible it is?
If someone implement such idea (Sql layer via Sqlite's virtual table mechanism), will the fact that SQLite supports only a single connection prevent such idea to become scale-able (in terms of foundationdb scalability)?
PS, the mORMot framework (for HTTP/REST, DB Access Engine and ORM) is already applying such idea for wrapping access to other database systems using the Sqlite's virtual table approach, and it's working like a charm.
I've been a desktop software developer for a long time so please forgive me if the questions might sound dump :)
branch: master
error desc:
In file included from ../../apple/foundationdb/bindings/go/src/fdb/cluster.go:27:
/usr/local/include/foundationdb/fdb_c.h:35:2: error: Requested API version requires a newer version of this header
#error Requested API version requires a newer version of this header
^
1 error generated.
From https://forums.foundationdb.org/t/setting-cpu-affinity-for-fdbservers/84, it was noted that we don't set CPU affinity for any of the processes we start, and there's likely advantages to doing so. It appears once upon a time, we had code in the resolver that would pin itself to a core, but it was commented out, and I agree doing it from fdbmonitor sounds like a better solution.
Implementing this doesn't sound like too much work, and a quick run through a performance test should tell us if it makes any noticeable impact.
Awesome, I am using CockroachDB for now because I need easy scalling.
It would be great to:
I also heard SQL performance suffer, it would be interesting to address those and benchmark it and compare to other NoSQL databases
I also heard Apple didn't actually used FoundationDB because of some limitations. Huh. :P
Have an amazing day!
In Python 3 bytes and string are different. Hence bytes need to be prefixed explicitly with 'b' in Python 3. This is not necessary in Python 2. The docs don't have the prefix causing error in Python 3.
➜ foundationdb-5.1.5 python
Python 2.7.14 (default, Mar 12 2018, 13:54:56)
[GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import fdb
>>> fdb.api_version(510)
>>> db = fdb.open()
>>> db['hello']
Python 3
➜ foundationdb-5.1.5 python3
Python 3.6.4 (default, Mar 12 2018, 13:42:53)
[GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import fdb
>>> fdb.api_version(510)
>>> db = fdb.open()
>>> db[b'hello']
>>> db['hello'] # Breaks
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/karthikeyansingaravelan/stuff/python/foundationdb-5.1.5/fdb/impl.py", line 893, in __getitem__
return Database.__database_getitem(self, key)
File "/Users/karthikeyansingaravelan/stuff/python/foundationdb-5.1.5/fdb/impl.py", line 265, in wrapper
ret = func(*largs, **kwargs)
File "/Users/karthikeyansingaravelan/stuff/python/foundationdb-5.1.5/fdb/impl.py", line 946, in __database_getitem
return tr[key].value
File "/Users/karthikeyansingaravelan/stuff/python/foundationdb-5.1.5/fdb/impl.py", line 450, in __getitem__
return self.get(key)
File "/Users/karthikeyansingaravelan/stuff/python/foundationdb-5.1.5/fdb/impl.py", line 410, in get
key = keyToBytes(key)
File "/Users/karthikeyansingaravelan/stuff/python/foundationdb-5.1.5/fdb/impl.py", line 1271, in keyToBytes
raise TypeError('Key must be of type ' + bytes.__name__)
TypeError: Key must be of type bytes
Are there any plans for official Swift bindings for FoundationDB?
It does not limit based on available memory but rather just on disk space (default storage is 1G).
Was there a re-organization of the package structure in the Java API before launch? Seems like the docs/examples use the prefix "com.apple.foundationdb.*" but the packages actually provided do not include the "apple" segment.
Perhaps you guys have a different setup, but I needed to make the following changes to compile on my Mac:
===================================================================
--- fdbserver/fdbserver.actor.cpp (revision f069fdf042094deecfdc65a31c6f4e8b03f0a81b)
+++ fdbserver/fdbserver.actor.cpp (revision bff15c68afe6a370436091e654e4bcd8a24728c0)
@@ -36,6 +36,7 @@
#include "DataDistribution.h"
#include "NetworkTest.h"
#include "IKeyValueStore.h"
+#include <fstream>
#include <stdarg.h>
#include <stdio.h>
#include "pubsub.h"
The link to FoundationDB in the README points to https://www.foundationdb.org/documentation/ (which returns a 404) and should probably be changed to https://apple.github.io/foundationdb/
(The Tutorial link is also broken)
Currently, FoundationDB does not have a make install
target for Linux based machines -- you're expected to use the dpkg
or rpm
which have the right layout (but I have no idea how this is generated). But most distros will be expecting this, and it would be good form and a lot easier to write installers, e.g. like the NixOS installer. It would also be good for macOS. Some of the file layout requirements surprised me (initially); copying from my initial post on the FoundationDB forums:
Please add a
make install
target. This should not rely on any particular packaging file, it should just runinstall
on the files in the right places. Unless I completely missed it (I don't think I did?), this is an oversight that makes 3rd party packaging much more annoying. In one particular annoyance,fdbrestore
,fdbdr
,dr_agent
etc are all actual symlinks to one binary (backup_agent
), and determine their operation based on the exe name. I didn't realize this until I had dug in and read the source code! So instead I had to 'extract' the fdb .dpkg and look at the filesystem hierarchy. Then I mostly replicated that. Butmake install
would have done it for me...Please note that it is essential that a command like
make install
respect the traditionalPREFIX=
andDESTDIR=
environment variables, so that maintainers can install them into arbitrary places, which is vital. These environment variables do not need to have any special meaning to the FoundationDB source code, they merely need to exist, and be respected, when finally copying files around duringmake install
. (FDB, as far as I can see, doesn't rely so much on fixed hardcoded paths outside of/etc/foundationdb
, and even that is fixable in the fdbmonitor file.) I'm sure you're aware of it, but I only mention it because addingmake install
but not adding that will just have distro maintainers right back here. :)
Is it possible to build this on Linux or other unixes?
Link: https://apple.github.io/foundationdb/architecture.html
Architecture.pdf is embedded well in other browsers, but not in Chrome. Possible fix can be to convert it to PNG format. Will be more consistent with other images in documentation (they all are in PNG format).
There is some cryptic information about the blobstore backup option and originally we figure you can just do something like: fdbbackup start --blob_credentials blob-creds -d blobstore://[email protected]/<my_s3_bucket>/<subfolder>
The first thing is that it's trying to connect in http (and not https) so we relaxed the S3 bucket so that it'd allow non-secure connections. Then, it seems like it's not really using whatever that's after the host:port and url params (the <my_s3_bucket>/<subfolder>
part from the example above) but instead tries to do a PUT against /FDB_BACKUP_V2
(that's a constant in the code) which of course fails. At this point, we started to wonder is this really S3 (but the code does expect the endpoint to respond with an S3-like API).
Bottom-line, how does one specify an S3 bucket as the backup destination?
Currently the only way to get the full picture of where shards are located is to get the shards (that's relatively fast, a single iteration through a system key space) and then ask for the storage servers for each of those shard one-by-one. It doesn't help that asking for IP addresses must be throttled so that you don't cause the responsible stateless processes to become pegged. There should be an easier way to accomplish this.
We need this API mainly to build coprocessors that live on the same machine as the DB and needing to figure out what data is local.
If the user adds an explicit read conflict range over one or more keys, then any keys in the write cache are skipped from the explicit conflict range insert. For example, suppose we have an increment counter method that looks like this:
private void incrementCounter(Transaction tr) {
long counterValue = Tuple.fromBytes(tr.snapshot().get(COUNTER_KEY).join()).getLong(0);
tr.set(COUNTER_KEY, Tuple.from(counterValue + 1).pack());
tr.addReadConflictKey(COUNTER_KEY);
}
(Presumably, we are reading at snapshot and then writing and adding a conflict range because we only want to conditionally update the counter, but we always want to know the counter value. This is a little contrived, but there are less contrived examples out there as well.) Then suppose we start two transactions and commit them concurrently:
try (Transaction tr1 = db.createTransaction(); Transaction tr2 = db.createTransaction()) {
CompletableFuture.allOf(tr1.getReadVersion(), tr2.getReadVersion()).join();
incrementCounter(tr1);
incrementCounter(tr2);
tr1.commit().join();
tr2.commit().join();
}
Then both commits succeed, but the counter still says "1" as its final value, despite two transactions concurrently incrementing it.
The user can fix this by correctly ordering their writes and read conflict range operations, but it probably isn't the expected semantics for such a call.
We at @wavefrontHQ, while we use Tuples extensively, we have developed tools to both pack and extract elements efficiently (saving memory allocations and improving performance). We'd like to donate those pieces (since it needs to be maintained with the mainline Java Tuple class) at some point.
Tracking it in this issue in case people are interested.
For instance, things like:
public static byte[] pack(Object... list)
to pack a bunch of elements together without creating an intermediate Tuple objectget(byte[] input, int skip)
calls with the ability to "output" the offset within tuple after extracting the element so that, again, one does not have to parse the whole tuple to get an element (often times, you know a fixed offset and would want to just extract, say, a string at offset 8 for every key you see)Just a tracking issue since we (Wavefront) have something pretty close in the 3.x branch. The idea is to have memory storage nodes abort recovery if the cluster is healthy (and just join as an empty node). It would also abort recovery (if in-flight) should the cluster turn healthy. Most of the time, they spend N minutes (roughly one minute per G of storage_memory) reading from disk for a healthy cluster only to just discard their data.
The Java client should be published to Maven Central so that it is easily accessible to users of Maven and Gradle.
Not sure if there's a place that is documenting the build steps but we have been using the Dockerfile under build and one critical step that's missing is the need to have
export CC=/opt/x-toolchain/bin/x86_64-nptl-linux-gnu-gcc
export CXX=/opt/x-toolchain/bin/x86_64-nptl-linux-gnu-g++
For it to work.
I found that db will be created from a global map in go sdk
when use db in multi go routine,too little connections can't process too many Transactions
how to improve connections count?
a systemd init script would allow easy installation and management on all modern linux platforms
Hello,
Could you please provide some more detailed steps how to build golang bindings. From your docker vm for build I get following error, while C and Python bindings were compiled without errors:
make fdb_go
Checking hgVersion.h
Compiling fdb
# github.com/apple/foundationdb/bindings/go/src/fdb
/foundationdb/lib/libfdb_c.so: undefined reference to `std::istream::ignore(long)'
/foundationdb/lib/libfdb_c.so: undefined reference to `std::basic_istream<wchar_t, std::char_traits<wchar_t> >::ignore(long)'
collect2: error: ld returned 1 exit status
bindings/go/include.mk:98: recipe for target '/foundationdb/bindings/go/build/pkg/linux_amd64/github.com/apple/foundationdb/bindings/go/src/fdb.a' failed
make: *** [/foundationdb/bindings/go/build/pkg/linux_amd64/github.com/apple/foundationdb/bindings/go/src/fdb.a] Error 2
We have historically maintained bindings in Node that wrapped our C bindings. However, it was initially developed for an older version of Node, and as time went on, it became incompatible with modern versions of the language. At this point, it is not particularly useful for Node users due to those incompatibilities. It also could probably benefit from some of the new Node features that have been developed in the intervening releases. As a result, it would seem like we should remove the Node bindings until such time as a suitable replacement is available. If there is demand for such a binding, the old Node bindings will still exist in history, so our previous work could plausibly be used as a jumping off point for someone who wanted to create their own.
How is FoundationDB reaching out to get the internet community on board after open sourcing?
If not yet decided, maybe a github issue thread here is good enough for now.
Sites to get on boarded (found by googling FoundationDB vs ... )
$ /usr/local/bin/fdbcli -C ./client.cluster
Using cluster file `./client.cluster'.
Internal Error @ fdbrpc/FlowTransport.actor.cpp 657:
atos -o fdbcli.debug -arch x86_64 -l 0x101fe7000 0x10233a5cd 0x1022a5e91 0x1022a514d 0x101feca78 0x102361916 0x102349631 0x1020bae5b 0x10200a6de 0x7fff59e48115
[lots more errors]
Server: Custom Docker container based on ubuntu:16.04
that has installed foundationdb-clients_5.1.5-1_amd64.deb
and foundationdb-server_5.1.5-1_amd64.deb
. Using default config.
Client: Installed with macOS installer, FoundationDB-5.1.5.pkg
. macOS 10.13.3.
Full fdbcli
output and logs from server here.
https://apple.github.io/foundationdb/architecture.html
It appears you're trying to use a PDF in an image tag, which won't work. I've attached a PNG you can use on the page, which you can link to the PDF
Hiya,
We make use of gevent's monkey.patch_all
functionality in order to make all of the pure-Python libraries play nicely with gevent. This is pretty successful, but it currently causes the fdb network thread to block the main application thread.
Demo: https://gist.github.com/zorkian/824e86d0d7f001965f62dc8ae7ad28db
This is tested in gevent 1.1.1 as well as the latest 1.3b1.
You can alleviate the blocking by using gevent.monkey.patch_all(thread=False)
, but this currently doesn't work in our environment due to certain modules we use that aren't implemented to use gevent and require threading.
I've been playing around with the fdb library to see if I can make it work, but haven't succeeded as yet. Figured I'd file the issue just to put it on the radar.
What are the current plans for the SQL layer?
As the API stands now, the SET_VERSIONSTAMP_KEY
mutation type allows users to choose an arbitrary location for their versionstamp by appending an offset to the end of their key. In contrast, the SET_VERSIONSTAMP_VALUE
mutation type forces users to place their version at the beginning. The proposal of this issue is thus the following two things:
Because values can be longer than 65 kB, the way to do this that keeps them consistent is to have both mutations use a 4 byte offset instead of a 2 byte one. (In theory, 3 bytes could be used, but probably inventing 24-bit integers for this use case is worse than just sending an extra byte over the wire.) We can API version guard this so that only users who have requested API version 520 or newer will get the change, so old code will continue to work as desired.
I'm interested in providing Rust bindings for FoundationDB. Would there be interest in including these within the FoundationDB repository along side the other languages?
In the file architecture.rst file (https://github.com/apple/foundationdb/blob/master/documentation/sphinx/source/architecture.rst), the link to the Architecture.pdf
file is broken.
Link is https://github.com/apple/foundationdb/blob/master/images/Architecture.pdf
but should be https://github.com/apple/foundationdb/blob/master/documentation/sphinx/source/images/Architecture.pdf
Would you like add asyncio python 3+ bindings?
I dont know why it is missing, according to the Makefile it should be in bin/
just like the other binaries.
Is this a critical process that I need to operate foundationdb? Because I dont see why this simple 1 process "cluster" keeps failing in various ways:
And if I try to add another process to the cluster, even the first fails to connect to the cluster (itself):
need some go examples
see the code comments below
package main
import (
"fmt"
"log"
"github.com/apple/foundationdb/bindings/go/src/fdb"
"github.com/apple/foundationdb/bindings/go/src/fdb/directory"
"github.com/apple/foundationdb/bindings/go/src/fdb/tuple"
)
func main() {
fdb.MustAPIVersion(510)
db := fdb.MustOpenDefault()
nmqDir, err := directory.CreateOrOpen(db, []string{"nmqtest"}, nil)
if err != nil {
log.Fatal(err)
}
testApp := nmqDir.Sub("testapp")
testTopic := "testtopic"
// store messages
_, err = db.Transact(func(tr fdb.Transaction) (ret interface{}, err error) {
for i := 1; i < 70; i++ {
SCKey := testApp.Pack(tuple.Tuple{testTopic, fmt.Sprintf("%02d", i), fmt.Sprintf("%02d", i-1)})
tr.Set(SCKey, []byte("11111"))
}
return
})
// query messages
var keys []string
_, err = db.Transact(func(tr fdb.Transaction) (ret interface{}, err error) {
// I want to query 5 messages from the testTopic from a specify position
// so, is there any solutions for this, may like the python code:
// tr.get_range(begin, fdb.KeySelector.first_greater_than(begin) + 5)
pr, _ := fdb.PrefixRange([]byte("testapp"))
pr.Begin = testApp.Pack(tuple.Tuple{testTopic, fmt.Sprintf("%02d", 4)})
pr.End = testApp.Pack(tuple.Tuple{testTopic, fmt.Sprintf("%02d", 69)})
ir := tr.GetRange(pr, fdb.RangeOptions{Limit: 5}).Iterator()
for ir.Advance() {
k := ir.MustGet().Key
keys = append(keys, string(k))
}
return
})
fmt.Println(keys)
// clear all messages
_, err = db.Transact(func(tr fdb.Transaction) (ret interface{}, err error) {
tr.ClearRange(testApp.Sub(testTopic))
return
})
}
so, please add more examples of tuples range and offset ,thans very much : )
Right now, the flow directory layer does not support snapshot reads. This in and of itself is a shortcoming of the flow bindings, but it is not so bad really. The flow binding tester attempts to get around this with the following bit of code:
if (isDirectory && isSnapshot) {
Version readVersion = wait(instruction->tr->getReadVersion());
instruction->tr = Reference<Transaction>(new Transaction(data->db));
instruction->tr->setVersion(readVersion);
}
The problem is that though this ensures that the "snapshot" transaction has the same view of the database as the regular transaction, it is missing read-your-writes support. This means that the tests can fail if there is something along the lines of:
The flow tester will miss the modifications, so it will print out different results.
I ran into this error when running the tests locally. Here is the input that I can give to the bindingtester to make it reproduce:
python bindingtester.py --test-name directory --num-ops 261 flow --seed 3583312077 --compare
(The number of ops, 261, is the minimum needed to get it to reproduce with that seed.) I was running it against commit 5fdcd62 which, at the time, was the tip of the release-5.2 branch, to the extent that that is important for reproduction. Here is a print out of the relevant section of adding --print-test --all
to the command:
2175. 'RESET'
2176. 'PUSH' 0
2177. 'DIRECTORY_REMOVE_DATABASE'
2178. 'COMMIT'
2179. 'WAIT_FUTURE'
2180. 'RESET'
2181. 'PUSH' 3
2182. 'DIRECTORY_CHANGE'
2183. 'PUSH' 'default-6\x1e\xde-M\\R:\xea\x19\xdd\xe7\xa3\x96\xaa\x9d'
2184. 'PUSH' ''
2185. 'PUSH' u'default1'
2186. 'PUSH' 1
2187. 'DIRECTORY_CREATE_DATABASE'
2188. 'PUSH' 4
2189. 'DIRECTORY_CHANGE'
2190. 'PUSH' u'1'
2191. 'PUSH' 1
2192. 'PUSH' 1
2193. 'DIRECTORY_REMOVE'
2194. 'PUSH' u'\u70c1\uc3f9\ufe98\u34e1\U0002a2b2\uffa3\u6283\u5f6a\u5a34\ufc6c\u7526\u3df9\u0c1e\u6b44\u532e\u7269\u2d1c\ub766\u583b\uacd9\u3b0d\uc34e\u4218\u21f4'
2195. 'PUSH' ()
2196. 'PUSH' 2
2197. 'DIRECTORY_PACK_KEY'
2198. 'DIRECTORY_UNPACK_KEY'
2199. 'PUSH' 27
2200. 'DIRECTORY_CHANGE'
2201. 'PUSH' '\x13\x11\x05'
2202. 'PUSH' 0
2203. 'DIRECTORY_CREATE_SUBSPACE'
2204. 'PUSH' 97
2205. 'DIRECTORY_CHANGE'
2206. 'PUSH' u'2'
2207. 'PUSH' u'3'
2208. 'PUSH' u'3'
2209. 'PUSH' 3
2210. 'PUSH' 0
2211. 'DIRECTORY_MOVE'
2212. 'PUSH' 0
2213. 'PUSH' 1
2214. 'DIRECTORY_LIST_SNAPSHOT'
This leads to the following discrepancy between flow and python:
Incorrect result:
flow - ('tester_output', 'stack', 269, 2214) = '\x01\x021\x00\xff\x00'
python - ('tester_output', 'stack', 269, 2214) = '\x01\x00'
I hacked up the binding tester to actual emit non-snapshot reads whenever it thinks it should actually emit a snapshot read, and then the test passed. So, I don't think it's actually a bug in the directory layer, just that the lack of RYW in snapshot tests.
This can cause spurious errors, so it would be nice to get it fixed. In some sense, the "correct" solution is to make the flow directory layer support snapshot reads. I'm not sure if there is an easier way. In theory, we could just give the snapshot tester a regular transaction. This can lead to a different kind of spurious error (additional not_committed
errors), so that's not air-tight. In theory, we could also log all of the writes and replay them within the snapshot transaction or do something like keep two transactions around and use both of them for writes, but perform non-snapshot reads and commits on one and "snapshot" reads on the other. But this might be harder than just supporting snapshot reads.
It looks like after this change the build container no longer pins the exact versions of every system package installed. It also appears to allow apt to pull in transitive dependencies, rather than specifying all packages explicitly.
This makes it a little bit harder to understand exactly what the build environment depends on, and also prevents our builds from being reproducible. In the worst case, you could imagine a situation where after all the testing has occurred during a release, Ubuntu upstream bumps a package version which causes a subtle change in behavior.
My guess is this was done because in practice, the team at Apple never rebuilds the container, but has a known good copy of it published somewhere? Is that right? Or was there some other rationale?
I think the two most straightforward solutions are either to pin the versions again, or to publish an "official" container, direct people to use it, and keep this file around mostly as documentation.
What do you think?
It will be good if we can publish Python bindings to PyPI so that we can do pip install foundationdb
. Currently bindings are present as a tar.gz file and can be installed as below :
pip install https://www.foundationdb.org/downloads/5.1.5/bindings/python/foundationdb-5.1.5.tar.gz
Similar issue with respect to Java and Maven : #219
This would allow us to inspect directories without adding read conflict ranges, which might be something someone would want to do.
In some sense, this is the root cause of #157. However, that issue points to a bug in the tester that is causing spurious errors. Fixing this issue would give us #157 basically for free, but this somewhat of a separate issue.
Hi,
I followed the build instructions for building on a mac and installed the necessary dependencies. However, the make command does not seem to be doing much. The OS I have is macOS Sierra. Is there something that I could be doing wrong here?
$ make
flow/coveragetool/generated.mk:40: warning: overriding commands for target `GENTARGET'
flow/actorcompiler/generated.mk:40: warning: ignoring old commands for target `GENTARGET'
fdbclient/vexillographer/generated.mk:40: warning: overriding commands for target `GENTARGET'
flow/coveragetool/generated.mk:40: warning: ignoring old commands for target `GENTARGET'
flow/generated.mk:113: warning: overriding commands for target `GENTARGET'
fdbclient/vexillographer/generated.mk:40: warning: ignoring old commands for target `GENTARGET'
fdbrpc/generated.mk:113: warning: overriding commands for target `GENTARGET'
flow/generated.mk:113: warning: ignoring old commands for target `GENTARGET'
fdbclient/generated.mk:113: warning: overriding commands for target `GENTARGET'
fdbrpc/generated.mk:113: warning: ignoring old commands for target `GENTARGET'
fdbbackup/generated.mk:113: warning: overriding commands for target `GENTARGET'
fdbclient/generated.mk:113: warning: ignoring old commands for target `GENTARGET'
fdbcli/generated.mk:113: warning: overriding commands for target `GENTARGET'
fdbbackup/generated.mk:113: warning: ignoring old commands for target `GENTARGET'
bindings/c/generated.mk:113: warning: overriding commands for target `GENTARGET'
fdbcli/generated.mk:113: warning: ignoring old commands for target `GENTARGET'
bindings/java/generated.mk:113: warning: overriding commands for target `GENTARGET'
bindings/c/generated.mk:113: warning: ignoring old commands for target `GENTARGET'
fdbmonitor/generated.mk:113: warning: overriding commands for target `GENTARGET'
bindings/java/generated.mk:113: warning: ignoring old commands for target `GENTARGET'
bindings/flow/tester/generated.mk:113: warning: overriding commands for target `GENTARGET'
fdbmonitor/generated.mk:113: warning: ignoring old commands for target `GENTARGET'
bindings/flow/generated.mk:113: warning: overriding commands for target `GENTARGET'
bindings/flow/tester/generated.mk:113: warning: ignoring old commands for target `GENTARGET'
FDBLibTLS/generated.mk:113: warning: overriding commands for target `GENTARGET'
bindings/flow/generated.mk:113: warning: ignoring old commands for target `GENTARGET'
make: *** No rule to make target `.objs/fdbserver/ClusterController.actor.g.cpp.o', needed by `bin/fdbserver'. Stop.
Building the TLS Plugin requires LibreSSL, which is provided within the docker image for building in Linux, but would need to be installed by hand for MacOS and Windows currently. Doing so, or making it easier to do so, needs to be documented in our build instructions.
installed with FoundationDB-5.1.5.pkg, then start fdbcli, enter set a b
, a error comed out : ERROR: Database is locked (1038)
I remember @ajbeamon mentioning that there's something that needs to be changed in the 3.x code (IIRC?) that will allow us to ship a foundationdb-client package that can talk to a both a 3.x cluster and a 5.x cluster. I can't seem to find that in the code, can someone point us in the right direction?
Perhaps this has already been addressed but we have always observed that 3.x clusters would need to have their process memory limit in /etc/foundationdb/foundationdb.conf increased beyond 8G (we have 10G for a cluster that has 30+TB of KVs (100+TB on disk). Perhaps at that size the recommendation would be to always have separate transaction processes.
As dissussed in the forum (https://forums.foundationdb.org/t/atomic-add-key-1-can-leak-keys-with-value-0-if-you-are-not-careful/148) I'm opening an issue here.
One of the issue we had in the past, was with atomic ADD leaking values decremented to 0, which was a common issue when updating static counters used in indexes.
Say we have an index that adds (..., 0, VALUE, ID) = ''
for each indexed document, and also atomically increments a counter like (..., 1, VALUE) = (64 bits counter)
when a document is indexed for VALUE, and decrements it when the document is later mutated to some other value.
The issue was that decrementing a value down to 0 would keep the key present in the database.
(..., TIMESTAMP) = 00 00 00 00 00 00 00 00
in the dataset which could consume a lot of space in the long run (I had to scrub these multiple times, and they would amount to more than 3/4 of the total data size in one extreme case).Has there been any thing done about this case? I remember asking if maybe decrementing the key down to 0 could maybe also remove it. Or maybe add having a COMPARE_AND_CLEAR
atomic operator which would only delete a key if it its value was equal to the argument.
ie: in pseudo code:
SET((..., 0, VALUE, ID), '')
ATOMIC_ADD((..., 1, VALUE), <FF FF FF FF>)
ATOMIC_COMPARE_AND_CLEAR((..., 1, VALUE), <00 00 00 00>)
I had to do some custom logic which would automatically disable this feature when the attribute was of type DateTime or Guid, but this was a bit clunky.
The specifications for the bindings tester are currently in "txt" formatted files. The files are found under bindings/bindingtester/spec
. Ideally, it could be converted to Markdown for cleaner rendering.
Will be the SQL-Parser part of the Open Source version ?
I would be super excited to see Lua(JIT) bindings.
Lua has coroutines, which appear to be a good match to FD's Futures.
Going to the FoundationDB.org SiteMap, I'd expected there to be some mention of the word "SSL" or "TLS" or "Administration", but there seems to be none.
Inevitably, I found both of these using the search bar (initially I could only find TLS in the repository code here, but I found it eventually by searching "Transport Layer Security" very specifically using the Sphinx search functionality). But these should really be linked up front -- they're quite important for real deployments!
Hi
It is most likely a user error.
$ c++ --version
Apple LLVM version 9.1.0 (clang-902.0.39.1)
Target: x86_64-apple-darwin17.5.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
fdbclient/MultiVersionAssignmentVars.h:135:2: note: destructor has a implicit non-throwing exception specification
~DLThreadSingleAssignmentVar() {
^
fdbclient/MultiVersionAssignmentVars.h:138:4: error: '~DLThreadSingleAssignmentVar' has a non-throwing exception specification but can still throw [-Werror,-Wexceptions]
ASSERT(futureRefCount == 1);
^
./flow/Error.h:84:81: note: expanded from macro 'ASSERT'
#define ASSERT( condition ) if (!((condition) || isAssertDisabled(LINE))) { throw internal_error(); }
I can work around it by adding noexcept(false) to the destructors, but is there a better way?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.