dain / leveldb Goto Github PK

Port of LevelDB to Java

License: Apache License 2.0

Java 100.00%

leveldb's Introduction

LevelDB in Java

This is a rewrite (port) of LevelDB in Java. This goal is to have a feature complete implementation that is within 10% of the performance of the C++ original and produces byte-for-byte exact copies of the C++ code.

Current status

Currently the code base is basically functional, but only trivially tested. In some places, this code is a literal conversion of the C++ code and in others it has been converted to a more natural Java style. The plan is to leave the code closer to the C++ original until the baseline performance has been established.

API Usage:

Recommended Package imports:

import org.iq80.leveldb.*;
import static org.iq80.leveldb.impl.Iq80DBFactory.*;
import java.io.*;

Opening and closing the database.

Options options = new Options();
options.createIfMissing(true);
DB db = factory.open(new File("example"), options);
try {
  // Use the db in here....
} finally {
  // Make sure you close the db to shutdown the 
  // database and avoid resource leaks.
  db.close();
}

Putting, Getting, and Deleting key/values.

db.put(bytes("Tampa"), bytes("rocks"));
String value = asString(db.get(bytes("Tampa")));
db.delete(bytes("Tampa"), wo);

Performing Batch/Bulk/Atomic Updates.

WriteBatch batch = db.createWriteBatch();
try {
  batch.delete(bytes("Denver"));
  batch.put(bytes("Tampa"), bytes("green"));
  batch.put(bytes("London"), bytes("red"));

  db.write(batch);
} finally {
  // Make sure you close the batch to avoid resource leaks.
  batch.close();
}

Iterating key/values.

DBIterator iterator = db.iterator();
try {
  for(iterator.seekToFirst(); iterator.hasNext(); iterator.next()) {
    String key = asString(iterator.peekNext().getKey());
    String value = asString(iterator.peekNext().getValue());
    System.out.println(key+" = "+value);
  }
} finally {
  // Make sure you close the iterator to avoid resource leaks.
  iterator.close();
}

Working against a Snapshot view of the Database.

ReadOptions ro = new ReadOptions();
ro.snapshot(db.getSnapshot());
try {
  
  // All read operations will now use the same 
  // consistent view of the data.
  ... = db.iterator(ro);
  ... = db.get(bytes("Tampa"), ro);

} finally {
  // Make sure you close the snapshot to avoid resource leaks.
  ro.snapshot().close();
}

Using a custom Comparator.

DBComparator comparator = new DBComparator(){
    public int compare(byte[] key1, byte[] key2) {
        return new String(key1).compareTo(new String(key2));
    }
    public String name() {
        return "simple";
    }
    public byte[] findShortestSeparator(byte[] start, byte[] limit) {
        return start;
    }
    public byte[] findShortSuccessor(byte[] key) {
        return key;
    }
};
Options options = new Options();
options.comparator(comparator);
DB db = factory.open(new File("example"), options);

Disabling Compression

Options options = new Options();
options.compressionType(CompressionType.NONE);
DB db = factory.open(new File("example"), options);

Configuring the Cache

Options options = new Options();
options.cacheSize(100 * 1048576); // 100MB cache
DB db = factory.open(new File("example"), options);

Getting approximate sizes.

long[] sizes = db.getApproximateSizes(new Range(bytes("a"), bytes("k")), new Range(bytes("k"), bytes("z")));
System.out.println("Size: "+sizes[0]+", "+sizes[1]);

Getting database status.

String stats = db.getProperty("leveldb.stats");
System.out.println(stats);

Getting informational log messages.

Logger logger = new Logger() {
  public void log(String message) {
    System.out.println(message);
  }
};
Options options = new Options();
options.logger(logger);
DB db = factory.open(new File("example"), options);

Destroying a database.

Options options = new Options();
factory.destroy(new File("example"), options);

Projects using this port of LevelDB

ActiveMQ Apollo: Defaults to using leveldbjni, but falls back to this port if the jni port is not available on your platform.

leveldb's People

Contributors

Stargazers

Watchers

Forkers

martint nathanchen ealliaume xiaoniudu2008 andywang85 hszander gnomix supertango seh gaul athens-team stevenschlansker abramsm gladwig jackiexl zettaset-kpelykh ammasajan dfrankow fenderstr tabish121 franklinsun kirill-sidorchuk netcomm tlvenn wasserfuhr sofartogo stonegao gridline abeep qp0609 omarbenhamid myaniu achun2080 xuzha jclawson xq0804200134 xiexi elandau paulscott56 duffqiu ibodrov mjohh dengliu placrosse savechina hekima electrum zhouheming clojure-cn jankotek jerryba zhangya lawrencewu mapbased kidnox yeyinqcri wmefan samsara00 tamkhanevivek johnkilloran coflys yangcheng shjgiser kingdouustc tinfoiled alexshvid tj1633 suxinde2009 hesy007 nathangwang phantomwl okhiwell ssyue chakra-coder uchenm dinesh-rt liaor binarytemple ligi hbf impulsehu tenderice techend apsaltis qq254963746 jziub licarth archiecobbs chengc017 zhenchuan applied-duality yynil codenfish maxboehm aahmed-se rlxrlxrlx ivywjhua zhengkejun gaoch023 camillepierre

leveldb's Issues

File Partitioning

Well this is not really an issue. I want to know if there is this feature.

When adding data to database, the file is going to grow size. If the size grows huge, is there the feature of file splitting? will the db be save in different files?

File format is not yet compatible with the C++ implementation of leveldb

Run the NativeInteropTest see the compatibility failure.

Question: Are all files static when compaction is suspended?

Hi, I'm looking to back-up a running database after ensuring that the storage and log files have ceased changing. To do this I first stop processing writes, and then call suspendCompaction(). Is this sufficient to ensure that the files have 'stopped moving'?

Related question, is it guaranteed that the compactions have stopped completely when suspendCompaction() returns?

thank you.

NullPointerException on Android

Hey Fellows,

LevelDB works fine for me on Android until
I bumped into this issue:

I get into VersionSet.numberOfFilesInLevel()
where current == null and then get an NPE ,
I will digg deeper by myself but I will appreciate
any hint about what kind role is for this
version files ? why possible it can be null ?

Thanks in advance.

read (get) performance at scale?

I'm hoping for some help, or at least some feedback.
I'm getting wonderful performance with small databases, say 4 million rows or less, but dramatically worse performance as the dataset grows. At 20 M rows, I get performance of about 3,000 random reads per second, with a large (4GB) cache. I don't use Compression because most of the data is already compressed. The original LDB C++ benchmark says they get better than 129,000 random reads per second with 100 byte keys and a million records.

Here is my configuration:
leveldb version 0.7
Java 8
new OSX MacBookPro. 16GB ram, SSD, 4 Cores.
values are 800 bytes each

option values:
private static final int MAX_OPEN_FILES = 7500;
private static final long CACHE_SIZE_MB = 4000;
private static final int WRITE_BUFFER_MB = 200;
private static final int BLOCK_RESTART_INTERVAL = 32;

Almost all the time is in the call to db.get(). Does this kind of performance seem reasonable?
Also wondering if anyone use this version for 10s to 100s of millions of records, or is this the best it can do?

LevelDB in read-only filesystem

Is it possible to migrate LevelDB dump on a read-only filesystem to do read operations??

Build Failure

I am getting a build failure. Here is the relevant info:

Test Methods
Passed Scenarios
Passed # skipped # failed Total
Time Included
Groups Excluded
Groups
Command line test 73 73 0 1 73.5 seconds
Class Method # of
Scenarios Time
(Msecs)
Command line test — failed
org.iq80.leveldb.impl.DbImplTest testCantCreateDirectoryReturnMessage 1 1
Command line test — passed
org.iq80.leveldb.util.PureJavaCrc32CTest testProducesDifferentCrcs 1 0
org.iq80.leveldb.impl.LogTest testMultipleLargeRecords 1 4
org.iq80.leveldb.util.PureJavaCrc32CTest testCrc 5 1
org.iq80.leveldb.impl.LogTest testSmallRecord 1 0
testReadWithoutProperClose 1 1
org.iq80.leveldb.impl.DbImplTest testApproximateSizes 1 1463
org.iq80.leveldb.util.VariableLengthQuantityTest testWriteVariableLengthLong 1 0
testWriteVariableLengthInt 1 0
org.iq80.leveldb.impl.DbImplTest testCompactionsGenerateMultipleFiles 1 857
testRecoveryWithEmptyLog 1 31
testSnapshot 1 9
testRecover 1 190
testEmpty 1 9
org.iq80.leveldb.util.PureJavaCrc32CTest testCrc 5 1
org.iq80.leveldb.impl.LogTest testEmptyBlock 1 1
org.iq80.leveldb.impl.DbImplTest testHiddenValuesAreRemoved 1 152
org.iq80.leveldb.util.PureJavaCrc32CTest testCrc 5 1
org.iq80.leveldb.table.FileChannelTableTest testEmptyBlock 1 1
org.iq80.leveldb.table.MMapTableTest testSingleEntrySingleBlock 1 2
org.iq80.leveldb.impl.DbImplTest testSymbolicLinkForFileWithoutParent 1 0
org.iq80.leveldb.table.MMapTableTest testMultipleEntriesWithSingleBlock 1 9
org.iq80.leveldb.impl.DbImplTest testEmptyBatch 1 14
testGetSnapshot 1 169
org.iq80.leveldb.table.BlockTest testMultipleEntriesWithSharedKey 1 1
org.iq80.leveldb.util.PureJavaCrc32CTest testCrc 5 1
org.iq80.leveldb.impl.DbImplTest testGetOrderedByLevels 1 14
testReadWrite 1 16
testEmptyDb 1 10
org.iq80.leveldb.table.MMapTableTest testMultipleEntriesWithMultipleBlock 1 4
org.iq80.leveldb.impl.DbImplTest testMultiPassMultipleEntries 1 20
org.iq80.leveldb.table.FileChannelTableTest testMultipleEntriesWithSingleBlock 1 12
org.iq80.leveldb.impl.DbImplTest testIteratorSingle 1 6
testPutDeleteGet 1 14
testBackgroundCompaction 1 3105
org.iq80.leveldb.table.MMapTableTest testEmptyFile 1 1
org.iq80.leveldb.table.BlockTest testEmptyBlock 1 0
org.iq80.leveldb.table.FileChannelTableTest testSingleEntrySingleBlock 1 2
org.iq80.leveldb.impl.DbImplTest testEmptyIterator 1 7
org.iq80.leveldb.table.BlockTest testMultipleEntriesWithNonSharedKey 1 0
org.iq80.leveldb.impl.DbImplTest testSparseMerge 1 16983
testGetFromImmutableLayer 1 11
testDeletionMarkers2 1 24
org.iq80.leveldb.impl.LogTest testMultipleSmallRecords 1 1
org.iq80.leveldb.table.BlockTest testMultipleEntriesWithNonSharedKeyAndRestartPositions 1 1
org.iq80.leveldb.util.PureJavaCrc32CTest testCrc 5 1
org.iq80.leveldb.util.SliceComparatorTest testSliceComparison 1 1
org.iq80.leveldb.table.BlockTest testSingleEntry 1 0
org.iq80.leveldb.impl.DbImplTest testRecoverDuringMemtableCompaction 1 139
testApproximateSizesMixOfSmallAndLarge 1 281
testMultipleEntries 1 44
org.iq80.leveldb.impl.LogTest testLargeRecord 1 1
org.iq80.leveldb.impl.DbImplTest testGetLevel0Ordering 1 16
org.iq80.leveldb.util.PureJavaCrc32CTest testMask 1 0
org.iq80.leveldb.table.BlockTest testEmptyBuffer 1 0
org.iq80.leveldb.table.FileChannelTableTest testMultipleEntriesWithMultipleBlock 1 6
org.iq80.leveldb.util.PureJavaCrc32CTest testComposes 1 1
org.iq80.leveldb.impl.TestMMapLogWriter testLogRecordBounds 1 2
org.iq80.leveldb.impl.DbImplTest testSymbolicLinkForFileWithParent 1 1
testMinorCompactionsHappen 1 656
testIteratorMultiple 1 10
org.iq80.leveldb.table.MMapTableTest testEmptyBlock 1 1
org.iq80.leveldb.impl.TestFileChannelLogWriter testLogRecordBounds 1 6
org.iq80.leveldb.impl.DbImplTest testRepeatedWritesToSameKey 1 3393
testGetFromVersions 1 13
testDBDirectoryIsFileRetrunMessage 1 1
testIteratorPinsRef 1 101
org.iq80.leveldb.table.FileChannelTableTest testEmptyFile 1 0
org.iq80.leveldb.impl.DbImplTest testGetPicksCorrectFile 1 16
testSingleEntrySingle 1 177
testCompactionsOnBigDataSet 1 44945
testRecoverWithLargeLog 1 23
org.iq80.leveldb.table.BlockTest testMultipleEntriesWithSharedKeyAndRestartPositions 1 3
org.iq80.leveldb.impl.DbImplTest testDeletionMarkers1 1 43
Command line test

org.iq80.leveldb.impl.DbImplTest:testCantCreateDirectoryReturnMessage

Expected exception java.lang.IllegalArgumentException but got org.testng.TestException: The exception was thrown with the wrong message: expected "Database directory '/foo/bar/doowop/idontexist'.*" but got "Database '\foo\bar\doowop\idontexist' exists and the error if exists option is enabled"

org.testng.internal.Invoker.handleInvocationResults(Invoker.java:1416)
at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1184)
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
at org.testng.TestRunner.runWorkers(TestRunner.java:1147)
at org.testng.TestRunner.privateRun(TestRunner.java:749)
at org.testng.TestRunner.run(TestRunner.java:600)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:317)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:312)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:274)
at org.testng.SuiteRunner.run(SuiteRunner.java:223)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1039)
at org.testng.TestNG.runSuitesLocally(TestNG.java:964)
at org.testng.TestNG.run(TestNG.java:900)
at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:70)
at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeMulti(TestNGDirectoryTestSuite.java:149)
at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:95)
at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:114)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.maven.surefire.booter.ProviderFactory$ClassLoaderProxy.invoke(ProviderFactory.java:103)
at $Proxy0.invoke(Unknown Source)
at org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:150)
at org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcess(SurefireStarter.java:91)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:69)
1 lines not shown
Caused by The exception was thrown with the wrong message: expected "Database directory '/foo/bar/doowop/idontexist'.*" but got "Database '\foo\bar\doowop\idontexist' exists and the error if exists option is enabled"

org.testng.internal.Invoker.handleInvocationResults(Invoker.java:1409)
at org.testng.internal.Invoker.invokeMethod(Invoker.java:722)
at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:846)
at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1170)
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
at org.testng.TestRunner.runWorkers(TestRunner.java:1147)
at org.testng.TestRunner.privateRun(TestRunner.java:749)
at org.testng.TestRunner.run(TestRunner.java:600)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:317)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:312)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:274)
at org.testng.SuiteRunner.run(SuiteRunner.java:223)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1039)
at org.testng.TestNG.runSuitesLocally(TestNG.java:964)
at org.testng.TestNG.run(TestNG.java:900)
at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:70)
at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeMulti(TestNGDirectoryTestSuite.java:149)
at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:95)
at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:114)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.maven.surefire.booter.ProviderFactory$ClassLoaderProxy.invoke(ProviderFactory.java:103)
at $Proxy0.invoke(Unknown Source)
at org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:150)
at org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcess(SurefireStarter.java:91)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:69)
1 lines not shown
Caused by Database '\foo\bar\doowop\idontexist' exists and the error if exists option is enabled

com.google.common.base.Preconditions.checkArgument(Preconditions.java:119)
at org.iq80.leveldb.impl.DbImpl.(DbImpl.java:175)
at org.iq80.leveldb.impl.DbImplTest$DbStringWrapper.(DbImplTest.java:963)
at org.iq80.leveldb.impl.DbImplTest$DbStringWrapper.(DbImplTest.java:952)
at org.iq80.leveldb.impl.DbImplTest.testCantCreateDirectoryReturnMessage(DbImplTest.java:800)
36 lines not shown
back to summary

org.iq80.leveldb.util.PureJavaCrc32CTest:testProducesDifferentCrcs

back to summary

org.iq80.leveldb.impl.LogTest:testMultipleLargeRecords

back to summary

org.iq80.leveldb.util.PureJavaCrc32CTest:testCrc

Parameter #1 Parameter #2
1188919630 [B@40ef965e
-1970194774 [B@756535fa
1655221059 [B@47baec4c
289397596 [B@317916a0
-644466090 [B@4bc7bc0
back to summary

org.iq80.leveldb.impl.LogTest:testSmallRecord

back to summary

org.iq80.leveldb.impl.LogTest:testReadWithoutProperClose

back to summary

org.iq80.leveldb.impl.DbImplTest:testApproximateSizes

back to summary

org.iq80.leveldb.util.VariableLengthQuantityTest:testWriteVariableLengthLong

back to summary

org.iq80.leveldb.util.VariableLengthQuantityTest:testWriteVariableLengthInt

back to summary

org.iq80.leveldb.impl.DbImplTest:testCompactionsGenerateMultipleFiles

back to summary

org.iq80.leveldb.impl.DbImplTest:testRecoveryWithEmptyLog

back to summary

org.iq80.leveldb.impl.DbImplTest:testSnapshot

back to summary

org.iq80.leveldb.impl.DbImplTest:testRecover

back to summary

org.iq80.leveldb.impl.DbImplTest:testEmpty

back to summary

org.iq80.leveldb.util.PureJavaCrc32CTest:testCrc

Parameter #1 Parameter #2
1188919630 [B@40ef965e
-1970194774 [B@756535fa
1655221059 [B@47baec4c
289397596 [B@317916a0
-644466090 [B@4bc7bc0
back to summary

org.iq80.leveldb.impl.LogTest:testEmptyBlock

back to summary

org.iq80.leveldb.impl.DbImplTest:testHiddenValuesAreRemoved

back to summary

org.iq80.leveldb.util.PureJavaCrc32CTest:testCrc

Parameter #1 Parameter #2
1188919630 [B@40ef965e
-1970194774 [B@756535fa
1655221059 [B@47baec4c
289397596 [B@317916a0
-644466090 [B@4bc7bc0
back to summary

org.iq80.leveldb.table.FileChannelTableTest:testEmptyBlock

back to summary

org.iq80.leveldb.table.MMapTableTest:testSingleEntrySingleBlock

back to summary

org.iq80.leveldb.impl.DbImplTest:testSymbolicLinkForFileWithoutParent

back to summary

org.iq80.leveldb.table.MMapTableTest:testMultipleEntriesWithSingleBlock

back to summary

org.iq80.leveldb.impl.DbImplTest:testEmptyBatch

back to summary

org.iq80.leveldb.impl.DbImplTest:testGetSnapshot

back to summary

org.iq80.leveldb.table.BlockTest:testMultipleEntriesWithSharedKey

back to summary

org.iq80.leveldb.util.PureJavaCrc32CTest:testCrc

Parameter #1 Parameter #2
1188919630 [B@40ef965e
-1970194774 [B@756535fa
1655221059 [B@47baec4c
289397596 [B@317916a0
-644466090 [B@4bc7bc0
back to summary

org.iq80.leveldb.impl.DbImplTest:testGetOrderedByLevels

back to summary

org.iq80.leveldb.impl.DbImplTest:testReadWrite

back to summary

org.iq80.leveldb.impl.DbImplTest:testEmptyDb

back to summary

org.iq80.leveldb.table.MMapTableTest:testMultipleEntriesWithMultipleBlock

back to summary

org.iq80.leveldb.impl.DbImplTest:testMultiPassMultipleEntries

back to summary

org.iq80.leveldb.table.FileChannelTableTest:testMultipleEntriesWithSingleBlock

back to summary

org.iq80.leveldb.impl.DbImplTest:testIteratorSingle

back to summary

org.iq80.leveldb.impl.DbImplTest:testPutDeleteGet

back to summary

org.iq80.leveldb.impl.DbImplTest:testBackgroundCompaction

back to summary

org.iq80.leveldb.table.MMapTableTest:testEmptyFile

File is corrupt: size must be at least 48 bytes

com.google.common.base.Preconditions.checkArgument(Preconditions.java:119)
at org.iq80.leveldb.table.Table.(Table.java:50)
at org.iq80.leveldb.table.MMapTable.(MMapTable.java:44)
at org.iq80.leveldb.table.MMapTableTest.createTable(MMapTableTest.java:29)
at org.iq80.leveldb.table.TableTest.testEmptyFile(TableTest.java:54)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:76)
at org.testng.internal.Invoker.invokeMethod(Invoker.java:673)
at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:846)
at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1170)
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
at org.testng.TestRunner.runWorkers(TestRunner.java:1147)
at org.testng.TestRunner.privateRun(TestRunner.java:749)
at org.testng.TestRunner.run(TestRunner.java:600)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:317)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:312)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:274)
at org.testng.SuiteRunner.run(SuiteRunner.java:223)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1039)
at org.testng.TestNG.runSuitesLocally(TestNG.java:964)
at org.testng.TestNG.run(TestNG.java:900)
at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:70)
at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeMulti(TestNGDirectoryTestSuite.java:149)
at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:95)
at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:114)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.maven.surefire.booter.ProviderFactory$ClassLoaderProxy.invoke(ProviderFactory.java:103)
at $Proxy0.invoke(Unknown Source)
at org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:150)
at org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcess(SurefireStarter.java:91)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:69)
1 lines not shown
back to summary

org.iq80.leveldb.table.BlockTest:testEmptyBlock

back to summary

org.iq80.leveldb.table.FileChannelTableTest:testSingleEntrySingleBlock

back to summary

org.iq80.leveldb.impl.DbImplTest:testEmptyIterator

back to summary

org.iq80.leveldb.table.BlockTest:testMultipleEntriesWithNonSharedKey

back to summary

org.iq80.leveldb.impl.DbImplTest:testSparseMerge

back to summary

org.iq80.leveldb.impl.DbImplTest:testGetFromImmutableLayer

back to summary

org.iq80.leveldb.impl.DbImplTest:testDeletionMarkers2

back to summary

org.iq80.leveldb.impl.LogTest:testMultipleSmallRecords

back to summary

org.iq80.leveldb.table.BlockTest:testMultipleEntriesWithNonSharedKeyAndRestartPositions

back to summary

org.iq80.leveldb.util.PureJavaCrc32CTest:testCrc

Parameter #1 Parameter #2
1188919630 [B@40ef965e
-1970194774 [B@756535fa
1655221059 [B@47baec4c
289397596 [B@317916a0
-644466090 [B@4bc7bc0
back to summary

org.iq80.leveldb.util.SliceComparatorTest:testSliceComparison

back to summary

org.iq80.leveldb.table.BlockTest:testSingleEntry

back to summary

org.iq80.leveldb.impl.DbImplTest:testRecoverDuringMemtableCompaction

back to summary

org.iq80.leveldb.impl.DbImplTest:testApproximateSizesMixOfSmallAndLarge

back to summary

org.iq80.leveldb.impl.DbImplTest:testMultipleEntries

back to summary

org.iq80.leveldb.impl.LogTest:testLargeRecord

back to summary

org.iq80.leveldb.impl.DbImplTest:testGetLevel0Ordering

back to summary

org.iq80.leveldb.util.PureJavaCrc32CTest:testMask

back to summary

org.iq80.leveldb.table.BlockTest:testEmptyBuffer

Block is corrupt: size must be at least 4 block

com.google.common.base.Preconditions.checkArgument(Preconditions.java:119)
at org.iq80.leveldb.table.Block.(Block.java:74)
at org.iq80.leveldb.table.BlockTest.testEmptyBuffer(BlockTest.java:37)
36 lines not shown
back to summary

org.iq80.leveldb.table.FileChannelTableTest:testMultipleEntriesWithMultipleBlock

back to summary

org.iq80.leveldb.util.PureJavaCrc32CTest:testComposes

back to summary

org.iq80.leveldb.impl.TestMMapLogWriter:testLogRecordBounds

back to summary

org.iq80.leveldb.impl.DbImplTest:testSymbolicLinkForFileWithParent

back to summary

org.iq80.leveldb.impl.DbImplTest:testMinorCompactionsHappen

back to summary

org.iq80.leveldb.impl.DbImplTest:testIteratorMultiple

back to summary

org.iq80.leveldb.table.MMapTableTest:testEmptyBlock

back to summary

org.iq80.leveldb.impl.TestFileChannelLogWriter:testLogRecordBounds

back to summary

org.iq80.leveldb.impl.DbImplTest:testRepeatedWritesToSameKey

back to summary

org.iq80.leveldb.impl.DbImplTest:testGetFromVersions

back to summary

org.iq80.leveldb.impl.DbImplTest:testDBDirectoryIsFileRetrunMessage

Database directory 'C:\Users\Alex\AppData\Local\Temp\leveldb-1362273429457-0\imafile' is not a directory

com.google.common.base.Preconditions.checkArgument(Preconditions.java:119)
at org.iq80.leveldb.impl.DbImpl.(DbImpl.java:162)
at org.iq80.leveldb.impl.DbImplTest$DbStringWrapper.(DbImplTest.java:963)
at org.iq80.leveldb.impl.DbImplTest$DbStringWrapper.(DbImplTest.java:952)
at org.iq80.leveldb.impl.DbImplTest.testDBDirectoryIsFileRetrunMessage(DbImplTest.java:809)
36 lines not shown
back to summary

org.iq80.leveldb.impl.DbImplTest:testIteratorPinsRef

back to summary

org.iq80.leveldb.table.FileChannelTableTest:testEmptyFile

File is corrupt: size must be at least 48 bytes

com.google.common.base.Preconditions.checkArgument(Preconditions.java:119)
at org.iq80.leveldb.table.Table.(Table.java:50)
at org.iq80.leveldb.table.FileChannelTable.(FileChannelTable.java:34)
at org.iq80.leveldb.table.FileChannelTableTest.createTable(FileChannelTableTest.java:29)
at org.iq80.leveldb.table.TableTest.testEmptyFile(TableTest.java:54)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:76)
at org.testng.internal.Invoker.invokeMethod(Invoker.java:673)
at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:846)
at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1170)
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
at org.testng.TestRunner.runWorkers(TestRunner.java:1147)
at org.testng.TestRunner.privateRun(TestRunner.java:749)
at org.testng.TestRunner.run(TestRunner.java:600)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:317)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:312)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:274)
at org.testng.SuiteRunner.run(SuiteRunner.java:223)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1039)
at org.testng.TestNG.runSuitesLocally(TestNG.java:964)
at org.testng.TestNG.run(TestNG.java:900)
at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:70)
at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeMulti(TestNGDirectoryTestSuite.java:149)
at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:95)
at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:114)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.maven.surefire.booter.ProviderFactory$ClassLoaderProxy.invoke(ProviderFactory.java:103)
at $Proxy0.invoke(Unknown Source)
at org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:150)
at org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcess(SurefireStarter.java:91)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:69)
1 lines not shown
back to summary

org.iq80.leveldb.impl.DbImplTest:testGetPicksCorrectFile

back to summary

org.iq80.leveldb.impl.DbImplTest:testSingleEntrySingle

back to summary

org.iq80.leveldb.impl.DbImplTest:testCompactionsOnBigDataSet

back to summary

org.iq80.leveldb.impl.DbImplTest:testRecoverWithLargeLog

back to summary

org.iq80.leveldb.table.BlockTest:testMultipleEntriesWithSharedKeyAndRestartPositions

back to summary

org.iq80.leveldb.impl.DbImplTest:testDeletionMarkers1

More Relevant Info:
Product Version: NetBeans IDE 7.0.1 (Build 201107282000)
Java: 1.7.0_02; Java HotSpot(TM) 64-Bit Server VM 22.0-b10
System: Windows 7 version 6.1 running on amd64; Cp1252; en_US (nb)

NPE after writting an empty batch, closing and re-opening

Hi,

in a simple set of integration tests i came across this issue:

open a store in a specific directory, using options to create it if it's missing
create a WriteBatch, leave it empty, and write it to the db
close the WriteBatch and the store
re-open the store, using the same options
crashes with an NPE, stack trace as below

java.lang.NullPointerException
    at org.iq80.leveldb.impl.DbImpl.writeLevel0Table(DbImpl.java:900)
at org.iq80.leveldb.impl.DbImpl.recoverLogFile(DbImpl.java:522)
at org.iq80.leveldb.impl.DbImpl.<init>(DbImpl.java:184)
at org.iq80.leveldb.impl.Iq80DBFactory.open(Iq80DBFactory.java:60)
at at.knowcenter.storage.leveldb.LeveldbStorage.<init>(LeveldbStorage.java:86)

Code to reproduce the issue:

Options options = new Options();
options.createIfMissing(true);                  
db = new Iq80DBFactory().open(directory, options);
WriteBatch batch = db.createWriteBatch();
batch.close();
db.write(batch);
db.close();
db = new Iq80DBFactory().open(directory, options);

Writting an empty batch to the db is of course a bit bollocks, however, i vote for db.write(WriteBatch batch) to check if there's actually something to be written. I can of course work around this issue easily, but it would lower the amount of surprises new users of this lib encounter :)

I didn't have time to look into the leveldb code itself yet, but i'd assume it's an easy to fix issue.

Thanks for your work!
Mario

Background-Compaction fails

I got following exception when datasize about to 110G, abount 1800,000,000 rows.
need help,thanks

       Options options = new Options();
    options.createIfMissing(false);

    options.compressionType(CompressionType.SNAPPY);

    DBFactory factory = Iq80DBFactory.factory;


    DB db;
    try {
        db = factory.open(from, options);
    } catch (IOException e) {
        throw new java.lang.RuntimeException(e);
    }

org.iq80.leveldb.impl.DbImpl$BackgroundProcessingException: java.lang.NullPointerException
at org.iq80.leveldb.impl.DbImpl.checkBackgroundException(DbImpl.java:411)
at org.iq80.leveldb.impl.DbImpl.get(DbImpl.java:572)
at org.iq80.leveldb.impl.DbImpl.get(DbImpl.java:565)
............................................................................................
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
at org.iq80.leveldb.impl.Compaction.totalFileSize(Compaction.java:129)
at org.iq80.leveldb.impl.Compaction.isTrivialMove(Compaction.java:120)
at org.iq80.leveldb.impl.DbImpl.backgroundCompaction(DbImpl.java:468)
at org.iq80.leveldb.impl.DbImpl.backgroundCall(DbImpl.java:426)
at org.iq80.leveldb.impl.DbImpl.access$100(DbImpl.java:83)
at org.iq80.leveldb.impl.DbImpl$2.call(DbImpl.java:396)
at org.iq80.leveldb.impl.DbImpl$2.call(DbImpl.java:390)

Large SST files (ignoring VersionSet.TARGET_FILE_SIZE?)

In my databases, SSTable file sizes are more or less the same within an individual db. Varying either the content or db options, can cause sst files to be ~2MB in one database, ~23MB in another, ~39MB in a third, etc. That's the approximate size of every sst file in the system, in databases with dozens to hundreds of SST files.

Is there something I can do to control this? The large db files perform poorly on read-heavy loads.

thanks,

NullPointerException in DbImpl.get

Using the current head with some code of mine causes an NPE in leveldb:

java.lang.NullPointerException
at org.iq80.leveldb.impl.DbImpl.get(DbImpl.java:588)
at org.iq80.leveldb.impl.DbImpl.get(DbImpl.java:562)
...

Debugging into the issue got me to MemTable.get[line 77], i.e. the MemTable returns a "deleted" result, which has the value field set to null. Consequently, DbImpl.get[line 588] throws the exception when calling getBytes() on the result.

I guess the fix would be just to add another check for lookupResult.getValue()==null in the code (just as in the lines before). However, as I'm new to the code-base, I might be missing something.

Unfortunately, I could not find a minimal example to reproduce this issue. All smaller tests I wrote ran fine and the full application only crashes after a couple of seconds. I don't know under which conditions the immutableMemTable is used at all.

BTW our code uses multiple threads, but seeing all those locks in your code, I guess this can not be the cause for the NPE.

Background-Compaction fails

Test-Scenario:

        Options options = new Options();
        options.maxOpenFiles(100);
        options.createIfMissing(true);
        File f = new File("/tmp/leveldb");
        org.iq80.leveldb.util.FileUtils.deleteRecursively(f);
        db = new DbImpl(options, f);
        int i=0;
        while(true)
        {
            db.put(RandomStringUtils.random(64).getBytes(), new byte[]{0x01}, new WriteOptions().sync(false));
            db.get(RandomStringUtils.random(64).getBytes());
            if (++i%1000==0) log.info(i+" rows written");
        }

output:

11/12/06 19:27:59 INFO impl.Foo: 43000 rows written
11/12/06 19:27:59 INFO impl.Foo: 44000 rows written
java.lang.NullPointerException
    at org.iq80.leveldb.impl.Compaction.totalFileSize(Compaction.java:127)
    at org.iq80.leveldb.impl.Compaction.isTrivialMove(Compaction.java:118)
    at org.iq80.leveldb.impl.DbImpl.backgroundCompaction(DbImpl.java:434)
    at org.iq80.leveldb.impl.DbImpl.backgroundCall(DbImpl.java:395)
    at org.iq80.leveldb.impl.DbImpl.access$100(DbImpl.java:79)
    at org.iq80.leveldb.impl.DbImpl$2.call(DbImpl.java:370)
    at org.iq80.leveldb.impl.DbImpl$2.call(DbImpl.java:364)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:722)
java.lang.NullPointerException
    at org.iq80.leveldb.impl.Compaction.totalFileSize(Compaction.java:127)
    at org.iq80.leveldb.impl.Compaction.isTrivialMove(Compaction.java:118)
    at org.iq80.leveldb.impl.DbImpl.backgroundCompaction(DbImpl.java:434)
    at org.iq80.leveldb.impl.DbImpl.backgroundCall(DbImpl.java:395)
    at org.iq80.leveldb.impl.DbImpl.access$100(DbImpl.java:79)
    at org.iq80.leveldb.impl.DbImpl$2.call(DbImpl.java:370)
    at org.iq80.leveldb.impl.DbImpl$2.call(DbImpl.java:364)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:722)
and so on...
and so on...
and some time later OOE

hope that helps, thanks!
Paul.

File size restriction

First of all, thanks a lot for this library, I've been using it with great pleasure!

One issue I have run into is the maximal file size: I see that in Table.java, there is a precondition check on the file size, see here.

Is this actually needed? I have a file that is larger than this limit and cannot open it.

corruption of log writer detected by log reader

When running the tests I noticed a log of "corruption of 35 bytes: Partial record without end"

This also happened with using the code :-(

I have tracked down the problem to an off-by-one bug in

leveldb/src/main/java/org/iq80/leveldb/impl/FileChannelLogWriter.java

and

leveldb/src/main/java/org/iq80/leveldb/impl/MMapLogWriter.java

The following patch fixes this:

git diff leveldb/src/main/java/org/iq80/leveldb/impl/FileChannelLogWriter.java leveldb/src/main/java/org/iq80/leveldb/impl/MMapLogWriter.java
diff --git a/leveldb/src/main/java/org/iq80/leveldb/impl/FileChannelLogWriter.java  b/leveldb/src/main/java/org/iq80/leveldb/impl/FileChannelLogWriter.java

index bd675bc..fdb85f1 100644
--- a/leveldb/src/main/java/org/iq80/leveldb/impl/FileChannelLogWriter.java
+++ b/leveldb/src/main/java/org/iq80/leveldb/impl/FileChannelLogWriter.java
@@ -142,7 +142,7 @@ public class FileChannelLogWriter implements LogWriter
// fragment the record; otherwise write to the end of the record
boolean end;
int fragmentLength;
- if (sliceInput.available() >= bytesAvailableInBlock) {
+ if (sliceInput.available() > bytesAvailableInBlock) {
end = false;
fragmentLength = bytesAvailableInBlock;
}
diff --git a/leveldb/src/main/java/org/iq80/leveldb/impl/MMapLogWriter.java b/leveldb/src/main/java/org/iq80/leveldb/impl/MMapLogWriter.java
index 42b6ca8..221a302 100755
--- a/leveldb/src/main/java/org/iq80/leveldb/impl/MMapLogWriter.java
+++ b/leveldb/src/main/java/org/iq80/leveldb/impl/MMapLogWriter.java
@@ -148,7 +148,7 @@ public class MMapLogWriter implements LogWriter
// fragment the record; otherwise write to the end of the record
boolean end;
int fragmentLength;
- if (sliceInput.available() >= bytesAvailableInBlock) {
+ if (sliceInput.available() > bytesAvailableInBlock) {
end = false;
fragmentLength = bytesAvailableInBlock;
}

I have a junit/hamcrest based test that shows the problem/fix:

package org.iq80.leveldb.impl;

import org.iq80.leveldb.util.Slice;
import org.junit.*;
import java.io.*;
import java.nio.ByteBuffer;
import java.nio.channels.FileChannel;
import java.nio.charset.Charset;
import java.util.*;
import org.junit.rules.*;

import static org.hamcrest.Matchers.*;
import static org.junit.Assert.*;


 public class MMapLogWriterTest {
   @Rule public TemporaryFolder folder = new TemporaryFolder();

   @Test public void simple() throws Exception {
      File file = new File(folder.getRoot(), "10.log");
      LogWriter w = new MMapLogWriter(file, 10);

      //int recordNumber = 7032;
    //int recordSize = 40101;

    int recordNumber = 1;
    int recordSize = LogConstants.BLOCK_SIZE - LogConstants.HEADER_SIZE;
    Slice record = new Slice(recordSize);

    for (int i = 0; i < recordNumber; ++i) {
        w.addRecord(record, false);
    }
    w.close();

    LogMonitor mon = new LogMonitor() {

        @Override
        public void corruption(long bytes, String reason) {
            fail("corruption at " + bytes + " reason: " + reason);
        }

        @Override
        public void corruption(long bytes, Throwable reason) {
            fail("corruption at " + bytes + " reason: " + reason.toString());
        }

    };

    FileChannel channel = new FileInputStream(file).getChannel();

    LogReader logReader = new LogReader(channel, mon, true, 0L);
    int count = 0;
    while (true) {
        Slice s = logReader.readRecord();
        if (s == null) {
            break;
        }
        assertThat(s.length(), is(recordSize));
        count++;
    }
    assertThat(count, is(recordNumber));
}

}

There is a similar bug reported (issue#14) but there seems to be other issues reported there.

DbImplTest.testHiddenValuesAreRemoved fails

I've encountered the issue that DbImplTest.testHiddenValuesAreRemoved fails on my computer with:

java.lang.AssertionError: null: lists don't have the same size expected [1] but found [2]

I've also seen that some builds on Travis failed for this reason. Is this a known issue?

just run the test error.what's wrong?

50000 rows written
100000 rows written
150000 rows written
Tests run: 80, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 630.078 sec <<<FAILURE!
testCantCreateDirectoryReturnMessage(org.iq80.leveldb.impl.DbImplTest) Time elapsed: 0.015 sec <<< FAILURE!
org.testng.TestException:
The exception was thrown with the wrong message: expected "Database directory '.foo.bar.doowop.idontexist'.*" but got "Database '\foo\bar\doowop\idontexist' exi
sts and the error if exists option is enabled"
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:145)
at org.iq80.leveldb.impl.DbImpl.(DbImpl.java:175)
at org.iq80.leveldb.impl.DbImplTest$DbStringWrapper.(DbImplTest.java:1073)
at org.iq80.leveldb.impl.DbImplTest$DbStringWrapper.(DbImplTest.java:1062)
at org.iq80.leveldb.impl.DbImplTest.testCantCreateDirectoryReturnMessage(DbImplTest.java:813)

Results :

Failed tests:
? Test
The exception was thrown with the wrong message: expected "Database d...

Tests run: 80, Failures: 1, Errors: 0, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] leveldb-project .................................... SUCCESS [ 5.453 s]
[INFO] leveldb-api ........................................ SUCCESS [ 1.656 s]
[INFO] leveldb ............................................ FAILURE [11:10 min]
[INFO] leveldb-benchmark .................................. SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 11:20 min
[INFO] Finished at: 2016-02-07T21:22:04+08:00
[INFO] Final Memory: 15M/43M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.14:test (default-test) on project leveldb: There are test failures.
[ERROR]
[ERROR] Please refer to F:\java\leveldb-master\leveldb\target\surefire-reports f
or the individual test results.
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command

[ERROR] mvn -rf :leveldb

"RuntimeException: Could not open table" occurs when you run the DbBenchmark

When you run the DbBenchmark with the following arguments:
--num=1000000 --benchmarks=fillrandom,readseq

It produces the following error:

LevelDB:    iq80 leveldb version 0.2-SNAPSHOT
Date:       Thu Jan 05 20:18:11 EST 2012
Keys:       16 bytes each
Values:     100 bytes each (50 bytes after compression)
Entries:    1000000
RawSize:    110.6 MB (estimated)
FileSize:   62.9 MB (estimated)
------------------------------------------------
fillrandom   :       9.326 micros/op;   11.9 MB/s
... finished 1700000 ops                              Exception in thread "main" java.lang.RuntimeException: Could not open table 74
  at org.iq80.leveldb.impl.TableCache.getTable(TableCache.java:95)
  at org.iq80.leveldb.impl.TableCache.newIterator(TableCache.java:77)
  at org.iq80.leveldb.impl.TableCache.newIterator(TableCache.java:72)
  at org.iq80.leveldb.util.LevelIterator.openNextFile(LevelIterator.java:118)
  at org.iq80.leveldb.util.LevelIterator.getNextElement(LevelIterator.java:94)
  at org.iq80.leveldb.util.AbstractSeekingIterator.hasNext(AbstractSeekingIterator.java:30)
  at org.iq80.leveldb.util.DbIterator$ComparableIterator.next(DbIterator.java:216)
  at org.iq80.leveldb.util.DbIterator.getNextElement(DbIterator.java:87)
  at org.iq80.leveldb.util.AbstractSeekingIterator.hasNext(AbstractSeekingIterator.java:30)
  at org.iq80.leveldb.impl.SnapshotSeekingIterator.findNextUserEntry(SnapshotSeekingIterator.java:98)
  at org.iq80.leveldb.impl.SnapshotSeekingIterator.getNextElement(SnapshotSeekingIterator.java:65)
  at org.iq80.leveldb.util.AbstractSeekingIterator.hasNext(AbstractSeekingIterator.java:30)
  at org.iq80.leveldb.impl.SeekingIteratorAdapter.hasNext(SeekingIteratorAdapter.java:34)
  at org.iq80.leveldb.DbBenchmark.readSequential(DbBenchmark.java:430)
  at org.iq80.leveldb.DbBenchmark.run(DbBenchmark.java:149)
  at org.iq80.leveldb.DbBenchmark.main(DbBenchmark.java:640)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:601)
  at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
Caused by: java.io.FileNotFoundException: /tmp/dbbench/000074.sst (No such file or directory)
  at java.io.FileInputStream.open(Native Method)
  at java.io.FileInputStream.<init>(FileInputStream.java:138)
  at org.iq80.leveldb.impl.TableCache$TableAndFile.<init>(TableCache.java:115)
  at org.iq80.leveldb.impl.TableCache$TableAndFile.<init>(TableCache.java:105)
  at org.iq80.leveldb.impl.TableCache$1.load(TableCache.java:65)
  at org.iq80.leveldb.impl.TableCache$1.load(TableCache.java:61)
  at com.google.common.cache.CustomConcurrentHashMap$ComputingValueReference.compute(CustomConcurrentHashMap.java:3426)
  at com.google.common.cache.CustomConcurrentHashMap$Segment.compute(CustomConcurrentHashMap.java:2322)
  at com.google.common.cache.CustomConcurrentHashMap$Segment.getOrCompute(CustomConcurrentHashMap.java:2291)
  at com.google.common.cache.CustomConcurrentHashMap.getOrCompute(CustomConcurrentHashMap.java:3802)
  at com.google.common.cache.ComputingCache.get(ComputingCache.java:46)
  at org.iq80.leveldb.impl.TableCache.getTable(TableCache.java:88)
  ... 20 more

Caused by: org.iq80.leveldb.impl.DbImpl$BackgroundProcessingException: java.io.FileNotFoundException: leveldb/mapdbcache/021748.sst (Too many open files)

@dain

This is occuring with v 0.7. We are inserting the data into leveldb, but after ever 3 hours, we see exceptions as

Caused by: org.iq80.leveldb.impl.DbImpl$BackgroundProcessingException: java.io.FileNotFoundException: leveldb/mapdbcache/021748.sst (Too many open files)
    at org.iq80.leveldb.impl.DbImpl.checkBackgroundException(DbImpl.java:411)
    at org.iq80.leveldb.impl.DbImpl.get(DbImpl.java:572)
    at org.iq80.leveldb.impl.DbImpl.get(DbImpl.java:565)
    at com.shn.logs.cache.impl.LevelDBByteStore.getBytes(LevelDBByteStore.java:79)
    at com.shn.logs.cache.impl.LevelDBGenericStore.get(LevelDBGenericStore.java:99)
    at com.shn.logs.cache.impl.FileBackedUniqueItemsStore$1.call(FileBackedUniqueItemsStore.java:89)
    at com.shn.logs.cache.impl.FileBackedUniqueItemsStore$1.call(FileBackedUniqueItemsStore.java:85)
    at com.shn.logs.cache.impl.LevelDBGenericStore.lockedWriteOp(LevelDBGenericStore.java:126)
    ... 16 more
Caused by: java.io.FileNotFoundException: leveldb/mapdbcache/021748.sst (Too many open files)
    at java.io.FileOutputStream.open0(Native Method)
    at java.io.FileOutputStream.open(FileOutputStream.java:270)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
    at org.iq80.leveldb.impl.DbImpl.openCompactionOutputFile(DbImpl.java:1134)
    at org.iq80.leveldb.impl.DbImpl.doCompactionWork(DbImpl.java:1085)
    at org.iq80.leveldb.impl.DbImpl.backgroundCompaction(DbImpl.java:478)
    at org.iq80.leveldb.impl.DbImpl.backgroundCall(DbImpl.java:426)
    at org.iq80.leveldb.impl.DbImpl.access$100(DbImpl.java:83)
    at org.iq80.leveldb.impl.DbImpl$2.call(DbImpl.java:396)
    at org.iq80.leveldb.impl.DbImpl$2.call(DbImpl.java:390)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    ... 3 more

I looked at the source for v 0.7, which points me to

        compactionState.outfile = new FileOutputStream(file).getChannel();

and the call chain tells me that levelDB is running the compaction in background (is that correct?)

Also, when we set up an instance of leveldb, we do not specify anything related to compaction.

 private DB init(File storeLocation, long cacheSizeInMb, DBComparator dbComparator) {
    Options options = new Options()
        .cacheSize(ByteUnit.MEGA.of(cacheSizeInMb))
        .comparator(dbComparator)
        .compressionType(CompressionType.NONE)
        .createIfMissing(true);
    try {
      return Iq80DBFactory.factory.open(storeLocation, options);
    } catch (IOException e) {
      throw new ShnRuntimeException("Failed to open LevelDB store", e);
    }
  }

Do I need to? Am I missing anything?

database corrupted sometimes

I found sometimes the database will be broken. I found these in collected error logs and cant easily to reproduce:

Could not open table xxx
CURRENT file does not end with newline

The first problem seems because the sst file is not created correctly. The 2nd is issued here:https://code.google.com/p/leveldb/issues/detail?id=69. Currently I only use leveldb as a cache, so when these problems occured, I just simplely destroy the database and rebuild it. But this is not a good solution, is there other way to fix?

Thanks

NullPointer in VersionSet.numberOfFilesInLevel()

I write data to levelDB in multi-threads, the exception below is thrown. why?
java.lang.NullPointerException
at org.iq80.leveldb.impl.VersionSet.numberOfFilesInLevel(VersionSet.java:227) ~[leveldb-0.7.jar:?]
at org.iq80.leveldb.impl.DbImpl.makeRoomForWrite(DbImpl.java:809) ~[leveldb-0.7.jar:?]
at org.iq80.leveldb.impl.DbImpl.writeInternal(DbImpl.java:676) ~[leveldb-0.7.jar:?]
at org.iq80.leveldb.impl.DbImpl.write(DbImpl.java:665) ~[leveldb-0.7.jar:?]

BUG java.lang.RuntimeException: Could not open table 4049

I've found a bug while query leveldb to get data

java.lang.RuntimeException: Could not open table 4049
at org.iq80.leveldb.impl.TableCache.getTable(TableCache.java:98)
at org.iq80.leveldb.impl.TableCache.newIterator(TableCache.java:79)
at org.iq80.leveldb.impl.TableCache.newIterator(TableCache.java:74)
at org.iq80.leveldb.impl.Level.get(Level.java:130)
at org.iq80.leveldb.impl.Version.get(Version.java:171)
at org.iq80.leveldb.impl.VersionSet.get(VersionSet.java:222)
at org.iq80.leveldb.impl.DbImpl.get(DbImpl.java:613)
at org.iq80.leveldb.impl.DbImpl.get(DbImpl.java:574)
Caused by: java.io.FileNotFoundException: ../LevelDBData/approvedKeyword/004049.sst (No such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.(FileInputStream.java:146)
at org.iq80.leveldb.impl.TableCache$TableAndFile.(TableCache.java:124)
at org.iq80.leveldb.impl.TableCache$TableAndFile.(TableCache.java:114)
at org.iq80.leveldb.impl.TableCache$1.load(TableCache.java:67)
at org.iq80.leveldb.impl.TableCache$1.load(TableCache.java:62)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
at org.iq80.leveldb.impl.TableCache.getTable(TableCache.java:91)

Snapshots are not preserved across compaction

see here as it differs from here

I have some unit tests that demonstrate this failing (so it's not just speculation on the source). If no one picks this up i'll probably fix it eventually

Unsafe usage cause coredump on HP-UX

Hi,

Running LevelDB on HP-UX cause coredump.
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) j sun.misc.Unsafe.getInt(Ljava/lang/Object;J)I+0 j org.iq80.snappy.UnsafeMemory.loadInt([BI)I+50 j org.iq80.snappy.SnappyInternalUtils.loadInt([BI)I+5 j org.iq80.snappy.SnappyCompressor.findCandidate([BIIII[SI)[I+19 j org.iq80.snappy.SnappyCompressor.compressFragment([BII[BI[S)I+157 j org.iq80.snappy.SnappyCompressor.compress([BII[BI)I+63 j org.iq80.snappy.Snappy.compress([BII[BI)I+6 j org.iq80.leveldb.util.Snappy$IQ80Snappy.compress([BII[BI)I+7 j org.iq80.leveldb.util.Snappy.compress([BII[BI)I+9 j org.iq80.leveldb.table.TableBuilder.writeBlock(Lorg/iq80/leveldb/table/BlockBuilder;)Lorg/iq80/leveldb/table/BlockHandle;+54 j org.iq80.leveldb.table.TableBuilder.flush()V+51 j org.iq80.leveldb.table.TableBuilder.add(Lorg/iq80/leveldb/util/Slice;Lorg/iq80/leveldb/util/Slice;)V+172 j org.iq80.leveldb.impl.DbImpl.buildTable(Lorg/iq80/leveldb/impl/SeekingIterable;J)Lorg/iq80/leveldb/impl/FileMetaData;+135 j org.iq80.leveldb.impl.DbImpl.writeLevel0Table(Lorg/iq80/leveldb/impl/MemTable;Lorg/iq80/leveldb/impl/VersionEdit;Lorg/iq80/leveldb/impl/Version;)V+53 j org.iq80.leveldb.impl.DbImpl.compactMemTableInternal()V+41 j org.iq80.leveldb.impl.DbImpl.backgroundCompaction()V+11 j org.iq80.leveldb.impl.DbImpl.backgroundCall()V+94 j org.iq80.leveldb.impl.DbImpl.access$100(Lorg/iq80/leveldb/impl/DbImpl;)V+1 j org.iq80.leveldb.impl.DbImpl$2.call()Ljava/lang/Void;+4 j org.iq80.leveldb.impl.DbImpl$2.call()Ljava/lang/Object;+1
the cause of this error is the use of UnsafeMemory.

in org.iq80.leveldb.util.Snappy it is appealed to the library Snappy
using UnsafeMemory.loadInt. This causes exception in the JVM on multiple processors.
https://www.google.com/search?q=sun.misc.Unsafe+COREDUMP

Snappy the project opened an issue
dain/snappy#24

Bye
JYT
I'm trying Snappy 0.4 it works fine

NoClassDefFoundError on calling factory.open()

Hi, I'm getting the following stacktrace when I try to execute the code in the readme:

Error occurs when runing the following line:

DB db = factory.open(new File("example"), options);

Exception in thread "main" java.lang.NoClassDefFoundError: com/google/common/collect/Lists
at org.iq80.leveldb.impl.DbImpl.(DbImpl.java:95)
at org.iq80.leveldb.impl.Iq80DBFactory.open(Iq80DBFactory.java:59)
at com.test.LevelDBDain.main(LevelDBDain.java:23)
Caused by: java.lang.ClassNotFoundException: com.google.common.collect.Lists
at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

Could it be something I'm doing wrong?

maxOpenFiles ignored?

After setting maxOpenFiles to 500, leveldb is still opening close to a thousand files (before the OS starts throwing "too many open files"). I verified the Options object by printing options.maxOpenFiles() prior to calling the factory open method.

Cannot open table, No such file XYZ.sst

Not sure if Bug or different LevelDB version. But in my DB there are no ...sst files only ...ldb files, so i get the following error:

java.lang.RuntimeException: Could not open table 3
at org.iq80.leveldb.impl.TableCache.getTable(TableCache.java:97)
at org.iq80.leveldb.impl.TableCache.newIterator(TableCache.java:78)
at org.iq80.leveldb.impl.TableCache.newIterator(TableCache.java:73)
at org.iq80.leveldb.impl.Version.getLevel0Files(Version.java:146)
at org.iq80.leveldb.impl.DbImpl.internalIterator(DbImpl.java:776)
at org.iq80.leveldb.impl.DbImpl.iterator(DbImpl.java:742)
at org.iq80.leveldb.impl.DbImpl.iterator(DbImpl.java:733)
at org.iq80.leveldb.impl.DbImpl.iterator(DbImpl.java:1)
at java.lang.Iterable.forEach(Iterable.java:74)
at com.github.wolfposd.StartupLevelDB.main(StartupLevelDB.java:30)
Caused by: java.io.FileNotFoundException: mydatabase.db/000003.sst (No such file or directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.(FileInputStream.java:138)
at org.iq80.leveldb.impl.TableCache$TableAndFile.(TableCache.java:122)
at org.iq80.leveldb.impl.TableCache$TableAndFile.(TableCache.java:117)
at org.iq80.leveldb.impl.TableCache$2.load(TableCache.java:66)
at org.iq80.leveldb.impl.TableCache$2.load(TableCache.java:1)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
at org.iq80.leveldb.impl.TableCache.getTable(TableCache.java:90)
... 9 more

However, when i change org.iq80.leveldb.impl.Filename.java#59
from "sst" to "ldb" it works fine

Db lock isn't released in case fail initialization

Class: org.iq80.leveldb.impl.DbImpl
Method: Constructor public DbImpl(Options options, File databaseDir)

Code part:

       dbLock = new DbLock(new File(databaseDir, Filename.lockFileName()));
        // verify the "current" file
        File currentFile = new File(databaseDir, Filename.currentFileName());
        if (!currentFile.canRead()) {
            Preconditions.checkArgument(options.createIfMissing(), "Database '%s' does not exist and the create if missing option is disabled", databaseDir);
        }
        else {
            Preconditions.checkArgument(!options.errorIfExists(), "Database '%s' exists and the error if exists option is enabled", databaseDir);
        }

Issue:
You create DbLock object which creates LOCK file and get channel access for it.
In case any check fail below, the channel isn't closed. You couldn't work with the folder until stop JVM.

Solution:
Catch exception, close DbLock and throw exception up.

Is openDB slow?

From some initial startup time profiling of the Android app I'm working on, this line seems to be the current biggest offender:

Options opts = new Options()
        .createIfMissing(true)
        .errorIfExists(false)
        .paranoidChecks(false)
        .compressionType(CompressionType.NONE)
        .cacheSize(cacheSize);

return Iq80DBFactory.factory.open(new File(path), opts);

It takes about a whole second or two on the open call.

Is this normal? What are some ways to optimize this?

in DbImp.get(byte[] key, ReadOptions options) has a bug

java.lang.NullPointerException
at org.iq80.leveldb.impl.DbImpl.get(DbImpl.java:588)
at org.iq80.leveldb.impl.DbImpl.get(DbImpl.java:562)

DbImp.get(byte[] key, ReadOptions options){
.......
if (immutableMemTable != null) {
lookupResult = immutableMemTable.get(lookupKey);
if (lookupResult != null) {
//this line will throw NullPointerException
return lookupResult.getValue().getBytes();

            }
        }

..........
}

the code can be changed like this:
if (immutableMemTable != null) {
lookupResult = immutableMemTable.get(lookupKey);
if (lookupResult != null) {
Slice value = lookupResult.getValue();
if (value == null) {
return null;
}
return value.getBytes();
}
}

"IllegalArgumentException: new file overlaps existing files in range" on put

When I run a put( key, value ) loop over index from 0 to 25 million (the error is raised soon after 300 000) I'm often (though not always) getting following exception:

org.iq80.leveldb.impl.DbImpl$BackgroundProcessingException: java.lang.IllegalArgumentException: new file overlaps existing files in range
    at org.iq80.leveldb.impl.DbImpl.checkBackgroundException(DbImpl.java:386)
    at org.iq80.leveldb.impl.DbImpl.writeInternal(DbImpl.java:636)
    at org.iq80.leveldb.impl.DbImpl.put(DbImpl.java:602)
    at org.iq80.leveldb.impl.DbImpl.put(DbImpl.java:595)
...

Keys/values are like that (converted to bytes via getBytes( "UTF-8" ):

keys:  "Key LOOOOOOOOOOOOOOOOOONG KEY " + index;
values:  "This is element " + index + "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABZASDFASDKLFJASDFKJSDFLKSDJFLKJSDHFLKJHSDJFSDFHJASDFLKJSDF";

The error was reproduced with and without compression. Also with default (4K) and with 32K block size. And in general - I'm testing with default options (just createIfMissing is changed to true). And I'm always starting with clean folder used for database path.

The same thing with shorter key/value ( "Key " + index, "Value " + index ) worked without any errors.

Reproduced with:

Oracle JRockit(R) (build R28.2.4-14-151097-1.6.0_33-20120618-1634-linux-x86_64, compiled mode)
Java(TM) SE Runtime Environment (build 1.7.0_07-b10) /Linux x86-64/
Java(TM) SE Runtime Environment (build 1.6.0_33-b03)

DB.put fails with java.lang.IllegalArgumentException: key must be greater than last key

The following code:
package testleveldb;

import java.io.File;
import java.util.Random;

import org.iq80.leveldb.CompressionType;
import org.iq80.leveldb.DB;
import org.iq80.leveldb.Options;
import org.iq80.leveldb.impl.Iq80DBFactory;

public class Test
{
private static final byte[] NULL = new byte[0];

public static void main( String[] args )
    throws Exception
{
    Options options = new Options();
    options.createIfMissing( true );
    options.compressionType( CompressionType.SNAPPY );
    options.writeBufferSize( 16 * 1024 * 1024 );
    DB db = new Iq80DBFactory().open( new File( "/var/tmp/test/testleveldb" ), options );

    int progress = 0;
    byte[] key = new byte[40];
    Random random = new Random();
    while ( true )
    {
        random.nextBytes( key );
        db.put( key, NULL );

        progress++;
        if ( ( progress % 100000 ) == 0 )
        {
            System.out.println( progress );
        }
    }
}

}

fails with this output:
100000
200000
300000
Exception in thread "main" org.iq80.leveldb.impl.DbImpl$BackgroundProcessingException: java.lang.IllegalArgumentException: key must be greater than last key
at org.iq80.leveldb.impl.DbImpl.checkBackgroundException(DbImpl.java:384)
at org.iq80.leveldb.impl.DbImpl.writeInternal(DbImpl.java:634)
at org.iq80.leveldb.impl.DbImpl.put(DbImpl.java:600)
at org.iq80.leveldb.impl.DbImpl.put(DbImpl.java:593)
at testleveldb.Test.main(Test.java:30)
Caused by: java.lang.IllegalArgumentException: key must be greater than last key
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
at org.iq80.leveldb.table.BlockBuilder.add(BlockBuilder.java:109)
at org.iq80.leveldb.table.TableBuilder.add(TableBuilder.java:143)
at org.iq80.leveldb.impl.DbImpl.buildTable(DbImpl.java:928)
at org.iq80.leveldb.impl.DbImpl.writeLevel0Table(DbImpl.java:891)
at org.iq80.leveldb.impl.DbImpl.compactMemTableInternal(DbImpl.java:860)
at org.iq80.leveldb.impl.DbImpl.backgroundCompaction(DbImpl.java:425)
at org.iq80.leveldb.impl.DbImpl.backgroundCall(DbImpl.java:399)
at org.iq80.leveldb.impl.DbImpl.access$100(DbImpl.java:66)
at org.iq80.leveldb.impl.DbImpl$2.call(DbImpl.java:369)
at org.iq80.leveldb.impl.DbImpl$2.call(DbImpl.java:363)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

Varying the writeBufferSize, the number of records where the error occurs changes.

I run this with leveldb 0.3 and 0.4-SNAPSHOT and the behavior is the same.

Compaction run goes into a loop.

Once it starts looping, all other read/writes block.

Error in README

I think the key is the first param

db.delete(wo, bytes("Tampa"));

Exception when doing the write test

key are 15 bytes long and values are 127 bytes long, both keys and values are random bytes.

Exception in thread "main" org.iq80.leveldb.impl.DbImpl$BackgroundProcessingException: java.lang.IllegalArgumentException: key must be greater than last key
at org.iq80.leveldb.impl.DbImpl.checkBackgroundException(DbImpl.java:411)
at org.iq80.leveldb.impl.DbImpl.writeInternal(DbImpl.java:664)
at org.iq80.leveldb.impl.DbImpl.put(DbImpl.java:630)
at org.iq80.leveldb.impl.DbImpl.put(DbImpl.java:623)
at WriteTest.main(WriteTest.java:49)
Caused by: java.lang.IllegalArgumentException: key must be greater than last key
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:92)
at org.iq80.leveldb.table.BlockBuilder.add(BlockBuilder.java:109)
at org.iq80.leveldb.table.TableBuilder.add(TableBuilder.java:143)
at org.iq80.leveldb.impl.DbImpl.buildTable(DbImpl.java:969)
at org.iq80.leveldb.impl.DbImpl.writeLevel0Table(DbImpl.java:932)
at org.iq80.leveldb.impl.DbImpl.compactMemTableInternal(DbImpl.java:896)
at org.iq80.leveldb.impl.DbImpl.backgroundCompaction(DbImpl.java:455)
at org.iq80.leveldb.impl.DbImpl.backgroundCall(DbImpl.java:426)
at org.iq80.leveldb.impl.DbImpl.access$100(DbImpl.java:83)
at org.iq80.leveldb.impl.DbImpl$2.call(DbImpl.java:396)
at org.iq80.leveldb.impl.DbImpl$2.call(DbImpl.java:390)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:680)

Files not loaded on certain computers but load on others just fine.

For some reason when I try to use the Iq80DB it just wont work at all, but when my team mates do it it works fine! (Yes we are all using the same code) And what's weirder is that if I run the same code on my Macintosh and not my Windows 10, it loads just fine.

Stack trace:

[23:31:51] [ERROR] [RedstoneLamp]: java.io.FileNotFoundException while loading LevelDB world db
 (The filename, directory name, or volume label syntax is incorrect).provider.LevelLoadException: net.redstonelamp.level.provider.LevelLoadException: java.io.FileNotFoundException: worlds\world\db\MANIFEST-000025
 (The filename, directory name, or volume label syntax is incorrect)velLoadException: net.redstonelamp.level.provider.LevelLoadException: java.io.FileNotFoundException: worlds\world\db\MANIFEST-000025
[23:31:51] [INFO] [RedstoneLamp]:   at net.redstonelamp.level.Level.<init>(Level.java:88)
[23:31:51] [INFO] [RedstoneLamp]:   at net.redstonelamp.level.LevelManager.init(LevelManager.java:75)
[23:31:51] [INFO] [RedstoneLamp]:   at net.redstonelamp.Server.<init>(Server.java:126)
[23:31:51] [INFO] [RedstoneLamp]:   at net.redstonelamp.RedstoneLamp.main(RedstoneLamp.java:52)
 (The filename, directory name, or volume label syntax is incorrect)provider.LevelLoadException: java.io.FileNotFoundException: worlds\world\db\MANIFEST-000025
[23:31:51] [INFO] [RedstoneLamp]:   at net.redstonelamp.level.provider.leveldb.LevelDBProvider.<init>(LevelDBProvider.java:78)
[23:31:51] [INFO] [RedstoneLamp]:   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
[23:31:51] [INFO] [RedstoneLamp]:   at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
[23:31:51] [INFO] [RedstoneLamp]:   at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
[23:31:51] [INFO] [RedstoneLamp]:   at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
[23:31:51] [INFO] [RedstoneLamp]:   at net.redstonelamp.level.Level.<init>(Level.java:84)
[23:31:51] [INFO] [RedstoneLamp]:   ... 3 more
 (The filename, directory name, or volume label syntax is incorrect)eption: worlds\world\db\MANIFEST-000025
[23:31:51] [INFO] [RedstoneLamp]:   at java.io.FileInputStream.open0(Native Method)
[23:31:51] [INFO] [RedstoneLamp]:   at java.io.FileInputStream.open(FileInputStream.java:195)
[23:31:51] [INFO] [RedstoneLamp]:   at java.io.FileInputStream.<init>(FileInputStream.java:138)
[23:31:51] [INFO] [RedstoneLamp]:   at org.iq80.leveldb.impl.VersionSet.recover(VersionSet.java:341)
[23:31:51] [INFO] [RedstoneLamp]:   at org.iq80.leveldb.impl.DbImpl.<init>(DbImpl.java:186)
[23:31:51] [INFO] [RedstoneLamp]:   at org.iq80.leveldb.impl.Iq80DBFactory.open(Iq80DBFactory.java:59)
[23:31:51] [INFO] [RedstoneLamp]:   at net.redstonelamp.level.provider.leveldb.LevelDBProvider.<init>(LevelDBProvider.java:75)
[23:31:51] [INFO] [RedstoneLamp]:   ... 8 more

it says the file is not found but when I check to see if it exists it clearly does. I did
System.out.println(new File("worlds\world\db\MANIFEST-000025").exists());
which returned true, I also made sure it could be read by using .canRead(); which also returned true. I have no idea why this is happening and don't know if this is caused by a library issue dependent per system or what.

Can't work on Android?

It throws java.lang.ExceptionInInitializerError while initializing Iq80DBFactory.

max_file_size

In google code in options.cc there is property "max_file_size". Is it possible to set it somehow? I want to play with this parameter because I want to store large amount of data in leveldb

error when a large amount of

create a database
add 4 billion rows
with "for ()"

when the database size reaches 1 Gb
or number of lines ~ 21692085

java.lang.RuntimeException: Could not open table 2700
at org.iq80.leveldb.impl.TableCache.getTable(TableCache.java:95)
at org.iq80.leveldb.impl.TableCache.newIterator(TableCache.java:77)
at org.iq80.leveldb.impl.DbImpl.finishCompactionOutputFile(DbImpl.java:1112)
at org.iq80.leveldb.impl.DbImpl.doCompactionWork(DbImpl.java:1035)
at org.iq80.leveldb.impl.DbImpl.backgroundCompaction(DbImpl.java:444)
at org.iq80.leveldb.impl.DbImpl.backgroundCall(DbImpl.java:395)
at org.iq80.leveldb.impl.DbImpl.access$100(DbImpl.java:79)
at org.iq80.leveldb.impl.DbImpl$2.call(DbImpl.java:370)
at org.iq80.leveldb.impl.DbImpl$2.call(DbImpl.java:364)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:849)
at org.iq80.leveldb.table.Table.<init>(Table.java:63)
at org.iq80.leveldb.impl.TableCache$TableAndFile.<init>(TableCache.java:118)
at org.iq80.leveldb.impl.TableCache$TableAndFile.<init>(TableCache.java:105)
at org.iq80.leveldb.impl.TableCache$1.load(TableCache.java:65)
at org.iq80.leveldb.impl.TableCache$1.load(TableCache.java:61)
at com.google.common.cache.CustomConcurrentHashMap$ComputingValueReference.compute(CustomConcurrentHashMap.java:3426)
at com.google.common.cache.CustomConcurrentHashMap$Segment.compute(CustomConcurrentHashMap.java:2322)
at com.google.common.cache.CustomConcurrentHashMap$Segment.getOrCompute(CustomConcurrentHashMap.java:2291)
at com.google.common.cache.CustomConcurrentHashMap.getOrCompute(CustomConcurrentHashMap.java:3802)
at com.google.common.cache.ComputingCache.get(ComputingCache.java:46)
at org.iq80.leveldb.impl.TableCache.getTable(TableCache.java:88)
... 13 more
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:846)
... 24 more 
java.lang.RuntimeException: Could not open table 2702
at org.iq80.leveldb.impl.TableCache.getTable(TableCache.java:95)
at org.iq80.leveldb.impl.TableCache.newIterator(TableCache.java:77)
at org.iq80.leveldb.impl.TableCache.newIterator(TableCache.java:72)
at org.iq80.leveldb.impl.DbImpl.buildTable(DbImpl.java:936)
at org.iq80.leveldb.impl.DbImpl.writeLevel0Table(DbImpl.java:881)
at org.iq80.leveldb.impl.DbImpl.compactMemTableInternal(DbImpl.java:847)
at org.iq80.leveldb.impl.DbImpl.backgroundCompaction(DbImpl.java:421)
at org.iq80.leveldb.impl.DbImpl.backgroundCall(DbImpl.java:395)
at org.iq80.leveldb.impl.DbImpl.access$100(DbImpl.java:79)
at org.iq80.leveldb.impl.DbImpl$2.call(DbImpl.java:370)
at org.iq80.leveldb.impl.DbImpl$2.call(DbImpl.java:364)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:849)
at org.iq80.leveldb.table.Table.<init>(Table.java:63)
at org.iq80.leveldb.impl.TableCache$TableAndFile.<init>(TableCache.java:118)
at org.iq80.leveldb.impl.TableCache$TableAndFile.<init>(TableCache.java:105)
at org.iq80.leveldb.impl.TableCache$1.load(TableCache.java:65)
at org.iq80.leveldb.impl.TableCache$1.load(TableCache.java:61)
at com.google.common.cache.CustomConcurrentHashMap$ComputingValueReference.compute(CustomConcurrentHashMap.java:3426)
at com.google.common.cache.CustomConcurrentHashMap$Segment.compute(CustomConcurrentHashMap.java:2322)
at com.google.common.cache.CustomConcurrentHashMap$Segment.getOrCompute(CustomConcurrentHashMap.java:2291)
at com.google.common.cache.CustomConcurrentHashMap.getOrCompute(CustomConcurrentHashMap.java:3802)
at com.google.common.cache.ComputingCache.get(ComputingCache.java:46)
at org.iq80.leveldb.impl.TableCache.getTable(TableCache.java:88)
... 15 more
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:846)
... 26 more

With a large amount of base, after executing the commands

    db.close();
    db = factory.open(patch, options);

flying bug

Exception in thread "main" java.io.FileNotFoundException: leveldb\db\000015.sst (The requested operation can not be executed for a file with a user-mapped section open)

Translate by Google*

Implement the compactRange method in the DbImpl class

Related to #29, right now compactRange just throws UnsupportedOperationException

File Size doesn't change?

how do I change the file sizes of the db?

Currently each file size is about 2MB. What if I want to increase this size?

==> options.blockSize(10 * 1024); doesn't have any effect

does the Java version even support this feature?

Publish to Maven repository

Please publish the snapshots into a public repository (Sonatype)

Comparator passed via Options.comparator() doesn't get invoked.

Custom Comparator passed in following manner get ignored :

Options opts = new Options();
opts.createIfMissing(true);
opts.comparator(new Mycomparator());

DBFactory fctry = Iq80DBFactory.factory;
DB db = fctry.open(new File("./data/leveldbfiles"), opts);

Is there any other way to pass a custom Comparator ?

Regards,
Tarun

Support reverse iteration/seeking.

Add a compactRange method to the DB interface

Something like:

public void compactRange(byte[] begin, byte[] end) throws DBException

LevelDB optimizations like Basho's?

I didn't know where else to post this question, but do you foresee making changes to your port of LevelDB like Basho - https://speakerdeck.com/basho/optimizing-leveldb-for-performance-and-scale-ricon-east-2013?

Thanks.

Iterator bug

I've found a bug while comparing how long it takes to insert 1000000 int keys and values (from 0 to 999999) using this lib and rocksdb. When forward iterating the first key should be 0 but it isn't.

@rnarubin - any thoughts on this? Your fork seem to be the one that's most active.

Here's a test case that verifies the bug.

package se.svt.sandbox;

import java.io.File;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.nio.file.FileVisitResult;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.SimpleFileVisitor;
import java.nio.file.attribute.BasicFileAttributes;
import java.util.Map;

import org.iq80.leveldb.CompressionType;
import org.iq80.leveldb.DB;
import org.iq80.leveldb.DBIterator;
import org.iq80.leveldb.Options;
import org.iq80.leveldb.impl.Iq80DBFactory;
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.Test;

import static org.hamcrest.CoreMatchers.is;
import static org.junit.Assert.assertThat;

public class LeveldbBugTest {
    private static Path tempDir;

    @BeforeClass
    public static void before() throws IOException {
        tempDir = Files.createTempDirectory(null);
    }

    @Test
    public void verifyBug() throws IOException {
        File file = tempDir.resolve("levelbug").toFile();

        Options options = new Options();
        options.compressionType(CompressionType.NONE);
        options.createIfMissing(true);

        DB db1 = Iq80DBFactory.factory.open(file, options);
        byte[] data = new byte[4];
        ByteBuffer buffer = ByteBuffer.wrap(data);
        for (int i = 0; i < 600_000; ++i) {
            buffer.putInt(i);
            db1.put(data, data);
            buffer.clear();
        }

        db1.close();

        options.createIfMissing(false);

        DB db2 = Iq80DBFactory.factory.open(file, options);
        DBIterator iterator = db2.iterator();
        iterator.seekToFirst();

        assertThat(iterator.hasNext(), is(true));

        Map.Entry<byte[], byte[]> entry = iterator.peekNext();
        int key = ByteBuffer.wrap(entry.getKey()).getInt();
        try {
            assertThat(key, is(0));
        } finally {
            iterator.close();
            db2.close();
        }
    }

    @AfterClass
    public static void after() throws IOException {
        Files.walkFileTree(tempDir, new SimpleFileVisitor<Path>() {
            @Override
            public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) throws IOException {
                Files.delete(file);
                return FileVisitResult.CONTINUE;
            }

            @Override
            public FileVisitResult postVisitDirectory(Path dir, IOException e) throws IOException {
                if (e == null) {
                    Files.delete(dir);
                    return FileVisitResult.CONTINUE;
                } else {
                    throw e;
                }
            }
        });
    }
}

NullPointerException in VersionSet.numberOfFilesInLevel

Exception in thread "thread-11: xxxx" java.lang.NullPointerException
        at org.iq80.leveldb.impl.VersionSet.numberOfFilesInLevel(VersionSet.java:232)
        at org.iq80.leveldb.impl.DbImpl.makeRoomForWrite(DbImpl.java:821)
        at org.iq80.leveldb.impl.DbImpl.writeInternal(DbImpl.java:685)
        at org.iq80.leveldb.impl.DbImpl.put(DbImpl.java:646)
        at org.iq80.leveldb.impl.DbImpl.put(DbImpl.java:639)
        ...

I should mention that I'm using leveldb in a multi-threaded szenario and with synchronized around
the level DB put operation.

Creating a seconday Index?

Hey @dain
I have a usecase where I wanted a BiDirectional Map, where I can search on both key and value (they both are unique). Currently, I store them twice in 2 different databases

String key = // generated from somewhere
String hashedKey = // encrypt key and save

mapDb.put(key, hashedKey)
decryptMapDb(hashKey, key)

Our key and hashedKey both are unique

I was reading on leveldb google groups, where someone recommended to create secondary index. The link is here

Build fails

Maven build fails with following error :

Tests run: 73, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 230.714 sec <<< FAILURE!

Results :

Failed tests:
testCantCreateDirectoryReturnMessage(org.iq80.leveldb.impl.DbImplTest):

Tests run: 73, Failures: 1, Errors: 0, Skipped: 0

DbImpl constructor throws exceptions with '%s' in it.

This is a minor issue. In the DbImpl constructor, it does this:

    Preconditions.checkArgument(databaseDir.exists(), "Database directory '%s' does not exist and could not be created");
    Preconditions.checkArgument(databaseDir.isDirectory(), "Database directory '%s' is not a directory");

The databaseDir parameter is missing.

dain / leveldb Goto Github PK

leveldb's Introduction

LevelDB in Java

Current status

API Usage:

Projects using this port of LevelDB

leveldb's People

Contributors

Stargazers

Watchers

Forkers

leveldb's Issues

Recommend Projects

Recommend Topics

Recommend Org