Comments (14)
Thanks for checking this. I was facing this issue and I can also confirm that the problem was related to not using db.Close
. Now the database is just around 1 MB
from clover.
I made the title of the issue more self-explainatory in order to help people facing the same issue to quickly locate this
from clover.
@thiko, okay thank you :=)
Let's leave the issue open. Your problem of the initial 2GB size was due to db.Close()
not being called.
However, the problem related to the RunValueLogGC()
is a separate problem and needs a fix.
I'll work on it and release a fix by tomorrow :=)
from clover.
Great - thanks for your help!
from clover.
Hi, @thiko. If you ended with an empty database after inserting several items, and then you deleted all of them, yes it can be normal. Internally clover uses badger, which is a log structured key value store. This means that when you delete an object, a "delete" record is appended to the value log.
Logs should be compacted in the background, once the database reaches a suitable size.
EDIT: It seems that inside badger, space must be reclaimed by explicitly calling the RunValueLogGC
function. Currently this function is never called. I will write a fix for this. Thank you for reporting this issue
from clover.
Hi @ostafen!
Thanks for your response!
On my local setup, badger seems to reserve 2 GB on disk space immediately before even apply any operation to it (beside open the connection).
Anyway - I'm looking forward for your fix using the GC call 👍
from clover.
On my local setup, badger seems to reserve 2 GB on disk space immediately before even apply any operation to it (beside open the connection).
So you get a 2GB file even just after calling Open()
on an empty directory?
Can you post a snipped of code showing your usage? It will help me to have the whole picture :=)
from clover.
Sure - it's a small playground project so far. Please find the related code parts below:
// package and other imports
import cloverdb "github.com/ostafen/clover"
type clover struct {
db *cloverdb.DB
}
func OpenCloverConnection(databaseName string) (*clover, error) {
con, err := cloverdb.Open(databaseName)
if err != nil {
return nil, err
}
return &clover{db: con}, nil
}
func (c *clover) CloseConnection() error {
return c.db.Close()
}
// package and other imports
func main() {
// Init. connection
db, err = storage.OpenCloverConnection("test-clover-db2")
if err != nil {
panic("Unable to open database connection!")
}
defer db.CloseConnection()
}
That's it ;)
Using go versio 1.18
mod.go contains github.com/ostafen/clover v1.0.1
Operating system is Ubuntu 18.04.6 LTS
When running go run .
it creates a new directory containing:
▶ ls -alth test-clover-db2
total 32K
drwxrwxr-x 2 thiko thiko 4,0K Apr 24 11:11 .
-rw-rw-r-- 1 thiko thiko 2,0G Apr 24 11:11 000001.vlog
-rw-rw-r-- 1 thiko thiko 128M Apr 24 11:11 00001.mem
-rw-rw-r-- 1 thiko thiko 1,0M Apr 24 11:11 DISCARD
-rw------- 1 thiko thiko 28 Apr 24 11:11 KEYREGISTRY
drwxrwxr-x 12 thiko thiko 4,0K Apr 24 11:11 ..
-rw-rw-r-- 1 thiko thiko 6 Apr 24 11:11 LOCK
-rw------- 1 thiko thiko 16 Apr 24 11:11 MANIFEST
from clover.
Are you calling db.Close()
on the clover db instance inside db.CloseConnection()
?
from clover.
Sorry, missed that part. I'll modify my original post to keep it together
from clover.
@thiko by doing some quick tests it turns out that the problem is related to not calling db.Close()
properly.
Consider this snippet:
db, err := c.Open("./db")
if err != nil {
panic(err)
}
It produces this database:
-rw-rw-r-- 1 stefano stefano 2147483646 apr 24 11:32 000001.vlog
-rw-rw-r-- 1 stefano stefano 134217728 apr 24 11:32 00001.mem
-rw-rw-r-- 1 stefano stefano 1048576 apr 24 11:32 DISCARD
-rw------- 1 stefano stefano 28 apr 24 11:32 KEYREGISTRY
-rw-rw-r-- 1 stefano stefano 6 apr 24 11:32 LOCK
-rw------- 1 stefano stefano 16 apr 24 11:32 MANIFEST
But, if i correctly call db.Close()
:
db, err := c.Open("./db")
defer db.Close()
if err != nil {
panic(err)
}
what I get is the following:
-rw-rw-r-- 1 stefano stefano 20 apr 24 11:29 000001.vlog
-rw-rw-r-- 1 stefano stefano 1048576 apr 24 11:29 DISCARD
-rw------- 1 stefano stefano 28 apr 24 11:29 KEYREGISTRY
-rw------- 1 stefano stefano 16 apr 24 11:29 MANIFEST
from clover.
@thiko, I run exactly the same snipped you posted. It should be fine. I'm getting the correct size since db.Close()
is called.
-rw-rw-r-- 1 stefano stefano 20 apr 24 11:29 000001.vlog
-rw-rw-r-- 1 stefano stefano 1048576 apr 24 11:29 DISCARD
-rw------- 1 stefano stefano 28 apr 24 11:29 KEYREGISTRY
-rw------- 1 stefano stefano 16 apr 24 11:29 MANIFEST
If i remove the call to db.Close()
I get the 2GB size
from clover.
@ostafen you are right - the issue relates to a missing db.close()
call. I wasn't aware about the fact, that defer
is not called when killing the app using os signals (CTRL + C
in fact).
I'll handle the signal for a graceful shutdown.
My bad - sorry. Shall we close this issue?
from clover.
With the last commit, I implemented a simple garbage collection strategy which basically calls RunValueLog()
periodically in the background every 5 minutes.
In the future, I will improve the GC strategy to run only during inactivity periods, but for now its does its job :=)
from clover.
Related Issues (20)
- Example Request: Unmarshal/Update/Replace HOT 6
- Cannot allocate initial memory HOT 1
- GT LT operation very slow HOT 3
- Question: Is there a way to search for a value in all fields? HOT 2
- Getting error when during bulk insert HOT 3
- Small v2 package annoyance HOT 10
- Question: Searching for all documents within time range HOT 1
- distributed? HOT 1
- Support struct in Save() method HOT 7
- Add `OpenWithOptions()` to badger store
- Memory efficient `ExportCollection()` HOT 1
- need update readme HOT 2
- Hope to support semantic query building function HOT 9
- Memory store was removed in v2 HOT 1
- doc.Unmarshal does not play well with `json` tag HOT 5
- Badger storage | RunValueLogGC() is managed with a fatal error HOT 1
- using BadgerDB: DropCollection returns a "Txn is too big to fit into one request" HOT 2
- Why is the _id a uuid
- Runtime error by using of function db.Insert HOT 2
- Unmarshalling with embedded structs is failing HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from clover.