Comments (11)
Hey - writing a re-implementation of this library and running into a similar challenge. The way I'm seeing the implementation is that the BTree database works on top of the SBBF format, and expanding blocks is basically the following process (correct me if I'm wrong):
Free block is not -1:
- Read the free block's next free block ID (last 4 bytes) and update the next free block pointer found in the header
- Convert the free block to either a leaf or index block (using the header bytes)
- Consume the block
Free block is -1:
ftruncate
(resize) the fileblock_size
bytes further- Initialize with the proper header bytes
- Consume the new block
And the alternate (freeing a block; when a file is shortened down enough to lose n blocks):
- Set the newly freed block's next free block ID to the current next free block pointer found in the header
- Set the newly freed block's type header (first two bytes) to
'FF'
- Set the next free block pointer in the header to the newly freed block's ID
Is this correct?
from py-starbound.
@Qix- sorry to hijack this but does starfuse reading already work? Or do you know if updating py-starbound to support the new format is easy enough? I am really out of my depth on this one and starcheat is kinda screwed without it working. I think @blixt is out of commission atm
from py-starbound.
I'd be more interested in reading from the database directly, as cool as mounting it with FUSE is. Not sure how cross-platform FUSE is without adding dependencies.
from py-starbound.
@wizzomafizzo reading from starfuse works, yes. Writing doesn't yet. FUSE is cross-platform sans Windows, and is very stable and mature on Unix systems.
FUSE allows you to read directly from the database itself without extracting anything. That's the whole point of the FUSE filesystem being glued to the Starfuse libs.
As well my library differs from Blixt's because I use mmap()
instead of read/write buffers, allowing a bit more direct modification as if the entire library was addressed in-memory. Reduces latency a bit more, too. I'm also in the process of refactoring some of the pieces I borrowed from this library to be better structured for when write access is implemented.
from py-starbound.
Thanks for getting back so quick. That's really good news. I had a quick check through your repo and yeah that looks good to me. I will give it a test and let you know how it goes
Unfortunately Windows is the no 1 platform for starcheat so no go there. It's pretty cool nonetheless
Sent from my iPhone
On 17 May 2016, at 11:49 AM, Josh Junon [email protected] wrote:
@wizzomafizzo reading from starfuse works, yes. Writing doesn't yet. FUSE is cross-platform sans Windows, and is very stable and mature on Unix systems.
FUSE allows you to read directly from the database itself without extracting anything. That's the whole point of the FUSE filesystem being glued to the Starfuse libs.
As well my library differs from Blixt's because I use mmap() instead of read/write buffers, allowing a bit more direct modification as if the entire library was addressed in-memory. Reduces latency a bit more, too.
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
from py-starbound.
So for starcheat you could use everything but the FUSE portions - the library itself isnt specific to FUSE operations and even when writing functionality is added it won't be coupled with FUSE ops at all. Without FUSE, starfuse is basically just py-starbound with mmap-ed files and soon to be writing capability.
from py-starbound.
Hey guys! Great to hear Starbound file format projects are still alive and breathing. :) Unfortunately, I have no time these days to work on this (starting a company does that) but I do want to revisit all these projects once Starbound finally hits 1.0 and file formats (presumably) won't change as much.
Regarding saving, the closest I got to doing anything here was with repair.py but that just rebuilds the file from scratch, attempting to keep the indexes balanced. That process might actually be useful down the line to speed up the loading of old worlds, but I presume Starbound already does this automatically in the background.
from py-starbound.
@blixt I was going to ask - what is the point of the alternating root nodes? Is that for transaction-based modification, or am I misinterpreting the code?
from py-starbound.
@Qix- My memory of the Starbound internals is a bit faded now, but yes I believe it's intended for being able to modify the on-disk representation while the game is running (and reading the world). I remember a common issue for broken worlds would be an incomplete write on the current root node and sometimes switching to the other one would fix it (but point at older blocks, some of which have been repurposed resulting in glitches).
You could possibly ask @kyren on Starbound's IRC about details as well.
from py-starbound.
@blixt Awesome, thanks :D
from py-starbound.
Good luck with your company dude!
Sent from my iPhone
On 18 May 2016, at 6:00 AM, Josh Junon [email protected] wrote:
@blixt Awesome, thanks :D
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
from py-starbound.
Related Issues (20)
- Make the package run in both Python 2 and 3 HOT 4
- Ensure everything works fine on Windows
- Allow recovery for worlds that use materials provided by mods that are no longer installed
- Create a repair script which will try to recover a .fail file
- Add writing to sbon module HOT 1
- SBON to YAML conversion? HOT 5
- Newbie Issue HOT 1
- Is this still alive HOT 1
- No API for saving changes made HOT 1
- StarNub Plugin HOT 3
- SBBF header version updated in nightly HOT 13
- Support updating and writing BTreeDB5 databases
- Possible explanation of "magic byte" in SBVJ01 files HOT 4
- Problem HOT 2
- Support directory-like "browsing" of SBAsset6 objects? HOT 3
- SBAsset6 files intended to be case-insensitive? HOT 2
- Reading from btree during BTreeDB5.get_all_keys and World.get_all_regions_with_tiles HOT 4
- Support for reading bookmarks, etc, from world data (types/layers 3 and 4) HOT 12
- setup.py packaging / PyPI? HOT 5
- do you plan to save?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from py-starbound.