Comments (6)
Looking through again there isn't that many string methods in use:
- that first
:find
to get the data section seperator :byte
to read a few bytes at a time:sub
to extract a substring- casting to an ffi
const char*
for speed improvements on the above.
You should be able to replace most of these with a :seek
and :read
, I'd be willing to accept a pull request that adds this.
from mmdblua.
You need to have much of it in memory to find the data section seperator. See https://github.com/daurnimator/mmdblua/blob/master/mmdb.lua#L20
Searching backwards is 'hard'.
After that, ipv6_find_ipv4_start
will need to traverse a reasonable amount of the file...
I'm not sure if it's worth putting the work in for this?
from mmdblua.
no it's not high priority. But would be nice, as right now basically my server thing takes 3MB and the database 30MB. With the other geoip bindings there was an option to map the file in memory, so multiple instances of the same program would allocate the memory only once. I'll look around if there's a decent lua library that can do this with arbitrary files and expose them as strings.
from mmdblua.
With the other geoip bindings there was an option to map the file in memory
Yeah; they'll probably be using mmap()
.
You could do this in luajit via the ffi; but I'd rather not bring that dependency in.
I'll look around if there's a decent lua library that can do this with arbitrary files and expose them as strings.
That won't be possible; lua strings are interned.
from mmdblua.
I know, I mean expose the mmaped region in some way semantically equivalent to a string.
from mmdblua.
To quote myself from #5 (comment)
Sorry, but after reflecting on this a while, I realised that seeking a file here is not the correct answer: it will result in un-necessary slowness due to seek syscall overhead.
What I suggest instead is an ffi-only optimisation that uses an mmap call (perhaps via ljsyscall).
from mmdblua.
Related Issues (9)
- Luarocks please HOT 3
- maxmind GeoIP2-Anonymous-database not working HOT 5
- Provide a release tarball HOT 8
- Do not use ABI to detect number endian in then database HOT 2
- can i integrate your module without luarocks? HOT 4
- Use new 5.3 string.unpack instead of home grown reading. HOT 3
- Support IPv6 mapped IPv4
- Support `safe` mode HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mmdblua.