Comments (5)
Hi,
I looked a bit into this, and you're right that it's not equal between the
different compression algorithms. This is most likely because Compress() is
used both for the comparative benchmarks and for the ones within Snappy itself;
and for the latter, we don't want the performance numbers to be polluted with
whatever malloc is in use. (I'm actually surprised it matters so much, but I'd
guess it depends on your malloc.)
I think the right solution is actually the opposite of what you're proposing,
though; remove the resize() calls from the other algorithms, not add them to
Snappy. This would require a bit more logic on the outside, though (it would
need to resize the buffer to something that gives sense for all the algorithms
in use).
Original comment by [email protected]
on 29 Mar 2011 at 10:27
- Changed state: Accepted
from snappy.
I don't particularly care which way the tester makes sure libraries have equal
conditions, just that it does.
There is code to that effect:
// Pre-grow the output buffer so we don't measure stupid string
// append time
compressed[b].resize(block_size * 2);
So the buffer should be large enough to satisfy any sane algorithm (and indeed
LZO works with the resize commands removed). I don't have the other libraries
besides zlib and lzo to test with, and zlib decompression for example doesn't
work with just the resize command removed, so I hesitate to touch code I can't
even test.
Original comment by [email protected]
on 30 Mar 2011 at 3:00
from snappy.
QuickLZ needs 36000 scratch bytes, though, so perhaps max(block_size * 2,
block_size + 36000)?
Why can't you test the other libraries? They are all freely available.
Original comment by [email protected]
on 30 Mar 2011 at 11:01
from snappy.
Original comment by [email protected]
on 30 Mar 2011 at 12:45
- Changed state: Started
from snappy.
Fixed in r24.
Original comment by [email protected]
on 30 Mar 2011 at 8:28
- Changed state: Fixed
from snappy.
Related Issues (20)
- Various MSVC x64 compiler size_t warnings (C4267) HOT 6
- Does not handle data larger than 4GB HOT 8
- Mistakes on the start page HOT 5
- cppcheck - Member variable is not initialized in the constructor. HOT 2
- Type 'ssize_t' not defined for MSVC builds HOT 4
- snappy needs a command line utility HOT 5
- MIsspelled in code HOT 1
- testdata/mapreduce-osd-1-pdf contains "DO NOT DISTRIBUTE" disclaimer HOT 3
- Patch for compiling Snappy with MSVC on Windows HOT 3
- Bug in IncrementalCopyFastPath HOT 6
- use ctypes.util.find_library HOT 3
- ARMv6 and unaligned access HOT 3
- Decompression issues with Snappy 1.1.2 HOT 3
- ahsan ullah HOT 2
- Seeing Null Values from Hive with Snappy Compression HOT 1
- No versioned link for current build HOT 2
- ppc64le entry is needed in config.guess file HOT 4
- performance issue in snappy.cc - I am using version 1.1.2 HOT 2
- Unnecessary memory allocation in snappy.cc:Compress HOT 1
- bad_alloc exception not caught in snappy.cc::Compress
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from snappy.