Comments (3)
I've tried to investigate the problem. It appear that for the specific pattern in the testcase, the prefilter used is RareBytesOne { byte1: 106, offset: RareByteOffset { max: 4 } })
What happens is that calling earliest_find_at
from ahocorasick::StreamChunkIter::next
will not find that byte in the buffer and will therefore skip everything. Then buf.roll
will be called to rotate the buffer and read from the Reader. But the next call to earliest_find_at
will only search from the newly read data (despite the match is still in the buffer)
RareBytesOne::next_candidate
will find the rare byte, but will then discard the match because its start position is earlier than the at
argument.
I've tried replacing this line
Line 654 in 39b029b
with
pos.saturating_sub(self.offset.max as usize)
so that RareBytesOne::next_candidate can return a match smaller than at
, also adjusting the call to update_skipped_bytes
to handle negative values. And this solve the testcase.
However it causes some other test to run into infinite loops, so that's not quite right.
from aho-corasick.
@ogoffart Thanks so much for that investigation! That was enough to tell me where the problem was and helped me craft a fix. See #67 for a bit more detail, but the short story is that I've disabled the prefilter in this case unfortunately. Fixing stream searching to work with the rare byte prefilter is delicate and I don't trust my test coverage enough to attempt that.
from aho-corasick.
A fix for this is on crates.io in aho-corasick 0.7.14
.
from aho-corasick.
Related Issues (20)
- For Python wrapper, is it worth switching to new NFA, or sticking to DFA? HOT 2
- [Note] Investigating "transmute slower than intrinsics for vector unpacking" HOT 1
- Serializing pre-built automaton HOT 7
- RFC: review API before 1.0 release HOT 15
- Benchmarks results aren't clear HOT 11
- Default `anchored` value in a new `Input`. HOT 2
- AhoCorasick::memory_usage doctests fail on i686 (and possibly most 32-bit architectures) HOT 3
- nfa::contiguous::tests::swar test failure on s390x (and possibly all big-endian architectures) HOT 3
- Question: Suggestions for handling . wildcards within query patterns HOT 1
- Go wrapper for aho-corasick HOT 10
- A performance issue vs pyahocorasick HOT 3
- Async support HOT 3
- Document order of returned matches HOT 1
- Miri test fails because of neon intrinsic in aho_corasick HOT 12
- `_mm256_loadu_si256` is failed to be inlined for ABI issues HOT 4
- issue with target_feature in Conditional Compilation Tags HOT 3
- Default automaton HOT 7
- Is it ok to replace memchr with stringzilla for better performance? HOT 1
- Make len() available on `Match` HOT 1
- question: comparisson with a regex union. HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from aho-corasick.