Hi! If there is using command 'use_frameworks!' in Podfile, BRFullTextSearch will not compile and huge amount of errors will appear (at least I faced such case). Removing that command from Podfile fixed me the issue. Maybe it will be good idea put this info to readme.
Thanks.

Hello

Is it possible to do a search in multiple properties at once ? For example, if my Class has a Title and a Description, can I search in the title and the description in one query ?

And is it possible to "filter" my results to "all the words were found" ?
And can I define a list of words that I don't want to be searched (like "the")

Thanks

Sometimes, exact matches don't show up as the first result. Is there a way to configure BRFullTextSearch/Lucene to make sure that exact matches show up as the first result? Sometimes it's the first one and sometimes not.

I'm seeing this crash very sporadically since updating to 2.0.0-beta1. Any clues as to whats causing this?

Crashed: us.bluerocket.CLucene.IndexWrite
0  App                         0x10e3c9281 lucene::index::SegmentMerger::appendPostings(lucene::index::SegmentMergeInfo**, int) (SegmentMerger.cpp:630)
1  App                         0x10e3c91c1 lucene::index::SegmentMerger::mergeTermInfo(lucene::index::SegmentMergeInfo**, int) (SegmentMerger.cpp:594)
2  App                         0x10e3c9036 lucene::index::SegmentMerger::mergeTermInfos() (SegmentMerger.cpp:550)
3  App                         0x10e3c7f90 lucene::index::SegmentMerger::mergeTerms() (SegmentMerger.cpp:449)
4  App                         0x10e3c72c9 lucene::index::SegmentMerger::merge(bool) (SegmentMerger.cpp:132)
5  App                         0x10e3a2e88 lucene::index::IndexWriter::mergeMiddle(lucene::index::MergePolicy::OneMerge*) (IndexWriter.cpp:2076)
6  App                         0x10e39f5f8 lucene::index::IndexWriter::merge(lucene::index::MergePolicy::OneMerge*) (IndexWriter.cpp:1791)
7  App                         0x10e3a8ca4 lucene::index::SerialMergeScheduler::merge(lucene::index::IndexWriter*) (MergeScheduler.cpp:28)
8  App                         0x10e39db45 lucene::index::IndexWriter::optimize(int, bool) (IndexWriter.cpp:854)
9  App                         0x10e396554 lucene::index::IndexModifier::optimize() (IndexModifier.cpp:149)
10 App                         0x10e3f034c __55-[CLuceneSearchService bulkUpdateIndex:queue:finished:]_block_invoke (CLuceneSearchService.mm:285)
11 libdispatch.dylib              0x7fff89dae93d _dispatch_call_block_and_release + 12
12 libdispatch.dylib              0x7fff89da340b _dispatch_client_callout + 8
13 libdispatch.dylib              0x7fff89da803b _dispatch_queue_drain + 754
14 libdispatch.dylib              0x7fff89dae707 _dispatch_queue_invoke + 549
15 libdispatch.dylib              0x7fff89da6d53 _dispatch_root_queue_drain + 538
16 libdispatch.dylib              0x7fff89da6b00 _dispatch_worker_thread3 + 91
17 libsystem_pthread.dylib        0x7fff97bd54de _pthread_wqthread + 1129
18 libsystem_pthread.dylib        0x7fff97bd3341 start_wqthread + 13

Unable to build project, used pod BRFullTextSearch. Please let me know how to proceed further

I think this is worth noting despite the fact it's based on a beta version of Xcode.

The BRFullTextSearchMacOS framework fails to build under Xcode 9.2 beta's experimental new build system. It complains that the nodes _FastCharStreamer.h, Scorer.h, and libstemmer.h are produced by multiple commands.

This is reproducible by downloading the latest version of master, opening it in Xcode 9.2b, switching the target to BRFullTextSearchMacOS, changing to the new build system using the File->Workspace Settings menu and attempting to build.

This method of querying can be quite useful when constructing a query out of user-supplied query text. For example, you could support prefix based queries (for example, searching for ca* to match cat), how can I do this? Thanks in advance.

How come we have 4f9ea38 (and its later merge to master) if the CLucene code is now in a different pod/project? Looks like the Pods folder was versioned which doesn't follow CocoaPods best practices.

Hi,

Been trying to get search to work in Swift. I've spent the last week or so figuring out how to scrape a lot of HTML and have about 220,000 documents in Lucene. Searching with Java works really well. Exactly what I need.

Now I'm trying to bring it to iOS with Swift. I've got the framework set up correctly, and the CLuceneSearchService class initializes fine.

func application(application: UIApplication, didFinishLaunchingWithOptions launchOptions: [NSObject: AnyObject]?) -> Bool {

        searchService = CLuceneSearchService(indexPath:  NSBundle.mainBundle().resourcePath!.stringByAppendingString("/LuceneIndexes"))
        if (searchService as BRSearchService?) == nil {
            print("unable to intialize search database")
        }

        let results:BRSearchResults = searchService.search("piano")
        NSLog("Results: %i", results.count())

        return true
    }

When this code runs, it always returns 0. I'm not sure if:

1. CLuceneSearchService isn't actually reading the imported indexes (I've checked that the path is correct and files are recognized to already exist)... How do I check how many documents CLuceneSearchService is looking at? IndexReader doesn't seem to be available?
2. If I need to recreate the indexes with iOS code and not Java code--although I thought the whole point was it wouldn't matter

Any pointers on why this isn't working? The term "piano" should hit many terms (134 hits in the Java code).

Thanks

p.s. Here's the Java code for the search:

public static void searchIndex(String searchString) throws IOException, ParseException {
        System.out.println("Searching for '" + searchString + "'");
        Directory directory = FSDirectory.getDirectory(INDEX_DIRECTORY);
        IndexReader indexReader = IndexReader.open(directory);
        IndexSearcher indexSearcher = new IndexSearcher(indexReader);

        Analyzer analyzer = new StopAnalyzer();

        QueryParser queryParser = new QueryParser(JSON_WORD_SEARCH, analyzer);
        Query query = queryParser.parse(searchString);
        Hits hits = indexSearcher.search(query);
        System.out.println("Number of hits: " + hits.length());

        Iterator<Hit> it = hits.iterator();
        while (it.hasNext()) {
            Hit hit = it.next();
            org.apache.lucene.document.Document document = hit.getDocument();
            String path = document.get(JSON_WORD_DISPLAY);
            System.out.println("Hit: " + path);
        }

    }

Hi, first of all, thanks for making this awesome project! It's exactly what I needed.

In order to adjust it to my app's needs, I had to make a similar implementation of the default analyzer, and use a KeywordTokenizer instead of a StandardTokenizer, because I have some codes that have leading numbers (1 digit) and those numbers were being discarded when indexing. KeywordTokenizer makes it work.

However, when using the same approach as the standard analyzer, there seems to be a crash on the StopFilter, when the stop work set is empty. It seems that the first index gets filled with invalid data (even though is set to NULL by code, like the standard analyzer does), therefore tries to compare pointers' data, and it crashes because of an invalid memory address.

Right now, I'm making it work by just never add the StopFilter, I'm not using that anyways. But I'm afraid it crashes on other parts later. So far it didn't, but you never know...

Did that ever happened to you? Did you have to use any special C++ compiler flag to make sure that doesn't happen? Thanks!

Tried using BRFullTextSearch in a simple sample Swift project and got the following error:

:1:9: note: in file included from :1:

import "Headers/BRFullTextSearch-umbrella.h"

    ^

/Users/adamgask/Library/Developer/Xcode/DerivedData/BRFullTextSearchTest-dishthlubmfdgmescfmvlyszdwot/Build/Products/Debug-iphonesimulator/BRFullTextSearch.framework/Headers/BRFullTextSearch-umbrella.h:16:9: note: in file included from /Users/adamgask/Library/Developer/Xcode/DerivedData/BRFullTextSearchTest-dishthlubmfdgmescfmvlyszdwot/Build/Products/Debug-iphonesimulator/BRFullTextSearch.framework/Headers/BRFullTextSearch-umbrella.h:16:

import "CJKAnalyzer.h"

    ^

/Users/adamgask/Library/Developer/Xcode/DerivedData/BRFullTextSearchTest-dishthlubmfdgmescfmvlyszdwot/Build/Products/Debug-iphonesimulator/BRFullTextSearch.framework/Headers/CJKAnalyzer.h:10:10: error: 'CLucene/analysis/AnalysisHeader.h' file not found

include "CLucene/analysis/AnalysisHeader.h"

     ^

/Users/adamgask/Downloads/BRFullTextSearchTest/BRFullTextSearchTest/ViewController.swift:10:8: error: could not build Objective-C module 'BRFullTextSearch'
import BRFullTextSearch
^

Seems like there is a public header file issue, which does get resolved if I comment out use_frameworks! of my Podfile, but this means I can't use it within my current Swift application.

Is it possible to use BRFullTextSearch in Swift?

Sample repo can be found here.
Simply run Pod install and open & build the workspace.

We're using BRFullTextSearch version 1.1.1.

Would you have an idea of why we might be experiencing crashes like the one below? We can't reproduce this locally and can't see what might be the cause – maybe we've been using the API wrong and have just been lucky so far...

Exception Type: SIGSEGV
Exception Codes: SEGV_ACCERR at 0x18
Crashed Thread: 7

Thread 7 Crashed:
0 libsystem_malloc.dylib 0x00000001822c3314 szone_free + 700
1 CoreFoundation 0x00000001825d8f74 CFRelease + 1084
2 CoreFoundation 0x00000001825e7ee0 -[__NSDictionaryI dealloc] + 140
3 App 0x0000000100255f40 -[BRSimpleIndexable .cxx_destruct] (BRSimpleIndexable.m:15)
4 libobjc.A.dylib 0x0000000181d62b54 object_cxxDestructFromClass(objc_object*, objc_class*) + 144
5 libobjc.A.dylib 0x0000000181d6e040 objc_destructInstance + 88
6 libobjc.A.dylib 0x0000000181d6e0a0 object_dispose + 24
7 libswiftCore.dylib 0x0000000100e03278 0x100c28000 + 1946232
8 libswiftCore.dylib 0x0000000100c46d7c 0x100c28000 + 126332
9 libswiftCore.dylib 0x0000000100c76c64 0x100c28000 + 322660
10 libswiftCore.dylib 0x0000000100c76ba8 0x100c28000 + 322472
11 App 0x00000001001495fc 0x100000000 + 1349116
12 libobjc.A.dylib 0x0000000181d62b54 object_cxxDestructFromClass(objc_object*, objc_class*) + 144
13 libobjc.A.dylib 0x0000000181d6e040 objc_destructInstance + 88
14 libobjc.A.dylib 0x0000000181d6e0a0 object_dispose + 24
15 Foundation 0x0000000183004ef4 -[NSOperation dealloc] + 88
16 Foundation 0x00000001830baa18 __NSOQSchedule_f + 448
17 libdispatch.dylib 0x000000018214947c _dispatch_client_callout + 12
18 libdispatch.dylib 0x00000001821554c0 _dispatch_queue_drain + 860
19 libdispatch.dylib 0x000000018214cf80 _dispatch_queue_invoke + 460
20 libdispatch.dylib 0x0000000182157390 _dispatch_root_queue_drain + 724
21 libdispatch.dylib 0x00000001821570b0 _dispatch_worker_thread3 + 108
22 libsystem_pthread.dylib 0x0000000182361470 _pthread_wqthread + 1088
23 libsystem_pthread.dylib 0x0000000182361020 start_wqthread + 0

The stack seems to describe an NSOperation being deallocated, which in turn causes a BRSimpleIndexable being deallocated along with its NSDictionary, which triggers the SEGV_ACCERR – presumably because the NSDictionary was already deallocated.

The implementation of the NSOperation in question looks like this:

class IndexOperation : Operation {
    let searchService: BRSearchService
    let updateBlock: ((BRSearchService) -> Void)

    init(searchService: BRSearchService, block: @escaping ((BRSearchService) -> Void)) {
        self.searchService = searchService
        self.updateBlock = block
        super.init()
        self.queuePriority = .low
    }

    override func main() {
        if self.isCancelled { return }
        self.updateBlock(self.searchService)
        if self.isCancelled { return }
    }
}

And we're using it like this:

let service: BRSearchService // this is a global instance of CLuceneSearchService
let fields = [kBRSearchFieldNameTitle:"content title", kBRSearchFieldNameValue: "content body"]
let indexable = BRSimpleIndexable(identifier: "contentIdentifier", data: fields)
let operation = IndexOperation(searchService: service, block: { (service) in
            var error: NSError? = nil
            service.addObject(toIndexAndWait: indexable, error: &error)
            if let error = error { debugPrint("Error in addDocumentToIndex: \(error)") }
        })
let queue: OperationQueue = SearchQueue.globalQueue
queue.add(operation)

Basically, we're creating a BRSimpleIndexable and passing it to an NSOperation that goes into an NSOperationQueue, along with the BRSearchService (a global instance of CLuceneSearchService that we hold onto for the lifetime of the app). We're not holding onto the dictionary the BRSimpleIndexable was created with so the failure upon deallocation would seem to be internal to BRSimpleIndexable.

Then again, looking at BRSimpleIndexable.m, I don't see how that could be the culprit except that it leaks a reference to the NSDictionary through - (NSDictionary *)indexFieldsDictionary. I tried looking at CLuceneSearchService.mm but can't see anything there that would be trouble either.

We're doing all the index operations (add/remove) as well as all search operations through the same NSOperationQueue that has maxConcurrentOperationCount = 1.

Any idea what might be the problem here?

Say that there is a document in the index that has a value of:

"This is a document"

I'd like [self.searchService search:query] to return the document if query = @"doc"

Is there a way to do this currently? If not, would it be difficult to add? I do believe that Lucene supports substring searches, but don't know much about it.

I'm running the SampleCocoaPodsProject and everywhere I'm trying to debug, I can't seem to be able to access local variables in the debugger (if I try to 'po some_local_var'). It gives some 'Couldn't materialize: couldn't get the value of variable --some var name goes here-- : variable not available
Errored out in Execute, couldn't PrepareToExecuteJITExpression' error.

I tried doing a separate project and it's the same issue, everything works in the debugger until i add this library and use it.

In the project settings, I looked at compiler flags, optimizations, stuff like that, to be set correctly (no optimizations etc), but didn't find anything wrong with them

Any ideas on why is this happening?

First of all thank you for providing an Objective-C wrapper around Lucene! This is excellent.

I'd like to use the static framework in a project that targets both Mac OS and iOS. Right now it's building for iOS only. I suspect there is nothing in the code that is iOS specific and that building for Mac OS is a matter of creating a new target with different build settings while ensuring everything else is the same.

I understand that you may not have a Mac OS use case, so I'm going to try this and may make a pull request, but I wanted to mention it because you might be able to add a Mac OS target more quickly, being familiar with the code already.

I keep running into this issue. I copied the example usage exactly and ran into this exception from CLucene

_CLTHROWA(CL_ERR_IO, (string("No segments* file found in ") + directory->toString()).c_str());

Is there some permission I'm not doing right? I even tried:

NSString *tmpDir = NSTemporaryDirectory();
service = [[CLuceneSearchService alloc] initWithIndexPath:tmpDir];

Instead of the default one in the example usage

Please help to use addIndexes() to add already indexed file with current indexed file . Thanks in advance.

On moving to XCode 10.1 from version 9, this can no longer be compiled as part of a project. Apparently, the libstdc++ library isn't being included as of this release.

Is there a chance of BRFullTextSearch getting an update to bring it to where it is compatible with XCode 10.1?

Is there an easy way to detect if the index file is corrupted? We're seeing a very small proportion of our users have a power/disk failure and on restarting, the index file becomes unreadable and they can't get back into the app since the app crashes on

_searchService = [[CLuceneSearchService alloc] initWithIndexPath:[self searchServicePath]];

I don't know much about this Pod, or CLucene, but I'm faced with either adding multi-field sorting or abandoning this for another solution. Before I waste too much time, can someone say whether this would this work?

In CLuceneSearchService changing the sortBy argument in the internal searchWithQuery:sortBy:sortType:ascending: from an NSString to an NSArray, and its implementation like:

- (id<BRSearchResults>)searchWithQuery:(std::auto_ptr<Query>)query
                              sortBy:(NSArray *)sortFieldNames
                            sortType:(BRSearchSortType)sortType
                           ascending:(BOOL)ascending {
    std::tr1::shared_ptr<Searcher> s = [self searcher];
    if ( sortFieldNames != nil && sortFieldNames.count > 0 ) {
        SortField *sortFields[sortFieldNames.count + 2];
        int idx = 0;
        for (NSString *sortFieldName in sortFieldNames) {
            SortField *sortField = new SortField([sortFieldName asCLuceneString], (sortType == BRSearchSortTypeInteger
                                                                                   ? SortField::INT
                                                                                   : SortField::STRING), !ascending);
            sortFields[idx++];
        }
        SortField *docFieldSort = ascending ? SortField::FIELD_DOC() : new SortField(NULL, SortField::DOC, true);
        sortFields[idx++] = docFieldSort; // ensures we have consistent results if a sort field has duplicate values
        sortFields[idx] = NULL; // requires NULL last element
        std::auto_ptr<Sort> sort(new Sort(sortFields)); // assumes ownership of each SortField
        std::auto_ptr<Hits> hits(s->search(query.get(), sort.get()));
        return [[CLuceneSearchResults alloc] initWithHits:hits sort:sort query:query searcher:s];
    }
    std::auto_ptr<Hits> hits(s->search(query.get()));
    std::auto_ptr<Sort> sort;
    return [[CLuceneSearchResults alloc] initWithHits:hits sort:sort query:query searcher:s];
}

Then making the obvious changes to search:sortBy:.. and searchWithPredicate:sortBy:... to call it with ... sortBy:@[sortFieldName]..., and making additional methods search:sortByMultiple:.. and searchWithPredicate:sortByMultiple:....

I can do a pull request if this works.

Hi I am trying to implement a search feature with your search engine, but I do not really understand what I supposed to return as indexObjectType in my custom indexable object implementing BRSearchObjectType protocol. I saw that the default BRSimpleIndexable implementation returns a question mark ('?') and in your Core Data example the StickyNote implementation returns an 'n' character. Could you add some reference to the documentation wether these characters have some special meaning or they are simple used to differentiate the object types, so I can give the arbitrary values?

First off all thank you for the nice API around CLucene, probably you can help me out to analyze a memory consuption problem while using your lib to index a lot of strings.
For example:

NSMutableArray* arrObj = [NSMutableArray new];
// add a document to the index
for (int i=0; i<=100000; i++) {
    id<BRIndexable> doc = [[BRSimpleIndexable alloc] initWithIdentifier:[NSString stringWithFormat:@"%i", i] data:@{
                                                                                                                    kBRSearchFieldNameTitle : @"Special document",
                                                                                                                    kBRSearchFieldNameValue : @"This is a long winded note with really important details in it."
                                                                                                                    }];
    [arrObj addObject:doc];

}
NSError *error = nil;
[service addObjectsToIndexAndWait:arrObj error:&error];

The memory grows over 200MB for indexing this and after indexing the memory stays at 185MB.
Looks like the objects are staying in the memory when they written to the index? Is there a posibillity to flush the indexed objects to free up memory?

Hi,

I'm looking to use BRFullTextSearch for my upcoming musician's dictionary app. What's the performance like? The HTML table of all the entries are about 27MB. I've yet to clean/import/etc into CoreData, but it would be nice to have some idea if BRFullTextSearch will be able to handle the amount of entries. I particularly liked seeing some numbers/graphs here.

Thanks for a wonderful library! Look forward to investigating a bit further.

keehun

Can something be done to avoid these warnings? We're wanting to turn on "Treat Warnings as Errors" and this is now our only blocker to that.

Warning: Multiple build commands for output file .../Build/Products/Debug-iphonesimulator/BRFullTextSearch.framework/Headers/_FastCharStream.h
Warning: Multiple build commands for output file .../Build/Products/Debug-iphonesimulator/BRFullTextSearch.framework/Headers/libstemmer.h
Warning: Multiple build commands for output file .../Build/Products/Debug-iphonesimulator/BRFullTextSearch.framework/Headers/Scorer.h

My predecessor created a seemingly drastic work-around to prevent these warnings, adding the following to the Podfile. I'm less comfortable with this approach, however, and hope there's a better solution.

  # Whether this is the fault of CocoaPods or BRFullTextSearch or Xcode, the
  # simplest workaround is to delete the copy headers build phase. This might
  # normally break a build due to missing headers, but CocoaPods copies all
  # headers to Pods/Headers during integration and configures search paths to
  # look there first, so the headers remain available. Given that, this script
  # deletes the copy headers build phase from all the CocoaPods targets.
  installer.pod_targets.each do |target|
    target.native_target.build_phases.reject! { |bp| bp.is_a?(Xcodeproj::Project::Object::PBXHeadersBuildPhase) } if !target.native_target.nil?
  end

By moving the CLucene subspecs into a top-level BRCLucene pod the CLucene code can more easily adapt, such as other forks.

I am receiving the following error when building a project with this pod installed.

-1: Multiple commands produce 'Build/Products/Debug-iphonesimulator/BRFullTextSearch/BRFullTextSearch.framework/Headers/_FastCharStream.h':

1. Target 'BRFullTextSearch' (project 'Pods') has copy command from
   'Pods/BRFullTextSearch/clucene/src/core/CLucene/queryParser/_FastCharStream.h' to
   '/Build/Products/Debug-iphonesimulator/BRFullTextSearch/BRFullTextSearch.framework/Headers/_FastCharStream.h'

2. Target 'BRFullTextSearch' (project 'Pods') has copy command from '
   Pods/BRFullTextSearch/clucene/src/core/CLucene/util/_FastCharStream.h' to '
   Build/Products/Debug-iphonesimulator/BRFullTextSearch/BRFullTextSearch.framework/Headers/_FastCharStream.h'

Any chance there is a simple fix.

Hi,

I'm getting crashes (EXC_CRASH with either SIGABRT or SIGSEGV or an EXC_BAD_ACCESS if the Xcode debugger is connected) from BRFullTextSearch inside lucene::search::PrefixQuery::rewrite(lucene::index::IndexReader *) or lucene::index::SegmentTermDocs::SegmentTermDocs(lucene::index::SegmentReader const *) and I have a hunch that it might relate to threads and concurrent modification of the searcher/query objects while a search is being made.

Here's what's happening:

• The "us.bluerocket.CLucene.IndexWrite" queue is executing [CLuceneSearchService bulkUpdateIndex:queue:finished:] (apparently just having finished modifying the index and thus triggering a call to resetSearcher in the UI thread)
• My own "search queue" is executing [CLuceneSearchService search:] (apparently trying to read the index with a SegmentReader)
• The UI thread is executing [CLuceneSearchService resetSearcher] (apparently closing the same SegmentReader my search queue is using)

The examples in README.md don't feature the use of NSOperationQueue or dispatch_async and CLuceneSearchService.h says that it "supports concurrent updates and queries across different threads", which makes me believe I should be fine doing what I'm doing, i.e. calling bulkUpdateIndexAndWait:error: and search: from an NSOperationQueue but it would be really nice to get confirmation whether I'm trying to use CLuceneSearchService in a way that's not at all intended.

Hello!

Congratulations, this is an excellent iOS port of CLucene, we are using it in our projects.

Any clue about how do we extract small snippets of texts from each match?

Thanks in advance,

Frankai