xubenben / galagosearch Goto Github PK

What steps will reproduce the problem?

1. I Just reference Sorter class like this, 'String name = 
Sorter.class.getSimpleName();'
2. I met java.lang.NoClassDefFoundError: org.galagosearch.tupleflow.Sorter 
Exception ...
3. So I try to reference other class in galago search pronect. But I didn't met 
any Exceptions like above, when I use or reference some other classes in 
org.galagosearch.tupleflow.

this is my 'build.gradle' settings snippets .. 

dependencies {
    compile 'com.android.support:appcompat-v7:18.0.0'


    compile files ('libs/antlr-2.7.7.jar')
    compile files ('libs/galagosearch-core-1.04.jar')
    compile files ('libs/jetty-6.1.5.jar')
    compile files ('libs/jetty-embedded-6.1.5.jar')
    compile files ('libs/jetty-util-6.1.5.jar')
    compile files ('libs/servlet-api-2.5-6.1.5.jar')
    compile files ('libs/stringtemplate-3.0.jar')
    compile files ('libs/xmlenc-0.52.jar')
    compile files ('libs/galagosearch-tupleflow-1.04.jar')
}


What is the expected output? What do you see instead?

 - I want to use Sorter Class for search function integration to Android Project


What version of the product are you using? On what operating system?
 - Recent checkout version of Trunk 

Please provide any additional information below.

 - Please Let me know how CAN apply this search project to Android Applications..

While a stopword list can be supplied during retrieval, this isn't in the
documentation.

Use this as a template for documentation:

To use the stopword remover, add an XML like this to a parameter file
when running queries:

<traversals>
 <traversal>

<class>org.galagosearch.core.retrieval.traversal.RemoveStopwordsTraversal</class
>
    <order>before</order>
    <parameters>
       <word>the</word>
       <word>but</word>
    </parameters>
 </traversal>
</traversals>

For instance:

<parameters>
  <traversals>
  ...
  </traversals>
  <query>
   ...
  </query>
  <query>
  ...
  </query>
  ...
</parameters>

When building  a corpus from TRECTEXT formatted documents, if no <DOCNO> is 
found, the process dies with a NullPointerException.  

TrecTextParser.waitFor can return null, and this condiiton needs to be handled 
in TrecTextParser.parseDocNumber.

This behavior was encountered when parsing a TRECTEXT file which had some 
leading whitespace on lines within the <DOC>..</DOC> tags.

What steps will reproduce the problem?

Follow the indexing steps of the tutorial.

What is the expected output? What do you see instead?
harvey:~/Development/galago-test$ /bin/sh ../galagosearch-1.01/galagosearch-
core/target/appassembler/bin/galago build wiki-small.index wiki-small.corpus 
2009-03-12 08:40:02.925::INFO:  Logging to STDERR via org.mortbay.log.StdErrLog
2009-03-12 08:40:02.926::INFO:  jetty-6.1.5
2009-03-12 08:40:02.995::INFO:  Started [email protected]:51999
Status: http://localhost:51999
Exception in thread "main" java.util.concurrent.ExecutionException: Stage threw 
an exception: 
    at 
org.galagosearch.tupleflow.execution.JobExecutor$JobExecutionStatus.waitForStage
s(JobExecutor
.java:1135)
    at 
org.galagosearch.tupleflow.execution.JobExecutor$JobExecutionStatus.run(JobExecu
tor.java:105
4)
    at org.galagosearch.tupleflow.execution.JobExecutor.runWithServer(JobExecutor.java:1191)
    at org.galagosearch.tupleflow.execution.JobExecutor.runLocally(JobExecutor.java:1215)
    at org.galagosearch.core.tools.App.handleBuild(App.java:121)
    at org.galagosearch.core.tools.App.main(App.java:422)
Caused by: java.io.IOException: Couldn't instantiate a step object: 
org.galagosearch.core.index.DocumentNameWriter
    at 
org.galagosearch.tupleflow.execution.StageInstanceFactory.instantiateStep(StageI
nstanceFactory.
java:147)
    at 
org.galagosearch.tupleflow.execution.StageInstanceFactory.instantiate(StageInsta
nceFactory.java:
100)
    at 
org.galagosearch.tupleflow.execution.StageInstanceFactory.instantiate(StageInsta
nceFactory.java:
80)
    at 
org.galagosearch.tupleflow.execution.ThreadedStageExecutor$InstanceRunnable.run(
ThreadedSt
ageExecutor.java:56)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
    at java.lang.Thread.run(Thread.java:613)
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorI
mpl.java:39)
    at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorA
ccessorImpl.
java:27)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:494)
    at 
org.galagosearch.tupleflow.execution.StageInstanceFactory.instantiateStep(StageI
nstanceFactory.
java:137)
    ... 6 more
Caused by: java.io.FileNotFoundException: wiki-small.index/documentNames (No 
such file or 
directory)
    at java.io.FileOutputStream.open(Native Method)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:179)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:70)
    at org.galagosearch.core.index.DocumentNameWriter.<init>(DocumentNameWriter.java:39)
    ... 11 more

What version of the product are you using? On what operating system?

The fault occurs with the binary 1.0 and 1.01 distributions, as with the src 
1.01 distribution.

harvey:~/Development/galago-test$ uname -a
Darwin harvey.cs.umass.edu 9.6.0 Darwin Kernel Version 9.6.0: Mon Nov 24 
17:37:00 PST 2008; 
root:xnu-1228.9.59~1/RELEASE_I386 i386

Core Solo 1.5GHz, 2GB memory.
Please provide any additional information below.
The fault does not occur on linux:
indri1:~> uname -a
Linux indri1.cs.umass.edu 2.6.12-2.3.legacy_FC3 #1 Sun Feb 19 07:48:10 EST 2006 
i686 i686 
i386 GNU/Linux

This appears to be a race where the top-level index directory has not yet been 
created when the 
DocumentNameWriter is instantiated. Creating the top-level index directory 
manually prior to 
running galago  resolves the FileNotFoundException issue and enables a 
successful index build.

Release a Perl or Python library for parsing Galago queries and modifying
them for quick experimentation.

Query terms that are not escaped should be case folded.

[deleted issue]

Galago supports turning stemming on and off on a term-by-term basis in
retrieval, but this is not documented.

Workaround:

Use this syntax to use unstemmed posting lists:
   #text:dog:part=postings()
Use this syntax to force stemmed posting lists:
   #text:dog:part=stemmedPostings()

It would be convenient to extract term counts from the index using
dump-index, but this isn't implemented.

Workaround:
Use dump-index to dump posting lists, then use a script to convert this
data into term count information.

What steps will reproduce the problem?

build an index.

What is the expected output? What do you see instead?

The /tmp/tupleflow<pid> directory removed after building.

The /tmp/tupleflow<pid> directory remains.

What version of the product are you using? On what operating system?

1.01, Mac OSX 10.5.6

Please provide any additional information below.

If there is a document with an unrecognised file extension in a collection
directory Galago crashes with a NullPointerException, after printing a
message about skipping it.

Running the command:
> bin/galago build /tmp/ap1.index collections/adhoc_colls/ap1/

2009-04-23 17:18:12.827::INFO:  Logging to STDERR via org.mortbay.log.StdErrLog
2009-04-23 17:18:12.828::INFO:  jetty-6.1.5
2009-04-23 17:18:12.843::INFO:  Started [email protected]:40875
Status: http://localhost:40875
Skipping: collections/adhoc_colls/ap1/file_list
Exception in thread "main" java.util.concurrent.ExecutionException: Stage
threw an exception: 
    at
org.galagosearch.tupleflow.execution.JobExecutor$JobExecutionStatus.waitForStage
s(JobExecutor.java:1135)
    at
org.galagosearch.tupleflow.execution.JobExecutor$JobExecutionStatus.run(JobExecu
tor.java:1054)
    at
org.galagosearch.tupleflow.execution.JobExecutor.runWithServer(JobExecutor.java:
1191)
    at
org.galagosearch.tupleflow.execution.JobExecutor.runLocally(JobExecutor.java:121
5)
    at org.galagosearch.core.tools.App.handleBuild(App.java:121)
    at org.galagosearch.core.tools.App.main(App.java:422)
Caused by: java.lang.NullPointerException
    at
org.galagosearch.core.parse.DocumentSource.processFile(DocumentSource.java:124)
    at
org.galagosearch.core.parse.DocumentSource.processDirectory(DocumentSource.java:
139)
    at org.galagosearch.core.parse.DocumentSource.run(DocumentSource.java:150)
    at
org.galagosearch.tupleflow.execution.ThreadedStageExecutor$InstanceRunnable.run(
ThreadedStageExecutor.java:57)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:636)

This occurs in the binary release of Galago 1.01 on Ubuntu 8.10.

The bug appears to be caused in the method processFile of DocumentSource ,
where it prints a message about skipping a file but attempts to index it
anyway.

Galago ships with -Xmx256m as its memory flag, which is much lower than it
was originally tested with.  There are reports of crashes in
ExtentsNumberer with OutOfMemory errors.

It would be good to both increase the memory limit and understand the
crashes better.

What steps will reproduce the problem?
1. start 'galago search <index>'
2. visit the url

What is the expected output? What do you see instead?
there's a 'broken image' placeholder on the web page above the search box 
instead of something 
pretty.

What steps will reproduce the problem?
build galago 1.01 from source via  'mvn package'

What is the expected output? What do you see instead?
scripts in 
galagosearch-1.01/galagosearch-core/target/appassembler/bin/ 
should be executable, but aren't

What version of the product are you using? On what operating system?
galago 1.01,   java 1.6.0, maven 2.0.9, on fedora core 7

Not very important, but annoying when looking at javadoc.

What steps will reproduce the problem?
1. Look at javadoc of for example FeatureFactory
2. In the example > and < should be swapped, this is incorrect in
more examples

Version: galagosearch-1.04-src

The #smoothinside operator is generated by the query parser, but we don't have 
an implementation 
for it.

Non-ASCII characters show up incorrectly in the web interface; in the snippet 
view and in document 
view.  A binary dump of the web server output indicates that the bytes have 
been changed.  
Dumping documents at the command line from corpus file produces correct output.

Also, request.getParameter() does not parse special characters correctly.  If 
there are non-ASCII 
characters in the parameter, they may get dropped.

Need to follow this up with the Jetty list.

What steps will reproduce the problem?
1. Crawl a series of pages with wget
2. Type: bin/galago make-corpus my.corpus crawl-directory
3. Throws an exception with this as the root cause:

Caused by: java.lang.NullPointerException
    at org.galagosearch.core.index.IndexWriter.<init>(IndexWriter.java:59)
    at
org.galagosearch.core.parse.DocumentIndexWriter.<init>(DocumentIndexWriter.java:
37)
    ... 11 more

The textbook indicates that quotes turn off stemming, but quotes aren't
currently allowed by the query parser.

Need to:
- Add quotes to the query parser; output them as #quote
- Add support to the transformations for #quote to #text:...:part=postings()

What steps will reproduce the problem?
1. build an index with the wiki-small dataset 
(http://www.search-engines-book.com/collections/).  This completes normally.
2. run 'galago search <wiki-small index>'
3. go to the URL specified and run a query

What is the expected output? What do you see instead?
Instead of search results, the returned page is empty with the following 
exception printed to std. out:

2009-01-28 13:18:10.348::WARN:  EXCEPTION 
javax.servlet.ServletException: Caught exception from handleSearch
    at org.galagosearch.core.tools.SearchWebHandler.handle(SearchWebHandler.java:303)
    at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139)
    at org.mortbay.jetty.Server.handle(Server.java:313)
    at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:506)
    at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:830)
    at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:514)
    at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:211)
    at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:381)
    at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:227)
    at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)
Caused by: java.lang.NoSuchMethodException: 
org.galagosearch.core.retrieval.traversal.AddCombineTraversal.<init>(org.galagos
earch.core.retrieval.structured.StructuredRetriev
al)
    at java.lang.Class.getConstructor0(Class.java:2706)
    at java.lang.Class.getConstructor(Class.java:1657)
    at org.galagosearch.core.retrieval.structured.FeatureFactory.getTraversals(FeatureFactory.java:371)
    at org.galagosearch.core.retrieval.structured.StructuredRetrieval.transformQuery(StructuredRetrieval.java:79)
    at org.galagosearch.core.tools.Search.runQuery(Search.java:82)
    at org.galagosearch.core.tools.SearchWebHandler.performSearch(SearchWebHandler.java:326)
    at org.galagosearch.core.tools.SearchWebHandler.handleSearch(SearchWebHandler.java:126)
    at org.galagosearch.core.tools.SearchWebHandler.handle(SearchWebHandler.java:301)
    ... 9 more
2009-01-28 13:18:10.353::WARN:  Nested in javax.servlet.ServletException: 
Caught exception from handleSearch:
java.lang.NoSuchMethodException: 
org.galagosearch.core.retrieval.traversal.AddCombineTraversal.<init>(org.galagos
earch.core.retrieval.structured.StructuredRetriev
al)
    at java.lang.Class.getConstructor0(Class.java:2706)
    at java.lang.Class.getConstructor(Class.java:1657)
    at org.galagosearch.core.retrieval.structured.FeatureFactory.getTraversals(FeatureFactory.java:371)
    at org.galagosearch.core.retrieval.structured.StructuredRetrieval.transformQuery(StructuredRetrieval.java:79)
    at org.galagosearch.core.tools.Search.runQuery(Search.java:82)
    at org.galagosearch.core.tools.SearchWebHandler.performSearch(SearchWebHandler.java:326)
    at org.galagosearch.core.tools.SearchWebHandler.handleSearch(SearchWebHandler.java:126)
    at org.galagosearch.core.tools.SearchWebHandler.handle(SearchWebHandler.java:301)
    at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139)
    at org.mortbay.jetty.Server.handle(Server.java:313)
    at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:506)
    at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:830)
    at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:514)
    at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:211)
    at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:381)
    at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:227)
    at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)


What version of the product are you using? On what operating system?
Linux Fedora Core 5, java version "1.6.0_10-rc2", latest Galago release from 
SVN. 
output from 'svn info':
Path: .
URL: http://galagosearch.googlecode.com/svn/trunk
Repository Root: http://galagosearch.googlecode.com/svn
Repository UUID: 029f083a-8788-11dd-9299-37fd5bf03096
Revision: 72
Node Kind: directory
Schedule: normal
Last Changed Author: trevor.strohman
Last Changed Rev: 72
Last Changed Date: 2009-01-26 10:13:59 -0500 (Mon, 26 Jan 2009)

What steps will reproduce the problem?
run 'galago search', 'galago search /some/file/that/does/not/exist', 'galago 
build', etc.

What is the expected output? What do you see instead?
it would be nice to see the help message for a particular command, but instead 
you get a variety of 
unhelpful exceptions:

$ galago search
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 1
    at org.galagosearch.core.tools.App.handleSearch(App.java:209)
    at org.galagosearch.core.tools.App.main(App.java:391)

$ galago search foo
gException in thread "main" java.io.IOException: java.io.FileNotFoundException: 
/usr0/jelsas/Data/foo/manifest (No such file or directory)
    at org.galagosearch.tupleflow.Parameters.parse(Parameters.java:641)
    at org.galagosearch.core.index.StructuredIndex.<init>(StructuredIndex.java:32)
    at 
org.galagosearch.core.retrieval.structured.StructuredRetrieval.<init>(Structured
Retrieval.java:37)
    at org.galagosearch.core.retrieval.Retrieval.instance(Retrieval.java:27)
    at org.galagosearch.core.tools.App.handleSearch(App.java:222)
    at org.galagosearch.core.tools.App.main(App.java:391)

$ galago build
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 1
    at org.galagosearch.core.tools.App.handleBuild(App.java:114)
    at org.galagosearch.core.tools.App.main(App.java:377)

- Documentation for "galago eval" refers to ireval.jar, which no longer exists.
- The output has some lines that are inappropriately labeled, like
h-signtest-0.1
- The output references a 'k' value that is not documented.

At query time, allow users to set stemming parameters globally instead of
on a per-term basis.

#wsyn is mentioned in the book but not implemented in the code.

Currently the smoothing parameters are dirichlet:mu=1500 and can be changed
in the query.  Add a default smoothing parameter so that it can be changed
from a parameters file.

Jetty logs information like this to stdout:
2009-05-09 21:09:53.313::INFO:  Logging to STDERR via org.mortbay.log.StdErrLog
2009-05-09 21:09:53.313::INFO:  jetty-6.1.5
2009-05-09 21:09:53.344::INFO:  Started [email protected]:4564

This is potentially confusing to users.  It'd be nice to turn this off.

What steps will reproduce the problem?
1. Run batch query on CACM corpus with

<parameters>
     <query>
        <number>CACM1a</number>
        <text>#combine(articles TSS jolly)</text>
     </query>
     <query>
        <number>CACM-1b</number>
        <text> #weight(1.0 articles 350.0 TSS 2.8 jolly)
</text>
     </query>
 <query>
        <number>CACM-1c</number>
        <text> #weight(1.0 articles 1.0 TSS 2.8 jolly)
</text>
     </query>
  </parameters>


2. Results and scores will not be sensitive to weights

What is the expected output? What do you see instead?

Results should vary with weights

What version of the product are you using? On what operating system?

svn revision 123 on Mac and Unix

Please provide any additional information below.

A simple one-line fix to 


org.galagosearch.core.retrieval.structured.ScaleIterator.java suffices. 

Change

weight = parameters.get("weight", 1.0);

to 

weight = parameters.get("default", 1.0);

N.B. This is fixing the SYMPTOM, not the problem. The problem is higher up, 
when the
parameters are being created. They should have "weight" as a key, but they do 
not.

Unrecognized file extensions, like .cgi, cause the indexer to throw
exceptions.  Instead, it should skip those files and log that they were
skipped. (reported by jelsas)

Allow users to see document scores in debug mode in the web interface.

What steps will reproduce the problem?
running 'galago build' , pointing to a directory of HTML documents (~300k) from 
a site mirror created through 'wget'.

Linux Fedora Core 5, java version "1.6.0_10-rc2", latest Galago release from 
SVN. 
output from 'svn info':
Path: .
URL: http://galagosearch.googlecode.com/svn/trunk
Repository Root: http://galagosearch.googlecode.com/svn
Repository UUID: 029f083a-8788-11dd-9299-37fd5bf03096
Revision: 72
Node Kind: directory
Schedule: normal
Last Changed Author: trevor.strohman
Last Changed Rev: 72
Last Changed Date: 2009-01-26 10:13:59 -0500 (Mon, 26 Jan 2009)



received exception near the end (?) of building the index:
Exception in thread "main" java.util.concurrent.ExecutionException: Stage threw 
an exception:
        at org.galagosearch.tupleflow.execution.JobExecutor$JobExecutionStatus.waitForStages(JobExecutor.java:1135)
        at org.galagosearch.tupleflow.execution.JobExecutor$JobExecutionStatus.run(JobExecutor.java:1073)
        at org.galagosearch.tupleflow.execution.JobExecutor.runWithServer(JobExecutor.java:1191)
        at org.galagosearch.tupleflow.execution.JobExecutor.runLocally(JobExecutor.java:1215)
        at org.galagosearch.core.tools.App.handleBuild(App.java:116)
        at org.galagosearch.core.tools.App.main(App.java:377)
Caused by: java.io.IOException: Key is too long.
        at org.galagosearch.core.index.IndexWriter.add(IndexWriter.java:454)
        at org.galagosearch.core.index.ExtentIndexWriter.processExtentName(ExtentIndexWriter.java:48)
        at 
org.galagosearch.core.types.NumberedExtent$ExtentNameNumberBeginOrder$DuplicateE
liminator.processExtentName(NumberedExtent.jav
a:802)
        at 
org.galagosearch.core.types.NumberedExtent$ExtentNameNumberBeginOrder$ShreddedBu
ffer.copyUntilExtentName(NumberedExtent.java:
457)
        at org.galagosearch.core.types.NumberedExtent$ExtentNameNumberBeginOrder$ShreddedBuffer.copyUntil(NumberedExtent.java:528)
        at org.galagosearch.core.types.NumberedExtent$ExtentNameNumberBeginOrder$ShreddedCombiner.run(NumberedExtent.java:587)
        at org.galagosearch.tupleflow.OrderedCombiner.run(OrderedCombiner.java:141)
        at org.galagosearch.tupleflow.execution.ThreadedStageExecutor$InstanceRunnable.run(ThreadedStageExecutor.java:57)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)

The galago build command takes some optional flags, but they currently must
come after the index name, e.g.
   galago build <index> (flags)* <input-files>+
This is unintuitive and does not match the help documentation.

Date operators (#datebefore, #dateafter, etc.) are missing.  This also
includes date extraction.

aaaaaaaaaa

[deleted issue]

What steps will reproduce the problem?
1.i download the galagosearch-1.04-bin package, but while i execute the command 
as the example "% bin/galago build /tmp/wiki-small.index 
/tmp/wiki-small.corpus" , i could not run it, since the file does not exist on 
my PC.  where i can get it, or who can give me a example?


error infor on my PC:


\galagosearch-1.04-bin\galagosearch-1.04>.\bin\galago.bat build 
/tmp/wiki-small.index /tmp/wiki-small.corpus
Exception in thread "main" java.io.IOException: Couldn't find file/directory: /t
mp/wiki-small.corpus
        at org.galagosearch.core.tools.BuildIndex.getSplitStage(BuildIndex.java:
89)
        at org.galagosearch.core.tools.BuildIndex.getIndexJob(BuildIndex.java:39
4)
        at org.galagosearch.core.tools.App.handleBuild(App.java:132)
        at org.galagosearch.core.tools.App.run(App.java:435)
        at org.galagosearch.core.tools.App.main(App.java:458)