dfki / leechcrawler Goto Github PK

Hi,

I got this exception while crawling directory with single .doc file.
java.lang.ClassNotFoundException: org.apache.http.util.Args.

Adding the following dependency to the maven build resoles the issue.

        <dependency>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpcore</artifactId>
            <version>4.4</version>
        </dependency>

Thanks,
Wazhar

I noticed several instances of 'de.dfki.km.leech.parser.incremental.IncrementCrawlingHistory' remains in memory and never get garbage collected. After bit of an investigation it seems an instance is created per session and sticks to memory forever.

Please refer to the attached screenshot.

Thanks.

This will make the core tool independent from the Lucene version - someone can implement an own LuceneContentHandler with an aribitrary Lucene version

Hi, Any plans for adding Tika 1.7 support ?

Thanks,
Wazhar

Hi,

Tika 1.7 support worked well for me, 1.8 is out. Any plans for the upgrade ?

Appreciate your efforts.

Wazhar

There appears to be a problem with the following construct

imaps://[email protected]:[email protected]:993/INBOX

which results in

May 09, 2021 6:50:36 AM de.dfki.km.leech.sax.PrintlnContentHandler processErrorData
INFO: ## PrintlnContentHandler ERROR data ##########################
no data entity id known - in the case of a sub-entity, set it inside the metadata at your implementation of getSubDataEntitiesInformation(..)
under the key CrawlerParser.SOURCEID. Otherwise you maybe try to process an unsupported/broken URL, or it is totally strange.

metadata:

errorStacktrace: 'javax.mail.AuthenticationFailedException: failed to connect, no password specified?
at javax.mail.Service.connect(Service.java:400)
at de.dfki.km.leech.parser.ImapCrawlerParser.connect2Server(ImapCrawlerParser.java:121)
at de.dfki.km.leech.io.ImapURLStreamProvider.addFirstMetadata(ImapURLStreamProvider.java:85)
at de.dfki.km.leech.Leech.parse(Leech.java:671)
at de.dfki.km.leech.Leech.parse(Leech.java:487)
at de.dfki.km.leech.Leech.main(Leech.java:110)

It is a requirement of the email server to use [email protected] as the username, aliases cannot be used as they also require the same construct.
The email server works ok with the mail client supplied as part of the operating system.

Debugging the code shows that the construct is altered to

imaps://foo%40myemailerserver.lan:[email protected]:993/INBOX

Running with this construct causes

metadata:

X-Parsed-By: 'org.apache.tika.parser.CompositeParser'
X-Parsed-By: 'de.dfki.km.leech.parser.ImapCrawlerParser'
errorStacktrace: 'org.apache.tika.exception.TikaException: Error while crawling 'imaps://[email protected]:[email protected]:993/INBOX'
at de.dfki.km.leech.parser.CrawlerParser.parse(CrawlerParser.java:230)
at de.dfki.km.leech.parser.ImapCrawlerParser.parse(ImapCrawlerParser.java:383)
at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:280)
at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:280)
at org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:143)
at de.dfki.km.leech.parser.incremental.IncrementalCrawlingParser.parse(IncrementalCrawlingParser.java:170)
at de.dfki.km.leech.parser.filter.URLFilteringParser.parse(URLFilteringParser.java:119)
at de.dfki.km.leech.Leech.parse(Leech.java:675)
at de.dfki.km.leech.Leech.parse(Leech.java:487)
at de.dfki.km.leech.Leech.main(Leech.java:110)
Caused by: java.lang.NullPointerException
at de.dfki.km.leech.parser.ImapCrawlerParser.getSubDataEntitiesInformation(ImapCrawlerParser.java:354)
at de.dfki.km.leech.parser.CrawlerParser.parse(CrawlerParser.java:170)
... 9 more

errorMessage: 'Error while crawling “imaps://foo%40myemailerserver.lan:[email protected]:993/INBOX''
dataEntityId: 'imaps://foo%40myemailerserver.lan:@myemailerserver.lan:993/INBOX'
resourceName: 'imaps://foo%40myemailerserver.lan:@myemailerserver.lan:993/INBOX'
source: 'imaps://foo%40myemailerserver.lan:@myemailerserver.lan:993/INBOX'
dataEntityContentFingerprint: 'INBOX'
Content-Type: 'remotedatasource/imapfolder'

May 09, 2021 6:58:40 AM de.dfki.km.leech.sax.CrawlReportContentHandler crawlFinished
INFO: Crawl finished:
Report: First handled data entity at 09/05/21 06:58, 1 processed entities, duration 0ms
New data entities: 0
Modified data entities: 0
Removed data entities: 0
Unmodified data entities: 0
Double data entities: 0
Error data entities: 1
remotedatasource/imapfolder:1

On a more positive note, tests with files and web pages work.

Crawling word document is failing with following exception

org.apache.tika.exception.TikaException: Unexpected RuntimeException from org.apache.tika.parser.microsoft.OfficeParser

Try same document with Tika 1.8 directly and it does not fail. I will attach the document.

WAzhar