There appears to be a problem with the following construct
imaps://[email protected]:[email protected]:993/INBOX
which results in
May 09, 2021 6:50:36 AM de.dfki.km.leech.sax.PrintlnContentHandler processErrorData
INFO: ## PrintlnContentHandler ERROR data ##########################
no data entity id known - in the case of a sub-entity, set it inside the metadata at your implementation of getSubDataEntitiesInformation(..)
under the key CrawlerParser.SOURCEID. Otherwise you maybe try to process an unsupported/broken URL, or it is totally strange.
metadata:
errorStacktrace: 'javax.mail.AuthenticationFailedException: failed to connect, no password specified?
at javax.mail.Service.connect(Service.java:400)
at de.dfki.km.leech.parser.ImapCrawlerParser.connect2Server(ImapCrawlerParser.java:121)
at de.dfki.km.leech.io.ImapURLStreamProvider.addFirstMetadata(ImapURLStreamProvider.java:85)
at de.dfki.km.leech.Leech.parse(Leech.java:671)
at de.dfki.km.leech.Leech.parse(Leech.java:487)
at de.dfki.km.leech.Leech.main(Leech.java:110)
It is a requirement of the email server to use [email protected] as the username, aliases cannot be used as they also require the same construct.
The email server works ok with the mail client supplied as part of the operating system.
Debugging the code shows that the construct is altered to
imaps://foo%40myemailerserver.lan:[email protected]:993/INBOX
Running with this construct causes
metadata:
X-Parsed-By: 'org.apache.tika.parser.CompositeParser'
X-Parsed-By: 'de.dfki.km.leech.parser.ImapCrawlerParser'
errorStacktrace: 'org.apache.tika.exception.TikaException: Error while crawling 'imaps://[email protected]:[email protected]:993/INBOX'
at de.dfki.km.leech.parser.CrawlerParser.parse(CrawlerParser.java:230)
at de.dfki.km.leech.parser.ImapCrawlerParser.parse(ImapCrawlerParser.java:383)
at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:280)
at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:280)
at org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:143)
at de.dfki.km.leech.parser.incremental.IncrementalCrawlingParser.parse(IncrementalCrawlingParser.java:170)
at de.dfki.km.leech.parser.filter.URLFilteringParser.parse(URLFilteringParser.java:119)
at de.dfki.km.leech.Leech.parse(Leech.java:675)
at de.dfki.km.leech.Leech.parse(Leech.java:487)
at de.dfki.km.leech.Leech.main(Leech.java:110)
Caused by: java.lang.NullPointerException
at de.dfki.km.leech.parser.ImapCrawlerParser.getSubDataEntitiesInformation(ImapCrawlerParser.java:354)
at de.dfki.km.leech.parser.CrawlerParser.parse(CrawlerParser.java:170)
... 9 more
errorMessage: 'Error while crawling “imaps://foo%40myemailerserver.lan:[email protected]:993/INBOX''
dataEntityId: 'imaps://foo%40myemailerserver.lan:@myemailerserver.lan:993/INBOX'
resourceName: 'imaps://foo%40myemailerserver.lan:@myemailerserver.lan:993/INBOX'
source: 'imaps://foo%40myemailerserver.lan:@myemailerserver.lan:993/INBOX'
dataEntityContentFingerprint: 'INBOX'
Content-Type: 'remotedatasource/imapfolder'
May 09, 2021 6:58:40 AM de.dfki.km.leech.sax.CrawlReportContentHandler crawlFinished
INFO: Crawl finished:
Report: First handled data entity at 09/05/21 06:58, 1 processed entities, duration 0ms
New data entities: 0
Modified data entities: 0
Removed data entities: 0
Unmodified data entities: 0
Double data entities: 0
Error data entities: 1
remotedatasource/imapfolder:1
On a more positive note, tests with files and web pages work.