URLs with trailing white space continuously re-fetched about news-crawl HOT 4 CLOSED

I would look at the seed URLs from #12 ; this looks like the 3-field CSV of [URL,Language,Region] was treated as a single URL.

from news-crawl.

Thanks, @john-hewitt! That explains how these URLs went into the crawl. A quick fix would be to just remove them.

However, they are currently re-fetched every 3-5 minutes which is definitely a bug. Better to analyze while it's hot. It's an unnecessary waste of resources, and what's worse, some of them are repeatedly written into the WARC files.

from news-crawl.

Ok, got it: it's a record with a trailing space

      "_source" : {
        "url" : "http://www.topix.com/world/burkina-faso/,ENG,Foreign ",
        "status" : "DISCOVERED",
        "metadata" : { },
        "hostname" : "topix.com",
        "nextFetchDate" : "2016-12-21T16:14:40.784Z"
      }

The trailing is lost in the topology when FetcherBolt converts the URL string into a java.net.URL object. In consequence, the original record never gets updated.

from news-crawl.

Removed all troublesome URLs from Elasticsearch.

from news-crawl.