Giter Club home page Giter Club logo

crawler's Introduction

Hi there 馃憢

Thanks for checking out my profile! Hope you find the repository you're looking for. If you have an open-source project and need help to get it done, give me a shout.

Rafael's GitHub stats

crawler's People

Contributors

rafaeljusto avatar

Stargazers

avatar

Watchers

avatar avatar

crawler's Issues

Race conditions

When running

go test -cover -race

we got many race conditions errors:

==================
WARNING: DATA RACE
Write by goroutine 21:
  github.com/rafaeljusto/crawler.func路001()
      github.com/rafaeljusto/crawler/_test/crawler.go:25 +0x4e

Previous write by goroutine 18:
  github.com/rafaeljusto/crawler.func路001()
      github.com/rafaeljusto/crawler/_test/crawler.go:25 +0x4e

Goroutine 21 (running) created at:
  github.com/rafaeljusto/crawler.Crawl()
      github.com/rafaeljusto/crawler/_test/crawler.go:28 +0x36a
  github.com/rafaeljusto/crawler.TestCrawlMustFollowLinks()
      /home/rafael/Projects/go/src/github.com/rafaeljusto/crawler/crawler_test.go:498 +0x1f71
  testing.tRunner()
      /usr/local/go/src/pkg/testing/testing.go:391 +0x10f

Goroutine 18 (finished) created at:
  github.com/rafaeljusto/crawler.Crawl()
      github.com/rafaeljusto/crawler/_test/crawler.go:28 +0x36a
  github.com/rafaeljusto/crawler.TestCrawlMustReturnErrorOnFetchProblems()
      /home/rafael/Projects/go/src/github.com/rafaeljusto/crawler/crawler_test.go:235 +0x29e
  testing.tRunner()
      /usr/local/go/src/pkg/testing/testing.go:391 +0x10f
==================
==================
WARNING: DATA RACE
Write by goroutine 22:
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:70 +0x4a
  github.com/rafaeljusto/crawler.crawlPage()
      github.com/rafaeljusto/crawler/_test/crawler.go:66 +0x289

Previous write by goroutine 20:
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:70 +0x4a
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.crawlPage()
      github.com/rafaeljusto/crawler/_test/crawler.go:66 +0x289

Goroutine 22 (running) created at:
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:106 +0xbf5
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.crawlPage()
      github.com/rafaeljusto/crawler/_test/crawler.go:66 +0x289

Goroutine 20 (finished) created at:
  github.com/rafaeljusto/crawler.Crawl()
      github.com/rafaeljusto/crawler/_test/crawler.go:21 +0x2b7
  github.com/rafaeljusto/crawler.TestCrawlMustFollowLinks()
      /home/rafael/Projects/go/src/github.com/rafaeljusto/crawler/crawler_test.go:498 +0x1f71
  testing.tRunner()
      /usr/local/go/src/pkg/testing/testing.go:391 +0x10f
==================
==================
WARNING: DATA RACE
Write by goroutine 22:
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:169 +0x7b5
  github.com/rafaeljusto/crawler.crawlPage()
      github.com/rafaeljusto/crawler/_test/crawler.go:66 +0x289

Previous write by goroutine 20:
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:169 +0x7b5
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.crawlPage()
      github.com/rafaeljusto/crawler/_test/crawler.go:66 +0x289

Goroutine 22 (running) created at:
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:106 +0xbf5
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.crawlPage()
      github.com/rafaeljusto/crawler/_test/crawler.go:66 +0x289

Goroutine 20 (finished) created at:
  github.com/rafaeljusto/crawler.Crawl()
      github.com/rafaeljusto/crawler/_test/crawler.go:21 +0x2b7
  github.com/rafaeljusto/crawler.TestCrawlMustFollowLinks()
      /home/rafael/Projects/go/src/github.com/rafaeljusto/crawler/crawler_test.go:498 +0x1f71
  testing.tRunner()
      /usr/local/go/src/pkg/testing/testing.go:391 +0x10f
==================
==================
WARNING: DATA RACE
Write by goroutine 22:
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:172 +0x807
  github.com/rafaeljusto/crawler.crawlPage()
      github.com/rafaeljusto/crawler/_test/crawler.go:66 +0x289

Previous write by goroutine 20:
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:172 +0x807
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.crawlPage()
      github.com/rafaeljusto/crawler/_test/crawler.go:66 +0x289

Goroutine 22 (running) created at:
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:106 +0xbf5
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.crawlPage()
      github.com/rafaeljusto/crawler/_test/crawler.go:66 +0x289

Goroutine 20 (finished) created at:
  github.com/rafaeljusto/crawler.Crawl()
      github.com/rafaeljusto/crawler/_test/crawler.go:21 +0x2b7
  github.com/rafaeljusto/crawler.TestCrawlMustFollowLinks()
      /home/rafael/Projects/go/src/github.com/rafaeljusto/crawler/crawler_test.go:498 +0x1f71
  testing.tRunner()
      /usr/local/go/src/pkg/testing/testing.go:391 +0x10f
==================
==================
WARNING: DATA RACE
Write by goroutine 22:
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:72 +0x94
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.crawlPage()
      github.com/rafaeljusto/crawler/_test/crawler.go:66 +0x289

Previous write by goroutine 20:
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:72 +0x94
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.crawlPage()
      github.com/rafaeljusto/crawler/_test/crawler.go:66 +0x289

Goroutine 22 (running) created at:
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:106 +0xbf5
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.crawlPage()
      github.com/rafaeljusto/crawler/_test/crawler.go:66 +0x289

Goroutine 20 (finished) created at:
  github.com/rafaeljusto/crawler.Crawl()
      github.com/rafaeljusto/crawler/_test/crawler.go:21 +0x2b7
  github.com/rafaeljusto/crawler.TestCrawlMustFollowLinks()
      /home/rafael/Projects/go/src/github.com/rafaeljusto/crawler/crawler_test.go:498 +0x1f71
  testing.tRunner()
      /usr/local/go/src/pkg/testing/testing.go:391 +0x10f
==================
==================
WARNING: DATA RACE
Write by goroutine 22:
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:159 +0xcbd
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.crawlPage()
      github.com/rafaeljusto/crawler/_test/crawler.go:66 +0x289

Previous write by goroutine 20:
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:159 +0xcbd
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.crawlPage()
      github.com/rafaeljusto/crawler/_test/crawler.go:66 +0x289

Goroutine 22 (running) created at:
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:106 +0xbf5
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.crawlPage()
      github.com/rafaeljusto/crawler/_test/crawler.go:66 +0x289

Goroutine 20 (finished) created at:
  github.com/rafaeljusto/crawler.Crawl()
      github.com/rafaeljusto/crawler/_test/crawler.go:21 +0x2b7
  github.com/rafaeljusto/crawler.TestCrawlMustFollowLinks()
      /home/rafael/Projects/go/src/github.com/rafaeljusto/crawler/crawler_test.go:498 +0x1f71
  testing.tRunner()
      /usr/local/go/src/pkg/testing/testing.go:391 +0x10f
==================
==================
WARNING: DATA RACE
Write by goroutine 22:
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:161 +0xd9a
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.crawlPage()
      github.com/rafaeljusto/crawler/_test/crawler.go:66 +0x289

Previous write by goroutine 20:
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:161 +0xd9a
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.crawlPage()
      github.com/rafaeljusto/crawler/_test/crawler.go:66 +0x289

Goroutine 22 (running) created at:
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:106 +0xbf5
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.crawlPage()
      github.com/rafaeljusto/crawler/_test/crawler.go:66 +0x289

Goroutine 20 (finished) created at:
  github.com/rafaeljusto/crawler.Crawl()
      github.com/rafaeljusto/crawler/_test/crawler.go:21 +0x2b7
  github.com/rafaeljusto/crawler.TestCrawlMustFollowLinks()
      /home/rafael/Projects/go/src/github.com/rafaeljusto/crawler/crawler_test.go:498 +0x1f71
  testing.tRunner()
      /usr/local/go/src/pkg/testing/testing.go:391 +0x10f
==================
==================
WARNING: DATA RACE
Write by goroutine 22:
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:163 +0xe18
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.crawlPage()
      github.com/rafaeljusto/crawler/_test/crawler.go:66 +0x289

Previous write by goroutine 20:
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:163 +0xe18
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.crawlPage()
      github.com/rafaeljusto/crawler/_test/crawler.go:66 +0x289

Goroutine 22 (running) created at:
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:106 +0xbf5
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.crawlPage()
      github.com/rafaeljusto/crawler/_test/crawler.go:66 +0x289

Goroutine 20 (finished) created at:
  github.com/rafaeljusto/crawler.Crawl()
      github.com/rafaeljusto/crawler/_test/crawler.go:21 +0x2b7
  github.com/rafaeljusto/crawler.TestCrawlMustFollowLinks()
      /home/rafael/Projects/go/src/github.com/rafaeljusto/crawler/crawler_test.go:498 +0x1f71
  testing.tRunner()
      /usr/local/go/src/pkg/testing/testing.go:391 +0x10f
==================
==================
WARNING: DATA RACE
Write by goroutine 28:
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:109 +0x4ea
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.crawlPage()
      github.com/rafaeljusto/crawler/_test/crawler.go:66 +0x289

Previous write by goroutine 26:
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:109 +0x4ea
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.crawlPage()
      github.com/rafaeljusto/crawler/_test/crawler.go:66 +0x289

Goroutine 28 (running) created at:
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:106 +0xbf5
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.crawlPage()
      github.com/rafaeljusto/crawler/_test/crawler.go:66 +0x289

Goroutine 26 (finished) created at:
  github.com/rafaeljusto/crawler.Crawl()
      github.com/rafaeljusto/crawler/_test/crawler.go:21 +0x2b7
  github.com/rafaeljusto/crawler.TestCrawlMustFollowLinks()
      /home/rafael/Projects/go/src/github.com/rafaeljusto/crawler/crawler_test.go:498 +0x1f71
  testing.tRunner()
      /usr/local/go/src/pkg/testing/testing.go:391 +0x10f
==================
==================
WARNING: DATA RACE
Write by goroutine 28:
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:113 +0x50a
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.crawlPage()
      github.com/rafaeljusto/crawler/_test/crawler.go:66 +0x289

Previous write by goroutine 26:
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:113 +0x50a
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.crawlPage()
      github.com/rafaeljusto/crawler/_test/crawler.go:66 +0x289

Goroutine 28 (running) created at:
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:106 +0xbf5
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.crawlPage()
      github.com/rafaeljusto/crawler/_test/crawler.go:66 +0x289

Goroutine 26 (finished) created at:
  github.com/rafaeljusto/crawler.Crawl()
      github.com/rafaeljusto/crawler/_test/crawler.go:21 +0x2b7
  github.com/rafaeljusto/crawler.TestCrawlMustFollowLinks()
      /home/rafael/Projects/go/src/github.com/rafaeljusto/crawler/crawler_test.go:498 +0x1f71
  testing.tRunner()
      /usr/local/go/src/pkg/testing/testing.go:391 +0x10f
==================
==================
WARNING: DATA RACE
Write by goroutine 28:
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:116 +0x55d
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.crawlPage()
      github.com/rafaeljusto/crawler/_test/crawler.go:66 +0x289

Previous write by goroutine 26:
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:116 +0x55d
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.crawlPage()
      github.com/rafaeljusto/crawler/_test/crawler.go:66 +0x289

Goroutine 28 (running) created at:
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:106 +0xbf5
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.crawlPage()
      github.com/rafaeljusto/crawler/_test/crawler.go:66 +0x289

Goroutine 26 (finished) created at:
  github.com/rafaeljusto/crawler.Crawl()
      github.com/rafaeljusto/crawler/_test/crawler.go:21 +0x2b7
  github.com/rafaeljusto/crawler.TestCrawlMustFollowLinks()
      /home/rafael/Projects/go/src/github.com/rafaeljusto/crawler/crawler_test.go:498 +0x1f71
  testing.tRunner()
      /usr/local/go/src/pkg/testing/testing.go:391 +0x10f
==================
==================
WARNING: DATA RACE
Write by goroutine 28:
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:122 +0x885
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.crawlPage()
      github.com/rafaeljusto/crawler/_test/crawler.go:66 +0x289

Previous write by goroutine 26:
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:122 +0x885
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.crawlPage()
      github.com/rafaeljusto/crawler/_test/crawler.go:66 +0x289

Goroutine 28 (running) created at:
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:106 +0xbf5
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.crawlPage()
      github.com/rafaeljusto/crawler/_test/crawler.go:66 +0x289

Goroutine 26 (finished) created at:
  github.com/rafaeljusto/crawler.Crawl()
      github.com/rafaeljusto/crawler/_test/crawler.go:21 +0x2b7
  github.com/rafaeljusto/crawler.TestCrawlMustFollowLinks()
      /home/rafael/Projects/go/src/github.com/rafaeljusto/crawler/crawler_test.go:498 +0x1f71
  testing.tRunner()
      /usr/local/go/src/pkg/testing/testing.go:391 +0x10f
==================
==================
WARNING: DATA RACE
Write by goroutine 28:
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:129 +0x923
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.crawlPage()
      github.com/rafaeljusto/crawler/_test/crawler.go:66 +0x289

Previous write by goroutine 26:
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:129 +0x923
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.crawlPage()
      github.com/rafaeljusto/crawler/_test/crawler.go:66 +0x289

Goroutine 28 (running) created at:
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:106 +0xbf5
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.crawlPage()
      github.com/rafaeljusto/crawler/_test/crawler.go:66 +0x289

Goroutine 26 (finished) created at:
  github.com/rafaeljusto/crawler.Crawl()
      github.com/rafaeljusto/crawler/_test/crawler.go:21 +0x2b7
  github.com/rafaeljusto/crawler.TestCrawlMustFollowLinks()
      /home/rafael/Projects/go/src/github.com/rafaeljusto/crawler/crawler_test.go:498 +0x1f71
  testing.tRunner()
      /usr/local/go/src/pkg/testing/testing.go:391 +0x10f
==================
==================
WARNING: DATA RACE
Write by goroutine 28:
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:135 +0x9c9
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.crawlPage()
      github.com/rafaeljusto/crawler/_test/crawler.go:66 +0x289

Previous write by goroutine 26:
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:135 +0x9c9
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.crawlPage()
      github.com/rafaeljusto/crawler/_test/crawler.go:66 +0x289

Goroutine 28 (running) created at:
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:106 +0xbf5
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.crawlPage()
      github.com/rafaeljusto/crawler/_test/crawler.go:66 +0x289

Goroutine 26 (finished) created at:
  github.com/rafaeljusto/crawler.Crawl()
      github.com/rafaeljusto/crawler/_test/crawler.go:21 +0x2b7
  github.com/rafaeljusto/crawler.TestCrawlMustFollowLinks()
      /home/rafael/Projects/go/src/github.com/rafaeljusto/crawler/crawler_test.go:498 +0x1f71
  testing.tRunner()
      /usr/local/go/src/pkg/testing/testing.go:391 +0x10f
==================
==================
WARNING: DATA RACE
Write by goroutine 28:
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:138 +0x5f1
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.crawlPage()
      github.com/rafaeljusto/crawler/_test/crawler.go:66 +0x289

Previous write by goroutine 26:
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:138 +0x5f1
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.crawlPage()
      github.com/rafaeljusto/crawler/_test/crawler.go:66 +0x289

Goroutine 28 (running) created at:
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:106 +0xbf5
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.crawlPage()
      github.com/rafaeljusto/crawler/_test/crawler.go:66 +0x289

Goroutine 26 (finished) created at:
  github.com/rafaeljusto/crawler.Crawl()
      github.com/rafaeljusto/crawler/_test/crawler.go:21 +0x2b7
  github.com/rafaeljusto/crawler.TestCrawlMustFollowLinks()
      /home/rafael/Projects/go/src/github.com/rafaeljusto/crawler/crawler_test.go:498 +0x1f71
  testing.tRunner()
      /usr/local/go/src/pkg/testing/testing.go:391 +0x10f
==================
==================
WARNING: DATA RACE
Write by goroutine 28:
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:144 +0x65c
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.crawlPage()
      github.com/rafaeljusto/crawler/_test/crawler.go:66 +0x289

Previous write by goroutine 26:
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:144 +0x65c
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.crawlPage()
      github.com/rafaeljusto/crawler/_test/crawler.go:66 +0x289

Goroutine 28 (running) created at:
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:106 +0xbf5
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.parseHTML()
      github.com/rafaeljusto/crawler/_test/crawler.go:173 +0x839
  github.com/rafaeljusto/crawler.crawlPage()
      github.com/rafaeljusto/crawler/_test/crawler.go:66 +0x289

Goroutine 26 (finished) created at:
  github.com/rafaeljusto/crawler.Crawl()
      github.com/rafaeljusto/crawler/_test/crawler.go:21 +0x2b7
  github.com/rafaeljusto/crawler.TestCrawlMustFollowLinks()
      /home/rafael/Projects/go/src/github.com/rafaeljusto/crawler/crawler_test.go:498 +0x1f71
  testing.tRunner()
      /usr/local/go/src/pkg/testing/testing.go:391 +0x10f
==================
PASS
coverage: 97.3% of statements
Found 16 data race(s)
exit status 66
FAIL    github.com/rafaeljusto/crawler  1.069s

Too many go routines

The strategy of creating a go routine for each new link must have a limitation, when running the command line program for "http://cnn.com" we get the message:

==13331==ThreadSanitizer: Thread limit (8192 threads) exceeded. Dying.

That's because we have too many active go routines at once.

On-the-fly statistics

For better user interaction we should show on-the-fly the scan progress of the crawler in a domain.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    馃枛 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 馃搳馃搱馃帀

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google 鉂わ笍 Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.