Should we add robots.txt? about lmms.io HOT 5 CLOSED

Received this email from Google today...

from lmms.io.

Sure, add a robots.txt? Is it that who makes google look like this?:

Or is that just autogenerated?

from lmms.io.

I don't know (and I haven't researched it enough to answer) but at quick glance the robots.txt seems to be a way to prevent search engines from crawling certain pages. This crawling can have good and bad side effects, such as google linking to our community page (such as community/#forum) rather than our real page (/forum) by telling it not to look at certain pages. It can also help prevent unnecessary bandwidth on heavy resource pages that we'd prefer to save for our users instead of for a search engine, but this is the first time I've actually researched it, so that's just a 60 second crash course on http://www.robotstxt.org/.

from lmms.io.

Yeah, I read for a few minutes on the linked site, and from what I learned, it is only to block certain sites, or add sites to a list for the robots to read (e.g. search term site).

from lmms.io.

Closed per 065ebdc.

from lmms.io.