Are you a SEO-conscioncious webmaster? Of course, you are!
And you already have a nice, standards-compliant XML sitemap! Congrats! (Maybe you used a Joomla component to generate it on the fly - Brilliant!) You even had submitted it to the Google! Great! And you think you can enjoy from now that well-deserved pina colada, right? Hold on, you have some important tasks to do!
Relatively recently the major search-engines agreed to extend the robots.txt's functionality. Until recently this little txt file in your root has served mostly to deny the search-engine robots to index files which aren't their business. And the file is up there in any recent Joomla install, and not without purpose, just remember the story when lot of Joomla webmasters where discovered their site backups, complete with admin passwords and other sensitive data readily indexed and publicized by Google. (Just not blame Google that don't index your site fully )
So, since mid May 2008 many SEO sites had reported, that all major webrobots are looking in the robots.txt file for the command: