New in AdSense: Site Crawler Diagnostics
There's a new tab in AdSense: Site Diagnostics. The page includes information about Google crawler's errors when accessing a page to determine its content. Using the site diagnostic section, you can see the errors encountered by Google, when they happened, and what you can do to troubleshoot them in order to allow Google's bot crawl the specific pages. Some of the problems are: robots.txt files that forbid access to web spiders, servers down, HTML errors.
Related:
Google Sitemaps show Google's penalties for your site
Hacking Google
GoogleBot can destroy sites