A lot of time since this quesiton and the answer have been posted. Since then it has changed a lot. The typical recommendation about non -displayed crawlers wp-content/themes” wp-content/plugins” wp-content/cache” wp-includesAnd any other folder that contains CSS or JS files that are needed on the site is no longer valid.
For example, let’s talk about Google. Googlebot made websites without CSS and without JS, but actually not. Googlebot Fective The full document and checks things such as responsiveness, number, location and size of the scripts, etc. So Google does not like it if you do not allow Googlebot to gain access to CSS and JS files. That means you should not be allowed wp-content/themes” wp-content/plugins” wp-content/cache And wp-includes Because of all those folders, CSS and JS files can serve.
From my point of view, the best robots.txt file is standard made by WordPress (standard (The Robots.txt below is the standard since WP 4.0):
User-agent: *
Disallow: /wp-admin/
If you have a CGI-BIN folder, it can be a good idea not to allow the CGI-BIN folder:
User-agent: *
Disallow: /wp-admin/
Disallow: /cgi-bin/
And if you use a sitemap, it is a good idea to include a sitemap reference in robots.txt (you still have to submit the Sitemap manual to Google and Bing Webmaster Tools, but the reference can be useful for other crawlers):
User-agent: *
Disallow: /wp-admin/
Disallow: /cgi-bin/
Sitemap: http://example.com/sitemap.xml
That is in general. Specific websites may need other folders and files that must be studied in each specific case. For Exmaple, you may need it or do you perhaps not want a specific plug -in -fold:
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-content/plugins/plugin-folder/
Use to change the robots.txt, use robots_txt Filter (using a real robots.txt file ensures that WordPress is no longer able to process robots.txt). For example:
add_filter( 'robots_txt', function( $output ) {
$output .= "Disallow: /cgi-bin/\n";
$output .= "Disallow: /wp-content/plugins/plugin-folder-i-want-to-block/\n";
$output .= "\nSitemap: " . site_url( 'sitemap.xml' ) . "\n";
return $output;
});
#good #robots.txt