What is the appropriate Robot.txt to unblock if Google cannot get all the resources from my homepage?
-
Hello everyone. I did some research as to why my website has decreased in the Google search rankings recently. After reading this Yoast article I believe it's because the robot.txt files I have set up on my wordpress website. The following is a screen shot of the results of a fetch & render query of my webpage.Googlebot couldn't get all resources for this page. Here's a list:
My current robot.txt settings are as follows;
User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: */xmlrpc.php Disallow: */wp-*.php Disallow: */trackback/ Disallow: *?wptheme= Disallow: *?comments= Disallow: *?replytocom Disallow: */comment-page- Disallow: *?s= Disallow: */wp-content/ Allow: */wp-content/uploads/ ```What to I need to allow/disallow to allow Google spiders to properly read my website?
-
Remove the following line:
Disallow: */wp-content/ And add: Allow: /wp-includes/js/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Indexed, though blocked by robots.txt: Need to bother?
Hi, We have intentionally blocked some of the website files which were indexed for years. Now we receive a message "Indexed, though blocked by robots.txt" in GSC. We can ignore as per my knowledge? Are any actions required about this? We thought of blocking them with meta tags but these are PDF files. Thanks
Algorithm Updates | | vtmoz1 -
January 10, 2017 - Intrusive interstitials Google Update
Hi all, As everyone is most likely aware, Google have recently announced that if a site has intrusive intersitals that push the main content below the fold, will be downgraded in the SERP's from January 10th. At the moment we have a range of international sites, .ca, .com.au, .co.uk, .fr etc - if a user from a UK IP goes to a .ca site - a country switcher dialog will appear. I am aware that this may affect our sites performance in mobile search when the update comes out - however, if we block Google from seeing this - will they still pick it up? Thanks.
Algorithm Updates | | Brett-S0 -
Google Penguin update
When Google Penguin update will run again. The last time was in October 2013 and I'm still really curious now. Or have they stopped this and this is now continuously just like the panda?
Algorithm Updates | | NECAnGeL0 -
Is Moz Domain Authority still relvant when it comes to Google ranking?
My understanding of Moz DA is that it is predominantly based on external links. Since Penguin I am noticing more and more websites ranking high in Google with a "low" number of links and certainly a low DA but quality and relevancy of content and also of offering. I understand that there was always more to ranking than DA but is it anymore even relevant to how a site will rank in Google?
Algorithm Updates | | halloranc0 -
Next big Google update. Andy ideas when?
I'm interested to find out if anyone has and idea of when it will be or good guesses I would be interested to hear your thoughts. I have put a big effort into an SEO upgrade on my site and I'm interested in what impact I can expect when the next big update comes. Assuming I have fixed the issues that had been causing my traffic drop.
Algorithm Updates | | mark_baird0 -
How to get global search results on Google ? Also, is it possible to get results based on some other geographic location?
I don't want results based on my geographic location. When I am in India, I don't want local search results. In fact, I want results which are not dependent on my current location. Also, can I change my current location to some other city and will it affect the results ? For eg: While I am in London, can my search results be modified as if I am sitting in New York ?
Algorithm Updates | | EricMoore0 -
New Google "Knowledge Graph"
So according to CNN an hour ago regarding new Google update: "With Knowledge Graph, which will begin rolling out to some users immediately, results will be arranged according to categories with which the search term has been associated" http://www.cnn.com/2012/05/16/tech/web/google-search-knowledge-graph/index.html?hpt=hp_t3 Does this mean we need to start optimizing for Categories as well as Keywords?
Algorithm Updates | | JFritton0 -
Google showing different pages for same search term in uk and usa
Hi Guys, I have an interesting question and think Google is being a bit strange.. Can anyone tell me why when I input the term design agency in Google.co.uk it shows one page, but when i tyupe in the same search term in Google.com (worldwide search) it shows another page.. Any ideas guys? Is this not bit strange?? Any help here be much appreciated.. Thanks Gareth
Algorithm Updates | | GAZ090