Robot.txt file issue on wordpress site.
-
I m facing the issue with robot.txt file on my blog.
Two weeks ago i done some development work on my blog. I just added few pages in robot file. Now my complete site seems to be blocked. I have checked and update the file and still having issue.
The search result shows that "A description for this result is not available because of this site's robots.txt – learn more."
Any suggestion to over come with this issue
-
What the problem is with your site I cant say because we don't have the url.
But having said that, Why block pages? I find it is very rare that you need to block pages, I remember reading that Google sees blocked resources as a spam signal, that by itself is not a problem but when mixed with other signals can be harmful.
so do you really need to block the pages?
if you really do, then use meta noindex, follow.
you want the links to be followed so that link juice flows back out of the links to indexed pages. or you will be pouring link juice away via any links pointing to noindex pages -
Hi
You are most likely still seeing "A description for this result...." etc in Google because they may not have re-cached the page yet.
If the site is crawlable with a tool like Screaming Frog SEO Spider, or a header checker like http://urivalet.com/ - it's also accessible to Google, you just need to wait for them to re-crawl.
Let us know if you need more help!
-
The most likely lines in your robots.txt to cause this:
User-agent: * Disallow: /
-
Hi there
Check out your robots.txt in Wordpress:
Admin Dashboard>Settings>General>Privacy -- what is setting?
I would also read up on Robots.txt and see if you can find anything that may stand out to you. Then, I would take a look at your XML sitemap and upload it to Google Webmaster Tools and Bing Webmaster Tools.
I, like the others, would like to see your robots.txt file as well to see a live example. This shouldn't be anything major!
-
It's so easy to block your entire site from Google with robots.txt. Are you using Yoast SEO as the SEO plugin? With this, there shouldn't really be any need to block anything yourself.
Drop your URL here and we can take a look.
-Andy
-
Could you post the URL to your robots.txt file or post the content of the file itself?
-
If you wish to block your pages from search engines, then you should use noindex and not disallow in robots.txt. In many cases, a robots,txt is doing more harm than good.
What is your URL?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Redirect issue
http://www.themorrisagency.co.uk/wedding-band redirects to http://www.themorrisagency.co.uk/wedding-band-cost-much-hire/ NO canonical in place. Nothing in Htaccess. I cant think where else to look to find it. If I amend it to http://www.themorrisagency.co.uk/we it still redirects. Any ideas? I have spent way too long on it now
On-Page Optimization | | agentmorris10 -
Redirect closing established site to another established site.
We have 2 ecommerce websites with duplicate product/category content that have been around for years. "Site A" since 2004 and "Site B" since 2011. "Site B" doesn't generate many sells and we want to close the site in fear that it may look spammy to google and since we don't want to upkeep the site. Couple questions:
On-Page Optimization | | Tuurbo
1. Is it possible that "Site B" has damaged "Site A" with google since they are so similar?
2. Can i 301 redirect all of "Site B" to "Site A" without hurting "Site A"?
3. If question 2 is ok, should i use the "Change of Address" tool in google webmaster tools to point "Site B" to "Site A"? Thanks!0 -
Site Cleanup Operation
Hi, I hope you can help, I have been asked to look at a friends site that is simply shocking, but somehow ranks for its main target keywords, mainly because they are easy. But going through the site he has like 350 links on his menus that are all follow so I need to change them to nofollow, but should anything else normally be marked such as no index etc on a menu link. Also upon doing a Moz scan there are something like 250 missing meta descriptions from old blog posts. When I looked closer they had been using the blog section for posting relevant news headlines, but thats it, So he has 250 useless, low quality blog posts. My question is, what should I really do with them, ie delete, redirect, canonical etc. Any help is much appreciated. Thanks Paul
On-Page Optimization | | propertyhunter0 -
Creating Duplicate Content on Shopping Sites
I have a client with an eCommerce site that is interested in adding their products to shopping sites. If we use the same information that is on the site currently, will we run into duplicate content issues when those same products & descriptions are published on shopping sites? Is it best practice to rewrite the product title and descriptions for shopping sites to avoid duplicate content issues?
On-Page Optimization | | mj7750 -
ECommerce Site Breadcrumbs Best Practice
I'm working on an Ecom website and I was wondering - For breadcrumbs - is there an SEO and/or UEx preference when it comes to taking them back to the homepage? I have the option of going CATEGORY > SUB CATEGORY > SUB CATEGORY or HOME > CATEGORY > SUBCATEGORY > SUBCATEGORY Each example is hyperlinked except for the lowest level. Thanks
On-Page Optimization | | Blenny0 -
Fixing Wordpress Title and Description Tag Placement
I know how important the title and description tag are and I know it is important to have them place at the top of the code. However, when I view the source on a Wordpress template, I see it looks messy. Here is an example of what the source looks like. Is there a plugin or an adjustment to be made so the code is cleaner? To get rid of spaces and blank lines in code. Or should I just not worry about the way the code looks? Notice the first example has a break after the <title>, then after the title there is this</p> <p style="color: #5e5e5e;"><strong><em><meta http-equiv="content-type" content="text/html; charset=UTF-8" /> </em></strong></p> <p style="color: #5e5e5e;"><strong><em> </em></strong>then the description? Can that be moved?</p> <p style="color: #5e5e5e;"> </p> <p style="color: #5e5e5e;"><span style="color: #5e5e5e;"><!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"></span><span style="color: #5e5e5e;"><br /></span><span style="color: #5e5e5e;"><html xmlns="http://www.w3.org/1999/xhtml" dir="ltr" lang="en-US"></span><span style="color: #5e5e5e;"><br /></span><span style="color: #5e5e5e;"><span> </span><head profile="http://gmpg.org/xfn/11"></span><span style="color: #5e5e5e;"><br /></span><span style="color: #5e5e5e;"><span> </span><title></span><span style="color: #5e5e5e;"><br /></span><span style="color: #5e5e5e;"><span> </span>tite text goes here <span> </span></title> " /> Here there are no breaks after <title>and the <title> is directly above the description</p> <p style="color: #5e5e5e;"> </p> <p style="color: #5e5e5e;"><span style="color: #5e5e5e;"><!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"></span></p> <p style="color: #5e5e5e;"><span style="color: #5e5e5e;"> </span><span style="color: #5e5e5e;"><html xmlns="http://www.w3.org/1999/xhtml" dir="ltr" lang="en-US"></span></p> <p style="color: #5e5e5e;"><span style="color: #5e5e5e;"><head profile="http://gmpg.org/xfn/11"></span></p> <p style="color: #5e5e5e;"><span style="color: #5e5e5e;"><meta http-equiv="content-type" content="text/html; charset=UTF-8" /></span></p> <p style="color: #5e5e5e;"><span style="color: #5e5e5e;"><title>tite text goes here</title>
On-Page Optimization | | hfranz0 -
Content for ecommerce site
How important on site/page contents are for ecommerce site. Keeping in mind the page layout. Its not that important to have page copy/content at all for ecommerce sites If yes, does position of content is an important factor? if putting page copy/content in upper fold of a page then the most important thing which is product itself will have less exposure if putting near the footer of the page, does that seem like doing just for the sake of SEs and ranking. How important internal linking form that content would be compare to left panel links or links at the header of a website Thanks Rick
On-Page Optimization | | RickGa0 -
How do we handle sitemaps in robots.txt when multiple domains point to same physical location?
we have www.mysite.net, www.mysite.se, www.mysite.fi and so on. all of these domains point to the same physical location on our webserver, and we replace texts given back to client depending on which domain he/she requested. My problem is this: How do i configure sitemaps in robots.txt when robots.txt is used by multiple domains? If I for instance put the rows Sitemap: http://www.mysite.net/sitemapNet.xml
On-Page Optimization | | nordicnetproducts
Sitemap: http://www.mysite.net/sitemapSe.xml in robots.txt, would that result in some cross submission error?0