Robot.txt file issue on wordpress site.
-
I m facing the issue with robot.txt file on my blog.
Two weeks ago i done some development work on my blog. I just added few pages in robot file. Now my complete site seems to be blocked. I have checked and update the file and still having issue.
The search result shows that "A description for this result is not available because of this site's robots.txt – learn more."
Any suggestion to over come with this issue
-
What the problem is with your site I cant say because we don't have the url.
But having said that, Why block pages? I find it is very rare that you need to block pages, I remember reading that Google sees blocked resources as a spam signal, that by itself is not a problem but when mixed with other signals can be harmful.
so do you really need to block the pages?
if you really do, then use meta noindex, follow.
you want the links to be followed so that link juice flows back out of the links to indexed pages. or you will be pouring link juice away via any links pointing to noindex pages -
Hi
You are most likely still seeing "A description for this result...." etc in Google because they may not have re-cached the page yet.
If the site is crawlable with a tool like Screaming Frog SEO Spider, or a header checker like http://urivalet.com/ - it's also accessible to Google, you just need to wait for them to re-crawl.
Let us know if you need more help!
-
The most likely lines in your robots.txt to cause this:
User-agent: * Disallow: /
-
Hi there
Check out your robots.txt in Wordpress:
Admin Dashboard>Settings>General>Privacy -- what is setting?
I would also read up on Robots.txt and see if you can find anything that may stand out to you. Then, I would take a look at your XML sitemap and upload it to Google Webmaster Tools and Bing Webmaster Tools.
I, like the others, would like to see your robots.txt file as well to see a live example. This shouldn't be anything major!
-
It's so easy to block your entire site from Google with robots.txt. Are you using Yoast SEO as the SEO plugin? With this, there shouldn't really be any need to block anything yourself.
Drop your URL here and we can take a look.
-Andy
-
Could you post the URL to your robots.txt file or post the content of the file itself?
-
If you wish to block your pages from search engines, then you should use noindex and not disallow in robots.txt. In many cases, a robots,txt is doing more harm than good.
What is your URL?
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Missing Meta description issue reported
Moz says missing meta description, however the meta description exists properly (using yoast plugin) eg of link reported by moz for missing meta description : https://www.basezap.com/reinstall-os-in-vps-using-solusvm-panel/ It is giving same error for many blog posts, as same time, for some articles the issue isn't reported. Although the method for meta description is same for all the posts. Can anyone explain if its a common bug in moz site analysis or I am following something wrong ?
On-Page Optimization | | basezap0 -
Consolidating a Large Site with Duplicate Content
I will be restructuring a large website for an OEM. They provide products & services for multiple industries, and the product/service offering is identical across all industries. I was looking at the site structure and ran a crawl test, and learned they have a LOT of duplicate content out there because of the way they set up their website. They have a page in the navigation for “solution”, aka what industry you are in. Once that is selected, you are taken to a landing page, and from there, given many options to explore products, read blogs, learn about the business, and contact them. The main navigation is removed. The URL structure is set up with folders, so no matter what you select after you go to your industry, the URL will be “domain.com/industry/next-page”. The product offerings, blogs available, and contact us pages do not vary by industry, so the content that can be found on “domain.com/industry-1/product-1” is identical to the content found on “domain.com/industry-2/product-1” and so-on and so-forth. This is a large site with a fair amount of traffic because it’s a pretty substantial OEM. Most of their content, however, is competing with itself because most of the pages on their website have duplicate content. I won’t begin my work until I can dive in to their GA and have more in-depth conversations with them about what kind of activity they’re tracking and why they set up the website this way. However, I don’t know how strategic they were in this set up and I don’t think they were aware that they had duplicate content. My first thought would be to work towards consolidating the way their site is set up, so we don’t spread the link-equity of “product-1” content, and direct all industries to one page, and track conversion paths a different way. However, I’ve never dealt with a site structure of this magnitude and don’t want to risk messing up their domain authority, missing redirect or URL mapping opportunities, or ruin the fact that their site is still performing well, even though multiple pages have the same content (most of which have high page authority and search visibility). I was curious if anyone has dealt with this before and if they have any recommendations for tackling something like this?
On-Page Optimization | | cassy_rich0 -
How to analysis Responsive VS Dynamic site?
Hello Experts, Can you please throw your knowledge on How to analysis Responsive VS Dynamic site? Also please suggest best tool to optimize both type of sites plus if I have to optimize manually then what is the best way? It will help lots of people as currently google focusing more on mobile site so we can optimize our responsive and dynamic site. Pls suggest. Thanks!
On-Page Optimization | | adamjack0 -
"irrelevant pages of a site"
Hi there! Some pages of my site like "contact" or "registration": Should they have a title and a description tag? They are pages that I don't want them to be shown in the SERPs....Could I be penalized by google If I don't do so? The SEOMOZ crawling tool warms me about this issue (to short titles, no meta-description tags....) Many thanks
On-Page Optimization | | juanmiguelcr0 -
Can we listed URL on Website sitemap page which are blocked by Robots.txt
Hi, I need your help here. I have a website, and few pages are created for country specific. (www.example.com/uk). I have blocked many country specific pages from Robots.txt file. It is advisable to listed those urls (blocked by robots.txt) on my website sitemap. (html sitemap page) I really appreciate your help. Thanks, Nilay
On-Page Optimization | | Internet-Marketing-Profs0 -
What precaution should we take to change the default page of the site
For some reason we wanted to change the default page of my site from example.com to example.com/default.aspx. We will be using 301 redirection to get the back link benefits. Do we need to make any changes in webmaster tool and sitemap too??
On-Page Optimization | | CyrilWilson0 -
SSL sites and SEO
I'm seeing a lot about how SSL sites can negatively affect SEO from the analytics perspective, but if duplicate content issues are addressed and load times are OK, what else is there to worry about? Any advice is much appreciated. Thank you!
On-Page Optimization | | PerfectPitchConcepts0 -
Canonical issue
Hi, Very new to seomoz but very impressed. First report has shown me that I have duplicate pages. Some seem to be duplicate titles and some were duplicates of pages i found on the server. however the main problem is it seems to be picking up pages with www and without it which I have a vague idea is a canonical issue. so it throws up pages like this: http://web-writer-articles.co.uk and http://www.web-writer-articles.co.uk I want it just to pick up pages with www Firstly should it be picking up both and if not how can I make amendments so that it is only picking up pages which include www ? thank you for your help, louandel15
On-Page Optimization | | louandel150