Robot.txt file issue on wordpress site.
-
I m facing the issue with robot.txt file on my blog.
Two weeks ago i done some development work on my blog. I just added few pages in robot file. Now my complete site seems to be blocked. I have checked and update the file and still having issue.
The search result shows that "A description for this result is not available because of this site's robots.txt – learn more."
Any suggestion to over come with this issue
-
What the problem is with your site I cant say because we don't have the url.
But having said that, Why block pages? I find it is very rare that you need to block pages, I remember reading that Google sees blocked resources as a spam signal, that by itself is not a problem but when mixed with other signals can be harmful.
so do you really need to block the pages?
if you really do, then use meta noindex, follow.
you want the links to be followed so that link juice flows back out of the links to indexed pages. or you will be pouring link juice away via any links pointing to noindex pages -
Hi
You are most likely still seeing "A description for this result...." etc in Google because they may not have re-cached the page yet.
If the site is crawlable with a tool like Screaming Frog SEO Spider, or a header checker like http://urivalet.com/ - it's also accessible to Google, you just need to wait for them to re-crawl.
Let us know if you need more help!
-
The most likely lines in your robots.txt to cause this:
User-agent: * Disallow: /
-
Hi there
Check out your robots.txt in Wordpress:
Admin Dashboard>Settings>General>Privacy -- what is setting?
I would also read up on Robots.txt and see if you can find anything that may stand out to you. Then, I would take a look at your XML sitemap and upload it to Google Webmaster Tools and Bing Webmaster Tools.
I, like the others, would like to see your robots.txt file as well to see a live example. This shouldn't be anything major!
-
It's so easy to block your entire site from Google with robots.txt. Are you using Yoast SEO as the SEO plugin? With this, there shouldn't really be any need to block anything yourself.
Drop your URL here and we can take a look.
-Andy
-
Could you post the URL to your robots.txt file or post the content of the file itself?
-
If you wish to block your pages from search engines, then you should use noindex and not disallow in robots.txt. In many cases, a robots,txt is doing more harm than good.
What is your URL?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Creating a .cn site with the existing site content
Hi all, I'm planning to create a .cn site. If I simply translate the existing content on my site (.com.au) into Chinese, do you think Google will see the .cn site as a duplicate of the main site? Will this cause any duplicate content issues? Thanks
On-Page Optimization | | QuantumWeb620 -
Linking to External Site In Nav Bar
Hi, we are a celebrity site but also own a separate sports site with its own URL. We have a link to that site in our Nav bar. Are we being penalized by having that link? thanks
On-Page Optimization | | Uinterview0 -
On Site Problem Caused Rankings To Drop?
I'm getting to the bottom of why my site dropped rankings on the 9th March. I noticed that in google there is a cached version of my site on 9th March at 8.37am which is when my rankings dissapeared. Presumably this is when google last crawled my site? I guess this means that google found something on the home page or on the site that it didn't like? I wonder if anyone can take a look and let me know if there's anything obvious. Could it be a duplicate content penalty as I have lots of categories pulling content from the same posts?
On-Page Optimization | | SamCUK0 -
Should I link my 3 E-Commerce sites?
Good Morning, I have 3 E-Commerce sites that all sell the same products, but have unique content on them (unique text, unique urls, same products).... Up until now, I have not put any links from one to any of the others... I just started to wonder about that since these are all related to the same industry, and are owned by 1 company, what would be the downside to linking them... Does anyone have any advice on if I should link to each site from the other 2 sites? Also, if you think I should be linking them, please advise how you would do it (on which pages, how many links, anchor texts, etc...) Thanks a lot!
On-Page Optimization | | Prime850 -
Site Architecture: How do I best Optimize for Similar Keywords?
Hello Moz Community! I'm really struggling trying to decide on an improved site architecture. I run an online proofreading & editing website. This leaves us targeting many different niche keywords. For example: blog editing/proofreading, essay editing/proofreading, book editing/proofreading, resume... you get the point. I feel like editing & proofreading are similar enough to target on the same page(s). However, the issue is that I'm also having to deal with what I'm calling derivative keywords. For example, when I try to optimize for 'essay editing/proofreading', I also have to think about: paper editing, paper editor, paper correction, edit my paper, etc. I would have no problem optimizing the page for 'essay editing' in the title, H1, etc. and then targeting these words as secondary keywords within the body text, etc., however, I keep thinking 'a large slice of a small pie is better than a small slice of a big one.' You see, the keyword 'essay correction' has only about one-third the monthly searches as 'essay editing', but it is 50% less competitive. The same is loosely true for the rest of the 'derivative' keywords. I'd have no problem building specific pages for these derivative keyword groups, however, I'm very concerned how this would effect my site from a user experience perspective. I don't want to have a master "services" page with links to book editing, resume editing, essay editing, etc. and then also show paper editing, essay correction, etc. To me, this would be confusing... "What's the difference between essay editing and paper editing?". Any guidance is much appreciated. This has got my head spinning! Thanks!
On-Page Optimization | | TBiz0 -
Home Page Optimization For Wordpress Blog
Hi, I'm developing a wordpress site and currently confused over home page optimization. I'm planning to use excerpts of posts on home page so that users can glance over all recent posts and at same time avoid content duplication. But I'm not sure how to optimize the home page for couple of targeted keywords . How can i make sure that there is enough content, proper H1, title tag etc on home page for target keywords even by using excerpts of different posts on home page with Read more links to complete posts? Should i use some sticky post on home page to take care of all onpage requirements for required keywords or there are some other better ways to do it? Regards, shaz
On-Page Optimization | | shaz_lhr0 -
Site Stucture Advice - Keyword Dillema
I am creating a new site and am looking for some advice on how to structure the site Using Google's keyword search tool it seems like I have a dilemma in that about 50% of the keyword pairs are contained in 10 keyword pairs that are similar The first two pairs have about 49% of the traffic and only differ between plural / singular, not quite sure how to handle that, or if google has a method to make these more or less synonomous The last 8 pairs are roughly similar in distribtuion As an example (not my case, just for visualization) Mountain Bike Classes Mountain Bike Instruction Mountain Bike Workshops Mountain Bike Training Etc ... which all more or less give the same results (yes some difference but they all deal with learning how to ride a mountain bike, again this is not my exact case, don't care a whit about mountain bikes 😉 I don't see giving each of those kinds of pairs their own page since the content would be pretty much the exact same, making it substantially different would also be problematic (if I am thinking about this correctly) I have a clean slate to work with from a site perspective so I am wondering how people here would, or better yet have handled similar situations
On-Page Optimization | | bThere0 -
On site optimization - anchor text in body text
Hi I am learning SEO from scratch and am only a couple of weeks in. My wife has just started a website selling children clothes. In the left bar there are links to the different producers. Clicking it takes you to the products from that producer. In the body text om the main page ( epleskrinet.no) the same producers are mentioned. By time i need to opitmize for all of these. Is there any point in making them anchor text - taking them to the same URL as is given in the side bar ? Or will it get discounted ? Dan Laerum
On-Page Optimization | | danlae0