Robot.txt file issue on wordpress site.
-
I m facing the issue with robot.txt file on my blog.
Two weeks ago i done some development work on my blog. I just added few pages in robot file. Now my complete site seems to be blocked. I have checked and update the file and still having issue.
The search result shows that "A description for this result is not available because of this site's robots.txt – learn more."
Any suggestion to over come with this issue
-
What the problem is with your site I cant say because we don't have the url.
But having said that, Why block pages? I find it is very rare that you need to block pages, I remember reading that Google sees blocked resources as a spam signal, that by itself is not a problem but when mixed with other signals can be harmful.
so do you really need to block the pages?
if you really do, then use meta noindex, follow.
you want the links to be followed so that link juice flows back out of the links to indexed pages. or you will be pouring link juice away via any links pointing to noindex pages -
Hi
You are most likely still seeing "A description for this result...." etc in Google because they may not have re-cached the page yet.
If the site is crawlable with a tool like Screaming Frog SEO Spider, or a header checker like http://urivalet.com/ - it's also accessible to Google, you just need to wait for them to re-crawl.
Let us know if you need more help!
-
The most likely lines in your robots.txt to cause this:
User-agent: * Disallow: /
-
Hi there
Check out your robots.txt in Wordpress:
Admin Dashboard>Settings>General>Privacy -- what is setting?
I would also read up on Robots.txt and see if you can find anything that may stand out to you. Then, I would take a look at your XML sitemap and upload it to Google Webmaster Tools and Bing Webmaster Tools.
I, like the others, would like to see your robots.txt file as well to see a live example. This shouldn't be anything major!
-
It's so easy to block your entire site from Google with robots.txt. Are you using Yoast SEO as the SEO plugin? With this, there shouldn't really be any need to block anything yourself.
Drop your URL here and we can take a look.
-Andy
-
Could you post the URL to your robots.txt file or post the content of the file itself?
-
If you wish to block your pages from search engines, then you should use noindex and not disallow in robots.txt. In many cases, a robots,txt is doing more harm than good.
What is your URL?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Do Google Know What People Are Doing On My Site?
I run an ecommerce site and am keep to increase the time on site and reduce bounce rate. I have had some good results from writing good articles and advertising them on Outbrain and Twitter; both of these metrics have improved. However, I am wondering if Google will see this. Clearly if people land on the pages from the SERPS and return there they will know what is happening but will they know if people get there from an advert?
On-Page Optimization | | T0BY0 -
Linking to External Site In Nav Bar
Hi, we are a celebrity site but also own a separate sports site with its own URL. We have a link to that site in our Nav bar. Are we being penalized by having that link? thanks
On-Page Optimization | | Uinterview0 -
Problem with getting a site to rank at all
We pushed this Word Press site live about a month ago www.primedraftarchitecture.com. Since then we've been adding regular content, blog posts 3 times a week with social posts on facebook, twitter, G+ and LinkedIn. We also submitted via Moz Local about 3 weeks ago. Yext about two weeks ago and have been adding about 5 listings to small local directories a week. Webmaster tools shows that the site map is valid and the pages of the site are getting indexed and it shows links from 7 sites, mostly directories. I'm just not seeing the site ranking for anything. We're getting zero organic traffic. I though we did a good job not over optimizing the pages. I'm just stymied trying to figure out what's wrong. Usually we push a site live and see at least some low rankings after just a couple of weeks. Can anyone see anything that looks bad or where we've gone wrong?
On-Page Optimization | | DonaldS0 -
Wordpress category url problem.
I have set up wordpress categories but the permalinks are showing as www.mydomain.com/?cat=12 as opposed to the category name. The child categories though work fine and show as www.mydomain.com/category/chidcatgegory I've obviously got my permalink settings wrong somewhere. How do I fix this?
On-Page Optimization | | SamCUK0 -
Photon for Wordpress experience
Has anyone tried switching on the Photon for Wordpress CDN/image caching feature? If so, did you experience any problems and have you noticed any improvements?
On-Page Optimization | | pugh0 -
Noindex tags in WordPress
I'm receiving 4 duplicate content warnings from Roger.They are all "www.mysite/tag/tag-name". I'm using WordPress and have Yoast WordPress SEO installed. Should I set the tags to "noindex" in the plugin settings?
On-Page Optimization | | brandzz0 -
Directory site with an URL structure dilemma
Hello, We run a site, which lists local businesses and tag them by their nature of business (similar to Yelp). Our problem is, that our category and sub-category(i.e.: www.example.com/budapest/restaurant or www.example.com/budapest/cars/spare-parts) pages are extremely weak, and get almost no traffic, but most of the traffic (95+ percent) goes for the actual business pages. While this might be a completely normal thing, I still would like to strengthen our category (listing) pages as well, as these should be the ones targeted by some of general keywords, like ‘restaurant’ or ‘restaurant+budapest’. One of the issues I have identified as a possible problem, that we do not have a clear hierarchy within the site, so while the main category pages are linked from the homepage (and the sub-categories from here), there is no bottom-up linking from the business pages back to the category pages, as the business page URLs look like this: www.example.com/business/onyx-restaurant-budapest. I think, that the good site- and url structure for the above would be like this: www.example.com/budapest/restaurant/hungarian/onyx-restaurant. My only issue is, perhaps not with the restaurants but with others, that some of the businesses have multiple tags, so they can be tagged i.e. as car saloon, auto repair and spare parts at the same time. Sometimes, they even have 5+ tags on them. My idea is, that I will try to identify a primary tag for all the businesses (we maintain 99 percent of them right now), and the rest of their tags would be secondary ones. I would then use canonicalization and mark the page with the primary tag in the url as the preferred one for that specific content. With this scenario, I might have several URLs with the same content (complete duplicates), but they would point to one page only as the preferred one, while our visitors could still reach the businesses in any preferred ways, so either by looking for car saloons, auto-repair or spare parts. This way, we could also have breadcrumbs on all the pages, which now we miss completely. Can this be a feasible scenario? Might it have a side-effect? Any hints on how to do it a better way? Many thanks, Andras
On-Page Optimization | | Dilbak0 -
Site Speed Expert Referral Needed
We're looking for someone who can diagnose some site speed issues AND adjust the code, servers, etc. as needed. Does anyone know of an expert in this area?
On-Page Optimization | | BryanPhelps-BigLeapWeb0