Issue with Robots.txt file blocking meta description
-
Hi,
Can you please tell me why the following error is showing up in the serps for a website that was just re-launched 7 days ago with new pages (301 redirects are built in)?
A description for this result is not available because of this site's robots.txt – learn more.
Once we noticed it yesterday, we made some changed to the file and removed the amount of items in the disallow list.
Here is the current Robots.txt file:
# XML Sitemap & Google News Feeds version 4.2 - http://status301.net/wordpress-plugins/xml-sitemap-feed/ Sitemap: http://www.website.com/sitemap.xml Sitemap: http://www.website.com/sitemap-news.xml User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ Other notes... the site was developed in WordPress and uses that followign plugins:
- WooCommerce All-in-One SEO Pack
- Google Analytics for WordPress
- XML Sitemap
- Google News Feeds
Currently, in the SERPs, it keeps jumping back and forth between showing the meta description for the www domain and showing the error message (above).
Originally, WP Super Cache was installed and has since been deactivated, removed from WP-config.php and deleted permanently.
One other thing to note, we noticed yesterday that there was an old xml sitemap still on file, which we have since removed and resubmitted a new one via WMT. Also, the old pages are still showing up in the SERPs.
Could it just be that this will take time, to review the new sitemap and re-index the new site?
If so, what kind of timeframes are you seeing these days for the new pages to show up in SERPs? Days, weeks? Thanks, Erin ```
-
At the moment, it doesn't seem that rel=publisher is doing all that much for sites (aside from sometimes showing better info ion the knowledge graph listing on Brand searches) but personally I believe it's functionality and influence are going to be greatly expanded fairly soon, so well worth doing. As far as it contributing anything to help speed up indexing... doubt it.
P.
-
Paul,
Thanks... you hit upon my hunch, that we will just have to wait.
Much of the information in the SERPs (metadescriptions, titles and urls) are still old,even though they redirect to the new pages when I click.
Thanks for the tip... and about social media.
Do you think it will help to get the rel=publisher link to the Google+ page on the site?
Erin
-
A lot of people, especially WP users use modules that may block certain spiders crawling your site, but in your case, you don't seem to have any.
-
If you just changed the robots.txt file yesterday, my guess is you're going to have to be patient while the site gets recrawled, Erin. Any of the pages that are in the index and were cached before yesterday's robots update will still include the directive not to include the metadescription (since that's the condition they were under when they were cached.)
I suspect the pages you're seeing with metadescriptions were crawled since the robots update. Are you seeing the same page change whether it shows metadescription or not?
As far as old pages showing in the SERPs, again they'll all have to be crawled before the 301 redirects can be discovered and the SEs can begin to understand they should be dropped. (Even then it can take days to weeks for the originals to drop out.)
Another very effective way to help get the new site indexed faster is to attract some good-quality new links to the new pages. Social Media can be especially effective for this, Google+ in particular.
Paul
-
Thanks!
What do I need to look for in the .htaccess file?
Here is what is there... and the rest (not shown) are redirects:
BEGIN WordPress <ifmodule mod_rewrite.c="">RewriteEngine On RewriteBase / RewriteRule ^index.php$ - [L] RewriteCond %{REQUEST_FILENAME} !-f RewriteCond %{REQUEST_FILENAME} !-d RewriteRule . /index.php [L]</ifmodule> # END WordPress
BEGIN WordPress <ifmodule mod_rewrite.c="">RewriteEngine On RewriteBase / RewriteRule ^index.php$ - [L] RewriteCond %{REQUEST_FILENAME} !-f RewriteCond %{REQUEST_FILENAME} !-d RewriteRule . /index.php [L]</ifmodule> # END WordPress
-
Thanks for the tips! Let me check it out.
-
I'd also insure its not something to do with your .htacess file.
-
Make sure the pages aren't blocked with meta robots noindex tag
Fetch as Google in WMT to request a full site recrawl.
Run brokenlinkcheck.com and see if their crawler is successfully crawling or if it's blocked.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How Important is Meta Description on Non-Entry Pages?
If a page is never used as an entry page to your website -- in other words it's an obscure, relatively unimportant page that never ranks high enough in the search engines to be in the first few pages of the search results for any significant number of searches -- does editing the META Description really have any significant benefit? I guess the question could also be phrased as, does the Google Search Algo factor in the META Description tag, or is it only used for display purposes on the search results and doesn't affect ranking?
Intermediate & Advanced SEO | | SeoJaz0 -
What does Disallow: /french-wines/?* actually do - robots.txt
Hello Mozzers - Just wondering what this robots.txt instruction means: Disallow: /french-wines/?* Does it stop Googlebot crawling and indexing URLs in that "French Wines" folder - specifically the URLs that include a question mark? Would it stop the crawling of deeper folders - e.g. /french-wines/rhone-region/ that include a question mark in their URL? I think this has been done to block URLs containing query strings. Thanks, Luke
Intermediate & Advanced SEO | | McTaggart0 -
Meta refresh for news site?
We have a news site that uses a meta refresh (<meta < span="">http-equiv="refresh" content="600" /> across all content. I understand the reasoning on the homepage and am trying to decide of the cons of using this (slows page, is it treated differently and loses pagerank/link ..) Does anyone have experience with meta refresh being a negative thing or does it no longer matter?</meta <>
Intermediate & Advanced SEO | | KristieWahlquist0 -
Utf-8 symbols in the Title or Meta Description?
Has somebody any experience (pros or cons) to using utf-8 symbols in the Title or in the Meta Description tags?
Intermediate & Advanced SEO | | Yosef
Expedia uses it:
http://prntscr.com/74ofrv 74ofrv0 -
Faulty title, meta description and version (https instead of http) on homepage
Hi there, I am working on a client (http://minibusshuttle.com/) whose homepage is not indexed correctly by Google. In details, the title & meta description are taken from another website (http://planet55.co.uk/). In addition, homepage is indexed as https instead of http. The rest of the URIs are correctly indexed (titles, meta descriptions, http etc). planet55.co.uk used to be hosted on the same server as minibusshuttle.com and an SSL certificate was activated for that domain. I have tried several times to manually "fetch by Google" the homepage, to no avail. The rest of the pages are indexed/refreshed normally and Google responds very fast when I perform any kind of changes there. Any suggestions would be highly appreciated. Kind regards, George
Intermediate & Advanced SEO | | gpapatheodorou0 -
Meta refresh
What does this mean? That isn’t an address on the website (fdmgroup.com)?All I can think of is that there may be some email address incorrectly entered on the blog somewhere – but it’s not a meta-refreshLooking at the referring page http://www.fdmgroup.com/fdm-group-speaks-out-against-the-revelation-that-one-in-four-graduates-fail-to-find-work/ - a blog entry from 2011 – it seems someone’s tried to attach google tracking code to the email address?Thanks in advance.
Intermediate & Advanced SEO | | fdmgroup0 -
Missing Title Tags on Include Files?
GWT is telling me 3 of my include files (Contact Form - Header - Footer) are missing a Title Tag. This has never happened to me before and don't know how to tackle it. On the other hand the warning refers to a subdirectory of my site to these respective include files… The main directory, with literally the same html structure and no Title, returns no errors. Any ideas as to why this error now? or how to fix it? Thanks,
Intermediate & Advanced SEO | | dhidalgo10 -
Blocking HTTP 1.0?
One of my clients believes someone is trying to hack their site. We are seeing the requests with a server protocol or HTTP 1.0 so they want to block 1.0 entirely. Will this cause any problems with search engines or regular, non-spamming visitors?
Intermediate & Advanced SEO | | BryanPhelps-BigLeapWeb0