Get a list of robots.txt blocked URL and tell Google to crawl and index it.
-
Some of my key pages got blocked by robots.txt file and I have made required changes in robots.txt file but how can I get the blocked URL's list.
My webmaster page Health>blocked URL's shows only number not the blocked URL's.My first question is from where can I fetch these blocked URL's and how can I get them back in searches,
One other interesting point I see is that blocked pages are still showing up in searches.Title is appearing fine but Description shows blocked by robots.txt file.
I need urgent recommendation as I do not want to see drop in my traffic any more.
-
"changing the lastmod of those pages to today".
How can I make these changes?
Right now the news is that Resubmitted the Sitemap and no warnings this time.
-
I imagine that since you've got a robots txt error you'll probably ended closing a whole directory to bots which you wanted to be indexed. You can easily spot the directory and resubmit a sitemap to google changing the lastmod of those pages to today and the priority to 1 but only of those pages.
If you still receive warnings it may be due to errors in your sitemap. You're probably including some directory you don't want. You can try it in GWT putting in the box at the bottom the url you want to maintain in the index and then trying to see if some urls are being blocked by your robots.
If you want you can post here your robots and the URIs you want to be indexed without knowing the domain so that won't be public. Hope this may help you
-
Ok Resubmitted it.but even with updated file it gives a lot of errors.I think it takes some time.20,016 warnings
I have not added no index attribute in my header region.It was all messy stuff with robots.txt file.It means that with site showing up in SERP the rank will probably be the same or it was deranked?
-
Go into GWMT and resubmit sitemap.xml files (with the URLs you want indexed) for recrawling and Google will digest the sitemaps again, instead of waiting for Googlebot to come around on their own, you are requesting it to come around, also include those new sitemap files in your robots.txt file.
-
In Google Webmaster Tools, go to Health -> Fetch As Google. Then add the previously blocked URL and click Fetch. Once you've done that, refresh the page and click "Submit to index". That should get Google indexing those pages again.
Getting external links to your pages also helps get pages crawled & indexed, so it may be worth submitting your pages to social bookmarking sites, or get other types of backlinks to your previously blocked pages if possible.
-
Since you fixed your robots.txt file you should be good to. It will probably take a few days for Google to recrawl your site and update the index with the URLs they are now allow to crawl.
Blocked URLs can still show up in SERPs if you haven't defined the no-index attribute in your section.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
We have a client that wants to apply UTM URL tagging to track local organic traffic in Google Analytics. Is there any benefit in doing this?
One of our clients requested that we apply UTM URL tagging to better track organic traffic in Google Analytics. We found this to be an odd request because we are most familiar with UTM tracking for special campaigns (referral tracking, PPC, email tracking, etc). Is there any benefit of applying UTM tags to urls to analyze local organic traffic in Google Analytics? Are there any resources out there about this? Thanks!
Reporting & Analytics | | RosemaryB0 -
Google Webmaster indicates robots.text access error
Seems that Google has not been crawling due to an access issue with our robots.txt
Reporting & Analytics | | jmueller0823
Late 2013 we migrated to a new host, WPEngine, so things might have changed, however this issue appears to be recent. A quick test shows I can access the file. This is the Google Webmaster Tool message: http://www.growth trac dot com/: Googlebot can't access your site January 17, 2014 Over the last 24 hours, Googlebot encountered 62 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 8.8% Note the above message says 'over the last 24 hours', however the date is Jan-17 This is the response from our host:
Thanks for contacting WP Engine support! I looked into the suggestions listed below and it doesn't appear that these scenarios are the cause of the errors. I looked into the server logs and I was only able to find 200 server responses on the /robots.txt. Secondly I made sure that the server wasn't over loaded. The last suggestion doesn't apply to your setup on WP Engine. We do not have any leads as to why the errors occurred. If you have any other questions or concerns, please feel free to reach out to us. Google is crawling the site-- should I be concerned? If so, is there a way to remedy this? By the way, our robots file is very lean, only a few lines, not a big deal. Thanks!0 -
Google Analytics Tracking Code Problems
I am having a problem with my Google Analytics tracking codes. The relevant GA tracking codes have been entered into the source code of certain sites I'm running campaigns on, but Google Analytics is still telling me that the tracking code hasn't been installed - yet it has. What could possibly be the problem here? I have checked that these sites don't run off multiple domains, and that the tracking code has been entered in the correct place within the source code - and yet I am still having problems with GA. Any suggestions?
Reporting & Analytics | | WebhouseISP0 -
Bing vs Google Keyword Research
Hi All, I have carried out keyword research based on Google Analytics, Keyword Planner, competitors and various other toolsets.
Reporting & Analytics | | Mark_Ch
My research has been based on performance metrics favored towards Google.
I'm looking to research keyword traffic metrics and suggestions based on Bing Ads Intelligence. My Logic
I have compiled my master list based on Google metrics.
I will carry out Bing search traffic metrics based on my Google master list.
I will carry out Bing keyword suggestions based on my Google master list. My Questions
#1 Will my logic yield me any favorable benefits based on the high number of Google users vs Bing users
#2 Should Bing favor a particular keyword with high search traffic compared to Google, is there a fear that creating a 'seo user friendly website page' could dilute link juice based on the high Google users. Any other useful advice would be welcome. Regards Mark0 -
AHHH... Google Analytics just changed!
There's now an acquisitions tab.. but I can't figure out how to sort referral traffic by referral path.
Reporting & Analytics | | S.S.N0 -
Google Analytics Tracking Code Queries
Hello, I have taken on a new client who has Google Analytics installed. The tracking code is set to 'single domain'. Recently they added a mobile site using a sub-domain (m.website.com) which means that Google Analytics is not picking up this traffic. I want to revise the account so that I have a master account (raw data) and then profiles for the mobile site, main domain (www.website.com) and one other for a sub-domain that they are using. I am aware that there is mobile specific tracking code however I thought it would be easier (re conversions/goals/eCommerce tracking) to not use this and by changing the account to 'multiple domains' we could also get data for another sub-domain that they are using . My questions are: Am I right to want to use individual profiles over web properties. If not please explain why. When installing the tracking code (where the profile number is changing) I believe that I need to add that code with the changing profile number to the sub-domain sections. So my question is a) is that correct, and b) if I use a profile number on a sub-domain section will the master account still gather the data for the main URL as well as all sub-domains. If I change the master account from using 'single domain' tracking code to 'multiple domain' tracking code will this affect historical data? Will I lose the data? When changing from 'single domain' tracking to 'multiple domain' tracking does this affect eCommerce tracking? Or do we only need to be adding the additional lines of tracking code that allow sub-domains to be tracked? The web developers are using asynchronous code however half is in the and the other half is at the bottom of the source code. Given that traffic is being reported in the Google Analytics account should I have any concerns that the code is split? I have done a lot of reading but seem to be going around in circles, so your help is much appreciated! Thanks,
Reporting & Analytics | | Unity
Dinny0 -
Analytics Filter for URL's
Hi Fellow Mozzers I am setting my analytics and need to set some filters and need some help. I have a number of Local Sites i need to include and can't find how to do it. some of the the paths are local.imsm.com/new-york/ local.imsm.com/chicago/ local.imsm.com/long-beach/ local.imsm.com/atlanta/ each of the local URL's are /name/ any help would be great
Reporting & Analytics | | imsmlouis0 -
Setting up Google Analytics default URL
If someone has set: the default url in Google Analytics to a non-www address (http://mysite.com) then placed the UA tracking script from that GA account within the CMS framework of the website... ... and then set the permanent 301 redirect in the htaccess file to redirect to the www address (http://www.mysite.com). How less accurrate will my GA analytics measurements be considering the default url within GA is non-www and the permanent 301 redirect in htacess is to the www-address? Anyone know how reliable GA reports are until the default url in GA analytics is changed to match what is the redirected url in htaccess file? _Cindy
Reporting & Analytics | | CeCeBar0