Get a list of robots.txt blocked URL and tell Google to crawl and index it.
-
Some of my key pages got blocked by robots.txt file and I have made required changes in robots.txt file but how can I get the blocked URL's list.
My webmaster page Health>blocked URL's shows only number not the blocked URL's.My first question is from where can I fetch these blocked URL's and how can I get them back in searches,
One other interesting point I see is that blocked pages are still showing up in searches.Title is appearing fine but Description shows blocked by robots.txt file.
I need urgent recommendation as I do not want to see drop in my traffic any more.
-
"changing the lastmod of those pages to today".
How can I make these changes?
Right now the news is that Resubmitted the Sitemap and no warnings this time.
-
I imagine that since you've got a robots txt error you'll probably ended closing a whole directory to bots which you wanted to be indexed. You can easily spot the directory and resubmit a sitemap to google changing the lastmod of those pages to today and the priority to 1 but only of those pages.
If you still receive warnings it may be due to errors in your sitemap. You're probably including some directory you don't want. You can try it in GWT putting in the box at the bottom the url you want to maintain in the index and then trying to see if some urls are being blocked by your robots.
If you want you can post here your robots and the URIs you want to be indexed without knowing the domain so that won't be public. Hope this may help you
-
Ok Resubmitted it.but even with updated file it gives a lot of errors.I think it takes some time.20,016 warnings
I have not added no index attribute in my header region.It was all messy stuff with robots.txt file.It means that with site showing up in SERP the rank will probably be the same or it was deranked?
-
Go into GWMT and resubmit sitemap.xml files (with the URLs you want indexed) for recrawling and Google will digest the sitemaps again, instead of waiting for Googlebot to come around on their own, you are requesting it to come around, also include those new sitemap files in your robots.txt file.
-
In Google Webmaster Tools, go to Health -> Fetch As Google. Then add the previously blocked URL and click Fetch. Once you've done that, refresh the page and click "Submit to index". That should get Google indexing those pages again.
Getting external links to your pages also helps get pages crawled & indexed, so it may be worth submitting your pages to social bookmarking sites, or get other types of backlinks to your previously blocked pages if possible.
-
Since you fixed your robots.txt file you should be good to. It will probably take a few days for Google to recrawl your site and update the index with the URLs they are now allow to crawl.
Blocked URLs can still show up in SERPs if you haven't defined the no-index attribute in your section.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Techy - Google Analytics, Google & Code Expert
We are trying to find someone to join our company but are having a very difficult time. Does anyone have any suggestions about where to find someone like this?
Reporting & Analytics | | ColoradoMarketingTeam0 -
Google Analytics, Wordpress, Sumo - who to trust
Something is wrong. I have a website that according to Google Analytics gets zero relevant clicks, with most of the clicks it does get coming from Russia. Wordpress says it gets a hundred or so visitors a month, with most traffic going to the pages that I know rank. Sumo says the heat map got 76 clicks. How would you figure out what's really going on?
Reporting & Analytics | | julie-getonthemap0 -
Uptick in not tracked conversions / anyone have a list of things that google analytics will not track
There seems to have been an uptick in users on our site not being tracked in Google Analytics cause I see a lot more un-tracked revenue in the last 6 months then I used to. I know analytics is still working as it has been tracking a normal amount of visits but I assumed there might be a reason less would be actually showing up in analytics (mabye a change is what is being reported as organic). I know a lot of stuff goes into "not provided" such as logged in search and stuff like that but is there a list of all of the ones that go into not provided and all that just do not get tracked (javascript not enabled, iOS?). If it could be something else as well let me know. Thanks for the help!
Reporting & Analytics | | Gordian0 -
Question about cannonical URLs for a site redesign
Hello folks, I've redesigned a site completely and I ended up changing their CMS to wordpress as well. So their URLs which mostly ended in .html and folder organization have been thrown completely out the window with wordpress' '/' format. I'm just wondering what the best way is to approach retaining all the site's previous "link juice". What should I be doing here? How do I make sure their organic rankings don't fall? (They've left their previous SEO firm so they can't help me out on this). Thanks!
Reporting & Analytics | | seonubblet0 -
Google Analytics (not provided)
Hi A greater and greater % of clicks is showing as "not provided" - does any one know of any good external tracking software which will show this information? Thanks
Reporting & Analytics | | jj34340 -
Google Webmaster Tools not displaying all backlinks
I was asked to take a look at a site that a friend redesigned and immediately plummeted in rankings. It was a flash site where the domain redirected to a flash landing page. When the site was redesigned, they removed the flash page and put in the new site on the domain URL without changing the redirect or putting a 301 in place on the old flash landing page URL. This site didn't have many backlinks to begin with, but it did ok for some important search phrases. Now when I look in Webmaster Tools, there are only 2 backlinks showing. Meanwhile I know there are more backlinks in existence, primarily from yellowpages and citysearch. I've had them put a 301 on the old flash landing page pointing at the domain URL and they've added a canonical URL. What have I missed here?
Reporting & Analytics | | BostonWright0 -
What to get from google Webmaster tools?
Hi everyone, I've been doing optimization for our websites and tracking the results regularly but don't really know what the results actually mean. I heard that I need to check the traffic and organic results from Google analytics and Google webmaster tools. Everybody says something and not sure what to do? Is there anyone clarify the SEO process for me - when to do what and why in simplest way? Thanks in advance,
Reporting & Analytics | | WTGEvents0 -
Disclaimer about using Google Analytic?
When using GA on my Website do I need to have a disclaimer that 'We are using Google Analtyic to Track information'?
Reporting & Analytics | | daracreative0