Blocked Production Site from Search Engines - How to get it Crawled by Moz Crawler
-
I have an 'under development' site hosted, (which is an exact replica of live site as working on to add new functionalities & modules) - but its password protected, excluded from robots.txt (Disallow) & also marked noindex on all pages in the index - so that Googlebot & other Search Engines can not crawl the site
At present the development work is almost 95% completed., Now - feel like to crawl the site through SEOMOZ Roger Bot - to know the errors and all indexed urls by Rogerbot.
What's the best way to get Moz Bot crawl the site - but simultaneously continue it blocking its access to Search Engines
I have gone through - https://support.google.com/webmasters/answer/93708?hl=en, it says
a) Save it in a password-protected directory. Googlebot and other spiders won't be able to access the content- But this way Moz will also not be able to crawl the site
b) Use a robots.txt to control access to files and directories on your server - However it also says - It's important to note that even if you use a robots.txt file to block spiders from crawling content on your site, Google could discover it in other ways and add it to our index.
c) Use a
noindex
meta tag to prevent content from appearing in our search results - It also says that a link to the page can still appear in their search results. Because we have to crawl your page in order to see the noindex tag, there's a small chance that Googlebot won't see and respect the noindex meta tagPassword Protected thus seems the best way to continue blocking. However, continuing with it will also block Moz bot to crawl the site. Any suggestions
Thanks
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Crawl Issue Question
Hey guys, I have run the crawl on my WordPress site and Moz finds a "Critical crawl issue" for my site on a broken link (404 error): mydomain.com/**%25s **, I can't seem to be able to find such a link anyway and I have run the website through several other tools that scan for broken links and such and there is no such result.
Moz Bar | | K.Net
This link doesn't exist on my site at all and I don't know where Moz got it from, I have made changes to my site and recrawled several times and the specific error persists. Does anyone have any ideas?0 -
Can Moz Keyword Explorer help target keywords for Google Images results?
I'm wondering if I can use Keyword Explorer (or maybe another tool?) to target keywords for image rankings. I'd like to play around with optimizing images so that they appear in search results and thus provide traffic - but wasn't sure the best way to track that kind of progress. My ultimate goal is to analyze the difficulty of ranking for a certain keyword via Google images. (I do know to optimize alt tag/title tag/place in relevant article etc, but wanted to know if I could research the difficulty). Any help is much appreciated. Thanks!
Moz Bar | | naturalsociety0 -
If we put the disavow links in google, does MOZ crawl the same links?
I have put bad or spam links in disavow file, but still showing in MOZ backlinks. So, I want to know that Why is MOZ not removing the spam links from their system?
Moz Bar | | insidewebanalytics0 -
Moz Keyword Tool Monthly Volume
Ive recently put together a Keyword List of about 100 keywords on the Moz Keyword Explorer tool. One keyword, aerial filming, stood out as very low search volume of 51 - 100. I took the same 100 keywords and passed them through the Google Keyword Planner by Google AdWords. Aerial Filming has an average search volume of 1k - 10k according to the Keyword Planner. Even though Keyword Planner gives me a range of 1k - 10k, the lowest number is still 10 times higher than what the Moz Keyword Explorer was indicating. This drastic difference of volume was consistent across all 100 keywords. All of the Monthly Volume numbers were divided by 10. Why does Moz Keyword Tool display a search volume that is 10x less than what Google Keyword Planner is suggesting?
Moz Bar | | fictionarts0 -
Has anyone had to deal with Moz crawl issues on their Zendesk support site?
If so - how did you end up resolving them? For instance we have 85 "temporary redirect" errors from our Zendesk support site in our crawl error report and we don't have access to the robots.txt file through Zendesk.
Moz Bar | | zspace0 -
Who and how does one get in Fresh Alerts?
Who and how does one get in Fresh Alerts? This is such a great tool! Thank, Moz! I would like to use this more often and to a better advantage. Can someone help me understand what criteria the tool uses to choose who it and what it picks up? Why would someone's personal family gathering turn up in my Moz Fresh Alerts("Minneapolis home buyers"? http://mydesultoryblog.com/2014/07/having-a-great-time-with-katelyn-and-drew-in-wayzata-mn/ My Desultory Blog Desultory thoughts on a variety of subjects … Having a great time with Katelyn and Drew in Wayzata, MN It seems completely random when and which of my blog posts show up in Moz Fresh Alerts. For example one that did ("Minneapolis real estate sellers"): "5 Critical Shifts in the Twin Cities Housing Market" http://www.homedestination.com/real-estate-blog/4-critical-shifts-in-the-twin-cities-housing-market Jeannie
Moz Bar | | jessential0 -
Moz Conversion Tracking
Is there a place here on Moz where goals and conversions can be set up or viewed? If the latter, does Moz just take the data from Analytics? Any information on this would be appreciated. Thanks,
Moz Bar | | xvpn9020 -
Confusing Moz Crawl?
Hi there, I am not sure if I am missing on something but the moz crawls are rather confusing. After singing in I have received 11 emails with crawls and today I have received again new, When I go to check there to the dashboard it shows 26 pages with issues. When I scroll down I see the pages with issue. Then when I click on the first page listed, to view the issues it says this: Rel Canonical
Moz Bar | | Rebeca1
Using rel=canonical suggests to search engines which URL should be seen as canonical. For this site: http://villasdiani.com/ but we have sorted out the canonical issues a long time ago. Is this a wrong information or is it really true that we do not specify the canonical for our site? Then the second page with issue is there listed http://villasdiani.com/beach-villas/ and it says: Duplicate Page Title
You should use unique titles for your different pages to ensure that they describe each page uniquely and don't compete with each other for keyword relevance. But it does not point out which page is duplicate with this one! I do not have any other page named the same way. It also says in Issues overview 26pages with issues, but it shows on the bottom only 5 under and when I click on view more it brings me to high priority issues where is 0. The most is freaking me out this report: When I click on links, there are listed on the bottom the pages with highest authority among which I found this http://villasdiani.com/db I have never created this kind of page! Funny enough when I click on it it really open that page! How this can be??? In issues overview it also shows on the bottom, right corner 11 page with duplicate content but when I click on it to review it it brings me to high priority issues windows where is not displayed anything Can somebody advice me regarding of this. I have sign up here to learn and sort out the problems with the site but so far I am only getting more confused here. Thank you very much for looking into this.0