How to stop crawls for product review pages? Volusion site
-
Hi guys, I have a new Volusion website. the template we are using has its own product review page for EVERY product i sell (1500+) When a customer purchases a product a week later they receive a link back to review the product. This link sends them to my site, but its own individual page strictly for reviewing the product. (As oppose to a page like amazon, where you review the product on the same page as the actual listing.)
**This is creating countless "duplicate content" and missing "title" errors. What is the most effective way to block a bot from crawling all these pages? Via robots txt.? a meta tag? **
Here's the catch, i do not have access to every individual review page, so i think it will need to be blocked by a robot txt file? What code will i need to implement? i need to do this on my admin side for the site? Do i also have to do something on the Google analytics side to tell google about the crawl block?
Note: the individual URLs for these pages end with: *****.com/ReviewNew.asp?ProductCode=458VB
Can i create a block for all url's that end with /ReviewNew.asp etc. etc.?
Thanks! Pardon my ignorance. Learning slowly, loving MOZ community
-
No you should be fine
-
thanks. you say "update on 4/21" you're talking about googles update requiring more mobile friendly sites? My volusion template has its own mobile version. It is not a responsive template. So i should not be affected correct?
-
Parameters are good for pages that are a result of a search or sort. I guess it isn't necessary really, I am just a little ocd about that kind of stuff. The parameters in WMT basically tell google that these things might appear in the URL, and then you can the bot to ignore it or let GoogleBot decide how to read the URL.
The mobile site is not the same as a responsive design and on of the main reasons I left Volusion. The mobile site will get you through the update on 4/21, but if possible you should ask them for a responsive site. Just call the support number, or your account manager and ask.
-
ive had the following in my robots.txt file. Do i need to add the astrisk like you have posted above?
Currently in my robot.txt:
User-agent:*
Disallow: /reviews.asp/User-agent:*
Disallow: /reviewnew.asp/ -
Thanks monica, can you elaborate a bit more on the webmaster tools parameter? what specifically does adding a parameter like that do? You did that as a backup in case the robot txt file was not working? we do have a mobile version enabled which came with our template. Ill keep an eye out for the 404's. where do i check for a responsive template? Ours is one of their premium templates so its possible we are already on a responsive one? Can you clarify what responsive template means?
thanks.
-
I did that in my Volusion store. I also added ReviewNew.asp?ProductCode= as a parameter in Google Webmaster Tools. Do you have an enabled mobile site as well? If you do there are several 404 errors that you will start to see from there. Make sure you are adding parameters accordingly. I am not sure if Volusion has started offering their responsive templates yet, but if they have I would see if you can implement that over the mobile site.
-
Hi ,
Yes you can block such URL by using below code in robots.txt file.
User-agent: *
Disallow: /*ReviewNew.aspThanks
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Hey guys, for some reason my homepage has gone down in rankings though other pages on my site have not.
This is not something I have ever seen before. The site is still indexed if I search for it directly, but not in top 100 rankings for keywords even though sub-pages are ranking for the given keyword. Changes I have made recently include site transfer to wordpress, force redirect http to https removal of www by redirect and adding new property instance in Google Search Console. I have checked htaccess file and sitemap and all seem fine. ideas? Site: https://dublinSEO.co
Technical SEO | | HappyApple840 -
On our site by mistake some wrong links were entered and google crawled them. We have fixed those links. But they still show up in Not Found Errors. Should we just mark them as fixed? Or what is the best way to deal with them?
Some parameter was not sent. So the link was read as : null/city, null/country instead cityname/city
Technical SEO | | Lybrate06060 -
Site Crawling with Firewall Plugin
Just wondering if anyone has any experience with the WordPress Simple Firewall plugin. I have a client who is concerned about security as they've had issues in that realm in the past and they've since installed this plugin: https://wordpress.org/support/view/plugin-reviews/wp-simple-firewall?filter=4 Problem is, even with a proper robots file and appropriate settings within the firewall, I still cannot crawl the site with site crawler tools. Google seems to be accessing the site fine, but I still wonder if it is in anyway potentially hindering search spiders.
Technical SEO | | BrandishJay0 -
How do we keep Google from treating us as if we are a recipe site rather than a product website?
We sell food products that, of course, can be used in recipes. As a convenience to our customer we have made a large database of recipes available. We have far more recipes than products. My concern is that Google may start viewing us as a recipe website rather than a food product website. My initial thought was to subdomain the recipes (recipe.domain.com) but that seems silly given that you aren't really leaving our website and the layout of the website doesn't change with the subdomain. Currently our URL structure is... domain.com/products/product-name.html domain.com/recipes/recipe-name.html We do rank well for our products in general searches but I want to be sure that our recipe setup isn't detrimental.
Technical SEO | | bearpaw0 -
Why is Google Webmaster Tools showing 404 Page Not Found Errors for web pages that don't have anything to do with my site?
I am currently working on a small site with approx 50 web pages. In the crawl error section in WMT Google has highlighted over 10,000 page not found errors for pages that have nothing to do with my site. Anyone come across this before?
Technical SEO | | Pete40 -
Will getting backlinks to landing page from low quality sites negatively affect SEO?
I've recently started an initiative at my company to get our customers to publish a blog post about our company and to include a link to a landing page which sits on a subdomain attached to our main domain. The reason for directing visitors to the post to a landing page is to help with conversion. I've recently been thinking that couldn't the backlinks to this landing page from our customers' blogs (generally small sites) have a negative impact on the overall SEO of my companies domain? Thanks in advance.
Technical SEO | | JustinButlion0 -
Does this page crawl well?
I just put up a page that uses an image map to illustrate a national currency note. http://www.antiquebanknotes.com/NationalCurrency/National-Bank-Note-Information.aspx My goal with this page is get results for National Bank Note. But I know image maps are wierd creatures and not good for linking. My question is, will Google index my tooltips and find this page useful and therefore worthy? I think the content is useful for my users but I just don't know if the implementation will work well. This screen will eventually have 5 or 6 notes on it and I don't want to do it the concensus is negative... Thanks for any advice.
Technical SEO | | Banknotes0 -
International Site, flow of page rank?
OK. I'm working on an international site. The site is setup with folders for UK, US, AU e.g www.site.com/UK/index.aspx The root (non folder based) is the international version of the site e.g www.site.com/index.aspx www.site.com/index.aspx has the lions share of links. Therefore, the pages immediately linked from www.site.com/index.aspx have page rank distributed between them. My UK, US and AU home pages are linked via a country selector from the www.site.com/index.aspx page via an aspx redirect page that 301's to the appropriate country home page. Therefore the home pages of UK, US, AU are recieving some of the 'juice' that is coming in to www.site.com/index.aspx (but only a fraction via the redirect links) Am I right in thinking that pages on the international version of the site will have much more potential to rank (because of their 'juice') than the pages on UK, US and AU versions of the site? If so, am I right in thinking that these will tend to rank over the equivalent UK, US and AU versions of the pages in each country version of Google despite having set directory level Geo-targetting in GWT?
Technical SEO | | QubaSEO1