Question about robots.txt
-
Solved!
-
Just a friendly reminder. Please don't delete your question after it's been answered. It's very likely that someone in the future could have the same question and they would have been able to find the answer if you hadn't deleted the question.
-
Consider deleting all of this:
Disallow: /&limit
Disallow: /?limit
Disallow: /&sort
Disallow: /?sort
Disallow: /?route=checkout/
Disallow: /?route=account/
Disallow: /?route=product/search
Disallow: /?route=affiliate/
Disallow: /?marca
Disallow: /&manufacturer
Disallow: /?manufacturer
Disallow: /?filter
Disallow: /&filter
Disallow: /?order
Disallow: /&order
Disallow: /?price
Disallow: /&price
Disallow: /?filter_tag
Disallow: /&filter_tag
Disallow: /?mode
Disallow: /&mode
Disallow: /?cat
Disallow: /&cat
Disallow: /?product_id
Disallow: /&product_id
Disallow: /?route=affiliate/
Disallow: /*?keywordThose rules are telling Google not to crawl domain.com/EVERYTHING(then the URL parameter). This could be where the issue stems from. If you're worried about URLs with these things ranking, consider implementing canonical tags instead to point to the proper pages
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Image Titles and Descriptions Question
Hello, I have a question about optimizing the SEO on my pages through image titles and descriptions. There are a few times on my website that I use the same image on multiple pages. I am under the impression that giving it a title such as "social-media-marketing-agency-graphic.jpg" will help the SEO for the phrase "social media marketing agency" on that page. My question was, if I want to use the same image on multiple pages, am I better off uploading an entirely new image with a new title to make it more relevant to the new page? Or will this not make large enough of a difference? Or is there an easier solution? Please let me know your thoughts on how to best optimize the pages
On-Page Optimization | | brightsocial0 -
Can Robots.txt on Root Domain override a Robots.txt on a Sub Domain?
We currently have beta sites on sub-domains of our own domain. We have had issues where people forget to change the Robots.txt and these non-relevant beta sites get indexed by search engines (nightmare). We are going to move all of these beta sites to a new domain that we disallow all in the root of the domain. If we put fully configured Robots.txt on these sub-domains (that are ready to go live and open for crawling by the search engines) is there a way for the Robots.txt in the root domain to override the Robots.txt in these sub-domains? Apologies if this is unclear. I know we can handle this relatively easy by changing the Robots.txt in the sub-domain on going live but due to a few instances where people have forgotten I want to reduce the chance of human error! Cheers, Dave.
On-Page Optimization | | davelane.verve0 -
SEO Content Revolution Question
I was wondering if articles written about questions people are asking will help my website rank better. For example let's say I wrote an article answering the query, "What Hair Dye Does Angela Merkel Use?" or, "Is Hillary Clinton Thinking of Running for President," and they rank well on google, and in turn they get viewed a lot by searchers because it answers their queries. Would this help my website as whole start ranking better? Thanks!
On-Page Optimization | | OOMDODigital0 -
Silly question about noindex and canonical
Hi, This is probably going to sound a bit stupid, but I nevertheless want to check. We have a site that's going to have identical pages (really not my choice) for a sales reason. The two examples would be example.com/profile-name and example.com/location/profile-name Users using the onsite navigation will always end up in the latter example naturally as they have to select a location before viewing content (plus having the location in the url is nice as there are multiple profiles across different locations that have the same name). However, it's easier to sell our services when we can offer just example.com/profile-name to users for their own marketing reasons. I'd like to make the example.com/profile-name noindex follow, and have just the example.com/location/profile-name indexed, but not sure if it would be better to implement canonical tags instead? Can anyone see any potential pitfalls of using either method or does it not really make a difference (which is what I suspect, but I'd rather look stupid than get this wrong)? Thanks!
On-Page Optimization | | LeahHutcheon0 -
URL question
When we type in the URL of www.JustBunkBeds.com on firefox we end up with (S) in URL https://www.justbunkbeds.com/ When we type in the URL of www.JustBunkBeds.com on Explorer we end up with http://www.justbunkbeds.com/ Appreciate answer to this question Tony
On-Page Optimization | | OCFurniture0 -
An ecomerce seo question
Looking for a few opinions on this please...Trying to reduce the number of pages I have to seo to rank on my websites and at the same time avoid the google over optimisation issues. Previously on our ecomerce websites we would have a category page for, say, 12 times, we would then seo that page for generic terms related to the page; ie, blue dress, cheap blue dress, blue party dress etc. The individual product pages would then be seoed with the title and h1 tags containing the exact product name and the url containing the product name too. This worked fine but we are suffering from some duplicate content issues of late (the products are mixture of few unique items and probably 95% imported affiliate datafeeds) as we have an average of 80,000 products per store we have neither the time nor the staff to rewrite everything (the products update daily directly from the merchants so would need to be done daily) What we are planning on moving toward is blocking the individual product pages from Google and instead putting all efforts into the category pages. The category page will contain plenty of quality unique content related to the category so the only duplicate content would be a line of the product name and price. Whilst we would still rank the category page for broad keywords we also would like to now rank the category page for 16 individual product names as there is a good profit to make made by the sheer volume of product names we plan on ranking for. Obviously we could not get all the products into the url and the page title as that would be silly but would it be acceptable to have multiple h2 tags on the page, each with a different entry, the product names (H1 will be saved for the category name). We can easily bold these keywords to help in the optimisation as per the seo moz onsite analysis tool and we can add image text to ensure the product name is featured at least twice on the page. As so few sites actually seo for the long tail product names, most retailers rank by virtue of their domain quality alone, our onsite seo doesn't have to be 100% but getting the best we can out of the page will help the efforts. Many thanks Carl
On-Page Optimization | | Grumpy_Carl0 -
Using meta robots 'noindex'
Alright, so I would consider myself a beginner at SEO. I've been doing merchandising and marketing for Ecommerce sites for about a year and a half now and am just now starting to attempt to apply some intermediate SEO techniques to the sites I work on so bear with me. We are currently redoing the homepage of our site and I am evaluating what links to have on it. I don't want to lose precious link juice to pages that don't need it, but there are certain pages that we need to have on the homepage that people just won't search for. My question is would it be a good move to add the meta robots 'noindex' tag to these pages? Is my understanding correct that if the only link on the page is back to the homepage it will pass back the linkjuice? Also, how many homepage links are too many? We have a fairly large ecommerce site with a lot of categories we'd like to feature, but don't want to overdo the homepage. I appreciate any help!
On-Page Optimization | | ClaytonKendall0 -
Product Title Formating Question
The majority of the products I sell require lengthy product titles, I have 600+ items that have titles over 70+ characters. In the interest of reducing the character count (Product Feeds) I'm trying to eliminate unnecessary "stop words" in the titles, such as "and", "with" etc.. I've listed an example of a current product listing below, followed by two different formats to reduce the character count. Which one if any is better for SEO or should I leave my titles alone and stick with my original format? Current Listing Example: DeWalt Dual Comfort Safety Glasses with Black Frame and Clear Anti-Fog Lens Suggested Listing Example: DeWalt Dual Comfort Safety Glasses-Black Frame-Clear Anti-Fog Lens DeWalt Dual Comfort Safety Glasses/Black Frame/Clear Anti-Fog Lens
On-Page Optimization | | MEldridge0