Can anyone help me diagnose an indexing/sitemap issue on a large e-commerce site?
-
Hey guys. Wondering if someone can help diagnose a problem for me.
Here's our site: https://www.flagandbanner.com/
We have a fairly large e-commerce site--roughly 23,000 urls according to crawls using both Moz and Screaming Frog. I have created an XML sitemap (using SF) and uploading to Webmaster Tools. WMT is only showing about 2,500 urls indexed. Further, WMT is showing that Google is indexing only about 1/2 (approx. 11,000) of the urls. Finally (to add even more confusion), when doing a site search on Google (site:) it's only showing about 5,400 urls found. The numbers are all over the place!
Here's the robots.txt file:
User-agent: *
Allow: /
Disallow: /aspnet_client/
Disallow: /httperrors/
Disallow: /HTTPErrors/
Disallow: /temp/
Disallow: /test/Disallow: /i_i_email_friend_request
Disallow: /i_i_narrow_your_search
Disallow: /shopping_cart
Disallow: /add_product_to_favorites
Disallow: /email_friend_request
Disallow: /searchformaction
Disallow: /search_keyword
Disallow: /page=
Disallow: /hid=
Disallow: /fab/*Sitemap: https://www.flagandbanner.com/images/sitemap.xml
Anyone have any thoughts as to what our problems are??
Mike
-
A site running ASP should be perfectly fine. I bet you will see substantial increases in a lot of positive metrics by just pairing down that navigation.
-
Thanks so much for your response, Russ.
You're confirming one of the many issues we have identified (too many internal links) but I had not connected it to indexing or site speed. When I use the Google Page Speed Tool, many of our pages are not even registering. It seems like it's taking too long to load them so it times out. Could the crazy amount of links have to do with this, too?
Moreover, our mobile speed is especially poor. This could be an even bigger problem in mobile, no?
Are you familiar with .asp sites, in particular, having indexing issues...or is that a false assumption?
Mike
-
Thanks for the question!
First, it is very common to get inconsistent answers from GSC, site:, sitemap and crawl results. Don't worry too much about that.
Your goal is to get as many of your pages indexed and that is a function of links pointing to your site and internal link structure. While it is an imperfect analogy, we often refer to this as "crawl budget". There are essentially 2 solutions to this...
1. Get more/better backlinks to a diversity of pages on your site.
2. Improve your internal link architecture so that Googlebot finds your pages more quickly.
I think the problem in your case is that the site inundates bots with generic navigational links. For example, this page...
http://www.flagandbanner.com/products/chrome-air-force-lt-general-flag-kit.asp
has 1400 internal links! That is crazy!
This page has 1500!
https://www.flagandbanner.com/products/citizenship-gifts.asp
You need to reel this back in dramatically. Your navigation should like to top level categories or maybe a handful of subcategories. Once in a category, you can reveal deeper categories. This will increase the likelihood that the related and "also" buy links that you find on product pages will get found and followed by Googlebot.
Finally, on a different note, you need to make sure you standardize the casing of URLs (ie: /Products/ or /products/) I noticed that you have links both internal and external that do not take this into account, causing unnecessary duplicate content.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Site traffic/sales have plummeted
About 2 months ago we relaunched our Ecommerce store on Shopify Plus and have since seen a massive drop in traffic, sales and our most valuable pages are nowhere to be found. Also, GWT is showing that Google is indexing about half of our pages and none of the images are being indexed. We did extensive keyword research, created/implemented a keyword framework, wrote brand new category/product page content, implemented schema markup, optimized our blog content and even did link building where we got some 90+ DA links. We are literally at a loss for what is causing this. Our experience with Shopify Plus has been very poor because it doesn't even do basic SEO stuff so we've had to do a lot of workarounds to make it "SEO friendly". Has anyone else ever switched to Shopify Plus and had similar issues? Is there a silver bullet that you can think of that we are missing that could get the site being indexed/ranking again?
Intermediate & Advanced SEO | | Aquatell0 -
Possible problem with new site (GWT no queries/very low index vs. submitted)
Hi everyone, I recently launched a new website for a small business loan company in the Dallas area. The site has been live for roughly a month and a half. I submitted everything to GWT as usual, including my sitemap. I am not sure what's going on with the site, as there is no activity from GWT in the impressions or queries. The submit vs. index is 24/3 (and hasn't moved). Also the queries graph on the overview stops at 3/18/2015... On another note, when I go to Crawl > Sitemaps, it shows that there were pages indexed during the month of march and then on April 3 it drops from 17 to 2 and never increases. Google says there are no errors or issues found, but I feel like there's something wrong. When I do site:, my URLs do pop up which makes me believe there's just a problem with my GWT. With that being said, I'm not happy THINKING there's something wrong. I need to actually know what the problem is. The only thing I can think of that I have done is purchase SSL for the site, but when I search what pages are indexed using www. it shows all the HTTPS URLS, so that would tell me that the site is getting indexed without a problem? Does anyone have a clue as to what might be happening? I will attach some screen shots so that you can get a better idea... KQ2366i D5xBNZf mF7kkgW
Intermediate & Advanced SEO | | jameswesleyhunt0 -
GWT url parameter issue/question
Hi Moz community, I'm having an issue with URL parameters in GWT. The tracking taxonomy for my websites is used as either /?izid=... (internal) OR /?dzid=... (external) I put tracking parameters in GWT as izid & dzid, but it hasn't picked up any URLs or examples in regards to these parameters. It's been about 2 months since we've started using this so I want to make sure Google isn't indexing as duplicate content. Side note: any page that uses a tracking parameter automatically adds rel="canonical" to the original page. Could this be the reason that GWT doesn't pick up any URLs for tracking parameters and/or do I not need to worry about adding paramters if I already have the canonical attribute automatically in place. Thanks for your help,
Intermediate & Advanced SEO | | IceIcebaby
-Reed0 -
XML Sitemap Questions For Big Site
Hey Guys, I have a few question about XML Sitemaps. For a social site that is going to have presonal accounts created, what is the best way to get them indexed? When it comes to profiles I found out that twitter (https://twitter.com/i/directory/profiles) and facebook (https://www.facebook.com/find-friends?ref=pf) have directory pages, but Google plus has xml index pages (http://www.gstatic.com/s2/sitemaps/profiles-sitemap.xml). If we go the XML route, how would we automatically add new profiles to the sitemap? Or is the only option to keep updating your xml profiles using a third party software (sitemapwriter)? If a user chooses to not have their profile indexed (by default it will be index-able), how do we go about deindexing that profile? Is their an automatic way of doing this? Lastly, has anyone dappled with google sitemap generator (https://code.google.com/p/googlesitemapgenerator/) if so do you recommend it? Thank you!
Intermediate & Advanced SEO | | keywordwizzard0 -
Why is this site not indexed by Google?
Hi all and thanks for your help in advance. I've been asked to take a look at a site, http://www.yourdairygold.ie as it currently does not appear for its brand name, Your Dairygold on Google Ireland even though it's been live for a few months now. I've checked all the usual issues such as robots.txt (doesn't have one) and the robots meta tag (doesn't have them). The even stranger thing is that the site does rank on Yahoo! and Bing. Google Webmaster Tools shows that Googlebot is crawling around 150 pages a day but the total number of pages indexed is zero. It does appear if you carry out a site: search on Google however. The site is very poorly optimised in terms of title tags, unnecessary redirects etc which I'm working on now but I wondered if you guys had any further insights. Thanks again for your help.
Intermediate & Advanced SEO | | iProspect-Ireland0 -
Mobile Sitemap Issue
Hi there, I am having some difficulty with an error on Webmaster Tools. I'm concerned with a possible duplicate content penalty following the launch of my mobile site. I have attempted to update my sitemap to inform Google that a different mobile page exists in addition to the desktop page. I have followed Google's guidelines as outlined here:
Intermediate & Advanced SEO | | DBC01
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=34648 I'm having problems with my sitemap.xml file. Webmaster tools is reporting that it is not able to read the file and when I validate it I am getting an error stating that the 'Namespace prefix xhtml on link is not defined'. All I am trying to do is to create a sitemap that uses the rel="alternate" to inform Google that their is a mobile version of that specific page in addition to the desktop version. An instance of the code I am using is below: xml version="1.0" encoding="UTF-8"?> xml-stylesheet type="text/xsl" href="gss.xsl"?> <urlset< span="">xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xsi:schemaLocation="http://www.google.com/schemas/sitemap/0.84 http://www.sitemaps.org/schemas/sitemap/0.9/sitemap.xsd"> http://www.mydomain/info/detail/ <xhtml:link< span="">rel="alternate" media="only screen and (max-width: 640px)" href="http://m.mydomain.com/info/detail.html"/> <lastmod></lastmod>2013-02-01T16:03:48+00:00<changefreq></changefreq>daily0.50</xhtml:link<></urlset<> Any help would be much appreciated. Thanks0 -
Http://blogsearch.google.com/ping
Is there any reason why a website would submit all their content (videos, photo galleries, articles) to this?
Intermediate & Advanced SEO | | MargaritaS0 -
My site is duplicated on the internet, please help.
I've been told that my site: "- your site is duplicated on the internet. Both www.joeyvalyphotography.com and joeyvalyphotography.com are valid internet addresses. This is a problem for SEO." I am wondering, what's the cause of this, and how it can fixed. Thanks In advanced, Joey
Intermediate & Advanced SEO | | gaji0