Single Page on my client's website is not crawling and indexing new changes. What could be the possible reason?
-
I made several changes on client's website on different pages, changed titles, add content on few pages, moved blog from subdomain to sub directory. Everything is crawled but there is one page on the website (not part of the blog) that isn't getting crawled in Google and picking up changes. The last crawl of the website is 2 days back whereas that page was last crawled on 30th sep.
I just wanted to know the possible reasons and has anyone encountered this before?
-
Hi Moosa, do you have an update on this issue?
-
Things I check:
- header check (I've had pages appear 200 OK in webconfs and then massive errors in Browseo so I often check both.)
- Screaming Frog check - does the direct URL start a proper crawl?
- Make sure it's not blocked in WMT under removed pages (some website owners and/or bad SEOs try to demote sitelinks by removing the URL.)
- Meta tags & robots.txt both of which I assume you already checked.
- Duplicate content being indexed. Take a random sentence & throw it in quotes on Google. If a 100% dupe version comes up on their facebook page, archive.is, or some directory it may just need a content refresh to push through.
If you check all 5 of those off, then I'd definitely like to see the URL.
-
If that doesn't work, please share the page URL.
-
Thanks Drik,
I have double check the accessibility of page to crawlers and bots and pages does have decent amount of internal and external links pointing to the page.
Thanks for the Google fetch idea, i am going to try it to see if it works and update you accordingly.
-
Hi
I guess you have already checked that the page is accessible for crawlers and has sufficient internal links to be discovered by the bots (if not - try crawling the site with Screaming Frog)
If it's only one page that isn't picked up you could try a fetch like google & submit to index to force the page being indexed.
It could be that Google considers this page as less important and crawls with lower frequency compared to the other pages (to deep in the site, to few links, not enough content,...)
Dirk
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Should I remove a high traffic page on my website?
For the last few years, a particular blog post on my site has gotten 3 times as much traffic than any other page, even the home page; however, the topic of the post is only moderately related to the website topic and I'm wondering if all that unrelated traffic is negatively effecting SEO for our primary keywords. Here's an example.... Site topic: Yoga retreats in Costa Rica (we want to attract people who are interested in booking a yoga retreat) Blog Topic: How to extend your visa in Costa Rica (it's related only because it's about Costa Rica and travel, and may help our visitors stay longer) Other Notes: In 4 years, visitors to that blog post have never converted. Blog post bounce rate is 56%, significantly higher than almost any other page Lots of comments on the blog post so visitors to it are engaged and find it very useful To get an accurate reading of interested visitors to the site, i always have to filter entrance visits to this post in my analytics because these users are not an accurate representation of the visitors we're trying to draw. My question: Because I get so much traffic from the blog post, which is about the visa renewal process, will Google consider the website less about yoga and more about visas? If so, will it make it more difficult to rank well for yoga in Costa Rica? Does Google say to itself, "Hey, this website can't be an authority about both yoga and visas in Costa Rica so we're going to consider it a visa site because of all the visits and engagement it gets for that topic." So should I remove the post or just leave it alone? It offers a lot of people valuable information so I would never delete it entirely, but would redirect it somewhere else. Thanks!
On-Page Optimization | | Cabaretti0 -
Is there a limit to the number of duplicate pages pointing to a rel='canonical ' primary?
We have a situation on twiends where a number of our 'dead' user pages have generated links for us over the years. Our options are to 404 them, 301 them to the home page, or just serve back the home page with a canonical tag. We've been 404'ing them for years, but i understand that we lose all the link juice from doing this. Correct me if I'm wrong? Our next plan would be to 301 them to the home page. Probably the best solution but our concern is if a user page is only temporarily down (under review, etc) it could be permanently removed from the index, or at least cached for a very long time. A final plan is to just serve back the home page on the old URL, with a canonical tag pointing to the home page URL. This is quick, retains most of the link juice, and allows the URL to become active again in future. The problem is that there could be 100,000's of these. Q1) Is it a problem to have 100,000 URLs pointing to a primary with a rel=canonical tag? (Problem for Google?) Q2) How long does it take a canonical duplicate page to become unique in the index again if the tag is removed? Will google recrawl it and add it back into the index? Do we need to use WMT to speed this process up? Thanks
On-Page Optimization | | dsumter0 -
I have more pages in my site map being blocked by the robot file than I have being allowed to be crawled. Is Google going to hate me for this?
Using some rules to block all pages which start with "copy-of" on my website because people have a bad habit of duplicating new product listings to create our refurbished, surplus etc. listings for those products. To avoid Google seeing these as duplicate pages I've blocked them in the robot file, but of course they are still automatically generated in our sitemap. How bad is this?
On-Page Optimization | | absoauto0 -
Penalty for Changing Home Page Title Tags
Hey Mozzers I'm certain of the answer to this question, however I wanted to get some input from the experts in Moz-land to hopefully provide some additional perspective. I recently disagree with a client's assertion that there is some penalty Google levels for changing the title tags of your home page. Now, I understand changing the title tags can influence serp rankings, however, is anyone aware of some penalty Google levels for simply changing the title tags? Most of what I've read and experienced has people changing them all the time without some phantom penalty. It seems to me a problem of correlation = causality, in that people often attribute a drop to an action that may not have actually been the cause. Anyway, if you have any particular insight on this top I would appreciate it greatly. thanks!
On-Page Optimization | | BrandLabs0 -
Corporate Re-Branding, How to best lead into new website?
I have a company that is going to be moving away from a name they have had for 40 years, into a name they have had for the past 10. They are becoming a bigger brand in Canada and want to move away from the west coast name. http://www.wcmachinery.net/ is becoming obsolete. http://shearforce.ca/ will be taking over for both sites. Both of the websites are identically built by a previous web company. The current root domain metrics are shown in the image provided. What do you Mozzers feel would be the best way to phase out the old site without losing the little bit of domain authority wcmachinery.net has gained? west coast machinery is also well know to its customers, so getting the branded search traffic to the shearforce site is also a priority. I am fairly certain this can be handled easily in the about page where it explains the relationship between the two companies. As a side note I have not performed any SEO work on either site as of yet. So please do not judge me based on this site! Thanks for your insights, Randy edit edit
On-Page Optimization | | PixelgemsCreative0 -
How do I avoid duplicate content and page title errors when using a single CMS for a website
I am currently hosting a client site on a CMS with both a Canadian and USA version of the website. We have the .com as the primary domain and the .ca is re-directed from the registrar to the Canadian home page. The problem I am having is that my campaign produces errors for duplicate page content and duplicate page titles. Is there a way to setup the two versions on the CMS so that these errors do not get produced? My concern is getting penalized from search engines. Appreciate any help. Mark Palmer
On-Page Optimization | | kpreneur0 -
Adding keywords to URL's
I understand the importance of having the keyword in the URL (at least now I do). When I created my site (www.enchantingquotes.com), I was completely ignorant about SEO. So....question is...how do I go about adding keywords to already done pages? Do I create a new section and then redirect - or do I have to basically recreate pages? Thx much 🙂
On-Page Optimization | | enchantedgirlz0 -
Got loads of pages, but none indexing?
I have a WordPress site with loads of pages on a url like this http://mysite.com.au However, Google has indexed http://www.mysite.com.au and as a result only indexing 2 pages. How do I fix this? Many thanks Dan
On-Page Optimization | | Pokodot0