My website pages are not crawled, what to do?
-
Hi all.
I have made some changes on the website so i like to crawled them by the search engines Google especially.
I have made these changes around 2 weeks ago. I have submitted my website on good bookmarking websites.
Also i used a tool available in Google webmasters "Fetch as Google", Resubmitted a sitemap.xml.
Still my pages are not crawled your opinion please.
Thanks
-
this is somenting I would love to know more!-)
I think news and updates, maybe a blog in your site can make google "feel" it has to come back more times to refresh your site again ( "hum dude, they are doing something there)... Links to your site and to new articles in social media could help a lot.
If you find anything let us know!-)
-
SEO Martin,
all these 93 results are index successfully also shows in our Google webmaster account.
But i am worried that why crawler is not visit our website pages on regular basis as we submit sitemap.xml in webmaster account after the changes have made.
what is the reason behind this.
also can you please explain me that what the other methods to force the search engine crawler to crawl our page.
Regards
Lucid
-
Hi Sha,
I just add this meta tag (as below) to help to crawl the page but not sure how it is.
I am just worried that if we make changes in our website pages and submit the site map in webmaster (that's mean that we are put request to search engine crawler to crawl the page ) but why they do not.
what is the actual reason behind this.
also can you please explain me that what the other methods to force the search engine crawler to crawl our page.
Regards
Lucid
-
HI Martin,
Exactly, all url are indexed in search engines but they are not crawled after making a changes that is the main concern for me now.
For example:
Take a look on Magento Development page that showing last crawled 17 Mar 2013. I have made some changes and that should be crawled on a regular basis same as previous crawling.
Url :http://www.lucidsoftech.com/magento-development.html
your thoughts?
Thanks
-
Lucid,
I´ve just made a search on google and I found 93 results for your site, http://www.lucidsoftech.com/ , is that url right?
How many results do you actually own?
-
HI Sha,
yes site is indexed and crawled on regular basis before making these changes. don't know why not this time and not crawled around 2 weeks. yes i checked for thart all have including tag.
Normally when u submitted your pages on few good social bookmarking, workout in webmasters, after generate few good back links all these thing are very helpful for page crawling.
Still waiting for the page crawling.Thanks Sha
-
HI Martin,
Before making these changes they crawled regular basis and I just changed a meta title keywords and description but don't know the reason why after making these changes on few pages my web pages are not crawled in past 2 weeks and and still they are not crawled.
Martin i have my robots.txt in public.html and i allow the page that should be crawled by the bots.
waiting fro the page crawling.Thanks
-
Hi lucidsoftech,
The site is definitely indexed, but without knowing the specific URLs you are waiting on, it is a little difficult.
I ran a quick crawl with Screaming Frog and there are actually quite a few pages on the site which have the noindex meta tag in place - have you checked that the pages you are wanting indexed are not excluded with a tag?
Just a thought,
Sha
-
make sure you have a robots.txt on public folder. You must allow bots and disallow some folders.
Example of robots.txt of wordpress site
User-agent: *
Allow: /Disallow: /*?
Disallow: /feed/
Disallow: /trackback/
Disallow: /wp-admin/
Disallow: /wp-content/
Disallow: /wp-includes/
Disallow: /xmlrpc.php
Disallow: /wp-
Disallow: /tmpAllow: /wp-content/uploads/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Page ranked then disappeared
Recently there have been a a couple of pages form my website that ranked well, in top 5 for a couple of days then they disappear suddenly, they are not at all seen in google search results no matter how narrow I search for them. I checked my search console, there seems to be no issues with the page, but when I check google analytics, I do not get any data from that page since the day it disappeared, and it does not even show up on the 'active pages' section no matter I keep the url open in multiple computers.
Technical SEO | | JoelssonMedia
Has anyone else faced this issue? is there a solution to it?0 -
Should we reinstate the old website?
In a nut shell we had a great site that performed well and grew month on month but it perhaps looked a bit dated. A decision was taken to build a new site and the job given to a PR agency for some reason. All the titles, H1 tags, page content and url structure was changed and now the site has drop 50% of organic traffic. I've been tasked with trying to rebuild rankings but so far it's not going well. A snapshot of the old website still exists and i'm very tempted to have it reinstated in the hopes that our traffic will recover. What are your thoughts?
Technical SEO | | etienneb0 -
Home Pages of Several Websites are disappearing / reappearing in Google Index
Hi, I periodically use the Google site command to confirm that our client's websites are fully indexed. Over the past few months I have noticed a very strange phenomenon which is happening for a small subset of our client's websites... basically the home page keeps disappearing and reappearing in the Google index every few days. This is isolated to a few of our client's websites and I have also noticed that it is happening for some of our client's competitor's websites (over which we have absolutely no control). In the past I have been led to believe that the absence of the home page in the index could imply a penalty of some sort. This does not seem to be the case since these sites continue to rank the same in various Google searches regardless of whether or not the home page is listed in the index. Below are some examples of sites of our clients where the home page is currently not indexed - although they may be indexed by the time you read this and try it yourself. Note that most of our clients are in Canada. My questions are: 1. has anyone else experienced/noticed this? 2. any thoughts on whether this could imply some sort of penalty? or could it just be a bug in Google? 3. does Google offer a way to report stuff like this? Note that we have been building websites for over 10 years so we have long been aware of issues like www vs. non-www, canonicalization, and meta content="noindex" (been there done that in 2005). I could be wrong but I do not believe that the site would keep disappearing and reappearing if something like this was the issue. Please feel free to scrutinize the home pages to see if I have overlooked something obvious - I AM getting old. site:dietrichlaw.ca - this site has continually ranked in the top 3 for [kitchener personal injury lawyers] for many years. site:burntucker.com - since we took over this site last year it has moved up to page 1 for [ottawa personal injury lawyers] site:bolandhowe.com - #1 for [aurora personal injury lawyers] site:imranlaw.ca - continually ranked in the top 3 for [mississauga immigration lawyers]. site:canadaenergy.ca - ranks #3 for [ontario hydro plans] Thanks in advance! Jim Donovan, President www.wethinksolutions.com
Technical SEO | | wethink0 -
Page Juice not moving???
Moved URL's from ldnwicklesscandles.com to ldnwicklesscandles.co.uk because I wanted to rank better for UK where I'm located and thought also the .co.uk for my competitors may have been giving them the advantage. Use Squarespace 7 (transferred over from SS5)----they told me to set primary domain to .co.uk and I've done it. I've also done a 301 redirect and done a change of address in webmaster tools although I'm not sure if all of this is needed? Squarespace seem to think just setting the primary domain is enough. My question is its been a couple of weeks, I've resubmited to Google webmaster to try to speed things up, the new URL is appearing in Google but none of my Page Juice seems to be transferring yet? How long will it take? I know not all the juice will move over but my PA/DA is non existent now and I have no idea if I'm just being impatient or I've done something wrong here. Not a Pro, Just a small biz owner here so forgive me if this has been asked before.
Technical SEO | | ldnwickless0 -
How to inform Google to remove 404 Pages of my website?
Hi, I want to remove more than 6,000 pages of my website because of bad keywords, I am going to drop all these pages and making them ‘404’ I want to know how can I inform google that these pages does not exists so please don’t send me traffic from those bad keywords? Also want to know can I use disavow tool of google website to exclude these 6,000 pages of my own website?
Technical SEO | | renukishor4 -
Two of Pages Have Been SendBoxed
Hello, I was number 1-2 for my local keyword term, but now im nowhere, those two urls dont even show up in Google search results, my other pages DO, so that is obvious Google sendboxed them, i dont remember doing aggressive non quality link building, and its not a competitive term, since i was number 1 in Google for over 3 months or so i checked this tool and found that two of my urls are in sendbox http://www.searchenginegenie.com/sandbox-checker.htm I was never sendboxed before, can you help me how can i get out of this, since its my client's website, and i have to get those pages up as soon as possible Thank You
Technical SEO | | tonyklu0 -
Can too many pages hurt crawling and ranking?
Hi, I work for local yellow pages in Belgium, over the last months we introduced a succesfull technique to boost SEO traffic: we have created over 150k of new pages, all targeting specific keywords and all containing unique content, a site architecture to enable google to find these pages through crawling, xml sitemaps, .... All signs (traffic, indexation of xml sitemaps, rankings, ...) are positive. So far so good. We are able to quickly build more unique pages, and I wonder how google will react to this type of "large scale operation": can it hurt crawling and ranking if google notices big volumes of content (unique content)? Please advice
Technical SEO | | TruvoDirectories0 -
Removing pages from website
Hello all, I am fairly new to the SEOmoz community. But i am working for a company which organizes exhibitons, events and training in Holland. A lot of these events are only given ones ore twice and then we do not organise them any more because they are no longer relevant. Every event has its own few webpages which provide information about the event and are being indexed by Google. In the past we did not remove any of these events. I was looking in the CMS and saw a lot of events of 2008 and older which are being indexed. To clean the website and the CMS i am thinking of removing these pages of old events. The risk is that these pages have some links to them and are getting some traffic, so if i remove them there is a risk of losing traffic and rankings. What would be the wise thing to do? Make a folder with archive or something? Regards, Ruud
Technical SEO | | RuudHeijnen0