Website Indexing Issues - Search Bots will only crawl Homepage of Website, Help!
-
Hello Moz World,
I am stuck on a problem, and wanted to get some insight. When I attempt to use Screaming Spider or SEO Powersuite, the software is only crawling the homepage of my website. I have 17 pages associated with the main domain i.e. example.com/home, example.com/sevices, etc. I've done a bit of investigating, and I have found that my client's website does not have Robot.txt file or a site map. However, under Google Search Console, all of my client's website pages have been indexed.
My questions, Why is my software not crawling all of the pages associated with the website? If I integrate a Robot.txt file & sitemap will that resolve the issue?
Thanks ahead of time for all of the great responses.
B/R
Will H.
-
Hi Will,
It'd be impossible to find a solution to the problem without having the domain. If you want to message me the URL, I can take a quick look for ya.
A lack of either of those files shouldn't create any crawl issues. Setting up those files won't fix your problem, but you should add them both anyway.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How will google respond to allowing multilingual search terms for a single language website?
We would like to set up a website in English language only and promote this in various European countries. As said the website will only be available in English language, but we will keep translations (google translate) in backend. When a user in France then enters search query in French language in browser, a search can be done in French content, but we will present relevant content in English. Does anyone have any experience with that? Will it be allowed given the fact that the result (in English language) will probably not include any of the terms that was searched on (in French language).
Intermediate & Advanced SEO | | hansdef0 -
Will google be able to crawl all of the pages given that the pages displayed or the info on a page varies according to the city of a user?
So the website I am working for asks for a location before displaying the product pages. There are two cities with multiple warehouses. Based on the users' location, the product pages available in the warehouse serving only in that area are shown. If the user skips location, default warehouse-related product pages are shown. The APIs are all location-based.
Intermediate & Advanced SEO | | Airlift0 -
Redirecting Old Websites to New Websites
Hi Everyone, We are about to take down a number of websites in favour of a new singular B2B hub and would be looking to redirect all of these sites to the new home. For SEO purposes, what would be the best way to do this? Due to the difference in setups and scale of the site, it would be difficult to correctly match up each page to page between the sites for individual 301 redirects. Could someone advise on the best plan of action? Thanks.
Intermediate & Advanced SEO | | chbiz0 -
Sitemap indexing
Hi everyone, Here's a duplicate content challenge I'm facing: Let's assume that we sell brown, blue, white and black 'Nike Shoes model 2017'. Because of technical reasons, we really need four urls to properly show these variations on our website. We find substantial search volume on 'Nike Shoes model 2017', but none on any of the color variants. Would it be theoretically possible to show page A, B, C and D on the website and: Give each page a canonical to page X, which is the 'default' page that we want to rank in Google (a product page that has a color selector) but is not directly linked from the site Mention page X in the sitemap.xml. (And not A, B, C or D). So the 'clean' urls get indexed and the color variations do not? In other words: Is it possible to rank a page that is only discovered via sitemap and canonicals?
Intermediate & Advanced SEO | | Adriaan.Multiply1 -
Indexation of internal search results from infinite scroll
Hello, I have an issue where we will have a website set up with dynamic (AJAX) result pages based on the selection of certain filters chosen by the user. The result page will have 12 results shown and if the user scrolls down, the page will lazy load (infinite scroll) additional results. So for example, with these filters: Filter A: Size Filter B: Color Filter 😄 Location We could potentially have a page for "Large, Blue, New York" results dynamically generated. My issue is that I want Google to potentially crawl and index all these variations, so that I can have a page that ranks for "Large Blue New York", another page that ranks for "Small Orange Miami" etc. However, I do not need all the products indexed--- just the page with the first set of dynamic results would be enough since the additional products would just be more of the same. In other words, I am trying to get these pages with filters applied indexed and not necessarily get every possible product indexed. Can anyone comment on the best way to Get Google to index all dynamic variations? The proper way of paginating pages? Thank you
Intermediate & Advanced SEO | | Digi12340 -
How to Submit My new Website in All Search Engines
Hello Everyone, Can Any body help to suggest Good software, or Any other to easily Submit my website , to All Search Engines ? ? Any expert Can help please, Thanx in Advance
Intermediate & Advanced SEO | | falguniinnovative0 -
Will using my Homepage as a KW target improve my Inner page Ranking?
Hello your help please! I have 2 KWs that i have targeted Inner pages for and they have got them to page 2 in SERPs, but now its getting difficult to move them up to page 1. Will targeting the home page with a higher authority, for the same terms, help or hinder the inner pages current position? Many Thanks Ash
Intermediate & Advanced SEO | | AshShep10 -
How can Google index a page that it can't crawl completely?
I recently posted a question regarding a product page that appeared to have no content. [http://www.seomoz.org/q/why-is-ose-showing-now-data-for-this-url] What puzzles me is that this page got indexed anyway. Was it indexed based on Google knowing that there was once content on the page? Was it indexed based on the trust level of our root domain? What are your thoughts? I'm asking not only because I don't know the answer, but because I know the argument is going to be made that if Google indexed the page then it must have been crawlable...therefore we didn't really have a crawlability problem. Why Google index a page it can't crawl?
Intermediate & Advanced SEO | | danatanseo0