Crawl test. Bot crawled only 200 or so links when it should have crawled thousands
-
Hi everyone,
I just recieved my crawl test report and its only given me 200 or so URL's when my site has thousands, any thoughts?
-
Hi Ryan,
I am the site owner and this is the precise reason im trying to take matters into my own hands.
<meta name="keywords" content="E60,Rear,lamp,set" /> I see what you mean, because this is actually ridiculous, not quite sure how it got into this state either. Whats that saying, "if you want something done you have to do it yourself" Looks like i have to take a crash course in SEO to sort it all out. Thanks very much for all your help.
-
I realize you may not have full control over the site. What I would share is:
"That's how the site is" is not an acceptable response, unless the site owner is satisfied with their current SEO ranking.
The keywords have NOTHING to do with the product being displayed on the page. The link I offered is for a Hella e60 Rear Lamp. The only related in the keyword section is "rear". I am quite certain that is by coincidence.
Your keywords are not dynamically generated to vary with the pages content, nor were they manually altered to fit the pages content. The keyword selection is awful. The numbers "3", "5", and "7" are listed as 3 of the key words.
I want to help you, so don't take this the wrong way. The best thing about that site is it probably qualifies as a textbook case of what NOT to do from a SEO perspective. Perhaps you can appeal to a SEO company to use the site in a case study and turn it around.
-
Thank you very much Ryan, the columns on two sides of the page cant be helped as thats, how the site is, only the central content changes. However the duplicate keywords are for the products themselves, for example i sell 50 different BMW oil filters. Theres not much i can do about duplicating keywords as all of the products are very very similar.
I think you might be right about the site redesign....
-
A few notes about your site:
-
you are using meta keyords in your header. It offers no benefit and I would suggest removing it. It's not related to your inquiry but is something I noticed.
-
your site has a 50 keyword TAG block with the same keywords on every page. This isn't good from a SEO perspective on many levels. You want your keywords to focus the unique content on a given page
-you site pages are likely viewed as all duplicates. I can recognize the main item in the center of the page changes, but would a crawler? Your left and right sidebars are identical on all pages, along with most of your header. The actual content you offer is only a small percentage of the total page.
The large image of the various car parts is not considered as part of the content, aside from the ALT tag.
Look at a random page from your site: http://www.incarmotorfactors.co.uk/content/16-hella-bmw-e60-rear-lamp-set
According to the Analyze tool there are 5975 words on the page. I estimate about 100 of them are unique words addressing your Rear Lamp product, and the remaining 5800+ words are exactly the same as every other product page.
A crawler will see your pages as 98% duplicated data and the result will likely be your site isn't going to be listed. I would recommend a site re-design. Before taking that advice, it would probably be best to hear from others who have a lot more SEO expertise then myself.
-
-
-
What is different about your site? Is it flash or javascript based? Can you share your site URL?
-
Hi Ryan,
I used
On-Page Optimization Tools: Crawl Test. However this problem may be deeper than i first thought, as SEOMoz is not able to read any of my site info properly.
Open site explorer cant read it
Linkscape cant read the links. Crawl test isnt read properly, however my server and robots.txt files are fine, theres no blocking attempts from the server. Very strange.
-
What Crawl Test tool did you use?
Depending on the crawl tool, it may not look at content blocked from your Robots.txt file. You may want to ensure it is configured correctly.
Are there any permission issues? The crawler will look at your site the way a guest would. Any content which requires users to log in would be hidden to the crawler.
Are there any other issues regarding your site's accessibility? Connection or firewall issues? Could a server admin have seen a server performance issue and kicked the crawler before it finished? You can check server logs for this information.
If you check everything and do not locate a definitive cause, I would suggest running the crawl once more and checking the results before pursuing the matter further.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz & Xenu Link Sleuth unable to crawl a website (403 error)
It could be that I am missing something really obvious however we are getting the following error when we try to use the Moz tool on a client website. (I have read through a few posts on 403 errors but none that appear to be the same problem as this) Moz Result Title 403 : Error Meta Description 403 Forbidden Meta Robots_Not present/empty_ Meta Refresh_Not present/empty_ Xenu Link Sleuth Result Broken links, ordered by link: error code: 403 (forbidden request), linked from page(s): Thanks in advance!
Moz Pro | | ZaddleMarketing0 -
SEO Crawl Report Images?
Does SEOMOZ crawl images in the report? Raven tools is showing me about 200 missing alt tags and title tags. I can not seem to find any of this information on the SEOMOZ report. Am I missing something?
Moz Pro | | jasonsixtwo0 -
Why are inbound links not showing?
I run the site http://www.eurocheapo.com and am finding that many inbound links are not showing up in OSE and on the toolbar. For example, check out this hotel review: http://www.eurocheapo.com/paris/hotel/hotel-esmeralda.html In OSE it shows only 2 links (from 1 domain), which is crazy. It has dozens of inbound links from many different domains (links:http://www.eurocheapo.com/paris/hotel/hotel-esmeralda.html). I notice this all over my site. Pages that we link between are also showing no internal links -- which is easy to disprove. Was there a problem with this crawl? Or is the problem in our code? Many thanks for your help, Tom
Moz Pro | | TomNYC0 -
Rogerbot did not crawl my site ! What might be the problem?
When I saw the new crawl for my site I wondered why there are no errors, no warning and 0 notices anymore. Then I saw that only 1 page was crawled. There are no Error Messages or webmasters Tools also did not report anything about crawling problems. What might be the problem? thanks for any tips!
Moz Pro | | inlinear
Holger rogerbot-did-not-crawl.PNG0 -
Total link graph question
how is it that our website and our two competitors basically have the exact same link graph, except for at different levels?
Moz Pro | | imageworks-2612900 -
SEOMOZ Crawling Our Site
Hi there, We get a report from SEOMOZ every week which shows our performance within search. I noticed for our website www.unifor.com.au that it looks through over 10,000 pages, however our website sells less than 500 products so not sure why or how so many pages are trawled? If someone could let me know that would be great. It uses up a lot of bandwidth doing each of these searches so if the amount of pages being trawled reduced it would definitely assist. Thanks, Geoff
Moz Pro | | BeerCartel750 -
Can someone explain why I have been seeing an increase in the number of Linking Page URLs in OSE that link directly to downloads?
Ever since the last couple Linkscape updates when doing competitive back link analysis I have noticed a large increase in the number of URLs of Linking Pages in OSE that result in an immediate file download. The majority of the time these downloads are not common files ie PDF, DOC files. For example, these were all in a competitors back link profile: http://download.unesp.br/linux/debian/pool/main/i/isc-dhcp/isc-dhcp-relay-dbg_4.1.1-P1-17_ia64.deb http://snow.fmi.fi/data/20090210_eurasia_sd_025grid.mat http://www.rose-hulman.edu/class/me/HTML/ES204_0708_S/working model examples/Le25 mad hatter.wm?a=p&id=145880&g=5&p=sia&date=iso&o=ajgrep These are just a few I came across for a single competitor. Is this sketchy black hat SEO, some sort of error, actual links, or something else? Any information on this subject would be helpful. Thank you.
Moz Pro | | Gyi0 -
Individual Link Value
We understand the PA, DA, trust and all of that. My question is, is there a process or formula anyone uses that shows an individual links value as to the link juice it passes. The old Domain Juice seemed to be that, but after further investigation (And Rand setting me straight) I now understand it's not a good metric. Today, we use PA divided by the number of external links on that page to get some sense of an individual links actual value to the site or page we link to. I understand this is a very sloppy system, but seems to be the only choice we have? It's based on this simple thought. If you get a back link on two different pages, and both are equal in every way, except one has 3 outbound links and the other has 30, the link from the page with 3 will be significantly stronger as far as passing juice. So... anyone using something to determine an individual links value? I did ask the SEO staff, and they do not current have it.
Moz Pro | | MBayes0