Strange Webmaster Tools Crawl Report
-
Up until recently I had robots.txt blocking the indexing of my pdf files which are all manuals for products we sell. I changed this last week to allow indexing of those files and now my webmaster tools crawl report is listing all my pdfs as not founds.
What is really strange is that Webmaster Tools is listing an incorrect link structure: "domain.com/file.pdf" instead of "domain.com/manuals/file.pdf"
Why is google indexing these particular pages incorrectly? My robots.txt has nothing else in it besides a disallow for an entirely different folder on my server and my htaccess is not redirecting anything in regards to my manuals folder either. Even in the case of outside links present in the crawl report supposedly linking to this 404 file when I visit these 3rd party pages they have the correct link structure.
Hope someone can help because right now my not founds are up in the 500s and that can't be good
Thanks is advance!
-
Hello,
Did you check the "linked From" tab? click on each error and see which are the sites that are linked from
-
Thanks for the help Wissam!
What I have done is changed all relative paths to direct- then I ran screaming frog and it did not pick up any 404s at all - this was last Thursday. Unfortunately webmaster tools is still reporting the same style 404s having been discovered since then. Is there a reason why screaming frog and webmaster tools would be seeing different crawl results?
-
all link reported in the GWT is based on a crawl.( so there is either an external or internal link pointing to these.com/file.pdf)
So what i would do is fire up Screaming Frog or Xenu and do a full site crawl and check the reports. You might find some pages linking or using relative urls in the a href elements.
If you land into a situation where you have external links pointing to wrong URLS I would recommend either by contacting them or just 301 /file.pdf to /manuals/file.pdf
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Webmaster tools showing 200 page load ok - all other testing tools show a 301
hey, on https://www.xxx.co we've setup a 301 redirect to xxx.us - > BUT in webmaster tools its still showing a 200 load ok, whereas on all other testing tools its showing a 301 redirect (screamingfrog etc) even https://dns.google.com/query?name=www.xxx.co is showing that its 301 redirected. Any ideas? as we want to trigger the change of address tool in WMT and its saying it cant as it loads the homepage still....
Technical SEO | | RobertN-London0 -
Bing Webmaster Shows Domain without WWW
One of our sites shows thousands of 301 redirects due to domain without www in Bing Webmaster under crawl Information page. It’s been like this for a long time. None of the internal pages have domain without www, it was tested through Screaming Frog. We do have www preference set in google webmaster, but unfortunately bing doesn’t have this option. We also specify URL with www preference through structural data, but that still doesn’t help. Did anyone have similar problems with Bing, and how did you resolve it?
Technical SEO | | rkdc1 -
Structured Data Markup Helper in Webmasters
Structured Data Markup Helper in Webmasters. Once i set the article or Movies with star rating etc should i replace the whole code from here into my page? is there some other way?
Technical SEO | | AlexisWithers0 -
Massive drop off in Google crawl stats
Hi Could i get a second opinion on the following please. ON a client site we seem to have had a massive drop off in google crawling in the past few weeks, this is linked with a drop in search impressions and a slight reduction in penalty. There are no warning messages in WMT to say the site is in trouble, and it shouldn't be, however cannot get to the bottom of what is going on. In Feb the Kilobytes downloaded per day was between 2200 and about 3800, all good there. However in the past couple of weeks it has peaked at 62 and most days are not even over 3! Something odd has taken place. For the same period, the Pages crawled per day has gone from 50 - 100 down to under 3. At the same time the site speed hasn't changed - it is slow and has always been slow (have advised the client to change this but you know how it is....) Unfortunately I am unable to give the site url out so i understand that may impact on any advice people could offer. Ive attached some screen shots from WMT below. Many thanks for any assistance. stats.png
Technical SEO | | daedriccarl0 -
Ajax Optimization in Mobile Site - Ajax Crawling
I'm working on a mobile site that has links embedded in JavaScript/Ajax in the homepage. This functionality is preventing the crawlers for accessing the links to mobile specific URLs. We're using an m. sub-domain. This is just an object in the homepage with an expandable list of links. I was wondering if using the following solution provided by Google will be a good way to help with this situation. https://developers.google.com/webmasters/ajax-crawling/ Thanks!
Technical SEO | | burnseo0 -
Sitemap as Referrer in Crawl Error Report
I have just downloaded the SEOMoz crawl error report, and I have a number of pages listed which all show FALSE. The only common denominator is the referrer - the sitemap. I can't find anything wrong, should I be worried this is appearing in the error report?
Technical SEO | | ChristinaRadisic0 -
Alternatives to SEOmoz's Crawl Diagnistics
I really like SEOmoz's Crawl diagnostics reports, it goes through the pages and finds all sorts of valuable information, I wanted to know if there are any other services that compete against this specific service, to test the accuracy of their crawl diagnistics. Thanks
Technical SEO | | BestOdds0 -
Application/x-msdownload in crawl report?
In a crawl report for my company's site, the "content_type_header" column usually contains "text/html". But there are some random pages with "application/x-msdownload"... what does "application/x-msdownload" mean? <colgroup><col width="121"></colgroup>
Technical SEO | | JimLynch
| |0