Kill your htaccess file, take the risk to learn a little
-
Last week I was browsing Google's index with "site:www.mydomain.com and wanted to scan over to see what Google had indexed with my site. I came across a URL that was mistakenly indexed. It went something like this
www.mydomain.com/link1/link2/link1/link4/link3
I didn't understand why Google had indexed a page like that of mine when the "link" pages were links that were on my main bar which were site wide links. It seemed to be looping infinitely over and over. So I started trying to see how many of these Google had indexed and I came across about 20 pages. I went through the process of removing the URL's in Webmaster Tools, but then I wanted to know why it was happening. I had discovered that I had mistakenly placed some links on my site in my header in such a manner
If you know HTML you will realize that by not placing the "/" in the front of the link I was telling that page to add that link in addition to the URL that is was currently on. What this did was create an infinite loop of links which is not good
Basically when Google went to www.mydomain.com/link1/ it found the other links which then told Google to add that url to the existing URL and then go to that link.
Something like: www.mydomain.com/links1/link2/...
When you do not add the "/" in front of the directory you are linking too it will do this. The "/" refers to the root so if you place that in front of your directory you are linking too it will always assume that first "/" as the root then the url will follow.
So what did I do?
Even though I was able to find about 20 URL's using the "site:" search method there had to be more out there. Even though I tried to search I was not able to find anymore, but I was not convinced.
The light bulb went on at this point
My .htaccess file contained many 301 redirects in my attempt to try and redirect those pages to a real page, they were not really relevant pages to redirect too. So how could I really find out what Google had indexed out there for me since Webmaster Tools only reports the top 1000 links.
I decided to kill my htaccess file. Knowing that Google is "forgiving" when major changes to your site happen I knew Google would not simply just kill my site for removing my htaccess file immediately.
I waited 3 days then BOOM! Webmaster Tools was reporting to me that it found a ton of 401's on my site. I looked at the Crawl Errors and there they were. All those infinite loop links that I knew had to be more out there, I was able to see.
How many were there?
Google found in the first crawl over 5,000 of them. OMG! Yeah could you imagine the "Low quality" score I was getting on those pages? By seeing all those links I was able to determine about 4 patterns in the links. For example:
Now my issue was I wanted to keep all the URL's that were pointing to www.mydomain.com/link1 but anything after that I needed gone. I went into my Robots.txt file and added this
Disallow: www.mydomain.com/link1/link2/
Disallow: www.mydomain.com/link1/link3/
Disallow: www.mydomain.com/link1/link4/
Disallow: www.mydomain.com/link1/link5/
Now there were many more pages indexed that went deeper into those links but I knew I wanted anything after the 2nd URL gone since it was the start of the loop that I detected. With that I was able to have from what I know at least 5k links if not more.
What did I learn from this?
Kill your htaccess file for a few days and see what comes back in your reports. You might learn something
After doing this I simply replaced my htaccess file and I am on my way to removing a ton of "low quality" links I didn't even know I had.
-
Interesting post. Yeah, the .htaccess file is the most important file out there and it is easy to mess up (as I am sure most everyone has at one time or another).
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Do old website files in the public_html effect SEO?
My client has about a dozen old folders filled with old websites including index files, robots, htaccess files. They are all located in separate files with in public_html. Does this effect them negatively?
Technical SEO | | Renalynd0 -
Google disavow tool ( how long does it take ? )
Hello, I disavowed some of my links about three months but still see them in my link profile, using OSE? How long does it take for Google to make them nofollow. Thanks
Technical SEO | | mezozcorp0 -
Large Companies Taking Over Rankings
Recently many large companies. (lowes, bed bath and beyond, walmart, target, ace hardware, etc.), have taken over the rankings of keywords we used to rank high in. Is there any way to improve our rankings and compete with these large companies? Is this due to recent Google changes? Sample keywords are: electrolux vacuum bags, oreck vacuum bags, vacuum cleaner bag, vacuum cleaner filter
Technical SEO | | totalvac0 -
Title tag not changing in Google. Can somebody take a look for me?
I'm using Yoast SEO plugin for the website. The website is http://www.emerypharmaservices.com. It appears on the webpage, the title tag is correct (home page should be Contract Laboratory Research Services for Analytical Chemistry and Microbiology), however, in Google it only says Emeryville Pharmaceutical Services. Could this be due to my settings? Please advise. Thank you
Technical SEO | | leopold49520 -
Htaccess Rewrites
Hi, I'm battling with duplicate content and would love to fix some redirections in my htaccess file: 1. I'm trying to use the www version of my site via 301 2. I'm trying to remove ALL /index.php from my url's I currently have the following code, but my home page /index.php is still not redirecting to the root? Options +FollowSymLinks
Technical SEO | | Klement69
Options +Indexes <ifmodule mod_rewrite.c="">RewriteEngine On
#RewriteBase / RewriteCond %{HTTP_HOST} ^funeralcoverfinder.co.za$ [NC]
RewriteRule ^(.*)$ http://www.FuneralCoverFinder.co.za/$1 [R=301,NC,L] RewriteCond %{QUERY_STRING} base64_encode[^(]([^)]) [OR]
RewriteCond %{QUERY_STRING} (<|%3C)([^s]s)+cript.(>|%3E) [NC,OR]
RewriteCond %{QUERY_STRING} GLOBALS(=|[|%[0-9A-Z]{0,2}) [OR]
RewriteCond %{QUERY_STRING} _REQUEST(=|[|%[0-9A-Z]{0,2})
RewriteRule .* index.php [F] RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]
RewriteCond %{REQUEST_URI} !^/index.php
RewriteCond %{REQUEST_URI} /component/|(/[^.]|.(php|html?|feed|pdf|vcf|raw))$ [NC]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . index.php [L]</ifmodule> Any Advice? - Thanks so much!0 -
Same image file with different alt text?
I have an image that represents 'widgets'. The image works for more than one kind of widget. I have two pages, one optimized for 'blue widgets' and one optimized for 'red widgets'. I would like to use the same 'widgets' image on both pages but change the alt text to be 'blue widgets' or 'red widgets' depending on the page it is used on. Should I: (1) use the same image on different pages with different alt text. (2) duplicate the image file and have two copies 'red_widgets.jpg' and 'blue_widgets.jpg' and then use each copy on the page optimized for the corresponding phrase. (3) create distinct, unique image files (where the pixels are different, not just the file names) for each kind of widget. This is a simplified example of a larger SEO problem where I have 1 image that can be useful on 20 pages that are each optimized for 20 different phrases. Should I use the same image with 20 different alt tags, or create 20 identical (but renamed) copies of the image, or create 20 slightly different image files (with different pixels in each image)? Thanks.
Technical SEO | | scanlin0 -
Does Google take user site blockings from Chrome as a spam signal?
When you perform a search in Chrome, click through to a result, then hit "back", you get a nice little option to "Block all example.com results" listed next to the result from which you backed out. I am assuming Google collects this information from Chrome users whose settings allow them to? I am assuming this is a spam signal (in aggregate)? Anyone know? Thanks!
Technical SEO | | TheEspresseo0 -
Reading Crawl Diagnostics and Taking Action on results
My site crawl diagnostics are showing a high number of duplicate page titles and content. When i look at the flagged pages, many errors are simply listed from multiple pages of product category search results. This looks pretty normal to me and I am at a loss for understanding how to fix this situation. Can I talk with someone? thanks, Gary
Technical SEO | | GaryQ0