How deep should I let my forum be spidered
-
I run quote a niche website that's been running since late 1999 and over that time I've built up something like 4000 resources which consist of either text articles or image galleries and reviews along side another few thousand news stories relating to the niche interest. On top of the main site I also have a forum which isn't especially optimised for SEO and I was wondering, whilst was cleaning it up, whether anyone has any tips / suggestion / best practices for forum SEO.
Because it is all UGC the quality of the posts can be quite weak so I was wondering whether I should block robots completely from the forum, which seems a little harsh, whether I should let the whole forum be spidered (which seems a little excessive and potentially a bad thing) or whether I should restrict things to that only the main index and perhaps one page of topics and their posts be accessible to robots and then nofollow the rest?
Any thoughts?
-
Hey mate.
UGC can actually be very high quality content, and subsequently I would let Google run wild through all the content.
Forums can be tricky and you would want to make sure Google doesnt index any duplicate content.
I would use this tool, http://home.snafu.de/tilman/xenulink.html to crawl the forum. Export to Excel and then assess all the URL structures and see if it's picking up any strange parameters or duplicate content.
I would block the weird paramters in Google webmaster tools, and if technically possible 301 all the duplicate content issues. The problem with restricting things in robots.txt is that you might actually block link juice accidentally.
Hope that helps - my background is more in ecommerce but im sure we'll have similar site structure issues.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Do Search Engine Spiders Read Commented Out Content?
Do Search Engine Spiders Read Commented Out Content? Is commented out content detrimental?
On-Page Optimization | | lbohen0 -
Does Commented HTML Code Get Spidered?
I have a change, perhaps temporary, to my web store that I will be making. I am wondering if the code that I will comment out () gets spidered?
On-Page Optimization | | lbohen0 -
Using Transcription Service For Videos - Have Question Around Search and Spiders
Hi All, So I have put together a weekly video series on security topics. I have read an SEOmoz post around how you can boost SEO by adding the transcription to the page, which makes perfectly good sense. My question is, can I include the first couple of paragraphs and then have a "read the full transcription" so when the user clicks, the rest of the content appears? Do the search engine spiders only crawl the first two paragraphs in this instance or do they crawl the whole thing even though the entire content is not on the page? I dont mind making the page longer and including the entire transcription if it is easier for SEO but if there is no difference, than I think the first option would be the best user experience. Thanks for the help Pat
On-Page Optimization | | PatBausemer0 -
Html and css errors - what do SE spiders do if they come across coding errors? Do they stop crawling the rest of the code below the error
I have a client who uses a template to build their websites (no problem with that) when I ran the site through w3c validator it threw up a number of errors, most of which where minor eg missing close tags and I suggested they fix them before I start their off site SEO campaigns. When I spoke to their web designer about the issues I was told that some of the errors where "just how its done" So if that's the case, but the validator still registers the error, do the SE spiders ignore them and move on, or does it penalize the site in some way?
On-Page Optimization | | pab10 -
Relating forum discussions to improve internal linking
Hi all, I have a matter i would like a discussion about, since I am looking for a good solution. First the case: I have a site with a large community based discussion board. We daily have 40 to 80 active forum threads with average of 200 posts. As it is right now each thread page shows, at the bottom of the thread, the 20 threads with latest activity. From a SEO point of view this is not the best solution, since all thread pages passes link juice to the latest 20 threads. However, the threads change daily so the juice is sprayed around all over the place. What I want to do is related forum threads, such that each thread at the bottom shows up to 10 threads which could be of interest to the reader. In this way one thread will have more or less the same threads at the bottom at all time, unless the relevancy is better for other threads, causing som minor changes to happen over time. The question is, how can one do this? In the backend, the forum has two tables. One holding threads and one holding posts relating to the threads. All in all the system has 66.500 threads, and in total 469.000 posts. Every thread has a title, and the posts are of varying length. The threads have categories, but they are not so distinct that a thread in one category can not be related to a thread in another category. So I would like to make the relevancy from the title of the thread and the content of the forum posts. As of yet I have not come up with a good solution, and i will look forward to reading any feedback for this. I will answer any questions as fast as possible, to get a good discussion goint here. Best regards, Rasmus
On-Page Optimization | | rasmusbang0 -
Getting 403 error in forum
Hi all, I am getting 403 error for my site where it is throwing error for the following url http://www.topuniversityforum.in/members/member id/ignore and it is showing 7 similar url for 7 user ids. I want to know how can i resolve it and if it is going to have any negative effect on its ranking.
On-Page Optimization | | akhilendra0 -
Yahoo small business host has two web sites and confuses the spiders
Both another consultant and my first seomoz test stated that I have two websites and that confuses the spiders. . One is www.mystore.com ( example) and the other is mystore.com. Yahoo states that's how they role and I can not delete one. I can do a 301 redirect and redirect one site to the other but they do not have a recommendaation as to which one to redicrect. Which one should I redirrect?
On-Page Optimization | | Wales0 -
How deep should I go with a directory site?
I am creating a new site which has a directory component. Based on what I have ready I am inclined to keep the site architecture as flat as possible. However, the natural layout that I have come up with in my head has the directory listings 5 or 6 pages deep in the site structure. I saw in another post that someone in a similar situation was suggesting that going deep like this is fine so long as there are many internal links to the deeper pages to indicate that they are important. Should I make a conscious effort to make the site architecture as flat as possible? Are there any specific guides/resources that address this particular issue that I should be aware of? Thanks!
On-Page Optimization | | fastestmanalive0