IPB

Welcome Guest ( Log In )

 
Reply to this topicStart new topic
> Site Crawler Submission To Google Webmaster Tools
msuz
post Oct 3 2008, 04:24 PM
Post #1





Group: Verified NS Member
Posts: 186
Joined: 13-August 08
Member No.: 1,898



I use Gsitecrawler to create my site indexes and in turn submit to webmaster tools.
In V.4 the index was approx 12000 urls.....my site now in V.7 is showing approx 21000 urls....?

Are there urls I should be removing from the crawl prior to submitting?

Any thoughts would be appreciated
Msuz.
Go to the top of the page
 
+Quote Post
Wackyjazz
post Oct 5 2008, 08:47 AM
Post #2





Group: Verified NS Member
Posts: 81
Joined: 27-June 08
From: em thar hills in Virginia
Member No.: 1,342



Msuz..

You will have to check your feed...I found that mine had /search.aspx as well as a few of the old .asp pages from my old site. I just deleted them before submission. If you submit with all the urls, G will just ignore the pages such as /search.aspx, or at least they did in my case. With my feed, anything that wasn't a product or category I removed from the feed.


QUOTE (msuz @ Oct 3 2008, 05:24 PM) *
I use Gsitecrawler to create my site indexes and in turn submit to webmaster tools.
In V.4 the index was approx 12000 urls.....my site now in V.7 is showing approx 21000 urls....?

Are there urls I should be removing from the crawl prior to submitting?

Any thoughts would be appreciated
Msuz.
Go to the top of the page
 
+Quote Post
tscalf
post Oct 6 2008, 09:31 AM
Post #3





Group: Verified NS Member
Posts: 21
Joined: 28-December 07
From: Chicago
Member No.: 437



Add search.aspx to Gsitecrawler (or other sitemap generator) exclusion list to reduce load and time to generate sitemaps.
Go to the top of the page
 
+Quote Post
msuz
post Oct 6 2008, 11:03 AM
Post #4





Group: Verified NS Member
Posts: 186
Joined: 13-August 08
Member No.: 1,898



QUOTE (tscalf @ Oct 6 2008, 09:31 AM) *
Add search.aspx to Gsitecrawler (or other sitemap generator) exclusion list to reduce load and time to generate sitemaps.


Thanks for the tips, also got tons of 'add product review' as well as 'email a friend' links showing up
Go to the top of the page
 
+Quote Post
msuz
post Oct 6 2008, 11:30 AM
Post #5





Group: Verified NS Member
Posts: 186
Joined: 13-August 08
Member No.: 1,898



QUOTE (tscalf @ Oct 6 2008, 09:31 AM) *
Add search.aspx to Gsitecrawler (or other sitemap generator) exclusion list to reduce load and time to generate sitemaps.



Noticed in V.7 file manager an extension called browseproducts. They all have .html extensions. Did these come over from V.4 site? I think I should get rid of these files all together?
Go to the top of the page
 
+Quote Post
MC Team - Eddie
post Oct 6 2008, 11:33 AM
Post #6





Group: Administrators
Posts: 794
Joined: 10-August 07
From: St. Louis
Member No.: 8



QUOTE (msuz @ Oct 6 2008, 10:30 AM) *
Noticed in V.7 file manager an extension called browseproducts. They all have .html extensions. Did these come over from V.4 site? I think I should get rid of these files all together?


The browseproducts folder in the file manager was brought over to v7 upon migration. There is no regeneration of this folder in v7 so any updates to products will not be shown in these files. You can delete them if you wish however if they are ranked high in the search engines, I would recommend either keeping them or adding them to the page redirects under operations.
Go to the top of the page
 
+Quote Post
msuz
post Oct 6 2008, 12:01 PM
Post #7





Group: Verified NS Member
Posts: 186
Joined: 13-August 08
Member No.: 1,898



QUOTE (MC Team - Eddie @ Oct 6 2008, 11:33 AM) *
The browseproducts folder in the file manager was brought over to v7 upon migration. There is no regeneration of this folder in v7 so any updates to products will not be shown in these files. You can delete them if you wish however if they are ranked high in the search engines, I would recommend either keeping them or adding them to the page redirects under operations.



In Gsitecrawler it asks for a list of file extensions to check:
asp,aspx,cfm,cgi,do,htm,html,jsp,mv,mvc,php,php5,phtml,pl,py,shtml

Should I rather just tell it to check .aspx only?
Go to the top of the page
 
+Quote Post
Wackyjazz
post Oct 7 2008, 11:48 AM
Post #8





Group: Verified NS Member
Posts: 81
Joined: 27-June 08
From: em thar hills in Virginia
Member No.: 1,342



That would depend on if you have any custome html pages...

QUOTE (msuz @ Oct 6 2008, 01:01 PM) *
In Gsitecrawler it asks for a list of file extensions to check:
asp,aspx,cfm,cgi,do,htm,html,jsp,mv,mvc,php,php5,phtml,pl,py,shtml

Should I rather just tell it to check .aspx only?
Go to the top of the page
 
+Quote Post

Reply to this topicStart new topic
Tags
No Tag inserted yet

1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 

RSS Lo-Fi Version    Network Solutions © 2008 Time is now: 8th January 2009 - 09:43 PM
Domain Names | Web Hosting | Web Design | Shopping Cart Software | Online Marketing | SSL Certificates