Can anyone recommend a specific sitemap generator? I want to submit it to Google, but would feel better if someone pointed me in the right direction.

I used this free tool
http://www.auditmypc.com/free-sitemap-generator.asp

on recommendation of somebody here. works great, found a couple of broken links that I was able to fix quickly. The only issue I had is that it wouldn't export the xml file on my vista/ie7 machine. I had to export it to a different computer on my network and then upload it from there. Worked fine in the end. Took about a half hour to generate btw. let google know where it is, and yahoo too. Neither had any problems with it!

good luck!

You can use Gsitecrawler. Its free also.

How do you upload the icon that is viewable in your URL?

It is called "favicon" you should name it favicon.ico and upload it to your file manager root directory

I posted this in another forum, but I will post it here as well.

I use:
http://www.xml-sitemaps.com/

From the results, I download 4 files: XML, ROR, TEXT, HTML

I upload all 4 of these to the file manager.

Before doing that, I delete from the file manager the previous 4 files that I had uploaded. I also delete the 4 files from my computer so that the new 4 files are not renamed.

After I upload all 4 files to the file manager, I go to Google Webmaster Tools and I resubmit (upload) the XML sitemap.

I do all of this every time I add products to my website. It also seems to trigger Google to re-crawl my website.

Thanks gymbratleotards for a very detailed explanation. I wanted to chime in here with an additional detail that our software creates an automated sitemap file that you can use to submit to Google Webmaster Tools. It's in your root directory -

http://www.example.com/xml-sitemap.ashx

This is an XML formatted sitemap. You can submit multiple sitemaps to Google for the same site. As an aside, I'm curious if anyone uses their sitemaps on Yahoo with - http://siteexplorer.search.yahoo.com

/xml-sitemap.ashx does it make changes when we make changes to website?
How does it work?

It works automatically, and updates when you update anything.

I don't see this specific file in my root directly. The sitemap I generated is in my root, but not one with an ashx. Why don't I have one and how can I get one?

Thank you

It is system generated and not directly accessible. You can't edit it, configure it, or look at it in the file manager.

It is accessed at www.yourdomain.com/xml-sitemap.ashx

All sites on version 7.x have it.

Hi Dave,

You might want to take a peek at my .ashx sitemap. This auto sitemap has only a fraction of the url's listed. Example: I have 817 products, yet the .ashx sitemap shows just 628 urls. Why is there such a huge differance in numbers?

I took a peek at that from this thread: http://forums.networksolutions.com/e-comme...ashx-t4487.html

and was looking into it but lost track. I'll look into it more. There are URLs that are excluded from the sitemap for various reasons, but I don't have the list and the reasons for exclusion at this time. I emailed myself the other post as a reminder.

Thanks, I searched for that post but for some reason couldn't locate it. I am showing over 900 URLs for the sitemap that I created using the gsitecrawler. I manually scroll through the gsite sitemap to ensure there are no links that are not products, categories or seo pages. I do not use urls such as search by price, email friend, etc.

One other thing, we have loaded 10 or so new items in the past week, yet I dont see the number of urls on the .ashx sitemap increasing.

I use Gsitecrawler as well. I have about 1500 products but when I run gsitecrawler I get approximately 8500 URL's? I guess it includes every image URL as well as some html url's from File manager that came over from the old Monster Commerce version.

I took a look at the auto generated ashx sitemap, how do you know how many URL's there are when looking at that report?

Also, someone said earlier in this thread that you can submit multiple sitemaps to google webmaster tools for the same website? is this true?

Msuz.

You submit it to google and it tells you the count



Yes, google only allows 10mb sitemaps, but it allows you to have multiple 10mb sitemaps. So if your autogenerated sitemap comes out to be over 10mb you can't use it as we don't currently split it into 10mb chunks.

Msuz..

I dont use the picture urls in my sitemap...

Just wondering are you importing your robot.txt file to the crawler?

Wacky,

Yes I am using the robot.txt file.....which is weird. Maybe we should compare our robots files?

In fact, I have to eliminate several thousand URL on top of that, I get the ?previous and ?next url's as well as various $$$ over/ under URL's.......

Msuz.

I have found that sometimes you have to reset the robot.txt file in the crawler. You and find the button to do this at the bottom of the page where it list your robots.txt file. Sometimes you will get an error message that the crawler is busy, when its not. Stop the crawler, delete any current crawl info, import the robot.txt file and re-crawl..

Feel free to check out/copy my robot.txt file....

Have you had any luck with the images in your sitemap?

Wacky,

I checked your robots file to mine and found the foll:

THINGS YOU HAVE AND I DON'T:

Disallow: /help.aspx
Disallow: /login.aspx
Disallow: /search.aspx (you have it twice, I have it once)

You also have a link to your sitemap.xml

THINGS I HAVE THAT YOU DON'T:

Disallow: /Admin/
Disallow: /affiliatewiz/
Disallow: /eproducts/
Disallow: /themes/

Disallow: /custom.css.aspx
Disallow: /images/view.aspx

Let me know your thoughts? Maybe someone at netsol can comment on the above, as to what we should vs shouldn't have? I personally have never edited the file, so it's exactly as netsol setup...

Thanks
Msuz.

I think I need to add some of the items that you have listed since the affiliatewiz is new. I guess every site is different in what the owner wants indexed vs not indexed.

I had to put the search.aspx in twice since the bots would not or did not follow the command and I was getting search pages indexed.

admin affiliatewiz and eproducts were migrated from your 4x site. They can be removed as they don't exist on the 7x site, but they aren't hurting anything because they don't exist on the 7x site. Nothing you have will really hurt anything as nothing you are blocking has much seo value and some things you are blocking don't exist anyway. Search.aspx we added as we had some customers who were linking to searches on every product page and it was causing their site performance to degrade when indexed by google as it would do thousands of searches at once, so I would suggest leaving that blocked unless you know you aren't linking to searches anywhere.

Dave,
Taking my robots file into account, gsite crawler indexes the following URL types that cause me to have 1000's of URLS in my sitemap, including the following types:

http://www.nameofmysite.com/search.aspx?ma...amp;category=10 (I get tons of these low-high urls)

http://www.nameofmysite.com/search.aspx?ma...2&log=false
(still getting these search ones? even thought I have Disallow Search in my robots file)

http://www.nameofmysite.com/productorcateg...ame.aspx?page=3 ( I get every page 2, 3, 4, 5 etc....)

Also getting every product three times, because of the ?next and ?previous page links....

Can you advise on some things to add to the robots file to cut this down? Also do I need to put Disallow: /Search in twice as it seems to still pickup some search.aspx

Thanks
Msuz.

I've never had much luck with gsitecrawler. That being said, there is a disallow section in gsitecrawler, you should probably use that instead of relying on it being able to parse robots.txt.

Disallowing something multiple times in robots.txt has the same effect as disallowing it once.

Can we use the same generator for yahoo as well?
xml-sitemap.ashx