16 replies to this topic

[Joseph]

https://www.google.c.../sitemaps/login

Google Sitemaps (BETA)

=================================

Quote

What is Google Sitemaps?
Google Sitemaps is an experiment in web crawling. Using Sitemaps to inform and direct our crawlers, we hope to expand our coverage of the web and improve the time to inclusion in our index. By placing a Sitemap-formatted file on your webserver, you enable our crawlers to find out what pages are present and which have recently changed, and to crawl your site accordingly.

Basically, the two steps to participating in Google Sitemaps are:

Generate a Sitemap in the correct format using Sitemap Generator.
Update your Sitemap when you make changes to your site.

Who can use Google Sitemaps?
Google Sitemaps is intended for all web site owners, from those with a single web page to companies with millions of ever-changing pages. If any of the following are true, then you may be especially interested in Google Sitemaps:

You want Google to crawl more of your web pages.
You want to be able to tell Google when content on your site changes.

How much does it cost?

Absolutely nothing. Google has never charged for placement in our search results, and we don't have any plans to do so.

Why is Google doing this?

In alignment with Google's mission to organize the world's information and make it universally accessible, this collaborative crawling system will allow our crawlers to optimize the usefulness of Google's index for users by improving its coverage and freshness.

I am really excited about this development!!! There is a lot of information available on the website so I won't repeat it all. Basically you have a choice of either using a Python script to create an XML sitemap for your website (which will reside on your own web server to be accessed by Google) OR you can submit a text file of links to achieve roughly the same thing.

I am currently experimenting with just the text file option. I just submitted the file and am now waiting to see if Google will come to my site to read the sitemap file and list all my pages!

I will let you all know if it works!

Cheers, Jo

[glen]

Hi Joseph,

As far as I can tell, submitting a txt file will not work. It has to be xml.

So you are best creating a XML as follows (see below my sitemap xml file which was just submitted and came up "OK"). I called it sitemap.xml. Make sure you save it as UTF-8 (this is easily done in Notepad if you are using windows).

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.google.com/schemas/sitemap/0.84">
<url>
<loc>http://www.adsenseforums.net/index.php</loc></url><url>
<loc>http://www.adsenseforums.net/index.php?showforum=2</loc></url><url>
<loc>http://www.adsenseforums.net/index.php?showforum=3</loc></url><url>
<loc>http://www.adsenseforums.net/index.php?showforum=4</loc></url><url>
<loc>http://www.adsenseforums.net/index.php?showforum=5</loc></url><url>
<loc>http://www.adsenseforums.net/index.php?showforum=6</loc></url><url>
<loc>http://www.adsenseforums.net/index.php?showforum=7</loc></url><url>
<loc>http://www.adsenseforums.net/index.php?showforum=8</loc></url><url>
<loc>http://www.adsenseforums.net/index.php?showforum=9</loc></url><url>
<loc>http://www.adsenseforums.net/index.php?showforum=10</loc></url><url>
<loc>http://www.adsenseforums.net/index.php?showforum=11</loc></url><url>
<loc>http://www.adsenseforums.net/index.php?showforum=12</loc></url><url>
<loc>http://www.adsenseforums.net/index.php?showforum=13</loc></url><url>
<loc>http://www.adsenseforums.net/index.php?showforum=14</loc></url><url>
<loc>http://www.adsenseforums.net/index.php?showforum=15</loc></url><url>
<loc>http://www.adsenseforums.net/index.php?showforum=16</loc></url><url>
<loc>http://www.adsenseforums.net/index.php?showforum=17</loc></url><url>
<loc>http://www.adsenseforums.net/index.php?showtopic=1</loc></url><url>
<loc>http://www.adsenseforums.net/index.php?showtopic=2</loc></url><url>
<loc>http://www.adsenseforums.net/index.php?showtopic=3</loc></url><url>
<loc>http://www.adsenseforums.net/index.php?showtopic=4</loc></url><url>
<loc>http://www.adsenseforums.net/index.php?showtopic=5</loc></url><url>
<loc>http://www.adsenseforums.net/index.php?showtopic=6</loc></url><url>
<loc>http://www.adsenseforums.net/index.php?showtopic=7</loc></url><url>
<loc>http://www.adsenseforums.net/index.php?showtopic=8</loc></url><url>
<loc>http://www.adsenseforums.net/index.php?showtopic=9</loc></url><url>
<loc>http://www.adsenseforums.net/index.php?showtopic=10</loc></url><url>
<loc>http://www.adsenseforums.net/index.php?showtopic=11</loc></url><url>
<loc>http://www.adsenseforums.net/index.php?showtopic=12</loc></url><url>
<loc>http://www.adsenseforums.net/index.php?showtopic=13</loc></url><url>
<loc>http://www.adsenseforums.net/index.php?showtopic=14</loc></url><url>
<loc>http://www.adsenseforums.net/index.php?showtopic=15</loc></url><url>
<loc>http://www.adsenseforums.net/index.php?showtopic=16</loc></url><url>
<loc>http://www.adsenseforums.net/index.php?showtopic=17</loc></url><url>
<loc>http://www.adsenseforums.net/index.php?showtopic=18</loc></url><url>
<loc>http://www.adsenseforums.net/index.php?showtopic=19</loc></url><url>
<loc>http://www.adsenseforums.net/index.php?showtopic=20</loc></url><url>
<loc>http://www.adsenseforums.net/index.php?showtopic=21</loc></url><url>
<loc>http://www.adsenseforums.net/index.php?showtopic=22</loc></url><url>
<loc>http://www.adsenseforums.net/index.php?showtopic=23</loc></url><url>
<loc>http://www.adsenseforums.net/index.php?showtopic=24</loc></url><url>
<loc>http://www.adsenseforums.net/index.php?showtopic=25</loc></url><url>
<loc>http://www.adsenseforums.net/index.php?showtopic=26</loc></url><url>
<loc>http://www.adsenseforums.net/index.php?showtopic=27</loc></url><url>
<loc>http://www.adsenseforums.net/index.php?showtopic=28</loc></url><url>
<loc>http://www.adsenseforums.net/index.php?showtopic=29</loc></url><url>
<loc>http://www.adsenseforums.net/index.php?showtopic=30</loc></url><url>
<loc>http://www.adsenseforums.net/index.php?showtopic=31</loc></url><url>
<loc>http://www.adsenseforums.net/index.php?showtopic=32</loc></url><url>
<loc>http://www.adsenseforums.net/index.php?showtopic=33</loc></url><url>
<loc>http://www.adsenseforums.net/index.php?showtopic=34</loc></url><url>
<loc>http://www.adsenseforums.net/lofiversion/index.php/f2.html</loc></url><url>
<loc>http://www.adsenseforums.net/lofiversion/index.php/f3.html</loc></url><url>
<loc>http://www.adsenseforums.net/lofiversion/index.php/f4.html</loc></url><url>
<loc>http://www.adsenseforums.net/lofiversion/index.php/f5.html</loc></url><url>
<loc>http://www.adsenseforums.net/lofiversion/index.php/f6.html</loc></url><url>
<loc>http://www.adsenseforums.net/lofiversion/index.php/f7.html</loc></url><url>
<loc>http://www.adsenseforums.net/lofiversion/index.php/f10.html</loc></url><url>
<loc>http://www.adsenseforums.net/lofiversion/index.php/f12.html</loc></url><url>
<loc>http://www.adsenseforums.net/lofiversion/index.php/f14.html</loc></url><url>
<loc>http://www.adsenseforums.net/lofiversion/index.php/f17.html</loc></url><url>
<loc>http://www.adsenseforums.net/lofiversion/index.php/f9.html</loc></url><url>
<loc>http://www.adsenseforums.net/lofiversion/index.php/f8.html</loc></url><url>
<loc>http://www.adsenseforums.net/lofiversion/index.php/f11.html</loc></url><url>
<loc>http://www.adsenseforums.net/lofiversion/index.php/f13.html</loc></url><url>
<loc>http://www.adsenseforums.net/lofiversion/index.php/f15.html</loc></url><url>
<loc>http://www.adsenseforums.net/lofiversion/index.php/f16.html</loc></url>
</urlset>

I should say that this is not every url on my site but I whipped this up fairly quickly and submitted it since AdsenseForums.net is yet to be indexed in the Google Search Engine and I thought this might be a good opportunity to get listed! I will let you know if that ever happens!

Cheers, Glen

[glen]

Hi guys,

I just posted the following message in the digitalpoint forums (edited slightly)-

http://forums.digita...ead.php?t=17891

=============================================

I have just made a Google sitemap using Microsoft Excel. I will describe it here if anyone is interested. This method is useful if you have a forum. In my case I use Invision Power Board and the topics have the following structure -

http://www.domain.ne...php?showtopic=1
http://www.domain.ne...php?showtopic=2
http://www.domain.ne...php?showtopic=3
etc...

To create the hundreds of required url's in a sitemap format, I created 3 columns in excel.

Column 1 contains

<loc>[url]http://www.domain.net/forums/index.php?showtopic=[/url]

(repeating in every row)

Column 2 contains -

1
2
3
etc...

(the numbers incrementally increasing on each line - you achieve this by putting in a 1, 2..then selecting them and dragging the controller in the bottom right corner)

Column 3 contains

</loc></url><url>

Then in column 4 I combine these using the "concatenate" function. The result will look as follows -

<loc>http://www.domain.net/forums/index.php?showtopic=1</loc></url><url>

You can then cut and paste them into a file as follows -

Quote

http://www.domain.ne...php?showtopic=1
http://www.domain.ne...php?showtopic=2
http://www.domain.ne...php?showtopic=3

Don't forget to delete the red (see above) entry from the end of the last url line!

Now you have a valid XML file to submit.

Here is a sample of what the excel spreadsheet would look like when you start out.

 sample_start.xls   57K   326 downloads

The following images use column 2 as an example on how to auto-populate each column. Column 2 is a bit more complicated that the others because you need to create a "pattern" first so that Excel knows you want to create a sequences of numbers (1, 2, 3, etc). In the following image you see we type "1" in the first row and "2" in second row. This is all we need to establish a pattern of 1,2,3,4 etc.

We need to select these 2 cells as follows -



Next (see image below) we drag downwards. As you can see, Excel knows that we want it to add in the number 3, 4, 5 etc.



For the other columns, we only complete one row before we drag downwards because we just want to duplicate row 1 in every other row (i.e. there is no need to establish any pattern)

In case you don't spot it in the above excel file, the concatenate formula is in cell D1 and it looks like this -

=CONCATENATE(A1,B1,C1)

Note that we only enter the formula in row 1 and then we can drag downward to concatenate all the rows we need.

Cheers
===================================================

NEWS FLASH (Updated 13 June 2005)

EDIT - Here is a little Flash Training Video I put together demonstrating how to make a XML Google Sitmemap manually with Excel and Notepad. I hope you find this useful. The file is about 600 kB. I start off using the excel file attached above (sample_start.xls) and then auto-generate however many rows of data as are required. In this demo I just make 10 rows for simplicity but you could just as easily make 10,000 rows. This is just an example of the method. Please make sure that the links you use are right for YOUR FORUM, whether it be IPB, vBulletin, phpBB etc.

Also, please note that you can use this method to generate both forum links and topic links! You will probably want to do both...certainly topic links are the most important to have indexed.

For IPB a forum link looks like this -

Quote

http://www.adsenseforums.net/index.php?showforum=3

and a topic link looks like this -

Quote

http://www.adsenseforums.net/index.php?showtopic=5

Here is the VIDEO TUTORIAL (you will need flash player installed on your computer to view it)

[Falcon]

Hi guys,

Here is an alternative method. This method is perhaps better if you know that you have deleted a fair few topics in your forum.

The Falcon Method

Step 1 - Using phpMyAdmin or whatever database program you use, and open up your forum database.

Step 2 - Find the table that stores topics (for Invision Power Board it will probably be called ibf_topics).

Step 3 - Export that table called ibf_topic as a CSV Excel File.

Step 4 - Open up that CSV file in Excel and cut and paste the column with all the topic numbers into the Excel spreadsheet Glen described above. In other words, instead of having a column filled automatically with 1,2,3 etc you would have the "real" topic numbers pasted into that column.

Hope this is clear enough. Happy to explain further. This is perhaps more work than Glen's method but will guarantee that you do not send dud URLs to Google in your XML Sitemap.

Falcon

[glen]

Glad you found the information helpful sarah and welcome to the forum.

I just received word that Google has been putting together a page on useful third party tools for developing your Google sitemaps. The list of 3rd party tools can be found here -

http://code.google.c...thirdparty.html

All the best, Glen

[glen]

Thank you Burnie for setting out all that info. Much appreciated,

Cheers, Glen

[glen]

For those of you using Invision Power Board, there is a new mod called Google Sitemap Generator 1.0 that can automatically generate google sitemap xml files for your forum topics ONLY -

http://mods.invision...ndex.php/f/4911

The good thing about this mod is that it pulls the topic ID's directly out of the table which achieves what Falcon set out to do above which was to ensure each link is a valid link.

However, the auto-generated file is automatically overwriten COMPLETELY ever time it is created. So if you had a number of other links in that file (e.g. for static pages outside of your forum) then they would be deleted when the xml file was re-generated.

However, there is a simple solution to this - SIMPLE - have more than one sitemap. One file will contain your forum links and the other will contain the links to your static pages. If you want to have more than one sitemap, then you will need a sitemap index file.

DON'T get confused here! A sitemap index file and a sitemap file are NOT the same thing!!!

A sitemap index file simply tells Google spiders that you have more than one sitemap file and it tells the spider where to find them!

To learn more about sitemap index files check out the following link -

https://www.google.c...n/protocol.html

Here is a modified extract of useful information that explains the relationship between sitemap files and sitemap indexes -

Quote

Providing Multiple Sitemap Files 

.....If you do provide multiple Sitemaps, you must list them in a Sitemap index file. Sitemap index files may not list more than 1,000 Sitemaps. Your Sitemap index file could be named sitemap_index.xml.

...

Sample XML Sitemap Index

The following example shows a Sitemap index in XML format. The Sitemap index lists two Sitemaps:



      http://www.mysite.com/sitemap1.xml


      http://www.mysite.com/sitemap2.xml

[glen]

Here is a link to an elegant Excel-based sitemap generation method -

http://www.better-bu...p-generator.htm

All you need to use this method is a valid lists of urls. The spreadsheet formulas will do the rest for you.

Cheers, Glen

[glen]

Cool thanks bobafind

[glen]

welcome john,

sounds like a very useful tool. how long would it take to do a scan of a large forum? are you planning on making a version that does not required root access to the server?

personally i am using shared hosting and so i think i cannot use this tool. is that correct?

cheers, glen

[glen]

sounds good, thanks John,

cheers, glen