Some hours ago, Six Apart announced that the Movable Type User Manual had been improved and extended. As MTLookup allows including the User Manual in search queries, I refreshed the MTLookup database immediately.

Both the Movable Type User Manual and the Movable Type Knowledge Base have been spidered completely. The information about these two websites has been updated in the MTLookup database.

Update: After having updated the MTLookup database, I did some reading. Yes, the Movable Type documentation is really good. Congratulations! And it is a perfect idea to create a manual with the help of a Movable Type itsself. My next user manual will be created with Movable Type, too.

Update Process

I have already been asked by private email about the way MTLookup updates its database. Maybe I should elaborate on that subject here.

Currently, MTLookup knows 26 websites with more than 2400 pages.

The MTLookupBot, which is the central component for spidering the websites, regularly tries to find out, whether new pages have been published. With most websites this is an easy job, as new articles are usually mentioned on the startpage. MTLookupBot looks at each website's startpage once a day. So finding a new page and storing the reference into the MTLookup database should not last longer than 24 hours.

However, if an author updates an old article, MTLookup will not find out immediately. There is another schedule, which is responsible for that task. It refreshes all pages from all websites - maybe once a week or even less often. This is not automated. I start this process manually if I think that it is neccessary.

The Movable Type User Manual is a special website. New entries are not published on the startpage. So with this website, I start a full scan manually.

The spidering process does not put heavy load on the server. There is a noticable intervall between two pages that are read from a server. The daily scan, which only touches the startpage and the new pages, is neglectible. And even the full scan, which is spread over several hours, is no special burden for the server.

mgs | 10/05/2005

Feedback is welcome!

What do you think about this entry? Was it interesting or boring? I would like to hear your comments. If the text was helpful, please consider setting a link to http://www.movable-type-weblog.com/.

No spam please!

For protecting this weblog I have installed the MT-Approval Plugin. You have to view a new comment in preview mode, before it is saved on the server. Moreover, I will view your comment manually, before it is published. You can find more information on the subject in the entry Weblog Spamming Basics.

With an active TypeKey session, your comment will be published immediately.

TypeKey has temporarily been disabled at this location. Please create your comment without using TypeKey or log in from the preview dialog.