LOK looksmart limited

grub joins forces with looksmart

  1. MJR
    1,085 Posts.
    Grub Joins Forces with Looksmart
    Posted: Mar 17, 2003
    Section: Site News
    Posted By: Kord Campbell
    Grub has joined forces with LookSmart to take community web crawling to the next level. Our mission to eventually crawl and assemble the latest state information for every document on the Internet remains unchanged.

    This is a great opportunity for Grub to grow our infrastructure, expand our community of users, and to begin putting the data that is crawled to use in updating the WiseNut and LookSmart search engines. In addition, we also plan to continue to make the crawl data available to the Grub community of users more directly through an XML API.

    As most of the users that run Grub already know, there still are bugs that need to be fixed in the client, and in the system. We have been working overtime trying to get these resolved by making the infrastructure more stable, the client signup and install smoothed out, and the interface features like local crawling and the XML interface working correctly. We are focused, over the next several weeks, on addressing the majority of these bugs/issues.

    To directly address top questions regarding Local Crawling, open access to the database, and robots.txt behavior, we have assembled a series of pages to cover these issues. We will continue to provide information about features and bug fixes as they become available....

    Local Crawling Support - The current feature allows you to crawl a single host. We are working on multi-host support, and streamlining the interface in the client. Expect a link on the new site soon, with a new page up that deals with multi-host local crawling. In the meantime, use the current Local Crawling page.

    Respecting robots.txt Files - We currently have the host table built, but are in progress of updating the robots files. You can use Robots Refresh page to force a visit by us to your site.

    Database Access - a beta version of the XML interface is operational. More features will be added, including a web page interface to the API and the XMLish errors will be fixed.

    Stay tuned for more announcements from us, and as always, thank you for running the client!

    Team Grub

arrow-down-2 Created with Sketch. arrow-down-2 Created with Sketch.