Article Preview
Top1. Portal Sites And Traditional Web Search Engines
WebSphere Portal allows people to interact with applications, processes, other people, documents and content in a personalized and role-based fashion. In a day and age when information overload has become commonplace, Portal allows context-relevant resources to be presented via common Web browsers, placing what people need to complete the job at hand at their fingertips, regardless of where they are.
Figure 1 shows the range of applications and services that integrate with Portal, as well as how that information and content is visualized to the end user.
Figure 1. Portal and application integration overview
Figure 1 also shows on a very high level how a search engine can integrate with the Portal:
- 1.
Traditionally a regular web crawler starts with the homepage and then follows all links to capture all information provided by the portal site. What options are available to optimize this process will be described in this article. We will also discuss the restrictions and limitations. as well as options to resolve some of these
- 2.
We will show how a new crawling technique can be enabled on the application level, which no longer requires the crawler to communicate and explore applications and repositories (pull technique). The application will rather send the crawler on request the list of what information the crawler is requested to fetch and process (push technique). Technical details as well as advantages of doing so foremost in a Portal environment is also discussed in more depth throughout this article
1.1 Content Centric Portal - Just Like “Any Other Website”
The simplest case to look at first is a Portal site which delivers static content through its pages and portlets. Portal provides a rich user experience in terms of consistent navigation, appearance and information delivery.
Crawling such a Portal site is a matter of pointing the crawler either at the homepage, a sitemap, or a site-directory. The crawler will then identify and record all links to referenced pages of that site, fetch their content and then send it for final processing to the indexing service. And once stored in the index, the user will then be able to perform his search requests.
That said: this is “business as usual” for the web-crawler, no real difference compared to crawling a standard web-server delivering more or less static HTML content.