Since its creation in 1996, Cybion has kept a permanent watch over new and emerging data collection tools. Experts in using information from the Internet, we have looked for, without real success, a data collection tool that would be powerful and flexible enough to adapt to all the specificities of Internet sources. Observing that no tool fully met our needs, we decided to develop our own flexible and highly customizable solution, Cybion Eye, that is well adapted to the diversity of Internet documents and possesses a powerful control system. Cybion Eye makes it possible to get around the main limits of search engines that only index 10-15% of the visible Web, with at least a 3 week delay from the publication date. Our project received support from the Ministry of Economics, Finance and Industry in 2002.
 CYBION EYE: A POWERFUL AND INTELLIGENT DATA COLLECTION TOOL
The current awareness cycle is composed of 4 successive phases:
 Identification of sources
Data collection
Analysis of information
Dissemination of results
Many monitoring tools automate one or another of these phases. Some are specialized in data analysis (indexing or text mining tools), others in the representation of data (cartographic or profiling tools).
Two types of collection tool are available on the market:
1) Crawlers, that literally suck up Web pages. They move around the Web by using the hyperlinks that connect pages between themselves. This type of browsing (and therefore data collection) remains relatively haphazard because these tools limit themselves to the visible Web without taking into account the other Internet spheres (e-mail, newsgroups, chat rooms…). What's more, these crawlers do no more than capture and copy unstructured pages.
Using this type of crawler to feed information into a current awareness system means having a random and unstructured corpus of documents, that is difficult to use.
2) Metasearch engines that use general search engines as sources of information. This technique is not satisfactory for two reasons: on one hand, current search engines are not able to cover the totality of pages published on the Web, and on the other hand the growth in the number of Web publications means that these tools can never be entirely up to date.
Using metasearch engines to carry out strategic watch activities means searching on just 10% of published information, that is at best 2 or 3 weeks old.
 Cybion Eye: a data collection solution
With Cybion Eye, it is at last possible to teach the system, for example, how to go twice a day to the pages of a specialized magazine in order to detect new articles, and extract their date, title and body text.
All the restructured documents are then indexed in a database. Data collection is not limited to only Web sources: any document published and accessible via the Internet can constitute a source (databases, informal spheres…)
he data necessary for collecting articles includes, amongst others:
The address of the source. This can be a Web address or an e-mail account, or even a database access
The publication frequency. In order to assure optimal updating of the article database, each source is crawled depending on the frequency of updates
Rules of navigation for finding articles. These rules describe the different steps that lead from the address of the source to the articles..
The method for detecting redundancies. Many techniques can be used to detect an article that is already present in the database. Checks can be carried out on the URL of a Web source, the title, the date… For each source, the most appropriate method is chosen.
The method for extracting fields. Cybion Eye is the most innovative tool on the market as far as reformatting documents from any source is concerned. Thanks to the page templates programmed for each source, Cybion Eye can clean up the articles that it finds and identify the useful fields (date, author, title, text body…).
 Cybion Eye: selection and filtering of articles
With Cybion Eye, Internet sources (press, Web, mailing lists, newsgroups…) become a source of structured information for the company, that can be used in a current awareness system.
Once the information has been collected, two means of communication are available: push and pull. Each time the database is refreshed, the new articles are sent to their recipients. Documents can either be exported by e-mail or in XML format.
In addition, a graphically customized Web interface, with password-protected access, brings together all the articles collected in a database. These articles are indexed by a search engine, that supports classic Boolean operators.
It is possible at any moment, with Cybion Eye, to modify the collection and monitoring profiles: adding a new source, changing the frequency of updates, or modifying article selection criteria (keywords) for one of the profiles.
 Cybion Eye: a technique
All the documents are stored then indexed in a database. Each user of the Cybion Eye service can save their search profiles in order to quickly identify the most relevant articles.
For the administrator, a monitoring system is delivered with the tool and enables the surveillance of its actions. A collection history is created so that the administrator is alerted when a source has not published any articles (probably because the format of the source or of the presentation has been changed) or on the contrary, when an unusual number of articles are published (the redundancy rules may need to be revised). As the sources that we monitor are very different, it is necessary to observe their evolution and adapt the configuration of the 'collectors' when modifications are made.
|
