This master’s thesis, conducted at Sectra Medical Systems AB, investigates the feasibility of integrating a search platform, built on modern search technology, into the complex architecture of existing products. This was done through the implementation and integration of a search platform prototype, called Sectra Enterprise Search. It was built upon the search engine library Lucene.NET, written in C# for the Microsoft .NET Framework. Lucene.NET originates from the Java library Lucene, which is highly regarded and widely used for similar purposes. During the development process a lot of requirements for the search platform were identified, including high availability, scalability and maintainability. Besides full text search for information in a variety of data sources, desirable features include autocompletion and highlighting. Sectra Enterprise Search was successfully integrated within the architecture of existing products. The architecture of the prototype consists of multiple layers, with the search engine functionality at the very bottom and a web service handling all incoming request at the top. To sum up, integrating a search platform based on modern search technology into the architecture of existing products infers full control of deployment, users searching in a more intuitive manner and reasonable search response times.

The creation of an Enterprise Search system involves many challenges that are not present in Web search. Searching a corporate collection is influenced both by the structure of the data present in the collection and by the policies of the corporation. These structures and policies may differ from corporation to corporation, and from collection to collection. In particular, an Enterprise Search system must take a document's genre into account. Examples of document genre within a corporate collection might include FAQs, white papers, technical reports, memos, emails and chat messages. Depending on an individual's current work task, it might be appropriate to give one genre a greater weight than another during the processing of a search request. Moreover, this weighting may change as the individual's work task changes. The work presented in this thesis adapts the Okapi BM25 scoring function to weight term frequency based on the relevance of a document genre to a work task. The method utilizes two user-provided resources, relevance judgments and clickthrough data, to estimate a realistic weight for each task-genre relationship. Using this approach, the method matches the purpose of each user search request with the purpose of each document. Therefore, the proper documents are returned to the user and her/his need can be fulfilled. The method has been incorporated into a prototype search engine, X-site, currently deployed on a corporate intranet. X-Site is a contextual search engine that uses the relationships between work tasks and document genres to improve search precision for software engineers. The system provides a customized and user-controlled means of refining search results to suit the task context of a user. Through X-Site, each employee can make a single search request and has access to documents from the Internet, a corporate intranet, and Lotus Notes databases.

The creation of an Enterprise Search system involves many challenges that are not present in Web search. Searching a corporate collection is influenced both by the structure of the data present in the collection and by the policies of the corporation. These structures and policies may differ from corporation to corporation, and from collection to collection. In particular, an Enterprise Search system must take a document's genre into account. Examples of document genre within a corporate collection might include FAQs, white papers, technical reports, memos, emails and chat messages. Depending on an individual's current work task, it might be appropriate to give one genre a greater weight than another during the processing of a search request. Moreover, this weighting may change as the individual's work task changes. The work presented in this thesis adapts the Okapi BM25 scoring function to weight term frequency based on the relevance of a document genre to a work task. The method utilizes two user-provided resources, relevance judgments and clickthrough data, to estimate a realistic weight for each task-genre relationship. Using this approach, the method matches the purpose of each user search request with the purpose of each document. Therefore, the proper documents are returned to the user and her/his need can be fulfilled. The method has been incorporated into a prototype search engine, X-site, currently deployed on a corporate intranet. X-Site is a contextual search engine that uses the relationships between work tasks and document genres to improve search precision for software engineers. The system provides a customized and user-controlled means of refining search results to suit the task context of a user. Through X-Site, each employee can make a single search request and has access to documents from the Internet, a corporate intranet, and Lotus Notes databases.

This thesis discusses relevance feedback including implicit parameters, explicit parameters and user query and how they could be used to build a recommender system to enhance the search performance in the enterprise. It presents an approach for the development of an adaptive fuzzy logic based recommender system for enterprise search. The system is designed to recommend documents and people based on the user query in a task specific search environment. The proposed approach provides a new mechanism for constructing and integrating a task, user and document profiles into a unified index thorough the use of relevance feedback and fuzzy rule based summarisation. The three profiles are fuzzy based and are created using the captured relevance feedback. In the task profile, each task was modelled as a sequence of weighted terms which were used by the users to complete the task. In the user profile, the user was modelled as a sequence of weighted terms which were used to search for the required information. In the document profile the document was modelled as a group of weighted terms which were used by the users to retrieve the document. Fuzzy sets and rules were used to calculate the term weight based on the term frequency in the user queries. An empirical research was carried out to capture the relevance feedback from 35 users on 20 predefined simulated enterprise search tasks and to investigate the correlation between the implicit and explicit relevance feedback. Based on the results, an adaptive linear predictive model was developed to estimate the document relevancy from the implicit feedback parameters. The predicted document relevancy was then used to train the fuzzy system which created and integrated the three profiles, as briefly described above. The captured data set was used to develop and train the fuzzy system. The proposed system achieved 89% accuracy performance classifying the relevant documents. With regard to the implementation, Apache Sorl, Apache Tikka, Oracle 11g and Java were used to develop a prototype system. The overall retrieval accuracy performance of the proposed system was tested by carrying out a comparative retrieval accuracy performance evaluation based on Precision (P), Recall (R) and ranking analysis. The values of P and R of the proposed system were compared with two other systems being the standard inverted index based Solr system and the semantic indexing based lucid system. The proposed system enhanced the value of P significantly where the average of P value has been increased from 0.00428 to 0.064 as compared with the standard Sorl and from 0.0298 to 0.064 compared with Lucid. In other words, the proposed system has managed to decrease the number of irrelevant documents in the search result which means that the ability of the system to show the relevant document has been enhanced. The proposed system has also enhanced the value of R. The average value of R has been increased significantly (doubling) from 0.436 to 0.828 as compared with the standard Solr and from 0.76804 to 0.828 as compared with Lucid. This means that the ability of the system to retrieve the relevant document has also been enhanced. Furthermore the ability of the system to rank higher the relevant documents has been improved as compared with the other two systems.

As organizations grow larger, so does the information base of the organization, and the need to findimportant information increases along with it. However, many organizations are neglecting theimportance of information. Enterprise search lets users search for information in multiple informationrepositories from one single application, without having to know where the information resides. The focalfirm of this research, Findwise AB, is a Swedish IT consultancy firm specialized in delivering searchsolutions to its customers. It has been shown that not many organizations have a strategy for their internalwork with enterprise search. Therefore, Findwise wants to be able to deliver enterprise search strategies toits customers. Such a strategy would facilitate the customers’ development of their internal enterprisesearch management. However, this requires the ability to define an organization’s current and future statewith regard to enterprise search management, something that, up until now has not been possible.Therefore, the purpose of this study was to develop an enterprise search management maturity (ESMM)model, which Findwise’s consultants can use for assessing an organization’s current state, as well aspossible future state within enterprise search management. Through an extensive literature study, incombination with an empirical study consisting of a questionnaire and semi-structured interviews, theESMM model was developed. The themes of the identified maturity levels of enterprise searchmanagement are; knowledge, management, priority, standardization, and sustainability. It is concludedthat the parameters to evaluate on each of these maturity levels are the three parameters information,technology, and user. Each of these parameters consists of three specific factors to evaluate, providingguidelines for what to look at in an assessment. This will enable Findwise to define an organization’scurrent and future state within enterprise search management, as well as enable organizations to gain themost benefit out of their enterprise search solution. / När organisationer växer ökar även deras interna informationsmängd. I takt med detta, ökar även behovetav att finna relevant information inom organisationen. Tyvärr är detta något som ofta försummas avdagens organisationer. Enterprise search är en sökapplikation som möjliggör sökning i flera olikainformationskällor från endast en applikation. Detta innebär att istället för att söka igenom olikainformationskällor via separata sökfunktioner, behöver användare av en enterprise search-lösning endastgöra en enda sökning. Findwise AB är ett svenskt IT-konsultbolag som är specialiserade på att utvecklasådana enterprise search-lösningar. Ett stort problem är att organisationer ofta saknar ett konkretarbetssätt och en strategi för enterprise search. Detta leder till att det blir mycket svårt att dels realiseravärdet av söklösningen, men även till svårigheter med att definiera hur det interna arbetet kringsöklösningen bör se ut. På grund av detta vill Findwise kunna erbjuda sina kunder enterprise searchstrategier.En sådan strategi skulle underlätta kunders interna utveckling av styrprocesser för enterprisesearch. Detta kräver dock förmågan att definiera en organisations nuvarande och framtida tillstånd medavseende på styrningen av enterprise search, något som fram tills nu ej har varit möjligt. Således varsyftet med detta arbete att ta fram en mognadsmodell för utvärdering av styrprocesser för enterprisesearch. Denna mognadsmodell ska bistå Findwise-konsulter med ett verktyg för att definiera vilkenmognadsnivå en organisation befinner sig på i dagsläget, men även se var man vill befinna sig iframtiden. Från denna utvärdering kan sedan de aktiviteter identifieras, som behöver utföras för att uppnåett framtida mål och skapa en strategi för enterprise search. Denna rapport beskriver utvecklandet av ensådan mognadsmodell för styrning av enterprise search. Modellen utvecklades genom en kvalitativfallstudie på Findwise AB, där resultat från en enkät och semistrukturerade intervjuer utgör grunden förstudien. Fakta och diverse koncept har även grundats i teori. Resultatet blev en mognadsmodell beståendeav fem olika mognadsnivåer: kunskap, hantering, prioritering, standardisering och hållbarhet. Dessaanvänds för att utvärdera mognaden inom tre huvudområden av enterprise search: information, teknik ochanvändare. Inom varje område är tre olika faktorer identifierade, som bidrar med riktlinjer för vad manbör titta på vid en utvärdering av enterprise search. Modellen möjliggör för Findwise att kunna definieraen organisations nuvarande- och framtida tillstånd med avseende på styrning av enterpris search. Dessutom skulle en sådan modell möjliggöra för organisationer att få maximalt utbyte av sin enterprisesearch-lösning.

Die automatische Informationsextraktion (IE) aus unstrukturierten Texten ermöglicht völlig neue Wege, auf relevante Informationen zuzugreifen und deren Inhalte zu analysieren, die weit über bisherige Verfahren zur Stichwort-basierten Dokumentsuche hinausgehen. Die Entwicklung von Programmen zur Extraktion von maschinenlesbaren Daten aus Texten erfordert jedoch nach wie vor die Entwicklung von domänenspezifischen Extraktionsprogrammen. Insbesondere im Bereich der Enterprise Search (der Informationssuche im Unternehmensumfeld), in dem eine große Menge von heterogenen Dokumenttypen existiert, ist es oft notwendig ad-hoc Programm-module zur Extraktion von geschäftsrelevanten Entitäten zu entwickeln, die mit generischen Modulen in monolithischen IE-Systemen kombiniert werden. Dieser Umstand ist insbesondere kritisch, da potentiell für jeden einzelnen Anwendungsfall ein von Grund auf neues IE-System entwickelt werden muss. Die vorliegende Dissertation untersucht die effiziente Entwicklung und Ausführung von IE-Systemen im Kontext der Enterprise Search und effektive Methoden zur Ausnutzung bekannter strukturierter Daten im Unternehmenskontext für die Extraktion und Identifikation von geschäftsrelevanten Entitäten in Doku-menten. Grundlage der Arbeit ist eine neuartige Plattform zur Komposition von IE-Systemen auf Basis der Beschreibung des Datenflusses zwischen generischen und anwendungsspezifischen IE-Modulen. Die Plattform unterstützt insbesondere die Entwicklung und Wiederverwendung von generischen IE-Modulen und zeichnet sich durch eine höhere Flexibilität und Ausdrucksmächtigkeit im Vergleich zu vorherigen Methoden aus. Ein in der Dissertation entwickeltes Verfahren zur Dokumentverarbeitung interpretiert den Daten-austausch zwischen IE-Modulen als Datenströme und ermöglicht damit eine weitgehende Parallelisierung von einzelnen Modulen. Die autonome Ausführung der Module führt zu einer wesentlichen Beschleu-nigung der Verarbeitung von Einzeldokumenten und verbesserten Antwortzeiten, z. B. für Extraktions-dienste. Bisherige Ansätze untersuchen lediglich die Steigerung des durchschnittlichen Dokumenten-durchsatzes durch verteilte Ausführung von Instanzen eines IE-Systems. Die Informationsextraktion im Kontext der Enterprise Search unterscheidet sich z. B. von der Extraktion aus dem World Wide Web dadurch, dass in der Regel strukturierte Referenzdaten z. B. in Form von Unternehmensdatenbanken oder Terminologien zur Verfügung stehen, die oft auch die Beziehungen von Entitäten beschreiben. Entitäten im Unternehmensumfeld haben weiterhin bestimmte Charakteristiken: Eine Klasse von relevanten Entitäten folgt bestimmten Bildungsvorschriften, die nicht immer bekannt sind, auf die aber mit Hilfe von bekannten Beispielentitäten geschlossen werden kann, so dass unbekannte Entitäten extrahiert werden können. Die Bezeichner der anderen Klasse von Entitäten haben eher umschreibenden Charakter. Die korrespondierenden Umschreibungen in Texten können variieren, wodurch eine Identifikation derartiger Entitäten oft erschwert wird. Zur effizienteren Entwicklung von IE-Systemen wird in der Dissertation ein Verfahren untersucht, das alleine anhand von Beispielentitäten effektive Reguläre Ausdrücke zur Extraktion von unbekannten Entitäten erlernt und damit den manuellen Aufwand in derartigen Anwendungsfällen minimiert. Verschiedene Generalisierungs- und Spezialisierungsheuristiken erkennen Muster auf verschiedenen Abstraktionsebenen und schaffen dadurch einen Ausgleich zwischen Genauigkeit und Vollständigkeit bei der Extraktion. Bekannte Regellernverfahren im Bereich der Informationsextraktion unterstützen die beschriebenen Problemstellungen nicht, sondern benötigen einen (annotierten) Dokumentenkorpus. Eine Methode zur Identifikation von Entitäten, die durch Graph-strukturierte Referenzdaten vordefiniert sind, wird als dritter Schwerpunkt untersucht. Es werden Verfahren konzipiert, welche über einen exakten Zeichenkettenvergleich zwischen Text und Referenzdatensatz hinausgehen und Teilübereinstimmungen und Beziehungen zwischen Entitäten zur Identifikation und Disambiguierung heranziehen. Das in der Arbeit vorgestellte Verfahren ist bisherigen Ansätzen hinsichtlich der Genauigkeit und Vollständigkeit bei der Identifikation überlegen. / The automatic information extraction (IE) from unstructured texts enables new ways to access relevant information and analyze text contents, which goes beyond existing technologies for keyword-based search in document collections. However, the development of systems for extracting machine-readable data from text still requires the implementation of domain-specific extraction programs. In particular in the field of enterprise search (the retrieval of information in the enterprise settings), in which a large amount of heterogeneous document types exists, it is often necessary to develop ad-hoc program-modules and to combine them with generic program components to extract by business relevant entities. This is particularly critical, as potentially for each individual application a new IE system must be developed from scratch. In this work we examine efficient methods to develop and execute IE systems in the context of enterprise search and effective algorithms to exploit pre-existing structured data in the business context for the extraction and identification of business entities in documents. The basis of this work is a novel platform for composition of IE systems through the description of the data flow between generic and application-specific IE modules. The platform supports in particular the development and reuse of generic IE modules and is characterized by a higher flexibility as compared to previous methods. A technique developed in this work interprets the document processing as data stream between IE modules and thus enables an extensive parallelization of individual modules. The autonomous execution of each module allows for a significant runtime improvement for individual documents and thus improves response times, e.g. for extraction services. Previous parallelization approaches focused only on an improved throughput for large document collections, e.g., by leveraging distributed instances of an IE system. Information extraction in the context of enterprise search differs for instance from the extraction from the World Wide Web by the fact that usually a variety of structured reference data (corporate databases or terminologies) is available, which often describes the relationships among entities. Furthermore, entity names in a business environment usually follow special characteristics: On the one hand relevant entities such as product identifiers follow certain patterns that are not always known beforehand, but can be inferred using known sample entities, so that unknown entities can be extracted. On the other hand many designators have a more descriptive character (concatenation of descriptive words). The respective references in texts might differ due to the diversity of potential descriptions, often making the identification of such entities difficult. To address IE applications in the presence of available structured data, we study in this work the inference of effective regular expressions from given sample entities. Various generalization and specialization heuristics are used to identify patterns at different syntactic abstraction levels and thus generate regular expressions which promise both high recall and precision. Compared to previous rule learning techniques in the field of information extraction, our technique does not require any annotated document corpus. A method for the identification of entities that are predefined by graph structured reference data is examined as a third contribution. An algorithm is presented which goes beyond an exact string comparison between text and reference data set. It allows for an effective identification and disambiguation of potentially discovered entities by exploitation of approximate matching strategies. The method leverages further relationships among entities for identification and disambiguation. The method presented in this work is superior to previous approaches with regard to precision and recall.

abstract: Ample evidence exists to support the conclusion that enterprise search is failing its users. This failure is costing corporate America billions of dollars every year. Most enterprise search engines are built using web search engines as their foundations. These search engines are optimized for web use and are inadequate when used inside the firewall. Without the ability to use popularity-based measures for ranking documents returned to the searcher, these search engines must rely on full-text search technologies. The Information Science literature explains why full-text search, by itself, fails to adequately discriminate relevant from irrelevant documents. This failure in discrimination results in far too many documents being returned to the searcher, which causes enterprise searchers to abandon their searches in favor of re-creating the documents or information they seek. This dissertation describes and evaluates a potential solution to the problem of failed enterprise search derived from the Information Science literature: subject-aided search. In subject-aided search, full-text search is augmented with a search of subject metadata coded into each document based upon a hierarchically structured subject index. Using the Design Science methodology, this dissertation develops and evaluates three IT artifacts in the search for a solution to the wicked problem of enterprise search failure. / Dissertation/Thesis / Ph.D. Business Administration 2012

Information retrieval within a company can be referred to as enterprise search. With the use of enterprise search, employees can find the information they need in company internal data. If a business can take advantage of the knowledge within the organization, it can save time and effort, and be a source for innovation and development within the company.  In this project, two open source search engines, Recoll and Apache Solr, are selected, set up, and evaluated based on requirements and needs at the pharmacometric consulting company Pharmetheus AB. A requirement analysis is performed to collect system requirements at the company. Through a literature survey, two candidate search engines are selected. Lastly, a Proof of Concept is performed to demonstrate the feasibility of the search engines at the company. The search tools are evaluated on criteria including indexing performance, search functionality and configurability. This thesis presents assessment questions to be used when evaluating a search tool. It is shown that the indexing time for both Recoll and Apache Solr appears to scale linearly for less than one hundred thousand pdf documents. The benefit of an index is demonstrated when search times for both search engines greatly outperforms the Linux command-line tools grep and find. It is also explained how the strict folder structure and naming conventions at the company can be used in Recoll to only index specific documents and sub-parts of a file share. Furthermore, I demonstrate how the Recoll web GUI can be modified to include functionality for filtering on document type.  The results show that Recoll meets most of the company’s system requirements and for that reason it could serve as an enterprise search engine at the company. However, the search engine lacks support for authentication, something that has to be further investigated and implemented before the system can be put into production.

Many organizations are trying to re-create the ‘Google experience’, to find and exploit their own corporate information. However, there is evidence that finding information in the workplace using search engine technology has remained difficult, with socio-technical elements largely neglected in the literature. Explication of the factors and generative mechanisms (ultimate causes) to effective search task outcomes (user satisfaction, search task performance and serendipitous encountering) may provide a first step in making improvements. A transdisciplinary (holistic) lens was applied to Enterprise Search and Discovery capability, combining critical realism and activity theory with complexity theories to one of the world’s largest corporations. Data collection included an in-situ exploratory search experiment with 26 participants, focus groups with 53 participants and interviews with 87 business professionals. Thousands of user feedback comments and search transactions were analysed. Transferability of findings was assessed through interviews with eight industry informants and ten organizations from a range of industries. A wide range of informational needs were identified for search filters, including a need to be intrigued. Search term word co-occurrence algorithms facilitated serendipity to a greater extent than existing methods deployed in the organization surveyed. No association was found between user satisfaction (or self assessed search expertise) with search task performance and overall performance was poor, although most participants had been satisfied with their performance. Eighteen factors were identified that influence search task outcomes ranging from user and task factors, informational and technological artefacts, through to a wide range of organizational norms. Modality Theory (Cybersearch culture, Simplicity and Loss Aversion bias) was developed to explain the study observations. This proposes that at all organizational levels there are tendencies for reductionist (unimodal) mind-sets towards search capability leading to ‘fixes that fail’. The factors and mechanisms were identified in other industry organizations suggesting some theory generalizability. This is the first socio-technical analysis of Enterprise Search and Discovery capability. The findings challenge existing orthodoxy, such as the criticality of search literacy (agency) which has been neglected in the practitioner literature in favour of structure. The resulting multifactorial causal model and strategic framework for improvement present opportunities to update existing academic models in the IR, LIS and IS literature, such as the DeLone and McLean model for information system success. There are encouraging signs that Modality Theory may enable a reconfiguration of organizational mind-sets that could transform search task outcomes and ultimately business performance.

Die Arbeit geht dem Status quo der unternehmensweiten Suche in österreichischen Großunternehmen nach und beleuchtet Faktoren, die darauf Einfluss haben. Aus der Analyse des Ist-Zustands wird der Bedarf an Enterprise-Search-Software abgeleitet und es werden Rahmenbedingungen für deren erfolgreiche Einführung skizziert. Die Untersuchung stützt sich auf eine im Jahr 2009 durchgeführte Onlinebefragung von 469 österreichischen Großunternehmen (Rücklauf 22 %) und daran anschließende Leitfadeninterviews mit zwölf Teilnehmern der Onlinebefragung. Der theoretische Teil verortet die Arbeit im Kontext des Informations- und Wissensmanagements. Der Fokus liegt auf dem Ansatz der Enterprise Search, ihrer Abgrenzung gegenüber der Suche im Internet und ihrem Leistungsspektrum. Im empirischen Teil wird zunächst aufgezeigt, wie die Unternehmen ihre Informationen organisieren und welche Probleme dabei auftreten. Es folgt eine Analyse des Status quo der Informationssuche im Unternehmen. Abschließend werden Bekanntheit und Einsatz von Enterprise-Search-Software in der Zielgruppe untersucht sowie für die Einführung dieser Software nötige Rahmenbedingungen benannt. Defizite machen die Befragten insbesondere im Hinblick auf die übergreifende Suche im Unternehmen und die Suche nach Kompetenzträgern aus. Hier werden Lücken im Wissensmanagement offenbar. 29 % der Respondenten der Onlinebefragung geben zudem an, dass es in ihren Unternehmen gelegentlich bis häufig zu Fehlentscheidungen infolge defizitärer Informationslagen kommt. Enterprise-Search-Software kommt in 17 % der Unternehmen, die sich an der Onlinebefragung beteiligten, zum Einsatz. Die durch Enterprise-Search-Software bewirkten Veränderungen werden grundsätzlich positiv beurteilt. Alles in allem zeigen die Ergebnisse, dass Enterprise-Search-Strategien nur Erfolg haben können, wenn man sie in umfassende Maßnahmen des Informations- und Wissensmanagements einbettet. / The dissertation pursues the status quo of enterprise search within large companies in Austria and examines which aspects take influence on it. From the analysis of the current state the demand for enterprise search software is derived and the conditions of how it can be implemented successfully are outlined. The study is based on an online survey in 2009 among 469 large companies in Austria (response rate of 22 %), which was complemented by semi-structured interviews with twelve participants of the online survey. The theoretical part puts the examination in the field of information and knowledge management. It focuses on the strategy of enterprise search, its differences compared with web search, and its technical features. The empirical part investigates how companies organize information and which problems might arise as a result. This is followed by an analysis of the state of information retrieval within the company. Finally, the degree of awareness and use of enterprise search software and the basic conditions for its installation are explored. Above all, the respondents and interviewees identify deficits concerning metasearch features as well as the search for experts within the company. This in turn reveals gaps in knowledge management efforts. Moreover, 29 % of the online survey’s respondents indicate that their company either sometimes or often take wrong decisions because of insufficient information. 17 % of the responding companies take advantage of enterprise search software. In principle, the changes due to enterprise search software are seen positively. In summary, the results show that enterprise search risks to fail as a success-ful strategy unless it is embedded in comprehensive information and knowledge management activities.