The main motivation for the study described in this dissertation is to benefit from the development in technology and the huge amount of available data which can be easily captured, stored and maintained electronically. We concentrate on Web usage (i.e., log) mining and Web structure mining. Analysing Web log data will reveal valuable feedback reflecting how effective the current structure of a web site is and to help the owner of a web site in understanding the behaviour of the web site visitors. We developed a framework that integrates statistical analysis, frequent pattern mining, clustering, classification and network construction and analysis. We concentrated on the statistical data related to the visitors and how they surf and pass through the various pages of a given web site to land at some target pages. Further, the frequent pattern mining technique was used to study the relationship between the various pages constituting a given web site. Clustering is used to study the similarity of users and pages. Classification suggests a target class for a given new entity by comparing the characteristics of the new entity to those of the known classes. Network construction and analysis is also employed to identify and investigate the links between the various pages constituting a Web site by constructing a network based on the frequency of access to the Web pages such that pages get linked in the network if they are identified in the result of the frequent pattern mining process as frequently accessed together. The knowledge discovered by analysing a web site and its related data should be considered valuable for online shoppers and commercial web site owners. Benefitting from the outcome of the study, a recommendation system was developed to suggest pages to visitors based on their profiles as compared to similar profiles of other visitors. The conducted experiments using popular datasets demonstrate the applicability and effectiveness of the proposed framework for Web mining and recommendation. As a by product of the proposed method, we demonstrate how it is effective in another domain for feature reduction by concentrating on gene expression data analysis as an application with some interesting results reported in Chapter 5.

The Internet and world wide web (WWW) have in recent years, grown exponentially in size and in terms of the volume of information that is available on it. In order to effectively deal with the huge amount of information on the web, so called web search engines have been developed for the task of retrieving useful and relevant information for its users. Unfortunately, these web search engines have not kept pace with the boom growth and commercialization of the web. The main goal of this dissertation is the development of a model for a collaborative personalized meta-search agent (COPEMSA) system for the WWW. This model will enable the personalization of web search for users. Furthermore, the model aims to leverage on current search engines on the web as well as enable collaboration between users of the search system for the purposes of sharing useful resources between them. The model also employs the use of multiple intelligent agents and web content mining techniques. This enables the model to autonomously retrieve useful information for it’s user(s) and present this information in an effective manner. In order to achieve the above stated, the COPEMSA model employs the use of multiple intelligent agents. COPEMSA consists of five core components: a user agent, a query agent, a community agent, a content mining agent and a directed web spider. The user agent learns about the user in order to introduce personal preference into user queries. The query agent is a scaled down meta-search engine with the task of submitting the personalized queries it receives from the user agent to multiple search services on theWWW. The community agent enables the search system to communicate and leverage on the search experiences of a community of searchers. The content mining agent is responsible for analysis of the retrieved results from theWWWand the presentation of these results to the system user. Finally, a directed web spider is used by the content mining agent to retrieve the actual web pages it analyzes from the WWW. In this dissertation an additional model is also presented to deal with a specific problem all web spidering software must deal with namely content and link encapsulation. / Prof. E.M. Ehlers

Recommendations are crucial for the success of large websites. While there are many ways to determine recommendations, the relative quality of these recommenders depends on many factors and is largely unknown. We present the architecture and implementation of AWESOME (Adaptive website recommendations), a data warehouse-based recommendation system. It allows the coordinated use of a large number of recommenders to automatically generate website recommendations. Recommendations are dynamically selected by efficient rule-based approaches utilizing continuously measured user feedback on presented recommendations. AWESOME supports a completely automatic generation and optimization of selection rules to minimize website administration overhead and quickly adapt to changing situations. We propose a classification of recommenders and use AWESOME to comparatively evaluate the relative quality of several recommenders for a sample website. Furthermore, we propose and evaluate several rule-based schemes for dynamically selecting the most promising recommendations. In particular, we investigate two-step selection approaches that first determine the most promising recommenders and then apply their recommendations for the current situation. We also evaluate one-step schemes that try to directly determine the most promising recommendations.

The main motivation for the study described in this dissertation is to benefit from the development in technology and the huge amount of available data which can be easily captured, stored and maintained electronically. We concentrate on Web usage (i.e., log) mining and Web structure mining. Analysing Web log data will reveal valuable feedback reflecting how effective the current structure of a web site is and to help the owner of a web site in understanding the behaviour of the web site visitors. We developed a framework that integrates statistical analysis, frequent pattern mining, clustering, classification and network construction and analysis. We concentrated on the statistical data related to the visitors and how they surf and pass through the various pages of a given web site to land at some target pages. Further, the frequent pattern mining technique was used to study the relationship between the various pages constituting a given web site. Clustering is used to study the similarity of users and pages. Classification suggests a target class for a given new entity by comparing the characteristics of the new entity to those of the known classes. Network construction and analysis is also employed to identify and investigate the links between the various pages constituting a Web site by constructing a network based on the frequency of access to the Web pages such that pages get linked in the network if they are identified in the result of the frequent pattern mining process as frequently accessed together. The knowledge discovered by analysing a web site and its related data should be considered valuable for online shoppers and commercial web site owners. Benefitting from the outcome of the study, a recommendation system was developed to suggest pages to visitors based on their profiles as compared to similar profiles of other visitors. The conducted experiments using popular datasets demonstrate the applicability and effectiveness of the proposed framework for Web mining and recommendation. As a by product of the proposed method, we demonstrate how it is effective in another domain for feature reduction by concentrating on gene expression data analysis as an application with some interesting results reported in Chapter 5.

Ο παγκόσμιος ιστός αποτελεί μία τεράστια αποθήκη πληροφοριών και αναπτύσσεται με τάχιστους ρυθμούς, ενώ η ανθρώπινη ικανότητα να εντοπίζει, να επεξεργάζεται και να αντιλαμβάνεται τις παρεχόμενες πληροφορίες παραμένει πεπερασμένη. Οι μηχανές αναζήτησης διευκολύνουν την αναζήτηση στον παγκόσμιο ιστό και έχουν γίνει αναπόσπαστο κομμάτι της καθημερινής ζωής των χρηστών του διαδικτύου. Οι χρήστες όμως χαρακτηρίζονται από διαφορετικές ανάγκες, προτιμήσεις, ιδιαιτερότητες και κατά την πλοήγησή τους μπορεί να χάσουν τον στόχο της αναζήτησής τους. Η προσωποποίηση στον παγκόσμιο ιστό, δηλαδή η εξατομίκευση των παρεχόμενων αποτελεσμάτων, αποτελεί μία πολλά υποσχόμενη προσέγγιση για την λύση του πληροφοριακού υπερφόρτου, παρέχοντας κατάλληλα προσαρμοσμένες εμπειρίες πλοήγησης. Στα πλαίσια αυτής της διπλωματικής εργασίας αναπτύχθηκε μία μεθοδολογία για την προσωποποίηση των αποτελεσμάτων μίας μηχανής αναζήτησης ώστε αυτά να ανταποκρίνονται στις προτιμήσεις των χρηστών και στα διαιτολογικά τους χαρακτηριστικά. Η μεθοδολογία αναπτύχθηκε σε δύο μέρη: στο εκτός σύνδεσης τμήμα και στο συνδεδεμένο. Στο πρώτο με την χρησιμοποίηση των αρχείων πρόσβασης μίας μηχανής αναζήτησης και των διαιτολογικών χαρακτηριστικών των χρηστών, έγινε εξαγωγή πληροφορίας για τις προτιμήσεις των τελευταίων. Στην συνέχεια με την χρήση μίας οντολογίας που κατασκευάστηκε για τα πλαίσια της διπλωματικής αυτής εργασίας, έγινε σημασιολογική κατηγοριοποίηση των επιλογών των χρηστών και κατασκευάστηκαν τα προφίλ που τους χαρακτηρίζουν. Έπειτα με την χρήση ενός αλγορίθμου ομαδοποίησης οι χρήστες κατηγοριοποιήθηκαν με βάση τα διαιτολογικά τους χαρακτηριστικά και τις επιλογές τους στην μηχανή αναζήτησης. Στο συνδεδεμένο τμήμα ο αλγόριθμος προσωποποίησης εκμεταλλευόμενος την σημασιολογική αντιστοίχιση των αποτελεσμάτων της μηχανής αναζήτησης και τις ομάδες των χρηστών που δημιουργήθηκαν στο εκτός σύνδεσης τμήμα αναδιοργανώνει τα παρεχόμενα από την μηχανή αναζήτησης αποτελέσματα. Η αναδιοργάνωση γίνεται προωθώντας στις υψηλότερες θέσεις των αποτελεσμάτων της μηχανής αναζήτησης τα αποτελέσματα που ταιριάζουν καλύτερα με τις προτιμήσεις και τα χαρακτηριστικά της ομάδας στην οποία εντάσσεται ο χρήστης. Στο τέλος έγιναν πειράματα και εξακριβώθηκαν τα επιθυμητά αποτελέσματα για την προσωποποίηση σύμφωνα με τις σημασιολογικές ομάδες των χρηστών. / The World Wide Web has become a huge data repository and it keeps growing exponentially, whereas the human capability to find, process and understand the provided content remains constant. Search engines facilitate the search process in the World Wide Web and they have become an integral part of the web users' daily lives. However users who are characterized by different needs, preferences and special characteristics, navigate through large Web structures and may lost their goal of inquiry. Web personalization, i.e. the customization of the search engines’ returned results, is one of the most promising approaches for alleviating information overload providing tailored navigation experiences to Web users. The present dissertation presents the methodology which was implemented in order to personalize a search engine’s results for corresponding users’ preferences and dietary characteristics. This methodology was implemented in two parts: the offline and the online part. The first one uses a search engines’ log files and the dietary characteristics of the users in order to extract information for the latter preferences. Afterwards, with the use of an ontology which was created explicitly for this work, semantic profiling of users’ interests was achieved and their corresponding profiles were formed. Then with the use of a clustering algorithm, users’ categorization was made based on their dietary profiles and their preferences in the search engine. In the online part the methodology re-ranks the search engines’ results, based on the semantic characterization of those results and the users’ clusters which were created at the offline part. Re-ranking is achieved by placing those results which match better the interests and the characteristics of the user’s cluster at the top of the list of the search engines’ returned results. Experimental evaluation of the presented methodology shows that the expected objectives from the semantic users’ clustering in search engines are achievable.

This doctoral work focuses on identifying and understanding the intents that motivate a user to perform a search on the Web. To this end, we apply machine learning models that do not require more information than the one provided by the very needs of the users, which in this work are represented by their queries. The knowledge and interpretation of this invaluable information can help search engines to obtain resources especially relevant to users, and thus improve their satisfaction. By means of unsupervised learning techniques, which have been selected according to the context of the problem being solved, we show that is not only possible to identify the user’s intents, but that this process can be conducted automatically. The research conducted in this thesis has involved an evolutionary process that starts from the manual analysis of different sets of real user queries from a search engine. The work passes through the proposition of a new classification of user’s query intents; the application of different unsupervised learning techniques to identify those intents; up to determine that the user’s intents, rather than being considered as an uni–dimensional problem, should be conceived as a composition of several aspects, or dimensions (i.e., as a multi–dimensional problem), that contribute to clarify and to establish what the user’s intents are. Furthermore, from this last proposal, we have configured a framework for the on–line identification of the user’s query intent. Overall, the results from this research have shown to be effective for the problem of identifying user’s query intent. / Este trabajo doctoral se enfoca en identificar y entender las intenciones que motivan a los usuarios a realizar búsquedas en la Web a través de la aplicación de métodos de aprendizaje automático que no requieren datos adicionales más que las necesidades de información de los mismos usuarios, representadas a través de sus consultas. El conocimiento y la interpretación de esta información, de valor incalculable, puede ayudar a los sistemas de búsqueda Web a encontrar recursos particularmente relevantes y así mejorar la satisfacción de sus usuarios. A través del uso de técnicas de aprendizaje no supervisado, las cuales han sido seleccionadas dependiendo del contexto del problema a solucionar, y cuyos resultados han demostrado ser efectivos para cada uno de los problemas planteados, a lo largo de este trabajo se muestra que no solo es posible identificar las intenciones de los usuarios, sino que este es un proceso que se puede llevar a cabo de manera automática. La investigación desarrollada en esta tesis ha implicado un proceso evolutivo, el cual inicia con el análisis de la clasificación manual de diferentes conjuntos de consultas que usuarios reales han sometido a un motor de búsqueda. El trabajo pasa a través de la proposición de una nueva clasificación de las intenciones de consulta de usuarios, y el uso de diferentes técnicas de aprendizaje no supervisado para identificar dichas intenciones, llegando hasta establecer que éste no es un problema unidimensional, sino que debería ser considerado como un problema de múltiples dimensiones, donde cada una de dichas dimensiones, o facetas, contribuye a clarificar y establecer cuál es la intención del usuario. A partir de este último trabajo, hemos creado un modelo para la identificar la intención del usuario en un escenario on–line.

Os últimos anos foram marcados pelo surgimento de diversas mídias sociais, desde o Orkut até o Facebook, assim como Twitter, Youtube, Google+ e tantos outros: cada um oferece novas funcionalidades como forma de atrair um maior número de usuários. Essas mídias sociais geram uma grande quantidade de dados, que se devidamente processados podem ser utilizados para se identificar tendências, padrões e mudanças. O objetivo deste trabalho é a descoberta dos principais temas abordados em uma rede social, caracterizados como agrupamentos de termos relevantes, restritos a determinado contexto e o estudo de sua evolução ao longo do tempo. Para tanto serão utilizados procedimentos fundamentados em Mineração de Dados e no Processamento de Textos. Em um primeiro momento são utilizadas técnicas de pré-processamento de textos com o objetivo de identificar os termos mais relevantes que aparecem nas mensagens textuais da rede social. Em seguida utilizam-se algoritmos clássicos de agrupamento - k-means, k-medoids, DBSCAN - e o recente NMF (Non-negative Matrix Factorization), para a identificação dos temas principais destas mensagens, caracterizados como agrupamentos de termos relevantes. A proposta foi avaliada sobre a rede Twitter, utilizando-se bases de tweets considerando diversos contextos. Os resultados obtidos evidenciam a viabilidade da proposta e sua aplicação na identificação de temas relevantes desta rede social. / Recent years have been marked by the emergence of various social media, from Orkut to Facebook, and Twitter, Youtube, Google+ and many others: each offers new features as a way to attract more users. These social media generate a large amount of data which is processed properly can be used to identify trends, patterns and changes. The objective of this work is the discovery of the key topics in a social network, characterized as relevant terms groupings, restricted to a particular context and the study of its evolution over time. For that will be used procedures based on Data Mining and Text Processing. At first techniques are used preprocessing of texts in order to identify the most relevant terms that appear in the text messages from the social network. Next are used grouping of classical algorithms - k-means, k-medoids, DBSCAN - and the recent NMF (Non-negative Matrix Factorization), to identify the main themes of these messages, characterized as relevant terms groupings. The proposal was evaluated on the Twitter network, using bases tweets considering different contexts. The results show the feasibility of the proposal and its application in the identification of relevant topics of this social network

Os últimos anos foram marcados pelo surgimento de diversas mídias sociais, desde o Orkut até o Facebook, assim como Twitter, Youtube, Google+ e tantos outros: cada um oferece novas funcionalidades como forma de atrair um maior número de usuários. Essas mídias sociais geram uma grande quantidade de dados, que se devidamente processados podem ser utilizados para se identificar tendências, padrões e mudanças. O objetivo deste trabalho é a descoberta dos principais temas abordados em uma rede social, caracterizados como agrupamentos de termos relevantes, restritos a determinado contexto e o estudo de sua evolução ao longo do tempo. Para tanto serão utilizados procedimentos fundamentados em Mineração de Dados e no Processamento de Textos. Em um primeiro momento são utilizadas técnicas de pré-processamento de textos com o objetivo de identificar os termos mais relevantes que aparecem nas mensagens textuais da rede social. Em seguida utilizam-se algoritmos clássicos de agrupamento - k-means, k-medoids, DBSCAN - e o recente NMF (Non-negative Matrix Factorization), para a identificação dos temas principais destas mensagens, caracterizados como agrupamentos de termos relevantes. A proposta foi avaliada sobre a rede Twitter, utilizando-se bases de tweets considerando diversos contextos. Os resultados obtidos evidenciam a viabilidade da proposta e sua aplicação na identificação de temas relevantes desta rede social. / Recent years have been marked by the emergence of various social media, from Orkut to Facebook, and Twitter, Youtube, Google+ and many others: each offers new features as a way to attract more users. These social media generate a large amount of data which is processed properly can be used to identify trends, patterns and changes. The objective of this work is the discovery of the key topics in a social network, characterized as relevant terms groupings, restricted to a particular context and the study of its evolution over time. For that will be used procedures based on Data Mining and Text Processing. At first techniques are used preprocessing of texts in order to identify the most relevant terms that appear in the text messages from the social network. Next are used grouping of classical algorithms - k-means, k-medoids, DBSCAN - and the recent NMF (Non-negative Matrix Factorization), to identify the main themes of these messages, characterized as relevant terms groupings. The proposal was evaluated on the Twitter network, using bases tweets considering different contexts. The results show the feasibility of the proposal and its application in the identification of relevant topics of this social network

Webbapplikationers komplexitet och mängden avancerade tjänster ökar. Loggning av aktiviteter kan öka förståelsen över användares beteenden och behov, men används i för stor mängd utan relevant information. Mer avancerade system medför ökade krav för prestandan och loggning blir än mer krävande för systemen. Det finns behov av smartare system, utveckling inom tekniker för prestandaförbättringar och tekniker för datainsamling. Arbetet kommer undersöka hur svarstider påverkas vid loggning av serverdata, enligt datainsamlingsfasen i web usage mining, beroende på lagringsstrategier. Hypotesen är att loggning kan försämra svarstider ytterligare. Experiment genomförs där fyra olika lagringsstrategier används för att lagra serverdata med olika tabell- och databasstrukturer, för att se vilken strategi som påverkar svarstiderna minst. Experimentet påvisar statistiskt signifikant skillnad mellan lagringsstrategierna enligt ANOVA. Lagringsstrategi 4 påvisar bäst effekt för prestandans genomsnittliga svarstid, jämfört med lagringsstrategi 2 som påvisar mest negativ effekt för den genomsnittliga svarstiden. Framtida arbete vore intressant för att stärka resultaten. / Web applications complexity and the amount of advanced services increases. Logging activities can increase the understanding of users behavior and needs, but is used too much without relevant information. More advanced systems brings increased requirements for performance and logging becomes even more demanding for the systems. There is need of smarter systems, development within the techniques for performance improvements and techniques for data collection. This work will investigate how response times are affected when logging server data, according to the data collection phase in web usage mining, depending on storage strategies. The hypothesis is that logging may degrade response times even further. An experiment was conducted in which four different storage strategies are used to store server data with different table- and database structures, to see which strategy affects the response times least. The experiment proves statistically significant difference between the storage strategies with ANOVA. Storage strategy 4 proves the best effect for the performance average response time compared with storage strategy 2, which proves the most negative effect for the average response time. Future work would be interesting for strengthening the results.

This master's thesis deals with creating of an application, goal of which is to perform data preprocessing of web logs and finding association rules in them. The first part deals with the concept of Web mining. The second part is devoted to Web usage mining and notions related to it. The third part deals with design of the application. The forth section is devoted to describing the implementation of the application. The last section deals with experimentation with the application and results interpretation.