Historians, like many types of scholars, are often researchers and educators, and both roles involve significant interaction with primary sources. Primary sources are not only direct evidence for historical arguments but also important materials for teaching historical thinking skills to students in classrooms, and engaging the broader public. However, finding high quality primary sources that are relevant to a historian's specialized topics of interest remains a significant challenge. Automated approaches to text analysis struggle to provide relevant results for these "long tail" searches with long semantic distances from the source material. Consequently, historians are often frustrated at spending so much time on manually the relevance of the contents of these archives other than writing and analysis. To overcome these challenges, my dissertation explores the use of crowdsourcing to support historians in analysis of primary sources. In four studies, I first proposed a class-sourcing model where historians outsource historical analysis to students as a teaching method and students learn historical thinking and gain authentic research experience while doing these analysis tasks. Incite, a realization of this model, deployed in 15 classrooms with positive feedback. Second, I expanded the class-sourcing model to a broader audience, novice (paid) crowds and developedthe Read-agree-predict (RAP) technique to accurately evaluate relevance between primary sources and research topics. Third, I presented a set of design principles for crowdsourcing complex historical documents via the American Soldier project on Zooniverse. Finally, I developed CrowdSCIM to help crowds learn historical thinking and evaluated the tradeoffs between quality, learning and efficiency. The outcomes of the studies provide systems, techniques and design guidelines to 1) support historians in their research and teaching practices, 2) help crowd workers learn historical thinking and 3) suggest implications for the design of future crowdsourcing systems. / Ph. D. / Historians, like many types of scholars, are often researchers and educators, and both roles involve significant interaction with primary sources. Primary sources are not only direct evidence for historical arguments but also important materials for teaching historical thinking skills to students in classrooms, and engaging the broader public. However, finding highquality primary sources that are relevant to a historian’s specialized topics of interest remains a significant challenge. Automated approaches to text analysis struggle to provide relevant results for these “long tail” searches with long semantic distances from the source material. Consequently, historians are often frustrated at spending so much time on manually the relevance of the contents of these archives other than writing and analysis. To overcome these challenges, my dissertation explores the use of crowdsourcing to support historians in analysis of primary sources. In four studies, I first proposed a class-sourcing model where historians outsource historical analysis to students as a teaching method and students learn historical thinking and gain authentic research experience while doing these analysis tasks. Incite, a realization of this model, deployed in 15 classrooms with positive feedback. Second, I expanded the class-sourcing model to a broader audience, novice (paid) crowds and developed the Read-agree-predict (RAP) technique to accurately evaluate relevance between primary sources and research topics. Third, I presented a set of design principles for crowdsourcing complex historical documents via the American Soldier project on Zooniverse. Finally, I developed CrowdSCIM to help crowds learn historical thinking and evaluated the tradeoffs between quality, learning and efficiency. The outcomes of the studies provide systems, techniques and design guidelines to 1) support historians in their research and teaching practices, 2) help crowd workers learn historical thinking and 3) suggest implications for the design of future crowdsourcing systems.

When genetic algorithms (GA) are used to produce music, the results are limited by a fitness bottleneck problem. To create effective music, the GA needs to be thoroughly trained by humans, but this takes extensive time and effort. Applying online collective intelligence or "crowdsourcing" to train a musical GA is one approach to solve the fitness bottleneck problem. The hypothesis was that when music was created by a GA trained by a crowdsourced group and music was created by a GA trained by a small group, the crowdsourced music would be more effective and musically sound. When a group of reviewers and composers evaluated the music, the crowdsourced songs scored slightly higher overall than the songs from the small-group songs, but with the small number of evaluators, the difference was not statistically significant.

Magíster en Ciencias, Mención Computación / Uno de los problemas más significativos a resolver en sistemas que generan información en base a crowdsourcing, ha sido el controlar la calidad de la misma. A pesar de la existencia de casos conocidos que han resuelto este problema en forma exitosa (por ejemplo, Wikipedia o ReCaptcha), las soluciones encontradas son ad hoc al problema abordado, por lo tanto carecen de generalidad y no se pueden aplicar de la misma forma a otros escenarios. Por otra parte, la masificación de la computación móvil ha llevado a que el problema de verificar la calidad de la información que se ingresa a través de crowdsourcing, se haga presente en diversos escenarios, por ejemplo en la validación de la información georreferenciada que ingresan los usuarios de aplicaciones móviles acerca de diversos puntos de interés como: bares, restaurantes, colegios, hospitales, servicios públicos, etc. Las soluciones disponibles para validar información georreferenciada usualmente no involucran mecanismos crowdsourcing, y las que lo hacen, plantean mecanismos de validación un tanto limitados. Debido a esto, este trabajo de tesis busca explorar ese camino como una nueva alternativa de solución al problema planteado. Particularmente este trabajo desarrolló una heurística que permite la validación de la información que ingresan los usuarios, mientras éstos se desplazan por la ciudad (por ejemplo, tags). La heurística utiliza conceptos de computación social y crowdsourcing para reducir la incertidumbre acerca de la validez de dicha información. Para evaluar la solución se desarrolló un sistema móvil y una API (Application Programming Interface). La heurística propuesta se implementó y se dejó disponible a través de esta API para que otros desarrolladores puedan hacer uso de ésta. Por una cuestión de factibilidad del proceso de evaluación, dicho proceso fue realizado en Lima, Perú, y contó con la participación de 30 usuarios. Estos usuarios contaban con experiencia en el uso de aplicaciones similares a la presentada en esta tesis. A pesar de que los resultados obtenidos son insuficientes para sacar conclusiones definitivas respecto a la efectividad de la heurística propuesta, los datos obtenidos nos dan fuertes indicios de que la estrategia de solución propuesta es factible de usar y es útil para la validación de información georreferenciada.

Introduction: Despite the method existing for centuries, uses of crowdsourcing have been rising rapidly since the term was coined a decade ago. Crowdsourcing refers to ‘outsourcing’ a problem or task to a large group of people (i.e., a crowd) rapidly and cheaply. Researchers debate over definitions of crowdsourcing, and it is often conflated with mHealth, web 2.0, or data mining. Due to the inexpensive and rapid nature of crowdsourcing, it may be particularly amenable to health research and practice, especially in a global health context, where health systems, human resources, and finances are often scarce. Indeed, one of the dominant methods of health research prioritization uses crowdsourcing, and in particular, wisdom of the crowds. This method, called the Child Health and Nutrition Research Initiative (CHNRI) method, employs researchers to generate and rank research options which are scored against pre-set criteria. Their scores are combined with weights for each criterion, set by a larger, diverse group of stakeholders, to create a ranked list of research options. Unfortunately, due to difficulties in defining and assembling a group of stakeholders that would be appropriate to each exercise, 75% of CHNRI exercises to-date did not involve stakeholders, and therefore presented unweighted ranks. Methods: First, a crowdsourcing was defined through a literature review. Benefits and challenges of crowdsourcing were explored, in addition to ethical issues with crowdsourcing. A second literature review was conducted to explore ways in which crowdsourcing has been already used in health and global health. As crowdsourcing could be a potential solution to data scarcity or act as a platform for intervention in global health settings, but its potential has never been systematically assessed, a CHNRI exercise was conducted to explore potential uses of crowdsourcing in global health and conflict. Experts from both global health and crowdsourcing participated in generation and scoring ideas. This CHNRI exercise was conducted in-line with previously described steps of the CHNRI method for setting health research priorities. As three quarters of CHNRI exercises have not utilized a larger reference group (LRG) of stakeholders, and the public was cited as the most difficult stakeholder group to involve, we conducted a survey using Amazon Mechanical Turk, an online crowdsourcing platform, that involved an international group of predominantly laypersons who, in essence, formed a public stakeholder group, scoring the most common CHNRI criteria using a 5-point Likert scale. The resulting means were converted to weights that can be used in upcoming exercises. Differences in geographic location, and whether the respondents were health stakeholders were assessed through the Fisher exact test and Wilcoxon rank-sum test, respectively. The influence of other demographic characteristics was explored through random-intercept modelling and logistic regression. Finally, an example of a national-level CHNRI exercise, which is the largest CHNRI conducted to-date, exploring research priorities in child health in India is described. Results: A comprehensive definition of crowdsourcing is given, along with its benefits, challenges, and ethical considerations for using crowdsourcing, based on a literature review. An overview of uses of crowdsourcing in health are discussed, and potential challenges and techniques for improving accuracy, such as introducing thresholds, qualifiers, introducing modular tasks and gamification. Crowdsourcing was frequently used as a diagnostics or surveillance tool. The CHNRI method was not identified in the second literature review. In re-weighting the CHNRI criteria using a public stakeholder group, we identified differences in relative importance of the criteria driven by geographic location and health status. When using random-intercept modelling to control for geographic location, we found differences due to health status in many criteria (n = 11), followed by gender (n = 10), ethnicity (n = 9), and religion (n = 8). We used the CHNRI method to explore potential uses of crowdsourcing in global health, and found that the majority of ideas were problem solving or data generation in nature. The top-ranked idea was to use crowdsourcing to generate more timely reports of future epidemics (such as in the case of Ebola), and other ideas relating to using crowdsourcing for the surveillance or control of communicable disease scored highly. Many ideas were related to the United Nations’ Sustainable Development Goals (SDGs). Finally, a national-level exercise to set research priorities in child health in India identified differential priorities for three regions (Empowered Action Group and North Eastern States, Northern States and Union Territories, and the Southern and Western States). The results will be very useful in developing targeted programmes for each region, enabling India to make progress towards SDG 3.2. Conclusion: Crowdsourcing has grown exponentially in the past decade. Integrating gamification, machine learning, simplifying tasks and introducing thresholds or trustworthiness scores increases accuracy of results. This research provides recommendations for improvements in the CHNRI method itself, and for crowdsourcing, generally. Crowdsourcing is a rapid, inexpensive tool for research, and thus, is a promising data collection method or intervention for health and global health.

Organizations that want to maintain a good innovation capability can seek knowledge both within and outside its boundaries. Opening up innovation processes, through so-called open innovation, allows organizations to combine internal and external knowledge sources in order to include previously excluded perspectives to the innovation process. This study examines how knowledge is created and transferred when conducting open innovation through crowdsourcing. It is done based on Nonaka’s (1991; 1994) theory of how knowledge is created through conversion processes, where implicit knowledge can be converted into explicit knowledge and vice versa. How the crowdsourcing process can generate knowledge transfer is analyzed with the support of Nonaka’s theory and a number of empirically based illustrations. The gathered empirics primarily consist of interviews conducted at companies working with different crowdsourcing platforms. The paper clarifies how the crowdsourcing process gives support for knowledge transfer according to Nonaka’s conversion processes, and that additional knowledge transfer is made possible through the collaborative nature of crowdsourcing. The analysis also shows that extensive planning for how crowdsourcing initiatives should be handled is required in order for them to strengthen the organization’s innovation capability. / Organisationer som vill upprätthålla en god innovationsförmåga kan söka kunskap både inom och utom dess gränser. Att öppna sina innovationsprocesser, genom s.k. öppen innovation, innebär att organisationer kombinerar interna och externa kunskapskällor för att inkludera tidigare uteslutna perspektiv till innovationsprocessen. I studien undersöks hur kunskap skapas och överförs vid öppen innovation genom crowdsourcing. Det görs med utgångspunkt från Nonakas (1991; 1994) teori om hur kunskap skapas genom omvandlingsprocesser, där tyst kunskap kan omvandlas till explicit kunskap och vice versa. Med stöd av Nonakas teori och ett flertal empiriskt baserade illustrationer analyseras hur crowdsourcingprocessen kan generera kunskapsöverföring. Den insamlade empirin består främst av intervjuer som genomförts på företag som arbetar med olika crowdsourcingplattformar. Uppsatsen tydliggör hur crowdsourcingprocessen ger möjlighet till kunskapsöverföring enligt Nonakas omvandlingsprocesser, och att ytterligare kunskapsöverföring möjliggörs med hjälp av crowdsourcingens kollaborativa karaktär. Analysen visar även att det krävs genomgående planering för hur crowdsourcinginitiativ ska hanteras för att de ska stärka organisationens innovationsförmåga.

Folkfinansiering via Internet har på senare tid växt fram som en finansieringskanal förnya projekt och företagsidéer som har svårt att bli finansierade genom de traditionellafinansieringskanalerna. Det har emellertid varit okänt hur småföretagare ser på dennafinansieringskanal och vad de upplever som dess för- och nackdelar. Syftet med dennauppsats är att skapa förståelse för småföretagares uppfattning och inställning till deolika finansieringskanalerna banklån, riskkapital, kapital från affärsänglar ochfolkfinansiering. Studien har ett särskilt fokus på hur folkfinansiering står sig ursmåföretagarens perspektiv.För att skapa oss en förståelse för småföretagares finansieringssituation har vi samlatteorier som berör finansiering av just småföretag. Den teoretiska referensramenbehandlar därför traditionella finansieringsalternativ som banklån, riskkapital ochaffärsänglar men innehåller även vad som är känt om finansieringsformenfolkfinansiering. Vårt arbete är en kvalitativ studie där vi har intervjuat småföretagare iUmeå om hur de ser på de olika finansieringsformerna samt deras tankar kringfolkfinansiering.Resultatet visar att folkfinansiering kan fungera både som ett komplement och substituttill traditionell finansiering. Respondenterna hade generellt sett en positiv inställning tillfolkfinansiering men hade ändå vissa farhågor om att en folkfinansieringskampanjkunde ta onödigt mycket tid i anspråk och inte ge samma avlastning till entreprenörensom de andra finansieringsalternativen gav. Våra resultat visar även att banklån är enväldigt oattraktiv finansieringsform för denna kategori av företagare. Däremot sesaffärsänglar och riskkapitalister som ett bättre alternativ om det får ske påentreprenörens egna villkor. Fördelarna med affärsänglar och riskkapital jämfört medfolkfinansiering upplever respondenterna är att de utöver kapital även får tillgång tillvärdefull kunskap och viss avlastning i sitt företagande. Detta upplevdes intefolkfinansiering kunna bidra med.

Dissertação de Mestrado, apresentada ao Programa de Pós-Graduação em Desenvolvimento Socioeconômico, da Universidade do Extremo Sul Catarinense – UNESC, para a obtenção do grau de Mestre em Desenvolvimento Socioeconômico. / A dinâmica do macroambiente induz às empresas a necessidade de inovar continuamente seus negócios. Isso permite acompanharem os movimentos do mercado e mudanças no comportamento de compra dos consumidores e desta forma realizar a manutenção de sua competitividade, rentabilidade e sustentabilidade do negócio. Neste contexto, o presente estudo busca analisar a interação entre o compartilhamento do conhecimento ao fenômeno do crowdsourcing interno na promoção da inovação em uma empresa do setor varejista supermercadista. Quanto aos procedimentos metodológicos, ao integrar campos da gestão do conhecimento, ao crowdsourcing e a inovação esta pesquisa tem caráter interdisciplinar com abordagem qualitativa. Quanto aos objetivos o trabalho é descritivo e exploratório. A pesquisa-ação mediante estudo de caso único – uma empresa varejista supermercadista – foram as estratégias de pesquisa adotadas. Como técnica se utilizou a análise documental. Entende-se que a participação dos diversos stakeholders é importante para capturar o conhecimento interno e externo das empresas. Ao final desse trabalho, verificou-se que por meio da inteligência coletiva, utilizando-se do fenômeno do crowdsourcing interno pode-se promover a estratégia e o desafio lançado pela empresa para inovar seu modelo de gestão. A inovação no modelo de gestão ocorreu por meio da criação de novos instrumentos e processos de trabalho, que foram construídos mediante engajamento e participação de seus colaboradores. As experiências e conhecimentos heterogêneos e específicos dos colaboradores, em um ambiente de confiança e compromisso mútuo entre as partes, possibilitaram que ao final de cada desenvolvimento das ideias e projetos, apresenta-se a inovação almejada. Percebeu-se no desenvolvimento e análise da pesquisa que ocorreu o devido reconhecimento das contribuições significativas dos colaboradores, por meio das políticas de recursos humanos da empresa também apresentadas neste trabalho, gerando satisfação dos colaboradores e um novo modelo de gestão mais participativo e atualizado da empresa, conectado com o momento atual da competitividade de mercado.

Submitted by Lucienne Costa (lucienneferreira@ufcg.edu.br) on 2018-05-03T21:05:40Z No. of bitstreams: 1 IANNA MARIA SODRÉ FERREIRA DE SOUSA – TESE (PPGCC) 2017.pdf: 3044330 bytes, checksum: 5ec0f15ac650d61c186921dd2c1ef9f7 (MD5) / Made available in DSpace on 2018-05-03T21:05:40Z (GMT). No. of bitstreams: 1 IANNA MARIA SODRÉ FERREIRA DE SOUSA – TESE (PPGCC) 2017.pdf: 3044330 bytes, checksum: 5ec0f15ac650d61c186921dd2c1ef9f7 (MD5) Previous issue date: 2017-07-19 / Muitos sistemas de computação por humanos usam mercados de trabalho crowdsourcing para recrutar trabalhadores. No entanto, devido à natureza aberta desses mercados, garantir que os resultados produzidos pelos trabalhadores possuam uma qualidade suficientemente alta ainda é uma tarefa desafiadora, particularmente em mercados de microtarefas, onde a avaliação precisa ser feita de forma automática. A pré-seleção de trabalhadores adequa- dos é um mecanismo que pode melhorar a qualidade dos resultados obtidos. Isso pode ser feito considerando as informações do cadastro pessoal do trabalhador, o comportamento histórico do trabalhador no sistema ou o uso de testes de qualificação customizados. En- tretanto, pouco se sabe sobre como os solicitantes usam testes de qualificação na prática e se estes tem influência na qualidade dos resultados apresentados pelos trabalhadores. Este estudo visa avançar esse conhecimento. Por meio de análise de distribuições, classificação e agrupamento, as tarefas e os solicitantes foram caracterizados utilizando dados obtidos da plataforma Amazon Mechanical Turk em dois períodos de tempo distintos. Os resultados mostram que a maioria das tarefas (94% e 87%, para a coleta de dados1 e 2,respectivamente) usa algum teste de qualificação para a pré-seleção de trabalhadores e que o tipo e o número de testes de qualificação não são determinados pela classe da tarefa. Os solicitantes, em sua maioria, submetem tarefas com apenas um único teste de qualificação do tipo reputação, no entanto, os solicitantes mais ativos na plataforma usam, exclusivamente, teste de qualificação customizado. Para avaliar o impacto do uso de testes de qualificação customizados na qualidade dos resultados produzidos, foram realiza dos experimentos com três tipos diferentes de tarefas usando tanto trabalhadores qualificados (mestres ou trabalhadores pré-selecionados) como não qualificados. Os resultados mostram que a pontuação média alcançada pelos trabalhadores pré-selecionados foi sempre maior que a alcançada por trabalhadores que não foram pré-selecionados. Além disso, o desempenho de trabalhadores pré-selecionados foi muito próximo dos trabalhadores considerados mestres e, em alguns cenários, melhor, indicando assim, que é possível obter resultados mais acurados em plataformas de trabalho on-line de microtarefas quando se usa testes de qualificação.

Coarse-Grained Reconfigurable Architectures (CGRAs) are promising in developing high performance low-power portable applications. In this research, we crowdsource a mapping problem using gamification to harnass human intelligence. A scientific puzzle game, Untangled, was developed to solve a mapping problem by encapsulating architectural characteristics. The primary motive of this research is to draw insights from the mapping solutions of players who possess innate abilities like decision-making, creative problem-solving, recognizing patterns, and learning from experience. In this dissertation, an extensive analysis was conducted to investigate how players' computational skills help to solve an open-ended problem with different constraints. From this analysis, we discovered a few common strategies among players, and subsequently, a library of dictionaries containing identified patterns from players' solutions was developed. The findings help to propose a better version of the game that incorporates these techniques recognized from the experience of players. In the future, an updated version of the game that can be developed may help low-performance players to provide better solutions for a mapping problem. Eventually, these solutions may help to develop efficient mapping algorithms, In addition, this research can be an exemplar for future researchers who want to crowdsource such electrical engineering problems and this approach can also be applied to other domains.

Crowdsourcing systems empower individuals and companies to outsource labor-intensive tasks that cannot currently be solved by automated methods and are expensive to tackle by domain experts. Crowdsourcing platforms are traditionally used to provide training labels for supervised machine learning algorithms. Crowdsourced tasks are distributed among internet workers who typically have a range of skills and knowledge, differing previous exposure to the task at hand, and biases that may influence their work. This inhomogeneity of the workforce makes the design of accurate and efficient crowdsourcing systems challenging. This dissertation presents solutions to improve existing crowdsourcing systems in terms of accuracy and efficiency. It explores crowdsourcing tasks in two application areas, political discourse and annotation of biomedical and everyday images. The first part of the dissertation investigates how workers' behavioral factors and their unfamiliarity with data can be leveraged by crowdsourcing systems to control quality. Through studies that involve familiar and unfamiliar image content, the thesis demonstrates the benefit of explicitly accounting for a worker's familiarity with the data when designing annotation systems powered by the crowd. The thesis next presents Crowd-O-Meter, a system that automatically predicts the vulnerability of crowd workers to believe \enquote{fake news} in text and video. The second part of the dissertation explores the reversed relationship between machine learning and crowdsourcing by incorporating machine learning techniques for quality control of crowdsourced end products. In particular, it investigates if machine learning can be used to improve the quality of crowdsourced results and also consider budget constraints. The thesis proposes an image analysis system called ICORD that utilizes behavioral cues of the crowd worker, augmented by automated evaluation of image features, to infer the quality of a worker-drawn outline of a cell in a microscope image dynamically. ICORD determines the need to seek additional annotations from other workers in a budget-efficient manner. Next, the thesis proposes a budget-efficient machine learning system that uses fewer workers to analyze easy-to-label data and more workers for data that require extra scrutiny. The system learns a mapping from data features to number of allocated crowd workers for two case studies, sentiment analysis of twitter messages and segmentation of biomedical images. Finally, the thesis uncovers the potential for design of hybrid crowd-algorithm methods by describing an interactive system for cell tracking in time-lapse microscopy videos, based on a prediction model that determines when automated cell tracking algorithms fail and human interaction is needed to ensure accurate tracking.