<p dir="ltr">The quality of crowdsource annotations has always been a challenge due to the variability in annotators backgrounds, task complexity, the subjective nature of many labeling tasks, and various other reasons. Hence, it is crucial to evaluate these annotations to ensure their reliability. Traditionally, human experts evaluate the quality of crowdsourced annotations, but this approach has its own challenges. Hence, this paper proposes to leverage large language models like ChatGPT-4 to evaluate one of the existing crowdsourced MAVEN dataset and explore its potential as an alternative solution. However, due to stochastic nature of LLMs, it is important to discern when to trust and question LLM responses. To address this, we introduce a novel approach that applies Rubin's framework for identifying and using linguistic cues within LLM responses as indicators of LLMs certainty levels. Our findings reveal that ChatGPT-4 successfully identified 63% of the incorrect labels, highlighting the potential for improving data label quality through human-AI collaboration on these identified inaccuracies. This study underscores the promising role of LLMs in evaluating crowdsourced data annotations offering a way to enhance accuracy and fairness of crowdsource annotations while saving time and costs.</p><p dir="ltr"><br></p>

In the modern industrialized society, a long commuting time is becoming more and more common. However, commuting results in a number of different costs, for example, external costs such as congestion and pollution as well as internal costs such as individual time consumption. On the other hand, increased commuting opportunities offer welfare gains, for example via larger local labor markets. The length of the commute that is acceptable to the workers is determined by the workers' preferences and the compensation opportunities in the labor market. In this thesis the value of travel time or commuting time changes, has been empirically analyzed in four self-contained essays. First, a large set of register data on the Swedish labor market is used to analyze the commuting time changes that follow residential relocations and job relocations. The average commuting time is longer after relocation than before, regardless of the type of relocation. The commuting time change after relocation is found to differ substantially with socio-economic characteristics and these effects also depend on where the distribution of commuting time changes is evaluated. The same data set is used in the second essay to estimate the value of commuting time (VOCT). Here, VOCT is estimated as the trade-off between wage and commuting time, based on the effects wage and commuting time have on the probability of changing jobs. The estimated VOCT is found to be relatively large, in fact about 1.8 times the net wage rate. In the third essay, the VOCT is estimated on a different type of data, namely data from a stated preference survey. Spouses of two-earner households are asked to individually make trade-offs between commuting time and wage. The subjects are making choices both with regard to their own commuting time and wage only, as well as when both their own commuting time and wage and their spouse's commuting time and wage are simultaneously changed. The results show relatively high VOCT compared to other studies. Also, there is a tendency for both spouses to value the commuting time of the wife highest. Finally, the presence of hypothetical bias in a value of time experiment without scheduling constraints is tested. The results show a positive but not significant hypothetical bias. By taking preference certainty into account, positive hypothetical bias is found for the non-certain subjects.

With cancer being one of the leading causes of death globally, and with oral cancers being among the most common types of cancer, it is of interest to conduct large-scale oral cancer screening among the general population. Deep Learning can be used to make this possible despite the medical expertise required for early detection of oral cancers. A bottleneck of Deep Learning is the large amount of data required to train a good model. This project investigates two topics: certainty calibration, which aims to make a machine learning model produce more reliable predictions, and Active Learning, which aims to reduce the amount of data that needs to be labeled for Deep Learning to be effective. In the investigation of certainty calibration, five different methods are compared, and the best method is found to be Dirichlet calibration. The Active Learning investigation studies a single method, Cost-Effective Active Learning, but it is found to produce poor results with the given experiment setting. These two topics inspire the further development of the cytological annotation tool CytoBrowser, which is designed with oral cancer data labeling in mind. The proposedevolution integrates into the existing tool a Deep Learning-assisted annotation workflow that supports multiple users.