{"id":{"repo_id":"marshall","oai_identifier":"oai:mds.marshall.edu:etd-2135"},"canonical_url":"https://search.ndltd.org/etd/marshall/oai:mds.marshall.edu:etd-2135","repository":{"repo_id":"marshall","name":"Marshall University","base_url":"https://mds.marshall.edu/do/oai/"},"display":{"title":"A Robust Framework for Mining YouTube Data","abstract":"YouTube is currently the most popular and successful video sharing website. As YouTube has broad and profound social impact, YouTube analytics has become a hot research area. The videos on YouTube have become a treasure of data. However, getting access to the immense and massive YouTube data is a challenge. Previous research, studies, and analysis so far, are only conducted on very small volumes of YouTube video data. To date, there exists no mechanism to systematically and continuously collect, process and store the rich set of YouTube data. This thesis presents a methodology to systematically and continuously mine and store the YouTube data. The methodology has two modules: a video discovery and a video metadata collection. YouTube provides an API to conduct search requests analogous to the search performed by a user on the YouTube website. However, the YouTube API’s ‘search’ operation was not designed to return large volumes of data and only provides limited search results (metadata) that can easily be handled by a human. The proposed discovery process makes the search process scalable, robust and swift by (a) initially taking a few carefully selected video IDs (seeds) as input from each of the video categories in YouTube (so to get a wider coverage), and (b) later, using each of them to find related videos over multiple generations. The thesis employed in-memory data management for the discovery process to suppress redundancy explosion and rapidly find new videos. Further, the batch-caching mechanism is introduced to ensure that the high velocity data generated by the discovery process do not result in memory explosion; thereby increasing the reliability of the methodology. The performance of the proposed methodology is gauged over the period of two months. Within two months, 16,000,000 videos were discovered and complete metadata of more than 42,000 videos was mined. The thesis also explores serveral new possible dimensions that can be possible extensions to the proposed framework. The two most promiment dimensions are (a) channel discovery: Every YouTube user that has ever made a comment contributes to a channel. A channel can hold hundreds of YouTube videos and related metadata. Discovering channels can speed up the video discovery up to 100-fold; and (b) channel metadata collection: Since the volume of videos in a channel is massive, therefore, a mechanism needs to be developed to use multiple machines running software agents that can collaborate and communicate with each other to collect metadata of billions of videos in a distributed fashion.","abstract_html":"YouTube is currently the most popular and successful video sharing website. As YouTube has broad and profound social impact, YouTube analytics has become a hot research area. The videos on YouTube have become a treasure of data. However, getting access to the immense and massive YouTube data is a challenge. Previous research, studies, and analysis so far, are only conducted on very small volumes of YouTube video data. To date, there exists no mechanism to systematically and continuously collect, process and store the rich set of YouTube data. This thesis presents a methodology to systematically and continuously mine and store the YouTube data. The methodology has two modules: a video discovery and a video metadata collection. YouTube provides an API to conduct search requests analogous to the search performed by a user on the YouTube website. However, the YouTube API’s ‘search’ operation was not designed to return large volumes of data and only provides limited search results (metadata) that can easily be handled by a human. The proposed discovery process makes the search process scalable, robust and swift by (a) initially taking a few carefully selected video IDs (seeds) as input from each of the video categories in YouTube (so to get a wider coverage), and (b) later, using each of them to find related videos over multiple generations. The thesis employed in-memory data management for the discovery process to suppress redundancy explosion and rapidly find new videos. Further, the batch-caching mechanism is introduced to ensure that the high velocity data generated by the discovery process do not result in memory explosion; thereby increasing the reliability of the methodology. The performance of the proposed methodology is gauged over the period of two months. Within two months, 16,000,000 videos were discovered and complete metadata of more than 42,000 videos was mined. The thesis also explores serveral new possible dimensions that can be possible extensions to the proposed framework. The two most promiment dimensions are (a) channel discovery: Every YouTube user that has ever made a comment contributes to a channel. A channel can hold hundreds of YouTube videos and related metadata. Discovering channels can speed up the video discovery up to 100-fold; and (b) channel metadata collection: Since the volume of videos in a channel is massive, therefore, a mechanism needs to be developed to use multiple machines running software agents that can collaborate and communicate with each other to collect metadata of billions of videos in a distributed fashion.","abstract_has_math":false,"creators":["Tian, Zifeng"],"institution":null,"degree_name":"Computer Science","degree_level":"Thesis","degree_discipline":"College of Information Technology and Engineering","degree_department":null,"school":null,"contributors":["Wook-Sung Yoo","Haroon Malik","Jamil M. Chaudri"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":null,"date_issued":"","date_published":null,"updated_at":"2026-11-21T19:57:03Z","subjects":["Computer Engineering","Computer Sciences","Engineering","Numerical Analysis and Scientific Computing","Physical Sciences and Mathematics"],"languages":[],"rights":[],"rights_urls":[],"identifier_entries":[],"person_identifier_entries":[]},"links":{"outbound_url":"https://mds.marshall.edu/etd/1129","outbound_label":"Repository record","outbound_source":"dc:identifier","record_url":"https://mds.marshall.edu/etd/1129","record_label":"Repository record","record_kind":"record","record_source":"dc:identifier","download_url":null,"download_label":null,"download_source":null,"access_status":null,"embargo_until":null,"access_note":null},"source_record":{"url":"https://mds.marshall.edu/do/oai/?verb=GetRecord&metadataPrefix=oai_etdms&identifier=oai%3Amds.marshall.edu%3Aetd-2135","prefix":"oai_etdms"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Wook-Sung Yoo","Haroon Malik","Jamil M. Chaudri"]},{"key":"dc:creator","label":"Author","values":["Tian, Zifeng"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date.available","label":"Dc Date Available","values":["2018-12-04T08:00:00Z"]},{"key":"dc:date.created","label":"Dc Date Created","values":["2017-01-01T08:00:00Z"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["College of Information Technology and Engineering"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Thesis"]},{"key":"thesis:degree_name","label":"Degree Name","values":["Computer Science"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Computer Engineering","Computer Sciences","Engineering","Numerical Analysis and Scientific Computing","Physical Sciences and Mathematics"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["https://mds.marshall.edu/etd/1129"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description.abstract","label":"Abstract","values":["YouTube is currently the most popular and successful video sharing website. As YouTube has broad and profound social impact, YouTube analytics has become a hot research area. The videos on YouTube have become a treasure of data. However, getting access to the immense and massive YouTube data is a challenge. Previous research, studies, and analysis so far, are only conducted on very small volumes of YouTube video data. To date, there exists no mechanism to systematically and continuously collect, process and store the rich set of YouTube data. This thesis presents a methodology to systematically and continuously mine and store the YouTube data. The methodology has two modules: a video discovery and a video metadata collection. YouTube provides an API to conduct search requests analogous to the search performed by a user on the YouTube website. However, the YouTube API’s ‘search’ operation was not designed to return large volumes of data and only provides limited search results (metadata) that can easily be handled by a human. The proposed discovery process makes the search process scalable, robust and swift by (a) initially taking a few carefully selected video IDs (seeds) as input from each of the video categories in YouTube (so to get a wider coverage), and (b) later, using each of them to find related videos over multiple generations. The thesis employed in-memory data management for the discovery process to suppress redundancy explosion and rapidly find new videos. Further, the batch-caching mechanism is introduced to ensure that the high velocity data generated by the discovery process do not result in memory explosion; thereby increasing the reliability of the methodology. The performance of the proposed methodology is gauged over the period of two months. Within two months, 16,000,000 videos were discovered and complete metadata of more than 42,000 videos was mined. The thesis also explores serveral new possible dimensions that can be possible extensions to the proposed framework. The two most promiment dimensions are (a) channel discovery: Every YouTube user that has ever made a comment contributes to a channel. A channel can hold hundreds of YouTube videos and related metadata. Discovering channels can speed up the video discovery up to 100-fold; and (b) channel metadata collection: Since the volume of videos in a channel is massive, therefore, a mechanism needs to be developed to use multiple machines running software agents that can collaborate and communicate with each other to collect metadata of billions of videos in a distributed fashion."]},{"key":"dc:title","label":"Title","values":["A Robust Framework for Mining YouTube Data"]}]}],"canonical_facts":{"dc:contributor":["Wook-Sung Yoo","Haroon Malik","Jamil M. Chaudri"],"dc:creator":["Tian, Zifeng"],"dc:date.available":["2018-12-04T08:00:00Z"],"dc:date.created":["2017-01-01T08:00:00Z"],"dc:description.abstract":["YouTube is currently the most popular and successful video sharing website. As YouTube has broad and profound social impact, YouTube analytics has become a hot research area. The videos on YouTube have become a treasure of data. However, getting access to the immense and massive YouTube data is a challenge. Previous research, studies, and analysis so far, are only conducted on very small volumes of YouTube video data. To date, there exists no mechanism to systematically and continuously collect, process and store the rich set of YouTube data. This thesis presents a methodology to systematically and continuously mine and store the YouTube data. The methodology has two modules: a video discovery and a video metadata collection. YouTube provides an API to conduct search requests analogous to the search performed by a user on the YouTube website. However, the YouTube API’s ‘search’ operation was not designed to return large volumes of data and only provides limited search results (metadata) that can easily be handled by a human. The proposed discovery process makes the search process scalable, robust and swift by (a) initially taking a few carefully selected video IDs (seeds) as input from each of the video categories in YouTube (so to get a wider coverage), and (b) later, using each of them to find related videos over multiple generations. The thesis employed in-memory data management for the discovery process to suppress redundancy explosion and rapidly find new videos. Further, the batch-caching mechanism is introduced to ensure that the high velocity data generated by the discovery process do not result in memory explosion; thereby increasing the reliability of the methodology. The performance of the proposed methodology is gauged over the period of two months. Within two months, 16,000,000 videos were discovered and complete metadata of more than 42,000 videos was mined. The thesis also explores serveral new possible dimensions that can be possible extensions to the proposed framework. The two most promiment dimensions are (a) channel discovery: Every YouTube user that has ever made a comment contributes to a channel. A channel can hold hundreds of YouTube videos and related metadata. Discovering channels can speed up the video discovery up to 100-fold; and (b) channel metadata collection: Since the volume of videos in a channel is massive, therefore, a mechanism needs to be developed to use multiple machines running software agents that can collaborate and communicate with each other to collect metadata of billions of videos in a distributed fashion."],"dc:identifier":["https://mds.marshall.edu/etd/1129"],"dc:subject":["Computer Engineering","Computer Sciences","Engineering","Numerical Analysis and Scientific Computing","Physical Sciences and Mathematics"],"dc:title":["A Robust Framework for Mining YouTube Data"],"thesis:degree_discipline":["College of Information Technology and Engineering"],"thesis:degree_level":["Thesis"],"thesis:degree_name":["Computer Science"]},"updated_at":"2026-11-21T19:57:03Z"}