京都大学 / 新制・課程博士 / 博士(工学) / 甲第24609号 / 工博第5115号 / 新制||工||1978(附属図書館) / 京都大学大学院工学研究科機械理工学専攻 / (主査)教授 小森 雅晴, 教授 松野 文俊, 教授 松原 厚 / 学位規則第4条第1項該当 / Doctor of Philosophy (Engineering) / Kyoto University / DFAM

In remote or teleoperational tasks involving humans and robots, various aspects of the remote display system may greatly influence the individual's interactions with the teleoperated entity. This dissertation examined various configurations of display systems on several measures of operator performance, physiological states, and perceptions of the task. Display configurations included altering the camera placement (attached to the robot or placed overhead), screen orientation (horizontal or vertical), and screen size (small or large). Performance was measured in terms of specific task goals, accuracies, strategies, and completion times. Physiological state was assessed through physiological markers of arousal, specifically heart rate and skin conductance. Operator perception of the task was measured with a self-reported perception of workload and frustration. Scale model live simulation was used to create a task driven environment to test the display configurations. Screen size influenced performance on complex tasks in mixed ways. Participants using a small screen exhibited better problem solving strategies in a complex driving task. However, participants using the large screen exhibited better driving precision when the task required continual attention. These findings have value in design decisions for teleoperated interfaces where the advantages and disadvantages of screen size must be considered carefully. Orientation of the visual information seems to have much less impact on the operator than the source of the information, though it was an important factor of the display system when taken together with screen size and camera view. Results show strong influence of camera placement on many of the performance variables. Interestingly, the participants rated a higher frustration in the overhead condition, but not a higher task load, indicating that while they realized that the task was frustrating and perhaps they could have done better, they did not recognize the task as overloading. This was the case even though they took longer to complete the task and experienced more errors related to turning in the overhead camera condition. This finding may indicate a potential danger for systems in which the operator is expected to recognize when he or she is being overloaded. This type of performance decrease due to added frames of reference may be too subtle to register in the operator's self awareness

Over the past few years, the surge in VR (Virtual Reality) video traffic on networks has been remarkable. Nonetheless, a key challenge remains: ensuring a top-notch quality of experience (QoE) for VR video playback, especially when network bandwidth is limited. Prior studies have mainly focused on tile-based adaptive bitrate (ABR) streaming operating at the application layer on the server/client side to improve QoE, using single viewport prediction to conserve bandwidth. However, single-viewpoint prediction models face limitations due to uncertainties linked with head movement, making it difficult to handle sudden user motions effectively. To overcome these constraints, we propose a lightweight multimodal spatial-temporal transformer architecture, which generates multiple viewpoint trajectories and their corresponding probabilities while leveraging historical trajectory information. Consequently, we introduce a multi-agent reinforcement learning (MARL)-based ABR algorithm that capitalizes on multiple viewport prediction for VR video streaming at the application layer. Our algorithm strives to optimize various QoE objectives under diverse network conditions. To address the ABR problem, we formulate it as a Decentralized Partially Observable Markov Decision Process (Dec-POMDP) problem. To tackle this effectively, we develop a MAPPO (Multi-Agent Proximal Policy Optimization) algorithm within a centralized training and decentralized execution (CTDE) framework. Meanwhile, we also improve QoE at the network layer by utilizing network resources in different network nodes during VR video streaming. We present an innovative system called tile-weighted rate-distortion (TWRD) packet scheduling optimization, which takes advantage of viewpoint prediction. The system dynamically assigns weights to tiles and their corresponding packets using the probability of viewpoint prediction. Due to limited bandwidth, the problem of packet scheduling arises, requiring the determination of which packets should be dropped. To address this challenge, we formulate the problem as an optimization task, taking into account error propagation in the video. Our system leverages the weighted rate-distortion information of packets and applies dynamic programming techniques to design an optimal packet scheduling scheme. By selectively dropping packets at network nodes, our proposed system effectively reduces network congestion and enhances the overall performance of VR video streaming systems operating within bandwidth limitations.

<p>This thesis investigates two significant problems in control and coordination of complex teleoperation systems as they relate to the operation of a mobile robotic manipulator. The first part of the thesis focuses on the design of a control framework to resolve kinematic redundancy in teleoperation of a mobile robotic manipulator. Apart from the redundancy, workspace considerations for the operator and robot and asymmetry of master and slave systems pose significant design challenges in such telerobotic systems . The second part of the thesis considers psychophysical aspects of teleoperation from the operator's perspective. This part presents a method for automatic {\em optimal} positioning of a single camera for a remotely navigated mobile robot in systems with a controllable camera platform. In each part, a constrained optimization problem is formulated and solved in real time. The solution of these optimization problems are integrated seamlessly into the teleoperation control framework in order to assist the operator in accomplishing the main task. The proposed control framework in the first part allows the operator to concentrate on the manipulation task while the mobile base and arm joint configurations are automatically {\em optimized} according to the needs of the task. Autonomous control subtasks are defined to guide the base and the arms towards this optimal configuration while the operator teleoperates the end-effector(s) of the mobile arm(s). The teleoperation and autonomous control tasks have adjustable relative priorities set by the system designer. The work in the second part enables the operator to focus mainly on navigation and manipulation while the camera viewpoint is automatically adjusted. The workspace and motion limits of the camera system and the location of the obstacles are taken into consideration in camera view planning. A head tracking system enables the operator to use his/her head movements as an extra control input to guide the camera placement, if and when necessary. Both proposed controllers have been implemented and evaluated in teleoperation experiments and user studies. The results of these experiments confirm the effectiveness of these controllers and demonstrate significant improvements compared to other existing controllers from the literature included in the studies.</p> / Master of Applied Science (MASc)

Image interpolation is the process of generating a new image utilizing a set of available images. The available images may be taken with a camera at different times, or with multiple cameras and from different viewpoints. Usually, the interpolation problem in the first scenario is called Frame Rate-Up Conversion (FRUC), and the second one view synthesis. This thesis focuses on image interpolation and addresses both FRUC and view synthesis problems. We propose a novel FRUC method using optical flow motion estimation and a patch-based reconstruction scheme. FRUC interpolates new frames between original frames of a video to increase the number of frames, and increases motion continuity. In our approach first, forward and backward motion vectors are obtained using an optical flow algorithm, and reconstructed versions of the current and previous frames are generated by our patch-based reconstruction scheme. Using the original and reconstructed versions of the current and previous frames, two mismatch masks are obtained. Then two versions of the middle frame are generated using a patch-based scheme, with estimated motion vectors and the current and previous frames. Finally, a middle mask, which identifies the mismatch areas of the two middle frames is reconstructed. Using these three masks, the best candidates for interpolation are selected and fused to obtain the final middle frame. Due to the patch-based nature of our interpolation scheme most of the holes and cracks will be filled. Although there is always a probability of having holes, the size and number of such holes are much smaller than those that would be generated using pixel-based mapping. The rare holes are filled using existing hole-filling algorithms. With fewer and smaller holes, simpler hole-filling algorithms can be applied to the image and the overall complexity of the required post processing decreases. View synthesis is the process of generating a new (virtual) view using available ones. Depending on the amount of available geometric information, view synthesis techniques can be divided into three categories: Image Based Rendering (IBR), Depth Image Based Rendering (DIBR), and Model Based Rendering (MBR). We introduce an adaptive version, patch-based scheme for IBR. This patch-based scheme reduces the size and number of holes during reconstruction. The size of patch is determined in response to edge information for better reconstruction, especially near the boundaries. In the first stage of the algorithm, disparity is obtained using optical flow estimation. Then, a reconstructed version of the left and right views are generated using our adaptive patch-based algorithm. The mismatches between each view and its reconstructed version are obtained in the mismatch detection steps. This stage results in two masks as outputs, which help with the refinement of disparities and the selection of the best patches for final synthesis. Finally, the remaining holes are filled using our simple hole filling scheme and the refined disparities. The adaptive version still benefits from the overlapping effect of the patches for hole reduction. However, compared with our fixed-size version, it results in better reconstruction near the edges, object boundaries, and inside the highly textured areas. We also propose an adaptive patch-based scheme for DIBR. The proposed method avoids unnecessary warping which is a computationally expensive step in DIBR. We divide nearby views into blocks, and only warp the center of each block. To have a better reconstruction near the edges and depth discontinuities, the block size is selected adaptively. In the blending step, an approach is introduced to calculate and refine the blending weights. Many of the existing DIBR schemes warp all pixels of nearby views during interpolation which is unnecessary. We show that using our adaptive patch-based scheme, it is possible to reduce the number of required warping without degrading the overall quality compared with existing schemes. / Thesis / Doctor of Philosophy (PhD)

Human pose estimation (HPE) is an ever-growing research field, with an increasing number of publications in the computer vision and deep learning fields and it covers a multitude of practical scenarios, from sports to entertainment and from surveillance to medical applications. Despite the impressive results that can be obtained with HPE, there are still many problems that need to be tackled when dealing with real-world applications. Most of the issues are linked to a poor or completely wrong detection of the pose that emerges from the inability of the network to model the viewpoint. This thesis shows how designing viewpoint-equivariant neural networks can lead to substantial improvements in the field of human pose estimation, both in terms of state-of-the-art results and better real-world applications. By jointly learning how to build hierarchical human body poses together with the observer viewpoint, a network can learn to generalise its predictions when dealing with previously unseen viewpoints. As a result, the amount of training data needed can be drastically reduced, simultaneously leading to faster and more efficient training and more robust and interpretable real-world applications.

Summaries in Afrikaans and English / Text in Afrikaans / Die indikatiewe (inisiele) aspek van die heiliging word genadiglik deur God aan die gelowiges geskenk tydens hul regverdiging en wedergeboorte sodat die imperatiewe (voortgaande) aspek daaruit kan voortspruit. God ken in eersgenoemde aspek aan die gelowiges die nuwe status van heiliges as gawe toe, en rig hulle op Homself deur hulle met Christus te verenig in mede-sterwe vir die sonde, en in mede-opstanding in sy nuwe lewe. In kontinui'teit hiermee leef die gelowiges die implikasies van hul ontvangde status uit deur in die Militia Christiana die sonde(s) toenemend af te le, en hulle toenemend met Christus te beklee, en in sy krag deur die Gees vir God te lewe. So groei hulle in laasgenoemde aspek. Die Gereformeerde leer, waarvolgens geglo word dat 'n tweede genadewerk of 'n latere Geesdoop onnodig is vir effektiewe groei, laat die beste reg geskied aan die kontinui'teit tussen die twee aspekte. / The indicative (initial) aspect of sanctification is graciously given to the believers by God when they are justified and regenerated, so that the imperative (ongoing) aspect can flow therefrom. In the first-mentioned aspect God confers upon the believers the new status of saints, and focuses them on Himself by uniting them with Christ in co-crucifixion for sin, and in co-resurrection with Him in his new life. In continuation thereof, the believers live out the implications of their received status in the Militia Christiana by putting away sin, and by putting on Christ in an ever-increasing measure, and in living for God in Christ's power by the Spirit, thus growing in the latter aspect. The Reformed teaching, according to which it is believed that a second work of grace or a subsequent baptism with the Spirit is unnecessary for effectual growth, is most in accordance with the continuation between the two aspects. / Philosophy, Practical & Systematic Theology / M. Th. (Systematic Theology)

Summaries in Afrikaans and English / Text in Afrikaans / Die indikatiewe (inisiele) aspek van die heiliging word genadiglik deur God aan die gelowiges geskenk tydens hul regverdiging en wedergeboorte sodat die imperatiewe (voortgaande) aspek daaruit kan voortspruit. God ken in eersgenoemde aspek aan die gelowiges die nuwe status van heiliges as gawe toe, en rig hulle op Homself deur hulle met Christus te verenig in mede-sterwe vir die sonde, en in mede-opstanding in sy nuwe lewe. In kontinui'teit hiermee leef die gelowiges die implikasies van hul ontvangde status uit deur in die Militia Christiana die sonde(s) toenemend af te le, en hulle toenemend met Christus te beklee, en in sy krag deur die Gees vir God te lewe. So groei hulle in laasgenoemde aspek. Die Gereformeerde leer, waarvolgens geglo word dat 'n tweede genadewerk of 'n latere Geesdoop onnodig is vir effektiewe groei, laat die beste reg geskied aan die kontinui'teit tussen die twee aspekte. / The indicative (initial) aspect of sanctification is graciously given to the believers by God when they are justified and regenerated, so that the imperative (ongoing) aspect can flow therefrom. In the first-mentioned aspect God confers upon the believers the new status of saints, and focuses them on Himself by uniting them with Christ in co-crucifixion for sin, and in co-resurrection with Him in his new life. In continuation thereof, the believers live out the implications of their received status in the Militia Christiana by putting away sin, and by putting on Christ in an ever-increasing measure, and in living for God in Christ's power by the Spirit, thus growing in the latter aspect. The Reformed teaching, according to which it is believed that a second work of grace or a subsequent baptism with the Spirit is unnecessary for effectual growth, is most in accordance with the continuation between the two aspects. / Philosophy, Practical and Systematic Theology / M. Th. (Systematic Theology)

In Modellen der Gesichtererkennung (Bruce & Young, 1986) wird eine Verarbeitungsstufe postuliert, auf welcher strukturelle Gesichterrepraesentationen im Langzeitgedaechtnis (LZG) innerhalb von sogenannten Face Recognition Units (FRUs), abgerufen werden. Frühere Studien (z.B. Pfütze, Sommer & Schweinberger, 2002) zeigten eine Komponente in den ereigniskorrelierten Potentialen (EKPs), welche die Aktivierung solcher FRUs anzeigen - der frühe Wiederholungseffekt (oder early repetition effect). Dieser zeigt sich bei wiederholter Darbietung bekannter Gesichter um 250-350 ms mit frontaler Positivierung und temporal inferiorer Negativierung als ein EKP auf die zweite Praesentation. In Experiment I und II dieser Dissertation wurde der ERE als Marker der Aktivierung struktureller Gesichterrepraesentationen im LZG durch ein Prime-Target-Paradigma mit einer perzeptuellen Rueckwaertsmaskierung validiert. Die Maskierung mit einem unbekannten Gesicht ergab im Vergleich zu einem zerwürfelten Gesicht und der unmaskierten Bedingung eine spezifische Topographie des ERE, die den gedächtnisbasierten Anteil des ERE widerspiegelt. Dieses Paradigma wurde in Experiment III eingesetzt, um die Blickwinkelabhängigkeit des ERE zu untersuchen. Die Ergebnisse deuten darauf hin, daß Gesichterdurch mehr als nur blickwinkelabhängige, zweidimensionale Abbilder repräsentiert sind, da zumindest für Frontalgesichter ein ERE sogar über eine Blickwinkelabweichung von 90 Grad gezeigt werden konnte. Dies ist vereinbar mit einer "dualen Strategie" der Gesichtererkennung (z.B. O''Toole, Edelman & Bülthoff, 1998) wobei sowohl Konfiguration als auch Einzelmerkmale eines Gesichts abgerufen werden. Die Ergebnisse sind vereinbar mit Bruce und Young (1986), welche FRUs als Einheiten konzipierten, in denen konfigurale und einzelne Merkmale verschiedener Blickwinkel miteinander verbunden sind, was in diesem Sinn einer objektzentrierten, dreidimensionalen Speicherung eines individuellen Gesichts entspricht. / Models of face recognition (Bruce & Young, 1986) postulate a processing stage where face recognition units (FRUs) are accessed. FRUs are thought to be structural representations of familiar faces stored in long-term memory (LTM). Previous work (e.g. Pfütze, Sommer & Schweinberger, 2002), suggested the existence of a component in the event-related-potential (ERP) which signals FRU activation - the early repetition effect (ERE). The ERE can be observed when familiar faces are shown repeatedly, as an ERP around 250-350 ms that is more positive at fronto-central and more negative at inferior temporal sites for the second presentation of the face. In Experiment I and II of the present dissertation the ERE was validated as a marker of FRU activation in LTM by using a repetition priming paradigm with backward masking. Compared to the scrambled and non-mask conditions, the ERE revealed a distinguishable topography when an unfamiliar face interspersed prime and target, demonstrating contributions of face identity codes to the ERE. In Experiment III, this paradigm was used to analyse the viewpoint-dependency of the ERE. Results of Experiment III indicate that facial representations are more than viewpoint-dependent, two-dimensional images of a face, because even under a 90° view deviance between prime and target an ERE was detectable at least for frontal targets. This reconciles a dual-strategy in face recognition (e.g. O’Toole, Edelman & Bülthoff, 1998). Accordingly, facial representations can be assumed as a combination of image-based views, reflecting their configuration and single features. Results are in line with Bruce and Young (1986) who postulated that FRUs contain both features and configurations of distinct head angles in an interlinked manner. In that sense, FRUs can be described as three-dimensional analogues of the face they represent making object-centered recognition on an individual level possible.

Depois do êxito de Cervantes, o romance na Espanha permaneceu quase dois séculos à margem de outras formas literárias até ser retomado por um de seus maiores representantes na história da literatura espanhola: Benito Pérez Galdós. Um aspecto que tem importância central no conjunto de sua obra é a construção do personagem. Fortunata y Jacinta, um de seus romances que melhor representa a escritura realista do século XIX, traz em sua estrutura a questão do processo de individualização de um personagem, Fortunata. A protagonista é apresentada na obra como um personagem-tipo representante do povo, ou mesmo de uma classe social em formação na Espanha do século XIX: o proletariado urbano. Durante grande parte de sua trajetória é um personagem espectador de si mesmo, na medida em que suas imagens são compostas por seu entorno constituído de personagens da alta e baixa burguesia, no geral protagonistas dos romances de Galdós que precederam Fortunata y Jacinta. A complexidade desse personagem se concretizará como resultado de sua assimilação das imagens objetivas de si, produzidas por distintos pontos de vista lançados pelas instâncias narrativas do romance. O ponto de apoio externo ao conhecimento objetivo de si do personagem central se constitui das diferentes perspectivas dos personagens que o rodeiam, do juízo e movimentações do narrador que se desdobra em ficcionalizador do relato, das marcas do autor na obra e da interpretação do leitor / After Cervantes\'s achievement, the novel in Spain lagged almost two centuries behind other literary forms, until its eventual retake by one of its foremost representatives in the Spanish literary history: Benito Pérez Galdós. A feature of crucial importance in the whole of his work is character construction. Fortunata y Jacinta, one of the most representative among his novels of nineteenth-century realistic writing, brings along in its structure the question about the individualisation process of a character (Fortunata). The protagonist is portrayed in this work as a type-character representative of the people, or rather of a social class which was emerging in nineteenth-century Spain: the urban working class. During a considerable portion of her course in life she is a beholder of herself, since her images are shaped by her surroundings, which comprise characters from both the high and the low bourgeoisie, generally protagonists of novels Galdós wrote before Fortunata y Jacinta. This character\'s complexity materialises as a result of her assimilating objective images of herself produced by different viewpoints cast by the novel\'s narrative instances. The outward support for the main character\'s objective self-knowledge is made up of the diverse perspectives of the characters around her, of the opinions and movements of the narrator (who ends up becoming a fictionalizer of the account), of marks left by the author in the work and of the reader\'s interpretation