Publications of Dr. Oscar Meruvia-Pastor & Collaborators

I work at the interphase between computer graphics and human-computer interaction. More precisely, I have done research in telepresence systems, multimedia applications, medical imaging, non-photorealistic rendering and visibility pre-processing algorithms.

My current research interest is in human-centered solutions for 3D telepresence, remote training, immersive Virtual Reality, Augmented Reality and telecollaboration. Together with several colleagues, Computer Science Graduate Students under my supervision are working or have worked on gesture-based interaction using depth-sensing devices, 3D-enhanced telecollaboration systems and algorithms for obtaining depth from stereoscopic imagery.


3D Telepresence for Education, Training and Games


A User Study on Augmented Virtuality Using Depth Sensing Cameras for Near-Range Awareness in Immersive VR

Smartphone access within immersive VR displays is severely limited by the fact that when users need to make use of their smartphone, they usually have to remove their HMDs. One way of dealing with this issue is to move towards an Augmented Virtuality (AV) design. AV usually incorporates part of the physical world into the virtual world with the assistance of one or more cameras. In this article we present Near-Range Augmented Virtuality (NRAV), where we use a portable RGBD camera mounted on an HMD to provide near-range awareness, facilitating access to the users' personal mobile device while remaining inside the VR. In this user study we compare NRAV against SDSC, the Smartphone Detector based on a Statistical Classifier developed by Desai et al. The study covers a variety of tasks associated with the daily operation of a smartphone within a VR context, including reading and typing messages, answering phone calls, and navigating within the VR environment. Results of the study show that NRAV allows users to perform the tasks presented to the users and is preferred by the majority for most of the tasks.
( extended manuscript; online article @ WEVR 2018; video (4:57); Seg3D code @ GitHUB; reference).

Augmented Reality as a Telemedicine Platform for Remote Procedural Training

Traditionally, rural areas in many countries are limited by a lack of access to health care due to the inherent challenges associated with recruitment and retention of healthcare professionals. Telemedicine, which uses communication technology to deliver medical services over distance, is an economically sensible and potentially effective way to address this problem. In this research, we develop a new telepresence application using an Augmented Reality (AR) system that enables remote learners to train for complex medical procedures such as Point of Care Ultrasound (PoCUS) without visual interference. This research uses the Microsoft HoloLens and the Leap Motion sensor to capture the first-person view of a simulated rural emergency room (ER) through mixed reality capture (MRC), serving as a novel telemedicine platform with remote pointing capabilities. We validated the system through a user study. The results obtained provide valuable insight and guidance for the development of an AR-supported telemedicine platform.
(accepted manuscript; online article @ Sensors; reference).

A Window to your Smartphone: Exploring Interaction and Communication in Immersive VR with Augmented Virtuality

A major drawback of most Head Mounted Displays (HMDs) used in immersive Virtual Reality (VR) is the visual and social isolation of users from their real-world surroundings when wearing these headsets. To address this issue, we present a new method to allow people wearing VR HMDs to use their smartphones or tablets without removing their HMDs. We augment the scene inside the VR HMD with a view of the user's smartphone so that the user can interact with the device without removing the headset using additional cameras, such as the Leap Motion sensor, on top of the Oculus Rift. Our systems allows VR users to have a window to their smartphone from within the virtual world and make user of much of the functionality provided by their smartphones.
(accepted manuscript; online article @ IEEE Xplore; reference).

Operating Virtual Panels with Hand Gestures in Immersive VR Games - Experiences with the Leap Motion Controller

In this research, we explore gesture-based interaction in immersive VR games and present a novel platform for research and rapid application development at an affordable cost, while avoiding constant Android deployment during development. Using the Unity game engine, we implemented a prototype to interact with a numeric dial panel and play a game in a virtual world and we compared placements of the leap motion sensor on the desktop and on the HMD. Our results suggest that even though the tracking quality of the Leap Motion sensor was rather limited when used in the HMD for pointing and selection tasks, its performance was much better in the desk-mounted setup.
(accepted manuscript; online article @ Springer; reference).

OMARC: An Online Multimedia Application for the Assessment of Respiratory Conditions

OMARC is a multimedia application designed to support the training of health care providers in the identification of common lung sounds heard in a patient's thorax as part of a health assessment session or group of sessions. The main goal of OMARC is to effectively help health-care students and professionals to become familiar with lung sounds that might be heard during the assessment of respiratory conditions. OMARC was developed using an online platform to facilitate access to users in remote locations and is designed in a way that is easy to use and accessible to students and practitioners over the internet.
(accepted manuscript; online article @ IJMI; reference).

Evaluation of Depth-Sensing Cameras against Off-the-Shelf Controllers for Pick and Drop Tasks Using a Simulated Robot Arm Manipulator

In this article, we investigate a depth-sensing camera controller based on inverse-kinematics for placement tasks using a robot arm simulator, and describe a user study to evaluate participants' performance using short-range depth-sensing camera controllers against comparable off-the-shelf controllers.
(conference article @ HCII; reference).

Automatic and Adaptable Registration of Live RGBD Video Streams

A novel method for automatic registration of video streams originated from two depth-sensing cameras is shown. The motivation of this work is to facilitate the use of RGBD sensors for non-expert users, so that cameras need not to be calibrated, and if cameras are moved, the system will automatically recover the alignment of the video streams. DeReEs is used, since it is fast and successful in registering scenes with small overlapping sections.
(conference article @ SIGGRAPH's MIG; poster @ SIGGRAPH; video (36.5MB); reference).

*Student Research Competition Semi-finalist Poster at SIGGRAPH 2015 *

Robot Arm Manipulation Using Depth-Sensing Cameras and Inverse Kinematics

This article presents a new method in which the user just needs to point towards the target and a robotic arm will reach the target itself using inverse kinematics algorithms and a depth-sensing camera to capture the user commands. The advantage of this approach is that manipulation of the arm needs less training and is easier to learn compared to more complex input devices. We test our approach with a robot arm simulator similar to those used in industrial applications.
(conference article @ MTS/IEEE's Oceans ; poster abstract @ GI; poster @ OERA-NS; reference).

*Best HCI Poster at Graphics Interface (AI/GI/CRV) 2014 *

A Comprehensive Framework for Evaluation of Stereo Correspondence Solutions in Immersive Augmented and Virtual Realities

In this work we present a comprehensive evaluation scheme for stereo correspondence methods in outdoor Augmented Reality applications based on the processing speed requirements in AR and the necessary accuracy of depth results for the human visual system (HVS) at different ages. For a given stereo-correspondence solution, various metrics are estimated and reported in the framework of outdoor augmented reality: Average disparity error, average number of outliers, average stereoacuity and execution times.
(Journal article @ JVRB, reference; conference article @ VISAPP, reference,Graphics Interface poster).

DeReEs: Real-Time Registration of RGBD Images Using Image-Based Feature Detection And Robust 3D Correspondence Estimation and Refinement

DeReEs, an algorithm to align color + depth image pairs obtained from commercial depth cameras to build an extended model of the scene captured by both cameras is presented. The challenge is to work with input images taken from the same scene from different positions, with small overlapping regions, with the potential to support real-time processing.
(conference article @ ICVNZ; Graphics Interface poster; reference).


Information Visualization

 

Improving Perception Accuracy in Bar Charts with Internal Contrast and Framing Enhancements

Depending on the layout and embellsihments of a bar chart, accuracy in the perception of the values represented can vary significantly. In this research we performed a study to analyze the effect of several minimalistic enhancements on bar charts. Results from a series of user studies described in this article demonstrate that people are able to provide more acurate estimates of the values represented in bar charts through the addition of minimalistic visual elements, such as a single gridline on the top of the chart, the use of quantized gradients encoded in the bars, or Tufte's internal divisor lines.
(accepted manuscript, reference).

 

GeNET: a web application to explore and share Gene Co-expression Network Analysis data

In this article we introduce GeNET, a web service designed to facilitate sharing and study of Gene Co-expression Network Analysis (GCNA) data. For researchers interested in sharing their GCNA data, GeNET provides a convenient interface to upload their data and automatically make it accessible online. For researchers interested in exploring GCNA data published by others, GeNET provides an intuitive online tool to interactively explore GCNA data. In addition, GeNET allows users to download all or part of the published data for further computational analysis.
(open access article @ Peerj, Genet Website, reference).

 


Biomedical Visualization

 

Estimating Cell Distribution in Histological Samples Based on Incremental Search and Color Filtering

Incremental Cell Search (ICS) is a software that expedites the analysis of relationships between morphological outgrowth and cell proliferation in embryonic tissues. Based on an estimated average cell size and stain color, ICS rapidly indicates the approximate location and amount of cells in histological images of labeled embryonic tissue and even provides estimates of cell counts in regions with saturated fluorescence and blurred cell boundaries.
(open access article @ IJBI, ICS software distribution , reference).

 

Integrative Visualization of Temporally Varying Medical Image Patterns

We present software for the visualization of temporal changes of disease patterns using stacks of medical images collected in time-series experiments. With this software users can generate 3D surface models representing disease patterns and observe changes over time in size, shape, and location of clinically significant image patterns. Statistical measurements of the volume of the observed disease patterns can also be obtained simultaneously.
(journal article, online article @JIB, reference).

Generation of Connectivity-Preserving Surface Models of Multiple Sclerosis Lesions

In this article we introduce a software tool to build 3D surface models of Multiple Sclerosis (MS) brain lesions from 2D image stacks typically obtained through Magnetic Resonance Imaging (MRI). This tool allows users to create 3D models of MS lesions quickly and visualize the lesions and brain tissue using various visual attributes and configurations.
(journal article, online version @ google books, online version @ SHTI, reference).

A Novel MRI Visualization Tool for White Matter Pathology in Multiple Sclerosis

We developed a tool to visualize MS lesions and their 3D surface models to show changes in the lesions over time. These can be shown as an animation to elucidate differences across scanning sessions. With this software, a volumetric sub-region can be selected from the 3D model for zooming or animation, and a point on the 3D model can be selected to highlight all lesions connected to it. The total volume of lesions can be calculated, displayed as a chart, and exported.
(poster, online abstract @ SfN, reference).

Concurrent Visualization of and Mapping between 2D and 3D Medical Images for Disease Pattern Analysis

We present a software tool to highlight and display regions of interest in 2D medical images and their 3D mesh model counterparts. This tool can help researchers visualize and compare brain lesions and tissues in 2D and 3D at the same time.
(conference article, online article @ VBL, reference).

Building generic anatomical models using virtual model cutting and iterative registration

Using 3D generic models to statistically analyze trends in biological structure changes is an important tool in morphometrics research. In this article we show how the creation of a generic model that captures spatial information exploitable in statistical analyses is facilitated by coupling our generalized segmentation method to existing automatic image registration algorithms.
(open access article @ BMC-MI, reference).

Fast Interactive Integration of Cross-sectional Image Datasets and Surface Data for Morphometric Analysis

A system to align a stack of images in parallel for the reconstruction of a 3D data structure and surface model from a set of histological cross-sections is presented, along with complementary tools for interactive alignment. introduced.
(journal article, online article @ SHTI, poster @ iCore, reference).

An Efficient Virtual Dissection Tool to Create Generic Models for Anatomical Atlases

A method is presented to combine 3D mesh cutting tools to extract a particular object from a polygonal mesh of amalgamated structures. With this technique it is possible to extract individual objects, such as the jawbone from a skull, from a polygonal model.
(journal article, online article @ SHTI, reference).


Non-Photorealistic Rendering

 

Adaptive Incremental Stippling Using the Poisson Disk Distribution

This paper presents a method to produce high-quality stippled renditions with an emphasis in detail enhancement, using edge information contained in the input image.
(full text paper, online article @ JGGPUGT, reference).

 

Poisson-Disk Distributed Dots and its Applications in Non-Photorealistic Rendering (Puntos con Distribución Poisson Disk y su Uso en NPR)

This paper, written in spanish and presented in Mexico, presents two techniques to produce stippled renditions faster than traditional methods at the expense of quality.
(full text paper, reference).

 

OPENNPAR: A System for Developing, Programming, and Designing Non-
Photorealistic Animation and Rendering

In this article, co-written with Nick Halper et al, we present an Open-Source project to implement a library of algorithms commonly used in non-photorealistic rendering.
(full text paper, reference).

Graph-Based Point Relaxation for 3D Stippling

A new way to distribute point particles on the surface of a model that also satisfy the requirement that they belong in a point hierarchy of well-distirbuted points on several levels of detail is proposed. We also introduce the concept of using stipples to convey deformable semitransparent surfaces.
(full text paper, videos, reference).

 

Real-Time Animated Stippling

A point hierarchy mesh is created on the surface of a deformable 3D model. The point hierarchy is used to convey the stippling style used by many artists. Because the point hierarchy is attached to the surface of the mesh, the points enable frame-coherent rendering on the particle level
(full text paper, videos, reference).

 

Frame-Coherent Stippling

The foundations for creating a hierarchy of points based on mesh simplification and mesh subdivision of steps that can be used to convey a 3D-stippling style is introduced here. The technique is applied on models with fixed geometry.
(full text paper,poster, reference).

 


Visibility Precomputation Techniques

 

Visibility Preprocessing Using Spherical Sampling of Polygonal Patches

This paper builds up on the concept of visibility precomputation, and presents a more robust approach, where polygons are organized in contiguous regions of polygonal patches. The size of this bitfields is relatively small and the technique yields a dramatic reduction in the number of polygons sent to the graphics processor, so the response time is improved significantly for models with many folds or self-occluding surfaces.

(full text paper, poster, videos, reference).

Approximated View Reconstruction Using Precomputed ID-Bitfields

A method to pre-compute visibility of the polygons of a 3D mesh in order to cull away the non-visible polygons is presented. The advantage of visibility precomputation is that, for any arbitrary viewpoint, a reduced numer of polygons is sent to the graphics processor. A list of pre-computed polygons is encoded in a long bit list called ID-bitfield.
(full text paper, presentation, reference).


Dissertation & Thesis

 

Frame-Coherent 3D Stippling for Non-photorealistic Computer Graphics
Oscar Meruvia-Pastor, Dr. Ingenieur (Ph.D.) Dissertation

The foundations for frame-coherent 3D stippling are described here, as well as methods for rendering animated and static models in real time. In addition, several applications for 3D stippling in archaeology, anatomy and animated semi-transparent surfaces are presented.
(full text of dissertation (25MB), videos, videos referred, reference).

Level of Detail Selection and Interactivity
Oscar Meruvia-Pastor, M.Sc. Thesis

A new method to switch between different levels of detail is proposed. The method aims at improving the quality of interaction by reducing the response time based on a measure of divergence between the expected and the actual positions of a subject.
(full text of thesis, presentation, reference).


Theses and Reports of the Graduates of the

3D Telepresence & Telecollaboration Group

 

A Window to your Smartphone: Augmented Virtuality Using Depth Sensing Cameras for Near-Range Awareness
Ghassem Alaee, M.Sc. Project

We present an alternative to allow people wearing VR HMDs to use their smartphones without removing their HMDs. Using an RGBD camera, we augment the VR scene with a view of the user's smartphone so that the user can use it without removing the headset. We call this method NRAV, for Near-Range Augmented Virtuality. We validate this method with a user study that compares NRAV with SDSC, the method developed by Desai et Co.
(full text, reference).

A Window to your Smartphone: Exploring Interaction and Communication in Immersive VR with Augmented Virtuality
Amit P. Desai, M.Sc. Thesis

We present a new method to allow people wearing VR HMDs to use their smartphones without removing their HMDs. To do this, we augment the VR scene with a view of the user's smartphone so that the user can use it without removing the headset. The idea involves the use of the Leap Motion sensor to capture the user's real-world surrounding and augment the virtual world with the content displayed on the smartphone's screen. This setup allows VR users to have a window to their smartphones from within the virtual world.
(full text, reference).

Augmented Reality as a Telemedicine Platform for Remote Procedural Training
Shiyao Wang, M.Sc. Thesis

We develop a new telepresence application using an augmented reality (AR) system that allows users to perform complex medical procedures without visual interference. We explore the use of the Microsoft HoloLens to facilitate and enhance remote medical training for Point of Care Ultrasound (PoCUS). The HoloLens is used to capture the first-person view of a simulated rural emergency room (ER) through mixed reality capture (MRC). The mentor's hand gestures are captured using a Leap Motion and virtually displayed in the AR space of the HoloLens, which allows mentors to provide remote instruction.
(full text, reference).

Automatic and Adaptable Registration of Live RGBD Video Streams Sharing Partially Overlapping Views
Afsaneh Rafighi, M.Sc. Thesis

This work introduces DeReEs-4v, an algorithm for unsupervised and automatic registration of RGBD video streams from two depth-sensing cameras placed side-by-side in an indoor space. DeReEs-4V registration can be used to find the transformation of the view of one camera with respect to the other at interactive rates. The motivation of this research is to employ multiple depth-sensing cameras to enlarge the field of view and acquire a more complete and accurate 3D information of the environment.
(full text, reference).

Robot Arm Manipulation Using Depth-Sensing Camera and Inverse Kinematics
Akhilesh K. Mishra, M.Sc. Thesis

This thesis describes a new method in which the user just needs to point towards the target and a robotic arm will reach the target itself using inverse kinematics algorithms and a depth-sensing camera to capture the user commands. In addition, a user study to evaluate participants' performance using short-range depth-sensing camera controllers against comparable off-the-shelf controllers is described in detail.
(full text, reference).

DeReEs: Real-Time Registration of RGBD Images Using Image-Based Feature Detection and Robust 3D Correspondence Estimation and Refinement
Sahand Seifi, M.Sc. Thesis

DeReEs, an algorithm to align color + depth image pairs obtained from commercial depth cameras to build an extended model of the scene captured by both cameras, is presented in this work. The challenge is to work with input images taken from the same scene from different positions, with small overlapping regions, with the potential to support real-time video processing.
(full text, reference).

A Comprehensive Approach for Evaluation of Stereo Correspondence Solutions in Outdoor Augmented Reality
Bahar Pourazar, M.Sc. Thesis

This thesis describes a comprehensive evaluation scheme for stereo correspondence methods in outdoor Augmented Reality applications based on the processing speed requirements in AR and the necessary accuracy of depth results for the human visual system (HVS) at different ages. For a given stereo-correspondence solution, various metrics are estimated and reported in the framework of outdoor augmented reality: Average disparity error, average number of outliers, average stereoacuity and execution times.
(full text, reference).


References