Abstracts of Published Works
Towards Mobile HDR Video
Computer Graphics forum, 2011.
Abstract: We present a novel method for High Dynamic Range video where the critical phases of the pipeline are based on histograms. It is possible to achieve high framerates, since the algorithm generates one HDR frame per captured frame. Also, the method is of low computational cost, making it particularly suited for devices with less powerful processors. An implementation of the capture process for the Nokia N900 smartphone, using the recent FCam API, is detailed.
In collaboration with Tássio Knop, Alexandre Chapiro and Luiz Velho.
Dance to the Music / Play to the Motion
Video, in SIGMAD Animation Festival, 2010 (Won the first prize for the Technical Video category.)
Abstract: In this video we illustrate how the analysis of a samba dancer's movement can be used to synthesize an original samba-style music piece. First, the dance is captured using motion capture technology. Then, the performance is segmented according to the boundaries of the musical phrases, and the similarities between movements are analyzed in order to build a measure-synchronous motion graph. Finally, a constrained random walk in such a graph guides the composition of a new song. The automatic composition algorithm makes use of high level parameters, such as the melody instrument and the number of notes per beat, which are provided by the random walk in the motion graph.
In collaboration with Adriana Schulz, Luiz Velho, Aldo Zang and Bruno Madeira.
Playing the QWERTY Keyboard
Poster, in Siggraph, 2010
Abstract: Computer hardware and music softwares have evolved to a so good level that, in our days, its possible to compose high quality music using only a simple laptop equipped with the proper applications. However, the interface for entering musical notes in softwares running on laptops has not evolved. If music writers and performers want a reasonable way to interact with musical applications, they usually have to make use of an external MIDI Keyboard, i.e., a piano-like interface, what can be a problem when mobility is a concern. This work has two purposes. First, to introduce a simple mapping between the keys of the QWERTY keyboard and the notes of pentatonic and heptatonic musical scales, this way presenting an alternative to the external musical interfaces. Second, to claim computer manufacturers to implement the necessary properties for the QWERTY keyboard to be fully used as a musical input device, especially regarding polyphony and key-down velocity.
In collaboration with Paulo Cezar Carvalho.
Motion Scoring
Poster, in Siggraph, 2010
Abstract: The intention of this work is to explore how motion can be used to guide music composition. We introduce a method for synthesizing an original music piece based on the analysis of a dancer's movement. First, the dance is captured using motion capture technology. Then, the performance is segmented according to the boundaries of the musical phrases, and the similarities between movements are analysed in order to build a measure-synchronous motion graph. Finally, a constrained random walk in such a graph guides the composition of a new song. The automatic composition algorithm makes use of high level parameters, such as the melody instrument and the number of notes per beat, which are provided by the random walk in the motion graph.
In collaboration with Adriana Schulz and Luiz Velho.
Guitar-Leading Band
Poster, in Siggraph, 2010 (Student Research Competition semifinalist.)
Abstract: Interfacing with the guitar using the audio signal is one of the oldest problems in Computer Music, and advances in the area were astonishing. In our days it is possible to simulate a huge range of amplifiers, apply many filter effects and evaluate the pitch of a plucked string robustly, to mention a few useful applications. However, there are problems very hard to solve using audio, like recognizing the chord being played when the musician is not down- or up-stroking all strings at once, but picking them one at a time. In this work we explore the visual interface of the guitar, a subject that only in recent years has received the proper attention. Three aspects are treated. The first has just been mentioned: the problem of chord recognition when not all notes of the chord are played. The second relates to the implementation of an automatic composition algorithm inspired on the bi-dimensional nature of the representation of the diatonic scale in the guitar fretboard. Finally, the knowledge about the current chord, or simply the rough position of the hand in the guitar fretboard, allows controlling some parameters of the automatic composition algorithm, what becomes specially interesting in live performance.
In collaboration with Luiz Velho, Paulo Cezar Carvalho and Giordano Cabral.
On Bimodal Guitar-Chord Recognition
Full Paper, in ICMC, 2010
Abstract: We discuss the use of visual information to aid the task of recognizing guitar-chords in real time. The video-based method is analogous to the state-of-the-art audio-based counterpart, relying on a supervised Machine Learning algorithm applied to a visual chord descriptor. The visual descriptor consists of the rough position of the fingers in the guitar fretboard, found by using special markers attached to the middle phalanges and fiducials attached to the guitar body. Experiments were conducted regarding classification accuracy comparisons between methods using audio, video and the combination of the two signals. Four different data-fusion techniques were evaluated: feature fusion, sum rule, product rule and an approach in which the visual information is used as prior distribution, which resembles the way humans recognize chords being played by a guitarist.
In collaboration with Paulo Cezar Carvalho and Luiz Velho.
Plane Tessellation With Musical Scale Tiles and Bidimensional Automatic Composition
Poster, in ICMC, 2010
Abstract: We present a method for arranging the notes of certain musical scales (pentatonic, heptatonic, Blues Minor and Blues Major) on the plane by using a plane tessellation with especially designed musical scale tiles. The motivation of such a representation is described, as well as the mathematical analysis of the possibility of its realization. As a main application of the idea we introduce a bidimensional automatic composition algorithm, at which we also explore the target-note improvisation paradigm, by using Markov Chains conditioned to certain events.
In collaboration with Tertuliano Franco and Paulo Cezar Carvalho.
The Song Picture: On Musical Information Visualization for Audio and Video Editing
Full Paper, in Ivapp, 2010
Abstract: Most audio and video editors have a representation of music pieces consisting of the Pulse Code Modulation waveform, from which no much information, besides the energy envelope, can be visually captured. In the present work we discuss the importance of presenting to the user some hint about the melodic content of the audio to improve the visual segmentation of a music piece, thereby speeding up the edition work. Such a representation, yet powerfull in the mentioned sense, is of simple computation, a property desirable because loading an audio or video can not be a process which takes much time.
In collaboration with Paulo Cezar Carvalho.
Visual Pitch Class Profile: A Video-Based Method for Real-Time Guitar Chord Identification
Poster, in Visigrapp, 2010
Abstract: We propose a video-based method for real-time guitar chord identification which is analogous to the state-of-the-art audio-based method. While the method based on audio data uses the Pitch Class Profile feature and supervised Machine Learning techniques to “teach” the machine about the chord “shape”, we use as feature the approximated positions of fingertips in the guitar fretboard (what we call Visual Pitch Class Profile), captured using especial hardware. We show that visual- and audio-based methods have similar classification performance, but the former outperforms the latter with respect to the immunity to noise caused by strumming.
In collaboration with Marcelo Gattass, Luiz Velho and Paulo Cezar Carvalho.
EigenSound: Song Visualization for Edition Purposes
Poster, in Sibgrapi, 2009
Abstract: In this work we propose a representation of audio files more suitable for the task of editing audio and video than the traditional waveform graph, in the sense that it helps finding clusters and onsets visually, due to the array of colors associated to each previously computed segment. Segmentation and color-mapping are done by means of well known image processing techniques: bidimensional convolution and spectral decomposition, respectively.
In collaboration with Paulo Cezar Carvalho.
The Blues Machine
Video Festival, in Sibgrapi, 2009
Shord description: A multi-touch tangible interface for 12-bar blues improvisation.
In collaboration with Ilana Paterman, Luiz Velho and Paulo Cezar Carvalho.
Métodos Matemáticos e Computacionais em Música (in Portuguese)
Book and Mini Course, in CNMAC, 2009
Short description: A brief survey about mathematical and computational techniques applied to music, covering subjects like signal processing, rhythm, percussion, melolody, harmony, automatic composition and new technologies.
In collaboration with Sergio Krakowski, Paulo Cezar Carvalho and Luiz Velho.
Blues Machine
iPhone and iPod Touch Software, in Apple's App Store, 2009
Short description: A smart-phone implementation of The Blues Machine (see below).
The Blues Machine
Poster and Talk, in Siggraph, 2009
Shord description: A multi-touch tangible interface for 12-bar blues improvisation.
In collaboration with Ilana Paterman, Luiz Velho and Paulo Cezar Carvalho.
Relativistic Visualization
Video Festival, in Sibgrapi, 2007 (Best video award.)
Short description: Scientific visualization of Special Relativity concepts, using Ray Tracing and Image Based Rendering techniques.
In collaboration with Luiz Velho and Ilana Paterman.