To work at scale, a complete image indexing system comprises two components: An inverted file index to restrict the actual search to only a subset that should contain most of the items relevant to the query; An approximate distance computation mechanism to rapidly scan these lists. While supervised deep learning has recently enabled improvements to the latter, the former continues to be based on unsupervised clustering in the literature. In this work, we propose a first system that learns both components within a unifying neural framework of structured

Learning a Complete Image Indexing Pipeline

This article presents an empirical study that investigated and compared two “big data” text analysis methods: dictionary-based analysis, perhaps the most popular automated analysis approach in social science research, and unsupervised topic modeling (i.e., Latent Dirichlet Allocation [LDA] analysis), one of the most widely used algorithms in the field of computer science and engineering. By applying two “big data” methods to make sense of the same dataset—77 million tweets about the 2012 U.S. presidential election—the study provides a starting point for scholars to evaluate the efficacy and validity of different computer-assisted methods for conducting journalism and mass communication research, especially in the area of political communication.

Big Social Data Analytics in Journalism and Mass Communication: Comparing Dictionary-based Text Analysis and Unsupervised Topic Modeling

The ability of multimedia data to attract and keep people’s interest for longer periods of time is gaining more and more importance in the fields of information retrieval and recommendation, especially in the context of the ever growing market value of social media and advertising. In this chapter we introduce a benchmarking framework (dataset and evaluation tools) designed specifically for assessing the performance of media interestingness prediction techniques. We release a dataset which consists of excerpts from 78 movie trailers of Hollywood-like movies. These data are annotated by human assessors according to their degree of interestingness. A real-world use scenario is targeted, namely interestingness is defined in the context of selecting visual content for illustrating a Video on Demand (VOD) website. We provide an in-depth analysis of the human aspects of this task, i.e., the correlation between perceptual characteristics of the content and the actual data, as well as of the machine aspects by overviewing the participating systems of the 2016 MediaEval Predicting Media Interestingness campaign. After discussing the state-of-art achievements, valuable insights, existing current capabilities as well as future challenges are presented.

Predicting Interestingness of Visual Content

Interestingness is the quantification of the ability of an imageto induce interest in a user. Because defining and interpretinginterestingness remain unclear in the literature, we introduce inthis paper two new notions, intra- and inter-interestingness, andinvestigate a novel set of dedicated experiments.More specifically, we propose four experimental protocols:1/ object ranking with a pre-defined word list, 2/ pair-wise com-parison, 3/ image ranking and 4/ eye-tracking. We take advantageof experimenting on the same dataset to draw potential links be-tween the collected data and to state on the agreement betweensubjects. While we do not evidence a relationship between thelocal (intra) and global (inter) notions of interestingness, we doobserve correlated outputs throughout the different protocols. Be-yond the low or moderate values obtained from inter-rater agree-ment metrics, we point out the experimental reproducibility to ar-gue about the universal nature of the interestingness notions.In addition, we bring deep insights on the relationships be-tween interestingness and 7 other criteria, some of them alreadypointed out in the literature as being linked with interestingness.Unusualness and emotion seem to be the strongest enablers forinterestingness. These insights are highly relevant for future workon modeling.

Experiencing the interestingness concept within and between pictures

Inductive matrix completion (IMC) is a model for incorporating side information in form of “features” of the row and column entities of an unknown matrix in the matrix completion problem. As side information, features can substantially reduce the number of observed entries required for reconstructing an unknown matrix from its given entries. The IMC problem can be formulated as a low-rank matrix recovery problem where the observed entries are seen as measurements of a smaller matrix that models the interaction between the column and row features. We take advantage of this property to study the optimization landscape of the factorized IMC problem. In particular, we show that the critical points of the objective function of this problem are either global minima that correspond to the true solution or are “escapable” saddle points. This result implies that any minimization algorithm with guaranteed convergence to a local minimum can be used for solving the factorized IMC problem.

Global Optimality in Inductive Matrix Completion

This paper deals with the unification of local and non-local signal processing on graphs within a single convolutional neural network (CNN) framework. Building upon recent works on graph CNNs, we propose to use convolutional layers that take as inputs two variables, a signal and a graph, allowing the network to adapt to changes in the graph structure. In this article, we explain how this framework allows us to design a novel method to perform style transfer.

Learn to unify local and non-local signal processings with graph CNN

This paper presents the work done at Technicolor regardingthe MediaEval 2016 Predicting Media Interestingness Task,which aims at predicting the interestingness of individual im-ages and video segments extracted from Hollywood movies.We participated in both the image and video subtasks.

Technicolor@MediaEval 2016 Predicting Media Interestingness Task

Easy accessibility can often lead to over-consumption, as seen in food and alcohol habits. On video on-demand (VOD) services, this has recently been referred to as binge watching, where potentially entire seasons of TV shows are consumed in a single viewing session. While a user viewership model may reveal this binging behavior, creating an accurate model has several challenges, including censored data, deviations in the population, and the need to consider external influences on consumption habits. In this paper, we introduce a novel statistical mixture model that incorporates these factors and presents a first of its kind characterization of viewer consumption behavior using a real-world dataset that includes playback data from a VOD service. From our modeling, we tackle various predictive tasks to infer the consumption decisions of a user in a viewing session, including estimating the number of episodes they watch and classifying if they continue watching another episode. Using these insights, we then identify binge watching sessions based on deviation from normal viewing behavior. We observe different types of binging behavior, that binge watchers often view certain content out-of-order, and that binge watching is not a consistent behavior among our users. These insights and our findings have application in VOD revenue generation, consumer health applications, and customer retention analysis.

Just One More: Modeling Binge Watching Behavior

We present a dual-view mixture model to cluster users based on their features and latent behavioral functions. Every component of the mixture model represents a probability density over a feature view for observed user attributes and a behavior view for latent behavioral functions that are indirectly observed through user actions or behaviors. Our task is to infer the groups of users as well as their latent behavioral functions. We also propose a non-parametric version based on a Dirichlet Process to automatically infer the number of clusters. We test the properties and performance of the model on a synthetic dataset that represents the participation of users in the threads of an online forum. Experiments show that dual-view models outperform single-view ones when one of the views lacks information.

Non-parametric clustering over user features and latent behavioral functions with dual-view mixture models

This paper summarizes the computational models that Technicolor proposes to predict interestingness of images and videos within the MediaEval 2017 PredictingMedia Interestingness Task. Our systems are based on deep learning architectures and exploit the use of both semantic and multimodal features. Based on the obtained results, we discuss our findings and obtain some scientific perspectives for the task.

Multimodality and Deep Learning when predicting Media

An infrastructure-less indoor localization system is proposed based on fingerprints of light signals acquired at high frequencies. In contrast to other systems that modulate lights, the proposed system distinguishes lights by learning from training samples. Due to slight differences in the electronic components used in the construction of compact fluorescent light (CFL) and light emitting diode (LED) bulbs, the optical signals emitted by each light bulb have slight differences with other light bulbs even within the same brand and model. Light signals are digitized with a fast and accurate analog-to-digital converter (ADC) at up to 1 mega-samples/second, segmented, and mapped into the frequency domain using the Fast Fourier Transform (FFT). Spectral features based on the FFT are filtered, normalized, and used as training data for supervised machine learning algorithms. Results are provided for two classifiers of varying complexity: (1) A k-Nearest Neighbor (KNN) classifier; (2) A Convolutional Neural Net (CNN) classifier. A hardware system for indoor localization was designed to analyze the performance of the classifiers. Under certain restrictions, results show that light bulbs may be identified with high accuracy without special infrastructure for modulation. Identifying a light bulb is meant to be synonymous with identifying its associated location

INFRASTRUCTURE-LESS INDOOR LOCALIZATION USING LIGHT FINGERPRINTS

This paper addresses the example-based stylization of videos. Style transfer aims at editing an image so that it matches the style of an example. This topic has recently been investigated massively, both in the industry and academia. The difficulty lies in how to capture the style of an image. For this work we build on our previous work " Split and Match " for still pictures, based on adaptive patch synthesis. We address the issue of extending that particular technique to video, ensuring that the solution is spatially and temporally consistent. Results show that our video style transfer is visually plausible, while being very competitive regarding computation time and memory when compared to neural network approaches.

Video Style Transfer by Adaptive Patch Sampling

As a new generation of smartwatches enters the market, one common use is for displaying information such as notifications. While some content might warrant immediately interrupting a user, there is also information that might be important to display yet less urgent. It would be useful to show this content on the watch but not immediately draw the user's attention away from their primary task. In this paper, we investigate how fast three visual parameters draw a user's attention. In particular, we present data from a smartwatch user study where we examine the size, frequency, and color of a visual prompt and the associated impact on reaction time. We find statistically significant differences for size and frequency where smaller and slower result in the less immediate reactions. We also present reaction time distributions that a designer can use to tailor expected notification response times to match their content.

Visual Parameters Impacting Reaction Times on Smartwatches

Accounts are often shared by multiple users, each of them having different item consumption and temporal habits. Identifying of the active user can lead to improvements in a variety of services by switching from account personalized services to user personalized services. To do so, we develop a topic model extending the Latent Dirichlet Allocation using a hidden variable representing the active user and assuming consumption times to be generated by latent time topics. We create a new dataset of composite accounts from real users to test the identification capabilities of our model. We show that our model is able to learn temporal patterns from the whole set of accounts and infer the active user using both the consumption time and the consumed item.

Time-Aware User Identification with Topic Models

Electroencephalography (EEG)-based emotion recognition is currently a hot issue in the affective computing community. Numerous studies have been published on this topic, following generally the same schema: 1) presentation of emotional stimuli to a number of subjects during the recording of their EEG, 2) application of machine learning techniques to classify the subjects' emotions. The proposed approaches vary mainly in the type of features extracted from the EEG and in the employed classifiers, but it is difficult to compare the reported results due to the use of different datasets. In this paper, we present a new database for the analysis of valence (positive or negative emotions), which is made publicly available. The database comprises physiological recordings and 257-channel EEG data, contrary to all previously published datasets, which include at most 62 EEG channels. Furthermore, we reconstruct the brain activity on the cortical surface by applying source localization techniques. We then compare the performances of valence classification that can be achieved with various features extracted from all source regions (source space features) and from all EEG channels (sensor space features), showing that the source reconstruction improves the classification results. Finally, we discuss the influence of several parameters on the classification scores.

Emotion recognition based on high-resolution EEG recordings and reconstructed brain sources

This paper tackles the task of storing a large collection of vectors, such as visual descriptors, and of searching in it. To this end, we propose to approximate database vectors by constrained sparse coding, where possible atom weights are restricted to belong to a finite subset. This formulation encompasses, as particular cases, previous state-of-the-art methods such as product or residual quantization. As opposed to traditional sparse coding methods, quantized sparse coding includes memory usage as a design constraint, thereby allowing us to index a large collection such as the BIGANN billion-sized benchmark. Our experiments, carried out on standard benchmarks, show that our formulation leads to competitive solutions when considering different trade-offs between learning/coding time, index size and search quality.

Approximate search with quantized sparse representations

Les liens DSL peuvent subir des pannes sporadiques entraînant des deconnexions ou un acces Internet dégradé. Ces pannes sont a l’origine d’une expérience utilisateur négative et générent des coûts pour les fournisseurs d’accès Internet (FAI) via des appels d’assistance technique. La prediction de pannes permet aux FAI de mettre en oeuvre des mesures proactives de mitigations de pannes. Dans cet article, nous discutons comment effectuer la prediction en ligne de pannes DSL en utilisant des mesures passives faites de maniere continue par des passerelles Internet domestiques. Contrairement aux travaux anterieurs, notre approche permet une pr ´ ediction fine des pannes, plusieurs heures ou jours avant leur apparition. En utilisant une collection longitudinale de metrologies DSL sur 2 ans dans 98 maisons nous ´ montrons que nous predisons les pannes avec de bonnes performances.

Prédiction de pannes DSL par mesure passive sur des passerelles domestiques

Neural node embedding has been recently developed as a powerful representation for supervised tasks with graph data. We leverage this recent advance and propose a novel approach for unsupervised community discovery in graphs. Through extensive experimental studies on simulated and real-world data, we demonstrate consistent improvement of the proposed approach over the current state-of-the-arts. Specifically, our approach empirically attains the information theoretic limits under the benchmark Stochastic Block Models and exhibits better stability and accuracy over the best known algorithms in the community recovery limits.

Node Embedding for Network Community Detection

In this work we propose a novel model-based deep convolutional autoencoder that addresses the highly challenging problem of reconstructing a 3D human face from a single in-the-wild color image. To this end, we combine a convolutional encoder network with an expert-designed generative model that serves as decoder. The core innovation is our new differentiable parametric decoder that encapsulates image formation analytically based on a generative model. Our decoder takes as input a code vector with exactly defined semantic meaning that encodes detailed face pose, shape, expression, skin reflectance and scene illumination. Due to this new way of combining CNN-based with model-based face reconstruction, the CNN-based encoder learns to extract semantically meaningful parameters from a single monocular input image. For the first time, a CNN encoder and an expert-designed generative model can be trained end-to-end in an unsupervised manner, which renders training on very large (unlabeled) real world data feasible. The obtained reconstructions compare favorably to current state-of-the-art approaches in terms of quality and richness of representation.

MoFA: Model-based deep convolutional face autoencoder for unsupervised monocular reconstruction

The success of Google’s PageRank algorithm popularized graphs as a tool to model the web’s navigability. At that time, the web topology was resulting from human edition of hyper-links. Nowadays, that topology is mostly resulting from algorithms. In this paper, we propose to study the topology realized by a class of such algorithms: recommenders. By modeling the output of recommenders as graphs, we show that a vast array of topological observations become easily accessible, using a simple webcrawler. We give models and illustrations for those graph representations.

We then propose a graph-based methodology for addressing an algorithmic transparency problem: recommendation bias detection. We illustrate this approach on YouTube crawls, targeting the prediction of ”Recommended for you” links.

The Topological Face of Recommendation

Efficient Nearest Neighbor (NN) search in high-dimensional spaces is a foundation of many multimedia retrieval systems. A common approach is to rely on Product Quantization, which allows the storage of large vector databases in memory and efficient distance computations. Yet, implementations of nearest neighbor search with Product Quantization have their performance limited by the many memory accesses they perform. Following this observation, Andre et al. proposed Quick ADC with up to 6x faster implementations of mx4 product quantizers (PQ) leveraging specific SIMD instructions. Quicker ADC is a generalization of Quick ADC not limited to mx4 codes and supporting AVX-512, the latest revision of SIMD instruction set. In doing so, Quicker ADC faces the challenge of using efficiently 5,6 and 7-bit shuffles that do not align to computer bytes or words. To this end, we introduce (i) irregular product quantizers combining sub-quantizers of different granularity and (ii) split tables allowing lookup tables larger than registers. We evaluate Quicker ADC with multiple indexes including Inverted Multi-Indexes and IVF HNSW and show that it outperforms the reference optimized implementations (i.e., FAISS and polysemous codes) for numerous configurations. Finally, we release an open-source fork of FAISS enhanced with Quicker ADC.

Quicker ADC : Unlocking the hidden potential of Product Quantization with SIMD

"A recently celebrated kind of deep neural networks is Generative Adversarial Networks. GANs are generators of samples from a distribution that has been learned; they are up to now centrally trained from local data on a single location.

We question the performance of training GANs using a spread dataset over a set of distributed machines, using a gossip approach shown to work on standard neural networks [1]. This performance is compared to the federated learning distributed method, that has the drawback of sending model data to a server. We also propose a gossip variant, where GAN components are gossiped independently. Experiments are conducted with Tensorflow with up to 100 emulated machines, on the canonical MNIST dataset.

The position of this paper is to provide a first evidence that gossip performances for GAN training are close to the ones of federated learning, while operating in a fully decentralized setup. Second, to highlight that for GANs, the distribution of data on machines is critical (i.e., i.i.d. or not). Third, to illustrate that the gossip variant, despite proposing data diversity to the learning phase, brings only marginal improvements over the classic gossip approach."

Gossiping GANs

We consider example-guided audio source separation approaches, where the audio mixture to be separated is supplied with source examples that are assumed matching the sources in the mixture both in frequency and time. These approaches were successfully applied to the tasks such as source separation by humming, score-informed music source separation, and music source separation guided by covers. Most of proposed methods are based on nonnegative matrix factorization (NMF) and its variants, including methods using NMF models pre-trained from examples as an initialization of mixture NMF decomposition, methods using those models as hyperparameters of priors of mixture NMF decomposition, and methods using coupled NMF models. Moreover, those methods differ by the choice of the NMF divergence and the NMF prior. However, there is no systematic comparison of all these methods. In this work, we compare existing methods and some new variants on the score-informed and cover-guided source separation tasks.

Comparative study of example-guided audio source separation approaches based on nonnegative matrix factorization

In this paper, the Predicting Media Interestingness task which is running for the second year as part of the MediaEval 2017 Benchmarking Initiative for Multimedia Evaluation, is presented. For the task, participants are expected to create systems that automatically select images and video segments that are considered to be the most interesting for a common viewer. All task characteristics are described, namely the task use case and challenges, the released data set and ground truth, the required participant runs and the evaluation metrics.

MediaEval 2017 Predicting Media Interestingness Task

Adaptive transform learning schemes have been extensively studied in the literature with a goal to achieve better compression efficiency compared to extensively used Discrete Cosine Transforms (DCT) inside a video codec. These transforms are learned offline on a large training set and are tested either in competition with or in place of the core transforms i.e. DCT. In our previous work, we proposed an alternative approach where a set of non-separable content-adaptive transforms are learned on-the-fly on a sequence and then tested on the same sequence in competition with the core transforms. In this paper, we propose to further improve the previously proposed learning scheme by improving the coding of transformed coefficients in context to the online learning of transforms. The first proposed method improves the convergence of the learning scheme by re-ordering the transformed coefficients at each iteration. The second proposed method improves the compression efficiency by modifying the coding of last significant coefficient position in context to adaptive transforms. The results shows that by combining the above proposed methods, one can achieve 1.2% gain on top of the previously proposed scheme.

Improved Coefficient Coding for Adaptive Transforms in HEVC

Zepeda and Perez [41] have recently demonstrated the promise of the exemplar SVM (ESVM) as a feature encoder for image retrieval. This paper extends this approach in several directions: We first show that replacing the hinge loss by the square loss in the ESVM cost function significantly reduces encoding time with negligible effect on accuracy. We call this model square-loss exemplar machine, or SLEM. We then introduce a kernelized SLEM which can be implemented efficiently through low-rank matrix decomposition, and displays improved performance. Both SLEM variants exploit the fact that the negative examples are fixed, so most of the SLEM computational complexity is relegated to an offline process independent of the positive examples. Our experiments establish the performance and computational advantages of our approach using a large array of base features and standard image retrieval datasets.

Kernel square-loss exemplar machines for image retrieval

The Imitation Game: Algorithm Selection by Exploiting Black-Box Recommenders

Learning parameters from voluminous data can be prohibitive in terms of memory and computational requirements. We propose a ‘compressive learning’ framework, where we estimate model parameters from a sketch of the training data. This sketch is a collection of generalized moments of the underlying probability distribution of the data. It can be computed in a single pass on the training set and is easily computable on streams or distributed datasets. The proposed framework shares similarities with compressive sensing, which aims at drastically reducing the dimension of high-dimensional signals while preserving the ability to reconstruct them. To perform the estimation task, we derive an iterative algorithm analogous to sparse reconstruction algorithms in the context of linear inverse problems. We exemplify our framework with the compressive estimation of a Gaussian mixture model (GMM), providing heuristics on the choice of the sketching procedure and theoretical guarantees of reconstruction. We experimentally show on synthetic data that the proposed algorithm yields results comparable to the classical expectation-maximization technique while requiring significantly less memory and fewer computations when the number of database elements is large. We further demonstrate the potential of the approach on real large-scale data (over 108 training samples) for the task of model-based speaker verification. Finally, we draw some connections between the proposed framework and approximate Hilbert space embedding of probability distributions using random features. We show that the proposed sketching operator can be seen as an innovative method to design translation-invariant kernels adapted to the analysis of GMMs. We also use this theoretical framework to derive preliminary information preservation guarantees, in the spirit of infinite-dimensional compressive sensing.

Sketching for Large Scale Learning of Mixture Models

This paper provides an overview of the Predicting MediaInterestingness task that is organized as part of the Media-Eval 2016 Benchmarking Initiative for Multimedia Evalua-tion. The task, which is running for the first year, expectsparticipants to create systems that automatically select images and video segments that are considered to be the mostinteresting for a common viewer. In this paper, we presentthe task use case and challenges, the proposed data set andground truth, the required participant runs and the evalua-tion metrics

MediaEval 2016 Predicting Media Interestingness Task

In this paper we tackle the problem of single channel audio source separation driven by descriptors of the sounding object's motion. As opposed to previous approaches, motion is included as a soft-coupling constraint within the nonnegative matrix factorization framework. The proposed method is applied to a multimodal dataset of instruments in string quartet performance recordings where bow motion information is used for separation of string instruments. We show that the approach offers better source separation result than an audio-based baseline and the state-of-the-art multimodal-based approaches on these very challenging music mixtures.

Motion Informed Source Separation

A large portion of data mining and analytic services use modern machine learning techniques, such as deep learning. The state-of-the-art results by deep learning come at the price of an intensive use of computing resources. The leading frameworks (e.g., TensorFlow) are executed on GPUs or on high-end servers in datacenters. On the other end, there is a proliferation of personal devices with possibly free CPU cycles; this can enable services to run in users' homes, embedding machine learning operations. In this paper, we ask the following question: Is distributed deep learning computation on WAN connected devices feasible, in spite of the traffic caused by learning tasks? We show that such a setup rises some important challenges, most notably the ingress traffic that the servers hosting the up-to-date model have to sustain. In order to reduce this stress, we propose adaComp, a novel algorithm for compressing worker updates to the model on the server. Applicable to stochastic gradient descent based approaches, it combines efficient gradient selection and learning rate modulation. We then experiment and measure the impact of compression, device heterogeneity and reliability on the accuracy of learned models, with an emulator platform that embeds TensorFlow into Linux containers. We report a reduction of the total amount of data sent by workers to the server by two order of magnitude (e.g., 191-fold reduction for a convolutional network on the MNIST dataset), when compared to a standard asynchronous stochastic gradient descent, while preserving model accuracy.

Distributed Deep Learning on Edge-Devices: Feasibility Via Adaptive Compression

Détecter au plus tôt les utilisateurs importants dans les réseaux sociaux est un problème majeur. Les services de classement d'utilisateurs (peer ranking) sont maintenant des outils bien établis, par des sociétés comme PeerIndex, Klout ou Kred. Leur fonction est de ``classer'' les utilisateurs selon leur influence. Cette notion est néanmoins abstraite, et les méthodes algorithmiques de ces services pour obtenir ce classement sont opaques (les métriques et paramétrages utilisés sont des recettes internes dissimulées). Suivant le récent intérêt pour les outils permettant une plus grande transparence du web, nous proposons d'explorer le problème du retro-ingéniering de l'influence topologique dans ces services de classement. Comme ces services exploitent l'activité en ligne des utilisateurs pour inférer leur influence (dont leur connectivité sur les réseaux sociaux), nous proposons une approche permettant d'estimer précisément l'influence d'un ensemble de métriques topologiques (ou centralités) et leur taux de prise en compte dans le résultat final (classement utilisateur). Pour ce faire, nous modélisons l'algorithme de classement comme une boite noire, avec laquelle nous interagissons via des modifications topologiques, afin d'inférer quelles sont les centralités en jeu dans l'évolution du résultats de classement. Nous montrons que dans certains cas il est possible de déterminer quelles sont ces métriques, et ceci via des opérations sur la topologie par un utilisateur souhaitant connaitre ces paramètres.

Retro-ingénierer les métriques topologiques dans les algorithmes de peer-ranking

Efficient Nearest Neighbor (NN) search in high-dimensional spaces is a foundation of many multimedia retrieval systems. Because it offers low responses times, Product Quantization (PQ) is a popular solution. PQ compresses high-dimensional vectors into short codes using several sub-quantizers, which enables in-RAM storage of large databases. This allows fast answers to NN queries, without accessing the SSD or HDD. The key feature of PQ is that it can compute distances between short codes and high-dimensional vectors using cache-resident lookup tables. The efficiency of this technique, named Asymmetric Distance Computation (ADC), remains limited because it performs many cache accesses. In this paper, we introduce Quick ADC, a novel technique that achieves a 3 to 6 times speedup over ADC by exploiting Single Instruction Multiple Data (SIMD) units available in current CPUs. Efficiently exploiting SIMD requires algorithmic changes to the ADC procedure. Namely, Quick ADC relies on two key modifications of ADC: (i) the use 4-bit sub-quantizers instead of the standard 8-bit sub-quantizers and (ii) the quantization of floating-point distances. This allows Quick ADC to exceed the performance of state-of-the-art systems, e.g., it achieves a Recall@100 of 0.94 in 3.4 ms on 1 billion SIFT descriptors (128-bit codes).

Accelerated Nearest Neighbor Search with Quick ADC

Single Index Models (SIMs) are simple yet flexible semi-parametric models for machine learning, where the response variable is modeled as a monotonic function of a linear combination of features. Estimation in this context requires learning both the feature weights and the nonlinear function that relates features to observations. While methods have been described to learn SIMs in the low dimensional regime, a method that can efficiently learn SIMs in high dimensions, and under general structural assumptions, has not been forthcoming. In this paper, we propose computationally efficient algorithms for SIM inference in high dimensions with structural constraints. Our general approach specializes to sparsity, group sparsity, and low-rank assumptions among others. Experiments show that the proposed method enjoys superior predictive performance when compared to generalized linear models, and achieves results comparable to or better than single layer feedforward neural networks with significantly less computational cost.

On Learning High Dimensional Structured Single Index Models

We propose a novel informed source separation method for audio object coding based on a recent sampling theory for smooth signals on graphs. Assuming that only one source is active at each time-frequency point, we compute an ideal map indicating which source is active at each time-frequency point at the encoder. This map is then sampled with a compressive graph signal sampling strategy that guarantees accurate and stable recovery at the decoder. The graph is built using feature vectors, computed using non-negative matrix factorization, that allows us to connect similar source activations in the time-frequency plane. We show that the proposed approach performs better than state-of-the-art methods at low bitrate.

INFORMED SOURCE SEPARATION BY COMPRESSIVE GRAPH SIGNAL SAMPLING

Several high-dimensional learning applications require the parameters to satisfy a “group sparsity” constraint, where clusters of coefficients are required to be simultaneously selected or rejected. The group lasso and its variants are common methods to solve problems of this form. Recently, in the standard sparse setting, it has been noted that tighter convex relaxations than the ℓ 1 norm can be obtained by using other regularizers, leading to better predictive performance. Motivated by these discoveries, we develop the group k-support norm to achieve group sparsity with a better predictive accuracy. We develop an efficient algorithm to solve the resulting optimization problems, and show that our approach outperforms traditional methods for prediction and recovery under group sparsity.

The Group k-Support Norm for Learning with Structured Sparsity

Predicting interestingness of media content remains an important, but challenging research subject. The difficulty comes first from the fact that, besides being a high-level semantic concept, interestingness is highly subjective and its global definition has not been agreed yet. This paper presents the use of up-to-date deep learning techniques for solving the task. We perform experiments with both social-driven (i.e., Flickr videos) and content-driven (i.e., videos from the MediaEval 2016 interestingness task) datasets. To account for the temporal aspect and multimodality of videos, we tested various deep neural network (DNN) architectures, including a new combination of several recurrent neural networks (RNNs), to handle several temporal samples at the same time. We then investigated different strategies for dealing with unbalanced datasets. Multimodality, as the mid-level fusion of audio and visual information, brought benefit to the task. We also established that social interestingness differs from content interestingness.

DEEP LEARNING FOR MULTIMODAL-BASED VIDEO INTERESTINGNESS PREDICTION

For large-scale visual search, highly compressed yet meaningful representations of images are essential. Structured vector quantizers based on product quantization and its variants are usually employed to achieve such compression while minimizing the loss of accuracy. Yet, unlike binary hashing schemes, these unsupervised methods have not yet benefited from the supervision, end-to-end learning and novel architectures ushered in by the deep learning revolution. We hence propose herein a novel method to make deep convolutional neural networks produce supervised, compact, structured binary codes for visual search. Our method makes use of a novel block-softmax non-linearity and of batch-based entropy losses that together induce structure in the learned encodings. We show that our method outperforms state-of-the-art compact representations based on deep hashing or structured quantization in single and cross-domain category retrieval, instance retrieval and classification. We make our code and models publicly available online.

Supervised Structured Binary Codes for Image Search

In this paper we propose a new method to automatically select the rank of linear transforms during supervised learning. Our approach relies on a sparsity-enforcing element-wise soft-thresholding operation applied after the linear transform. This novel approach to supervised rank learning has the important advantage that it is very simple to implement and incurs no extra complexity relative to linear transform learning. Furthermore, we propose a simple Stochastic Gradient Descent (SGD) implementation suitable for large scale learning, where SGD solvers have established themselves as the default workhorse. We compare our method to various other metric learning techniques in the application of image retrieval. This is one of the remaining few areas where supervised learning of low-rank linear transforms has not been fully exploited. The main reason for this is the lack of adequate datasets that are large enough, and hence we further introduce a new dataset consisting of groups of matching images derived from Cable News Network (CNN) videos using geometric verification and manual selection to find matching frames with adequate variability.

Supervised Learning Of Low-Rank Transforms For Image Retrieval

This work concerns sampling of smooth signals on arbitrary graphs. We first study a structured sampling strategy for such smooth graph signals that consists of a random selection of few pre-defined groups of nodes. The number of groups to sample to stably embed the set of $k$-bandlimited signals is driven by a quantity called the \emph{group} graph cumulative coherence. For some optimised sampling distributions, we show that sampling $O(k\log(k))$ groups is always sufficient to stably embed the set of $k$-bandlimited signals but that this number can be smaller -- down to $O(\log(k))$ -- depending on the structure of the groups of nodes. Fast methods to approximate these sampling distributions are detailed. Second, we consider $k$-bandlimited signals that are nearly piecewise constant over pre-defined groups of nodes. We show that it is possible to speed up the reconstruction of such signals by reducing drastically the dimension of the vectors to reconstruct. When combined with the proposed structured sampling procedure, we prove that the method provides stable and accurate reconstruction of the original signal. Finally, we present numerical experiments that illustrate our theoretical results and, as an example, show how to combine these methods for interactive object segmentation in an image using superpixels.

Structured sampling and fast reconstruction of smooth graph signals

La recommandation joue un rôle central dans le e-commerce et dans l'industrie du divertissement. L'intérêt croissant pour la transparence algorithmique nous motive dans cet article à observer les résultats de recommandations sous la forme d'un graphe capturant les navigations proposées dans l'espace des items. Nous argumentons qu'une telle approche en "boite noire" est utile dans le cas d'une exploration limitée à un utilisateur: nous illustrons une topologie tirée de recommandations à un utilisateur de Youtube, fournissons ses caractéristiques clés, et montrons qu'elle renseigne sur la connaissance de cet utilisateur par le système. Nous montrons ensuite que l'analyse de cette topologie d'aborder la question du \text{biais} potentiel dans ces recommandations. Nous postulons que les systèmes de recommandation produisent naturellement des topologies cohérentes, et qu'une manipulation de ces résultats par l'ajout de liens biaisés a toutes les chances de violer cette cohérence (à la manières des liens longs d'un modèle "petit monde"). Ce postulat est supporté par l'analyse d'un modèle génératif basé sur les kNN et par l'exploitation du crawl Youtube, en ciblant la prédiction de liens "Recommandé pour vous" (i.e., biaisés ou non par Youtube).

La face topologique des recommandations

Estimating the inverse covariance matrix of p variables from n observations is challenging when n  p, since the sample covariance matrix is singular and cannot be inverted. A popular solution is to optimize for the `1 penalized estimator; however, this does not incorporate structure domain knowledge and can be expensive to optimize. We consider finding inverse covariance matrices with group structure, defined as potentially overlapping principal submatrices, determined from domain knowledge (e.g. categories or graph cliques). We propose a new estimator for this problem setting that can be derived efficiently via the Frank-Wolfe method, leveraging chordal decomposition theory for scalability. Simulation results show significant improvement in sample complexity when the correct group structure is known. We also apply these estimators to 14,910 stock closing prices, with noticeable improvement when group sparsity is exploited.

Inverse Covariance Estimation with Structured Groups

Ensuring the early detection of important social network users is a challenging task. Some peer ranking services are now well established, such as PeerIndex, Klout, or Kred. Their function is to rank users according to their influence. This notion of influence is however abstract, and the algorithms achieving this ranking are opaque. Following the rising demand for a more transparent web, we explore the problem of gaining knowledge by reverse engineering such peer ranking services, with regards to the social network topology they get as an input. Since these services exploit the online activity of users (and therefore their connectivity in social networks), we provide a precise evaluation of how topological metrics of the social network impact the final user ranking. Our approach is the following : we first model the ranking service as a black-box with which we interact by creating user profiles and by performing operations on them. Through those profiles, we trigger some slight topological modifications. By monitoring the impact of these modifications on the rankings of those profiles, we infer the weight of each topological metric in the black-box, thus reversing the service influence cookbook.

Uncovering Influence Cookbooks : Reverse Engineering the Topological Impact in Peer Ranking Services

This paper presents an open database of annotated film clips together with an analysis of elements of film style related to how the shots are composed, how the transitions are performed between shots and how the shots are sequenced to compose a film unit. The purpose is to initiate a shared repository pertaining to elements of film style which can be used by computer scientists and film analysts alike. Though both research communities rely strongly on the availability of such information to foster their findings, current databases are either limited to low-level features (such as shots lengths, color and luminance information), contain noisy data, or are not available to the communities. The data and analysis we provide open exciting perspectives as to how computational approaches can rely more thoroughly on information and knowledge extracted from existing movies, and also provide a better understanding of how elements of style are arranged to construct a consistent message.

Analysing elements of styles from annotated film clips

We consider inverse covariance estimation with group sparsity. The groups areoverlapping principal submatrices, which may correspond to structural similarity(e.g. pixels in adjacent regions) or categories (e.g. voter party loyalties). Wepropose a scalable method that makes use of chordal decomposition and appliesthe Frank-Wolfe algorithm. For small simulated problems with block sparsity, weoften perform better than vanilla LASSO. We also apply the method on votingrecords of various congress members, with gender and party as groups

Submatrix-constrained inverse covariance estimation

Matrix completion (MC) with additional information has found wide applicability in several machine learning applications. Among algorithms for solving such problems, Inductive Matrix Completion(IMC) has drawn a considerable amount of attention, not only for its well established theoretical guarantees but also for its superior performance in various real-world applications. However, IMC based methods usually place very strong constraints on the quality of the features (side information) to ensure accurate recovery, which might not be met in practice. In this paper, we pro-pose Goal-directed Inductive Matrix Completion(GIMC) to learn a nonlinear mapping of the features so that they satisfy the required properties. A key distinction between GIMC and IMC is that the feature mapping is learnt in a supervised manner, deviating from the traditional approach of un-supervised feature learning followed by model training. We establish the superiority of our method on several popular machine learning applications including multi-label learning, multi-class classiﬁcation, and semi-supervised clustering.

Goal Directed Inductive Matrix Completion

In this work, we propose a framework, dubbed Union-of-Subspaces SVM (US-SVM), to learn linear classifiers as sparse codes over a learned dictionary. In contrast to discriminative sparse coding with a learned dictionary, it is not the data but the classifiers that are sparsely encoded. Experiments in visual categorization demonstrate that, at training time, the joint learning of the classifiers and of the over-complete dictionary allows the discovery and sharing of mid-level attributes. The resulting classifiers further have a very compact representation in the learned dictionaries, offering substantial performance advantages over standard SVM classifiers for a fixed representation sparsity. This high degree of sparsity of our classifier also provides computational gains, especially in the presence of numerous classes. In addition, the learned atoms can help identify several intra-class modalities.

Maximum Margin Linear Classifiers in Unions of Subspaces

The aggregation of image statistics – the so-called pooling step of image classification algorithms – as well as the construction of part-based models, are two distinct and well-studied topics in the literature. The former aims at leveraging a whole set of local descriptors that an image can contain (through spatial pyramids or Fisher vectors for instance) while the latter argues that only a few of the regions an image contains are actually useful for its classification. This paper bridges the two worlds by proposing a new pooling framework based on the discovery of useful parts involved in the pooling of local representations. The key contribution lies in a model integrating a boosted non-linear part clas-sifier as well as a parametric soft-max pooling component, both trained jointly with the image classifier. The experimental validation shows that the proposed model not only consistently surpasses standard pooling approaches but also improves over state-of-the-art part-based models, on several different and challenging classification tasks.

SPLeaP: Soft Pooling of Learned Parts for Image Classification

“To be considered for the 2017 IEEE Jack Keil Wolf ISIT Student Paper Award.” In this paper we study the problem of noisy tensor completion for tensors that admit a canonical polyadic or CANDE-COMP/PARAFAC (CP) decomposition with one of the factors being sparse. We present general theoretical error bounds for an estimate obtained by using a complexity-regularized maximum likelihood principle and then instantiate these bounds for the case of additive white Gaussian noise. We also provide an ADMM-type algorithm for solving the complexity-regularized maximum likelihood problem and validate the theoretical finding via experiments on synthetic data set.

Noisy Tensor Completion for Tensors With a Sparse Canonical Polyadic Factor

Time series prediction problems are becoming increasingly high-dimensional in modern applications, such as climatology and demand forecasting. For example, in the latter problem, the number of items for which demand needs to be forecast might be as large as 50,000. In addition, the data is generally noisy and full of missing values. Thus, modern applications require methods that are highly scalable, and can deal with noisy data in terms of corruptions or missing values. However, classical time series methods usually fall short of handling these issues. In this paper, we present a temporal regularized matrix factorization (TRMF) framework which supports data-driven temporal learning and forecasting. We develop novel regularization schemes and use scalable matrix factorization methods that are eminently suited for high-dimensional time series data that has many missing values. Our proposed TRMF is highly general, and subsumes many existing approaches for time series analysis. We make interesting connections to graph regularization methods in the context of learning the dependencies in an autoregressive framework. Experimental results show the superiority of TRMF in terms of scalability and prediction quality. In particular, TRMF is two orders of magnitude faster than other methods on a problem of dimension 50,000, and generates better forecasts on real-world datasets such as Wal-mart E-commerce datasetsMachine/Deep Learning/AI

Temporal Regularized Matrix Factorization for High-dimensional Time Series Prediction

Style transfer' among images has recently emerged as a very active research topic, fuelled by the power of convolution neural networks (CNNs), and has become fast a very popular technology in social media. This paper investigates the analogous problem in the audio domain: How to transfer the style of a reference audio signal to a target audio content? We propose a flexible framework for the task, which uses a sound texture model to extract statistics characterizing the reference audio style, followed by an optimization-based audio texture synthesis to modify the target content. In contrast to mainstream optimization-based visual transfer method, the proposed process is initialized by the target content instead of random noise and the optimized loss is only about texture, not structure. These differences proved key for audio style transfer in our experiments. In order to extract features of interest, we investigate different architectures, whether pre-trained on other tasks, as done in image style transfer, or engineered based on the human auditory system. Experimental results on different types of audio signal confirm the potential of the proposed approach.

Audio Style Transfer

Some structural characteristics of online discussions have been successfully modeled in the recent years. When parameters of these models are properly estimated, the models are able to generate synthetic discussions that are structurally similar to the real discussions. A common aspect of these models is that they consider that all users behave according to the same model. In this paper, we combine a growth model with an Expectation–Maximization algorithm that finds different parameters for different latent groups of users. We use this method to find the different roles that coexist in the community. Moreover, we analyze whether we can predict users behaviors based on their roles. Indeed, we show that predictions are improved for some of the roles when compared with a simple growth model.

Role detection in online forums based on growth models for trees

Image retrieval in large image databases is an important problem that drives a number of applications. Yet the use of supervised approaches that address this problem has been limited due to the lack of large labeled datasets for training. Hence, in this paper we introduce two new datasets composed of images extracted from publicly available videos from the Cable News Network (CNN). The proposed datasets are particularly suited to supervised learning for image retrieval and are larger than any other existing dataset of a similar nature. The datasets are further provided with a set of pre-computed, state-of-the-art image feature vectors, as well as baseline results. In order to facilitate research in this important topic, we also detail a generic, supervised learning formulation for image retrieval and a related stochastic solver.

The CNN News Footage Dataset: Enabling Supervision in Image Retrieval

We consider the problem of identifying people on the basis of their walk (gait) pattern. Classical approaches to tackle this problem are based on, e.g., video recordings or piezoelectric sensors embedded in the floor. In this work, we rely on acoustic and vibration measurements, obtained from a microphone and a geophone sensor, respectively. The contribution of this work is twofold. First, we propose a feature extraction method based on an (untrained) shallow scattering network, specially tailored for the gait signals. Second, we demonstrate that fusing the two modalities improves identification in the practically relevant open set scenario.

Scattering Features for Multimodal Gait Recognition

Memorability of media content such as images and videos has recently become an important research subject in computer vision. This paper presents our computation model for predicting image memorability, which is based on a deep learning architecture designed for a classification task. We exploit the use of both convolutional neural network (CNN) - based visual features and semantic features related to image captioning for the task. We train and test our model on the large-scale benchmarking memorability dataset: LaMem. Experiment result shows that the proposed computational model obtains better prediction performance than the state of the art, and even outperforms human consistency. We further investigate the genericity of our model on other memorability datasets. Finally, by validating the model on interestingness datasets, we reconfirm the uncorrelation between memorability and interestingness of images.

Deep Learning for Image Memorability Prediction

Several learning applications require solving high-dimensional regression problems where the relevant features belong to a small number of (overlapping) groups. For very large datasets and under standard sparsity constraints, hard thresholding methods have proven to be extremely efficient, but such methods require NP hard projections when dealing with overlapping groups. In this paper, we show that such NP-hard projections can not only be avoided by appealing to submodular optimization, but such methods come with strong theoretical guarantees even in the presence of poorly conditioned data (i.e. say when two features have correlation ) >= 0.99, which existing analyses cannot handle. These methods exhibit an interesting computation-accuracy trade-off and can be extended to significantly harder problems such as sparse overlapping groups. Experiments on both real and synthetic data validate our claims and demonstrate that the proposed methods are orders of magnitude faster than other greedy and convex relaxation techniques for learning with group-structured sparsity.

Structured Sparse Regression via Greedy Hard Thresholding

Wi-Fi is the preferred way of accessing the Internet for many devices at home, but it is vulnerable to performance problems. The analysis of Wi-Fi quality metrics such as RSSI or PHY rate may indicate a number of problems, but users may not notice many of these problems if they don't degrade the performance of the applications they are using. In this work, we study the effects of the home Wi-Fi quality on Web browsing experience. We instrument a commodity access point (AP) to passively monitor Wi-Fi metrics and study the relationship between Wi-Fi metrics and Web QoE through controlled experiments in a Wi-Fi testbed. We use support vector regression to build a predictor of Web QoE when given Wi-Fi quality metrics available in most commercial APs. Our validation shows root-mean square errors on MOS predictions of 0.6432 in a controlled environment and of 0.9283 in our lab. We apply our predictor on Wi-Fi metrics collected in the wild from 4,880 APs to shed light on how Wi-Fi quality affects Web QoE in real homes.

Predicting the effect of home Wi-Fi on Web QoE

Recent work in video compression has shown that using multiple 2D transforms instead of a single transform in order to de-correlate residuals provides better compression efficiency. These transforms are tested competitively inside a video encoder and the optimal transform is selected based on the Rate Distortion Optimization (RDO) cost. However, one needs to encode a syntax to indicate the chosen transform per residual block to the decoder for successful reconstruction of the pixels. Conventionally, the transform index is binarized using fixed length coding and a CABAC context model is attached to it. In this work, we provide a novel method that utilizes Convolutional Neural Network to predict the chosen transform index from the quantized coefficient block. The prediction probabilities are used to binarize the index by employing a variable length coding instead of a fixed length coding. Results show that by employing this modified transform index coding scheme inside HEVC, one can achieve up to 0.59% BD-rate gain.