David Grangier

Machine Learning Research

david grangier

News - Research - Publications - Code - Colleagues

I am David Grangier, welcome to my homepage. I am currently with the Machine Learning Group at Apple Research, in Cupertino, California. I am interested in large scale Machine Learning and its application to pattern analysis tasks, such as Information Retrieval, Speech Recognition and Natural Language Processing. My research is described in below, while my resume and my linkedin give an overview of my previous experiences.

Research Interests

Learning to Generate Text
The generation of text by computers is relevant to machine translation, summarization, question answering... From a machine learning perspective, it involves conditional language modeling, learning text representation and learning to search a discrete space of sequence for decoding.
Learning to Rank
Learning to rank has increasingly received attention during the last decade, due to applications in domains such as information retrieval. These types of problems aim at assigning a confidence value to each example of a set, such that the values induce a specific ordering over the set. This task hence significantly differs from classical machine learning problem, since the example outputs cannot be considered independently.
Distance and Similarity Learning
Many effective learning algorithms, such as nearest neighbor classifiers or support vector machines, crucially relies on a function to compare examples. Such a function can be referred to as a distance metric, a similarity measure or a kernel, depending on its mathematical properties. Rather than defining this function relying only on prior knowledge, new learning techniques have been introduced to learn it from training data providing information on the desired example proximity.
Learning with Structured Data
In many problems, specific relations exist between the different inputs or between the labels. For instance, computer vision features extracted from the same image share spatial relations, or the phoneme classes in speech belong to a hierarchical structure. Encoding prior information about such structures, or learning such relationships offer great opportunities to improve machine learning approaches for several pattern recognition problems, and several techniques has been introduced toward this objective in the recent years.

Publications

2024

Specialized Language Models with Cheap Inference from Limited Domain Data
David Grangier, Angelos Katharopoulos, Pierre Ablin, Awni Hannun - arxiv 2402.01093. 2024.
Large language models have emerged as a versatile tool but are challenging to apply to tasks lacking large inference budgets and large in-domain training sets. This work formalizes these constraints and distinguishes four important variables: the pretraining budget (for training before the target domain is known), the specialization budget (for training after the target domain is known), the inference budget, and the in-domain training set size. Across these settings, we compare different approaches from the machine learning literature. Limited by inference cost, we find better alternatives to the standard practice of training very large vanilla transformer models. In particular, we show that hyper-networks and mixture of experts have better perplexity for large pretraining budgets, while small models trained on importance sampled datasets are attractive for large specialization budgets.
pdf
```
@misc{grangier2024specialized,
      title={Specialized Language Models with Cheap Inference from Limited Domain Data}, 
      author={David Grangier and Angelos Katharopoulos and Pierre Ablin and Awni Hannun},
      year={2024},
      eprint={2402.01093},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}
```
Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling
Pratyush Maini, Skyler Seto, He Bai, David Grangier, Yizhe Zhang, Navdeep Jaitly - arxiv 2401.16380. 2024.
Large language models are trained on massive scrapes of the web, which are often unstructured, noisy, and poorly phrased. Current scaling laws show that learning from such data requires an abundance of both compute and data, which grows with the size of the model being trained. This is infeasible both because of the large compute costs and duration associated with pre-training, and the impending scarcity of high-quality data on the web. In this work, we propose Web Rephrase Augmented Pre-training () that uses an off-the-shelf instruction-tuned model prompted to paraphrase documents on the web in specific styles such as "like Wikipedia" or in "question-answer format" to jointly pre-train LLMs on real and synthetic rephrases. First, we show that using WRAP on the C4 dataset, which is naturally noisy, speeds up pre-training by . At the same pre-training compute budget, it improves perplexity by more than 10% on average across different subsets of the Pile, and improves zero-shot question answer accuracy across 13 tasks by more than 2%. Second, we investigate the impact of the re-phrasing style on the performance of the model, offering insights into how the composition of the training data can impact the performance of LLMs in OOD settings. Our gains are attributed to the fact that re-phrased synthetic data has higher utility than just real data because it (i) incorporates style diversity that closely reflects downstream evaluation style, and (ii) has higher 'quality' than web-scraped data.
pdf
```
@misc{maini2024rephrasing,
      title={Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling}, 
      author={Pratyush Maini and Skyler Seto and He Bai and David Grangier and Yizhe Zhang and Navdeep Jaitly},
      year={2024},
      eprint={2401.16380},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```

2023

AudioLM: A Language Modeling Approach to Audio Generation
Zalan Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, Neil Zeghidour - IEEE/ACM Transactions on Audio, Speech, and Language Processing (TASLP). 2023.
We introduce AudioLM, a framework for high-quality audio generation with long-term consistency. AudioLM maps the input audio to a sequence of discrete tokens and casts audio generation as a language modeling task in this representation space. We show how existing audio tokenizers provide different trade-offs between reconstruction quality and long-term structure, and we propose a hybrid tokenization scheme to achieve both objectives. Namely, we leverage the discretized activations of a masked language model pre-trained on audio to capture long-term structure and the discrete codes produced by a neural audio codec to achieve high-quality synthesis. By training on large corpora of raw audio waveforms, AudioLM learns to generate natural and coherent continuations given short prompts. When trained on speech, and without any transcript or annotation, AudioLM generates syntactically and semantically plausible speech continuations while also maintaining speaker identity and prosody for unseen speakers. Furthermore, we demonstrate how our approach extends beyond speech by generating coherent piano music continuations, despite being trained without any symbolic representation of music.
pdf
```
@article{audio-lm-generation-2023,
  author={Borsos, Zalán and Marinier, Raphaël and Vincent, Damien and Kharitonov, Eugene and Pietquin, Olivier and Sharifi, Matt and Roblek, Dominik and Teboul, Olivier and Grangier, David and Tagliasacchi, Marco and Zeghidour, Neil},
  journal={IEEE/ACM Transactions on Audio, Speech, and Language Processing}, 
  title={AudioLM: A Language Modeling Approach to Audio Generation}, 
  year={2023},
  volume={31},
  number={},
  pages={2523-2533},
  keywords={Semantics;Acoustics;Training;Computational modeling;Codecs;Predictive models;Task analysis;Computer generated music;speech synthesis},
  doi={10.1109/TASLP.2023.3288409}
}
```
Transfer Learning for Structured Pruning under Limited Task Data
Lucio Dery, David Grangier, Awni Hannun - Third Workshop on Efficient Natural Language and Speech Processing (ENLSP-III). 2023.
Pre-trained models are growing increasingly large which can be problematic for applications with strong inference constraints. Fortunately, task-aware structured pruning offers a solution. While existing pruning algorithms can be efficient, the common practical setting where task-specific data is limited is yet to be addressed. To ameliorate the data scarcity problem, we propose a structured pruning strategy that leverages transfer learning. Detailed analyses of simple transfer learning based remedies lead us to a simple, flexible formulation of what, how and when to transfer, resulting in pruned models with improved generalization over strong baselines.
pdf
```
@inproceedings{dery-grangier-hannun-structured-pruning,
      title={Transfer Learning for Structured Pruning under Limited Task Data}, 
      author={Lucio Dery and David Grangier and Awni Hannun},
      year={2023},
      booktitle = {Third Workshop on Efficient Natural Language and Speech Processing (ENLSP-III)},
}
```
Bilevel Optimization to Learn Training Distributions for Language Modeling under Domain Shift
David Grangier, Pierre Ablin, Awni Hannun - NeurIPS 2023 Workshop on Distribution Shifts. 2023.
Language models trained on very large web corpora have become a central piece of modern language processing. In this paradigm, the large, heterogeneous training set rarely matches the distribution of the application domain. This work considers modifying the training distribution in the case where one can observe a small sample of data reflecting the test conditions. We propose an algorithm based on recent formulation of this problem as an online, bilevel optimization problem. We show that this approach compares favorably with alternative strategies from the domain adaptation literature.
pdf
```
@inproceedings{grangier-ablin-hannun-bilevel-learn-lm-distribution,
      title={Bilevel Optimization to Learn Training Distributions for Language Modeling under Domain Shift}, 
      author={David Grangier and Pierre Ablin and Awni Hannun},
      year={2023},
      booktitle = {NeurIPS 2023 Workshop on Distribution Shifts},
}
```

2022

The Trade-offs of Domain Adaptation for Neural Language Models
David Grangier and Dan Iter - Annual Meeting of the Association for Computational Linguistics (ACL). 2022.
This work connects language model adaptation with concepts of machine learning theory. We consider a training setup with a large out- of-domain set and a small in-domain set. We derive how the benefit of training a model on either set depends on the size of the sets and the distance between their underlying distributions. We analyze how out-of-domain pre-training before in-domain fine-tuning achieves better generalization than either solution independently. Finally, we present how adaptation techniques based on data selection, such as importance sampling, intelligent data selection and influence functions, can be presented in a common framework which highlights their similarity and also their subtle differences.
pdf
```
@inproceedings{grangier2022tradeoffs,
      title={The Trade-offs of Domain Adaptation for Neural Language Models}, 
      author={David Grangier and Dan Iter},
      year={2022},
      booktitle = {Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)},
}
```
A Natural Diet: Towards Improving Naturalness of Machine Translation
Markus Freitag, David Vilar, David Grangier, Colin Cherry and George Foster - Findings of the Annual Meeting of the Association for Computational Linguistics (ACL). 2022.
Machine translation (MT) evaluation often focuses on accuracy and fluency, without paying much attention to translation style. This means that, even when considered accurate and fluent, MT output can still sound less {\em natural} than high quality human translations or text originally written in the target language. Machine translation output notably exhibits lower lexical diversity, and employs constructs that mirror those in the source sentence. In this work we propose a method for training MT systems to achieve a more natural style, i.e. mirroring the style of text originally written in the target language. Our method tags parallel training data according to the naturalness of the target side by contrasting language models trained on natural and translated data. Tagging data allows us to put greater emphasis on target sentences originally written in the target language. Automatic metrics show that the resulting models achieve lexical richness on par with human translations, mimicking a style much closer to sentences originally written in the target language. Furthermore, we find that their output is preferred by human experts when compared to the baseline translations.
pdf
```
@inproceedings{freitag-natural-diet-translation-2022,
    title={A Natural Diet: Towards Improving Naturalness of Machine Translation},
    author={Markus Freitag and David Vilar and David Grangier and Colin Cherry and George Foster},
    booktitle={Findings of the Annual Meeting of the Association for Computational Linguistics (ACL)},
    year={2022},
}
```
Learning Strides in Convolutional Neural Networks
Rachid Riad, Olivier Teboul, David Grangier and Neil Zeghidour - International Conference on Learning Representation (ICLR). 2022.
Convolutional neural networks typically contain several downsampling operators, such as strided convolutions or pooling layers, that progressively reduce the resolution of intermediate representations. This provides some shift-invariance while reducing the computational complexity of the whole architecture. A critical hyperparameter of such layers is their stride: the integer factor of downsampling. As strides are not differentiable, finding the best configuration either requires cross-validation or discrete optimization (e.g. architecture search), which rapidly become prohibitive as the search space grows exponentially with the number of downsampling layers. Hence, exploring this search space by gradient descent would allow finding better configurations at a lower computational cost. This work introduces DiffStride, the first downsampling layer with learnable strides. Our layer learns the size of a cropping mask in the Fourier domain, that effectively performs resizing in a differentiable way. Experiments on audio and image classification show the generality and effectiveness of our solution: we use DiffStride as a drop-in replacement to standard downsampling layers and outperform them. In particular, we show that introducing our layer into a ResNet-18 architecture allows keeping consistent high performance on CIFAR10, CIFAR100 and ImageNet even when training starts from poor random stride configurations. Moreover, formulating strides as learnable variables allows us to introduce a regularization term that controls the computational complexity of the architecture. We show how this regularization allows trading off accuracy for efficiency on ImageNet.
pdf
```
@inproceedings{riad-learning-strides-convnets-2022,
    title={Learning Strides in Convolutional Neural Networks},
    author={Rachid Riad and Olivier Teboul and David Grangier and Neil Zeghidour},
    booktitle={International Conference on Learning Representation (ICLR)},
    year={2022},
}
```

2021

Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation
Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan and Wolfgang Macherey - Transactions of the Association for Computational Linguistics (TACL). 2021.
Human evaluation of modern high-quality machine translation systems is a difficult problem, and there is increasing evidence that inadequate evaluation procedures can lead to erroneous conclusions. While there has been considerable research on human evaluation, the field still lacks a commonly-accepted standard procedure. As a step toward this goal, we propose an evaluation methodology grounded in explicit error analysis, based on the Multidimensional Quality Metrics (MQM) framework. We carry out the largest MQM research study to date, scoring the outputs of top systems from the WMT 2020 shared task in two language pairs using annotations provided by professional translators with access to full document context. We analyze the resulting data extensively, finding among other results a substantially different ranking of evaluated systems from the one established by the WMT crowd workers, exhibiting a clear preference for human over machine output. Surprisingly, we also find that automatic metrics based on pre-trained embeddings can outperform human crowd workers. We make our corpus publicly available for further research.
pdf
```
@article{freitag2021experts,
      title={Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation}, 
      author={Markus Freitag and George Foster and David Grangier and Viresh Ratnakar and Qijun Tan and Wolfgang Macherey},
      journal={Transactions of the Association for Computational Linguistics (TACL)},
      year={2021},
}
```
DIVE: End-to-end Speech Diarization via Iterative Speaker Embedding
Neil Zeghidour, Olivier Teboul and David Grangier - IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). 2021.
We introduce DIVE, an end-to-end speaker diarization algorithm. Our neural algorithm presents the diarization task as an iterative process: it repeatedly builds a representation for each speaker before predicting the voice activity of each speaker conditioned on the extracted representations. This strategy intrinsically resolves the speaker ordering ambiguity without requiring the classical permutation invariant training loss. In contrast with prior work, our model does not rely on pretrained speaker representations and optimizes all parameters of the system with a multi-speaker voice activity loss. Importantly, our loss explicitly excludes unreliable speaker turn boundaries from training, which is adapted to the standard collar-based Diarization Error Rate (DER) evaluation. Overall, these contributions yield a system redefining the state-of-the-art on the standard CALLHOME benchmark, with 6.7% DER compared to 7.8% for the best alternative.
pdf
```
@inproceedings{zeghidour-teboul-grangier-dive-2021,
    title={DIVE: End-to-end Speech Diarization via Iterative Speaker Embedding},
    author={Neil Zeghidour and Olivier Teboul and David Grangier},
    booktitle   = {{IEEE} Automatic Speech Recognition and Understanding Workshop ({ASRU})},
    year={2021},
}
```
Auxiliary Task Update Decomposition: The Good, The Bad and The Neutral
Lucio Dery, Yann Dauphin, David Grangier - International Conference on Learning Representation (ICLR). 2021.
While deep learning has been very beneficial in data-rich settings, tasks with smaller training set often resort to pre-training or multitask learning to leverage data from other tasks. In this case, careful consideration is needed to select tasks and model parameterizations such that updates from the auxiliary tasks actually help the primary task. We seek to alleviate this burden by formulating a model-agnostic framework that performs fine-grained manipulation of the auxiliary task gradients. We propose to decompose auxiliary updates into directions which help, damage or leave the primary task loss unchanged. This allows weighting the update directions differently depending on their impact on the problem of interest. We present a novel and efficient algorithm for that purpose and show its advantage in practice. Our method leverages efficient automatic differentiation procedures and randomized singular value decomposition for scalability. We show that our framework is generic and encompasses some prior work as particular cases. Our approach consistently outperforms strong and widely used baselines when leveraging out-of-distribution data for Text and Image classification tasks.
pdf
```
@inproceedings{ldery-aux-taks-iclr21,
    title={Auxiliary Task Update Decomposition: The Good, The Bad and The Neutral},
    author={Lucio Dery and Yann Dauphin and David Grangier},
    booktitle={International Conference on Learning Representation (ICLR)},
    year={2021},
}
```
Wavesplit: End-to-End Speech Separation by Speaker Clustering
Neil Zeghidour and David Grangier - Transaction on Audio Speech and Language Processing (TASLP). 2021.
We introduce Wavesplit, an end-to-end source separation system. From a single mixture, the model infers a representation for each source and then estimates each source signal given the inferred representations. The model is trained to jointly perform both tasks from the raw waveform. Wavesplit infers a set of source representations via clustering, which addresses the fundamental permutation problem of separation. For speech separation, our sequence-wide speaker representations provide a more robust separation of long, challenging recordings compared to prior work. Wavesplit redefines the state-of-the-art on clean mixtures of 2 or 3 speakers (WSJ0-2/3mix), as well as in noisy and reverberated settings (WHAM/WHAMR). We also set a new benchmark on the recent LibriMix dataset. Finally, we show that Wavesplit is also applicable to other domains, by separating fetal and maternal heart rates from a single abdominal electrocardiogram.
pdf
```
@article{zeghidour-grangier-wavesplit-2021,
    title={Wavesplit: End-to-End Speech Separation by Speaker Clustering},
    author={Neil Zeghidour and David Grangier},
    journal   = {{IEEE} {ACM} Transaction on Audio Speech and Language Processing (TASLP)},
    year={2021},
}
```
Contrastive Learning of General-Purpose Audio Representations
Aaqib Saeed, David Grangier, Neil Zeghidour - International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2021.
We introduce COLA, a self-supervised pre-training approach for learning a general-purpose representation of audio. Our approach is based on contrastive learning: it learns a representation which assigns high similarity to audio segments extracted from the same recording while assigning lower similarity to segments from different recordings. We build on top of recent advances in contrastive learning for computer vision and reinforcement learning to design a lightweight, easy-to-implement self-supervised model of audio. We pre-train embeddings on the large-scale Audioset database and transfer these representations to 9 diverse classification tasks, including speech, music, animal sounds, and acoustic scenes. We show that despite its simplicity, our method significantly outperforms previous self-supervised systems. We furthermore conduct ablation studies to identify key design choices and release a library to pre-train and fine-tune COLA models.
pdf
```
@inproceedings{saeeds-contrastive-audio-2021,
    title={Contrastive Learning of General-Purpose Audio Representations},
    author={Aaqib Saeed and David Grangier and Neil Zeghidour},
    booktitle={International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
    year={2021},
}
```
Learning from Heterogeneous EEG Signals with Differentiable Channel Reordering
Aaqib Saeed, David Grangier, Olivier Pietquin, Neil Zeghidour - International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2021.
We propose CHARM, a method for training a single neural network across inconsistent input channels. Our work is motivated by Electroencephalography (EEG), where data collection protocols from different headsets result in varying channel ordering and number, which limits the feasibility of transferring trained systems across datasets. Our approach builds upon attention mechanisms to estimate a latent reordering matrix from each input signal and map input channels to a canonical order. CHARM is differentiable and can be composed further with architectures expecting a consistent channel ordering to build end-to-end trainable classifiers. We perform experiments on four EEG classification datasets and demonstrate the efficacy of CHARM via simulated shuffling and masking of input channels. Moreover, our method improves the transfer of pre-trained representations between datasets collected with different protocols.
pdf
```
@article{saeeds-eeg-reordering-2021,
    title={Learning from Heterogeneous EEG Signals with Differentiable Channel Reordering},
    author={Aaqib Saeed and David Grangier and Olivier Pietquin and Neil Zeghidour},
    booktitle={International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
    year={2021},
}
```
On the Complementarity of Data Selection and Fine Tuning for Domain Adaptation
Dan Iter and David Grangier - arXiv:2109.07591. 2021.
Domain adaptation of neural networks commonly relies on three training phases: pretraining, selected data training and then fine tuning. Data selection improves target domain generalization by training further on pretraining data identified by relying on a small sample of target domain data. This work examines the benefit of data selection for language modeling and machine translation. Our experiments assess the complementarity of selection with fine tuning and result in practical recommendations: (i) selected data must be similar to the fine-tuning domain but not so much as to erode the complementary effect of fine-tuning; (ii) there is a trade-off between selecting little data for fast but limited progress or much data for slow but long lasting progress; (iii) data selection can be applied early during pretraining, with performance gains comparable to long pretraining session; (iv) data selection from domain classifiers is often more effective than the popular contrastive data selection method.
pdf
```
@misc{iter2021complementarity,
      title={On the Complementarity of Data Selection and Fine Tuning for Domain Adaptation}, 
      author={Dan Iter and David Grangier},
      year={2021},
      eprint={2109.07591},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```
What Can Unsupervised Machine Translation Contribute to High-Resource Language Pairs?
Kelly Marchisio, Markus Freitag and David Grangier - arXiv:2106.15818. 2021.
Whereas existing literature on unsupervised machine translation (MT) focuses on exploiting unsupervised techniques for low-resource language pairs where bilingual training data is scare or unavailable, we investigate whether unsupervised MT can also improve translation quality of high-resource language pairs where sufficient bitext does exist. We compare the style of correct translations generated by either supervised or unsupervised MT and find that the unsupervised output is less monotonic and more natural than supervised output. We demonstrate a way to combine the benefits of unsupervised and supervised MT into a single system, resulting in better human evaluation of quality and fluency. Our results open the door to discussions about the potential contributions of unsupervised MT in high-resource settings, and how supervised and unsupervised systems might be mutually-beneficial.
pdf
```
@misc{marchisio2021unsupervised,
      title={What Can Unsupervised Machine Translation Contribute to High-Resource Language Pairs?}, 
      author={Kelly Marchisio and Markus Freitag and David Grangier},
      year={2021},
      eprint={2106.15818},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```
Minimum Bayes Risk Decoding with Neural Metrics of Translation Quality
Markus Freitag, David Grangier, Qijun Tan and Bowen Liang - arXiv:2111.09388. 2021.
This work applies Minimum Bayes Risk (MBR) decoding to optimize diverse automated metrics of translation quality. Automatic metrics in machine translation have made tremendous progress recently. In particular, neural metrics, fine-tuned on human ratings (e.g. BLEURT, or COMET) are outperforming surface metrics in terms of correlations to human judgements. Our experiments show that the combination of a neural translation model with a neural reference-based metric, BLEURT, results in significant improvement in automatic and human evaluations. This improvement is obtained with translations different from classical beam-search output: these translations have much lower likelihood and are less favored by surface metrics like BLEU.
pdf
```
@misc{freitag-mbr-2021,
      title={Minimum Bayes Risk Decoding with Neural Metrics of Translation Quality}, 
      author={Markus Freitag and David Grangier and Qijun Tan and Bowen Liang},
      year={2021},
      eprint={2111.09388},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```

2020

Efficient Content-Based Sparse Attention with Routing Transformers
Aurko Roy, Mohammad Saffar, Ashish Vaswani, David Grangier - Transactions of the Association for Computational Linguistics (TACL). 2020.
Self-attention has recently been adopted for a wide range of sequence modeling problems. Despite its effectiveness, self-attention suffers from quadratic compute and memory requirements with respect to sequence length. Successful approaches to reduce this complexity focused on attending to local sliding windows or a small set of locations independent of content. Our work proposes to learn dynamic sparse attention patterns that avoid allocating computation and memory to attend to content unrelated to the query of interest. This work builds upon two lines of research: it combines the modeling flexibility of prior work on content-based sparse attention with the efficiency gains from approaches based on local, temporal sparse attention. Our model, the Routing Transformer, endows self-attention with a sparse routing module based on online k-means while reducing the overall complexity of attention to O(n1.5d) from O(n2d) for sequence length n and hidden dimension d. We show that our model outperforms comparable sparse attention models on language modeling on Wikitext-103 (15.8 vs 18.3 perplexity) as well as on image generation on ImageNet-64 (3.43 vs 3.44 bits/dim) while using fewer self-attention layers.
pdf
```
@article{aurkoroy-routing-transformer-2020,
    title={Efficient Content-Based Sparse Attention with Routing Transformers},
    author={Aurko Roy and Mohammad Saffar and Ashish Vaswani and David Grangier},
    journal={Transactions of the Association for Computational Linguistics (TACL) },
    year={2020},
}
```
Human-Paraphrased References Improve Neural Machine Translation
Markus Freitag, George Foster, David Grangier, Colin Cherry - Conference on Machine Translation (WMT). 2020.
Automatic evaluation comparing candidate translations to human-generated paraphrases of reference translations has recently been proposed by Freitag et al. (2020). When used in place of original references, the paraphrased versions produce metric scores that correlate better with human judgment. This effect holds for a variety of different automatic metrics, and tends to favor natural formulations over more literal (translationese) ones. In this paper we compare the results of performing endto-end system development using standard and paraphrased references. With state-of-the-art English-German NMT components, we show that tuning to paraphrased references produces a system that is significantly better according to human judgment, but 5 BLEU points worse when tested on standard references. Our work confirms the finding that paraphrased references yield metric scores that correlate better with human judgment, and demonstrates for the first time that using these scores for system development can lead to significant improvements.
pdf
```
@inproceedings{freitag-paraphrase-mert-2020,
    title={Human-Paraphrased References Improve Neural Machine Translation},
    author={Markus Freitag and George Foster and David Grangier and Colin Cherry},
    booktitle={Conference on Machine Translation (WMT)},
    year={2020},
}
```
BLEU might be Guilty but References are not Innocent
Markus Freitag, David Grangier, Isaac Caswell - Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020.
The quality of automatic metrics for machine translation has been increasingly called into question, especially for high-quality systems. This paper demonstrates that, while choice of metric is important, the nature of the references is also critical. We study different methods to collect references and compare their value in automated evaluation by reporting correlation with human evaluation for a variety of systems and metrics. Motivated by the finding that typical references exhibit poor diversity, concentrating around translationese language, we develop a paraphrasing task for linguists to perform on existing reference translations, which counteracts this bias. Our method yields higher correlation with human judgment not only for the submissions of WMT 2019 English to German, but also for Back-translation and APE augmented MT output, which have been shown to have low correlation with automatic metrics using standard references. We demonstrate that our methodology improves correlation with all modern evaluation metrics we look at, including embedding-based methods. To complete this picture, we reveal that multi-reference BLEU does not improve the correlation for high quality output, and present an alternative multi-reference formulation that is more effective.
pdf
```
@article{freitag-bleu-paraphrase-references-2020,
    title={BLEU might be Guilty but References are not Innocent},
    author={Markus Freitag and David Grangier and Isaac Caswell},
    booktitle={Conference on Empirical Methods in Natural Language Processing (EMNLP)},
    year={2020},
}
```
Translationese as a Language in "Multilingual" NMT
Parker Riley, Isaac Caswell, Markus Freitag, David Grangier - Annual Meeting of the Association for Computational Linguistics (ACL). 2020.
Machine translation has an undesirable propensity to produce "translationese" artifacts, which can lead to higher BLEU scores while being liked less by human raters. Motivated by this, we model translationese and original (i.e. natural) text as separate languages in a multilingual model, and pose the question: can we perform zero-shot translation between original source text and original target text? There is no data with original source and original target, so we train sentence-level classifiers to distinguish translationese from original target text, and use this classifier to tag the training data for an NMT model. Using this technique we bias the model to produce more natural outputs at test time, yielding gains in human evaluation scores on both accuracy and fluency. Additionally, we demonstrate that it is possible to bias the model to produce translationese and game the BLEU score, increasing it while decreasing human-rated quality. We analyze these models using metrics to measure the degree of translationese in the output, and present an analysis of the capriciousness of heuristically-based train-data tagging.
pdf
```
@article{riley-translationese-2020,
    title={Translationese as a Language in "Multilingual" {NMT}},
    author={Parker Riley and Isaac Caswell and Markus Freitag and David Grangier},
    booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
    year = {2020},
}
```
Towards Better Storylines with Sentence-Level Language Models
Daphne Ippolito, David Grangier, Douglas Eck, Chris Callison-Burch - Annual Meeting of the Association for Computational Linguistics (ACL). 2020.
This work proposes a sentence-level language model which predicts the next sentence in a story given the embeddings of the previous sentences. The model operates at the sentence-level and selects the next sentence within a fine set of fluent alternatives. By working with sentence embeddings instead of word embeddings, our model is able to efficiently consider a large number of alternative sentences. By considering only fluent sentences, our model is relieved from modeling fluency and can focus on longer range dependencies. Our method achieves state-of-the-art accuracy on the StoryCloze task in the unsupervised setting.
pdf
```
@inproceedings{ippolito-storyline-2020,
    title={Towards Better Storylines with Sentence-Level Language Models},
    author={Daphne Ippolito and David Grangier and Douglas Eck and Chris Callison-Burch},
    booktitle={Annual Meeting of the Association for Computational Linguistics (ACL)},
    year={2020},
}
```

2019

ELI5: Long Form Question Answering
Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, Michael Auli - Annual Meeting of the Association for Computational Linguistics (ACL). 2019.
We introduce the first large-scale corpus for long form question answering, a task requiring elaborate and in-depth answers to open-ended questions. The dataset comprises 270K threads from the Reddit forum “Explain Like I’m Five” (ELI5) where an online community provides answers to questions which are comprehensible by five year olds. Compared to existing datasets, ELI5 comprises diverse questions requiring multi-sentence answers. We provide a large set of web documents to help answer the question. Automatic and human evaluations show that an abstractive model trained with a multi-task objective outperforms conventional Seq2Seq, language modeling, as well as a strong extractive baseline.However, our best model is still far from human performance since raters prefer gold responses in over 86% of cases, leaving ample opportunity for future improvement.
pdf
```
@inproceedings{fan-etal-2019-eli5,
    title = "{ELI}5: Long Form Question Answering",
    author = "Fan, Angela  and Jernite, Yacine and Perez, Ethan and Grangier, David and Weston, Jason and Auli, Michael",
    booktitle = "Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics",
    year = "2019",
    url = "https://www.aclweb.org/anthology/P19-1346"
}
```
Tagged Back-Translation
Isaac Caswell, Ciprian Chelba, David Grangier - Conference on Machine Translation (WMT). 2019.
Recent work in Neural Machine Translation (NMT) has shown significant quality gains from noised-beam decoding during back-translation, a method to generate synthetic parallel data. We show that the main role of such synthetic noise is not to diversify the source side, as previously suggested, but simply to indicate to the model that the given source is synthetic. We propose a simpler alternative to noising techniques, consisting of tagging back-translated source sentences with an extra token. Our results on WMT outperform noised back-translation in English-Romanian and match performance on English-German, re-defining state-of-the-art in the former.
pdf
```
@inproceedings{caswell:tagbt:2019,
  author    = {Isaac Caswell and Ciprian Chelba and David Grangier},
  title     = {Tagged Back-Translation},
  year      = {2019},
  booktitle   = {Conference on Machine Translation (WMT)},
}
```
Unsupervised Paraphrasing without Translation
Aurko Roy, David Grangier - Conference of the Association for Computational Linguistics (ACL). 2019.
Paraphrasing exemplifies the ability to abstract semantic content from surface forms. Recent work on automatic paraphrasing is dominated by methods leveraging Machine Translation (MT) as an intermediate step. This contrasts with humans, who can paraphrase without being bilingual. This work proposes to learn paraphrasing models from an unlabeled monolingual corpus only. To that end, we propose a residual variant of vector-quantized variational auto-encoder. We compare with MT-based approaches on paraphrase identification, generation, and training augmentation. Monolingual paraphrasing outperforms unsupervised translation in all settings. Comparisons with supervised translation are more mixed: monolingual paraphrasing is interesting for identification and augmentation; supervised translation is superior for generation.
pdf
```
@inproceedings{aurkoroy:paraphrase:2019,
  author    = {Aurko Roy and David Grangier},
  title     = {Unsupervised Paraphrasing without Translation},
  year      = {2019},
  booktitle   = {Conference of the Association for Computational Linguistics (ACL)},
}
```
fairseq: A Fast, Extensible Toolkit for Sequence Modeling
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, Michael Auli - Demo of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL Demo). 2019.
fairseq is an open-source sequence modeling toolkit that allows researchers and developers to train custom models for translation, summarization, language modeling, and other text generation tasks. The toolkit is based on PyTorch and supports distributed training across multiple GPUs and machines. We also support fast mixed-precision training and inference on modern GPUs.

pdf
```
@inproceedings{ottedunov:fairseq:2019,
  author    = {Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, Michael Auli},
  title     = {fairseq: A Fast, Extensible Toolkit for Sequence Modeling},
  year      = {2019},
  booktitle   = {Demo of Conference of the North American Chapter of the Association for Computational Linguistics (NAACL Demo)},
}
```
Unsupervised Hierarchical Story Infilling
Daphne Ippolito, David Grangier, Chris Callison-Burch, Douglas Eck - Workshop on Narrative Understanding @ NAACL. 2019.
Story infilling involves predicting words to go into a missing span from a story. This challenging task has the potential to transform interactive tools for creative writing. However, state-of-the-art conditional language models have trouble balancing fluency and coherence with novelty and diversity. We address this limitation with a hierarchical model which first selects a set of rare words and then generates text conditioned on that set. By relegating the high entropy task of picking rare words to a word-sampling model, the second-stage model conditioned on those words can achieve high fluency and coherence by searching for likely sentences, without sacrificing diversity.
pdf
```
@inproceedings{ippolito-etal-2019-unsupervised,
    title = "Unsupervised Hierarchical Story Infilling",
    author = "Ippolito, Daphne and Grangier, David and Callison-Burch, Chris and Eck, Douglas",
    booktitle = "Proceedings of the First Workshop on Narrative Understanding @ NAACL",
    year = "2019",
    url = "https://www.aclweb.org/anthology/W19-2405"}
  
```
3D human pose estimation in video with temporal convolutions and semi-supervised training
Dario Pavllo, Christoph Feichtenhofer, David Grangier, Michael Auli - Conference on Computer Vision and Pattern Recognition (CVPR). 2019.
In this work, we demonstrate that 3D poses in video can be effectively estimated with a fully convolutional model based on dilated temporal convolutions over 2D keypoints. We also introduce back-projection, a simple and effective semi-supervised training method that leverages unlabeled video data. We start with predicted 2D keypoints for unlabeled video, then estimate 3D poses and finally back-project to the input 2D keypoints. In the supervised setting, our fully-convolutional model outperforms the previous best result from the literature by 6 mm mean per-joint position error on Human3.6M, corresponding to an error reduction of 11%, and the model also shows significant improvements on HumanEva-I. Moreover, experiments with back-projection show that it comfortably outperforms previous state-of-the-art results in semi-supervised settings where labeled data is scarce. Code and models are available.

pdf
```
@inproceedings{pavllo:3dpose:2018,
  author    = {Dario Pavllo and Christoph Feichtenhofer and David Grangier and Michael Auli},
  title     = {3D human pose estimation in video with temporal convolutions and semi-supervised training},
  year      = {2019},
  booktitle = {Conference on Computer Vision and  Pattern Recognition (CVPR)},
}
```

2018

Understanding Back-Translation at Scale
Sergey Edunov, Myle Ott, Michael Auli, David Grangier - Conference on Empirical Methods in Natural Language Processing (EMNLP). 2018.
An effective method to improve neural machine translation with monolingual data is to augment the parallel training corpus with back-translations of target language sentences. This work broadens the understanding of back-translation and investigates a number of methods to generate synthetic source sentences. We find that in all but resource poor settings back-translations obtained via sampling or noised beam outputs are most effective. Our analysis shows that sampling or noisy synthetic data gives a much stronger training signal than data generated by beam or greedy search. We also compare how synthetic data compares to genuine bitext and study various domain effects. Finally, we scale to hundreds of millions of monolingual sentences and achieve a new state of the art of 35 BLEU on the WMT'14 English-German test set.

pdf
```
@article{edunov:backtranslation:2018,
  author    = {Sergey Edunov Myle Ott and Michael Auli and David Grangier},
  title     = {Understanding Back-Translation at Scale},
  year      = {2018},
  booktitle = {Conference on Empirical Methods in Natural Language Processing ({EMNLP})},
}
```
Scaling Neural Machine Translation
Myle Ott, Sergey Edunov, David Grangier, Michael Auli - Workshop on Machine Translation (WMT@EMNLP). 2018.
Sequence to sequence learning models still require several days to reach state of the art performance on large benchmark datasets using a single machine. This paper shows that reduced precision and large batch training can speedup training by nearly 5x on a single 8-GPU machine with careful tuning and implementation. On WMT'14 English-German translation, we match the accuracy of Vaswani et al (2017) in under 5 hours when training on 8 GPUs and we obtain a new state of the art of 29.3 BLEU after training for 91 minutes on 128 GPUs. We further improve these results to 29.8 BLEU by training on the much larger Paracrawl dataset.

pdf
```
@inproceedings{ott:scaling:2018,
  author    = {Myle Ott and Sergey Edunov and David Grangier and Michael Auli},
  title     = {Scaling Neural Machine Translation},
  year      = {2018},
  booktitle = {Workshop on Machine Translation ({WMT@EMNLP})},
}
```
QuaterNet: A Quaternion-based Recurrent Model for Human Motion
Dario Pavllo, David Grangier, Michael Auli - British Machine Vision Conference (BMVC). 2018.
Deep learning for predicting or generating 3D human pose sequences is an active research area. Previous work regresses either joint rotations or joint positions. The former strategy is prone to error accumulation along the kinematic chain, as well as discontinuities when using Euler angle or exponential map parameterizations. The latter requires re-projection onto skeleton constraints to avoid bone stretching and invalid configurations. This work addresses both limitations. Our recurrent network, QuaterNet, represents rotations with quaternions and our loss function performs forward kinematics on a skeleton to penalize absolute position errors instead of angle errors. On short-term predictions, QuaterNet improves the state-of-the-art quantitatively. For long-term generation, our approach is qualitatively judged as realistic as recent neural strategies from the graphics literature.

pdf
```
@inproceedings{pavllo:quaternet:2018,
  author    = {Dario Pavllo and David Grangier and Michael Auli},
  title     = {QuaterNet: A Quaternion-based Recurrent Model for Human Motion},
  year      = {2018},
  booktitle   = {British Machine Vision Conference (BMVC)},
}
```
Analyzing Uncertainty in Neural Machine Translation
Myle Ott, Michael Auli, David Grangier, Marc’Aurelio Ranzato - International Conference on Machine Learning (ICML). 2018.
Machine translation is a popular test bed for research in neural sequence-to-sequence models but despite much recent research, there is still a lack of understanding of these models. Practitioners report performance degradation with large beams, the under-estimation of rare words and a lack of diversity in the final translations. Our study relates some of these issues to the inherent uncertainty of the task, due to the existence of multiple valid translations for a single source sentence, and to the extrinsic uncertainty caused by noisy training data. We propose tools and metrics to assess how uncertainty in the data is captured by the model distribution and how it affects search strategies that generate translations. Our results show that search works remarkably well but that the models tend to spread too much probability mass over the hypothesis space. Next, we propose tools to assess model calibration and show how to easily fix some shortcomings of current models. We release both code and multiple human reference translations for two popular benchmarks.

pdf
```
@inproceedings{ott:uncertainty:2018,
  author    = {Myle Ott and Michael Auli and David Grangier and Marc’Aurelio Ranzato},
  title     = {Analyzing Uncertainty in Neural Machine Translation},
  year      = {2018},
  booktitle   = {International Conference on Machine Learning {(ICML)}},
}
```
Controllable Abstractive Summarization
Angela Fan, David Grangier, Michael Auli - ACL Workshop on Neural Machine Translation and Generation (NMT@ACL) 2018.
Current models for document summarization ignore user preferences such as the desired length, style or entities that the user has a preference for. We present a neural summarization model that enables users to specify such high level attributes in order to control the shape of the final summaries to better suit their needs. With user input, we show that our system can produce high quality summaries that are true to user preference. Without user input, we can set the control variables automatically and outperform comparable state of the art summarization systems despite the relative simplicity of our model.

pdf
```
@inproceedings{fan:summarization:2018,
  author    = {Angela Fan and David Grangier and Michael Auli},
  title     = {Controllable Abstractive Summarization},
  year      = {2018},
  booktitle   = {ACL Workshop on Neural Machine Translation and Generation (NMT@ACL)},
}
```
Classical Structured Prediction Losses for Sequence to Sequence Learning
Sergey Edunov, Myle Ott, Michael Auli, David Grangier, Marc’Aurelio Ranzato - Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). 2018.
There has been much recent work on training neural attention models at the sequence-level using either reinforcement learning-style methods or by optimizing the beam. In this paper, we survey a range of classical objective functions that have been widely used to train linear models for structured prediction and apply them to neural sequence to sequence models. Our experiments show that these losses can perform surprisingly well by slightly outperforming beam search optimization in a like for like setup. We also report new state of the art results on both IWSLT 2014 German-English translation as well as Gigaword abstractive summarization.

pdf
```
@inproceedings{edunovott:structured:2018,
  author    = {Sergey Edunov and Myle Ott and Michael Auli and David Grangier and Marc’Aurelio Ranzato},
  title     = {Classical Structured Prediction Losses for Sequence to Sequence Learning},
  year      = {2018},
  booktitle   = {Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)},
}
```
QuickEdit: Editing Text and Translations by Crossing Words Out
David Grangier and Michael Auli - Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). 2018.
We propose a framework for computer-assisted text editing. It applies to translation post-editing and to paraphrasing and relies on very simple interactions: a human editor modifies a sentence by marking tokens they would like the system to change. Our model then generates a new sentence which reformulates the initial sentence by avoiding the words from the marked tokens. Our approach builds upon neural sequence-to-sequence modeling and introduces a neural network which takes as input a sentence along with deleted token markers. Our model is trained on translation bitext by simulating post-edits. Our results on post-editing for machine translation and paraphrasing evaluate the performance of our approach. We show +11.4 BLEU with limited post-editing effort on the WMT-14 English-German translation task (25.2 to 36.6), which represents +5.9 BLEU over the post-editing baseline (30.7 to 36.6).

pdf
```
@article{grangier:quickedit:2018,
  author    = {David Grangier and Michael Auli},
  title     = {QuickEdit: Editing Text and Translations by Crossing Words Out},
  year      = {2018},
  booktitle   = {Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)},
}
```

2017

Convolutional Sequence to Sequence Learning
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, Yann N. Dauphin - International Conference on Machine Learning (ICML). 2017.
The prevalent approach to sequence to sequence learning maps an input sequence to a variable length output sequence via recurrent neural networks. We introduce an architecture based entirely on convolutional neural networks. Compared to recurrent models, computations over all elements can be fully parallelized during training and optimization is easier since the number of non-linearities is fixed and independent of the input length. Our use of gated linear units eases gradient propagation and we equip each decoder layer with a separate attention module. We outperform the accuracy of the deep LSTM setup of Wu et al. (2016) on both WMT'14 English-German and WMT'14 English-French translation at an order of magnitude faster speed, both on GPU and CPU.

pdf
```
@inproceedings{gehring:convs2s:2017,
  author    = {Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, Yann N. Dauphin},
  title     = {Convolutional Sequence to Sequence Learning},
  year      = {2017},
  booktitle = {International Conference on Machine Learning {(ICML)}}
}
```
Language Modeling with Gated Convolutional Networks
Yann N. Dauphin, Angela Fan, Michael Auli and David Grangier - International Conference on Machine Learning (ICML). 2017.
The pre-dominant approach to language model- ing to date is based on recurrent neural networks. In this paper we present a convolutional approach to language modeling. We introduce a novel gating mechanism that eases gradient propagation and which performs better than the LSTM- style gating of Oord et al. (2016b) despite being simpler. We achieve a new state of the art on WikiText-103 as well as a new best single-GPU result on the Google Billion Word benchmark. In settings where latency is important, our model achieves an order of magnitude speed-up compared to a recurrent baseline since computation can be parallelized over time. To our knowledge, this is the first time a non-recurrent approach out- performs strong recurrent models on these tasks.

pdf
```
@inproceedings{dauphin:gatedlm:2017,
  author    = {Yann N. Dauphin and Angela Fan and Michael Auli and David Grangier},
  title     = {Language Modeling with Gated Convolutional Networks},
  year      = {2017},
  booktitle = {International Conference on Machine Learning {(ICML)}}
}
```
Efficient softmax approximation for GPUs
E. Grave and A. Joulin and M. Cisse and D. Grangier and H. Jegou - International Conference on Machine Learning (ICML). 2017.
We propose an approximate strategy to efficiently train neural network based language models over very large vocabularies. Our approach, called adaptive softmax, circumvents the linear dependency on the vocabulary size by exploiting the unbalanced word distribution to form clusters that explicitly minimize the expectation of computational complexity. Our approach further reduces the computational cost by exploiting the specificities of modern architectures and matrix-matrix vector operations, making it particularly suited for graphical processing units. Our experiments carried out on standard benchmarks, such as EuroParl and One Billion Word, show that our approach brings a large gain in efficiency over standard approximations while achieving an accuracy close to that of the full softmax.

pdf
```
@inproceedings{grave:softmax:2017,
  author    = {Edouard Grave and Armand Joulin and Moustapha Cisse and David Grangier and Herve Jegou},
  title     = {Strategies for Training Large Vocabulary Neural Language Models},
  year      = {2017},
  booktitle = {International Conference on Machine Learning {(ICML)}}
}
```
A Convolutional Encoder Model for Neural Machine Translation
Jonas Gehring, Michael Auli, David Grangier, Yann N. Dauphin - Conference of the Association for Computational Linguistics (ACL). 2017.
The prevalent approach to neural machine translation relies on bi-directional LSTMs to encode the source sentence. In this paper we present a faster and conceptually simpler architecture based on a succession of convolutional layers. This allows to encode the entire source sentence simultaneously compared to recurrent networks for which computation is constrained by temporal dependencies. We achieve a new state-of-the-art on WMT'16 English-Romanian translation and outperform several recently published results on the WMT'15 English-German task. We also achieve almost the same accuracy as a very deep LSTM setup on WMT'14 English-French translation. Our convolutional encoder speeds up CPU decoding by more than two times at the same or higher accuracy as a strong bi-directional LSTM baseline.

pdf
```
@inproceedings{gehring:convnmt:2017,
  author    = {Jonas Gehring and Michael Auli and David Grangier and Yann N. Dauphin},
  title     = {A Convolutional Encoder Model for Neural Machine Translation},
  year      = {2017},
  booktitle = {Conference of the Association for Computational Linguistics ({ACL})}
}
```

2016

Iterative Refinement for Machine Translation
Roman Novak, Michael Auli, David Grangier - arXiv:1610.06602. 2016.
Existing machine translation decoding algorithms generate translations in a strictly monotonic fashion and never revisit previous decisions. As a result, earlier mistakes cannot be corrected at a later stage. In this paper, we present a translation scheme that starts from an initial guess and then makes iterative improvements that may revisit previous decisions. We parameterize our model as a convolutional neural network that predicts discrete substitutions to an existing translation based on an attention mechanism over both the source sentence as well as the current translation output. By making less than one modification per sentence, we improve the output of a phrase-based translation system by up to 0.4 BLEU on WMT15 German-English translation.

pdf
```
@article{novak:refinement:2016,
  author    = {Roman Novak and Michael Auli and David Grangier},
  title     = {Iterative Refinement for Machine Translation},
  year      = {2016},
  booktitle = {arxiv}
}
```
Vocabulary Selection Strategies for Neural Machine Translation
Gurvan L'Hostis, David Grangier, Michael Auli - arXiv:1610.00072. 2016.
Classical translation models constrain the space of possible outputs by selecting a subset of translation rules based on the input sentence. Recent work on improving the efficiency of neural translation models adopted a similar strategy by restricting the output vocabulary to a subset of likely candidates given the source. In this paper we experiment with context and embedding-based selection methods and extend previous work by examining speed and accuracy trade-offs in more detail. We show that decoding time on CPUs can be reduced by up to 90% and training time by 25% on the WMT15 English-German and WMT16 English-Romanian tasks at the same or only negligible change in accuracy. This brings the time to decode with a state of the art neural translation system to just over 140 msec per sentence on a single CPU core for English-German.

pdf
```
@article{lhostis:selectionmt:2016,
  author    = {Gurvan L'Hostis and David Grangier and Michael Auli},
  title     = {Vocabulary Selection Strategies for Neural Machine Translation},
  year      = {2016},
  booktitle = {arxiv}
}
```
Neural Generation of Text from Structured Data with Application to the Bibliography Domain
Remi Lebret, David Grangier, Michael Auli - Conference on Empirical Methods in Natural Language Processing (EMNLP). 2016.
This paper introduces a neural model for concept-to-text generation that scales to large, rich domains. We experiment with a new dataset of biographies from Wikipedia that is an order of magnitude larger than existing resources with over 700k samples. The dataset is also vastly more diverse with a 400k vocabulary, compared to a few hundred words for Weathergov or Robocup. Our model builds upon recent work on conditional neural language model for text generation. To deal with the large vocabulary, we extend these models to mix a fixed vocabulary with copy actions that transfer sample-specific words from the input database to the generated output sentence. Our neural model significantly out-performs a classical Kneser-Ney language model adapted to this task by nearly 15 BLEU.

pdf
```
@inproceedings{lebret:emnlp:2016,
  author    = {Remi Lebret and David Grangier and Michael Auli},
  title     = {Neural Generation of Text from Structured Data with Application to the Bibliography Domain},
  year      = {2016},
  booktitle = {Conference on Empirical Methods in Natural Language Processing (EMNLP)}
}
```
Strategies for Training Large Vocabulary Neural Language Models
W. Chen, D. Grangier and M. Auli - Conference of the Association for Computational Linguistics (ACL). 2016.
Training neural network language models over large vocabularies is still computationally very costly compared to count-based models such as Kneser-Ney. At the same time, neural language models are gaining popularity for many applications such as speech recognition and machine translation whose success depends on scalability. We present a systematic comparison of strategies to represent and train large vocabularies, including softmax, hierarchical softmax, target sampling, noise contrastive estimation and self normalization. We further extend self normalization to be a proper estimator of likelihood and introduce an efficient variant of softmax. We evaluate each method on three popular benchmarks, examining performance on rare words, the speed/accuracy trade-off and complementarity to Kneser-Ney.

pdf
```
@inproceedings{chen:acl:2016,
  author    = {Wenlin Chen and David Grangier and Michael Auli},
  title     = {Strategies for Training Large Vocabulary Neural Language Models},
  year      = {2016},
  booktitle = {Conference of the Association for Computational Linguistics (ACL)}
}
```
Interactive Semantic Featuring for Text Classification
C. Jandot, P. Simard, M. Chickering, D. Grangier, J. Suh - ICML Workshop on Human Interpretability in Machine Learning (WHI). 2016.
In text classification, dictionaries can be used to define human-comprehensible features. We propose an improvement to dictionary features called smoothed dictionary features. These features recognize document contexts instead of n-grams. We describe a principled methodology to solicit dictionary features from a teacher, and present results showing that models built using these human-comprehensible features are competitive with models trained with Bag of Words features.

pdf
```
@inproceedings{jandot:2016:whi,
  author    = {Camille Jandot and Patrice Simard and Max Chickering and David Grangier and Jina Suh},
  title     = {Interactive Semantic Featuring for Text Classification},
  year      = {2016},
  booktitle = { ICML Workshop on Human Interpretability in Machine Learning (WHI)}
}
```
Predicting distributions with Linearizing Belief Networks
Y. Dauphin, D. Grangier - International Conference on Learning Representation (ICLR). 2016.
Conditional belief networks introduce stochastic binary variables in neural networks. Contrary to a classical neural network, a belief network can predict more than the expected value of the output Y given the input X. It can predict a distribution of outputs Y which is useful when an input can admit multiple outputs whose average is not necessarily a valid answer. Such networks are particularly relevant to inverse problems such as image prediction for denoising, or text to speech. However, traditional sigmoid belief networks are hard to train and are not suited to continuous problems. This work introduces a new family of networks called linearizing belief nets or LBNs. A LBN decomposes into a deep linear network where each linear unit can be turned on or off by non-deterministic binary latent units. It is a universal approximator of real-valued conditional distributions and can be trained using gradient descent. Moreover, the linear pathways efficiently propagate continuous information and they act as multiplicative skip-connections that help optimization by removing gradient diffusion. This yields a model which trains efficiently and improves the state-of-the-art on image denoising and facial expression generation with the Toronto faces dataset.

pdf
```
@inproceedings{dauphin:2016:iclr,
  author    = {Yann N. Dauphin and David Grangier},
  title     = {Predicting distributions with Linearizing Belief Networks},
  year      = {2016},
  booktitle = {International Conference on Learning Representation (ICLR)}
}
```

2014

ICE: Enabling Non-Experts to Build Models Interactively for Large-Scale Lopsided Problems
P. Simard, M. Chickering, A. Lakshmiratan, L. Bottou, C. Garcia, D. Grangier, S. Amershi, J. Verwey, J. Suh - arXiv:1409.4814. 2014.
Quick interaction between a human teacher and a learning machine presents numerous benefits and challenges when working with web-scale data. The human teacher guides the machine towards accomplishing the task of interest. The learning machine leverages big data to find examples that maximize the training value of its interaction with the teacher. When the teacher is restricted to labeling examples selected by the machine, this problem is an instance of active learning. When the teacher can provide additional information to the machine (e.g., suggestions on what examples or predictive features should be used) as the learning task progresses, then the problem becomes one of interactive learning. To accommodate the two-way communication channel needed for efficient interactive learning, the teacher and the machine need an environment that supports an interaction language. The machine can access, process, and summarize more examples than the teacher can see in a lifetime. Based on the machine's output, the teacher can revise the definition of the task or make it more precise. Both the teacher and the machine continuously learn and benefit from the interaction. We have built a platform to (1) produce valuable and deployable models and (2) support research on both the machine learning and user interface challenges of the interactive learning problem. The platform relies on a dedicated, low-latency, distributed, in-memory architecture that allows us to construct web-scale learning machines with quick interaction speed. The purpose of this paper is to describe this architecture and demonstrate how it supports our research efforts. Preliminary results are presented as illustrations of the architecture but are not the primary focus of the paper.

pdf
```
@article{simard:2014:ice,
  author    = {Patrice Y. Simard and
               David Maxwell Chickering and
               Aparna Lakshmiratan and
               Denis Xavier Charles and
               L{\'{e}}on Bottou and
               Carlos Garcia Jurado Suarez and
               David Grangier and
               Saleema Amershi and
               Johan Verwey and
               Jina Suh},
  title     = {{ICE:} Enabling Non-Experts to Build Models Interactively for Large-Scale
               Lopsided Problems},
  journal   = {CoRR},
  volume    = {abs/1409.4814},
  year      = {2014},
  url       = {http://arxiv.org/abs/1409.4814},
  timestamp = {Thu, 02 Oct 2014 07:52:03 +0200},
  biburl    = {http://dblp.uni-trier.de/rec/bib/journals/corr/SimardCLCBSGAVS14},
  bibsource = {dblp computer science bibliography, http://dblp.org}
}
```

2013

GBC: gradient boosting consensus model for heterogeneous data
X. Shi, JF. Paiement, D. Grangier and P. Yu - Statistical Analysis and Data Mining (SADM). 2013.
With the rapid development of database technologies, multiple data sources may be available for a given learning task (e.g. collaborative ﬁltering). However, the data sources may contain diﬀerent types of features. For example, users' proﬁles can be used to build recommendation systems. In addition, a model can also use users' historical behaviors and social networks to infer users' interests on related products. We argue that it is desirable to collectively use any available multiple heterogeneous data sources in order to build eﬀective learning models. We call this framework heterogeneous learning. In our proposed setting, data sources can include (i) nonoverlapping features, (ii) nonoverlapping instances, and (iii) multiple networks (i.e. graphs) that connect instances. In this paper, we propose a general optimization framework for heterogeneous learning, and devise a corresponding learning model from gradient boosting. The idea is to minimize the empirical loss with two constraints: (1) there should be consensus among the predictions of overlapping instances (if any) from diﬀerent data sources; (2) connected instances in graph datasets may have similar predictions. The objective function is solved by stochastic gradient boosting trees. Furthermore, a weighting strategy is designed to emphasize informative data sources, and deemphasize the noisy ones. We formally prove that the proposed strategy leads to a tighter error bound. This approach consistently outperforms a standard concatenation of data sources on movie rating prediction, number recognition, and terrorist attack detection tasks. Furthermore, the approach is evaluated on AT&T's distributed database with over 500 000 instances, 91 diﬀerent data sources, and over 45 000 000 joined features. We observe that the proposed model can improve out-of-sample error rate substantially.

pdf
```
@article{shi:2013:gbc,
  title={GBC: gradient boosting consensus model for heterogeneous data},
  author={Shi, Xiaoxiao and Paiement, Jean-Francois and Grangier, David and Yu, Philip S},
  journal={Statistical Analysis and Data Mining},
  year={2013},
  publisher={Wiley Online Library}
}
```

2012

Learning from Heterogeneous Sources via Gradient Boosting Consensus
X. Shi, JF. Paiement, D. Grangier and P. Yu - SIAM International Conference on Data Mining (SDM). 2012.
Multiple data sources containing different types of features may be available for a given task. For instance, users’ profiles can be used to build recommendation systems. In addition, a model can also use users’ historical behaviors and social networks to infer users’ interests on related products. We argue that it is desirable to collectively use any available multiple heterogeneous data sources in order to build effective learning models. We call this framework heterogeneous learning. In our proposed setting, data sources can include (i) non-overlapping features, (ii) non-overlapping instances, and (iii) multiple networks (i.e. graphs) that connect instances. In this paper, we propose a general optimization framework for heterogeneous learning, and devise a corresponding learning model from gradient boosting. The idea is to minimize the empirical loss with two constraints: (1) There should be consensus among the predictions of overlapping instances (if any) from different data sources; (2) Connected instances in graph datasets may have similar predictions. The objective function is solved by stochastic gradient boosting trees. Furthermore, a weighting strategy is designed to emphasize informative data sources, and deemphasize the noisy ones. We formally prove that the proposed strategy leads to a tighter error bound. This approach consistently outperforms a standard concatenation of data sources on movie rating prediction, number recognition and terrorist attack detection tasks. We observe that the proposed model can improve out-of-sample error rate by as much as 80%.

pdf - ps
```
@inproceedings{shi:2012:heterogeneous_gbdt_sdm,
	author = "X. Shi and JF. Paiement and D. Grangier and P. Yu",
	title = "Learning from Heterogeneous Sources via Gradient Boosting Consensus",
	booktitle = "SIAM International Conference on Data Mining (SDM)",
	year = "2012",
}
```

2010

Feature Set Embedding for Incomplete Data
D. Grangier and I. Melvin - Advances in Neural Information Processing Systems (NIPS). 2010.
We present a new learning strategy for classification problems in which train and/or test data suffer from missing features. In previous work, instances are represented as vectors from some feature space and one is forced to impute missing values or to consider an instance-specific subspace. In contrast, our method considers instances as sets of (feature,value) pairs which naturally handle the missing value case. Building onto this framework, we propose a classification strategy for sets. Our proposal maps (feature,value) pairs into an embedding space and then non-linearly combines the set of embedded vectors. The embedding and the combination parameters are learned jointly on the final classification objective. This simple strategy allows great flexibility in encoding prior knowledge about the features in the embedding step and yields advantageous results compared to alternative solutions over several datasets.

pdf - ps
```
@inproceedings{grangier:2010:missing_nips,
	author = "D. Grangier and I. Melvin",
	title = "Feature Set Embedding for Incomplete Data",
	booktitle = "Advances in Neural Information Processing Systems (NIPS)",
	year = "2010",
}
```
Label Embedding Trees for Large Multi-Class Tasks
J. Weston, S. Bengio and D. Grangier - Advances in Neural Information Processing Systems (NIPS). 2010.
Multi-class classification becomes challenging at test time when the number of classes is very large and testing against every possible class can become computationally infeasible. This problem can be alleviated by imposing (or learning) a structure over the set of classes. We propose an algorithm for learning a tree-structure of classifiers which, by optimizing the overall tree loss, provides superior accuracy to existing tree labeling methods. We also propose a method that learns to embed labels in a low dimensional space that is faster than non-embedding approaches and has superior accuracy to existing embedding approaches. Finally we combine the two ideas resulting in the label embedding tree that outperforms alternative methods including One-vs-Rest while being orders of magnitude faster.

pdf - ps
```
@inproceedings{weston:2010:label_trees_nips,
	author = "J. Weston and S. Bengio and D. Grangier",
	title = "Label Embedding Trees for Large Multi-Class Tasks",
	booktitle = "Advances in Neural Information Processing Systems (NIPS)",
	year = "2010",
}
```
Half Transductive Ranking
B. Bai, J. Weston, D. Grangier, R. Collobert, C. Cortes and M. Mohri - Artificial Intelligence and Statistics (AISTATS). 2010.
We study the standard retrieval task of ranking a fixed set of items given a previously unseen query and pose it as the half-transductive ranking problem. Transductive representations (where the vector representation of each example is learned) allow the generation of highly nonlinear embeddings that capture the characteristics of object relationships without relying on a specific choice of features, and require only relatively simple optimization. Unfortunately, they have no direct out-of-sample extension. Inductive approaches on the other hand allow for the representation of unknown queries. We describe algorithms for this setting which have the advantages of both transductive and inductive approaches, and can be applied in unsupervised (either reconstruction-based or graph-based) and supervised ranking setups. We show empirically that our methods give strong performance on all three tasks.

pdf - ps
```
@inproceedings{bai:2010:halftrans_aistats,
	author = "B. Bai and J. Weston and D. Grangier and R. Collobert and C. Cortes and M. Mohri",
	title = "Half Transductive Ranking",
	booktitle = "Artificial Intelligence and Statistics (AISTATS)",
	year = "2010",
}
```

2009

Ranking with Half Transductive Models
B. Bai, J. Weston, D. Grangier, R. Collobert, C. Cortes and M. Mohri - NIPS Workshop on Advances in Ranking. 2009.
We study the standard retrieval task of ranking a fixed set of documents given a previously unseen query and pose it as the half-transductive ranking problem. The task is partly transductive as the document set is fixed. Existing transductive approaches are natural non-linear methods for this set, but have no direct out-of-sample extension. Functional approaches, on the other hand, can be applied to the unseen queries, but fail to exploit the availability of the document set in its full extent. This work introduces a half-transductive approach to benefit from the advantages of both transductive and functional approaches and show its empirical advantage in supervised ranking setups.

pdf - ps
```
@inproceedings{bai:2009:halftrans_nips,
	author = "B. Bai and J. Weston and D. Grangier and R. Collobert and C. Cortes and M. Mohri",
	title = "Ranking with Half Transductive Models",
	booktitle = "NIPS Workshop on Advances in Ranking",
	year = "2009",
}
```
Polynomial Semantic Indexing
B. Bai, J. Weston, D. Grangier, R. Collobert, K. Sadamasa, Y. Qi, C. Cortes and M. Mohri - Advances in Neural Information Processing Systems (NIPS). 2009.
We present a class of nonlinear (polynomial) models that are discriminatively trained to directly map from the word content in a query-document or document-document pair to a ranking score. Dealing with polynomial models on word features is computationally challenging. We propose a low-rank (but diagonal preserving) representation of our polynomial models to induce feasible memory and computation requirements. We provide an empirical study on retrieval tasks based on Wikipedia documents, where we obtain state-of-the-art performance while providing realistically scalable methods.

pdf - ps
```
@inproceedings{bai:2009:psi_nips,
	author = "B. Bai and J. Weston and D. Grangier and R. Collobert and K. Sadamasa and Y. Qi and C. Cortes and M. Mohri",
	title = "Polynomial Semantic Indexing",
	booktitle = "Advances in Neural Information Processing Systems (NIPS)",
	year = "2009",
}
```
Learning to Rank with (a Lot of) Word Features
B. Bai, J. Weston, D. Grangier, R. Collobert, Y. Qi, K. Sadamasa, O. Chapelle and K. Weinberger - Information Retrieval -- Special Issue on Learning to Rank, Springer. 2009.
In this article we present Supervised Semantic Indexing (SSI) which defines a class of nonlinear (quadratic) models that are discriminatively trained to directly map from the word content in a query-document or document-document pair to a ranking score. Like Latent Semantic Indexing (LSI), our models take account of correlations between words (synonymy, polysemy). However, unlike LSI our models are trained from a supervised signal directly on the ranking task of interest, which we argue is the reason for our superior results. As the query and target texts are modeled separately, our approach is easily generalized to different retrieval tasks, such as cross-language retrieval or online advertising placement. Dealing with models on all pairs of words features is computationally challenging. We propose several improvements to our basic model for addressing this issue, including low rank (but diagonal preserving) representations, correlated feature hashing (CFH) and sparsification. We provide an empirical study of all these methods on retrieval tasks based on Wikipedia documents as well as an Internet advertisement task. We obtain state-of-the-art performance while providing realistically scalable methods.

pdf - ps
```
@article{bai:2009:ssi_jir,
	author = "B. Bai and J. Weston and D. Grangier and R. Collobert and Y. Qi and K. Sadamasa and O. Chapelle and K. Weinberger",
	title = "Learning to Rank with (a Lot of) Word Features",
	journal = "Information Retrieval -- Special Issue on Learning to Rank",
	publisher = "Springer",
	year = "2009",
}
```
Supervised Semantic Indexing
B. Bai, J. Weston, D. Grangier, R. Collobert, Y. Qi, K. Sadamasa, O. Chapelle and K. Weinberger - ACM Conference on Information and Knowledge Management (CIKM). 2009.
In this article, we propose Supervised Semantic Indexing (SSI), an algorithm that is trained on (query, document) pairs of text documents to predict the quality of their match. Like Latent Semantic Indexing (LSI), our models take account of correlations between words (synonymy, polysemy). However, unlike LSI our models are trained with a supervised signal directly on the ranking task of interest, which we argue is the reason for our superior results. As the query and target texts are modeled separately, our approach is easily generalized to different retrieval tasks, such as online advertising placement. Dealing with models on all pairs of words features is computationally challenging. We propose several improvements to our basic model for addressing this issue, including low rank (but diagonal preserving) representations, and correlated feature hashing (CFH).We provide an empirical study of all these methods on retrieval tasks based on Wikipedia documents as well as an Internet advertisement task. We obtain state-of-the-art performance while providing realistically scalable methods.

pdf - ps
```
@inproceedings{bai:2009:ssi_cikm,
	author = "B. Bai and J. Weston and D. Grangier and R. Collobert and Y. Qi and K. Sadamasa and O. Chapelle and K. Weinberger",
	title = "Supervised Semantic Indexing",
	booktitle = "ACM Conference on Information and Knowledge Management (CIKM)",
	year = "2009",
}
```
Supervised Semantic Indexing
B. Bai, J.Weston, R. Collobert and D. Grangier - European Conference on Information Retrieval (ECIR). 2009.
We present a class of models that are discriminatively trained to directly map from the word content in a query-document or document-document pair to a ranking score. Like Latent Semantic Indexing (LSI), our models take account of correlations between words (synonymy, polysemy). However, unlike LSI our models are trained with a supervised signal directly on the task of interest, which we argue is the reason for our superior results. We provide an empirical study on Wikipedia documents, using the links to define document-document or query-document pairs, where we obtain state-of-the-art performance using our method.

pdf - ps
```
@inproceedings{bai:2009:ssi_ecir,
	author = "B. Bai and J.Weston and R. Collobert and D. Grangier",
	title = "Supervised Semantic Indexing",
	booktitle = "European Conference on Information Retrieval (ECIR)",
	year = "2009",
}
```
Discriminative Keyword Spotting
D. Grangier, J. Keshet and S. Bengio - Automatic Speech and Speaker Recognition: Large Margin and Kernel Methods, J. Keshet and S. Bengio (Ed.), Wiley. 2009.
This chapter introduces a discriminative approach for the detection of keywords in speech utterances. Specifically, this work proposes a learning algorithm, which aims at maximizing the area under the receiver operating curve, given a set of training spotting problems. This algorithm relies on a large margin formulation of the spotting task, and adopts an efficient online learning strategy. This approach contrasts with standard spotting strategies based on Hidden Markov Models (HMMs), for which the training procedure does not maximize a loss directly related to the spotting performance. Different experiments performed over TIMIT and WSJ data show the advantage of our approach over HMM-based alternatives.

pdf - ps
```
@incollection{grangier:2009:kws_book,
	author = "D. Grangier, J. Keshet and S. Bengio",
	title = "Discriminative Keyword Spotting",
	booktitle = "Automatic Speech and Speaker Recognition: Large Margin and Kernel Methods",
	editor = "J. Keshet and S. Bengio",
	publisher = "Wiley",
	year = "2009",
}
```

2008

Machine Learning for Information Retrieval
D. Grangier - Ecole Polytechnique Federale de Lausanne - PhD Thesis 4088. 2008.
In this thesis, we explore the use of machine learning techniques for information retrieval. More specifically, we focus on ad-hoc retrieval, which is concerned with searching large corpora to identify the documents relevant to user queries. This identification is performed through a ranking task. Given a user query, an ad-hoc retrieval system ranks the corpus documents, so that the documents relevant to the query ideally appear above the others. In a machine learning framework, we are interested in proposing learning algorithms that can benefit from limited training data in order to identify a ranker likely to achieve high retrieval performance over unseen documents and queries. This problem presents novel challenges compared to traditional learning tasks, such as regression or classification. First, our task is a ranking problem, which means that the loss for a given query cannot be measured as a sum of an individual loss suffered for each corpus document. Second, most retrieval queries present a highly unbalanced setup, with a set of relevant documents accounting only for a very small fraction of the corpus. Third, ad-hoc retrieval corresponds to a kind of ``double'' generalization problem, since the learned model should not only generalize to new documents but also to new queries. Finally, our task also presents challenging efficiency constraints, since ad-hoc retrieval is typically applied to large corpora. The main objective of this thesis is to investigate the discriminative learning of ad-hoc retrieval models. For that purpose, we propose different models based on kernel machines or neural networks adapted to different retrieval contexts. The proposed approaches rely on different online learning algorithms that allow efficient learning over large corpora.

pdf - ps
```
@phdthesis{grangier:2008:phd_thesis,
	author = "D. Grangier",
	title = "Machine Learning for Information Retrieval",
	number = "4088",
	school = "Ecole Polytechnique Federale de Lausanne",
	year = "2008",
}
```
Discriminative Keyword Spotting
J. Keshet, D. Grangier and S. Bengio - Speech Communication. 2008.
This paper proposes a new approach for keyword spotting, which is not based on HMMs. Unlike previous approaches, the proposed method employs a discriminative learning procedure, in which the learning phase aims at maximizing the area under the ROC curve, as this quantity is the most common measure to evaluate keyword spotters. The keyword spotter we devise is based on mapping the input acoustic representation of the speech utterance along with the target keyword into a vector space. Building on techniques used for large margin and kernel methods for predicting whole sequences, our keyword spotter distills to a classifier in this vector-space, which separates speech utterances in which the keyword is uttered from speech utterances in which the keyword is not uttered. We describe a simple iterative algorithm for training a keyword spotter and discuss its formal properties. Experiments with the TIMIT corpus show that our method outperforms the conventional HMM-based approach. Further experiments using the TIMIT trained model, but tested on the WSJ dataset, show that without further training our method outperforms the conventional HMM-based approach.

pdf - ps
```
@article{grangier:2008:kws_journal,
	author = "J. Keshet and D. Grangier and S. Bengio",
	title = "Discriminative Keyword Spotting",
	journal = "Speech Communication",
	year = "2008",
}
```
A Discriminative Kernel-based Model to Rank Images from Text Queries
D. Grangier and S. Bengio - IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI). 2008.
This paper introduces a discriminative model for the retrieval of images from text queries. Our approach formalizes the retrieval task as a ranking problem, and introduces a learning procedure optimizing a criterion related to the ranking performance. The proposed model hence addresses the retrieval problem directly and does not rely on an intermediate image annotation task, which contrasts with previous research. Moreover, our learning procedure builds upon recent work on the online learning of kernel-based classifiers. This yields an efficient, scalable algorithm, which can benefit from recent kernels developed for image comparison. The experiments performed over stock photography data show the advantage of our discriminative ranking approach over state-of-the-art alternatives (e.g. our model yields 26.3% average precision over the Corel dataset, which should be compared to 22.0%, for the best alternative model evaluated). Further analysis of the results shows that our model is especially advantageous over difficult queries such as queries with few relevant pictures or multiple-word queries.

pdf - ps
```
@article{grangier:2008:tpami,
	author = "D. Grangier and S. Bengio",
	title = "A Discriminative Kernel-based Model to Rank Images from Text Queries",
	journal = "IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)",
	year = "2008",
}
```

2007

Learning the Inter-frame Distance for Discriminative Template-based Keyword Detection
D. Grangier and S. Bengio - International Conference on Speech Processing (INTERSPEECH). 2007.
This paper proposes a discriminative approach to template-based keyword detection. We introduce a method to learn the distance used to compare acoustic frames, a crucial element for template matching approaches. The proposed algorithm estimates the distance from data, with the objective to produce a detector maximizing the Area Under the receiver operating Curve (AUC), i.e. the standard evaluation measure for the keyword detection problem. The experiments performed over a large corpus, SpeechDatII, suggest that our model is effective compared to an HMM system, e.g. the proposed approach reaches 93.8% of averaged AUC compared to 87.9% for the HMM.

pdf - ps
```
@inproceedings{grangier:2007:rr_07-15,
	author = "D. Grangier and S. Bengio",
	title = "Learning the Inter-frame Distance for Discriminative Template-based Keyword Detection",
	booktitle = "International Conference on Speech Processing (INTERSPEECH)",
	year = "2007",
}
```
Discriminative Keyword Spotting
J. Keshet, D. Grangier and S. Bengio - International Workshop on Non-LInear Speech Processing (NOLISP). 2007.
This paper proposes a new approach for keyword spotting, which is not based on HMMs. The proposed method employs a new discriminative learning procedure, in which the learning phase aims at maximizing the area under the ROC curve, as this quantity is the most common measure to evaluate keyword spotters. The keyword spotter we devise is based on non-linearly mapping the input acoustic representation of the speech utterance along with the target keyword into an abstract vector space. Building on techniques used for large margin methods for predicting whole sequences, our keyword spotter distills to a classifier in the abstract vector-space which separates speech utterances in which the keyword was uttered from speech utterances in which the keyword was not uttered. We describe a simple iterative algorithm for learning the keyword spotter and discuss its formal properties. Experiments with the TIMIT corpus show that our method outperforms the conventional HMM-based approach.

pdf - ps
```
@inproceedings{keshet:2007:nolisp,
	author = "J. Keshet, D. Grangier and S. Bengio",
	title = "Discriminative Keyword Spotting",
	booktitle = "International Workshop on Non-LInear Speech Processing (NOLISP)",
	year = "2007",
}
```

2006

A Neural Network to Retrieve Images from Text Queries
D. Grangier and S. Bengio - International Conference on Artificial Neural Networks (ICANN)}. 2006.
This work presents a neural network for the retrieval of images from text queries. The proposed network is composed of two main modules: the first one extracts a global picture representation from local block descriptors while the second one aims at solving the retrieval problem from the extracted representation. Both modules are trained jointly to minimize a loss related to the retrieval performance. This approach is shown to be advantageous when compared to previous models relying on unsupervised feature extraction: average precision over Corel queries reaches 26.2% for our model, which should be compared to 21.6% for PAMIR, the best alternative.

pdf - ps
```
@inproceedings{grangier:2006:icann,
	author = "D. Grangier and S. Bengio",
	title = "A Neural Network to Retrieve Images from Text Queries",
	booktitle = "International Conference on Artificial Neural Networks (ICANN)}",
	year = "2006",
}
```
Learning to Retrieve Images from Text Queries with a Discriminative Model
D. Grangier, F. Monay and S. Bengio - International Workshop on Adaptive Multimedia Retrieval (AMR). 2006.
This work presents a discriminative model for the retrieval of pictures from text queries. The core idea of this approach is to minimize a loss directly related to the retrieval performance of the model. For that purpose, we rely on a ranking loss which has recently been successfully applied to text retrieval problems. The experiments performed over the Corel dataset show that our approach compares favorably with generative models that constitute the state-of-the-art (e.g. our model reaches 21.6% mean average precision with Blob and SIFT features, compared to 16.7% for PLSA, the best alternative).

pdf - ps
```
@inproceedings{grangier:2006:amr,
	author = "D. Grangier and F. Monay and S. Bengio",
	title = "Learning to Retrieve Images from Text Queries with a Discriminative Model",
	booktitle = "International Workshop on Adaptive Multimedia Retrieval (AMR)",
	year = "2006",
}
```
A Discriminative Approach for the Retrieval of Images from Text Queries
D. Grangier, F. Monay and S. Bengio - European Conference on Machine Learning (ECML). 2006.
This work proposes a new approach to the retrieval of images from text queries. Contrasting with previous work, this method relies on a discriminative model: the parameters are selected in order to minimize a loss related to the ranking performance of the model, i.e. its ability to rank the relevant pictures above the non-relevant ones when given a text query. In order to minimize this loss, we introduce an adaptation of the recently proposed Passive-Aggressive algorithm. The generalization performance of this approach is then compared with alternative models over the Corel dataset. These experiments show that our method outperforms the current state-of-the-art approaches, e.g. the average precision over Corel test data is 21.6% for our model versus 16.7% for the best alternative, Probabilistic Latent Semantic Analysis.

pdf - ps
```
@inproceedings{grangier:2006:ecml,
	author = "D. Grangier and F. Monay and S. Bengio",
	title = "A Discriminative Approach for the Retrieval of Images from Text Queries",
	booktitle = "European Conference on Machine Learning (ECML)",
	year = "2006",
}
```

2005

A Discriminative Decoder for the Recognition of Phoneme Sequences
D. Grangier and S. Bengio - IDIAP Research Report RR2005-67. 2005.
In this report, we propose a discriminative decoder for phoneme recognition, i.e. the identification of the uttered phoneme sequence from a speech recording. This task is solved as a 3 step process: a phoneme classifier first classifies each accoustic frame, then temporal consistency features (TCF) are extracted from the phoneme classifier outputs, and finally a sequence decoder identifies the phoneme sequence according to the TCF.

pdf - ps
```
@techreport{grangier:2005:idiap-05-67,
	author = "D. Grangier and S. Bengio",
	title = "A Discriminative Decoder for the Recognition of Phoneme Sequences",
	number = "67",
	institution = "IDIAP",
	year = "2005",
}
```
Exploiting Hyperlinks to Learn a Retrieval Model
D. Grangier and S. Bengio - NIPS Workshop on Learning to Rank. 2005.
Information Retrieval (IR) aims at solving a ranking problem: given a query q and a corpus C, the documents of C should be ranked such that the documents relevant to q appear above the others. This task is generally performed by ranking the documents d in C according to their similarity with respect to q, sim (q,d). The identification of an effective function a,b -> sim(a,b) could be performed using a large set of queries with their corresponding relevance assessments. However, such data are especially expensive to label, thus, as an alternative, we propose to rely on hyperlink data which convey analogous semantic relationships. We then empirically show that a measure sim inferred from hyperlinked documents can actually outperform the state-of-the-art Okapi approach, when applied over a non-hyperlinked retrieval corpus.

pdf - ps
```
@inproceedings{grangier:2005:nips_workshop,
	author = "D. Grangier and S. Bengio",
	title = "Exploiting Hyperlinks to Learn a Retrieval Model",
	booktitle = "NIPS Workshop on Learning to Rank",
	year = "2005",
}
```
Inferring Document Similarity from Hyperlinks
D. Grangier and S. Bengio - ACM Conference on Information and Knowledge Management (CIKM). 2005.
Assessing semantic similarity between text documents is a crucial aspect in Information Retrieval systems. In this work, we propose to use hyperlink information to derive a similarity measure that can then be applied to compare any text documents, with or without hyperlinks. As linked documents are generally semantically closer than unlinked documents, we use a training corpus with hyperlinks to infer a function a,b -> sim(a,b) that assigns a higher value to linked documents than to unlinked ones. Two sets of experiments on different corpora show that this function compares favorably with OKAPI matching on document retrieval tasks.

pdf - ps
```
@inproceedings{grangier:2005:cikm,
	author = "D. Grangier and S. Bengio",
	title = "Inferring Document Similarity from Hyperlinks",
	booktitle = "ACM Conference on Information and Knowledge Management (CIKM)",
	year = "2005",
}
```
Effect of Segmentation Method on Video Retrieval Performance
D. Grangier and A. Vinciarelli - IEEE International Conference on Multimedia and Expo (ICME). 2005.
This paper presents experiments that evaluate the effect of different video segmentation methods on text-based video retrieval. Segmentations relying on modalities like speech, video and text or their combination are compared with a baseline sliding window segmentation. The results suggest that even with the sliding window segmentation, acceptable performance can be obtained on a broadcast news retrieval task. Moreover, in the case where manually segmented data are available for training, the approach combining the different modalities can lead to IR results close to those obtained with a manual segmentation.

pdf - ps
```
@inproceedings{grangier:2005:icme,
	author = "D. Grangier and A. Vinciarelli",
	title = "Effect of Segmentation Method on Video Retrieval Performance",
	booktitle = "IEEE International Conference on Multimedia and Expo (ICME)",
	year = "2005",
}
```

2004

Effect of Recognition Errors on Text Clustering
D. Grangier and A. Vinciarelli - IDIAP Research Report RR2004-82. 2004.
This paper presents clustering experiments performed over noisy texts (i.e. texts that have been extracted through an automatic process like character or speech recognition). The effect of recognition errors is investigated by comparing clustering results performed over both clean (manually typed data) and noisy (automatic speech transcriptions) versions of the same speech recording corpus.

pdf - ps
```
@techreport{grangier:2004:idiap-04-82,
	author = "D. Grangier and A. Vinciarelli",
	title = "Effect of Recognition Errors on Text Clustering",
	number = "82",
	institution = "IDIAP",
	year = "2004",
}
```

2003

Information Retrieval on Noisy Text
D. Grangier, A. Vinciarelli and H. Bourlard - IDIAP Research Report RR2003-8. 2003.
Spoken Document Retrieval (SDR) consists in retrieving segments of a speech database that are relevant to a query. The state-of-the-art approach to the SDR problem consists in transcribing the speech data into digital text before applying common Information Retrieval (IR) techniques. The transcription, produced by an Automatic Speech Recognition system, contains recognition errors. These errors can be referred to as noise. This thesis investigates the effect of this noise on the retrieval process. We compare the results obtained with clean and noisy data at different steps of the retrieval process. To perform such a task, standard IR measures (precision, recall, break-even point, etc.) are used. It is shown that even with very different error rates (10% vs 30%), the performances obtained over noisy text are only slightly lower than those over clean text (9% degradation of average precision for our complete IR system, 45.2% vs 41.2%).

pdf - ps
```
@techreport{com-03-08,
	author = "D. Grangier and A. Vinciarelli and H. Bourlard",
	title = "Information Retrieval on Noisy Text",
	number = "8",
	Keywords = "Information Retrieval, Speech, Spoken Documents Retrieval, Noisy Text",
	institution = "IDIAP",
	year = "2003",
}
```

Code, Data & Demos

Fairseq in lua/torch and pytorch
Fairseq is a sequence-to-sequence learning toolkit tailored to Neural Machine Translation (NMT). It implements the convolutional NMT models proposed as well as a standard LSTM-based model. It features multi-GPU training and fast beam search generation for CPU and GPU. We provide pre-trained models for English to French, English to German and English to Romanian translation.
Wikipedia biography dataset
WikiBio is a dataset of over 700K biography paragraphs paired with structured data from Wikipedia. It has been introduced in our EMNLP'16 paper to evaluate the generation of text from structured data.
Deep Convolutional Networks for Scene Parsing
We use deep convolutional networks for modeling the complex label structure of street scenes, relying on a supervised greedy learning strategy.
Discriminative Keyword Spotter
Our algorithm learns a keyword spotter with the objective to maximize the area under the ROC curve. The code is available under BSD licence.
PAMIR Code
PAMIR is a model to retrieve images from text queries, introduced last year. The code is available under BSD licence.

Tutorials and Workshops

ICML'16 Workshop: Neural Nets Back to the Future
A workshop linking the past, present and future research on neural networks, co-organized with John Platt, Leon Bottou and Tomas Mikolov.
NIPS'14 Workshop on Learning Semantics
co-organized with Cedric Archambeau, Antoine Bordes, Leon Bottou and Chris Burges.
Introduction to Machine Learning and Rankings
This is a lecture I gave to the UC Berkeley School of Information in September 2011. Jean-Francois Paiement prepared an associated lab on Boosted Trees.
NIPS'06 Workshop on Learning to Compare Examples
Samy Bengio and I organized a workshop on learning distances and similarity measures, both the proceedings and videos of the talks are available online.

Colleagues and Co-Authors

Michael Auli, Facebook AI Research, Menlo Park, CA, USA
Bing Bai, NEC Labs America, Princeton, NJ, USA
Samy Bengio, Apple Inc, Cupertino CA, USA
Léon Bottou, Facebook AI Research, New York, NY, USA
Ronan Collobert, Facebook, Menlo Park, CA, USA
Yann Dauphin, Google Research, Accra, Ghana
Angela Fan, Facebook AI Research, Paris, France
Markus Freitag, Google Research, Mountain View, CA, USA
Jonas Gehring, Facebook, Paris, France
Joseph Keshet, Bar Ilan University, Ramat Gan, Israel
Florent Monay, ams AG, Bern, Switzerland
Jean François Paiement, AT&T Labs Research, San Francisco, CA
Marc'Aurelio Ranzato, DeepMind, London, UK
Patrice Simard, Microsoft Research, Redmond, WA, USA
Jason Weston, Facebook AI Research, New York, NY, USA
Neil Zeghidour,Google Research, Paris, France