In the conference paper "Deep Reinforcement Learning with Hierarchical Recurrent Encoder-Decoder for Conversation", we trained an open domain dialogue agent to carry out an “engaging conversation” marked by topic coherent responses to the speaker’s queries, followed by open ended questions to prolong the conversation. The dialogue agent was trained end-to-end where a hierarchical recurrent neural net learned the grammar model, and small talk etiquette was learned over time using deep reinforcement learning with handcrafted reward functions.

This is part of a multi-year project supervised by professor Chris Callison-Burch. We develop a pipeline that ranks adjectives that are semantically similar but differ in intensity (i.e., fine, good, great). In particular, we leverage the Paraphrase database to construct a directed graph where vertices are adjectives and edges are adverbs, so that if there is an edge from vertex s to t with label u, then the string “u s” is a paraphrase of the string “t”. Then we developed an l1-penalized logistic regression model that inferred the relative strength of adjectives based on relationship with their neighbors. The ground truth is procured by Amazon mechanical turks. Preliminary tests show that our approach outperform the state of the art by significant margins measured by both pairwise accuracy and Kendall’s tau score.

In this paper published in ACL 2017, we model bilingual lexicon induction (learning word translations without bilingual parallel copora) as a matrix completion problem leverage diverse set of data from monolingual corpora to images. Our model achieves state-of-the-art performance in both high and low resource languages.

Based on the paper "A Large Multilingual Corpus for Learning Translations from Images" submitted to EMNLP 2017, we introduce a new large scale multilingual corpus of labeled images collected to facilitate research into learning translations through visual similarity. We have collected 100 images for up to 10,000 words in each of 100 foreign languages, plus images for each of their translations into English. In addition to the images, we collected the text from the web pages where each of the images appeared. Our dataset contains 35 million images and web pages, totaling 25 terabytes of data. We also release vectors represent each image using Scale Invariant Feature Transform (SIFT) features, color histograms and convolutional neural network (CNN) based features.

For a few years in my life I was obsessed with becoming better at mathmatics. This was one of my first proofs, and to this day still my favorite.