Daniel D. Johnson (Posts about neural network)

Introducing the GGT-NN

Daniel D. Johnson — Sun, 06 Nov 2016 00:00:00 GMT

Note to present-day readers: The use of the word "transformer" in this post, and in the corresponding ICLR paper, is unrelated to the now-popular Transformer architecture introduced in the paper "Attention is all you need", which was released shortly after this post.

Recently, I submitted a paper titled "Learning Graphical State Transitions" to the ICLR conference. (Update: My paper was accepted! I will be giving an oral presentation at ICLR 2017 in Toulon, France. See here for more details.) In it, I describe a new type of neural network architecture, called a Gated Graph Transformer Neural Network, that is designed to use graphs as an internal state. I demonstrate its performance on the bAbI tasks as well as on some other tasks with complex rules. While the main technical details are provided in the paper, I figured it would be worthwhile to describe the motivation and basic ideas here.

Note: Before I get too far into this post, if you have read my paper and are interested in replicating or extending my experiments, the code for my model is available on GitHub.

Another thing that I've noticed is that almost all of the papers on machine learning are about successes. This is an example of an overall trend in science to focus on the positive results, since they are the most interesting. But it can also be very useful to discuss the negative results as well. Learning what doesn't work is in some ways just as important as learning what does, and can save others from repeating the same mistakes. During my development of the GGT-NN, I had multiple iterations of the model, which all failed to learn anything interesting. The version of the model that worked was thus a product of an extended cycle of trial and error. In this post I will try to describe the failed models as well, and give my speculative theories for why they may not have been as successful.

Summer Research on the HMC Intelligent Music Software Team

Daniel D. Johnson — Mon, 08 Aug 2016 00:00:00 GMT

This summer, I had the chance to do research at Mudd as part of the Intelligent Music Software team. Every year, under the advisement of Prof. Robert Keller, members of the Intelligent Music Software team work on computational jazz improvisation, usually in connection with the Impro-Visor software tool. Currently, Impro-Visor uses a grammar-based generation approach for most of its generation ability. The goal this summer was to try to integrate neural networks into the generation process.

Reading an Artificial Mind: Visualizing the Neural Net

Daniel D. Johnson — Tue, 15 Sep 2015 19:51:51 GMT

In my previous post, I described my music-composing neural network. In some of the time since then, I extracted some interesting images to visualize the neural network's internal state. And this is what I got:

Composing Music With Recurrent Neural Networks

Daniel D. Johnson — Mon, 03 Aug 2015 00:53:44 GMT

(Update: A paper based on this work has been accepted at EvoMusArt 2017! See here for more details.)

It's hard not to be blown away by the surprising power of neural networks these days. With enough training, so called "deep neural networks", with many nodes and hidden layers, can do impressively well on modeling and predicting all kinds of data. (If you don't know what I'm talking about, I recommend reading about recurrent character-level language models, Google Deep Dream, and neural Turing machines. Very cool stuff!) Now seems like as good a time as ever to experiment with what a neural network can do.

For a while now, I've been floating around vague ideas about writing a program to compose music. My original idea was based on a fractal decomposition of time and some sort of repetition mechanism, but after reading more about neural networks, I decided that they would be a better fit. So a few weeks ago, I got to work designing my network. And after training for a while, I am happy to report remarkable success!

Here's a taste of things to come: