Musi

# .title-slide-h1[Human + AI Music Generation]
### .title-slide-h2[An analytical approach to original music creation]
.title-slide-h3[Terry Wang, Rima Mittal, Joshua Goldberg]

---

# Business opportunity

]

]

---

# Research Purpose

The purpose of this project is to develop a deep learning based music generator (instrumental music) which will help musicians and non-musicians alike to develop and refine musical ideas, and facilitate them in the music composition process.

## Deliverable outline

* A GAN/Variational AutoEncoder music generation model either using a time series approach or another analytical method.

* A framework (genetic algorithm) to update weights/parameters of the model. We plan to have the human user choose k "good" samples from n generated samples for each iteration.

* A user interface where the user can interact with the model.

---

# Two key parts

]

]

---

# Data and model description

* 130,000 midi files curated by a Reddit user, with a size of 3.65GB uncompressed. The collection has a great amount of variety representing all major genres in classical, jazz, metal, pop, etc.

* The generative algorithm will be of recurrent nature, where a snippet of monophonic midi track consisting of n number of quarter or `\(\frac{1}{16^\text{th}}\)` notes will be fed at a time into a neural network of input matrix dimension `\(129 \times n\)`.

* This neural network will then output a matrix `\(129 \times 1\)`, which will be compared to the immediate next quarter or `\(\frac{1}{16^\text{th}}\)` note of the input for back propagation. Thus, the model would take n quarter or `\(\frac{1}{16^\text{th}}\)` notes as input to predict the immediate next note.

* The adversarial algorithm is a simple discriminator to find whether a given input is from one of the original training samples or not.

???

First bullet point: The music data we have is a collection of 130,000 music files. The Genres include: Pop, Classical (Piano/Violin/Guitar), EDM, VideoGame, Movie/TV Theme.

Second bullet: The length is yet to be determined and will be determined through experimentation

Last bullet: The length of the generated music is yet to be determined, but in theory we should be able to generate music of unlimited length given the recurrent structure.

---

# Variables

Python libraries like __Music21__, __Librosa__, __PrettyMIDI__ are the toolkits we are using to extract meaningful information from our MIDI files.

<br>

The stream object extracted using __Music21__ can then be used to identify:

* Instruments

* Key Signatures

* Overlaps

* Time Signatures

* Measures

From the measure, we were then able to identify:

* Notes

* Chord

???

We are also working with other metrics like the Pitch Histogram and other Composition Parameters Analysis.

---

# Data structure example

<div id="htmlwidget-f4ca85bdf4c31db9dfe5" style="width:100%;height:auto;" class="datatables html-widget"></div>
<script type="application/json" data-for="htmlwidget-f4ca85bdf4c31db9dfe5">{"x":{"filter":"none","data":[["1","2","3","4","5","6","7","8","9","10","11","12","13","14"],["070527 (Oh for a Poet).MID","070110 (The Raven Days).MID","070606 (Hail the Comic Song).MID","Celine Dion - Fly 2.mid","070102 (I'll Love You Always).MID","051230.MID","050620B (The Children of the Night).MID","009count.mid","070107 (Music of the Third Kind).MID","060107-Sonnet 73.MID","060222-Tribute to Mozart.MID","09carnivalcuckoo.mid","040 Feat Erica Baxter - Dreams (Lost Witness Remix).mid","040FeatEricaBaxter_-_DreamsLostWitnessRemix__SamCera_20120708165703.mid"],["C major","G major","C major","E major","G major","G major","D major","G major","G major","F major","g minor","c# minor","d minor","d minor"],["['-', 'I6', 'V42', 'I6', 'vii6', 'I6', 'I53', 'v43', 'I64', 'vii65', 'i', 'I53', 'V65', 'I53', 'V7', 'I64', 'I53', 'V65', 'V', 'I53', 'V', 'i', 'vii', 'ii7', 'i', 'i', 'IV53', 'V7', 'ii53', 'I6', 'IV53', 'I53', 'vii53', 'V7', 'V7', 'vi65', 'i', 'I53', 'v43', 'I64', 'vii65', 'i', 'I53', 'V65', 'I53', 'V7', 'I64', 'I53', 'V65', 'V', 'I53', 'V', 'i', 'vii', 'ii7', 'i', 'i', 'IV53', 'V7', 'ii53', 'I6', 'IV53', 'I53', 'vii53', 'V7', 'V7', 'vi65', 'i']","['-', '-', 'I64', 'V53', 'I64', 'vii', 'vi53', 'ii64', 'V53', 'i', 'IV42', 'I53', 'V64', 'I53', 'V6', 'vi53', 'ii64', 'V53', 'iv43', 'I', 'I', 'V64', 'I53', 'V6', 'vi53', 'ii64', 'V53', 'iv43', 'i', 'V53', 'I53', 'V7', 'iii53', 'vi7', 'v7', 'I64', 'V7', 'vi53', 'ii64', 'V53', 'IV53', '-VI', '-vi', '-vii', 'I', 'I53', 'IV42', 'I53', 'V64', 'I53', 'V6', 'vi53', 'ii64', 'V53', 'iv43', 'I', 'I', 'V64', 'I53', 'V6', 'vi53', 'ii64', 'V53', 'iv43', 'i', 'V53', 'I53', 'V7', 'iii53', 'vi7', 'v7', 'I64', 'V7', 'vi53', 'ii64', 'V53', 'IV53', '-VI', '-vi', '-vii', 'I', 'I53', 'vi53', 'IV', 'i']","['-', '-', '-', '-', 'vii42', 'vii43', 'iii53', 'vii42', 'vii53', 'I64', 'vii', 'vi', 'ii', 'vii53', 'I64', 'vii', 'vi', 'V7', 'I', 'IV43', 'I53', 'V42', 'IV', 'I53', 'II42', 'V7', 'I53', 'I53', 'V7', 'I64', 'V43', 'I53', 'V43', 'I64', 'vi', 'I53', 'vii42', 'vii65', 'I64', 'vii', 'vi', 'IV', 'vii53', 'I64', 'vii', 'vi', 'V7', 'I', 'IV43', 'I53', 'V42', 'IV', 'I53', 'II42', 'V7', 'I53', 'I53', 'V7', 'I64', 'V43', 'I53', 'V43', 'I64', 'vi', 'I53']","['-', 'I', 'iii', 'IV', 'I64', 'I53', 'I', 'vi7', 'vi7', 'IV53', 'IV7', 'I', '-i', 'I', 'I53', 'vi53', 'vi7', 'IV', 'v42', 'ii7', 'v', 'I', 'I', 'vi7', 'vi7', 'IV', 'IV53', 'ii53', 'v7', 'iv', 'i', 'I64', 'vi', 'vi42', 'IV', 'vi', 'iv', 'i', 'I', 'vi64', 'vi7', 'IV', 'vi42', 'I']","['-', 'I53', 'ii53', 'I53', 'iii', 'IV53', 'ii', 'V7', 'I64', 'V65', 'I64', 'I', 'ii', 'i', 'iii', 'IV', 'ii', 'V', 'I53', 'v7', 'I6', 'ii', 'I', 'ii', 'V', 'I53', 'ii43', 'v7', 'I53', 'vii64', 'iv64', '-VII53', 'V7', 'I64', 'I6', 'I64', 'V65', 'I64', 'I', 'ii', 'i', 'iii', 'IV', 'ii', 'V', 'I53', 'v7', 'I6', 'ii', 'I', 'ii', 'V', 'I53', 'ii43', 'v7', 'I53', 'vii64', 'iv64', '-VII53', 'V7', 'I64', 'I6', 'I64', 'V65', 'I64', 'I', 'ii', 'i', 'iii', 'IV', 'ii', 'V', 'I53', 'v7', 'I6', 'ii', 'I', 'ii', 'V', 'I53', 'ii43', 'v7', 'I53', 'vii64', 'iv64', '-VII53', 'V7', 'I64', 'I6', 'I64', 'V65', 'I64', 'I', 'ii', 'i', 'iii', 'IV', 'ii', 'V', 'I53', 'v7', 'I6', 'ii', 'I', 'ii', 'V', 'I53', 'ii43', 'v7', 'I53', 'vii64', 'iv64', '-VII53', 'V7', 'I64', 'I6']","['-', 'I53', 'IV6', 'iv', 'VI6', 'ii', 'I53', 'I53', 'IV6', 'iv', 'VI6', 'ii64', 'I53', '-vii', 'iv64', 'VI53', 'I64', 'vii', 'vi53', 'V53', 'ii6', 'iii53', 'IV53', 'I', 'I53', 'I64', 'I64', 'I64', 'I53', '-vii', '-vii', 'iv53', '-VII53', 'vii7', 'vi65', 'I53', '-vi']","['-', 'I64', 'I53', 'I64', 'I64', 'iv', 'I64', 'iii', 'I64', 'iv', 'I53', 'I42', 'iv43', 'I53', 'I53', 'i', 'V7', 'v65', 'vi', 'ii53', 'I53', 'v42', 'I53', 'IV53', 'V65', 'ii42', 'IV43', 'V7', 'I53', 'IV', 'I53', 'I64', 'I64', 'iv', 'I64', 'iii', 'I64', 'iv', 'I53', 'I42', 'iv43', 'I53', 'I53', 'i', 'V7', 'v65', 'vi', 'ii53', 'I53', 'v42', 'I53', 'IV53', 'V65', 'ii42', 'IV43', 'V7', 'I53', 'IV', 'I53', 'I64', 'I64', 'iv', 'I64', 'iii', 'I64', 'iv', 'I53', 'I42', 'iv43', 'I53', 'I53', 'i', 'V7', 'v65', 'vi', 'ii53', 'I53', 'v42', 'I53', 'IV53', 'V65', 'ii42', 'IV43', 'V7', 'I53', 'IV', 'I53', 'I64', 'I64', 'iv', 'I64', 'iii', 'I64', 'iv', 'I53', 'I42', 'iv43', 'I53', 'I53', 'i', 'V7', 'v65', 'vi', 'ii53', 'I53', 'v42', 'I53', 'IV53', 'V65', 'ii42', 'IV43', 'V7', 'I53', 'IV', 'I53', 'I64', 'I53']","['vi65', 'I53', 'V64', 'I53', 'I53', 'I6', 'V64', 'I53', 'i', 'I', 'v', 'V53', 'i', 'i', 'v', 'v42', 'i', 'I53', 'V64', 'I53', 'I53', 'I6', 'V64', 'I53', 'i', 'I', 'v', 'V53', 'i', 'i', 'v', 'v42', 'i', 'I53', 'V64', 'I53', 'I53', 'I6', 'V64', 'I53', 'i', 'IV', 'I65', 'IV', 'vi64', 'v', 'I53', 'V64', 'I53', 'I53', 'I6', 'V64', 'I53', 'i', 'I', 'v', 'V53', 'i', 'i', 'v', 'v42', 'i', 'I53', 'V64', 'I53', 'I53', 'I6', 'V64', 'I53', 'vi65', 'I53', 'V64', 'I53', 'I53', 'I6', 'V64', 'I53', 'i', 'i', 'i', 'i', 'i', 'iv', 'iv', 'iv', 'iv', 'I64', 'I64']","['-', 'vi65', 'vi65', 'vi65', 'vi65', 'i', 'i', 'I', 'i', 'i', 'i', 'I53', 'I', 'I', 'i', 'I53', 'i', 'I6', 'I7', 'i', 'I6', 'vi65', 'vi65', 'i', 'i', 'I', 'i', 'i', 'i', 'I53', 'I', 'I', 'i', 'I53', 'i', 'I6', 'I7', 'i', 'I6']","['-', 'I', 'I53', 'I', 'V53', 'v', 'I53', 'I', 'V', 'v7', 'i', 'IV', 'V53', 'V53', 'V7', 'I53', 'IV53', 'I53', 'V53', 'I53', 'IV53', 'IV53', 'V7', 'V42', 'I53', 'I53', 'I53', 'vii64', 'V53', 'I53', 'IV53', 'II64', 'V53', 'I64']","['-', '-', 'i', 'i53', 'i53', 'i53', 'ii53', 'ii42', 'V65', 'V7', 'i', '#vii43', 'i6', '#vii43', 'III53', '#vi7', 'It6', 'v7', 'v7', 'v7', 'v7', 'v', 'i', 'i6', 'i53', 'iv43', 'iv7', '-VII65', 'vii43', 'III53', '-VII6', '-VI6', 'III64', '-VI', 'III6', 'IV', '-VII53', 'iii', '-VII53', 'iii', '-VII', 'III64', 'V43', '#i', '#iii6', 'II', '#vii', '#vii53', '#vii53', 'iv64', '#i43', '#iv43', '-vii', '#iv43', 'ii6', '#iii43', '#vii7', '-vi', '#vi6', 'IV', 'iv', '-VII', 'i42', 'III53', 'i', 'iv', '#vii', 'vi', '#vi', 'II53', 'II53', 'II53', 'II53', 'II53', 'vi42', 'i', 'iv53', 'iv53', 'iv', '#vii', '#vii43', 'III', 'III', 'III', 'V7', 'i64', 'i53', 'i']","['i', '-i6', 'i', '-i53', '-iii', 'iii', 'i53', '-i6', 'i', '-i6', 'i', '-i6', '-iii', 'iii', 'i53', '-i6', '-iii', '-I6', '-iii', '-i', '-I6', '-I6', '-I6', '-I6', '-I6', '-I6', '-I6', '-I6', '-I6', 'III53', 'III53', '-I6', '-I', 'III53', 'iii53', '-i', 'iii53', '-i', '-i', '-i', '-i', '-i', 'III53']","['v7', 'III65', 'v7', 'III65', 'v7', 'III65', 'v7', 'III65', 'iii', 'ii', 'i', 'ii42', 'v', 'v7', 'iv', 'vii43', 'iii', 'i65', 'i', 'iv43', 'v53', 'v', 'vii43', 'IV53', 'iii', 'i65', 'i', 'iv43', 'v53', 'v', 'vii43', 'IV53']","['v7', 'III65', 'v7', 'III65', 'v7', 'III65', 'v7', 'III65', 'iii', 'ii', 'i', 'ii42', 'v', 'v7', 'iv', 'vii43', 'iii', 'i65', 'i', 'iv43', 'v53', 'v', 'vii43', 'IV53', 'iii', 'i65', 'i', 'iv43', 'v53', 'v', 'vii43', 'IV53']"]],"container":"<table class=\"display\">\n  <thead>\n    <tr>\n      <th> <\/th>\n      <th>midi_name<\/th>\n      <th>key_signature<\/th>\n      <th>harmonic_reduction<\/th>\n    <\/tr>\n  <\/thead>\n<\/table>","options":{"pageLength":1,"dom":"tip","order":[],"autoWidth":false,"orderClasses":false,"columnDefs":[{"orderable":false,"targets":0}],"lengthMenu":[1,10,25,50,100]}},"evals":[],"jsHooks":[]}</script>

---

# Visualizing model structure

---

# To-date progress

* Gathered data and performed initial analysis

* Explored Google’s Magenta and explored Performance-RNN to generate some music

* Analyzed song structure with time series approaches (ARIMA, VAR, Markov)

---

# Future work

* Build 2-3 variations of GANs and Variational Auto-encoders during the implementation quarter

* Work on specific Genre-based generation and also personalized music generation, sending out surveys and using preferences to adjust the model weights

---

# Technical challenges

* Novelty of the project results in less formulated resources to rely on

* Determining a flexible model architecture to allow user feedback

* Collecting more data for training

---

# Project Timeline

* Capstone Implementation in Autumn 2019

* Capstone Writing in Winter 2020

* Utilize the Summer Quarter(Vacation/Internship) for exploring 2-3 variations of GANs and Variational Auto-Encoders.

* Work on specific Genre based generation and personalized Music Generation in the Autumn Quarter.