Harry Potter and the Deep Learning Experiment

For each character, the model looks up the embedding, runs the GRU one timestep with the embedding as input, and applies the dense layer to generate logits predicting the log-likelihood of the next character. ¹

Three years ago (back when teaching RNNs to generate text was cool) I had this idea of training a TensorFlow model with Harry Potter’s books. This project was on my list and it was hunting me.

This weekend I finally did it and also made a simple Streamlit app for you to play around with the model: https://harry-dl.uc.r.appspot.com/

Streamlit app. Code: https://github.com/santiviquez/harry-potter-rnn

Some of the generated text

Harry sat down next to the went to see when Harry reading the stairs.

“So sorry, your bloodiness, Mr. Baron, Snape’s classe on your cheek, Potter, or I wouldn’t both their far worse than even Dudley. Dudley had been a couple of the library; he swallowed a lot of new questions that had just occaurt their ends.

So put me on! Don’t be afraid! And don’t get in our first week. See you.”

Fred and George Weasley were responsible for trying to stop Gryffindor for the first time something that had been worrying him a lot like Uncle Vernon. He had a lot of trouble keeping his letter. He handed silently. He mustn’t want to tell Harry?”

Harry heard the little bronze coins, and the owl swooped in and dropped the newspaper on top of Hagrid, who dived on top of it and managed to get through his exams when he half expected Voldemort to come back.

“What is a wizard’s duel?” said Harry and Ron.

As you can see most of the sentences don’t make sense, but all of them were generated by a computer and that’s the impressive part.

Isn’t this just re-arranging the words?

Nope. Let’s see an example of the learning process to understand why.

These are some of the model’s generated texts.

Epoch 1:at the first epoch (full training pass over the entire dataset) we see what it appears to be just random letters and characters.

sngns yiiy I rad yit se ke sery Ro, "udGct touveid. samtd ccift, the. fpe gincy lindos ys Towe feeng beunat 
)X h so-int rines Hufons nsmtheis anmin f oideyhend wrratt hit le therf arerre he fo ke thomecAg voutud unfed fow suk rond thet 
macughilg wares thirstof hy youMl hed folh,

Epoch 2:almost the same as before.

"Mald sined ro fastrong orryfe frourd hot'dilg trimt wa thin ware ne bamly oue ntho feringseps thag catkirgen tu hime baicf sait yoveece way yonerbyedene nomt thaney an'thing ve fuba carpouflit  Serofy," saided one whizciof Harr wiole, I it the pie pevainigrofle pors

Epoch 5:here we see some progress. It is learning about how to create paragraphs and it learned how to spell Ron correctly and it almost did it Hermione. Another interesting fact is that it learned that each paragraph needs to end with a period.

daven lack a grivert He mad eets plaiced. The inares, ruther Inard in brigt tuffing a to kinares." 

"Haw do get brousl purny to seize whis with hem looking by. We's if ge. 

"Haryy, you, wande -- recatbladred got Herwione, and Harry ando, him some wich nover Durn Ron. Ho said abeary own exard to ke to nextsen, mowned megrmings.

Epoch 10:It learned to spell Snape, Harry, Dudley and Hermione!

Harry fast roy winged and same forward that Snape heard fat looked at all. Small not about is but he didn't makber, wh they furged him. 

"It's going to lim, I'll never thought that Curse for dong tone ach ins other, could poney. Harry flocked Dudley this if he soll that Hermione had be before wouldn't meville. 'veryto his rrom -- and he stop its head. I've reed Lonnem. Snape was gonew like Harry's gettid himmered in elor without hair.

Epoch 50:notice that in the second paragraph it learned how to use quotation marks correctly (orthographically).

Neville had been hand stretching to watch as Harry sped straight at Snape -- she didn't even noticed in tim again. After burning all the letters, he was made difficult by Fathers s had been waiting noise and the sheets a hote there, with right dewild!""But what's only shop."Harry nodded, but stopped quickly, because it made different time was protected wherever he went.

So it basically tries to replicate J.K. Rowling’s writing style and tone.

How this works

One of the best resources to understand how this works is Andrej Karpathy’s blog post from 2015. I’ll take the freedom to use one of his examples.

Let’s imagine that our whole vocabulary consists of the four letters: h , e , l , o . And we divide our text in chunks of length=4 and we want to train our RNN on the sequence “hello”.

In this case, our input sequence would be “hell” and the target sequence “ello”. So given the context that initially there’s only a lonely “h” the model will guess that the next word is probably an “e” and then given the context that the first two letters are“h” and “e” the model will guess an “l” for the third letter and so on.

The relation among all the previous letters helps predicting the next one. The RNN remembers all these relationships for each of the sequences.

https://karpathy.github.io/2015/05/21/rnn-effectiveness/

To learn more about RNNs I truly recommend Karpathy’s blog post and if you want to build your own this TensorFlow 2 Tutorial is a great resource.

[1]: https://www.tensorflow.org/tutorials/text/text_generation