Knowledge
  • My Knowledge Wiki
  • Courses
    • Courses
    • Coursera
      • Machine Learning
        • Week 1
          • Introduction
        • Week 2
          • Linear Regression with Multiple Variables
    • fast.ai
      • fast.ai
      • Deep Learning Part 1: Practical Deep Learning for Coders
        • Deep Learning Part 1: 2018 Edition (v2)
          • Lesson 1
        • Deep Learning Part 1: 2019 Edition (v3)
          • Lesson 1 - Image Recognition
          • Lesson 2 - Computer Vision: Deeper Applications
          • Lesson 3 - Multi-label, Segmentation, Image Regression, and More
          • Lesson 4 - NLP, Tabular, and Collaborative Filtering
          • Lesson 5 - Foundations of Neural Networks
          • Lesson 6 - Foundations of Convolutional Neural Networks
          • Lesson 7 - ResNets, U-Nets, GANs and RNNs
      • Deep Learning Part 2: Cutting Edge Deep Learning for Coders
        • Deep Learning Part 2: 2017 Edition (v1)
          • Lesson 8 - Artistic Style
          • Lesson 9 - Generative Models
          • Lesson 10 - Multi-modal & GANs
          • Lesson 11 - Memory Networks
          • Lesson 12 - Attentional Models
          • Lesson 13 - Neural Translation
          • Lesson 14 - Time Series & Segmentation
        • Deep Learning Part 2: 2018 Edition (v2)
          • Lesson 8 - Object Detection
          • Lesson 9 - Single Shot Multibox Detector (SSD)
          • Lesson 10 - Transfer Learning for NLP and NLP Classification
          • Lesson 11 - Neural Translation; Multi-modal Learning
          • Lesson 12 - DarkNet; Generative Adversarial Networks (GANs)
          • Lesson 13 - Image Enhancement; Style Transfer; Data Ethics
          • Lesson 14 - Super Resolution; Image Segmentation with U-Net
      • Machine Learning: Intro to Machine Learning for Coders
        • Machine Learning: 2017 Edition
          • Lesson 1 - Introduction to Random Forests
          • Lesson 2 - Random Forest Deep Dive
          • Lesson 3 - Feature Engineering
          • Lesson 4 - Random Forest Interpretation
          • Lesson 5 - Train vs Test
          • Lesson 6 - What is Machine Learning and Why Do We Use It
          • Lesson 7 - Decision Trees Ensemble
          • Lesson 8 - Basic Neural Networks
          • Lesson 9 - SGD; Neural Network Training; Broadcasting
          • Lesson 10 - Logistic Regression; NLP; Naive Bayes
          • Lesson 11 - Structured and Time-Series Data
          • Lesson 12 - Entity Embeddings; Data Science and Ethics
  • Books
    • Deep Work
  • Programming Languages
    • Programming Languages
      • JavaScript
        • JS Libraries
Powered by GitBook
On this page
  • Coursework
  • Jupyter Notebook Used
  • Reading: Paper [TODO]
  • My Notes
  1. Courses
  2. fast.ai
  3. Deep Learning Part 2: Cutting Edge Deep Learning for Coders
  4. Deep Learning Part 2: 2017 Edition (v1)

Lesson 12 - Attentional Models

PreviousLesson 11 - Memory NetworksNextLesson 13 - Neural Translation

Last updated 6 years ago

Topics:

  • Clustering in TensorFlow

  • Attentional models

Lesson

Coursework

Jupyter Notebook Used

  • [time: 00:00:00]

  • [time: 00:40:00]

Reading: Paper [TODO]

  • [time: 01:37:28] by Dzmitry Bahdanau, et. al

  • [time: 01:44:08] by Oriol Vinyals et. al

    • This is a very recent research paper on the subject (released last week 21 Mar 2017)

    • Open source code:

My Notes

  • [time: 00:00:00] Clustering again. Kmeans.

  • [time: 00:40:00] Intro to next step: NLP and translation deep-dive, with CMU pronouncing dictionary via spelling_bee_RNN.ipynb

  • [time: 01:32:00] Attention models. I actually really like these, I think they're great. And really the paper that introduced these, quite an extraordinary paper, introduced both GRUs and attention models at the same time. I think it might even be before the guy had his PhD, if I remember correctly. It was just a wonderful paper, very successful.

  • [time: 01:28:38] Even though I try to teach things which I think are going to stand the test of time, I'm not at all convinced that any technique for reinforcement learning is going to stand the test of time. So I don't think we're going to touch that.

  • [time: 01:44:08] This whole thing is summarized in another paper, actually a very cool paper, Grammar as a Foreign Language. Lots of names you probably recognize here, Geoffrey Hinton, who's kind of the father of deep learning; Illya Stuskever, who's Director of Science at OpenAI; Oriel Vinyals, who's done lots of cool stuff. This paper is kind of neat and fun anyway. It basically says, What if you didn't know anything about grammar and you attempted to build a neural net which assigned grammar to sentences. It turns out you actually end up with something more accurate than any rule-based grammar system that's been built.

  • [time: 01:49:50] AMA - Question: Any advice on imbalanced datasets? Seems to be a recurring issue with real world data.

Website
Video
Wiki
Forum discussion
kmeans_test.ipynb
spelling_bee_RNN.ipynb
Neural Machine Translation by Jointly Learning to Align and Translate
Grammar as a Foreign Language
Massive Exploration of Neural Machine Translation Architectures
https://github.com/google/seq2seq/