Yekun's Note

Flow models are used to learn continuous data.

The brain has about 10¹⁴ synapses and we only live for about 10⁹ seconds. So we have a lot more parameters than data. This motivates the idea that we must do a lot of unsupervised learning since the perceptual input (including proprioception) is the only place we can get 10⁵ dimensions of constraint per second.
(Geoffrey Hinton)

Techniques of NN training. Keep updating.

This is an introduction of recent BERT families.

Softmax encounters large computing cost when the output vocabulary size is very large. Some feasible approaches will be explained under the circumstance of skip-gram pretraining task.

Reasoning the relations between objects and their properties is a hallmark of intelligence. Here are some notes about the relational reasoning neural networks.

This is an introduction of variant Transformers.^[1]

FLOPs is a measure of model complexity in deep learning.

A guide to calculate the number of trainable parameters by hand.

Dynamic Programming (DP) is ubiquitous in NLP, such as Minimum Edit Distance, Viterbi Decoding, forward/backward algorithm, CKY algorithm, etc.