Labelling Unsegmented Sequence Data with Recurrent Neural Networks (Graves et al.)

The Problem

When training models for sequence-to-sequence tasks like speech recognition or handwriting recognition, we often face a challenge: the input and output sequences have different lengths, and we don't know the alignment between them.

For example, in speech recognition:

• Input: Audio frames (e.g., 1000 frames)
• Output: Text ("hello" - 5 characters)

Traditional approaches require frame-level alignments, which are expensive to obtain.

CTC Solution

CTC introduces a blank token (often denoted \(\epsilon\)) and defines a many-to-one mapping from frame-level predictions to output sequences. Multiple frame-level paths can map to the same output.

For example, these all map to "cat":

• \(\epsilon c a a t \epsilon\)
• \(c c \epsilon a t t\)
• \(\epsilon \epsilon c a t\)

The CTC Loss

The CTC loss is the negative log probability of the correct output sequence, marginalized over all possible alignments:

\[ L_{CTC} = -\log \sum_{\pi \in \mathcal{B}^{-1}(y)} P(\pi | x) \]

where \(\mathcal{B}^{-1}(y)\) is the set of all paths that map to output \(y\).

Key Properties

• Alignment-free: No need for pre-segmented training data
• Differentiable: Uses dynamic programming (forward-backward algorithm) for efficient computation
• Monotonic: Assumes input-output alignment is monotonic (left-to-right)

Resources

• Distill: Sequence Modeling with CTC
• Intuitively Understanding CTC

Gut Instinct: Creating Scientific Theories with Online Learning

Our intuition is that scientific crowdsourcing will most usefully contribute to domains where science is nascent and/or highly contextual. The human microbiome project is both. This paper explores the potential of coupling online citizen science with learning materials to create scientific questions. Example: Foldit players discovered protein structures that helped scientists understand how the AIDS virus reproduces. The main contribution of this paper is demonstrating that a crowd of online non-expert learners can collaboratively perform useful scientifc work. Gut Instinct, which brings together learnes to perform useful collaborative brainstorming on a citizen science project while developing expertise. Collectively aggregating many people's responses can produce faster, better, and more reliable results - at much larger scale - than lone individuals can, at least errors andd biases are independent events. Our novel contribution is an explicit integration of learning. Hypotheses: Learning improves quality of work on relevant problems. Working on relevant real-world problems improves learning. Working while learning improves learners’ en- gagement with the learning material.

Reading Group: Asynchronous Methods for Deep Reinforcement Learning (Mnih et al.)

Motivation

The sequence of observed data encountered by an online RL agent is non-stationary, and online RL updates are strongly correlated. By storing the agent's data in an experience replay memory, the data can be batched or randomly sampled from different time steps.

Drawbacks of Experience Replay

• Uses more memory and computation per interaction
• Requires off-policy learning algorithms

Asynchronous RL Framework

The paper presents multi-threaded asynchronous variants of:

• One-step SARSA
• One-step Q-learning
• N-step Q-learning
• Advantage actor-critic (A3C)

Key insight: Actor-critic is an on-policy search method while Q-learning is an off-policy value-based method. Running multiple agents in parallel on different threads provides diverse, decorrelated experience without replay memory.

Key Benefits

• Decorrelated updates: Different threads explore different parts of the environment
• No replay memory needed: Enables on-policy methods like actor-critic
• CPU-friendly: Runs on multi-core CPUs rather than requiring GPUs

Related Work

• Gorila Framework: Distributed RL with parameter servers
• Hogwild!: Lock-free parallel SGD