What is Attention?

Attention is focus.

It answers a simple question:

Where should I focus?

Different types of attention provide different signals for deciding where to focus:

• Positional attention: Where did it appear?
• Temporal attention: When did it happen?
• Relation attention: What is it related to?
• Semantic attention: What does it mean?
• Discourse attention: What does it refer to in the current conversation?
• Novelty attention: Is it new or already known?

Multi-head attention combines the different attention signals that matter for the current task.

The goal is not to process everything.

The goal is:

Focus on what matters, ignore what does not, and reason over the selected context.