Deep Transformer Expressivity

SIGNAL ANALYSIS

Deep transformers form hierarchical representations, but their expressivity is not well understood. This analysis uses bounded-depth grammars to study how they capture abstract features. The findings could impact language modeling and beyond.