Name: Seminar at NAVER LABS Europe: Analyzing Information Flow in Transformers
Start: 2020-01-30T11:00:00+01:00
End: 2020-01-30T12:00:00+01:00
Location: NAVER LABS Europe

Seminar at NAVER LABS Europe: Analyzing Information Flow in Transformers

NAVER LABS Europe (Grenoble, France) is the biggest industrial AI research lab in France with fundamental and applied research in the fields of machine learning and optimization, computer vision, natural language processing and human robot interaction. The main area of application  is  AI for Robotics where they work closely with  NAVER LABS Korea the R&D subsidiary of NAVER, Korea’s leading internet company. NAVER LABS is responsible for creating future technology in robotics, AI, autonomous driving, 3D/HD mapping and AR.LinkedIn-Bluesky

NAVER LABS Europe

Speaker: Lena Voita, Yandex Research, University of Amsterdam

Abstract : We will discuss what, how and why Transformers learn by analyzing1. the mechanisms the model uses to encode different kinds of information;2. how training objective defines information flow in a model.First, we will start with an in-depth analysis of multi-head attention. Using attribution methods, we will assess the importance of individual heads and will show that the most important heads play interpretable roles. Surprisingly, all the rest of the heads are redundant and, using our novel heads-pruning method, can be pruned with almost no loss in translation quality.Then, we will look at how the representations of individual tokens in the Transformer evolve between layers under different learning objectives: MT, LM and MLM (BERT-style). While previous work mostly used so-called 'probing tasks' and has made some interesting observations, an explanation of the process behind the observed behavior has been lacking. I will attempt to explain more generally why such behavior is observed by characterizing how the learning objective determines the information flow in the model. I look at this task from the information bottleneck perspective on learning in neural networks and will show that patterns in information flow are substantially different. For example, while LMs gradually forget past when forming predictions about future, for MLMs the evolution proceeds in two stages of context encoding and token reconstruction.

Our {communityGuidelinesLink} describe the sort of content we prohibit on Eventbrite. If you suspect an event may be in violation, you can report it to us so we can investigate.

To report other categories of prohibited or illegal content, submit {link}

Seminar at NAVER LABS Europe: Analyzing Information Flow in Transformers

More events from NAVER LABS Europe

Follow organisers to get events picked for you

Seminar at NAVER LABS Europe: Analyzing Information Flow in Transformers

More events from NAVER LABS Europe

Follow organisers to get events picked for you

You might also like...

Browse more events with different dates, prices, and formats to find your next great experience.