Learning Syntax with Deep Neural Networks @ Cambridge University Linguistic Society, Cambridge [25 May]

Learning Syntax with Deep Neural Networks


46
25
May
16:30 - 18:30

 Facebook event page
Cambridge University Linguistic Society
9 West Road, CB3 9DP Cambridge, Cambridgeshire
Prof Shalom Lappin (University of Gothenburg, King's College London and Queen Mary University of London)

Joint work with Jean-Philippe Bernardy, University of Gothenburg

We consider the extent to which different deep neural network (DNN) configurations can learn syntactic relations, by taking up Linzen et al.’s (2016) work on subject-verb agreement with LSTM RN Ns. We test their methods on a much larger corpus than they used (a 24 million example part of the WaCky corpus, instead of their ~1.35 million example corpus, both drawn from Wikipedia). We experiment with several different DNN architectures (LSTM RNNs, GRUs, and CNNs), and alternative parameter settings for these systems (vocabulary size, training to test ratio, number of layers, memory size, and drop out rate). We also try out our own unsupervised DNN language model. Our results are broadly compatible with those that Linzen et al. report. However, we discovered some interesting, and in some cases, surprising features of DNNs and language models in their performance of the agreement learning task. In particular, we found that DNNs require large vocabularies to form substantive lexical embeddings in order to learn structural patterns. This finding has significant consequences for our understanding of the way in which DNNs represent syntactic information. We also achieved significantly better accuracy with our language model for unsupervised prediction of agreement than Linzen et al. report in their LM experiments.
Discussion
image
Only registered users may comment.
Pass a quick registration or authorization.

The most anticipated events in Cambridge :

Cambridge May Ball Marketplace 2017
Cambridge, United Kingdom
Strawberries & Creem Festival 2017: Shaggy, Wiley & 40+ artists
Cambridge, United Kingdom
Strawberry Fair Official Event Page
Midsummer Common
Senator Bernie Sanders | Cambridge Union
The Cambridge Union
Strawberry fair 2017
Midsummer Common
Murray Edwards Garden Party 2017
Murray Edwards Garden Party
Cambridge Union Society Garden Party 2017
Sidney Sussex College
Warning 22nd Birthday Part II – Outlook Festival Launch Party
Cambridge Junction
Jesus May Ball 2017 - 'Between the Lines'
Jesus College, Cambridge
Matty G & Li'l Winter Live At Strawberry Fair
Strawberry Fair, Cambs