MechanismosfI mplicLieta rnign NeuraNle tworMko delinagn dC onnectionism JeffrLe.Ey l man, Editor ConnectiMoondieslti anngdB raiFnu nctiTohneD: e velopIinntge rface StepheJno sHea nsona ndC arRl .O lsone,d itors NeuraNle twoDreks igann dt heC omplexoifLt eya rniJn.Sg t epheJnu dd NeuraNle tworfkosCr o ntrWo.l T homasM illeRirc,h ardS uttoann,d PaulJ .W erbose,d itors TheP erceptofiM ounl tipOlbej ecAt Cso:n nectionist Approach MichaMeolz er NeuraClo mputatoifPo ant teMront ioMno:d eliSntga goesf M otoin Analysiints h eP rimaVties uCaolr tMeaxr gareEtu phrasia Sereno SubsymboNlaitcu aLla nguaPgreo cessAinn Ign:t egrMaotdeedol f Scripts, LaenxdMi ecmoonr,Riy s toM iikkulainen Analogy-MakaisnP ge rceptiCoonm:p uAt Meord eMle laniMei tchell MechanisomfIs m pliLceiatr niCnogn:n ectiMoondieslotsf S equence ProcessAixneglC leeremans MechanismosfI mplicLieta rnign ConnectioMnoidsetl osf S equenPcreo cessing AxelC leeremans A BradfoBrodo k TheM IT Press CambridgMea,s sachusetts LondonE,n gland © 199Ma3s sachusIentsttistu toefT echnology All righrtess erveNdo. p arotf t hibso okm ayb er eproduinc eandy formb ya nye lectronic orm echanimceaaln s( includpihnogt ocopyriencgo,r dinogr,i nformatsitoonr aagned retrievwailt)h opuetr missini ownr itingfr om thep ublisher. This bookw ass eitn P alatibnoyTh e MITP resasn dw asp rinteandd bound in the United StatoefsA merica. LibraorfyC ongreCsast aloging-in-PubliDcaattai on CleeremaAnxse,l . Mechanismosfi mpliclietarnin g :c onnectiomnoidsetl osfs equenpcreo cess/i ng AxelC leeremans. p. em.- (NeurNaeltw orkm odelianngd c onnectionism) "AB radfobrodo k," Includbeisb liograrpehfiecraeln acnedis n dex. ISBN0 -262-03205-B 1N.e uranle twor(kCso mputsecri enc2eI.)m pliclietarnin g. 3C.o nnectmiaocnh ines. IT.i tlIIe..S eries. QA7.6B.7C541 993 006.3'3-dc20 92-35739 CIP A I'enfant invisible Contents SeriFeosr eworidx Preface xi Acknowledgmenxtisii Chapte1r ImpliLceiatrn inEgx:p loratiinoB nass iCco gnitio1n Introduct1i on EmpiriSctauled sio fI mpliLceiatr nin5g Modelasn dM echanis1ms9 Chapte2r The SRN ModelC:o mputatioAnsaple cotfsS equence Processi3n5g LearnianF gi nite-GSrtaamtmea r3 9 DiscovearnidnU gs inPga tIhn formati4o9n Learnin5g8 Encoding NoCnolnotceax6lt3 Discussi7o0n Chapte3r SequenLceea rniansga ParadifgomrS tudyiInmgp licit 75 Learning LearnitnhgSe t ructoufEr veen tS equenc7e6s Experime1n t8 1 Experime2n t9 1 Simulatoifto hneE xperimenDtaatla 9 5 GenerDails cussi1o0n3 Conclusi1o1n2 Chapte4r SequenLceea rniFnugr:t hEexrp lorati1o1n3s SequenLceea rnianngdM emoryD isorde1r1s4 AttentainodnS equenScter uctu1r1e6 ElementSaeryq uenLceea rnianngdt hEeff ecotfsE xpliKcniotw ledg1e2 3 GenerDails cussi1o3n4 viii Contents 5 Chapter 137 EncodiRnegm otCeo ntext Long-DistCaonnctei ngeanncdiP erse dictionL-eBaarnsiendg1 37 CompariwsiotnOh t heArr chitecftoruS reqeuse ncPer ocess1in3g9 An EmpiriTceaslt 1 44 Experim3e n1t5 7 Discussi1o64n 6 Chapter 169 ExpliSceiqtu enLceea rning An ExpliPcrietd icTtaisokn1 69 Tranesrtf oN ew Materi1al8 3 7 Chapter 189 GenerDails cussion Prediction-RaenldCe ovnatnecRxeept r esentat1i9o1n AttentAiwoanr,e neasnsdt, he R oloef E xpliKcniotw ledg1e9 4 ConsciaonudsU nconsciKonuosw ledg1e97 Tranesraf ndA bstract2i0o0n On Modelin2g0 5 Conclusi2o0n6 209 Notes 231 References 221 Index SeriFeosr eword Theg oaolf t hisse riNeesu,r aNle tworMko delianngd C onnectionism, ist oi dentainfdyb rintgot hep ubltihceb eswto rki nt hee xcitfiinegl d ofn euranle tworakn dc onnectiomnoidsetl inTgh.e s eriiensc ludes monograpbhass eodn d issertaetxitoennsd,re edp orotfws o rkb yl ead­ ersin thef ieledd,i tevdo lumeasn dc ollectoinot nosp iocfss pecial intermeasjto,rr e ferewnocrek s, uanndde rgradaunadtg er aduate-level textTsh.ef ieilsdh ighilnyt erdiscipalnidwn oarrkyps,u blishine tdh e seriweisl tlo ucohn a widev arieotfyt o pircasn gifnrgo ml ow-level visitoont hep hilosophfiocuanld atiooftn hse oroifer se presentation. JeffrLe.Ey l manE,d itor AssociEadtiet ors: JameAsn dersoBnr,o wnU niversity Andrew BartUon,i versoifMt ays sachusAemhtetrss,t GaryD ellU,n iversoiflt lyl inois JeromFee ldmaUnn,i versoifCt ayl ifornBiear,k eley StephGerno ssbeBrogs,t oUnn iversity StephHeann sonP,r inceUtniovne rsity GeoffrHeiyn toUnn,i versoifTty o ronto MichaJeolrdaMnf,T JameMsc ClellaCnadr,n egMieel loUnn iversity DomenicPoa riIsnis,t itduiPt soi colodgeilCa N R DaviRdu melhaSrtta,n foUrndi versity TerrenSceej nowsThkei ,S alIkn stitute PauSlm olensUknyi,v ersoifCty o lorado StephPe.nS tihc,R utgeUrnsi versity DaviTdo uretzCkayr,n egMieel loUnn iversity DaviZdi pseUrn,i versoifCty a liforSnainDa i,e go Preface Il ikteo t hinokft hibso oka sa booka bouetl ementlaerayrnin g pro­ cesseesv,e nt hougshu cahc laitmog eneraliisut nyq uestionoavbe­lry stateIdn.d eetdh,i bso oki sr ealalbyo uatp henomencoanl liemdp licit learning-p"rtohcee bsysw hickhn owledagbeo utth er ule-governed complexiotfti heess timuleunsv ironmaerneat c quiriendd ependently 1989). ofc onsciaotutse mpttods o s o"( Reber, Butt hinkionfig m plicit learniansag n e lementaabriyl tiote yx traacntdt op rocesstsr ucture­ rathetrh ana ss omem ysterifoaucsu lftoyrl earninwgi thouetv en knowinigt -hatsh em eriotf p uttitnhgee mphasoinst hen aturoef processriantgh tehra onn t hen atureo fkn owledgeT.h ipse rspective formtsh eb asimco tivatfioortn h ibso okt:o e xplowrhea tk indosf mechanismmasyb es uffictioae cncto ufnotir m plilceiatr ndiantgaa ,n d toi nstanttihaetsmeee chanisimnts h ef ormo fc omputatimoondaell s ofp erformaTnhcieas.p proatcoht hefi elids d ifferfernotmp revious researicnsh e verwaaly sF.i rsetv,e tnh ougihm plilceiatr nirnegs earch isr apidglayi niinngc rearseecdo gnittihoefn i,e lidss timlalr rebdy numeroudse bataebso utth en aturoefk nowledagceq uiriemdp licitly. Eversyo o ftenne,w s tudieasp peatrh afto rcues t or econsitdheer validoiftp yr evious methodtooel loigceiixetps l icit knowledge (e.g., PerruchGeatl,l eg&o S,a vy1,9 9P1er;r uc&h eAtm orim1,9 9o2rt) h,a t demonstrtahtaept r eviouessltya blirsehseudld tons o th olvde rwye ll whenl earninogrt esticnogn ditiaorneas l tereevde ns ligh(tDluyl any, Carlso&n D,e wey1,9 8P4er;r uc&h ePta ctea1u9,9 P0e)rh.a tphsem ost enduridnegb athea st od ow itrhe solvtihnefg o llowqiunegs tiHoonw: muchc onsciaocucse dsosw er ealhlayv teo k nowledtghea tth ee xperi­ mentecrl aimwse acquiriemdp liciTtold ya?t et,h iqsu estihoansn ot beena nswerseadt isfactIob reilliyet.vh ee riesa v ergyo odr easofno r thiIsti: st hew ronqgu estitooan s kS.t uditehsar te vemaelt hodological shortcomianrgeis m portabnuttt, h efya itloh elfpo rmultatheer ight kinodf r esearscthr ateIgnsy h.o rItb ,e lietvheat th iksin do fa pproach isn otv eryp roductiIvtme a.y j usbte i mpossitbold ei sentantghlee contribuotfie oxnpsl iacnidit m plilceiatrn itnogp erformainnac g ei ven tasks,i mplbye causwee cannottu rn offc onsciousinnen sosr mal xii Preface subjec(tNse.ws tudiweist ha mnesiacnsd o tehrb rain-damapgae­d tiensthso wm uchp romiisneh elpitnogr esolsvoem eo ft heesi ssues, however.) Ana pproatchha hta sb eesno mewhamto re succeisnts hfirusel s pect hasb eetno t rtyo i dentify experimentuanld ecwroh nidciihmt piloincsi t learninpgr ocesasreems o stl iketlooy p erate-ifnosrt anbcyee ,x plor­ ingp aradigimnws h icohn em ayo btadiins sociabteitwoenesnp erfor­ mancea nda bilittovy e rbaluinzdee sro mec onditibountns o to thers (e.gB.e,r r&y B roadbe1n9t8,1 49,8T8h)ih.sa sl edt os oliadd vanciens ourkn owledgoef w hicfha ctoarfsf epcetr formaanncdew hicahf fect abilittovy e rbalbiuzteo ;fc oursieti, ss tiplols sibtlooe b jetchta tth e dissociarteisounla trsea reflecmtoiroeno ft hep articucloanrd itions usedt hano fan actucaolg nitdiivset incbteitwoene ntw o modeso f learnianngdp rocessIni ntgh.em eantimwee, h avem adev eyr little progreosns t hen aturoef i mplicpirto cessiitnsge Wlhfa.t a ret he mechanisimnsv olvCeadn?w ep ropoasd ee taicloemdp utatimoondaell ofa tl eassotm ea speocfti mplilceiatr nipnegr formatnhcaeit sa blteo learanss ubjedcota sn dt hauts emse chanissmoes l ementtahraytt h e complemxa chineorfcy o nsciousdnoeesnsso ta ppeatrob en ecessary? In thibso okI,p reseanntd e xplosruec ha modeli,n t hec onteoxft sequence-procteassskiTsnh.ge m odeli sb y no meansa complete architecftourir mep lilceiatr niinngd;e eidtl, a ckmsa nyf eatutrheast wouldm akei tg eneraanld i se vend emonstrawbrloyn gi ns ome instanBcuetsi .ti sa fi rsstt eipn t hed irectoifio dne ntifpylianugs ible mechanisfmosri mplilceiatrn inpge rforma.n tcIhe intkh aotn cew e havea repertooifsr uec hm echaniswmes w,i lble i na muchb etter posititoosn t aarttt acktihnerg e alhlayr dq uestiosnusc,ha sw hatt he relationissbh eitpw eeinm pliacnidet x plipcriotc esshionwgc ,o nscious we areo fk nowledagceq uiruendd eri mpliccointd itioornw sh,a t exacttlhyen aturoefk nowledagceq uiriemdp licmiatylb ye .