{"id":113,"date":"2022-07-22T10:07:29","date_gmt":"2022-07-22T01:07:29","guid":{"rendered":"https:\/\/jaeinlab.com\/?p=113"},"modified":"2023-11-02T19:59:13","modified_gmt":"2023-11-02T10:59:13","slug":"cross-lingual-language-transfer-learning-overview","status":"publish","type":"post","link":"https:\/\/jaeinlab.com\/?p=113","title":{"rendered":"Cross-lingual Language Transfer Learning – Overview"},"content":{"rendered":"\n

\uc774 \ud3ec\uc2a4\ud2b8\ub294 \ucd5c\uadfc Eleuther.AI \uc758 Language Transferring \ud504\ub85c\uc81d\ud2b8\uc5d0 \ucc38\uc5ec\ud558\uba74\uc11c \uacf5\ubd80\ud55c \ub0b4\uc6a9\uc744 \uc815\ub9ac\ud558\uace0\uc790 \uc791\uc131\ud558\uac8c \ub418\uc5c8\ub2e4.<\/p>\n\n\n\n

BERT<\/a>, GPT-3<\/a>, GPT-Neo<\/a>\uc640 \uac19\uc740 \ud604\uc7ac\uae4c\uc9c0 \uacf5\uac1c\ub41c \ud070 \uc2a4\ucf00\uc77c\uc758 PLM (Pre-trained Languae Model) \uc740 \uc601\uc5b4 \ub370\uc774\ud130\ub97c \uae30\ubc18\uc73c\ub85c \ud55c \ubaa8\ub378\uc774 \uac00\uc7a5 \ub9ce\ub2e4. PLM \uc5f0\uad6c\uac00 \uc131\uc219\ub428\uc5d0 \ub530\ub77c \ud55c\uad6d\uc5b4, \uc911\uad6d\uc5b4, \ub3c5\uc77c\uc5b4\uc640 \uac19\uc740 \ube44\uc601\uc5b4 \ubaa8\ub378\ub4e4\uc774 \ub9ce\uc774 \uacf5\uac1c\ub418\uc5c8\ub2e4. \uadf8\ub7fc\uc5d0\ub3c4 \ubd88\uad6c\ud558\uace0 \uc601\uc5b4 \uae30\ubc18 \ubaa8\ub378\uc758 \uc218\uac00 \uc5ec\uc804\ud788 \uac00\uc7a5 \ub9ce\uc740\ub370, \uadf8 \uc774\uc720\ub294 \ub370\uc774\ud130\uc758 \uc591\uc5d0\uc11c \ucc3e\uc744 \uc218 \uc788\ub2e4. \uc778\ud130\ub137\uc5d0\uc11c \uc138\uac8c \uacf5\uc6a9\uc5b4\uc778 \uc601\uc5b4\uac00 \uc555\ub3c4\uc801\uc73c\ub85c \ub9ce\uc774 \uc0ac\uc6a9\ub418\uace0 \uc788\uc73c\uba70 \uadf8\uc5d0 \ub530\ub77c PLM\uc744 \ud6c8\ub828\uc2dc\ud0a4\ub294\ub370 \uc601\uc5b4 \ub370\uc774\ud130\ub97c \uc0c1\ub300\uc801\uc73c\ub85c \ub354 \uc27d\uac8c, \uadf8\ub9ac\uace0 \ub354 \ub9ce\uc774 \ud655\ubcf4\ud560 \uc218 \uc788\ub294 \uac83\uc774\ub2e4. <\/p>\n\n\n\n

\ub2e4\uad6d\uc5b4\ub97c \ud0c0\uac9f\ud305\ud558\ub294 \uc5b8\uc5b4 \ubaa8\ub378\uc758 \uc5f0\uad6c \ub610\ud55c \ucd5c\uadfc \uba87\ub144\ub3d9\uc548 \uafb8\uc900\ud788 \ud65c\ubc1c\ud558\uac8c \uc9c4\ud589\ub418\uace0 \uc788\ub2e4. Multilingual BERT (mBERT)<\/a> \uc758 \uacf5\uac1c\uac00 \uadf8\uc758 \uc2dc\uc791\uc774\ub2e4. 104\uac1c \uc5b8\uc5b4\uc758 \ub370\uc774\ud130\ub85c Pre-training \ud55c \ub2e4\uad6d\uc5b4 \ubc84\uc804 BERT \ubaa8\ub378\uc774\ub2e4. \uc989, \ucc98\uc74c\ubd80\ud130 \ud558\ub098\uc758 \uc5b8\uc5b4\ub97c \uc704\ud55c monolingual representation\uc774 \uc544\ub2cc \ub2e4\uad6d\uc5b4\uc758 \uc815\ubcf4\ub97c \ub2f4\uc740 cross-lingual representation \uc744 \ud559\uc2b5\ud558\uc600\ub2e4. mBERT \uc678\uc5d0\ub3c4 XLM<\/a>, XLM-R<\/a> \uc640 \uac19\uc740 multilingual PLM \ubaa8\ub378\ub4e4\uc774 \uc788\ub2e4. \uc774\ub7ec\ud55c \ucd08\uae30 multilingual PLM \ub4e4\uc740 “Curse of Multilinguality”\ub77c \ubd88\ub9ac\uc6b0\ub294 \ud070 \ud55c\uacc4\uc810\uc774 \uc788\ub294\ub370, \uadf8\uac83\uc740 \uace0\uc815\ub41c \ubaa8\ub378 \uc0ac\uc774\uc988\uc5d0\uc11c pre-training \uc5b8\uc5b4\b\uac00 \uc77c\uc815 \uac2f\uc218\ub97c \ucd08\uacfc\ud560 \uc2dc \ubaa8\ub378\uc758 \uc131\ub2a5\uc774 \ud558\ud5a5\ub418\uae30 \uc2dc\uc791<\/a>\ud55c\ub2e4\ub294 \uac83\uc774\ub2e4. \ubaa8\ub378\uc758 \ud06c\uae30\ub97c \ub298\ub824 robustness\ub97c \uc62c\ub9ac\ub294 \uc77c\uc2dc\uc801 \ud574\uacb0 \ubc29\ubc95\uc774 \uc788\uae30\ub294 \ud558\uc9c0\ub9cc \ubb34\ud55c\uc73c\ub85c \uacc4\uc0b0 \uc790\uc6d0\uc744 \ub298\ub9b4 \uc218\ub294 \uc5c6\uc73c\ub2c8 \ubcf8\uc9c8\uc801\uc778 \uc194\ub8e8\uc158\uc740 \uc544\ub2c8\ub2e4. \ub370\uc774\ud130\uc758 \ubd88\uade0\ud615 \ubb38\uc81c\ub3c4 \uc788\ub2e4. \uac00\ub839, \uc601\uc5b4 \ub370\uc774\ud130\ub97c 100\uac1c \uc218\uc9d1\ud560 \uc218 \uc788\uc744 \ub54c \uc5b4\ub5a4 \uc5b8\uc5b4\ub294 50\uac1c, \uc5b4\ub5a4 \uc5b8\uc5b4\ub294 10\uac1c\ub3c4 \ubaa8\uc73c\uae30\uac00 \ud798\ub4e4 \uc218 \uc788\ub2e4. \ubaa8\ub378 \uc131\ub2a5\uc740 \uc77c\ubc18\uc801\uc73c\ub85c \ub370\uc774\ud130\uc758 \uc591\uc5d0 \uc815\ube44\ub840\ud55c\ub2e4. \ub530\ub77c\uc11c pre-training\uc5d0\uc11c \ub370\uc774\ud130\ub97c \uc801\uac8c \uc0ac\uc6a9\ud55c \uc5b8\uc5b4\uc758 \uc131\ub2a5\uc740 \ub5a8\uc5b4\uc9c8 \uc218 \ubc16\uc5d0 \uc5c6\ub2e4.<\/p>\n\n\n\n

\uadf8\ub807\ub2e4\uba74 \ub370\uc774\ud130\uac00 \uc801\uc740 low-resource \uc5b8\uc5b4\uc758 \uc815\ubcf4\ub294 high-resource \uc5b8\uc5b4\ub9cc\ud07c \ud070 \uc2a4\ucf00\uc77c\uc758 \uc5b8\uc5b4 \ubaa8\ub378\ub85c \ud6c8\ub828\uc2dc\ud0ac \uc218 \uc5c6\ub294\uac78\uae4c? \uc774\ubc88 \ud3ec\uc2a4\ud2b8\uc5d0\uc11c\ub294 high-resource \uc5b8\uc5b4\ub97c \uc774\uc6a9\ud558\uc5ec low-resource \uc5b8\uc5b4\ub97c \ubaa8\ub378\uc774 \uc798 \ud559\uc2b5\ud558\ub3c4\ub85d \ud558\ub294 \ubc29\ubc95\uc778 Cross-lingual Transfer Learning<\/strong> \uc744 \uc54c\uc544\ubcf8\ub2e4. \ube44\uc2b7\ud55c \uc5f0\uad6c\uc778 Monolingual Language Transfer Learning<\/strong> \uc740 \ub2e4\uc74c \ud3ec\uc2a4\ud2b8\uc5d0\uc11c \ub2e4\ub8e8\ub824\uace0 \ud55c\ub2e4.<\/p>\n\n\n\n

Cross-lingual Transfer Learning Objective<\/h2>\n\n\n\n
\"\"
Cross-lingual transfer learning \uac1c\uc694.<\/figcaption><\/figure>\n\n\n\n

Cross-lingual representation<\/strong> \uc774\ub780 cross-lingual \uc815\ubcf4\ub97c \ub2f4\uc740 representation\uc744 \ub73b\ud55c\ub2e4. \uc989, \uc5ec\ub7ec\uac1c\uc758 \ubaa8\ub378\uc774 \uac01\uac01 \ub2e4\ub978 \uc5b8\uc5b4\ub97c \ub530\ub85c \ud559\uc2b5\ud558\ub294 \uac83\uc774 \uc544\ub2cc, \ud558\ub098\uc758 \ubaa8\ub378\uc774 \ub2e4\uad6d\uc5b4\ub97c \ud45c\ud604\ud558\ub294 universal embedding\uc744 \ud559\uc2b5\ud55c\ub2e4. \uc55e\uc11c \uc5b8\uae09\ud55c mBERT \uacfc XLM \ubaa8\ub378\uc774 cross-lingual representation\uc744 \ud559\uc2b5\ud558\ub294 multilingual pre-trained model \uc774\ub2e4. <\/p>\n\n\n\n

Cross-lingual Transfer Learning <\/strong>(CLTL) \uc740 \uc774\ub7f0 cross-lingual PLM \ubaa8\ub378\uc5d0 fine-tuning \uc9c4\ud589\ud55c\ub2e4. \uc6d0\ub798 PLM\uc774 \uc9c0\ub2cc \uc815\ubcf4\uc640 \uc0c8\ub85c\uc6b4 \ub370\uc774\ud130\ub97c \ud65c\uc6a9\ud558\uc5ec \uc0c8\ub85c\uc6b4 \uc9c0\uc2dd\uc744 \ud559\uc2b5\ud558\ub294 \uac83\uc774 \ubaa9\uc801\uc774\ub2e4. PLM \uc5d0\uc11c \ucda9\ubd84\ud788 \ud559\uc2b5\ub418\uc9c0 \ubabb\ud55c \uc5b8\uc5b4\ub098 \uc544\uc608 \uc0c8\ub85c\uc6b4 \uc5b8\uc5b4\ub97c \ubaa8\ub378\ub85c \ud558\uc5ec\uae08 \ud559\uc2b5\ud558\ub3c4\ub85d \ud55c\ub2e4. Source \uc5b8\uc5b4\uc758 \uc815\ubcf4\ub97c \uc774\uc6a9\ud55c\ub2e4\ub294 \uc810\uc774 \uc911\uc694\ud558\ub2e4. \uc798 \ud559\uc2b5\ub41c source \uc5b8\uc5b4\ub97c \uc5b4\ub5a0\ud55c \ubc29\ubc95\uc744 \ud1b5\ud574 target \uc5b8\uc5b4\ub85c “transfer” \ud55c\ub2e4. \uc608\ub97c \ub4e4\uc5b4, mBERT \ub97c target \uc5b8\uc5b4\uc778 \ud55c\uad6d\uc5b4 \ub370\uc774\ud130\ub85c fine-tuning \ud55c\ub2e4\uace0 \ud574\ubcf4\uc790. mBERT\uac00 \uae30\uc874\uc5d0 \ud559\uc2b5\ud588\ub358 104\uac1c\uc758 \uc5b8\uc5b4\uac00 source \uc774\ub2e4. \uc6b0\ub9ac\uac00 \uae30\ub300\ud558\ub294 \uacb0\uacfc\ub294 training from scratch \ud55c \ud55c\uad6d\uc5b4 \ubaa8\ub378 (\uc774\ud558 scratch \ubaa8\ub378) \ubcf4\ub2e4, mBERT \uc5d0 CLTL \ud558\uc5ec \uc5bb\uc740 \ubaa8\ub378\uc758 \ud55c\uad6d\uc5b4 \uc131\ub2a5\uc774 \ub354 \uc88b\uc740 \uac83\uc774\ub2e4. \uc989, \ub611\uac19\uc774 10\ub9cc \uac1c\uc758 \ud55c\uad6d\uc5b4 \ub370\uc774\ud130\ub97c \uc774\uc6a9\ud55c\ub2e4\uace0 \ud588\uc744 \ub54c, scratch model\ubcf4\ub2e4 CLTL\ub85c fine-tuning\ub41c \ubaa8\ub378\uc774 \ud55c\uad6d\uc5b4\ub97c \ub354 \uc798 \ud559\uc2b5\ud558\uac8c\ub054 \ud55c\ub2e4. <\/p>\n\n\n\n

\ucc38\uace0\ub85c, \ud55c\uad6d\uc5b4\ub294 low-resource \uc5b8\uc5b4\ub77c\uace0 \ubcf4\uae30 \ud798\ub4e4\ub2e4\ub294 \uc758\uacac\ub3c4 \ub9ce\ub2e4. Scratch \ubaa8\ub378\uc744 \ud559\uc2b5\ud560 \uc815\ub3c4\uc758 \ub370\uc774\ud130\ub294 \ucda9\ubd84\ud788 \uc218\uc9d1\uc774 \uac00\ub2a5\ud558\uae30 \ub54c\ubb38\uc774\ub2e4. CLTL \uc740 \ud55c\uad6d\uc5b4\ubcf4\ub2e4 \ub354 \uc790\uc6d0\uc774 \uc801\uc740 \uc5b8\uc5b4\ub97c \ub300\uc0c1\uc73c\ub85c \ud560 \ub54c \ub354 \ub6f0\uc5b4\ub09c \ud6a8\uacfc\ub97c \uae30\ub300\ud560 \uc218 \uc788\ub2e4.<\/p>\n\n\n\n

Cross-lingual Transfer Methods<\/h2>\n\n\n\n

Transfer \ubc29\uc2dd\uc740 \ud06c\uac8c \uc138 \ubd84\ub958\ub85c \ub098\ub20c \uc218 \uc788\ub2e4. \uac01\uac01 instance transfer<\/strong>, parameter transfer<\/strong>, feature transfer<\/strong> \uc774\ub2e4.<\/p>\n\n\n\n

Instance transfer\uc740 target \uc5b8\uc5b4\uc758 \ub2e4\uc591\ud55c instance\ub97c transfer \ud558\uc5ec source \uc5b8\uc5b4\uc758 instance\ub97c \ud48d\ubd80\ud654\ud55c\ub2e4. \ud604\uc7ac\uae4c\uc9c0 \ub2e4\uc591\ud55c instance transfer \ubc29\ubc95\uc774 \uc81c\uc2dc\ub418\uc5c8\uc9c0\ub9cc, \uac00\uc7a5 \uc9c1\uad00\uc801\uc774\uba74\uc11c\ub3c4 \ud6a8\uacfc\uc801\uc778 \ubc29\ubc95\uc740 translation data augmentation\b\uc774\ub2e4. Source \uc5b8\uc5b4\uc758 \ub370\uc774\ud130\ub97c target \uc5b8\uc5b4\ub85c NMT \ub4f1\uc744 \uc774\uc6a9\ud574 \ubc88\uc5ed\ud558\ub294 \ubc29\uc2dd\uc73c\ub85c \ub354 \ub9ce\uc740 target \uc5b8\uc5b4\ub97c \ud655\ubcf4\ud55c\ub2e4. \uac00\uc7a5 \uc26c\uc6b4 \uc608\uc2dc\ub97c \uc0b4\ud3b4\ubcf4\uc790\uba74, R\ub294 real text data \ubaa8\uc74c\uc744, A\ub294 translation data \ubaa8\uc74c\uc744 \ub098\ud0c0\ub0b4\uba70, Source \uc5b8\uc5b4\ub97c S\ub85c, target \uc5b8\uc5b4\ub97c T\ub85c \ud45c\uae30\ud55c\ub2e4\uace0 \ud558\uc790. Source \uc5b8\uc5b4\uc758 \ubcd1\ub82c \ub370\uc774\ud130 (real text data) \ubaa8\uc74c\uc740 RT<\/sub>Train<\/sup> = (XS<\/sub>Train<\/sup>, YS<\/sub>Train<\/sup>) \ub85c \uc815\uc758\ud560 \uc218 \uc788\ub2e4. RT<\/sub>Train<\/sup>\uc5d0 Instance transfer\uc744 \uc218\ud589\ud558\uba74 \ubc88\uc5ed\uc758 \uacb0\uacfc\ubb3c\uc778 target \uc5b8\uc5b4\uc758 pseudo-training data AT<\/sub>Train<\/sup> = (XT<\/sub>Train<\/sup>, YT<\/sub>Train<\/sup>) \uc744 \uc5bb\ub294\ub2e4. Test \ub370\uc774\ud130\uc5d0\ub3c4 \ub9c8\ucc2c\uac00\uc9c0\ub85c transfer \uc744 \uc801\uc6a9\ud55c\ub2e4.<\/p>\n\n\n\n

\"\"
Example of parameter transfer.
(Effective Cross-lingual Transfer of Neural Machine Translation Models without Shared Vocabularies)
<\/a><\/figcaption><\/figure>\n\n\n\n

Parameter transfer\uc740 share-private \ubaa8\ub378 \uc544\ud0a4\ud14d\ucc98\ub97c \ud559\uc2b5\ud558\ub294 \ub4f1\uc758 \ubc29\ubc95\uc73c\ub85c parameter\uc744 transfer\ud55c\ub2e4. \uac00\ub839, PLM \uc774 \ud559\uc2b5\ud55c parameter \b\uc744 \uadf8\ub300\ub85c \uac00\uc838\uc640 \uc7ac\ud559\uc2b5 \uc2dc\ud0a4\ub294 \ubc29\ubc95\uc774 \uc774\uc5d0 \ud574\ub2f9\ud55c\ub2e4. \uc77c\ubd80 parameter\uc740 freeze \ud558\uc5ec \uae30\uc874\uc5d0 \ubc30\uc6b4 \uc815\ubcf4\ub97c \uc77c\ubd80 \uace0\uc815\ud55c \uc0c1\ud0dc\uc5d0\uc11c \ubaa8\ub378 \ud559\uc2b5\uc744 \uc9c4\ud589\ud560 \uc218\ub3c4 \uc788\ub2e4.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

\ub9c8\uc9c0\ub9c9\uc73c\ub85c feature transfer\uc740 source \uc5b8\uc5b4\uc758 \ud2b9\uc9d5\uc744 target \uc5b8\uc5b4\ub85c transfer \ud558\ub294 \ubc29\uc2dd\uc73c\ub85c \ud559\uc2b5\ud55c\ub2e4. \uac00\ub839 \uc5b8\uc5b4\uac04 representation \uac2d\uc744 align \ud558\ub294 \ubc29\ubc95\uc774 \uc5ec\uae30\uc5d0 \uc18d\ud55c\ub2e4. \uc608\ub97c \ub4e4\uc5b4, \uc704 \uadf8\ub9bc\uc740 \ubcf4\uba74 \ub2e4\ub978 \uc5b8\uc5b4\ub4e4\uc758 sentence representation \uac19\uc740 space\uc5d0 \ubfcc\ub9b0 \ubaa8\uc2b5\uc778\ub370, \uc5b8\uc5b4\ub9c8\ub2e4 \uc704\uce58\ud558\ub294 \uc790\ub9ac\uac00 \uc870\uae08\uc529 \ub2e4\ub974\ub2e4. \ud2b9\ud788 \uc5b8\uc5b4\uc758 \ud2b9\uc9d5\uc774 \ub2e4\ub978 \uc5b8\uc5b4\uc77c\uc218\ub85d \ud574\ub2f9 \uacf5\uac04\uc5d0\uc11c\uc758 \uac70\ub9ac\ub3c4 \uba40\uc5b4\uc9c4\ub2e4. feature transfer \uc5d0 \uc18d\ud558\ub294 representation gap reducing \ubc29\ubc95\uc740, \uc11c\ub85c \ub2e4\ub978 \uc5b8\uc5b4\uac00 \ucd5c\uc885\uc801\uc73c\ub85c \ube44\uc2b7\ud55c \uc704\uce58\uc5d0 \ubfcc\ub824\uc9c8 \uc218 \uc788\ub3c4\ub85d \ud55c\ub2e4. <\/p>\n\n\n\n

\"\"
Example of feature transfer.
(Enhancing Cross-lingual Transfer by Manifold Mixup<\/a>)<\/figcaption><\/figure>\n\n\n\n

\uc704 \uadf8\ub9bc\uc5d0\uc11c Translate-train \uacfc X-MIXUP \uc740 \ubaa8\ub450 feature transfer \ubc29\uc2dd\uc744 \uc774\uc6a9\ud558\ub294 \uba54\uc3d8\ub4dc\uc774\ub2e4. \ub450 \ubc29\uc2dd \ubaa8\ub450 \ub2e4\uad6d\uc5b4 \ub370\uc774\ud130\ub97c \uac19\uc740 \uacf5\uac04\uc5d0 \ubfcc\ub838\uc744 \ub54c, \ube44\uc2b7\ud55c \uc704\uce58\uc5d0 \uc704\uce58\ud560 \uc218 \uc788\ub3c4\ub85d \ud559\uc2b5\ud55c\ub2e4. X-MIXUP \uc744 \uc801\uc6a9\ud588\uc744 \ub54c \ub2e4\ub978 \uc5b8\uc5b4\ub4e4\uc758 \ubd84\ud3ec\uac00 \uc880 \ub354 \ube44\uc2b7\ud55c \uac83\uc744 \ud655\uc778\ud560 \uc218 \uc788\ub2e4.<\/p>\n\n\n\n

Model Evaluation<\/h2>\n\n\n\n

Cross-lingual representation transfer learning \ud3c9\uac00\uc5d0\uc11c \ubd10\uc57c \ud560 \uac83\uc740 transfer learning\uc744 \uac70\uce5c \ubaa8\ub378\uc774 \uae30\uc874 \ubaa8\ub378\uacfc \ube44\uad50\ud588\uc744 \ub54c \uc5bc\ub9c8\ub098 \uc131\ub2a5\uc774 \uc99d\uac00\ud558\uc600\ub294\uac00\uc774\ub2e4. \uc989, task evaluation \ubc29\uc2dd\uc73c\ub85c \uae30\uc874 \ubaa8\ub378\uacfc \uc131\ub2a5\uc744 \ube44\uad50\ud558\uae30\ub9cc \ud558\uba74 \ub41c\ub2e4. \uc608\ub97c \ub4e4\uc5b4, [\ub17c\ubb38]\uc5d0\uc11c\ub294 mBERT\uc640 XLM-R\uc744 backbone \ubaa8\ub378\ub85c \ud558\uc5ec transfer learning\uc744 \uc9c4\ud589\ud55c\ub2e4. \ud0dc\uc2a4\ud06c\ub294 XTREME \uc758 \ud0dc\uc2a4\ud06c, XNLI, PAWS-X, POS \ub4f1\uc744 \ud3ec\ud568\ud55c\ub2e4.<\/p>\n\n\n\n

\"\"
Example of CLTL model evaluation.
((Enhancing Cross-lingual Transfer by Manifold Mixup<\/a>))<\/figcaption><\/figure>\n\n\n\n

Instance transfer \uacfc feature transfer\uc744 \uc801\uc6a9\ud55c X-MIXUP \uc774 \ub2e4\uc591\ud55c task\uc5d0\uc11c backbone \ubaa8\ub378\uacfc \ub2e4\ub978 baseline \ubaa8\ub378\ubcf4\ub2e4 \ub192\uc740 \uc131\ub2a5\uc744 \uc5bb\uc5c8\uc74c\uc774 \ud655\uc778\ub418\uc5c8\ub2e4. \uc5ec\uae30\uc11c \uc0ac\uc6a9\ud55c \ud3c9\uac00 \uc9c0\ud45c\ub294 \uae30\uc874 task\uc5d0\uc11c \uc0ac\uc6a9\ub418\ub294 \uc9c0\ud45c\uc640 \uac19\ub2e4. \uc608\ub97c \ub4e4\uc5b4, NER \ud0dc\uc2a4\ud06c\ub294 F1 score \ub85c \uc131\ub2a5\uc744 \ube44\uad50\ud588\ub2e4.<\/p>\n\n\n\n

Summary<\/h2>\n\n\n\n

\uc774\ubc88 \ud3ec\uc2a4\ud2b8\uc5d0\uc11c\ub294 CLTL\uc744 \uac04\ub7b5\ud788 \uc815\ub9ac\ud574\ubcf4\uc558\ub2e4. CLTL\uc740 \uae30\uc874 PLM\uc744 \ud65c\uc6a9\ud558\uc5ec \uc801\uc740 \uc5b8\uc5b4 \ub370\uc774\ud130\ub9cc\uc73c\ub85c\ub3c4 large scale language model\uc744 \ub9cc\ub4e4 \uc218 \uc788\ub294 \ub9e4\uc6b0 \ud6a8\uacfc\uc801\uc778 \ubc29\ubc95\uc774\ub2e4. \ub2e4\uc74c \ud3ec\uc2a4\ud2b8\uc5d0\uc11c\ub294 CLTL\uacfc \ube44\uc2b7\ud558\uc9c0\ub9cc \ub2e4\ub978 Monolingual Language Transfer Learning\uc5d0 \ub300\ud574 \ub2e4\ub904\ubcf4\ub824\uace0 \ud55c\ub2e4. <\/p>\n","protected":false},"excerpt":{"rendered":"

\uc774 \ud3ec\uc2a4\ud2b8\ub294 \ucd5c\uadfc Eleuther.AI \uc758 Language Transferring \ud504\ub85c\uc81d\ud2b8\uc5d0 \ucc38\uc5ec\ud558\uba74\uc11c \uacf5\ubd80\ud55c \ub0b4\uc6a9\uc744 \uc815\ub9ac\ud558\uace0\uc790 \uc791\uc131\ud558\uac8c \ub418\uc5c8\ub2e4. BERT, GPT-3, GPT-Neo\uc640 \uac19\uc740 \ud604\uc7ac\uae4c\uc9c0 \uacf5\uac1c\ub41c \ud070 \uc2a4\ucf00\uc77c\uc758 PLM (Pre-trained Languae Model) \uc740 \uc601\uc5b4 \ub370\uc774\ud130\ub97c \uae30\ubc18\uc73c\ub85c \ud55c \ubaa8\ub378\uc774 \uac00\uc7a5 \ub9ce\ub2e4. PLM \uc5f0\uad6c\uac00 \uc131\uc219\ub428\uc5d0 \ub530\ub77c \ud55c\uad6d\uc5b4, \uc911\uad6d\uc5b4, \ub3c5\uc77c\uc5b4\uc640 \uac19\uc740 […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"nf_dc_page":"","_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[14],"tags":[37],"class_list":["post-113","post","type-post","status-publish","format-standard","hentry","category-natural-language-processing","tag-nlp"],"yoast_head":"\nCross-lingual Language Transfer Learning - Overview - Jaein's Lab<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/jaeinlab.com\/?p=113\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Cross-lingual Language Transfer Learning - Overview - Jaein's Lab\" \/>\n<meta property=\"og:description\" content=\"\uc774 \ud3ec\uc2a4\ud2b8\ub294 \ucd5c\uadfc Eleuther.AI \uc758 Language Transferring \ud504\ub85c\uc81d\ud2b8\uc5d0 \ucc38\uc5ec\ud558\uba74\uc11c \uacf5\ubd80\ud55c \ub0b4\uc6a9\uc744 \uc815\ub9ac\ud558\uace0\uc790 \uc791\uc131\ud558\uac8c \ub418\uc5c8\ub2e4. BERT, GPT-3, GPT-Neo\uc640 \uac19\uc740 \ud604\uc7ac\uae4c\uc9c0 \uacf5\uac1c\ub41c \ud070 \uc2a4\ucf00\uc77c\uc758 PLM (Pre-trained Languae Model) \uc740 \uc601\uc5b4 \ub370\uc774\ud130\ub97c \uae30\ubc18\uc73c\ub85c \ud55c \ubaa8\ub378\uc774 \uac00\uc7a5 \ub9ce\ub2e4. PLM \uc5f0\uad6c\uac00 \uc131\uc219\ub428\uc5d0 \ub530\ub77c \ud55c\uad6d\uc5b4, \uc911\uad6d\uc5b4, \ub3c5\uc77c\uc5b4\uc640 \uac19\uc740 […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/jaeinlab.com\/?p=113\" \/>\n<meta property=\"og:site_name\" content=\"Jaein's Lab\" \/>\n<meta property=\"article:published_time\" content=\"2022-07-22T01:07:29+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-11-02T10:59:13+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/jaeinlab.com\/wp-content\/uploads\/2022\/07\/Screen-Shot-2022-07-16-at-11.30.45-AM-1024x407.png\" \/>\n<meta name=\"author\" content=\"Jaein\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Jaein\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/jaeinlab.com\\\/?p=113#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/jaeinlab.com\\\/?p=113\"},\"author\":{\"name\":\"Jaein\",\"@id\":\"http:\\\/\\\/jaeinlab.com\\\/#\\\/schema\\\/person\\\/70edb94493e55ab8edbafc48bdf56883\"},\"headline\":\"Cross-lingual Language Transfer Learning – Overview\",\"datePublished\":\"2022-07-22T01:07:29+00:00\",\"dateModified\":\"2023-11-02T10:59:13+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/jaeinlab.com\\\/?p=113\"},\"wordCount\":274,\"commentCount\":0,\"image\":{\"@id\":\"https:\\\/\\\/jaeinlab.com\\\/?p=113#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/jaeinlab.com\\\/wp-content\\\/uploads\\\/2022\\\/07\\\/Screen-Shot-2022-07-16-at-11.30.45-AM-1024x407.png\",\"keywords\":[\"NLP\"],\"articleSection\":[\"Natural Language Processing\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/jaeinlab.com\\\/?p=113#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/jaeinlab.com\\\/?p=113\",\"url\":\"https:\\\/\\\/jaeinlab.com\\\/?p=113\",\"name\":\"Cross-lingual Language Transfer Learning - Overview - Jaein's Lab\",\"isPartOf\":{\"@id\":\"http:\\\/\\\/jaeinlab.com\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/jaeinlab.com\\\/?p=113#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/jaeinlab.com\\\/?p=113#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/jaeinlab.com\\\/wp-content\\\/uploads\\\/2022\\\/07\\\/Screen-Shot-2022-07-16-at-11.30.45-AM-1024x407.png\",\"datePublished\":\"2022-07-22T01:07:29+00:00\",\"dateModified\":\"2023-11-02T10:59:13+00:00\",\"author\":{\"@id\":\"http:\\\/\\\/jaeinlab.com\\\/#\\\/schema\\\/person\\\/70edb94493e55ab8edbafc48bdf56883\"},\"breadcrumb\":{\"@id\":\"https:\\\/\\\/jaeinlab.com\\\/?p=113#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/jaeinlab.com\\\/?p=113\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/jaeinlab.com\\\/?p=113#primaryimage\",\"url\":\"https:\\\/\\\/i0.wp.com\\\/jaeinlab.com\\\/wp-content\\\/uploads\\\/2022\\\/07\\\/Screen-Shot-2022-07-16-at-11.30.45-AM.png?fit=1368%2C544&ssl=1\",\"contentUrl\":\"https:\\\/\\\/i0.wp.com\\\/jaeinlab.com\\\/wp-content\\\/uploads\\\/2022\\\/07\\\/Screen-Shot-2022-07-16-at-11.30.45-AM.png?fit=1368%2C544&ssl=1\",\"width\":1368,\"height\":544},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/jaeinlab.com\\\/?p=113#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"http:\\\/\\\/jaeinlab.com\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Cross-lingual Language Transfer Learning – Overview\"}]},{\"@type\":\"WebSite\",\"@id\":\"http:\\\/\\\/jaeinlab.com\\\/#website\",\"url\":\"http:\\\/\\\/jaeinlab.com\\\/\",\"name\":\"Jaein's Lab\",\"description\":\"Natural Language Processing, Knowledge Graph, and others!\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"http:\\\/\\\/jaeinlab.com\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"http:\\\/\\\/jaeinlab.com\\\/#\\\/schema\\\/person\\\/70edb94493e55ab8edbafc48bdf56883\",\"name\":\"Jaein\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/6a23798f753744134b0f342a342b0fb0d4035a7313d8b7c54faf930978eaa942?s=96&d=wp_user_avatar&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/6a23798f753744134b0f342a342b0fb0d4035a7313d8b7c54faf930978eaa942?s=96&d=wp_user_avatar&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/6a23798f753744134b0f342a342b0fb0d4035a7313d8b7c54faf930978eaa942?s=96&d=wp_user_avatar&r=g\",\"caption\":\"Jaein\"},\"description\":\"I'm a Computer Science PhD candidate from South Korea, currently based in Beijing, China. My academic journey is dedicated to pioneering research in Natural Language Processing and Knowledge Graphs, striving to break new ground in these dynamic fields.\",\"sameAs\":[\"http:\\\/\\\/jaeinlab.com\",\"https:\\\/\\\/www.linkedin.com\\\/in\\\/jaeinlab\\\/\"],\"url\":\"https:\\\/\\\/jaeinlab.com\\\/?author=1\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Cross-lingual Language Transfer Learning - Overview - Jaein's Lab","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/jaeinlab.com\/?p=113","og_locale":"en_US","og_type":"article","og_title":"Cross-lingual Language Transfer Learning - Overview - Jaein's Lab","og_description":"\uc774 \ud3ec\uc2a4\ud2b8\ub294 \ucd5c\uadfc Eleuther.AI \uc758 Language Transferring \ud504\ub85c\uc81d\ud2b8\uc5d0 \ucc38\uc5ec\ud558\uba74\uc11c \uacf5\ubd80\ud55c \ub0b4\uc6a9\uc744 \uc815\ub9ac\ud558\uace0\uc790 \uc791\uc131\ud558\uac8c \ub418\uc5c8\ub2e4. BERT, GPT-3, GPT-Neo\uc640 \uac19\uc740 \ud604\uc7ac\uae4c\uc9c0 \uacf5\uac1c\ub41c \ud070 \uc2a4\ucf00\uc77c\uc758 PLM (Pre-trained Languae Model) \uc740 \uc601\uc5b4 \ub370\uc774\ud130\ub97c \uae30\ubc18\uc73c\ub85c \ud55c \ubaa8\ub378\uc774 \uac00\uc7a5 \ub9ce\ub2e4. PLM \uc5f0\uad6c\uac00 \uc131\uc219\ub428\uc5d0 \ub530\ub77c \ud55c\uad6d\uc5b4, \uc911\uad6d\uc5b4, \ub3c5\uc77c\uc5b4\uc640 \uac19\uc740 […]","og_url":"https:\/\/jaeinlab.com\/?p=113","og_site_name":"Jaein's Lab","article_published_time":"2022-07-22T01:07:29+00:00","article_modified_time":"2023-11-02T10:59:13+00:00","og_image":[{"url":"https:\/\/jaeinlab.com\/wp-content\/uploads\/2022\/07\/Screen-Shot-2022-07-16-at-11.30.45-AM-1024x407.png","type":"","width":"","height":""}],"author":"Jaein","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Jaein","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/jaeinlab.com\/?p=113#article","isPartOf":{"@id":"https:\/\/jaeinlab.com\/?p=113"},"author":{"name":"Jaein","@id":"http:\/\/jaeinlab.com\/#\/schema\/person\/70edb94493e55ab8edbafc48bdf56883"},"headline":"Cross-lingual Language Transfer Learning – Overview","datePublished":"2022-07-22T01:07:29+00:00","dateModified":"2023-11-02T10:59:13+00:00","mainEntityOfPage":{"@id":"https:\/\/jaeinlab.com\/?p=113"},"wordCount":274,"commentCount":0,"image":{"@id":"https:\/\/jaeinlab.com\/?p=113#primaryimage"},"thumbnailUrl":"https:\/\/jaeinlab.com\/wp-content\/uploads\/2022\/07\/Screen-Shot-2022-07-16-at-11.30.45-AM-1024x407.png","keywords":["NLP"],"articleSection":["Natural Language Processing"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/jaeinlab.com\/?p=113#respond"]}]},{"@type":"WebPage","@id":"https:\/\/jaeinlab.com\/?p=113","url":"https:\/\/jaeinlab.com\/?p=113","name":"Cross-lingual Language Transfer Learning - Overview - Jaein's Lab","isPartOf":{"@id":"http:\/\/jaeinlab.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/jaeinlab.com\/?p=113#primaryimage"},"image":{"@id":"https:\/\/jaeinlab.com\/?p=113#primaryimage"},"thumbnailUrl":"https:\/\/jaeinlab.com\/wp-content\/uploads\/2022\/07\/Screen-Shot-2022-07-16-at-11.30.45-AM-1024x407.png","datePublished":"2022-07-22T01:07:29+00:00","dateModified":"2023-11-02T10:59:13+00:00","author":{"@id":"http:\/\/jaeinlab.com\/#\/schema\/person\/70edb94493e55ab8edbafc48bdf56883"},"breadcrumb":{"@id":"https:\/\/jaeinlab.com\/?p=113#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/jaeinlab.com\/?p=113"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/jaeinlab.com\/?p=113#primaryimage","url":"https:\/\/i0.wp.com\/jaeinlab.com\/wp-content\/uploads\/2022\/07\/Screen-Shot-2022-07-16-at-11.30.45-AM.png?fit=1368%2C544&ssl=1","contentUrl":"https:\/\/i0.wp.com\/jaeinlab.com\/wp-content\/uploads\/2022\/07\/Screen-Shot-2022-07-16-at-11.30.45-AM.png?fit=1368%2C544&ssl=1","width":1368,"height":544},{"@type":"BreadcrumbList","@id":"https:\/\/jaeinlab.com\/?p=113#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"http:\/\/jaeinlab.com\/"},{"@type":"ListItem","position":2,"name":"Cross-lingual Language Transfer Learning – Overview"}]},{"@type":"WebSite","@id":"http:\/\/jaeinlab.com\/#website","url":"http:\/\/jaeinlab.com\/","name":"Jaein's Lab","description":"Natural Language Processing, Knowledge Graph, and others!","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"http:\/\/jaeinlab.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"http:\/\/jaeinlab.com\/#\/schema\/person\/70edb94493e55ab8edbafc48bdf56883","name":"Jaein","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/6a23798f753744134b0f342a342b0fb0d4035a7313d8b7c54faf930978eaa942?s=96&d=wp_user_avatar&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/6a23798f753744134b0f342a342b0fb0d4035a7313d8b7c54faf930978eaa942?s=96&d=wp_user_avatar&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/6a23798f753744134b0f342a342b0fb0d4035a7313d8b7c54faf930978eaa942?s=96&d=wp_user_avatar&r=g","caption":"Jaein"},"description":"I'm a Computer Science PhD candidate from South Korea, currently based in Beijing, China. My academic journey is dedicated to pioneering research in Natural Language Processing and Knowledge Graphs, striving to break new ground in these dynamic fields.","sameAs":["http:\/\/jaeinlab.com","https:\/\/www.linkedin.com\/in\/jaeinlab\/"],"url":"https:\/\/jaeinlab.com\/?author=1"}]}},"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/jaeinlab.com\/index.php?rest_route=\/wp\/v2\/posts\/113","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/jaeinlab.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/jaeinlab.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/jaeinlab.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/jaeinlab.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=113"}],"version-history":[{"count":5,"href":"https:\/\/jaeinlab.com\/index.php?rest_route=\/wp\/v2\/posts\/113\/revisions"}],"predecessor-version":[{"id":128,"href":"https:\/\/jaeinlab.com\/index.php?rest_route=\/wp\/v2\/posts\/113\/revisions\/128"}],"wp:attachment":[{"href":"https:\/\/jaeinlab.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=113"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/jaeinlab.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=113"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/jaeinlab.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=113"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}