政大機構典藏-National Chengchi University Institutional Repository(NCCUR):Item 140.119/124122

政大機構典藏-National Chengchi University Institutional Repository(NCCUR):Item 140.119/124122

English | 正體中文 | 简体中文 | Post-Print筆數 : 27 | Items with full text/Total items : 109951/140887 (78%)
Visitors : 46273006 Online Users : 1333

RC Version 6.0 © Powered By DSPACE, MIT. Enhanced by NTU Library IR team.

Scope

please add "double quotation mark" for query phrases to get precise results

please goto advance search for comprehansive author search

Adv. Search

Home ‧ Login ‧ Upload ‧ Help ‧ About ‧ Administer

Goto mobile version

政大典藏 > College of Commerce > Department of Statistics > Theses > Item 140.119/124122

Please use this identifier to cite or link to this item: https://nccur.lib.nccu.edu.tw/handle/140.119/124122

Title:	基於兩詞彙的序列關係建造非監督式 SeqWORDS 斷詞方法 SeqWORDS: an unsupervised Chinese segmentation method using relationship of two consecutive words.
Authors:	吳冠輝 Wu, Guan-Hui
Contributors:	薛慧敏 Hsueh, Huey-Miin 吳冠輝 Wu, Guan-Hui
Keywords:	中文斷詞文本探勘動態規劃法文字詞典模型 EM演算法詞彙序列關係 Chinese texts mining Dynamic programming EM algorithm Word dictionary model Words dependency Word segmentation
Date:	2019
Issue Date:	2019-07-01 10:43:44 (UTC+8)
Abstract:	由於中文文本中的詞彙之間沒有任何標記或空格，所以斷詞被認為是中文文本探勘前必要且重要的預處理步驟。而目前中文斷詞方法多屬監督式方法，當沒有適當的詞典時難以發揮，例如針對新世代的文章或特定專業領域的文本。Deng等人在2016年提出非監督式斷詞方法TopWORDS，利用文字詞典模型(Word D ictionary Model, WDM)建構文本之概似函數，並且將斷詞資訊當作遺失變數，以EM演算法估計出各詞彙的使用機率，更利用動態規劃法(dynamic programm ing)計算，除了運算上相當具有效率，TopWORDS應用在許多文本上有良好的結果。然而，TopWORDS假設文本中每個位置的詞彙獨立且分配相同，這樣的假設恐怕忽略了詞彙在文意上的相連。此研究假設每個詞彙出現的概率與前一個詞彙有關，因此文本的概似函數可表示為兩詞彙的序列關係的函數，故將此研究提出的方法稱為「SeqWORDS」。在運用三種不同斷詞法於紅樓夢文本上後，我們觀察到 SeqWORDS雖然在探索新詞彙的能力較弱，然而當接續使用文本探勘工具如詞向量分析後發現，SeqWORDS 能提供最佳的解釋性。 Unlike alphabet-based language, there exists no space between words in Chinese corpus. The first step in Chinese text mining is to segment words in a sentence. Many existing segmentation methods are supervised in terms of requiring an adequate dictionary. However, Chinese language has developed so long and growing so fast. A suitable dictionary may not be available or easily accessed. In 2016, Deng et al. proposed an unsupervised method called “TopWORDS”, which needs no dictionary in hand. The authors derived the likelihood function of the corpus via word dictionary model (WDM). Further, they regard unknown segmentation information as missing data and utilize EM algorithm to estimate occurrence probability of words. To enhance computational efficiency, the estimates are computed by dynamic programming. In the article, the TopWORDS is found to perform well in several corpus. However, the iid assumption of TopWORDS ignores words dependency, which frequently occurs in consecutive words. Therefore, in this research we assume that a word’s occurrence depends on previous one and modify the TopWORDS method. By considering the sequential association of consecutive words, the proposed method is named “SeqWORDS”. The new method and two other existing methods are evaluated by their performance on the famous classical novel Story-of-Stone. We find that SeqWORDS is less capable to find new, rare words and is much time consuming. However, when we further implement some advance text mining analysis on the segmented corpus, the segmented corpus by SeqWORDS produces the most reasonable, interpretable results.
Reference:	[1] The Stanford Natural Language. Processing Group, Chinese Natural Language Processing and Speech Processing. Retrieved May 24, 2019, from https://nlp.stanford.edu/projects/chinese-nlp.shtml#cws [2] J. Lafferty, A. McCallum, F. C.N. Pereira, (2001), Conditional random fields: Probabilistc models for segmenting and labeling sequence data. Proceedings of the 18th International Conference on Machine Learning 2001(ICML 2001), pp 282–289. [3] fxsjy, Jieba, Retrieved May 27, 2019, from https://github.com/fxsjy/ji eba [4] L. R. Rabiner, B. H. Juang, (1986), An introduction to hidden Markov models, IEEE ASSP MAGAZINE, vol 3, no 1, pp. 4-16. [5] A. Chen, (2003), Chinese word segmentation using minimal linguistic knowledge. Proceeding SIGHAN `03 Proceedings of the second SIGHAN workshop on Chinese language processing, Vol 17, pp 148–151. [6] K. J. Chen, S. H. Liu, (1992), Word identification for Mandarin Chinese sentences. Proceeding COLING `92 Proceedings of the 14th conference on Computational linguistics, Vol 1, pp 101–107. [7] K. Deng, P. K. Bol, K. J. Li, and J. S. Liu, (2016). On the unsupervised analysis of domain-specific Chinese texts. Proceedings of the National Academy of Sciences of the United States of America, vol 113, pp 6154–6159. [8] X. Ge, W. Pratt, P. Smyth, (1999), Discovering Chinese words from unsegmented text. Proceeding SIGIR `99 Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, pp 271–272. [9] A. P. Dempster, N. M. Laird, D. B. Rubin, (1977), Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society. Series B, vol 39, no 1, pp 1-38. [10] R. Bellman, (1954), The theory of dynamic programming, Bulletin of the American Mathematical Society, vol 60, no 6, pp 503-515. [11] X. Cao, Story-of-Stone. [12] 胡適，(1988)，胡適紅樓夢研究論述全編，上海古籍出版社。 [13] T. Mikolov, K. Chen, G. Corrado, J. Dean, (2013). Efficient Estimation of Word Representations in Vector Space, arXiv:1301.3781v3. [14] T. Mikolov, I. Sutskever, K. Chen, G. Corrado, J. Dean, (2013). Distributed Representations of Words and Phrases and their Compositionality, NIPS 2013,3111-3119. [15] K. Pearson, (1901), On Lines and Planes of Closest Fit to Systems of Points in Space. Philosophical Magazine, vol 2, pp 559-572.
Description:	碩士國立政治大學統計學系 106354027
Source URI:	http://thesis.lib.nccu.edu.tw/record/#G0106354027
Data Type:	thesis
DOI:	10.6814/NCCU201900115
Appears in Collections:	[Department of Statistics] Theses

Files in This Item:

File	Size	Format
402701.pdf	2480Kb	Adobe PDF2	194	View/Open

All items in 政大典藏 are protected by copyright, with all rights reserved.

社群 sharing

著作權政策宣告 Copyright Announcement

1.本網站之數位內容為國立政治大學所收錄之機構典藏，無償提供學術研究與公眾教育等公益性使用，惟仍請適度，合理使用本網站之內容，以尊重著作權人之權益。商業上之利用，則請先取得著作權人之授權。
The digital content of this website is part of National Chengchi University Institutional Repository. It provides free access to academic research and public education for non-commercial use. Please utilize it in a proper and reasonable manner and respect the rights of copyright owners. For commercial use, please obtain authorization from the copyright owner in advance.

2.本網站之製作，已盡力防止侵害著作權人之權益，如仍發現本網站之數位內容有侵害著作權人權益情事者，請權利人通知本網站維護人員(nccur@nccu.edu.tw)，維護人員將立即採取移除該數位著作等補救措施。
NCCU Institutional Repository is made to protect the interests of copyright owners. If you believe that any material on the website infringes copyright, please contact our staff(nccur@nccu.edu.tw). We will remove the work from the repository and investigate your claim.

DSpace Software Copyright © 2002-2004 MIT & Hewlett-Packard / Enhanced by NTU Library IR team Copyright © - Feedback