English  |  正體中文  |  简体中文  |  Items with full text/Total items : 88266/117736 (75%)
Visitors : 23392431      Online Users : 180
RC Version 6.0 © Powered By DSPACE, MIT. Enhanced by NTU Library IR team.
Scope Tips:
  • please add "double quotation mark" for query phrases to get precise results
  • please goto advance search for comprehansive author search
  • Adv. Search
    HomeLoginUploadHelpAboutAdminister Goto mobile version
    政大機構典藏 > 商學院 > 統計學系 > 學位論文 >  Item 140.119/124122
    Please use this identifier to cite or link to this item: http://nccur.lib.nccu.edu.tw/handle/140.119/124122


    Title: 基於兩詞彙的序列關係建造非監督式 SeqWORDS 斷詞方法
    SeqWORDS: an unsupervised Chinese segmentation method using relationship of two consecutive words.
    Authors: 吳冠輝
    Wu, Guan-Hui
    Contributors: 薛慧敏
    Hsueh, Huey-Miin
    吳冠輝
    Wu, Guan-Hui
    Keywords: 中文斷詞
    文本探勘
    動態規劃法
    文字詞典模型
    EM演算法
    詞彙序列關係
    Chinese texts mining
    Dynamic programming
    EM algorithm
    Word dictionary model
    Words dependency
    Word segmentation
    Date: 2019
    Issue Date: 2019-07-01 10:43:44 (UTC+8)
    Abstract: 由於中文文本中的詞彙之間沒有任何標記或空格,所以斷詞被認為是中文文本探勘前必要且重要的預處理步驟。而目前中文斷詞方法多屬監督式方法,當沒有適當的詞典時難以發揮,例如針對新世代的文章或特定專業領域的文本。Deng等人在2016年提出非監督式斷詞方法TopWORDS,利用文字詞典模型(Word D ictionary Model, WDM)建構文本之概似函數,並且將斷詞資訊當作遺失變數,以EM演算法估計出各詞彙的使用機率,更利用動態規劃法(dynamic programm ing)計算,除了運算上相當具有效率,TopWORDS應用在許多文本上有良好的結果。然而,TopWORDS假設文本中每個位置的詞彙獨立且分配相同,這樣的假設恐怕忽略了詞彙在文意上的相連。此研究假設每個詞彙出現的概率與前一個詞彙有關,因此文本的概似函數可表示為兩詞彙的序列關係的函數,故將此研究提出的方法稱為「SeqWORDS」。在運用三種不同斷詞法於紅樓夢文本上後,我們觀察到 SeqWORDS雖然在探索新詞彙的能力較弱,然而當接續使用文本探勘工具如詞向量分析後發現,SeqWORDS 能提供最佳的解釋性。
    Unlike alphabet-based language, there exists no space between words in
    Chinese corpus. The first step in Chinese text mining is to segment words in a sentence. Many existing segmentation methods are supervised in terms of requiring an adequate dictionary. However, Chinese language has developed so long and growing so fast. A suitable dictionary may not be available or easily accessed. In 2016, Deng et al. proposed an unsupervised method called “TopWORDS”, which needs no dictionary in hand. The authors derived the likelihood function of the corpus via word dictionary model (WDM). Further, they regard unknown segmentation information as missing data and utilize EM algorithm to estimate occurrence probability of words. To enhance computational efficiency, the estimates are computed by dynamic programming. In the article, the TopWORDS is found to perform well in several corpus. However, the iid assumption of TopWORDS ignores words dependency, which frequently occurs in consecutive words. Therefore, in this research we assume that a word’s occurrence depends on previous one and modify the TopWORDS method. By considering the sequential association of consecutive words, the proposed method is named “SeqWORDS”. The new method and two other existing methods are evaluated by their performance on the famous classical novel Story-of-Stone. We find that SeqWORDS is less capable to find new, rare words and is much time consuming. However, when we further implement some advance text mining analysis on the segmented corpus, the segmented corpus by SeqWORDS produces the most reasonable, interpretable results.
    Reference: [1]
    The Stanford Natural Language. Processing Group, Chinese Natural Language Processing and Speech Processing. Retrieved May 24, 2019, from https://nlp.stanford.edu/projects/chinese-nlp.shtml#cws
    [2]
    J. Lafferty, A. McCallum, F. C.N. Pereira, (2001), Conditional random fields: Probabilistc models for segmenting and labeling sequence data. Proceedings of the 18th International Conference on Machine Learning 2001(ICML 2001), pp 282–289.
    [3]
    fxsjy, Jieba, Retrieved May 27, 2019, from https://github.com/fxsjy/ji eba
    [4]
    L. R. Rabiner, B. H. Juang, (1986), An introduction to hidden Markov models, IEEE ASSP MAGAZINE, vol 3, no 1, pp. 4-16.
    [5]
    A. Chen, (2003), Chinese word segmentation using minimal linguistic knowledge. Proceeding SIGHAN '03 Proceedings of the second SIGHAN workshop on Chinese language processing, Vol 17, pp 148–151.
    [6]
    K. J. Chen, S. H. Liu, (1992), Word identification for Mandarin Chinese sentences. Proceeding COLING '92 Proceedings of the 14th conference on Computational linguistics, Vol 1, pp 101–107.
    [7]
    K. Deng, P. K. Bol, K. J. Li, and J. S. Liu, (2016). On the unsupervised analysis of domain-specific Chinese texts. Proceedings of the National Academy of Sciences of the United States of America, vol 113, pp 6154–6159.
    [8]
    X. Ge, W. Pratt, P. Smyth, (1999), Discovering Chinese words from unsegmented text. Proceeding SIGIR '99 Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, pp 271–272.
    [9]
    A. P. Dempster, N. M. Laird, D. B. Rubin, (1977), Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society. Series B, vol 39, no 1, pp 1-38.
    [10]
    R. Bellman, (1954), The theory of dynamic programming, Bulletin of the American Mathematical Society, vol 60, no 6, pp 503-515.
    [11]
    X. Cao, Story-of-Stone.
    [12] 胡適,(1988),胡適紅樓夢研究論述全編,上海古籍出版社。
    [13]
    T. Mikolov, K. Chen, G. Corrado, J. Dean, (2013). Efficient Estimation of Word Representations in Vector Space, arXiv:1301.3781v3.
    [14]
    T. Mikolov, I. Sutskever, K. Chen, G. Corrado, J. Dean, (2013). Distributed
    Representations of Words and Phrases and their Compositionality, NIPS 2013,3111-3119.
    [15]
    K. Pearson, (1901), On Lines and Planes of Closest Fit to Systems of Points in Space. Philosophical Magazine, vol 2, pp 559-572.
    Description: 碩士
    國立政治大學
    統計學系
    106354027
    Source URI: http://thesis.lib.nccu.edu.tw/record/#G0106354027
    Data Type: thesis
    DOI: 10.6814/NCCU201900115
    Appears in Collections:[統計學系] 學位論文

    Files in This Item:

    File SizeFormat
    402701.pdf2480KbAdobe PDF19View/Open


    All items in 政大典藏 are protected by copyright, with all rights reserved.


    社群 sharing

    著作權政策宣告
    1.本網站之數位內容為國立政治大學所收錄之機構典藏,無償提供學術研究與公眾教育等公益性使用,惟仍請適度,合理使用本網站之內容,以尊重著作權人之權益。商業上之利用,則請先取得著作權人之授權。
    2.本網站之製作,已盡力防止侵害著作權人之權益,如仍發現本網站之數位內容有侵害著作權人權益情事者,請權利人通知本網站維護人員(nccur@nccu.edu.tw),維護人員將立即採取移除該數位著作等補救措施。
    DSpace Software Copyright © 2002-2004  MIT &  Hewlett-Packard  /   Enhanced by   NTU Library IR team Copyright ©   - Feedback