政大機構典藏-National Chengchi University Institutional Repository(NCCUR):Item 140.119/131480
English  |  正體中文  |  简体中文  |  Post-Print筆數 : 27 |  Items with full text/Total items : 109952/140887 (78%)
Visitors : 46280748      Online Users : 1222
RC Version 6.0 © Powered By DSPACE, MIT. Enhanced by NTU Library IR team.
Scope Tips:
  • please add "double quotation mark" for query phrases to get precise results
  • please goto advance search for comprehansive author search
    •  Adv. Search
    Home ‧ Login ‧ Upload ‧ Help ‧ About ‧ Administer Goto mobile version


    Please use this identifier to cite or link to this item: https://nccur.lib.nccu.edu.tw/handle/140.119/131480


    Title: 中文關鍵詞偵測的探討
    A Study of Chinese Keyword Extraction
    Authors: 林晏辰
    Lin, Yen-Chen
    Contributors: 余清祥
    鄭文惠
    Yue, Ching-Syang
    Cheng, Wen-Huei
    林晏辰
    Lin, Yen-Chen
    Keywords: 文字探勘
    關鍵詞偵測
    空間統計
    非監督學習
    交叉驗證
    Text Mining
    Keyword Extraction
    Spatial Statistics
    Unsupervised Learning
    Cross-Validation
    Date: 2020
    Issue Date: 2020-09-02 11:43:39 (UTC+8)
    Abstract: 在資訊爆炸的大數據時代，需要快速有效率獲取關鍵訊息，透過搜尋引擎之類的資料檢索，配合適當字詞（或關鍵詞）可準確及有效率地找出目標。然而定義關鍵詞取決於搜尋者的目標需求，也與各文章所屬領域有關，多半仍需仰賴專家意見決定哪些詞彙是關鍵詞。近年也有不少研究聚焦於關鍵詞的數位分析，藉由相關領域專家的標示與協助，以監督學習的方式決定關鍵詞，但成效仍有待加強，猜測與沒有挑選適當的解釋變數有關。本文以量化模型決定關鍵詞為研究目標，尤其著重於測試哪些變數與關鍵詞關聯性較高；此外，本文也將提出一種非監督學習的關鍵詞挑選方法，不需依賴專家意見。
    本文以關鍵詞偵測為研究主題，依據詞頻及文本頻率（TF-IDF，Term Frequency Inverse Document Frequency）、卡方檢定值（Chi-Square Test）、RAKE（Rapid Automatic Keyword Extraction）、吉尼係數（Gini Index）等測量值，比較監督學習、非監督學習模型的關鍵詞偵測效果。實證分析為《新青年》、《人民日報》、《自由時報》、《蘋果日報》等屬於白話文的文本，先由人文學者標記關鍵詞（視為標準答案），再以交叉驗證比較各監督學習模型的優劣。其中，監督學習包括常見的統計學習、機器學習模型，包括：羅吉士迴歸、分類樹、隨機森林、類神經網路、支持向量機、極限梯度提升。本文也考量在沒有專家意見輔助下，結合空間分析、逐次分析等統計方法，嘗試建立非監督學習的關鍵詞偵測方法，再與上述監督學習模型比較詞偵測準確性。研究結果顯示適當變數在偵測關鍵詞扮演重要角色，而成效評估的部分由監督學習模型結果較佳，F1值高出非監督學習模型約20%，而非監督學習模型的F1值也能達到45%左右的水準。
    It becomes an essential skill to effectively search and obtain important information in the era of big data. With suitable words and phrases (or keywords), the search engines can help us to acquire most relevant information. However, there are no specific rules yet for defining keywords, depending on the study goal and the nature of texts, and people usually rely on expert’s opinions to determine the keywords. Even with the expert’s feedback, the accuracy of applying quantitative models for keyword extraction is usually lower than 50%. We think the key reason is lacking appropriate explanatory variables. In this study, we should explore if including more variables can increase the accuracy. Also, we will propose an unsupervised learning method, based on the ideas from spatial analysis and sequential analysis, to detect keywords.
    For the variables of keyword extraction, we choose Chi-Square Test, RAKE(Rapid Automatic Keyword Extraction), and Gini’s Index, in addition to the term frequency and text frequency (TF-IDF, Term Frequency Inverse Document Frequency) which is often used in the past studies. Both statistical and machine learning models are used in this study, including logistic regression, classification tree, Random Forest, Neural Network, Support Vector Machine, and Extreme Gradient Boosting. We choose articles from “New Youth Magazine”, “People’s Daily”, “The Liberty Times”, and “Apple Daily” in empirical study, and all texts belong to modern Chinese writing style. First, humanity scholars mark keywords (treated as true answers) and then we use cross-validation to evaluate the model performance. We found that adding more variables can increase the accuracy of keyword extraction and the supervised learning models have higher accuracy. Still, the proposed unsupervised learning model can still achieve about 45% of accuracy.
    Reference: ㄧ、中文文獻
    1. 何立行、余清祥、鄭文惠（2014），「從文言到白話：《新青年》雜誌語言變化研究」，東亞觀念史期刊，第七期，頁427-454。
    2. 余清祥、葉昱廷（2020）。「以文字探勘技術分析臺灣四大報文字風格」，《數位典藏與數位人文》，第六卷。
    3. 吳冠輝（2019），「基於兩詞彙的序列關係建造非監督式SeqWORDS斷詞方法」，國立政治大學統計學研究所碩士論文。
    4. 許承恩（2019），「關鍵詞偵測方法的比較與應用」，國立政治大學統計學 系碩士論文。
    5. 黃于珊（2017），「文字探勘在總體經濟上之應用-以美國聯準會會議紀錄為例」，國立政治大學金融學系研究所碩士論文。
    6. 黃培軒（2017），「關鍵詞與階乘式詞彙文本分群之應用」，國立政治大學統計學系碩士論文。
    7. 黃臆榤（2018），「結合語意關鍵詞與卷積神經網路之文本分類研究」，國立清華大學資訊工程學系碩士論文。
    8. 謝博行（2013），「局部最長連續共同子序列與新詞組收集」，國立清華大學統計學研究所碩士論文。

    二、 英文文獻
    1. Chengzhi, Z. & Qingguo, Z. (2008). “Automatic Chinese Extraction Based on KNN for Implicit Subject Extraction. “International Symposium on Knowledge Acquisition and Modeling, Wuhan, China, ISBN 978-0-7695-3488-6, 689-692.
    2. Herrera, J.P. & Pury, P.A. (2008). “Statistical Keyword Detection in Literary Corpora. “ The European Physical Journal.
    3. Lu, B. & Huang, C.N. & Li, M. & Liang, W. (2009). “Extracting Keyphrases from Chinese News Articles Using TextRank and Query Log Knowledge.” In Proceedings of PACLIC 2009, 733-740.
    4. Mikhailova, E.G. & Sandul, M. (2018). “Keyword Extraction from Single Russian Document.” CEUR Workshop Proceedings, 2135:30-36.
    5. Sharan, A. & Siddiqi, S. (2015). “Keyword and Keyphrase Extraction Techniques: A Literature Review.” International Journal of Computer Applications, 109(2):18-23. doi: 10.5120/19161-0607.
    6. Tarau, P. & Mihalcea, R. (2004). “TextRank - Bringing Order into Texts.” In Proceedings of the Conference on Empirical Methods in Natural Language Processing. EMNLP 2004, Barcelona, Spain.
    7. Willyan, D.A. & Leandro, N.D.C. (2014). “A Keyword Extraction Method from Twitter Messages Represented as Graphs.” Applied Mathematics and Computation, 240: 308-325.
    Description: 碩士
    國立政治大學
    統計學系
    107354026
    Source URI: http://thesis.lib.nccu.edu.tw/record/#G0107354026
    Data Type: thesis
    DOI: 10.6814/NCCU202001387
    Appears in Collections:[Department of Statistics] Theses

    Files in This Item:

    File Description SizeFormat
    402601.pdf2773KbAdobe PDF20View/Open


    All items in 政大典藏 are protected by copyright, with all rights reserved.


    社群 sharing

    著作權政策宣告 Copyright Announcement
    1.本網站之數位內容為國立政治大學所收錄之機構典藏，無償提供學術研究與公眾教育等公益性使用，惟仍請適度，合理使用本網站之內容，以尊重著作權人之權益。商業上之利用，則請先取得著作權人之授權。
    The digital content of this website is part of National Chengchi University Institutional Repository. It provides free access to academic research and public education for non-commercial use. Please utilize it in a proper and reasonable manner and respect the rights of copyright owners. For commercial use, please obtain authorization from the copyright owner in advance.

    2.本網站之製作，已盡力防止侵害著作權人之權益，如仍發現本網站之數位內容有侵害著作權人權益情事者，請權利人通知本網站維護人員(nccur@nccu.edu.tw)，維護人員將立即採取移除該數位著作等補救措施。
    NCCU Institutional Repository is made to protect the interests of copyright owners. If you believe that any material on the website infringes copyright, please contact our staff(nccur@nccu.edu.tw). We will remove the work from the repository and investigate your claim.
    DSpace Software Copyright © 2002-2004  MIT &  Hewlett-Packard  /   Enhanced by   NTU Library IR team Copyright ©   - Feedback