Loading...
|
Please use this identifier to cite or link to this item:
https://nccur.lib.nccu.edu.tw/handle/140.119/83535
|
| Title: | 應用記憶體內運算於多維度多顆粒度資料探勘之研究―以醫療服務創新為例
A Research Into In-memory Computing In Multidimensional, Multi-granularity Data Mining ― With Healthcare Services Innovation |
| Authors: | 朱家棋
Chu, Chia Chi |
| Contributors: | 姜國輝
季延平
Chiang, Kuo Huie
Chi, Yan Ping
朱家棋
Chu, Chia Chi |
| Keywords: | 資料探勘
多維度關聯分析
記憶體內運算
創新醫療服務應用
Data Mining
Multidimensional and Multi-granularity Data Mining Association Analysis
In-Memory Computing
Apriori Algorithm |
| Date: | 2015 |
| Issue Date: | 2016-04-01 10:40:25 (UTC+8) |
| Abstract: | 全球面臨人口老化與人口不斷成長的壓力下，對於醫療服務的需求不斷提升。醫療服務領域中常以資料探勘「關聯規則」分析，挖掘隱藏在龐大的醫學資料庫中的知識(knowledge)，以支援臨床決策或創新醫療服務。隨著醫療服務與應用推陳出新(如，電子健康紀錄或行動醫療等)，與醫療機構因應政府政策需長期保存大量病患資料，讓醫療領域面臨如何有效的處理巨量資料。
然而傳統的關聯規則演算法，其效能上受到相當大的限制。因此，許多研究提出將關聯規則演算法，在分散式環境中，以Hadoop MapReduce框架實現平行化處理巨量資料運算。其相較於單節點 (single-node) 的運算速度確實有大幅提升。但實際上，MapReduce並不適用於需要密集迭帶運算的關聯規則演算法。
本研究藉由Spark記憶體內運算框架，在分散式叢集上實現平行化挖掘多維度多顆粒度挖掘關聯規則，實驗結果可以歸納出下列三點。第一點，當資料規模小時，由於平行化將資料流程分為Map與Reduce處理，因此在小規模資料處理上沒有太大的效益。第二點，當資料規模大時，平行化策略模式與單機版有明顯大幅度差異，整體運行時間相差100倍之多；然而當項目個數大於1萬個時，單機版因記憶體不足而無法運行，但平行化策略依舊可以運行。第三點，整體而言Spark雖然在小規模處理上略慢於單機版的速度，但其運行時間仍小於Hadoop的4倍。大規模處理速度上Spark依舊優於Hadoop版本。因此，在處理大規模資料時，就運算效能與擴充彈性而言，Spark都為最佳化解決方案。
Under the population aging and population growth and rising demand for Healthcare. Healthcare is facing a big issue how to effectively deal with huge amounts of data. Cased by new healthcare services or applications (such as electronic health records or health care, etc), and also medical institutions in accordance with government policy for long-term preservation of a large number of patient data.
But the traditional algorithms for mining association rules, subject to considerable restrictions on their effectiveness. Therefore, many studies suggest that the association rules algorithm in a distributed computing, such as Hadoop MapReduce framework implements parallel to process huge amounts of data operations. But in fact, MapReduce does not apply to require intensive iterative computation algorithm of association rules.
Studied in this Spark in-memory computing framework, implemented on a distributed cluster parallel mining association rules mining multidimensional granularity, the experimental results can be summed up in the following three points. 1th, when data is small, due to the parallel data flow consists of Map and Reduce, so not much in the small-scale processing of benefits. 2nd, when the data size is large, parallel strategy models and stand-alone obviously significant differences overall running time is 100 times as much when the item number is greater than 10,000, however, stand-alone version cannot run due to insufficient memory, but parallel strategies can still run. 3rd, overall Spark though somewhat slower than the single version in small scale processing speed, but the running time is less than 4 times times the Hadoop. Massive processing speed Spark is still superior to the Hadoop version. Therefore, when working with large data, operational efficiency and expansion elasticity, Spark for optimum solutions. |
| Reference: | [1] Agrawal, Rakesh, and Ramakrishnan Srikant. "Fast algorithms for mining association rules." Proc. 20th int. conf. very large data bases, VLDB. Vol. 1215. 1994.
[2] Agrawal, Rakesh, Tomasz Imieliński, and Arun Swami. "Mining association rules between sets of items in large databases." ACM SIGMOD Record. Vol. 22. No. 2. ACM, 1993.
[3] Apache Hadoop, https://hadoop.apache.org/,2015
[4] Chae, Young Moon, et al. "Data mining approach to policy analysis in a health insurance domain." International journal of medical informatics 62.2 (2001): 103-111.
[5] Cheung, David W., et al. "A fast distributed algorithm for mining association rules." Parallel and Distributed Information Systems, 1996., Fourth International Conference on. IEEE, 1996.
[6] Chiang, Johannes, and Chia-Chi Wu. "Mining multi-dimensional association rules in multiple database segmentation." Proceedings of International Conference on Information Management. 2005.
[7] Chuang, Kun-Ta, Ming-Syan Chen, and Wen-Chieh Yang. "Progressive sampling for association rules based on sampling error estimation." Advances in Knowledge Discovery and Data Mining. Springer Berlin Heidelberg, 2005. 505-515.
[8] Das, Amitabha, Wee-Keong Ng, and Yew-Kwong Woon. "Rapid association rule mining." Proceedings of the tenth international conference on Information and knowledge management. ACM, 2001.
[9] Dean, Jeffrey, and Sanjay Ghemawat. "MapReduce: simplified data processing on large clusters." Communications of the ACM 51.1 (2008): 107-113.
[10] Do, Tien Dung, Siu Cheung Hui, and Alvis Fong. "Mining frequent itemsets with category-based constraints." Discovery Science. Springer Berlin Heidelberg, 2003.
[11] Ghemawat, Sanjay, Howard Gobioff, and Shun-Tak Leung. "The Google file system." ACM SIGOPS operating systems review. Vol. 37. No. 5. ACM, 2003.
[12] Han, Jiawei, Jian Pei, and Yiwen Yin. "Mining frequent patterns without candidate generation." ACM SIGMOD Record. Vol. 29. No. 2. ACM, 2000.
[13] Khare, Neelu, Neeru Adlakha, and K. R. Pardasani. "An Algorithm for Mining Multidimensional Association Rules using Boolean Matrix." Recent Trends in Information, Telecommunication and Computing (ITC), 2010 International Conference on. IEEE, 2010.
[14] Lent, Brian, Arun Swami, and Jennifer Widom. "Clustering association rules." Data Engineering, 1997. Proceedings. 13th International Conference on. IEEE, 1997.
[15] Li, Lingjuan, and Min Zhang. "The strategy of mining association rule based on cloud computing." Business Computing and Global Informatization (BCGIN), 2011 International Conference on. IEEE, 2011.
[16] Li, Ning, et al. "Parallel implementation of apriori algorithm based on MapReduce." Software Engineering, Artificial Intelligence, Networking and Parallel & Distributed Computing (SNPD), 2012 13th ACIS International Conference on. IEEE, 2012.
[17] Lin, Ming-Yen, Pei-Yu Lee, and Sue-Chen Hsueh. "Apriori-based frequent itemset mining algorithms on MapReduce." Proceedings of the 6th international conference on ubiquitous information management and communication. ACM, 2012.
[18] Qiu, Hongjian, et al. "YAFIM: A Parallel Frequent Itemset Mining Algorithm with Spark." Parallel & Distributed Processing Symposium Workshops (IPDPSW), 2014 IEEE International. IEEE, 2014.
[19] Schuster, Assaf, and Ran Wolff. "Communication-efficient distributed mining of association rules." Data Mining and Knowledge Discovery 8.2 (2004): 171-196.
[20] Srikant, Ramakrishnan, and Rakesh Agrawal. "Mining quantitative association rules in large relational tables." ACM SIGMOD Record. Vol. 25. No. 2. ACM, 1996.
[21] Wojciechowski, Marek, and Maciej Zakrzewicz. "Dataset filtering techniques in constraint-based frequent pattern mining." Pattern detection and discovery. Springer Berlin Heidelberg, 2002. 77-91.
[22] Yang, Xin Yue, Zhen Liu, and Yan Fu. "MapReduce as a programming model for association rules algorithm on Hadoop." Information Sciences and Interaction Sciences (ICIS), 2010 3rd International Conference on. IEEE, 2010.
[23] Zaharia, Matei, et al. "Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing." Proceedings of the 9th USENIX conference on Networked Systems Design and Implementation. USENIX Association, 2012.
[24] “Frequent Itemset Mining Dataset Repository”, http://fimi.ua.ac.be/data/ |
| Description: | 碩士
國立政治大學
資訊管理學系
102356020 |
| Source URI: | http://thesis.lib.nccu.edu.tw/record/#G1023560202 |
| Data Type: | thesis |
| Appears in Collections: | [Department of MIS] Theses
|
Files in This Item:
| File |
Description |
Size | Format | |
|---|
| 020201.pdf | | 6432Kb | Adobe PDF2 | 127 | View/Open |
|
All items in 政大典藏 are protected by copyright, with all rights reserved.
|
