ࡱ> [ +bjbj 8Jΐΐ# AAAAAUUU84LUN 000LNNNNNNNNNNNNN$PWSVrNA0"000rNAA[Nvvv0vAALNv0LNvvEDI`8ܦUF&8NN0NGfSSLII&SAI00v00000rNrNv000N0000S000000000 : Performance Measures for Classifiers Objective To construct a confusion matrix, compute the precision, recall, and f1 scores for a classifier, and to construct a precision/recall chart in R to compare the relative strengths and weaknesses of different classifiers. Background Classifiers can be developed using many techniques, such as neural networks, logistic regression, heuristics, decision trees, and Bayesian methods. Because classifiers can be developed in so many ways, and using so many different combinations of parameters and architectures, it can be useful to compare the performance of classifiers to see which one works better in practice. True positives (TP) - We predicted that a certain outcome would be true, and indeed it was False positives (FP) - We predicted that an outcome would be true, but it was really false, giving us an unexpected result or a false detection. This is the equivalent of Type I (alpha) errors in statistics (e.g. convicting an innocent person, or a pregnancy test that's positive when in fact you're not pregnant) False negatives (FN) - We predicted that an outcome would be false, but it really turned out to be true, giving us a missing result or a non-detection. This is the equivalent of Type II (beta) errors in statistics (e.g. not convicting a guilty person; a pregnancy test that says you're not pregnant, when in fact you are). True negatives (TN) - We predicted that a certain outcome would be false, and indeed it was. The first step before you start computing parameters or plotting them to see which classifier is the best is to develop a confusion matrix (Figure X.1). This is a contingency table containing two categorical variables: the actual state of the item classified (in this case, was it really spam or really not spam), and the predicted state of the item as determined by the classifier (that is, did the classifier think it was spam, or think it was not spam). really is spamreally is not spamspam test is positive91 (true positives - TP)1 (false positives - FP)spam test is negative18 (false negatives - FN)110 (true negatives - TN) Figure X.1: Confusion matrix created using made-up data from spam and not-spam email classifiers. Each cell of the table contains counts of how often the situation was observed. Using the counts of each of these occurrences from our contingency table, we can compute precision, recall, or the f1 score (a combination of precision and recall): Precision (P) - How many classifications did you get right based upon the number of attempts you made? P = TP / (TP + FP) Recall (R) - How many classifications did you get right compared to the total number of correct classifications you could have made? R = TP / (TP + FN) F1 - This score combines P and R into one measure that can be used to evaluate the overall performance of the classifier, rather than having to evaluate the trade-offs between precision and recall. F1 = (2 x P x R) / (P + R) The precision/recall chart that we will produce will have f1 contour lines that can be used as a guide to determine which classifiers are better or worse than one another. Data Format This example doesn't use the most elegant code, but it works, and you can use it to construct your own precision/recall charts. (A few lines of code come from http://www.dfki.uni-kl.de/~grimnes/2009/06/fmeasure/fmeasurePlot.R.) # set number of classifiers num.cl <- 4 # our first classifier tp[1] <- 290 fp[1] <- 210 tn[1] <- 442 fn[1] <- 58 # our second classifier tp[2] <- 471 fp[2] <- 29 tn[2] <- 409 fn[2] <- 91 # our third classifier tp[3] <- 120 fp[3] <- 380 tn[3] <- 110 fn[3] <- 390 # our fourth classifier tp[4] <- 388 fp[4] <- 112 tn[4] <- 499 fn[4] <- 1 Code and Results Next, load these functions into R by cutting and pasting them into the console. They are used to compute precision, recall, and f1 for any combination of counts. Even though they do not require counts from all four cells of the confusion matrix, I've written the functions to take all four inputs just so we can keep track of the full dataset going in: precision <- function(tp,fp,fn,tn) { x <- tp/(tp+fp) return(x) } recall <- function(tp,fp,fn,tn) { x <- tp/(tp+fn) return(x) } f1 <- function(precision,recall) { x <- (2 * precision * recall) / (precision + recall) return(x) } Now, we can initialize three arrays to store the computed values for precision, recall, and f1, which will ultimately go into the plot. Once they are initialized, we cycle through the number of classifiers we are evaluating to perform the computations and create a precision/recall data frame (pr.df) containing the results: my.precisions <- rep(NA,num.cl) my.recalls <- rep(NA,num.cl) my.f1s <- rep(NA,num.cl) for (n in 1:num.cl) { my.precisions[n] <- precision(tp[n],fp[n],fn[n],tn[n]) my.recalls[n] <- recall(tp[n],fp[n],fn[n],tn[n]) my.f1s[n] <- f1(my.precisions[n],my.recalls[n]) } pr.df <- cbind(my.precisions,my.recalls) Next, we create the canvas for our precision/recall plot: # this sets up the graph curve(1000+1000*x, xlim=c(0,1), ylim=c(0.0,1),xlab="Precision",ylab="Recall") # this draws the f1 contour curves for(f in seq(0.1,0.9,0.1)) { curve(f*x/(2*x-f), f/2,1.1, add=TRUE, n=1000, col="lightgrey") text(1,f/(2-f), label=sprintf("f=%.1f",f), pos=1, col="grey") } The following code will plot one point for each classifier on the chart, using a different plotting character (pch) for each point, starting with the character whose ID is 21. points(pr.df, bg=rainbow(num.cl), col="black", pch=21:(21+num.cl), cex=2) It is useful to be able to label the points on the precision/recall plot so you can tell which classifier each point is referring to. Labeling the points in a straightforward way requires that we install the calibrate package from Packages -> Install package(s) and then load it into memory so we can use the textxy command to label the plot: library(calibrate) for (n in 1:nrow(pr.df)) { textxy(my.precisions[n],my.recalls[n],n,cx=1.5) } Instead of just plotting the classifier number on the plot, we can also plot longer descriptions, as long as we set them up first. (Before you do this, be sure to use the code above to re-draw the canvas, otherwise your descriptions will be plotted on top of the numbers you plotted previously.) The cx argument increases or decreases the size of the labels. The plot is shown in Figure X.2 below. #or you can give them descriptions too: description <- rep(NA,num.cl) description[1] <- "NN1" description[2] <- "NN2" description[3] <- "LOG REG" description[4] <- "BAYES" for (n in 1:nrow(pr.df)) { textxy(my.precisions[n],my.recalls[n],description[n],cx=1) }  Figure X.2: Precision/recall chart showing that BAYES and NN2 are comparable. Conclusions With the examples you have just completed, you should be able to enter counts of true positives, false positives, false negatives, and true negatives, and be able to plot precision/recall charts and compare alternatives to see which classifier is best. The classifiers that fall in higher F1 regimes are typically the higher quality classifiers, but sometimes you will need to qualitatively assess whether precision (sensitivity) or recall (specificity) are more important to you in the context of your problem. In addition, any of the measures that can be derived from the confusion matrix (such as  HYPERLINK "http://www2.cs.uregina.ca/~dbd/cs831/notes/confusion_matrix/confusion_matrix.html" http://www2.cs.uregina.ca/~dbd/cs831/notes/confusion_matrix/confusion_matrix.html) can be used as the basis for improvement activities. Other Resources:  HYPERLINK "http://en.wikipedia.org/wiki/Confusion_matrix" http://en.wikipedia.org/wiki/Confusion_matrix  HYPERLINK "http://en.wikipedia.org/wiki/Precision_and_recall" http://en.wikipedia.org/wiki/Precision_and_recall - includes information about the f1 measure  HYPERLINK "http://en.wikipedia.org/wiki/Sensitivity_and_specificity" http://en.wikipedia.org/wiki/Sensitivity_and_specificity  HYPERLINK "http://en.wikipedia.org/wiki/F1_score" http://en.wikipedia.org/wiki/F1_score  HYPERLINK "http://en.wikipedia.org/wiki/Type_I_and_type_II_errors" http://en.wikipedia.org/wiki/Type_I_and_type_II_errors - the confusion matrix is closely related to Type I (alpha) and Type II (beta) errors in statistics  HYPERLINK "http://www.h5.com/solutions/h5-trec-results" http://www.h5.com/solutions/h5-trec-results - notes for interpreting the precision-recall chart  HYPERLINK "http://uberpython.wordpress.com/2012/01/01/precision-recall-sensitivity-and-specificity/" http://uberpython.wordpress.com/2012/01/01/precision-recall-sensitivity-and-specificity/ - precision/recall vs. sensitivity/specificity (used in medical fields)  HYPERLINK "http://cran.r-project.org/web/packages/rocplus/vignettes/rocplus.pdf" http://cran.r-project.org/web/packages/rocplus/vignettes/rocplus.pdf - ROC and precision/recall package in R     %&'014@P^rs~   4   T f k z $ % 9 < = g h { ~  ÿǖh=jhu~hu~5 hUhjhh3HhE*h|CJ^JhZ: hu~6hu~hu~6hu~ hFQf6hFQfhhih65hihi5h%s>h$*5h%s>h%s>5 h85CJ !jh%s>5CJ UmHnHu2'1  % h $$Ifa$gdI $xa$gdqD $ & Fxa$gdqD $ & Fa$gd$a$gdj$a$gdhm$a$gda ?O*+DEFGMPQRryRmpv!"#&ûûhz>h;,hu~hA5 hA5hAh>M ht$hohoho6hohnho5 ho5 h>5 h>h>h>hIh>5 ht$ht$ ht$6h3Hht$;$$Ifa$gdIikd$$IflgF \* t6    44 laytI+/E$$Ifa$gdIikd$$IflgF \* t6    44 laytIEFG?Kzmze$a$gda $ & Fa$gdz> $ & Fa$gdA$a$gd3Hgdoikd($$IflgF \* t6    44 laytI ?JK,./12ij (ŶХжжtБetetetehqDhqDCJOJQJ^JhqDCJOJQJ^J hqDhqDCJOJPJQJ^JhqDhB,h hd?hd?5 hd?5hnCJOJQJ^Jh[ch[cCJOJQJ^Jh,z h[ch[ch[ch=h;,hih65hihi5hz>hA hA5 hz>hA'K/KWXo|,9:R_ly $a$gd[c$a$gd,z0=?at2Ro$a$gda $^a$gd $a$gdn$a$gdd? $a$gd[c =?@ij  /L&9 $^a$gdqD$a$gdqDgdqD ^gdqD$a$gda $a$gd[c  & !!^!s!v!!!"" "!"""#")"+"-"."ٵٵٵٵٍ}qldldhnhd?5 hd?5h(CJOJQJ^Jjhd?CJOJQJU^Jh[cCJOJQJ^Jhd?hd?CJhd?hd?CJOJQJ^Jhd?hd?CJOJQJ^JhqDhqDCJhqDhqDCJOJQJ^Jh[ch[cCJOJQJ^JhqDCJOJQJ^Jhh hqD'9T!@!^!v!!!!!!" "!"#"q"}"}$%%$a$gda $a$gdd? $a$gd[c$a$gdqD $^a$gdd? $^a$gdqD."p"q"|"}""y#|$}$$$$5%6%%%%%%%%% & &:&;&<&=&|&}&&&&&&#'$'\']'^'_'''ɵڱ֧ڗ|v|m|vm|h h CJ h 0Jh jh UhhhhCJ hh0JjhhUh##h65hh_56h1h1hh0JCJh1hhCJjh1hhCJUh\,hhhbhAhhihi5 hQ9R5 hd?5hd?*%<&&^''(8)B*+++ + + + +++++ dgdG $ & Fa$gd?s $ & Fa$gda''''''(6(7(((((())7)8)9)))))A*B*C*****+++++ + + ++++帰h,kjh,kUh?sh:CJh hq`CJ hq`0Jhq`jhq`Uhq`h CJh h h CJjh U h 0J(6&P 1h:pEq0* 4!"#$% $$If!vh55\5*#v#v\#v*:V lg t655\5*/ aytI$$If!vh55\5*#v#v\#v*:V lg t655\5*/ / aytI$$If!vh55\5*#v#v\#v*:V lg t655\5*/ aytIzDd /.vv0  # Abz>U"ZBky0 ny>U"ZBkPNG  IHDR#"7sRGByIDATx^1lG׮9B3&\@H lN8c '6X)*$ fꮙaOOSUO[䩪S93Tu݌  @y.!@ (- "E~!@J @|_@ " @/(-_d @ JY @(-r "E~!@J @|_@ " @/(-_d @ JY @(-r "E~!@J @|_@ " @/(-_d @ JY @(-r "E~!@J @|_@ " @/(-_d @ JY @(-r "E~!@J @|_@ " @/(-_d @ JY @(-r "E~!@J @|_@ " @/(-_d @ JY @(-r "E~!@J @|_@ " @/(-_d @ JY @(-r "E~!@J @|_@ " @/(-_d @ JY @(-r "E~!@J @|_@ " @/(-_d @ JY @(-r "E~!@J @|_@ " @/(-_d @ JY @(-r "E~!@J @|_@ " @/(-_d @ JY @(-r "E~!@J @|_@ " @/(-_d @ JY @;.X jjәjܥ;_.%aDFyؗ@6 8!9a'&[f 2V>~xppjjsjܥ;\.e`D6N!v73=;ᷢ`,| ]LN@흚Cy ,#/Bc}sGicKAdʌgN1WVʌ$@ yQ%IhZbS>~=}uiayW璆m=uRnCB'PZN0  My7#,軆xbQϜ9+s⏺{츹ix{pnJFׯcG@iGr8͖@koO &T 7kūvuެ]h#(ˊw/~ieGiP@OPz8z]_jk:ן֬Yk,W_N޾_,gݶ)q"SZvѻb)z\":fc+%w 6WBJ{CQi]k[_7q{SZ_JV,Jk.hU,Eo5/;@=I%C1BMUvߊR)JZ/fՀY{^HWǡϦ,zkYm^$C@[LB'$hd-êaY_ @}n.~[6H7묵q ˘Z\Ev&Dbr:wv[Cw r" \CyVBj?hkQs/GԳ!17»r[*l>g" N>(gS=xkg @`-h (k*|k+_gǕI 'Vች=Uk [t_mYuࢺ5g" U.V͏.&CC9Hrq+W7o?5k~/ZY߻]PBsكxZzlV2)qPZf{}F bgj-k=a 8vEŞWGޏ?Ze_@.hS5z :F eggVӅZA 84Da ~lY֣i~8Ŋy l6 S4۲|FWg` !Ғ@@@ŢJ=ݡ|4tt^wNy(z-2)F8;-S 6v(@c ^j~N5n >4OcFOJ|D: c6=R XGmNRMS^!WbbWB, IIi d(c (,$-1[7̙K))-{Jۆ74k}b.WbhCCm: t+@ YHƸf*,hټ\֊I&Ҋ!J@2dHe( KV(S"_)ES\H5DĖtUJKU8pCUFذ*t:@(15F ,1&sHPZ+ʥyH~dO Ųe-<C@Jח潜\ 016@8i~ȋfN7K7Mj";" 47'Tu󸼍4%:sfosil2k:ɸ¢w(jZe|͹//^9}+6GNƎLiGr$o:Ǚy8g)@tJ%nrÑ^ P'j2-U θjιZs >}uXK\ U)(Iá @ D9b(h }Zaj5Egʖ'QtҊ"L8 @QX#o#t@$a')MvV_b)٣R&s 0 V'wV'T PZFyA ܤ #k6Wt"J^L 01<li$J@ }J -j{Vy4K*[{.]C} Jžt6Ai ^A`,mc :mO8 N)Ҧ/1(-qV|aQcq7UPZ3@#Ò:V(-=@ 8<` q@|l JKO,H"`t7DiM `Snچ(i3: tF^v=[ލhPZ!(3&!M=^' tt1!Cl7I[$@[cC/|[Vx@QZ3|:Hwb%ն>QZ!i3 MPtcAOV0  'v8TZ;y$bk0ӭ:=viBmWWΧCVp @= P Ljnbw 4+Y :N:;%v(-% %@M/_wo6 4[h^p7=(XGlv˻uqǚR&H&A=ڀ Ь05ńZBL1_xxhPwu]~.u֟^=76(-1@m_"FsE2Iot◎%.צ^g/Zq_(@6o:k̬X3,dVi5z+9N,Vlee䧣i5!XIT 6 _ˡ1K(-yEW@hqG̓ή YΉ5a(nQӚ0:5|QaiF">rV-)nQrTt@;"Fl%>VDJtɧl*PMiz@:qEضtw\2xjf[܊ի;˷?uW'oYrkjO2ȁvêM\n]ts|6~yu~ڶta1 <J2Cdxi!f:Qg2O76Ml},5Xw/Va$'I6$B_bm6-|no)Iד"su6on)6hkmbjZšO*^rw^:gfr3,L7>|H]Ō;HkYeG7fqB'Fn+k-V&; ~PZW_ yb ˈcpoYkaQUJ!H=x;6SBkwZ 慳ݕ-` ~PZ/̷ţ[|ԣ d \ry+HͦuZ.+Ŭl^]hKBiĻUoZwq%3&;GCāPܺ'l*J`]ZC(no[>CBibWW\̖r/OG1 r vǻUV;j׻5{O#"܆qoQ(rŋW-/Eh9Nq|SHà ”t󠨝7>,Y~šfFk b?̥= 8%[ SOkX bYyi Y}װEiYVo5]H,9B@*XCty=a}}a8 >(>,u"q) 1,2 p' HLh لM+$즫cv{9)MEt+J+%o(-oh2#`mI#"_^UY ͛Z ?5?2vjWp(@vGZ6H#u[͓fM*_BR5S{j1/@XC"zFiE2@Xd5c%Q)q+Vfz6@ly{E((„D n(!;mJ;n2'C q?-XI,(T"< Dp#'gbkh C~F+$I Y(-m@;$EbxOr[a8+$@@V0P Y((- Q+ gwK2gsy1VӎҚ?C@#Tƨ@JġnCb9J+$mƂ@Pn>Tm`!Ci},p:$pP N'Jk @ &IEqkEi9GJ pD-YĎ`w8(J!L$CJlXLw(-߄bK$Ė?3J+^:@qσ˕,}ӥ-&MF(-M@`:J9{XY僪>QZ3 ܓȃT A~ȲS~ZQZrVXB@̞-9+-xtrN , a6#@<Jk0:B KH%ƌa8՗@2,㍉(O#-#!Jk!@`b+`=mPZny ,Z#C˹#nr$@@2d&2 es\M`d~[93@Azaz#@IIٶcN[Àq&:&r2t7Vo- yV'" @hJȻ"9I?QZBPi'O{|sOm9+V'sV'" @J28 FWkJ+S@5UZ2. 7+\@l"9a?ljҚ0'=cۖ[F 5a*2tR741zmٶ焽J\z2(i3: 7}ďDT'3S!PZ^"CSH$A 6 ʝF{3[\n~lNְ2;a?,QZ-2%l y3qq)[ךl?x\j1PZ#p/m+&YLbTxr%]˳iwZ(ێBV5`Fiy=:@ZWI> ,&~û/+zv!*]5Wb >owwQ 2@  MΒ),XLtI}ٚUᙕIl6̦NC4Bi8||`1[ *7^ ޕ+׆}]&ػcVod4(kmf?b"8l +xX>zhW>Ai gGK⺻ΈbKCؼ: }ZC\{p6[GIfZ+;咢Lj- s# eUgO_]l/rt4 &.=Fly;aPZα[_rGz"y|q;vqśq(Z#æ/oϻ6s*<ϊ@5v9p[apEiXt!  aap[a*슍 R\J+"  !a1o&YVe"b|CiE&@^xa)n9+VՆ8|˚@=-HvR 3يg'OqU}KOhPZzb' '-Hw,q-c>d("L5De叭枩rMT -߄"U.Bi&L(FkoJ'tbHf  !y]5!Iӧ ŭ>eakhQZ^9 0@Zs,>A{[HY4V(*=Jk/:{pv@ PrtD5M05!TZ+.=:0p}Bxw 9d}H["? t*͊ﯬ*uc]Cƈ $vNl: 2a $HSi=dfVqC~8;;.EQ,,tHbKgl[:W t* _^?ZcDh&ؐQYb6q60Cݧ<8,ޮ8A;(wOjތ@>nym)xP}|z{po>|w4 Q}RZP 2TW?F|ZȬl{N{k`ӽ2Ӎm\f9%~ۻRayi&Tm¿'ZSd 6INjV r_oԷٳLeYUȳe%V{Դ*ߟ`ΟI S1uOýlLM y-tȬO ;<6kqY'tr2VJk~9Ix Zk10 d$ $$"7L)ʮ~Ůͭ1jymZ A:=##)ķ/;̀j p?S#9ʁ#<O̚\-b^Iw]ͷB>@gfJm}"D@L&9)YeAq_uge?gskDܩ~wd9A =J(q9jH Ze.- իYˊ@l %%.aԉrlwWNΚ1\_Aky[0)tX@^EJM1ntB{p[*"bFc@ V/V4m%E;Eg?{ߛVȖF>|U{Z#iivtIP2  ·iN2:gN<@tI=XAf} VjԒ2puzni6o$\Hxj h؆qgwwqX,R9@/zSܒӲZ2v.6ue =<9*?M6zDEUVK-9-TWX~?<& ŗ?I̦$8)V>f_MwTW-fxFyP~RV==y1;5dG7"W34pʝLp:svXAAěs@흚V^B 51`17g_WQ潇8UԴD2ʰƓᔻ)n"qԺN}X%_Q#Yw傯ߣ|;tj(-TmPZu9w/&9ީQZD2SiY*9Ͻ+/+JxSDPCi'0Vpd;qARx4AoyK2e+SDJCi'0Yp@K%;Ci'6\md>}qx7DoyG (\mT{FiMmPZ2řph(zSDY6~(-QF+- | a A1<ީQZR?(~b#DbK,A [Aq3;5JKPjůb˖MT>YԖ( n ".F~SD 6~VP(-Z/8k:N:?:>;|yP?#[>y#CK%+Ci39% 3~ԵPZeJ #%%j}:RŨ4:i0@oi>LH@s_r}C%`v4C#0'L꒽Q '0ojZnt[Jk6;z}ww򰘵ْeʃW)*?t_r:X?d 2*@[e%jln[orCg7>RRZWE#>!P!J/ȕ^L(KTqUjLGygvKIk&Ϟ<,Y_-b:l03:] @l%eCPIkY7F4ſBR8]_n}-uER}ZS潞=4z<E() 7 Y'ꃊJvw ZXG/ՍZe{ީQZմON*=y{Դ,-(1A7%geMiRpsY\h5,UXKjԱ7/Ã&HʖG7nP)t)XTn~|buyơO2˼\g$r'\"ύV-%IB5Sizz}w$ꡨ;~E|_^nZ4/B,ٛsWgkfGiL`{8 ˞u߭]5ޅNQ(G?u-. dK՗XԶ{[_Yu{smVj 1>GyEJ]:S.1rAJ:j%PFcEU1 Z_?,τTMhf;cJ~_n4VB8D94 TO%yQ maV*g(K<vcqLW3gqԕb+}`ӵTET{Cif;&-6&/AJf"c>Bo00ULz瘮#ީcQZ6/5vCi>B.aNqk <ۢYSǥwDmPZnܧVZ sn9Gsj(jϩJkj/b+8r丑D;5JKP?(~S!ޡXR<6L%vCi71&K KJSDI6~)-p>Ş|f*18 RMXp$ Ez;5JKjo:#tG $0RSDI6~(-Q3b q:=FFr)#0RM8JKD%$3ʰ̓e ɥ.$81w! m[v疑\='΂A07on%[6jHICQ\$7xyJodR׈UEu!!P{EB *[X;*StP[ReζExr+<ͅ;55\Ry!@eKO,{^"ai@i_f6u Ke+DFH読Iz(J#UōS/qÔ^ E@흚VW=`ډ;5qPECReZ2s*!>IQJ"Lͧwov=b۹be;wjjZ|  3Uccᚔͧ̚Ͷ63^f6}߷ Y&C5-Q*ejZ6;>~)-ķ_4 JRUe<-=|vӯީiy;:O#Yg,tC+X'>#y-qsm>Y.4<~ e(4EiiBK c!~)E5ۯޙ&[޳Bkw*wv7K[+zHz(ښfo9(?dݰȘ`PӘRe+|1!SzsS^ea`eZ4){p`_U{F3,׏R spJ`"#NIYO;=+yfY#ټ,i}BZ.z'c넚Z{MOYk=0},1* P"&бQ!5 +e,[Q[W{5u2> BHrf~PVf;:wk55-Q*ejZiĶ8uA"7Vy٧jw<,U흚V i$I5j/Xp(ؓƫ).t⥦%Z@M~wv[(l> %@k(9- ߧeZ̰W~NMMO ThQ@gы-qV;\'o>r3q[do_׏R8sjZUԴDKˆVaoJ"t|񐚖YշKϽ[yZqXYDwj(%_*F{TB<^QͣÔVPӵK WV;5JK}j㗆2`(K#[rVX# SDI6~(-Q3Bl%'Զl1%wj(A/EYKFh29!Hwj(%_F譤Û_D<ީQZJKԍ[G8}SDT \l7BoHl8}`ѭjԜ\6Spz.1zVf˷5 ~7YjK:Ivj <3DլQZ3X!P XΎyjk fe ki?u)ݻD@0Q$?(-l7=ӹ@9y{Wޞ&/hw?aӳFuաQmkҮ::KR40bC1%rj z?秅l:|y]fssv\Z־TYֱ_[t )=s`K#=R} lU=ڦ}VPZY;RֲAz{|/4-/h?7?yb翝vD5 zmdy.?Yޏ?UϗM?|_)[uNνoVCOh'Y3MQcfJ+8e-SR|hC jQ4\/_P_fjh &rIobg;S,Vʌl=M/j57pz~Kg=\-HD?DWbJ+7*Mu'\s5VU]fzD@/;@˳. ^md^ /GOYwwSG% 5N+ַF˴.z@&5̸,m~fAi, h(kYUZlo8 6u[Atü?}g&}>$ulZe֒Kyh\9=ԜxفUlTv0?Oۤ}?n3gݏƺԙ@@NK6~Vt`BdՕVZ c]GգWvP-Qg_Ի_Oo>h>ҪLaK#@^,[>6&1odsvx=h?'`_t4o7+_PM}}\_v|Ugnv)y@ 0wji4@ZRݛvܳUy.J%|gi`zovW4 h@`z((ba\-#j=_iPI@%>LdAYYv2- `)[cқj(-Q|O}"qc) YJD\cқj(-Q|Oo"aY Y8>F@% IQ$;12 `$̧+@@B@% ِ1(ED͵[15 ÙQ 9wDȔR? OE2sOs.Y=c s@@J@% IN$C;7!b 5"yzqKBwj(AO?u"ݹ[,wEucIOͧwȻ۹qErj#wj(O?i"݇I2G\>Q]1|'OlKF/正PZǼzAo$1&*}_;*y)@p!4eUr FWEYh,S*`VϞ]t#ruZd;Q$JQ/.yom=}u5=@<==b 6 C loק͵Y\hy)]_ VE>NCiQź*޹{b ޯ7O7-dy:Uf]w?-v]meNU,m스VvE8(%ԣww/grab <; /Ymhi0܍j]YMXGJPjGliHI}z7l!&dZ.1K 'l*J`]ZCgŤŽDevi|. Oϊ_ ڗ#רx=r{bF(j94֎u('@}ujve5.6meĕ-I*۴W$;"^bZw>}~Xvwwz EMkY2ak5lai5?@Gy0M~e3FNt@;ZMk8z+[PZ+ETӊEi`kˈs;!@[6UDWJ˞^_'l_T7ʷ2È֔y6~B(L) :f |Hߧ5Ϊ$PZ7QZcl I H QZUyKbu6cd04CiEQ{cY'e6L iRZU;DDiMK{23Ϫ؃봜"l 0U(-Wr@eTċ4VnRDJaFP[aG(j'4Mi)}H$XH)cwj(j'>Wؒ bAp;5JKj'>cؒgH1 qIީ#SZGh j' WpXI h#JP{EiK[j'O̕-y` 䂌RH4P{Ci-t!%0j'J˲(%V[ g'JkLK %XEmzMU^Ƀ1 0L-'N!MySGQfCREmzMmQEqW` jr_SGfMX٧՝(FFpD\ $>{I@mzM Նkǟ2 c@`Znu[u.1~SQZ]oXƯzo:a$I*ީRZ%$bb'" 'C,šDقaJP0  wj(O}ňV/b'…1 SkB( L!֢QV>}܃ iU"ibϒ#t@=wj(j'F{3@nSDV?Flm z F: ;5JKbyI%F! ީQZ{L 3 gީQZ{L# F) ީQZ{L.TyrY ީQZP?kFlmL+Tz I@@mDޣaӪv*.?Iӻv݃gwc~XpS wj(O=Jk u G5~Ƀm`V2Т:;5JKմ̜Y@>+,oFu&QI~jZ{ݳy9kb{ViC'yF'p@f^MAt@t~smHZ.4<~Д®oZ6댻-nog)liU"i F7I[k@0( B'];,Ս{j ީQZOGi Ƥ2 .sS^UW-QCW;5JKP?-F-"li xu?jUL%,{LѶm^'C@-9"i xa2)kőD~cIO%Zi-מFT{%v#H4oS` 17W+Y)SsʃܣcCzDmg~x h"]mz{8~^+m'z toFMKTRyOMk$&P F@ Wg ;yh|0>8?T%,F, gH60ۄ)HO`ҲR3NunoZT{Fi\mDw!{/$R A=B%AJk!VYyz὇2Қ?gN@ɨe*L ÔVWSo8G~1w-SS2ED,fSh SD6~"F-$L%0@-Y,ٴ`bP{FiRImD PZH@VFzA`$wj(j'^fؒq ^Rl7/SD6~"F-1* !m}T"H<\T{FiPmDދPZbTBsaH=rP{FiRTmD1Bl- 01udhA 6j(-QpO}O#VO`CD 98 ;5JK]j'J'0!H@&lZD!b??:SV0 @ 3j,LWD%„ @JP7bKae8h0o@J+ȅ3@@ا%_ ?!@ wjjZ!,#F> i 舭8℗ Iwd@  'NMM}ST˼ @jZ"j{F\?@K@% `lqƩt @ SDT?ތ[1 |z{o>|V2Ͷ>y`{1;5YSb@ s72ˈ4hYwn65lm5%V)bD}+f LJRϋSVMr^EjԴl=y^!@/y-,6մ\hyYn+k,3n?||ZnQ kmwiU"0*-vOƄ 꾩ƩfZ`Jwj(OFHI38 Me{}lZ4U}قC,W -F9\ I`g岟Pޯu6opYݕ>8~a=Pq%$6)-0 @YᢓZL흚R6 ZrW$! 0?a lmԴD*e-^!|jZ+Ǒ)اe履ީQZߨЅ?H@5݂xx;5JKj'>Q 댈Dcδ 2i 8a2\ީQZU?unBD`ҲR3⻏R{Fi2OmDcJSUd )pZxeQVUm8oN}O=FC d賡iDM`*\UOk;WصN(8SjȀ@|ީQZdR?Lh@ Aa!N%= L"zlFtN #wj(O=Fȃ@1e# VnSD6~"1 rHB@_(-QQZ"LA&"N' @ ( !@Jk" @@PZ)B 0D 22S @`"(3, @@ied@DPZgX@2 L &ϰ d@mJ"1 x$pk;¡n}vnܧlV8[J흚00 @ͧ̚Ͷ22Z^f6}5 ~DY)PZSgl@@no)IדۢټճghBh"SZ秦:xO!@ײJ@7fqB'FX2u-s:ҚK6lgDžCoH+ ,wF}[0|J 8 P>8<2x?Xc-$6:ȳ҅y)͇!@5'b3g|*_ͳ+}gbDX=(B A`Aq6DC{1=8U|j(-9@@ ʗί@JiqLΑ8i gGK@jԴf3V~?C  TJ wjjZ$B Vl_@!2G\@S@2!ҚK,Ν츐`LiB :?˜v/F4oFkőxx @Ȃ@Jїw_m^S7,#fL (Q ~*Z\ @@ (VɪXuuڸ@@^PZXuvUkt̖r/O+yB 09XΈ_`f3]=yv\h@H;u5-fûMW_\^dVzȌ @C VUsax@jԱԴt/ @@(> @} !C [@ ЇJ-l!@@(> @} !yZ"ZF LD`>țEi sՂR2`.u0uHJ;`@ dG]ș0 @f @#.L `PZP3 @@ier& @@0(`  9 @ V0 @@vPZم C J+j @ ;03a@ 5A Vv!g # @ȎJ+3a@@iC@ dG]ș0 @f @#.L `PZP3 @@ier& @@0(`  9 @ V0 @@vPZم C J+j @ ;(-!ztkyKDm6fk髫9`ԕK겏#w#f&$8^!Ũ?!Nw\=Y}hٸaÇ+_-W~7wǢ 6QZ.bt|с/[4qi}8t @=YW4SlavwԳ~WXg``PtSEi'0;k~<{,ូIi?f_ŋ}]=ȩ#e#tS!sJsEi gGX IoI,?|+,p׻n*iW^ պ7$"LSL%Far Q}V|sgmW~%J*=غppsڨ>4p rEWv#l5(mY^n촶#~v~JmDM{ ~Ԣa>(-P7o훏7 Oc7WYfk,QG}_fPShqDH?gW;[is9%0 :NJ ȋ>fho;8 h;'Q_T' Ms+-qSg$OYu,X[G]=r?ppiި*1~؃ 9#b~mgW%O' t"1v=u{wkO5@ ˅ߝ ;#l[`&ΎzA:o 6vdEQQ.`ϧwO9@x-HÏʈ @ Դr2s @`(i3* @9@ie@4PZpgT@r ! iΨ @C# @@iMÝQ!@ȁJ+(3G@!Қ;B VQf LC5 wF @ (!@Jk @@PZ9D9B 0 4 r2s @`(i3* @9@ie@4PZpgT@r ! iΨ @C# @@iMÝQ!@ȁJ+(3G@!Қ;B VQf LC5 wF @ (!@Jk @@PZ9D9B 0 4 r2sjWn5_~#KF[! C g[O_]3_f $H`.i1%@ ^Of_35< @M8z}p6;{y Qq HJ+82 $I`ʭRfO)uͯm5.7V~Y}Ug}Z++{Jc3PeI&(JKB @`W_>}sZ(_B,~TH*N-gXm|}ЙQV36͚袷a'ˠ@8(p E*'?Uo4K?+tV?*~P[,6ҟQ0(#/^ѰyfK;heԲ[jle_W` 4҈#@53ejyPXG'oK5E5cg/)t[m~˲ϕQ_fG>DLB5m PUDc<U3+Yd m^/a3g[joþ*J觺[4.ƀT@i NA ?0UWYkP.?/7BfRvMZȃJ+83K$I`B5huFlb\ 冯$2)@?ƌx"`Op5XٖUΰЅrn߹5Dv+"MJ+:'`wGW=j,. ?^%ŕVOxj:AI@`6Ci@^K}mUfv{iã_'0؝VVyVq>ykW1`:[uIň: JE&#`*;ٍ'W6 ݼT/V:{Y%C@309:@@i? @@3 @q@i? @@3 @q@i? @@3 @q@i? @@3 @q@i? @@3 @q@i? @@3 @q@i? @@3 @q@i? @@3 @q@i? @@3 @q@i? @@3 @q@i? @@3 @q@i? @@3 @q@i? @@3 @q@i? @@3 @q@i? @@3 @q򃮮(IENDB`j 666666666vvvvvvvvv666666>6666666666666666666666666666666666666666666666666hH6666666666666666666666666666666666666666666666666666666666666666662 0@P`p2( 0@P`p 0@P`p 0@P`p 0@P`p 0@P`p 0@P`p8XV~ OJPJQJ_HmH nH sH tH J`J $*Normal dCJ_HaJmH sH tH DA`D Default Paragraph FontRi@R 0 Table Normal4 l4a (k ( 0No List 4U@4 60 Hyperlink >*phjj  q Table Grid7:V0e ++s0HTML PreformattedA 2( Px 4 #\'*.25@9dCJOJPJQJ^JaJT!T ++s0HTML Preformatted CharOJPJQJ^J424 G0Header  H$6A6 G0 Header CharCJaJ4 R4 G0Footer  H$6a6 G0 Footer CharCJaJPK![Content_Types].xmlj0Eжr(΢Iw},-j4 wP-t#bΙ{UTU^hd}㨫)*1P' ^W0)T9<l#$yi};~@(Hu* Dנz/0ǰ $ X3aZ,D0j~3߶b~i>3\`?/[G\!-Rk.sԻ..a濭?PK!֧6 _rels/.relsj0 }Q%v/C/}(h"O = C?hv=Ʌ%[xp{۵_Pѣ<1H0ORBdJE4b$q_6LR7`0̞O,En7Lib/SeеPK!kytheme/theme/themeManager.xml M @}w7c(EbˮCAǠҟ7՛K Y, e.|,H,lxɴIsQ}#Ր ֵ+!,^$j=GW)E+& 8PK!Ptheme/theme/theme1.xmlYOo6w toc'vuر-MniP@I}úama[إ4:lЯGRX^6؊>$ !)O^rC$y@/yH*񄴽)޵߻UDb`}"qۋJחX^)I`nEp)liV[]1M<OP6r=zgbIguSebORD۫qu gZo~ٺlAplxpT0+[}`jzAV2Fi@qv֬5\|ʜ̭NleXdsjcs7f W+Ն7`g ȘJj|h(KD- dXiJ؇(x$( :;˹! I_TS 1?E??ZBΪmU/?~xY'y5g&΋/ɋ>GMGeD3Vq%'#q$8K)fw9:ĵ x}rxwr:\TZaG*y8IjbRc|XŻǿI u3KGnD1NIBs RuK>V.EL+M2#'fi ~V vl{u8zH *:(W☕ ~JTe\O*tHGHY}KNP*ݾ˦TѼ9/#A7qZ$*c?qUnwN%Oi4 =3ڗP 1Pm \\9Mؓ2aD];Yt\[x]}Wr|]g- eW )6-rCSj id DЇAΜIqbJ#x꺃 6k#ASh&ʌt(Q%p%m&]caSl=X\P1Mh9MVdDAaVB[݈fJíP|8 քAV^f Hn- "d>znNJ ة>b&2vKyϼD:,AGm\nziÙ.uχYC6OMf3or$5NHT[XF64T,ќM0E)`#5XY`פ;%1U٥m;R>QD DcpU'&LE/pm%]8firS4d 7y\`JnίI R3U~7+׸#m qBiDi*L69mY&iHE=(K&N!V.KeLDĕ{D vEꦚdeNƟe(MN9ߜR6&3(a/DUz<{ˊYȳV)9Z[4^n5!J?Q3eBoCM m<.vpIYfZY_p[=al-Y}Nc͙ŋ4vfavl'SA8|*u{-ߟ0%M07%<ҍPK! ѐ'theme/theme/_rels/themeManager.xml.relsM 0wooӺ&݈Э5 6?$Q ,.aic21h:qm@RN;d`o7gK(M&$R(.1r'JЊT8V"AȻHu}|$b{P8g/]QAsم(#L[PK-![Content_Types].xmlPK-!֧6 +_rels/.relsPK-!kytheme/theme/themeManager.xmlPK-!Ptheme/theme/theme1.xmlPK-! ѐ' theme/theme/_rels/themeManager.xml.relsPK] #J  ."'+ "$EK9%+!#5 :<|#\^6 !8!!!B"""#XXXXXXXXX8@(  P  3 "?B S  ? #to q | ~ !RT_aln%').R]ikmr2?GPR\dm}*.8@EINOg8;|?B JOW]^koy}S\t{###### # # # ####1j&ss!"#qy|##- 1 . b+,^ u%d0rK_ e.gV`f ʆ*]f ""&.d,BVlT@dRFNH5KDn27TTec6KrXJtpR| ?5}d.f,h^`OJQJo(hHh^`OJQJ^Jo(hHohp^p`OJQJo(hHh@ ^@ `OJQJo(hHh^`OJQJ^Jo(hHoh^`OJQJo(hHh^`OJQJo(hHh^`OJQJ^Jo(hHohP^P`OJQJo(hH ^`OJPJQJ^Jo(^`OJQJ^Jo(hHop^p`OJQJo(hH@ ^@ `OJQJo(hH^`OJQJ^Jo(hHo^`OJQJo(hH^`OJQJo(hH^`OJQJ^Jo(hHoP^P`OJQJo(hH^`OJPJQJ^Jo(^`OJQJ^Jo(hHop^p`OJQJo(hH@ ^@ `OJQJo(hH^`OJQJ^Jo(hHo^`OJQJo(hH^`OJQJo(hH^`OJQJ^Jo(hHoP^P`OJQJo(hH^`OJPJQJ^Jo(^`OJQJ^Jo(hHop^p`OJQJo(hH@ ^@ `OJQJo(hH^`OJQJ^Jo(hHo^`OJQJo(hH^`OJQJo(hH^`OJQJ^Jo(hHoP^P`OJQJo(hHh^`OJQJo(hHh^`OJQJ^Jo(hHohp^p`OJQJo(hHh@ ^@ `OJQJo(hHh^`OJQJ^Jo(hHoh^`OJQJo(hHh^`OJQJo(hHh^`OJQJ^Jo(hHohP^P`OJQJo(hHh^`OJQJo(hHh^`OJQJ^Jo(hHohp^p`OJQJo(hHh@ ^@ `OJQJo(hHh^`OJQJ^Jo(hHoh^`OJQJo(hHh^`OJQJo(hHh^`OJQJ^Jo(hHohP^P`OJQJo(hH^`OJPJQJ^Jo(^`OJQJ^Jo(hHop^p`OJQJo(hH@ ^@ `OJQJo(hH^`OJQJ^Jo(hHo^`OJQJo(hH^`OJQJo(hH^`OJQJ^Jo(hHoP^P`OJQJo(hHh^`OJPJQJ^Jo(h^`OJQJ^Jo(hHohp^p`OJQJo(hHh@ ^@ `OJQJo(hHh^`OJQJ^Jo(hHoh^`OJQJo(hHh^`OJQJo(hHh^`OJQJ^Jo(hHohP^P`OJQJo(hHh^`OJQJo(hHh^`OJQJ^Jo(hHohp^p`OJQJo(hHh@ ^@ `OJQJo(hHh^`OJQJ^Jo(hHoh^`OJQJo(hHh^`OJQJo(hHh^`OJQJ^Jo(hHohP^P`OJQJo(hH^`OJPJQJ^Jo(^`OJQJ^Jo(hHop^p`OJQJo(hH@ ^@ `OJQJo(hH^`OJQJ^Jo(hHo^`OJQJo(hH^`OJQJo(hH^`OJQJ^Jo(hHoP^P`OJQJo(hH ^`OJPJQJ^Jo(^`OJQJ^Jo(hHop^p`OJQJo(hH@ ^@ `OJQJo(hH^`OJQJ^Jo(hHo^`OJQJo(hH^`OJQJo(hH^`OJQJ^Jo(hHoP^P`OJQJo(hHh^`OJQJo(hHh^`OJQJ^Jo(hHohp^p`OJQJo(hHh@ ^@ `OJQJo(hHh^`OJQJ^Jo(hHoh^`OJQJo(hHh^`OJQJo(hHh^`OJQJ^Jo(hHohP^P`OJQJo(hH^`OJPJQJ^Jo(^`OJQJ^Jo(hHop^p`OJQJo(hH@ ^@ `OJQJo(hH^`OJQJ^Jo(hHo^`OJQJo(hH^`OJQJo(hH^`OJQJ^Jo(hHoP^P`OJQJo(hH ^`OJPJQJ^Jo(^`OJQJ^Jo(hHop^p`OJQJo(hH@ ^@ `OJQJo(hH^`OJQJ^Jo(hHo^`OJQJo(hH^`OJQJo(hH^`OJQJ^Jo(hHoP^P`OJQJo(hH^`OJPJQJ^Jo(^`OJQJ^Jo(hHop^p`OJQJo(hH@ ^@ `OJQJo(hH^`OJQJ^Jo(hHo^`OJQJo(hH^`OJQJo(hH^`OJQJ^Jo(hHoP^P`OJQJo(hH^`OJPJQJ^Jo(^`OJQJ^Jo(hHop^p`OJQJo(hH@ ^@ `OJQJo(hH^`OJQJ^Jo(hHo^`OJQJo(hH^`OJQJo(hH^`OJQJ^Jo(hHoP^P`OJQJo(hHlT@ecR|. 27T u%K_f "f,XJtH5K?5}&.d,- 1 e`f          nD                 ^.l                          BC2w        7p                 l|        ]                  O        }        7p                 ?Q`C6/?+?@@_AB\B\{BqD{DjbGH3H0JdJ:K/aLb[OWP{PX"QQ9RQ9SX^SWXPZhZ_zO` Z`[i`[cpdseXNeSeFQf_#h,kz m%m@pmnVnXtop8Xpq qEqGqa}q++s ttuhwx yc2z,{s{|W}Sg~u~v~ j4d?~awZ^1/`g#q`-j1J >GB0|xW#K] <Gi4qPW.-FB>Mhm:NQ?sOo} Kdm.H Z6;ubhq>6##IK~ t$UjBcQ>ANLcQ|![8 }Su++RI 'ECbrh&opzaiW4X>*'E~Cv%,z CS Z0az>BQ"u MTz9y=Re=j\M+!kr1##@||||#p@UnknownG*Ax Times New Roman5Symbol3. *Cx Arial7.@ Calibri?= *Cx Courier New;= |i0BatangChe;WingdingsA$BCambria Math"1hM GQ g~9?9?!n20"" 2QHP $P62!xxJIT51Nicole M RadziwilL           Oh+'0`   ( 4@HPXJIT51NormalNicole M Radziwil15Microsoft Office Word@t@,z@iܦ9՜.+,D՜.+,8 hp  Hewlett-Packard?"  TitleL 8@ _PID_HLINKSA6Ehttp://cran.r-project.org/web/packages/rocplus/vignettes/rocplus.pdf0 O^Yhttp://uberpython.wordpress.com/2012/01/01/precision-recall-sensitivity-and-specificity/0 %r,http://www.h5.com/solutions/h5-trec-results0 n7http://en.wikipedia.org/wiki/Type_I_and_type_II_errors0 7 &http://en.wikipedia.org/wiki/F1_score0 Y 9http://en.wikipedia.org/wiki/Sensitivity_and_specificity0 R2http://en.wikipedia.org/wiki/Precision_and_recall0 l.http://en.wikipedia.org/wiki/Confusion_matrix0 8%Rhttp://www2.cs.uregina.ca/~dbd/cs831/notes/confusion_matrix/confusion_matrix.html0   !"#$%'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdfghijklmnopqrstuvwxyz{|}~Root Entry FJܦData &@|1TableeSWordDocument8JSummaryInformation(DocumentSummaryInformation8CompObjy  F'Microsoft Office Word 97-2003 Document MSWordDocWord.Document.89q