ࡱ> #% !"%` )bjbj"x"x .B@@h P P P d ,S,S,S,S\Td U8ZZZZfJw{XVXXXXXX$ hqF|P qweNfqq| ZZ q> ZP Z qV hP PZU 0,S?,PP Ph~ a~~~||~~~qqqqd d d d=Jdd d d Jd d d  Table of Contents to Introduction to ROC5 (08/26/08)  TOC \o "1-3" \h \z \u  HYPERLINK \l "_Toc207527162" Table of Contents to Introduction to ROC5 (08/26/08)  PAGEREF _Toc207527162 \h 1  HYPERLINK \l "_Toc207527163" 1. Introduction to ROC5 (08/26/08)  PAGEREF _Toc207527163 \h 2  HYPERLINK \l "_Toc207527164" 1.1 Files included with download (at mirecc.Stanford.edu)  PAGEREF _Toc207527164 \h 2  HYPERLINK \l "_Toc207527165" 1.2 What is this Program Good for?  PAGEREF _Toc207527165 \h 2  HYPERLINK \l "_Toc207527166" 1.2.1 Producing a Decision Tree  PAGEREF _Toc207527166 \h 3  HYPERLINK \l "_Toc207527167" 1.2.2 Weighing the Importance of False Positives versus False Negatives  PAGEREF _Toc207527167 \h 3  HYPERLINK \l "_Toc207527168" 1.3 Who Owns this Program?  PAGEREF _Toc207527168 \h 4  HYPERLINK \l "_Toc207527169" 1.4 Where Does the Theory Behind the Program Come From?  PAGEREF _Toc207527169 \h 4  HYPERLINK \l "_Toc207527170" 2. Overview of Programming Strategy  PAGEREF _Toc207527170 \h 4  HYPERLINK \l "_Toc207527171" 3. Data Preparation  PAGEREF _Toc207527171 \h 5  HYPERLINK \l "_Toc207527172" 3.1 The Gold Standard versus Predictors  PAGEREF _Toc207527172 \h 5  HYPERLINK \l "_Toc207527173" 3.2 Details of Data Preparation  PAGEREF _Toc207527173 \h 5  HYPERLINK \l "_Toc207527174" 3.2.1 Missing data  PAGEREF _Toc207527174 \h 5  HYPERLINK \l "_Toc207527175" 3.2.2 ID Numbers  PAGEREF _Toc207527175 \h 5  HYPERLINK \l "_Toc207527176" 3.2.3 Note on Data Recoding  PAGEREF _Toc207527176 \h 5  HYPERLINK \l "_Toc207527177" 4. Running the ROC Program  PAGEREF _Toc207527177 \h 5  HYPERLINK \l "_Toc207527178" 4.1 How do you Run the Program?  PAGEREF _Toc207527178 \h 5  HYPERLINK \l "_Toc207527179" 4.1.1 Batch Files Basics  PAGEREF _Toc207527179 \h 5  HYPERLINK \l "_Toc207527180" 4.1.2 Batch Files Quirks  PAGEREF _Toc207527180 \h 6  HYPERLINK \l "_Toc207527181" 4.2 What does the ROC Output Mean? (and how to read it)  PAGEREF _Toc207527181 \h 6  HYPERLINK \l "_Toc207527182" 4.3 How to Change Emphasis on Sensitivity versus Specificity  PAGEREF _Toc207527182 \h 7  HYPERLINK \l "_Toc207527183" 4.4 How to Get Results for Plots, i.e. ROC Curves  PAGEREF _Toc207527183 \h 7  HYPERLINK \l "_Toc207527184" 4.4.1 How to Actually Get a ROC Plot out of the Data  PAGEREF _Toc207527184 \h 7  HYPERLINK \l "_Toc207527185" 4.4.2 How do I get SAS or Better yet What is SAS?  PAGEREF _Toc207527185 \h 7  HYPERLINK \l "_Toc207527186" 5. Run it Again Sam? More on Decision Trees  PAGEREF _Toc207527186 \h 8  HYPERLINK \l "_Toc207527187" 6. FAQ (Frequently Asked Questions)  PAGEREF _Toc207527187 \h 9  HYPERLINK \l "_Toc207527188" Appendix 1: Note on Memory Allocation and Run Time  PAGEREF _Toc207527188 \h 10  HYPERLINK \l "_Toc207527189" Appendix 2: Note on Data Recoding  PAGEREF _Toc207527189 \h 12  HYPERLINK \l "_Toc207527190" IF  PAGEREF _Toc207527190 \h 12  HYPERLINK \l "_Toc207527191" Even more important is the operator AND in Excel:  PAGEREF _Toc207527191 \h 14  HYPERLINK \l "_Toc207527192" Even more important is the operator = in Excel:  PAGEREF _Toc207527192 \h 14  HYPERLINK \l "_Toc207527193" Rotate.exe:  PAGEREF _Toc207527193 \h 14  HYPERLINK \l "_Toc207527194" Appendix 3: Formulae  PAGEREF _Toc207527194 \h 15  HYPERLINK \l "_Toc207527195" Appendix 4: Example SAS Program for Graphics  PAGEREF _Toc207527195 \h 16  1. Introduction to ROC5 (08/26/08)4 This READ_ME is designed to cover all aspects of our program designed to perform a number of signal detection functions. 1.1 Files included with download (at mirecc.Stanford.edu) Download the file ROC54.ZIP. It can be unzipped by programs such as WinZip. ROC54.ZIP contains the files: ROC4.2118.exe The program old version(100 variables x 15000 cases max) The current version as of 12/29/2003 is ROC4.18.exe. Demo.txt Demo dataset runDemoData.bat The batch file that does all the housekeeping and runs the program on the right dataset with the right settings. outrunDemoData.doc Output from the run ROC4yymmddhhhh.doc A word file of the actual c code for version ROC4. It is dated by yy year, mm month and day and sometimes hhhh hour. Change the .doc to .c for a C compiler. READ_ME.yymmdd.doc An explanation of all this (what you are reading) ROC5.xx.exe The new version for single processor ROC5.xx.dual.exe The new version for dualsingle processor ROC5.xx.quad.exe The new version for quadsingle processor ROC5.cc Source code for ROC5 1.2 What is this Program Good for? This program is designed to help the average clinician/researcher with a PC to evaluate clinical databases and discover the characteristics of patients, including genetics. that best predict a binary outcome. That outcome may be any binary outcome such as: Whether or not the patient has a certain disorder (medical test evaluation) Whether or not the patient is likely to develop a certain disorder (risk factor evaluation) Whether or not the patient is likely to respond to a certain treatment (evaluation of treatment moderators) When the predictors considered are themselves all binary (e.g., male/female; inpatient/outpatient; symptoms present/absent), the program identifies the optimal predictor. When one or more of the predictors are ordinal (e.g., age, severity of symptoms) it identifies the optimal cutpoint for each or the ordinal predictors, as well as the overall optimal predictor. 1.2.1 Producing a Decision Tree The program can be applied to different subsets of the same dataset, thus producing a "decision tree", which combines various predictors with "and/or" rules to best predict the binary outcome. The bottom line of the output is one of these trees. This is a schematized example from a hypothetical study predicting conversion to Alzheimers Disease using age and the Mini-Mental State Exam (MMSE) as potential predictors:          In this example, subjects who are less than 75 years old have a 10% conversion rate. Those who are at least 75 AND have an MMSE score less than 27 have a 20% conversion rate. Finally, subjects older than 75 AND have an MMSE score of at least 27 have a 40% conversion rate. These cutpoints are significant at the p=.01 level. 1.2.2 Weighing the Importance of False Positives versus False Negatives This program (a type of recursive partitioning) differs from other such programs in that the criterion for splitting is based on a CLINICAL judgment of the relative clinical or policy importance of false positive versus false negative identifications via weights called r. The program automatically considers three possibilities: Optimal Sensitivity: Here r=1, and the total emphasis is placed on avoiding false negatives. This would be appropriate, for example, for self-examination for breast or testicular lumps. Optimal Efficiency: Here r=1/2, and equal emphasis is placed on both types of errors. This would be appropriate, for example, for mammagraphy. Optimal Specificity: Here r=0, and total emphasis is placed on avoiding false positives. This would be appropriate, for example, for frozen tissue biopsy done during breast surgery to decide on whether or not a mastecomy were to be done. When the user does not have reason to favor either false positives or false negatives, use of r=1/2 is advised. It is also possible that a user might want to choose a weight of, say, 0.70 to put more emphasis on avoiding false negatives, but not total emphasis. The program has an option for the user to input the value of r (between 0 and 1) to obtain the optimal predictor for that cutpoint. How you do this is described below in Section 4.3: How to Change Emphasis on Sensitivity versus Specificity. 1.3 Who Owns this Program? It is in the public domain. The work that went into this was mostly paid for by the Department of Veterans Affairs and the National Institute of Aging of the United States of America. 1.4 Where Does the Theory Behind the Program Come From? From HC Kraemer, Evaluating Medical TestsSage Publications, Newbury Park, CA 1992. The formulae for the calculations are taken from page X from the book and are presented in Appendix 3. 2. Overview of Programming Strategy The ROC4 program is designed to perform basic signal detection computations in a Windows environment. The program is written in C++ Microsoft version 6.0. Original Mark 4 version was written circa October 2001. Likely it can be recompiled on other platforms that use C++ or C, such as Sun, SGI or other UNIX workstations, and maybe the Macintosh. For details on capacity of the program see Appendix 1, but basically it has been tested on datasets of 50 variables and 8000 cases on a Dell Inspiron 5000 laptop. ROC5 is the industrial strength version designed to perform similar work on very large datasets such as those from whole gene scans containing 600,000 variables and 200 subjects. It has been tested on A Dell Xeon Precision Workstation with 32GB of RAM and a 64-bit processor for the whole genome analyses. This size of workstation appears to be the physical and programming limitation, but luckily seems also to be within the size limitations of typical large-scale genetic analyses. .A big version designed for huge datasets and runs slower as it uses virtual memory, is available on request. To get the full benefit from ROC4this program and 5 you MUST be able to use Excel a bit and if you want sophisticated graphics, SAS. It is a waste of time to recreate the editing and statistical capabilities of Excel and SAS, especially the latter for plotting ROC curves and the former for creating a clean dataset. Dealing with genetic datasets is a whole nother level of complexity. So, the basic idea is that however you prepare your data it should be able one way or another to get to Excel and be output as a text tab-delimited (separated) dataset. Then, after going through ROC4, you also get a text tab-delimited dataset, this time readable by SAS (SAS Institute Inc., Cary NC) for sophisticated plots and graphics, or, more likely, you just look at the results that come out of ROC. So, the basic idea is: Dataprep (Excel) -> signal detection calculations (ROC) -> graphics (SAS)  The ROC output is an ASCII file that can be read into MS Word or Excel. 3. Data Preparation 3.1 The Gold Standard versus Predictors The ROC program reads in data via a text tab-delimited format. The last column is a set of 0s and 1s representing the gold standard. This is the criterion for success. The other columns are the predictors. This can all be arranged in an Excel file and then output to a tab-delimited .txt file. 3.2 Details of Data Preparation 3.2.1 Missing data Represent missing data only with a 9999.99. If you have blanks, edit it in Word first and do a global replace of ^t^t (two tabs) with ^t-9999.99^t. Version 5 also takes the integer -9999 as missing. 3.2.2 ID Numbers Remove any columns of data that will not be analyzed (e.g. ID numbers). 3.2.3 Note on Data Recoding This should be done in Excel before submitting the data to ROC4. See Appendix 2 for information on recoding. A Demonstration dataset (Demo.txt) is also enclosed as part of the Zip package. A separate program is includedrequired if the dataset needs to be rotated: we want the columns to be the variables. If subjects are the column, the dataset must be rotated before it can be analyzed. 4. Running the ROC Program 4.1 How do you Run the Program? 4.1.1 Batch Files Basics It is easiest to run the program out of a batch file (.bat), i.e. you tap on the icon. This is like a UNIX script and basically is a place that keeps all your files and commands straight. rDemoData.bat is a simple one liner edited as a text file in Word: roc4 Demo.txt 50> runDemoData.doc This tells ROC4 to use Demo.txt as the data file and output (the >) it to runDemoData.doc as a word (.doc) file. The 50 is explained in Section 4.3. Note on Versions. As newer versions come out, you may have to modify the script to carry the right version number. Version 5 Notes: Several additional command line arguments are included: (SENS/SPEC) 50 = equal trade off of sensitivity and specificity ... check README... default 50 (PLOT) create dataset for plotting in SAS. Default is OFF, no plot data, turn on with PLOT (NO_PRINT) suppresses printing intermediate values.. lots of them.. default in ON suggest NO_PRINT if more than 10-15 variables (DE BUG) various debugging information... tons of it... default is OFF turn on with DE_BUG (P CRITERION VALUE) 05 01 or 001 default is 01 (no decimals in command line please) EXAMPLE: roc5 pgen.txt 50 NO_PRINT > run_pgen.out.doc 4.1.2 Batch Files Quirks Batch files seem a bit quirky in Windows. We have found that if you write one from scratch you may have to run it first in the DOS (Windows 98) or Command Prompt (Windows 2000, XP or Vista) window. After that it seems to run if you just tap on it. In Windows 2000 the prompt is found under Start: Programs: Accessories. If you have one that works, you can just edit it in Word. It is saved as a .bat file in text format. Note Well: The batch file will not run if either the data.txt or the output.doc files are open. So do not waste a lot of time debugging the .bat file if these are open. How do I know it is running? Look in the folder to which the output file is directed. Under the View menu hit REFRESH, note the file size, hit REFRESH again. If it is larger, take heart that it is working and go get some coffee (or a good nights sleep if you have a slow processor). To get a rough idea of how long it may take to run your ROC program please see Appendix 1. 4.2 What does the ROC Output Mean? (and how to read it) The output should be readable in Word. It is designed to be read in and printed using the following format: Font: Courier New size 7 or 6 with good eyes Move margin all the way to the right. Under File/Page Set-up and Paper Size Tab, set to Landscape There are five segments to the output: The output starts with some general description of the data you put in hopefully with the same n for each variable that you started with, i.e. this is here for a data check. You then get a listing of the various signal detection results for each variable and for each value of each variable in your dataset. Finally, you get a summary of the results for the five highest kappa (50) values for each variable. In general the best cut point to separate successes from failures will be the value of the variable with this highest kappa over all the variables. The program will do a series of iterations basically taking the best cut point identified above and rerun the data that are above and below that cut point. This step is repeated until all cutpoints (up to three-way interactions) are identified. If you would like to identify interactions beyond three-way, see Section 5. A summary of the results The Decision tree (a simplified version was presented in Section 1.2.1) 4.3 How to Change Emphasis on Sensitivity versus Specificity The program has an option for the user to input the value of r (between 0 and 1) to obtain the optimal predictor for that cutpoint. Why you might want to do this is described in Section 1.2.2 Weighing the Importance of False Positives versus False Negatives. Note how the script is changed to accomplish the change in emphasis: roc4 Demo.txt 70> runDemoData.doc This version of the script has a little 70 added this will calculate a 70/30 split to kappa emphasizing sensitivity (70%) versus specificity (30%). Default is 50/50. You can use any proportion as long as it is a multiple of 10, e.g. 0, 10, 20, 30, 40, 50, 60, 70, 80, 90 and 100 are acceptable. Note that optimal sensitivity and specificity (100 and 0, respectively) are automatically calculated by the program, regardless of what is chosen here. 4.4 How to Get Results for Plots, i.e. ROC Curves 4.4.1 How to Actually Get a ROC Plot out of the Data Several programs such as Excel or SAS can simply read in the ROC output.doc file. The output from ROC4 has some means at the top of the file and headers at the top of each variable, which needs to be stripped off (easily done in Excel and saved as a tab-delimited text file) before performing graphics. This is there as a data check that you have the right N and means. Note that in Windows 2000 if you right click on the output.doc file, you have the option of opening the file directly into Excel. Although there are many programs that do graphics, programs such as Excel may only allow relatively simple plots. The SAS program supplied in Appendix 4 will read in the data and create classic ROC Plots, after a couple of lines in the supplied code are modified. See Appendix 3 for the SAS program and further details. 4.4.2 How do I get SAS or Better yet What is SAS? SAS (Statistical Analysis System) is the most widely used professional statistical language in the world. It used to run only on mainframes and UNIX workstations but now runs on PCs. Most Universities have contacts to use it. Pharmaceutical company clinical trial data are submitted to the FDA in SAS datasets. If you do not have SAS and still want a graph, maybe some other language will do it. The output can be read by a number of programs. I have failed to get Excel to do a nice graph of the results. Deltagraph, Statview or other programs perhaps might work? Whatever you try, the basic output from the ROC4 program is a tab-delimited text file, which any worthwhile statistical/graphics program should be able to input. Finally, to facilitate SAS direct input, unnecessary lines of the output file are preceeded by a **. This allows these lines to be ignored by SAS. 5. Run it Again Sam? More on Decision Trees One the first optimal cut is made, yielding the first positive and negative test groups, the program runs again on those with a positive first test, and those with a negative first test, to find the optimal cuts within those subpopulations. Thus if the optimal predictor at the first stage is gender, one might want to consider males and females separately in the next two runs of the program. Then, of course, when the second cuts are made (yielding 4 groups), one could run the program again on these four subpopulations. Doing this produces a "decision tree", a series of "and/or" rules that identify subgroups at different risks of the binary outcome. Since the program will not run when the sample size is inadequate (10 or fewer in each of the marginal positions), this can only be done when the sample sizes are large enough. Moreover, a "stopping rule" should be set in advance. For example, one might stop when the optimal predictor produces a result that is not statistically significant at the 1% level (which is what this program does). Note: in the printout *, **, ***, mean p < .05, .01 and .001. This "significance" is the result of multiple testing and should not be regarded as a true significance level, but is useful as a stopping rule. This in fact is how the program works. It will result in a maximum of three levels of cuts and a final maximum of eight (2**3) subgroups. The reason it stops at this point is two-fold: 1) programmer exhaustion after having created 2,422 lines of C++ code and 2) we are in fact generating three-way interactions of predictors. Note: In Vversion 4.21 by popular demand I added the option of making a l arger decision tree by over-riding the 0.01 criterion for continuing the tree. If you end the command line arguments with, 0.05, it will use 0.05 as criterion. Note well that this gives you a lot more output. Note also well that this will only work for 0.05, not 0.04 or other criteria. This is a monumental programming task to get all of the printing etc. out correctly if this one wanted multiple criteria. Here at least you have the choice of 0.05 or 0.01. Please note that this has not been fully debugged as of 4/1/06, thus, we consider 4.21 Betaware and 4.20 the definitive model. tThis modeificationmodification of to got to 4.21 required about 200 additional lines of code, and you never know, so watch output carefully. It seems to work in our hands. This modification is carried forward into Version 5. If you would like a four-way (or more interaction), divide the initial dataset on the cutpoints identified by the three-way interaction and rerun the program. We have successfully identified up to a ten-way interaction using this method and program. More details on all aspects of ROC including decision trees can be found in the mirecc.stanford.edu web site RealPresenter show:  HYPERLINK "http://video-server.stanford.edu:8080/ramgen/realcontent3/mirecc/ROC/trainer.smi" http://video-server.stanford.edu:8080/ramgen/realcontent3/mirecc/ROC/trainer.smi If you do not have this capability, we can send the actual PowerPoint slides but this is a very large file. 6. FAQ (Frequently Asked Questions) Q: How is this program different from CART, SPSS Answer Tree, etc: A: This program is more concise as it uses kappa to minimize false positives and false negatives. We believe other programs too many branches (and thus sometimes requires pruning afterwards), and / or use the odds ratio, which we do not favor. Q: In the SAS graphing program in Appendix 3 is there a specific reason why only the top 25 points (sorted by descending k0_50) are plotted: A: We arbitrarily decided to plot the top 25 points because we felt plotting more would clutter the graph. Of course it is easy to change to program to plot more or less points if you desire. Q: Does the ROC program use the empirical ROC curve rather than a fitted curve? A: Yes, that is correct. The fitted curve assumptions are usually not true. Q: I am curious how the weighting is done when you select differing values for the sensitivity/specificity emphasis. I have been taught that the bias parameter often represents the ratio of the ordinate of the S distribution over the ordinate of the SN distribution (I may have these backwards). Does emphasis in the ROC program map onto that concept in some way? A: No. The ROC program makes no distribution assumptions. It is non parametric Appendix 1: Note on Memory Allocation and Run Time Historical Information (circa 2001): The version 4 program is designed to make maximum use of the memory capacity of your machine. This is industrial-design programming, rather than convenience-designed programming. As written it uses about 12MB of memory to run a matrix of 25 variables for 35000 cases (largest test so far). In this situation, it may take the program over an hour to run data on a 650mHz Intel processor, but at least it runs in the background, though it takes up 90% of my processor. The maximum initialization is for 1000 variables and 50,000 cases. It does not initialize to a larger number because a larger initial data matrix may crash on a machine with small memory capacity or at least make the program use virtual memory (disk instead of RAM). If you have a large dataset, a big machine and want to edit the program, just change the values for NCOL and NROW to the number of variables and cases you need, then recompile, or ask me and I can do it. The standard program, however, allocates memory only as needed and will run much faster on small datasets, such those with as 10 variables and 500 cases (a few seconds). Slowness appears to increase exponentially with N because computations involve not only each case, but every other case too, etc. etc... To see how all this memory allocation is done, look for the function malloc() in the program. Clever programmers with time on their hands could do this more efficiently and free memory once it is no longer used. The way it is written, you use about 6 bytes/case times the number of variables. Stingier allocation of memory would get it down from 5MB. FYI, on my machine Windows 2000 toys seem to use about 40MB, Microsoft Word 20 MB and Outlook 15MB. Bottom line, forget running this on a machine with 64MB main storage (it will have to use slow virtual or paging memory). I may compile a larger capacity version and leave it on the web site for those who want to deal with US census data and have the machine to complete the task. How long might the program take to run? We have successfully run the ROC program with about 20 variables (half of which were binary) and 35000 observations. This took 2 hours on a Dell M60 laptop with Windows XP, 512MB memory and 1.7 Gz P4m processor, BUT, only by using the new NO_PRINT option. Adding this to the command line will supress the initial intemediate values. The sorting of these intermediate values could take literally DAYS in large datasets. The suggestion is, if it runs slowly on a large dataset, use this option. Recent Information (circa late 2007-8): Version 5 was designed to read large dataset such as those from whole genome scan, i.e. 600,000 variables. This is not easy, but you can also use Version 5 on smaller datasets and it will run much faster due to many improvements in the programming. A dataset of 600,000 SNPs on 210 patients takes less than a day on a Dell Xenon Precision Workstation. To run Version 5 with genetic data you need a big computer like that Dell Workstation: 64bit processor, 6 if not 8 or more GB of RAM and a fast processor does not hurt. We have precompiled a couple versions of the program but the code is different depending on your processor, so the precompiled code may not work on your machine.. ask us for help or get somebody to compile itproduced an installer due to the multiple processors running under windows. This version was compiled using Microsoft Visual Studio StandardPro 20085 for 32bit versions and the Beta 2008 version of this program for 64bit versions. It has also been compiled and run on a Linux 32bit workstation running Fedora 6. The big computer is necessary because you simply cannot do all the computations this program does unless all data are in memory (RAM). Furthermore, a 32bit processor is limited to 2GBs for the program. 64bit programs can be bigger. An unpleasant nuance to this is that with the 64bit processor you may think you have a lot of RAM if you have 6GB, but it turns out that floating point numbers and even integers require 4 bytes to represent one number (32bit computing just takes 1). So, the 6GB 64bit processor actually gives you fewer floating point numbers than a 32bit processor with 2GB! Buyer beware. We are in the process of obtainingnow have a 64bit Dell with 32GB of RAM and we will need it all. Thanks to the folks at Dell and Microsoft who helped me on this project, especially Scot Brennecke of Microsoft who got the Version 4 program to compile without errors the first time on a 64bit processor. Appendix 2: Note on Data Recoding I think it is best to do this in Excel for two reasons: By the time you indicate how you should recode data in command line arguments, they get too big and confusing and prone to error. Same thing for gold standard computations. Better to get the dataset fixed up nice in Excel before running the program. You can just delete any offending columns and you can compute any gold standard you want using conditional statements i.e. ifs. Here is an example. Say you want to consider an infant feeding program success if children are over 1800g birthweight, well just put: =IF(C2<1800,0,1) In the last column first cell assuming C2 is where the birth weight is located. Then highlight the whole column down to the bottom and hit Control-d, or copy down the column. Voila, data recoding with 0s and 1s in that column. Now, people might say I am just lazy, but Microsoft has put a whole lot of options in those conditional statements and they work for text. Let the following help page for the one conditional if from Microsoft Excel be a testimonial: IF Returns one value if a condition you specify evaluates to TRUE and another value if it evaluates to FALSE. Use IF to conduct conditional tests on values and formulas. Syntax IF(logical_test,value_if_true,value_if_false) Logical_test is any value or expression that can be evaluated to TRUE or FALSE. For example, A10=100 is a logical expression; if the value in cell A10 is equal to 100, the expression evaluates to TRUE. Otherwise, the expression evaluates to FALSE. This argument can use any  HYPERLINK "xlrefUnderstandinghowOperatorswork.htm" comparison calculation operator. Value_if_true is the value that is returned if logical_test is TRUE. For example, if this argument is the text string "Within budget" and the logical_test argument evaluates to TRUE, then the IF function displays the text "Within budget". If logical_test is TRUE and value_if_true is blank, this argument returns 0 (zero). To display the word TRUE, use the logical value TRUE for this argument. Value_if_true can be another formula. Value_if_false is the value that is returned if logical_test is FALSE. For example, if this argument is the text string "Over budget" and the logical_test argument evaluates to FALSE, then the IF function displays the text "Over budget". If logical_test is FALSE and value_if_false is omitted, (that is, after value_if_true, there is no comma), then the logical value FALSE is returned. If logical_test is FALSE and value_if_false is blank (that is, after value_if_true, there is a comma followed by the closing parenthesis), then the value 0 (zero) is returned. Value_if_false can be another formula. Remarks Up to seven IF functions can be nested as value_if_true and value_if_false arguments to construct more elaborate tests. See the following last example. When the value_if_true and value_if_false arguments are evaluated, IF returns the value returned by those statements. If any of the arguments to IF are arrays, every element of the array is evaluated when the IF statement is carried out. Microsoft Excel provides additional functions that can be used to analyze your data based on a condition. For example, to count the number of occurrences of a string of text or a number within a range of cells, use the  HYPERLINK "xlfctCOUNTIF.htm" COUNTIF worksheet function. To calculate a sum based on a string of text or a number within a range, use the  HYPERLINK "xlfctSUMIF.htm" SUMIF worksheet function. Learn  HYPERLINK "xlconCondFormulas.htm" about calculating a value based on a condition. Examples On a budget sheet, cell A10 contains a formula to calculate the current budget. If the result of the formula in A10 is less than or equal to 100, then the following function displays "Within budget". Otherwise, the function displays "Over budget". IF(A10<=100,"Within budget","Over budget") In the following example, if the value in cell A10 is 100, then logical_test is TRUE, and the total value for the range B5:B15 is calculated. Otherwise, logical_test is FALSE, and empty text ("") is returned that blanks the cell that contains the IF function. IF(A10=100,SUM(B5:B15),"") Suppose an expense worksheet contains in B2:B4 the following data for "Actual Expenses" for January, February, and March: 1500, 500, 500. C2:C4 contains the following data for "Predicted Expenses" for the same periods: 900, 900, 925. You can write a formula to check whether you are over budget for a particular month, generating text for a message with the following formulas: IF(B2>C2,"Over Budget","OK") equals "Over Budget" IF(B3>C3,"Over Budget","OK") equals "OK" Suppose you want to assign letter grades to numbers referenced by the name AverageScore. See the following table. If AverageScore isThen returnGreater than 89AFrom 80 to 89BFrom 70 to 79CFrom 60 to 69DLess than 60FYou can use the following nested IF function: IF(AverageScore>89,"A",IF(AverageScore>79,"B", IF(AverageScore>69,"C",IF(AverageScore>59,"D","F")))) In the preceding example, the second IF statement is also the value_if_false argument to the first IF statement. Similarly, the third IF statement is the value_if_false argument to the second IF statement. For example, if the first logical_test (Average>89) is TRUE, "A" is returned. If the first logical_test is FALSE, the second IF statement is evaluated, and so on. Even more important is the operator AND in Excel: IF(AND(1 Outfile. Appendix 3: Formulae Taken from Kraemer (1992) page X: TP=true positive=% positive test and diagnosis TN=true negative=% negative test, negative diagnosis FP=false positive=% positive test, negative diagnosis FN=false negative=% negative test, positive diagnosis P=prevalence=TP+FN (P=1-P) Q=level of test=TP+FP (Q=1-Q) SENS=sensitivity=TP/P SPEC=specificity=TN/P PVP=predictive value of positive test=TP/Q PVN=predictive value of negative test=TN/Q EFFIC=efficiency=TP+TN k0_0, k_50, k1_0 are weighted kappa coefficients where: k0_0=(SPEC-Q')/Q k1_0=(SENS-Q)/Q' k0_50=(EFFIC-PQ-P'Q')/(1-PQ-P'Q') Appendix 4: Example SAS Program for Graphics (As of 12/12/2002 this program should still be considered preliminary) Note that you will need to have a site license for SAS/Graph in order to run this program successfully. Your SAS Site Representative should know whether you have this. In order to run this program, two lines of code may need to be modified: In the infile statement about half a page down below (infile 'C:\art\QROC\people\Jerry\011101 Helena Data\run_controlANDtreatment_501.txt'), change the directory structure and filename to that corresponding on your PC. Towards the end of the program there is a goptions statement (goptions device= /* WIN */ LJIV600). To print the plots directly to an HP Printer, leave the statement as is. To view the plots on your monitor, remove the /* */ surrounding WIN and add them to /* LJIV600 */. The slash-star pair causes SAS to ignore the code within. Its actually more complicated than this because what if the user has a different printer from what we are using. There are hundreds of different choices. This is a problem--Ill fill in the details later- Art Noda. Once these steps are complete, the program can be run and a listing showing the top 25 points (sorted by descending k_r) will either be plotted or displayed on your screen and these top 25 will also be shown in the SAS output window, where they can be printed. /*************************/ /* QROCPLOT.011030.SAS */ /*************************/ /* 10/30/01 Read in output from Jerry's QROC program and create two plots: 1) K1_0 vs K0_0 (ROC Curve) 2) Sensitivity vs Specificity (QROC Curve) */ /********************************/ /* SET UP DEFAULT SAS OPTIONS */ /********************************/ options ls=110 ps=50 nocenter pageno=1 errors=2 mprint; /*****************************************/ /* READ IN OUTPUT FROM JERRY's PROGRAM */ /*****************************************/ data qroc; infile 'C:\art\QROC\people\Jerry\011101 Helena Data\run_controlANDtreatment_501.txt' dlm='09'x lrecl=500 missover; input varnum n ltge $ critraw $ p q sens spec effic k0_0 k0_r k1_0 pvp pvpnum pvpden pvn pvnnum pvnden chi_square sig $; /* Set up new variable CRITERION which concatenates LTGE and CRITRAW */ criterion=compress(ltge||critraw); run; proc print data=qroc uniform; var varnum n criterion p q sens spec effic k0_0 k0_r k1_0 pvp pvpnum pvpden pvn pvnnum pvnden chi_square sig; title "Output from Jerry's QROC Program"; run; /* Keep top 25 obs, sorted by descending k0_r (r=0.5 by default) */ proc sort data=qroc; by descending k0_r; run; data qroc; set qroc; if _n_ le 25; run; proc print data=qroc uniform; var varnum n criterion p q sens spec effic k0_0 k0_r k1_0 pvp pvpnum pvpden pvn pvnnum pvnden chi_square sig; title "Top 25 records, sorted by descending k0_r"; run; /* Set up reference lines for plots as follows: */ /* 1) for k1_0 vs k0_0 plot, add k1_0=k0_0 line */ /* 2) for sens vs spec plot, add */ /* (a) sens=1-spec line and */ /* (b) line from (P Prime,P) to (1,1) */ data qroc; set qroc; if _n_=1 then do; k1_0ref=0; k0_0ref=0; sensref=1; specref=0; pref=p; pprimeref=1-p; end; else do; k1_0ref=1; k0_0ref=1; sensref=0; specref=1; pref=1; pprimeref=1; end; run; /*********************************************************************************/ /*****************************/ /* SET UP PLOTTING OPTIONS */ /*****************************/ /*------------------------------------------------------------------* | Summary: | | Creating a simple plot using the table | | SASUSER.SENSSPEC and plotting | | sens against spec. | | Generated: 29OCT2001 1:05:01 | *------------------------------------------------------------------*/ /*------------------------------------------------------------------* | The GOPTIONS statement allows you to have more control over the | | final appearance of your output such as fonts, colors, text | | height and so on. The output device and destination is also | | specified in the goptions statement. | *------------------------------------------------------------------*/ /* HSIZE and VSIZE are different, but give a square plot */ goptions reset=(axis, legend, pattern, symbol, title, footnote) norotate hpos=0 vpos=0 htext= ftext= ctext= target= gaccess= gsfmode= hsize=6in vsize=7in; /* Use device = WIN to print to monitor, LJIV600 to print directly to HP Printer */ goptions device= /* WIN */ LJIV600 ctext=black graphrc interpol=none; /*------------------------------------------------------------------* | SYMBOL statements allow you to supply information such as plot | | character, plot lines, color and interpolation. | *------------------------------------------------------------------*/ /* POINTLABEL is the variable number as specified in Jerry's output */ symbol1 color=black value=star pointlabel = ("#varnum") ; symbol2 interpol=join width=3.0 color=black ; symbol3 interpol=join width=3.0 color=black ; /*------------------------------------------------------------------* | AXIS statements allow you to supply information on how your | | vertical and horizontal axes will appear on the graph. | *------------------------------------------------------------------*/ axis1 order=0.0 to 1.0 by 0.1 color=black width=3.0 style=1 ; axis2 order=0.0 to 1.0 by 0.1 color=black width=3.0 style=1 ; axis3 color=black width=3.0 ; /*------------------------------------------------------------------* | This section produces the actual plot and any options that | | directly relate to the data and the axis area. | *------------------------------------------------------------------*/ proc gplot data=qroc; where (0 le k1_0 le 1) and (0 le k0_0 le 1); plot k1_0 * k0_0 k1_0ref * k0_0ref / haxis=axis1 vaxis=axis2 frame overlay ; title1 'QROC Plot'; title2 'K1_0 vs. K0_0'; run; proc gplot data=qroc; where (0 le sens le 1) and (0 le spec le 1); plot sens * spec sensref*specref pref*pprimeref / haxis=axis1 vaxis=axis2 frame overlay ; title1 'ROC Plot'; title2 'Sensitivity vs. Specificity'; run; quit; 20% Convert 40% Convert MMSE < 27 30% Convert 10% Convert Age < 75 All Subjects (20% convert) %456MNOPlmno $ ˨˝}c˝2jhrXhrX>*B*UmHnHphu j}hrXUmHnHujhrXUmHnHuhrXmHnHu2jhrXhrX>*B*UmHnHphuhrXmHnHuhGohrX0JmHnHu$jhGohrX0JUmHnHuhjhUh/\h 5* p X 8 Z `zOB- !  !  ! ($ % & ' ( ) * + , H I J K   ¹««Հ¹«f«U jkhrXUmHnHu2jhrXhrX>*B*UmHnHphu jqhrXUmHnHu2jhrXhrX>*B*UmHnHphuhGohrX0JmHnHuhrXmHnHu$jhGohrX0JUmHnHuhrXmHnHujhrXUmHnHu jwhrXUmHnHu!   ) * + , N O P j k l m n o p q r һɻһһɻvһeһ j_hrXUmHnHu2jhrXhrX>*B*UmHnHphu jehrXUmHnHu2jhrXhrX>*B*UmHnHphuhGohrX0JmHnHuhrXmHnHu$jhGohrX0JUmHnHuhrXmHnHujhrXUmHnHu'     6 7 8 R S T U V W X Y Z v w x y λ谡v谡e jShrXUmHnHu2jhrXhrX>*B*UmHnHphu jYhrXUmHnHujhrXUmHnHuhrXmHnHu$jhGohrX0JUmHnHu2jhrXhrX>*B*UmHnHphuhGohrX0JmHnHuhrXmHnHu"    2 3 4 5 6 7 8 9 : V W X Y l m n źӑőwźfӑő jG hrXUmHnHu2jhrXhrX>*B*UmHnHphuhrXmHnHu jMhrXUmHnHujhrXUmHnHuhrXmHnHuhGohrX0JmHnHu$jhGohrX0JUmHnHu2jhrXhrX>*B*UmHnHphu     8 9 : T U V W X Y Z [ \ x y źӑőwźfӑő j; hrXUmHnHu2j hrXhrX>*B*UmHnHphuhrXmHnHu jA hrXUmHnHujhrXUmHnHuhrXmHnHuhGohrX0JmHnHu$jhGohrX0JUmHnHu2j hrXhrX>*B*UmHnHphu y z {  !źӑőwźfӑő j/ hrXUmHnHu2j hrXhrX>*B*UmHnHphuhrXmHnHu j5 hrXUmHnHujhrXUmHnHuhrXmHnHuhGohrX0JmHnHu$jhGohrX0JUmHnHu2j hrXhrX>*B*UmHnHphu !"#>?@Z[\]^_`ab~źӑőwźfӑő j#hrXUmHnHu2jhrXhrX>*B*UmHnHphuhrXmHnHu j)hrXUmHnHujhrXUmHnHuhrXmHnHuhGohrX0JmHnHu$jhGohrX0JUmHnHu2j hrXhrX>*B*UmHnHphu  !=>?@XYZtuvwxyz{|źӑőwźfӑő jhrXUmHnHu2jhrXhrX>*B*UmHnHphuhrXmHnHu jhrXUmHnHujhrXUmHnHuhrXmHnHuhGohrX0JmHnHu$jhGohrX0JUmHnHu2jhrXhrX>*B*UmHnHphu -./IJKLMNOPQmnźӑőwźfӑő j hrXUmHnHu2jhrXhrX>*B*UmHnHphuhrXmHnHu jhrXUmHnHujhrXUmHnHuhrXmHnHuhGohrX0JmHnHu$jhGohrX0JUmHnHu2jhrXhrX>*B*UmHnHphu nop !"<=>?@ABCD`aźӑőwźfӑő jhrXUmHnHu2jhrXhrX>*B*UmHnHphuhrXmHnHu jhrXUmHnHujhrXUmHnHuhrXmHnHuhGohrX0JmHnHu$jhGohrX0JUmHnHu2jhrXhrX>*B*UmHnHphu abc   '()*+,-./KLźӑőwźfӑő jhrXUmHnHu2jvhrXhrX>*B*UmHnHphuhrXmHnHu jhrXUmHnHujhrXUmHnHuhrXmHnHuhGohrX0JmHnHu$jhGohrX0JUmHnHu2j|hrXhrX>*B*UmHnHphu LMNz{| !źӑőwźfӑő jhrXUmHnHu2jjhrXhrX>*B*UmHnHphuhrXmHnHu jhrXUmHnHujhrXUmHnHuhrXmHnHuhGohrX0JmHnHu$jhGohrX0JUmHnHu2jphrXhrX>*B*UmHnHphu !"#VWXrstvwxyz{źӑőwźfӑő jhrXUmHnHu2j^hrXhrX>*B*UmHnHphuhrXmHnHu jhrXUmHnHujhrXUmHnHuhrXmHnHuhGohrX0JmHnHu$jhGohrX0JUmHnHu2jdhrXhrX>*B*UmHnHphu y% \&(Lm` p@ ^@ ` pp^p`& p^` !  !  "#$%&'CDEFwxyźӑőwźfӑő jhrXUmHnHu2jRhrXhrX>*B*UmHnHphuhrXmHnHu jhrXUmHnHujhrXUmHnHuhrXmHnHuhGohrX0JmHnHu$jhGohrX0JUmHnHu2jXhrXhrX>*B*UmHnHphu     +,-.9:;UVWYZ[\]^z{źӑőwźfӑő jhrXUmHnHu2jFhrXhrX>*B*UmHnHphuhrXmHnHu jhrXUmHnHujhrXUmHnHuhrXmHnHuhGohrX0JmHnHu$jhGohrX0JUmHnHu2jLhrXhrX>*B*UmHnHphu {|} !#$%&'>?źӑőwźfӑ^ZPHhH<h|r<hjhU j hrXUmHnHu2j: hrXhrX>*B*UmHnHphuhrXmHnHu jhrXUmHnHujhrXUmHnHuhrXmHnHuhGohrX0JmHnHu$jhGohrX0JUmHnHu2j@hrXhrX>*B*UmHnHphu?JKRSTmrtv޸vaavNF<HhFhFhmH sH %hk-h|r<cHdhdhdhH<)hhFcHcHdhdhH<dhF%hhFcHdhdhdhF)hk-hFcHcHdhdhH<dhF2HhH<h|r<hFcHdhdhdhF%hk-hFcHdhdhdhF%hhFcHdhdhdhFh%hh|r<cHdhdhdhH<HhFhF`8<BM\̲jP4jP4j6HhJ<h|r<hFcHcHdhdhɃFdhF2HhɃFh({hFcHdhdhdhF2HhJ<h|r<hFcHdhdhdhF2HhJ<h|r<hFcHdhdhdhFHhJ<h|r<HhI<h2HhI<h|r<hFcHdhdhdhF%hhFcHdhdhdhF%hhFcHdhdhdhFHhFhFhMklZXR`P ppC$EƀI<^p`S ppC$EƀJ<^p`gd|r<\akl&(1; +++,E,̹}si_UHhFh6|HhFh6|Hh̓Fh({HhM<h|r<hZ&:jhZ&:CJUmHnHujhCJUmHnHuHhL<h|r<HhK<h|r<h%h|r<h|r<cHdhdhdhJ<2HhJ<h|r<hFcHdhdhdhF2HhɃFh({hFcHdhdhdhF9q*$h`hF & FEƀG<qF & FEƀG<qF & FEƀG<q9F!!"gdZ&:`h`h"#)$%%q*(F & FEƀG<qF & FEƀG<qF & FEƀG<q%'0''$(()c-.00111M1% $0]0`a$]kd4!$$Ifl&&0&64 lal0$If]0`E,Y,t,,,,,b------..00000122V2`233333455Ѿ}}sssY2HhP<hhFcHdhdhdhFHhP<h h>* h5\HhO<hHhFhF%hhvFcHdhdhdhӃFHhӃFhvFh%hh|r<cHdhdhdhL<2HhM<h|r<h({cHdhdhdh̓FHhFh6|HhFh6| M1a112223342455567477F88GC$EƀR<gd$a$55J5Q55555;6E67?7C7K7S7i7o77777E8F89999V9`9a9f9i999::::};;y<<'=B=_?s???GCҶ~~zhJHhV<h hPJHhT<hHhU<hHhS<hHhR<hHhR<hPJ h>*PJh5PJ\ hPJ h>*hHhQ<hHhP<hHhFhF088K99':|::lGC$EƀR<gdK & FC$EƀS<gd:::y<'=>>G?t????^DC$EƀR< ?@5A0Bq*F & FEƀG<()F & FEƀG<()F & FEƀG<()GCSCCC^ElEnEDGKXNZNNTUUU>U?U@UAUBUGUVVV%WWWWWWWWWWĸězěěeYezeHhFhF5(hJhF5cHdhdhdhF(hJhF5cHdhdhdhFHhFhF5 hJ5.hJhJhF5cHdhdhdhFHhFhF5hJhJ5!h5CJ\fHq hfHq h5PJ\ hPJhhJ"0BuCCCDq*(F & FEƀG<()F & FEƀG<()F & FEƀG<()D^EEDGvGGIJKRLMNNOQTzXuYYZ[;[[y\]]^h^_h`h$a$WXBXCXyXzXuYYYUZVZWZZZZ[o[p[h^,`T`U`a``êyd`Z`E)HhX<hNha<*>* haJh)hhrX0JcHdhdhdhF7j!hhrX>*UcHdhdhdhF(hhrX>*cHdhdhdhF1jhhrX>*UcHdhdhdhF%hhrXcHdhdhdhFhhJhJ5HhFhF5HhFhF5 hJ5_,`a`neRhzhljknoDC$EƀZ<DC$EƀX<````iaatbbd8dhhhh ii$i*HHhX<hNhhrXcHdhdhdhFa<*>*)HhX<hNha<*>*HhX<h%hh.EcHdhdhdhd<hwvhHhX<h"3lNllllllGmtmmmnnn n nZnwnnnnnnnEoGoMoYoǭyoeyoeoy[e[eQe[eQGQGHhc<hNHh]<h]Hhb<hNHh\<h]HhFhrX2Hh\<h]hrXcHdhdhdhF2Hha<hNhrXcHdhdhdhF2Hh[<h]hrXcHdhdhdhF2Hh]<h]hrXcHdhdhdhFHh[<h]HhZ<h]Hha<hNYo{o|ooooppqq5q=qwqxqqqqFrGrkryr}rrrrrrr&sRséÕtmi_i_i_iUiUKHhZfhHhwZfhHhvZfhh h.Eh.EHhg<h.Eh.EHhg<h.EHhe<h.EHhd<h]HhFhrX2Hhh<h.EhrXcHdhdhdhFHhh<h.EHhd<h.EHh_<h]Hhc<h.EHh^<h]Hh]<h]nxqGrkrrttvvv7w>wusqqlqjhhf$a$DC$Eƀe<DC$Eƀ]< Rs\sst!tCtathttt|tttttttu uuu8u>uGuHuNuOucuiukuuuuvvv-v?v{vvvv7w״ê~Ò~~tttjHhZfhHhZfhHhZfhHhZfhHhZfhhHhZfhHhZfhHhZfh0J5\HhZfhHhZfhHhZfhHhZfhHhZfhHhwZfh)7w=w>w@wAwMwNw[wlwxwxxxxxxxxx||RSqrsz{!"EFGuvyzÂ>Zp HhZfh0Jjx'hJUj3&hJUj$hJUHhZfh0Jju#hJUjHhZfhUHhZfhHhZfh5\5>wlwxz||}}aL & FdEƀG<[$L & FdEƀG<[$}w~yzĂ>pecaaaaaaaaM & FddEƀG<[$\$L & FdEƀG<[$ *+;=>LNVKkd6)$$If00B N 6340a$IfKkd($$If00B N 6340a $$Ifa$ )*+:;<=>KLMNO\]^_`mnopq}~ԈՈKɺɺɺɺɺɺɺɺɺɺɮɤzvzhHhZfhHhZfhHhZfhHhZfh0JHhZfhHhZfh0JHhZfhB*phHhZfhHhZfhCJaJHhZfh5\"HhZfh5B*\ph+NO]_`npaKkd *$$If00B N 6340a$IfKkd)$$If00B N 6340apq~ޅa_]][Y_Kkd*$$If00B N 6340a$IfKkdw*$$If00B N 6340a nL̊cPC$Eƀi<gd\ynoi<d&FGC$Eƀi<gd\ynKLVʉsʊˊ̊͊7 /͎ӎԎrثv[@v4hB* CJOJQJ^JaJfHphq 4hB*CJOJQJ^JaJfHphq 4hB*CJOJQJ^JaJfHphq h5\hZ&:haJmH sH  haJh h\ynh\ynHhl<h\ynHhk<h\ynHhj<h\ynh\yn%h\ynh\yncHdhdhdhi<Hhi<h\ynHhi<h̊͊7l؋+Bm d7$8$H$[$` d7$8$H$[$ d[$`d[$d[$DC$Eƀi< -./^NrF & FEƀG<) d7$8$H$[$` ɏʏRUŐǐǒȒߓ寑ssojosoK<hB* CJOJQJ^JaJfHmH phq sH  *hh:h5B*CJOJQJ\^JaJfHphq :h5B* CJOJQJ\^JaJfHphq 4hB* CJOJQJ^JaJfHphq 4hB*CJOJQJ^JaJfHphq 4hB*CJOJQJ^JaJfHphq rȒ Mi 7$8$H$` ^`F & FEƀG<)<=>?kÔĔϔ$IϕЕ=>CDbǖݖ 7$8$H$` )*23?jk”ĔȔϔjjjLj:h5B* CJOJQJ\^JaJfHphq 4hB*CJOJQJ^JaJfHphq 4hB* CJOJQJ^JaJfHphq Bh5B* CJOJQJ\^JaJfHmH phq sH <hB*CJOJQJ^JaJfHmH phq sH <hB*CJOJQJ^JaJfHmH phq sH ϔՔ֔#+./45:;>?GINЕ>ʯʯʑsU:4hB* CJOJQJ^JaJfHphq :hZ&:hB*CJOJQJ^JaJfHphq :hZ&:hB*CJOJQJ^JaJfHphq :h5B* CJOJQJ\^JaJfHphq 4hB* CJOJQJ^JaJfHphq 4hB*CJOJQJ^JaJfHphq 4hB*CJOJQJ^JaJfHphq >ADHINOSY`beݖ SUYZ^_cjlmw~ǬǬǬǬǑvǬǬǬǬǬ4hB* CJOJQJ^JaJfHphq 4hB* CJOJQJ^JaJfHphq 4hB*CJOJQJ^JaJfHphq 4hB*CJOJQJ^JaJfHphq :h5B* CJOJQJ\^JaJfHphq & TUj~˗0Fy~T 7$8$H$`—ɗ˗ΗFKLwy|STǩǩǩǎǎǎǎsǩXXXXXǩǎ4hB* CJOJQJ^JaJfHphq 4hB* CJOJQJ^JaJfHphq 4hB*CJOJQJ^JaJfHphq :h5B* CJOJQJ\^JaJfHphq 4hB*CJOJQJ^JaJfHphq :h5B* CJOJQJ\^JaJfHphq "̙ޙ'08ASew2Rr 7$8$H$`șəڙۙ!"+.26<>OPabst12Qʬʬʬʬʬʬʬʬʬʬʬʬʎsss4hB* CJOJQJ^JaJfHphq :h5B* CJOJQJ\^JaJfHphq :h5B* CJOJQJ\^JaJfHphq 4hB*CJOJQJ^JaJfHphq 4hB*CJOJQJ^JaJfHphq -QRqsceMOWX]v|Ÿ̟ӟ՟ܟSU]^dfo|:h5B* CJOJQJ\^JaJfHphq 4hB*CJOJQJ^JaJfHphq 4hB* CJOJQJ^JaJfHphq 4hB*CJOJQJ^JaJfHphq 9rsG՜d9ǞNOߟTU 7$8$H$`8ǡȡ*8TYZbsʢϢТѢҢ` 7$8$H$`ơȡ#-2;EIRZav{|âӢ "'(ʯʯʯʯʔʯʯvʯʯʯvʯʯʯvvvʯʯ:h5B* CJOJQJ\^JaJfHphq 4hB* CJOJQJ^JaJfHphq 4hB*CJOJQJ^JaJfHphq 4hB*CJOJQJ^JaJfHphq 4hB* CJOJQJ^JaJfHphq .`,7<B]lyCҥӥ 7$8$H$`(+/456<AEJKNRUY\`eotux|ѥӥץإݥޥǬǬǬǬǬǬǬǬǬǑssǬ:h5B* CJOJQJ\^JaJfHphq 4hB* CJOJQJ^JaJfHphq 4hB*CJOJQJ^JaJfHphq 4hB*CJOJQJ^JaJfHphq :h5B* CJOJQJ\^JaJfHphq *FKX]joszģģģģĈmmmmmmRmmRm4hB* CJOJQJ^JaJfHphq 4hB*CJOJQJ^JaJfHphq 4hB*CJOJQJ^JaJfHphq @hZ&:h5B* CJOJQJ\^JaJfHphq :hZ&:hB*CJOJQJ^JaJfHphq :hZ&:hB*CJOJQJ^JaJfHphq @Rdp{Ϧ3EWcnvw`gd 7$8$H$`¦ƦѦ֦ئ٦9>KP]bfmw}ǬǍnLnLnLnLnǬǬǬǬǬBh5B* CJOJQJ\^JaJfHmH phq sH <hB*CJOJQJ^JaJfHmH phq sH <hB*CJOJQJ^JaJfHmH phq sH 4hB*CJOJQJ^JaJfHphq 4hB*CJOJQJ^JaJfHphq :h5B* CJOJQJ\^JaJfHphq }~̧ͧ٧ $%')十uuuqhZ&:hCJOJQJaJ0hZ&:CJOJQJaJ0h:h5B* CJOJQJ\^JaJfHphq 4hB*CJOJQJ^JaJfHphq 4hB* CJOJQJ^JaJfHphq 4hB*CJOJQJ^JaJfHphq ̧ͧ٧ڧ $%&'()`gd $7$8$H$`a$,1h/ =!"#8$8% }DyK _Toc207527162}DyK _Toc207527162}DyK _Toc207527163}DyK _Toc207527163}DyK _Toc207527164}DyK _Toc207527164}DyK _Toc207527165}DyK _Toc207527165}DyK _Toc207527166}DyK _Toc207527166}DyK _Toc207527167}DyK _Toc207527167}DyK _Toc207527168}DyK _Toc207527168}DyK _Toc207527169}DyK _Toc207527169}DyK _Toc207527170}DyK _Toc207527170}DyK _Toc207527171}DyK _Toc207527171}DyK _Toc207527172}DyK _Toc207527172}DyK _Toc207527173}DyK _Toc207527173}DyK _Toc207527174}DyK _Toc207527174}DyK _Toc207527175}DyK _Toc207527175}DyK _Toc207527176}DyK _Toc207527176}DyK _Toc207527177}DyK _Toc207527177}DyK _Toc207527178}DyK _Toc207527178}DyK _Toc207527179}DyK _Toc207527179}DyK _Toc207527180}DyK _Toc207527180}DyK _Toc207527181}DyK _Toc207527181}DyK _Toc207527182}DyK _Toc207527182}DyK _Toc207527183}DyK _Toc207527183}DyK _Toc207527184}DyK _Toc207527184}DyK _Toc207527185}DyK _Toc207527185}DyK _Toc207527186}DyK _Toc207527186}DyK _Toc207527187}DyK _Toc207527187}DyK _Toc207527188}DyK _Toc207527188}DyK _Toc207527189}DyK _Toc207527189}DyK _Toc207527190}DyK _Toc207527190}DyK _Toc207527191}DyK _Toc207527191}DyK _Toc207527192}DyK _Toc207527192}DyK _Toc207527193}DyK _Toc207527193}DyK _Toc207527194}DyK _Toc207527194}DyK _Toc207527195}DyK _Toc207527195$$Ifl!vh5&#v&:V l0&65&4alDyK Qhttp://video-server.stanford.edu:8080/ramgen/realcontent3/mirecc/ROC/trainer.smiyK http://video-server.stanford.edu:8080/ramgen/realcontent3/mirecc/ROC/trainer.smiuDyK yK ../../../Administrator/Local Settings/Temporary Internet Files/011105 download/xlrefUnderstandinghowOperatorswork.htmyX;H,]ą'c%IDyK yK ../../../Administrator/Local Settings/Temporary Internet Files/011105 download/xlfctCOUNTIF.htmyX;H,]ą'c%EDyK yK ../../../Administrator/Local Settings/Temporary Internet Files/011105 download/xlfctSUMIF.htmyX;H,]ą'c%SDyK yK ../../../Administrator/Local Settings/Temporary Internet Files/011105 download/xlconCondFormulas.htmyX;H,]ą'c%i$$If!vh5 5W#v #vW:V 065N 5340i$$If!vh5 5W#v #vW:V 065N 5340i$$If!vh5 5W#v #vW:V 065N 5340i$$If!vh5 5W#v #vW:V 065N 5340i$$If!vh5 5W#v #vW:V 065N 5340i$$If!vh5 5W#v #vW:V 065N 5340)L@L Normalx`CJ_HaJmH sH tH \@\ Heading 1$$<@&`a$5CJ KH \aJ V@V Heading 2$<@&`5CJPJ\aJ\@\ Heading 3$$<@&`a$5>*PJ\aJ\@\ Heading 4$$7$8$@&H$`a$CJOJQJaJ0N@N Heading 5$7$8$@&H$`aJDA@D Default Paragraph FontVi@V  Table Normal :V 44 la (k@(No List DZ@D Plain TextCJOJQJ^JaJL^@L Normal (Web)dd[$\$ B*phPOPlJerome YesavageB* CJOJQJ^Jph6O"6 tdd[$\$ B*ph8O28 rldd[$\$ B*ph:OB: syndd[$\$ B*ph6U@Q6 Hyperlink >*B*ph:Ob: dt1dd[$\$ B*phBb@qB HTML CodeCJOJPJQJ^JaJe@ HTML Preformatted; 2( Px 4 #\'*.25@9!B*CJOJPJQJ^JaJph<C@< Body Text Indent&@& TOC 1.@. TOC 2 ^.@. TOC 3 ^.@. TOC 4 ^.@. TOC 5 ^.@. TOC 6 ^.@. TOC 7 ^.@. TOC 8 !^.@. TOC 9 "^*W@1* Strong5\FV@AF FollowedHyperlink >*B* ph>B@R> Body Text%0]0`XR@bX Body Text Indent 2& pp^p`LP@rL Body Text 2'7$8$H$`aJH@H Z&: Balloon Text(CJOJQJ^JaJ %2?Iefg)D=7,+)( %2?Iefgj)B5* pX8Z`zOB -  y % \ &(Lm`Mkl99F)0$ !c%&(()))M)a))***++,2,---./4//F000K11'2|2222y4'566G7t77778590:u;;;<^==D?v??ABCRDEFFOILzPuQQRS;SSyTUUVhVW,XaXn]R`z`lbcfxiGjkjjllnnn7o>oloprttuuwvyxxzyyzz{>|p|| }}*}+};}=}>}L}N}O}]}_}`}n}p}q}~}}}}}~nL̂͂7l؃+Bm -./^NrȊ Mi<=>?kÌČό$IύЍ=>CDbǎݎ TUj~ˏ0Fy~T̑ޑ'08ASew2RrsGՔd9ǖNOߗTU8Ǚș*8TYZbsʚϚКњҚ`,7<B]lyCҝӝ@Rdp{Ϟ3EWcnvw̟ٟ͟ڟ $%&'*00000000000000000000000000000000000000(0(&000000000(0(0(0(0(0l 0l 0l 0l0l(0l00000000000000000000(0l0F 0F 0F 0F0F0F0(00(000 0 0 0 0 0 0 %0 00M)0a)0M)(0*0*(0*0+(0*0,00-(0-0-0-0-0-0- 0- 0- 0- 0- 0-0-0-(0-0202020-0606060606 06 06 06 06 06 060+0909090+(0e=0=0=(0e=0A000000000D0D00000000000000000000g0g0g0g0g0k0k0k0k0k0k0k0k 0k 0k 0k 0k0k0k0k0k0k0k0k0k0k0k0k0k0k<0k00k00k0k00k00k0k00k00k0k00k00k0k00k00k0k0k0k0k0g0|0|0|0g0}0g0~000H00N0NH00000000000000000u0u0u 0u 0u000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000X00 0X00 80X000X000X0080X000X00X00X00X005* pX8(Lm`$ !c%&))++,2,-FFOILzPuQQRSn]R`z`lbcfxiGj*}+}=}>}N}O}_}`}p}q}}}*X00$X0*X00X00"X00"X00"X00"X00"X00"X00"X00"X00X00X00X00X00X00X00X00X00X00X00X0000X00hX0 0X0 0 X0 0 X0 0X0 0X0X0 0X0 0 X0 0X0 00X00X00(cX00X00~8 0\X00X0( X0(X0(X0(X0()1X0(X0(X0(X0(X0(X00X00X067\X00 X089X00 X0:;X00 X0<=X00 X0>?<X00 X0@AtX00 X00 $  y !naL!{?\E,5GCW`3lYoRs7wKϔ>Q(})UXYZ[\]^_`abcdfghijlqswz|}~"%M18:?0BD_n>w}Np̊rr`)Vekmnoprtuvxy{(W5MNn %'(*J  +Okmnp7SUVXx3568Xm9UWXZz"?[]^`?Yuwxz.JLMOo! = ? @ B b ( * + - M {  " W s v w y   " # % E x  - : V Y Z \ |  #$&QVRRpppRwrwzwwwx!xFxux) X%ĕX%ĕX%ĕX%ĕX%ĕX%ĕX%ĕX%ĕX%ĕX%ĕX%ĕX%ĕX%ĕX%ĕX%ĕX%ĕX%ĕX%ĕX%ĕX%ĕX%ĕX%ĕX%ĕX%ĕX%ĕX%ĕX%ĕX%ĕX%ĕX%ĕX%ĕX%ĕX%ĕX%ĕ̕XXTXTXTXT8DE@^ D (   ( N1?"`  ) N1?"` lB * <D8c? + N1?"`  , N1?"` :n  -3 #" lB . 0D8c?PfB / s *D8c?PPlB 0 0D8c?PfB 1 s *D8c?  P:n  23 #" lB 3 0D8c?PfB 4 s *D8c?PPlB 5 0D8c?PfB 6 s *D8c?  P 7 N1?"` lB : <D8c? = H1?"`   D H1?"`  B S  ?)(\t*ssVt) t-<t,1$t+$ t:Yt7t2t=tN!tD t}t _Toc529581209 _Toc529581354 _Toc529581417 _Toc530878726 _Toc207527162 _Toc529581355 _Toc529581418 _Toc530878727 _Toc207527163 _Toc529581210 _Toc529581356 _Toc529581419 _Toc530878728 _Toc207527164 _Toc529581211 _Toc529581357 _Toc529581420 _Toc530878729 _Toc207527165 _Toc529581212 _Toc529581358 _Toc529581421 _Toc530878730 _Toc207527166 _Toc530878731 _Toc207527167 _Toc530878732 _Toc207527168 _Toc529581213 _Toc529581359 _Toc529581422 _Toc530878733 _Toc207527169 _Toc529581214 _Toc529581360 _Toc529581423 _Toc530878734 _Toc207527170 _Toc529581215 _Toc529581361 _Toc529581424 _Toc530878735 _Toc207527171 _Toc529581216 _Toc529581362 _Toc529581425 _Toc530878736 _Toc207527172 _Toc529581217 _Toc529581363 _Toc529581426 _Toc530878737 _Toc207527173 _Toc529581218 _Toc529581364 _Toc529581427 _Toc530878738 _Toc207527174 _Toc529581219 _Toc529581365 _Toc529581428 _Toc530878739 _Toc207527175 _Toc207527176 _Toc529581220 _Toc529581366 _Toc529581429 _Toc530878740 _Toc207527177 _Toc529581221 _Toc529581367 _Toc529581430 _Toc530878741 _Toc207527178 _Toc529581222 _Toc529581368 _Toc529581431 _Toc530878742 _Toc207527179 _Toc529581223 _Toc529581369 _Toc529581432 _Toc530878743 _Toc207527180 _Toc529581224 _Toc529581370 _Toc529581433 _Toc530878744 _Toc207527181 _Toc529581225 _Toc529581371 _Toc529581434 _Toc530878745 _Toc207527182 _Toc529581226 _Toc529581372 _Toc529581435 _Toc530878746 _Toc207527183 _Toc529581228 _Toc529581374 _Toc529581437 _Toc530878748 _Toc207527184 _Toc529581229 _Toc529581375 _Toc529581438 _Toc530878749 _Toc207527185 _Toc529581230 _Toc529581376 _Toc529581439 _Toc530878750 _Toc207527186 _Hlt207525463 _Hlt207525464 _Toc207527187 _Toc529581231 _Toc529581377 _Toc529581440 _Toc530878751 _Toc207527188 _MailData _Toc529581232 _Toc529581378 _Toc529581441 _Toc530878752 _Toc207527189 _Toc529581233 _Toc529581442 _Toc530878753 _Toc207527190 _Toc529581234 _Toc529581379 _Toc529581443 _Toc530878754 _Toc207527191 _Toc529581235 _Toc529581380 _Toc529581444 _Toc530878755 _Toc207527192 _Toc207527193 _Toc529581236 _Toc529581381 _Toc529581445 _Toc530878756 _Toc207527194 _Toc207527195((((lllllFF M)M)M)M)M)a)a)a)a)a)**********+++++,---------------2222266666;;;;;D?D?D?D?D?v?v?v?v?v?BBBBBFFFFFRRS-X-X-X-X-XaXGjHjHjHjHjnnnnL͂΂΂΂΂0*  !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr@s@tuvwxyz{|}~44JKKKK88/////# # # # # !!!!!`)`)`)`)`))))))**********+++++1,----------.....2222266666<<<<<u?u?u?u?u??????CCCCCFFFFFRR:SSXSXSXSX`Xjjjjjjjjjjnnnn߀߀߀߀߀K]]]]]*FFFFdFF$FF!!D"""###$$` ` n ,"(((e*     l p p /"(((o * B*urn:schemas-microsoft-com:office:smarttagscountry-region8*urn:schemas-microsoft-com:office:smarttagstime? *urn:schemas-microsoft-com:office:smarttags stockticker8 *urn:schemas-microsoft-com:office:smarttagsdate9*urn:schemas-microsoft-com:office:smarttagsState9*urn:schemas-microsoft-com:office:smarttagsplace8*urn:schemas-microsoft-com:office:smarttagsCity>*urn:schemas-microsoft-com:office:smarttags PersonName H- 1012132001200229305DayHourMinuteMonthYear            !&H Q ""&&((e+h+..::<<E"E$E,EHFQFPP]]=a?aaaaaccllAojoloxopp qqiquqqqqqfrsrrrrrs(ssssssst t.tzJzM|X|||||}}}}}}}}}~R~`~~~~=IȂ͆ӆUWxz !"(+14:Ɍ͌όՌ+.5:?GOUX\_fmqw|Ǎ-:TXbefl}Ǝ͎׎!dhˏΏϏՏ !"()/6@Őǐ)-kr  kr}OW—̗ӗ՗ܗU]|;EKQem؝ݝFKX]Ǟ˞ў֞ !"&'09>KP*<@&FU$ v ++--..//}11115566<<AA C CGHH>M?MMMNNzS~SYY]]9aNa}aaii&kPklln6o>oAozy}yyyzz{{>|A|p|s|}}}}~~0 ֆ؆LjʈɊߊċ lsČȌόՌ+.INǍэ׍$>ADHbe͎׎ݎ UYjl~ˏΏ 6@FKy|pr !+.26<>ks}39=COWU]<Eə֙$-3;Eenv|Ědl"(/5EK`fou|GOӝםFLX^joszў֞9?KQ]bfm*333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333a)u;;ttiuuxx|}˟͟؟ڟ** !J"` '`mu2DV*o(()^`.pLp^p`L.@ @ ^@ `.^`.L^`L.^`.^`.PLP^P`L.hh^h`o(.hh^h`CJOJQJo(q^`CJOJQJo(^`CJOJQJo(opp^p`CJOJQJo(@ @ ^@ `CJOJQJo(^`CJOJQJo(^`CJOJQJo(^`CJOJQJo(^`CJOJQJo(PP^P`CJOJQJo(\V<!J"mu2PJjNI'.                 X        }L}N}O}]}_}`}n}p}q}~}}}*dgnword-docGUIDj dgnword-eventsinkZ&:2wThis READ_ME is designed to cover all aspects of our program designed to perform a number of signal detection functionsJerome Yesavageyesavage,        Oh+'0 ,DP\p     xThis READ_ME is designed to cover all aspects of our program designed to perform a number of signal detection functionsJerome Yesavage Normal.dot yesavage3Microsoft Office Word@q@r@V @՜.+,D՜.+,x hp  XDell Computer Corporation!Qo' xThis READ_ME is designed to cover all aspects of our program designed to perform a number of signal detection functions Title 8@ _PID_HLINKSA e../../../Administrator/Local Settings/Temporary Internet Files/011105 download/xlconCondFormulas.htmAK^../../../Administrator/Local Settings/Temporary Internet Files/011105 download/xlfctSUMIF.htm=?`../../../Administrator/Local Settings/Temporary Internet Files/011105 download/xlfctCOUNTIF.htmT\v../../../Administrator/Local Settings/Temporary Internet Files/011105 download/xlrefUnderstandinghowOperatorswork.htm Qhttp://video-server.stanford.edu:8080/ramgen/realcontent3/mirecc/ROC/trainer.smi6_Toc2075271956_Toc2075271946_Toc2075271936_Toc2075271926_Toc2075271916_Toc2075271906_Toc2075271896_Toc2075271886_Toc2075271876_Toc2075271866_Toc2075271856_Toc2075271846_Toc2075271836z_Toc2075271826t_Toc2075271816n_Toc2075271806h_Toc2075271796b_Toc2075271786\_Toc2075271776V_Toc2075271766P_Toc2075271756J_Toc2075271746D_Toc2075271736>_Toc20752717268_Toc20752717162_Toc2075271706,_Toc2075271696&_Toc2075271686 _Toc2075271676_Toc2075271666_Toc2075271656_Toc2075271646_Toc2075271636_Toc207527162  !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~     $Root Entry Fq&Data M+1TableWordDocument.BSummaryInformation(DocumentSummaryInformation8CompObjq  FMicrosoft Office Word Document MSWordDocWord.Document.89q