{"text_format":"1","workbook":"xlsx","n_sheets":7,"sheets":["0 Quick Start","1 Summary Table","2 Relevance Index","3 Credibility Index (Single)","3 Credibility Index","4 Background & References","Reference Table"],"not_rendered":{"charts":0,"images":0,"macros":1,"external_links":0,"defined_names":1,"comments":1,"pivot_tables":0},"unopened":[],"bounds":{"document_bytes":16777216,"sheet_cells":100000,"rows_first":2000,"rows_last":500,"rows_between":2000,"halvings":0,"cell_chars":16384,"cell_head_chars":4096},"cells_cut":0,"cells_raw":0,"read":"whole","date_system":"1900"}

{"sheet":"0 Quick Start","index":0,"n_rows":10,"n_cols":2,"rows_kept":10,"cut":false,"hidden":false,"merged":[[0,0,0,1]]}
["Relevance and Credibility Indices for Studies of Educational Interventions"]
["Hollands, F. M., Escueta, M.M., Pan, Y., Levin, H. M., Corter, J., & Wang, K. (2019). Relevance and credibility indices for studies of educational interventions. Teachers College, Columbia University."]
["Notes"]
[null,"These indices are currently designed to be used to assess studies which focus on single investigations rather than for studies which summarize the results of many investigations, such as literature reviews or meta-analyses. \n\nWhile such summaries are useful for providing a general idea of “what works,” it would be hard to assess the relevance of an overview study to your own context or to evaluate the credibility of all studies at once.  \n\nYou may wish to identify individual studies included in an overview to which you can apply these indices."]
["Steps to follow"]
[1,"Identify studies you want to evaluate for relevance and credibility."]
[2,"List each study in the tab “1 Summary Table”."]
[3,"Go to the tab “2 Relevance Index” (RI) and complete the worksheet for each study."]
[4,"Go to the tab “3 Credibility Index” (CI) and complete the worksheet for any studies that you found relevant."]
[5,"Return to “1 Summary Table” to review overall results."]

{"sheet":"1 Summary Table","index":1,"n_rows":17,"n_cols":14,"rows_kept":17,"cut":false,"hidden":false,"merged":[[0,0,0,3],[9,0,9,3],[1,0,1,3],[3,0,3,3],[4,0,4,3],[5,0,5,3],[6,0,6,3],[7,0,7,3],[8,0,8,3],[2,0,2,3]]}
["Relevance and Credibility Indices for Studies of Educational Interventions"]
["Instructions (Click add/subtract button in the left-hand margin to expand or collapse the detailed instructions)"]
["1. For first time users, review tab 4 \"Background & References\""]
["2. Fill in Columns B-D with information about the studies you want to evaluate. Column A is optional."]
["3. Proceed to Relevance Index tab and complete the worksheet for each study."]
["4. Return to Summary Table tab. Columns E-G in Summary Table will self-populate based on your inputs in Relevance Index tab."]
["5. In Column H, use the dropdown to indicate whether you want to use this study as relevant evidence."]
["6. Proceed to Credibility Index tab and complete the worksheet for any studies that you found relevant."]
["7. Return to Summary Table tab. Columns I-K will populate based on your inputs in the Credibility Index."]
["8. (Optional) Enter each study's effect size, and decide whether you want to adjust the effect size found in the study for your context using our suggested adjustment factor (“Credibility Parameter”). "]
[]
["Full name of the study \n(Optional)","Nickname of the study\n(This will appear in RI & CI tab as reference while you fill in the worksheets)","Authors","Year of study","Relevance Index","Relevance rating","Recommendation on proceeding to evaluate CI?","Use study as relevant evidence for decision?","Credibility Index","Credibility parameter","Credibility rating","Effect size reported in study","Do you want to adjust the effect size?","Adjusted effect size"]
[null,null,null,null,{"$f":"'2 Relevance Index'!H34","v":""},{"$f":"'2 Relevance Index'!G35","v":""},{"$f":"IF(OR(F13=\"Moderate Relevance\",F13=\"High Relevance\"),\"Yes\",IF(F13=\"Low Relevance\",\"No\",\"\"))","v":""},null,{"$f":"IF('3 Credibility Index'!C66=0,\"\",'3 Credibility Index'!C66)","v":""},{"$f":"IF(I13<10,0.2,IF(I13<20,0.6,IF(I13<=27,1,IF(I13=\"\",\"\",\"\"))))","v":""},{"$f":"IF(J13<=0.2,\"Low Credibility\",IF(J13<=0.6,\"Moderate Credibility\",IF(J13<=1,\"High Credibility\",IF(J13=\"\",\"\"))))","v":""},null,null,{"$f":"IF(M13=\"Yes\",L13*J13,IF(M13=\"No\",\"N.A.\",\"\"))","v":""}]
[null,null,null,null,{"$f":"'2 Relevance Index'!L34","v":""},{"$f":"'2 Relevance Index'!K35","v":""},{"$f":"IF(OR(F14=\"Moderate Relevance\",F14=\"High Relevance\"),\"Yes\",IF(F14=\"Low Relevance\",\"No\",\"\"))","v":""},null,{"$f":"IF('3 Credibility Index'!E66=0,\"\",'3 Credibility Index'!E66)","v":""},{"$f":"IF(I14<10,0.2,IF(I14<20,0.6,IF(I14<=27,1,IF(I14=\"\",\"\",\"\"))))","v":""},{"$f":"IF(J14<=0.2,\"Low Credibility\",IF(J14<=0.6,\"Moderate Credibility\",IF(J14<=1,\"High Credibility\",IF(J14=\"\",\"\"))))","v":""},null,null,{"$f":"IF(M14=\"Yes\",L14*J14,IF(M14=\"No\",\"N.A.\",\"\"))","v":""}]
[null,null,null,null,{"$f":"'2 Relevance Index'!P34","v":""},{"$f":"'2 Relevance Index'!O35","v":""},{"$f":"IF(OR(F15=\"Moderate Relevance\",F15=\"High Relevance\"),\"Yes\",IF(F15=\"Low Relevance\",\"No\",\"\"))","v":""},null,{"$f":"IF('3 Credibility Index'!G66=0,\"\",'3 Credibility Index'!G66)","v":""},{"$f":"IF(I15<10,0.2,IF(I15<20,0.6,IF(I15<=27,1,IF(I15=\"\",\"\",\"\"))))","v":""},{"$f":"IF(J15<=0.2,\"Low Credibility\",IF(J15<=0.6,\"Moderate Credibility\",IF(J15<=1,\"High Credibility\",IF(J15=\"\",\"\"))))","v":""},null,null,""]
[null,null,null,null,{"$f":"'2 Relevance Index'!T34","v":""},{"$f":"'2 Relevance Index'!S35","v":""},{"$f":"IF(OR(F16=\"Moderate Relevance\",F16=\"High Relevance\"),\"Yes\",IF(F16=\"Low Relevance\",\"No\",\"\"))","v":""},null,{"$f":"IF('3 Credibility Index'!I66=0,\"\",'3 Credibility Index'!I66)","v":""},"","",null,null,""]
[null,null,null,null,{"$f":"'2 Relevance Index'!X34","v":""},{"$f":"'2 Relevance Index'!W35","v":""},{"$f":"IF(OR(F17=\"Moderate Relevance\",F17=\"High Relevance\"),\"Yes\",IF(F17=\"Low Relevance\",\"No\",\"\"))","v":""},null,{"$f":"IF('3 Credibility Index'!K66=0,\"\",'3 Credibility Index'!K66)","v":""},"","",null,null,""]

{"sheet":"2 Relevance Index","index":2,"n_rows":36,"n_cols":25,"rows_kept":36,"cut":false,"hidden":false,"merged":[[35,10,35,12],[31,14,32,14],[34,0,34,3],[13,0,18,0],[19,0,25,0],[31,2,32,2],[31,0,32,1],[31,15,32,15],[34,14,34,16],[10,0,10,4],[31,4,32,4],[31,6,32,6],[31,8,32,8],[10,10,10,12],[31,7,32,7],[31,11,32,11],[28,0,30,0],[10,6,10,8],[34,6,34,8],[31,10,32,10],[31,12,32,12],[35,22,35,24],[34,10,34,12],[10,14,10,16],[31,16,32,16],[35,14,35,16],[10,18,10,20],[31,18,32,18],[31,19,32,19],[31,20,32,20],[34,18,34,20],[35,18,35,20],[10,22,10,24],[31,22,32,22],[31,23,32,23],[31,24,32,24],[34,22,34,24],[0,0,0,4],[1,0,1,4],[35,0,35,4],[2,0,2,4],[3,0,3,4],[4,0,4,4],[5,0,5,4],[6,0,6,4],[7,0,7,4],[8,0,8,4],[9,0,9,1]]}
["Relevance and Credibility Indices for Studies of Educational Interventions"]
["Instructions (Click add/subtract button in the left-hand margin to expand or collapse the detailed instructions) "]
["1. Review factors listed in Column B and identify around 5 or 6 important considerations for the specific decision you are using these studies to make."]
["2. Use the dropdown menu in the cells in Column C to indicate how important each of these factors is for this decision context."]
["3. (Optional) Enter details about this factor from your own context in Column E."]
["4. (Optional) Enter information provided in the study about this factor in Column G for the first study."]
["5. Use Column H to indicate how similar the study's context is to yours on this factor."]
["6. Return to Summary Table tab and decide whether you want to use this study as evidence for this decision."]
["7. Repeat steps 4 to 6 for each additional study you review."]
["For a detailed explanation of the calculations used in the RI, click here."]
["General Information",null,null,null,null,null,{"$f":"IF('1 Summary Table'!B13=\"\",\"Please leave this table blank\",'1 Summary Table'!B13)","v":"Please leave this table blank"},null,null,null,{"$f":"IF('1 Summary Table'!B14=\"\",\"Please leave this table blank\",'1 Summary Table'!B14)","v":"Please leave this table blank"},null,null,null,{"$f":"IF('1 Summary Table'!B15=\"\",\"Please leave this table blank\",'1 Summary Table'!B15)","v":"Please leave this table blank"},null,null,null,{"$f":"IF('1 Summary Table'!B16=\"\",\"Please leave this table blank\",'1 Summary Table'!B16)","v":"Please leave this table blank"},null,null,null,{"$f":"IF('1 Summary Table'!B17=\"\",\"Please leave this table blank\",'1 Summary Table'!B17)","v":"Please leave this table blank"}]
["Contextual factor","Things to look for and think about","Important consideration for me* ","Important consideration for me ","Details of my own context",null,"Details of the study","Study context is similar to mine","Study context is similar to mine",null,"Details of the study","Study context is similar to mine","Study context is similar to mine",null,"Details of the study","Study context is similar to mine","Study context is similar to mine",null,"Details of the study","Study context is similar to mine","Study context is similar to mine",null,"Details of the study","Study context is similar to mine","Study context is similar to mine"]
["Recency of study","Extent to which the educational activity, strategy or intervention (Solution Option) is still likely to be applicable in my context today","not at all/unavailable",{"$f":"IF(C13=\"very\",3,IF(C13=\"moderately\",2,IF(C13=\"slightly\",1,IF(C13=\"not at all/unavailable\",0))))","v":0},null,null,null,null,{"$f":"IF(H13=\"very\",3,IF(H13=\"moderately\",2,IF(H13=\"slightly\",1,IF(H13=\"not at all/unavailable\",0))))","v":false},null,null,null,{"$f":"IF(L13=\"very\",3,IF(L13=\"moderately\",2,IF(L13=\"slightly\",1,IF(L13=\"not at all/unavailable\",0))))","v":false},null,null,null,{"$f":"IF(P13=\"very\",3,IF(P13=\"moderately\",2,IF(P13=\"slightly\",1,IF(P13=\"not at all/unavailable\",0))))","v":false},null,null,null,{"$f":"IF(T13=\"very\",3,IF(T13=\"moderately\",2,IF(T13=\"slightly\",1,IF(T13=\"not at all/unavailable\",0))))","v":false},null,null,null,{"$f":"IF(X13=\"very\",3,IF(X13=\"moderately\",2,IF(X13=\"slightly\",1,IF(X13=\"not at all/unavailable\",0))))","v":false}]
["Student demographics","Age of students on which the program/ intervention was tested","not at all/unavailable",{"$f":"IF(C14=\"very\",3,IF(C14=\"moderately\",2,IF(C14=\"slightly\",1,IF(C14=\"not at all/unavailable\",0))))","v":0},null,null,null,null,{"$f":"IF(H14=\"very\",3,IF(H14=\"moderately\",2,IF(H14=\"slightly\",1,IF(H14=\"not at all/unavailable\",0))))","v":false},null,null,null,{"$f":"IF(L14=\"very\",3,IF(L14=\"moderately\",2,IF(L14=\"slightly\",1,IF(L14=\"not at all/unavailable\",0))))","v":false},null,null,null,{"$f":"IF(P14=\"very\",3,IF(P14=\"moderately\",2,IF(P14=\"slightly\",1,IF(P14=\"not at all/unavailable\",0))))","v":false},null,null,null,{"$f":"IF(T14=\"very\",3,IF(T14=\"moderately\",2,IF(T14=\"slightly\",1,IF(T14=\"not at all/unavailable\",0))))","v":false},null,null,null,{"$f":"IF(X14=\"very\",3,IF(X14=\"moderately\",2,IF(X14=\"slightly\",1,IF(X14=\"not at all/unavailable\",0))))","v":false}]
[null,"Baseline performance of students before implementing the program","not at all/unavailable",0,null,null,null,null,false,null,null,null,false,null,null,null,false,null,null,null,false,null,null,null,false]
[null,"Percent economically disadvantaged, e.g., as indicated by FRPL","not at all/unavailable",0,null,null,null,null,false,null,null,null,false,null,null,null,false,null,null,null,false,null,null,null,false]
[null,"Percent classified as minority","not at all/unavailable",0,null,null,null,null,false,null,null,null,false,null,null,null,false,null,null,null,false,null,null,null,false]
[null,"Percent ELL","not at all/unavailable",0,null,null,null,null,false,null,null,null,false,null,null,null,false,null,null,null,false,null,null,null,false]
[null,"Percent Special Education","not at all/unavailable",0,null,null,null,null,false,null,null,null,false,null,null,null,false,null,null,null,false,null,null,null,false]
["School context","Charter school vs. district school vs. private school","not at all/unavailable",0,null,null,null,null,false,null,null,null,false,null,null,null,false,null,null,null,false,null,null,null,false]
[null,"Selective admissions vs. open to all applicants","not at all/unavailable",0,null,null,null,null,false,null,null,null,false,null,null,null,false,null,null,null,false,null,null,null,false]
[null,"Qualifications of teachers (e.g., degree level, certification, tenure status, average experience)","not at all/unavailable",0,null,null,null,null,false,null,null,null,false,null,null,null,false,null,null,null,false,null,null,null,false]
[null,"Availability of necessary staff","not at all/unavailable",0,null,null,null,null,false,null,null,null,false,null,null,null,false,null,null,null,false,null,null,null,false]
[null,"Availability of necessary technology ","not at all/unavailable",0,null,null,null,null,false,null,null,null,false,null,null,null,false,null,null,null,false,null,null,null,false]
[null,"Alignment with School Improvement Plan ","not at all/unavailable",0,null,null,null,null,false,null,null,null,false,null,null,null,false,null,null,null,false,null,null,null,false]
[null,"Urban, rural or suburban","not at all/unavailable",0,null,null,null,null,false,null,null,null,false,null,null,null,false,null,null,null,false,null,null,null,false]
["Relevance of measure used","Whether the outcome measure reported in the study is relevant for your goals, e.g., if you are trying to reduce behavior incidents, a study that reports suspension rates might be relevant, but one that reports attendance might be less relevant","not at all/unavailable",0,null,null,null,null,false,null,null,null,false,null,null,null,false,null,null,null,false,null,null,null,false]
["Scale","Is the scale at which the study was conducted similar to your context and needs (e.g., if you want to implement a reading program at large scale across several schools, did the study look at the effect of the intervention across many schools, or only at small scale in one school?)","not at all/unavailable",0,null,null,null,null,false,null,null,null,false,null,null,null,false,null,null,null,false,null,null,null,false]
["Additional factors that you want to consider",null,"not at all/unavailable",0,null,null,null,null,false,null,null,null,false,null,null,null,false,null,null,null,false,null,null,null,false]
[null,null,"not at all/unavailable",0,null,null,null,null,false,null,null,null,false,null,null,null,false,null,null,null,false,null,null,null,false]
[null,null,"not at all/unavailable",0,null,null,null,null,false,null,null,null,false,null,null,null,false,null,null,null,false,null,null,null,false]
["Total possible score (Y) = Total importance score x 3 points =",null,{"$f":"SUM(D13:D31)*3","v":0},null,null,null,"Total points you gave this study (X):  ",{"$f":"SUMPRODUCT($D$13:$D$31,I13:I31)","v":0},null,null,"Total points you gave this study (X):  ",{"$f":"SUMPRODUCT($D$13:$D$31,M13:M31)","v":0},null,null,"Total points you gave this study (X):  ",{"$f":"SUMPRODUCT($D$13:$D$31,Q13:Q31)","v":0},null,null,"Total points you gave this study (X):  ",{"$f":"SUMPRODUCT($D$13:$D$31,U13:U31)","v":0},null,null,"Total points you gave this study (X):  ",{"$f":"SUMPRODUCT($D$13:$D$31,Y13:Y31)","v":0}]
[]
[null,null,null,null,null,null,"Relevance Index = ",{"$f":"IFERROR(IF(H32=0,\"\",H32/$C$32),\"\")","v":""},null,null,"Relevance Index = ",{"$f":"IFERROR(IF(L32=0,\"\",L32/$C$32),\"\")","v":""},null,null,"Relevance Index = ",{"$f":"IFERROR(IF(P32=0,\"\",P32/$C$32),\"\")","v":""},null,null,"Relevance Index = ",{"$f":"IFERROR(IF(T32=0,\"\",T32/$C$32),\"\")","v":""},null,null,"Relevance Index = ",{"$f":"IFERROR(IF(X32=0,\"\",X32/$C$32),\"\")","v":""}]
[null,null,null,null,null,null,{"$f":"IF(H34<=0.3,'Reference Table'!$G$4,IF(H34<0.7,'Reference Table'!$G$5,IF(H34<=1,'Reference Table'!$G$6,\"\")))","v":""},null,null,null,{"$f":"IF(L34<=0.3,'Reference Table'!$G$4,IF(L34<0.7,'Reference Table'!$G$5,IF(L34<=1,'Reference Table'!$G$6,\"\")))","v":""},null,null,null,{"$f":"IF(P34<=0.3,'Reference Table'!$G$4,IF(P34<0.7,'Reference Table'!$G$5,IF(P34<=1,'Reference Table'!$G$6,\"\")))","v":""},null,null,null,{"$f":"IF(T34<=0.3,'Reference Table'!$G$4,IF(T34<0.7,'Reference Table'!$G$5,IF(T34<=1,'Reference Table'!$G$6,\"\")))","v":""},null,null,null,{"$f":"IF(X34<=0.3,'Reference Table'!$G$4,IF(X34<0.7,'Reference Table'!$G$5,IF(X34<=1,'Reference Table'!$G$6,\"\")))","v":""}]
["*Weights are assigned to these factors as follows: Very = 3; Moderately = 2; Slightly = 1; Not at all = 0"]

{"sheet":"3 Credibility Index (Single)","index":3,"n_rows":72,"n_cols":4,"rows_kept":72,"cut":false,"hidden":true,"merged":[[62,0,62,1],[65,0,65,1],[66,0,68,0],[34,0,38,0],[40,0,42,0],[44,0,47,0],[48,0,48,1],[49,0,51,0],[52,0,52,1],[39,0,39,1],[63,0,64,0],[43,0,43,1],[53,0,55,0],[56,0,61,0],[33,0,33,1],[30,0,32,0],[0,0,0,1],[1,0,1,1],[2,0,2,1],[3,0,3,1],[10,0,10,1],[15,0,15,1],[20,0,20,1],[25,0,25,1],[29,0,29,1],[11,0,14,0],[16,0,19,0],[21,0,24,0],[26,0,28,0]]}
["Instructions (Click add/subtract button in the left-hand margin to expand or collapse the detailed instructions)"]
["1. For each question, enter the relevant details about the study in Column A."]
["2. Select ONE statement per question based on the details of the study."]
["3. Complete Steps 1 and 2 for Q1 through Q12."]
[]
[]
[]
[]
[]
["Details of the Study","What to look for and think about"]
["Q1. Who conducted the study?",null,1,{"$f":"C11-2","v":-1}]
[]
[]
[]
[]
["Q2. Who published the study?",null,0,{"$f":"C16-2","v":-2}]
[]
[]
[]
[]
["Q3. Length of time participants are exposed to the educational program/strategy being studied",null,0,{"$f":"C21-2","v":-2}]
[]
[]
[]
[]
["Q4. Components of the program: Is it clear what program participants were expected to do compared with business as usual conditions?",null,0,{"$f":"IF(C26=1,-1,IF(C26=2,1,2))","v":2}]
[]
[]
[]
["Q5. Implementation of the program:  Is it clear how the program was implemented in the study, e.g., how much time participants actually spent engaging in the program activities, how often, whether it was implemented in/out of the classroom, before/ after school hours etc.?",null,0,{"$f":"IF(C30=1,-1,IF(C30=2,1,2))","v":2}]
[]
[]
[]
["Q6. Meaningful outcomes",null,0,{"$f":"C34-2","v":-2}]
[]
[]
[]
[]
[]
["Q7. Sample characteristics: Are the characteristics of the study participants clear? (e.g., grade, gender, FRPL, ELL and Special Education status). ",null,0,{"$f":"IF(C40=1,-1,IF(C40=2,1,2))","v":2}]
[]
[]
[]
["Q8. Sample size: Does the size of the sample, i.e., the number of participants in the study, seem adequate? (For researchers: do you think there is enough power to detect an effect if indeed there is one?)",null,0,{"$f":"C44-2","v":-2}]
[]
[]
[]
[]
["Q9. First determine whether there is a comparison group of any kind.",null,0,{"$f":"IF(C49=1,-1,IF(C49=2,-1,1))","v":1}]
[]
[]
[]
["Q10. Then, identify how that comparison group was selected. (only answer 10a. and 10b.  if the study includes a comparison group)"]
[null,"10a. Clarity of selection process for program v. comparison groups.",0,{"$f":"C54-2","v":-2}]
[]
[]
[null,"10b. Strength of study design to ensure the comparison group is similar to the program participants",2,{"$f":"IF(C57=1,0,IF(C57=2,1,IF(C57=3,3,IF(C57=4,5,IF(C57=5,6)))))","v":1}]
[]
[]
[]
[]
[]
["Q11. Addressing other possible explanations for the results: If the study does not assign students/teachers/schools at random to participate in the program or in the comparison group, do the authors attempt to account for, or at least discuss the possibility of, other factors besides the treatment explaining the difference in outcomes between treatment and control groups?",null,2,{"$f":"C63-1","v":1}]
[]
[]
["Q12. Measuring outcomes over time",null,0,{"$f":"C66-1","v":-1}]
[]
[null,null,null,{"$f":"IF(D71<=0.2,\"Low Credibility\",IF(D71<=0.6,\"Moderate Credibility\",IF(D71<=1,\"High Credibility\",IF(D71=\"\",\"\"))))","v":""}]
[]
["Points earned:",{"$f":"IF(COUNTIF(C11:C69,\">0\")<13,\"\",SUM(D11:D69))","v":""},"CI:",{"$f":"IF(COUNTIF(C11:C69,\">0\")<13,\"\",SUM(D11:D69))","v":""}]
["Max points:",27,"CP:",{"$f":"IF(D70<10,'Reference Table'!Q4,IF(D70<20,'Reference Table'!Q5,IF(D70=\"\",\"\",'Reference Table'!Q6)))","v":""}]
[null,null,"CR:",{"$f":"IF(D71<=0.2,\"Low Credibility\",IF(D71<=0.6,\"Moderate Credibility\",IF(D71<=1,\"High Credibility\",IF(D71=\"\",\"\"))))","v":""}]

{"sheet":"3 Credibility Index","index":4,"n_rows":68,"n_cols":83,"rows_kept":68,"cut":false,"hidden":false,"merged":[]}
["Relevance and Credibility Indices for Studies of Educational Interventions"]
["Instructions (Click add/subtract button in the left-hand margin to expand or collapse the detailed instructions)"]
["1. For each question below (Q1-11), select one relevant statement that most accurately reflects the study you are reviewing and click on the corresponding radio button."]
["2. The score earned by the study for each question will appear in the blue band at the top of each set of radio buttons."]
["3. Repeat steps 1-2 for each additional study you want to evaluate for credibility."]
["For a detailed explanation of the calculations used in the CI, click here."]
[null,null,{"$f":"IF('1 Summary Table'!B13=\"\",\"\",'1 Summary Table'!B13)","v":""},null,{"$f":"IF('1 Summary Table'!B14=\"\",\"\",'1 Summary Table'!B14)","v":""},null,{"$f":"IF('1 Summary Table'!B15=\"\",\"\",'1 Summary Table'!B15)","v":""},null,{"$f":"IF('1 Summary Table'!B16=\"\",\"\",'1 Summary Table'!B16)","v":""},null,{"$f":"IF('1 Summary Table'!B17=\"\",\"\",'1 Summary Table'!B17)","v":""}]
["Q1. Who conducted the study?",null,{"$f":"IF(BX8=0,\"blank\",IF(BX8=1,-1,IF(BX8=2,0,IF(BX8=3,1,IF(BX8=4,2)))))","v":"blank"},null,{"$f":"IF(BY8=0,\"blank\",IF(BY8=1,-1,IF(BY8=2,0,IF(BY8=3,1,IF(BY8=4,2)))))","v":"blank"},null,{"$f":"IF(CA8=0,\"blank\",IF(CA8=1,-1,IF(CA8=2,0,IF(CA8=3,1,IF(CA8=4,2)))))","v":"blank"},null,{"$f":"IF(CC8=0,\"blank\",IF(CC8=1,-1,IF(CC8=2,0,IF(CC8=3,1,IF(CC8=4,2)))))","v":"blank"},null,{"$f":"IF(CE8=0,\"blank\",IF(CE8=1,-1,IF(CE8=2,0,IF(CE8=3,1,IF(CE8=4,2)))))","v":"blank"},null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,0,0,null,0,null,0,null,0]
["It is not clear who conducted the study. "]
["The study was conducted by the program vendor."]
["The study was conducted by an external evaluator hired by the program vendor."]
["The study was conducted by an external evaluator acting as an independent third party, i.e., not paid by the vendor. This may include collaborations between the evaluator and implementing partners such as school districts or other educational or research institutions, but excluding the program vendor. The key distinction is that those conducting and/or commissioning the evaluation should not stand to make a profit from the sale of the product/ intervention being tested."]
["Q2. Who published the study?",null,{"$f":"IF(BX13=0,\"blank\",IF(BX13=1,-1,IF(BX13=2,0,IF(BX13=3,1,IF(BX13=4,2)))))","v":"blank"},null,{"$f":"IF(BY13=0,\"blank\",IF(BY13=1,-1,IF(BY13=2,0,IF(BY13=3,1,IF(BY13=4,2)))))","v":"blank"},null,{"$f":"IF(CA13=0,\"blank\",IF(CA13=1,-1,IF(CA13=2,0,IF(CA13=3,1,IF(CA13=4,2)))))","v":"blank"},null,{"$f":"IF(CC13=0,\"blank\",IF(CC13=1,-1,IF(CC13=2,0,IF(CC13=3,1,IF(CC13=4,2)))))","v":"blank"},null,{"$f":"IF(CE13=0,\"blank\",IF(CE13=1,-1,IF(CE13=2,0,IF(CE13=3,1,IF(CE13=4,2)))))","v":"blank"},null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,0,0,null,0,null,0,null,0]
["It is not clear where the study was published. "]
["The study was published by the vendor."]
["The study was published by a third party (i.e., other than the vendor) but not a peer-reviewed journal, e.g., a university, research organization, school district or other government agency. Keep in mind some technical reports are also later published in a peer-reviewed journal, so you may wish to check Google Scholar to see if there is a later published version."]
["The study was published in a peer-reviewed journal. "]
["Q3. Length of time participants are exposed to the educational program/strategy being studied",null,{"$f":"IF(BX18=0,\"blank\",IF(BX18=1,-1,IF(BX18=2,0,IF(BX18=3,1,IF(BX18=4,2)))))","v":"blank"},null,{"$f":"IF(BY18=0,\"blank\",IF(BY18=1,-1,IF(BY18=2,0,IF(BY18=3,1,IF(BY18=4,2)))))","v":"blank"},null,{"$f":"IF(CA18=0,\"blank\",IF(CA18=1,-1,IF(CA18=2,0,IF(CA18=3,1,IF(CA18=4,2)))))","v":"blank"},null,{"$f":"IF(CC18=0,\"blank\",IF(CC18=1,-1,IF(CC18=2,0,IF(CC18=3,1,IF(CC18=4,2)))))","v":"blank"},null,{"$f":"IF(CE18=0,\"blank\",IF(CE18=1,-1,IF(CE18=2,0,IF(CE18=3,1,IF(CE18=4,2)))))","v":"blank"},null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,0,0,null,0,null,0,null,0]
["Length of exposure is not clear from the study. "]
["Length of exposure is too short to make a difference."]
["Length of exposure is too long to reflect likely effect in regular practice."]
["Length of exposure is about right."]
["Q4. Components of the program: Is it clear what program participants were supposed to do compared with business as usual conditions?",null,{"$f":"IF(BX23=0,\"blank\",IF(BX23=1,-1,IF(BX23=2,1,IF(BX23=3,2))))","v":"blank"},null,{"$f":"IF(BY23=0,\"blank\",IF(BY23=1,-1,IF(BY23=2,1,IF(BY23=3,2))))","v":"blank"},null,{"$f":"IF(CA23=0,\"blank\",IF(CA23=1,-1,IF(CA23=2,1,IF(CA23=3,2))))","v":"blank"},null,{"$f":"IF(CC23=0,\"blank\",IF(CC23=1,-1,IF(CC23=2,1,IF(CC23=3,2))))","v":"blank"},null,{"$f":"IF(CE23=0,\"blank\",IF(CE23=1,-1,IF(CE23=2,1,IF(CE23=3,2))))","v":"blank"},null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,0,0,null,0,null,0,null,0]
["The components of the program/treatment are not clear."]
["The components of the program/treatment are partially clear."]
["The components of the program/treatment are completely clear."]
["Q5. Implementation of the program:  How clear are the details of implementation (dosage, frequency, whether it was implemented in/out of the classroom, before/ after school hours etc.)?",null,{"$f":"IF(BX27=0,\"blank\",IF(BX27=1,-1,IF(BX27=2,1,IF(BX27=3,2))))","v":"blank"},null,{"$f":"IF(BY27=0,\"blank\",IF(BY27=1,-1,IF(BY27=2,1,IF(BY27=3,2))))","v":"blank"},null,{"$f":"IF(CA27=0,\"blank\",IF(CA27=1,-1,IF(CA27=2,1,IF(CA27=3,2))))","v":"blank"},null,{"$f":"IF(CC27=0,\"blank\",IF(CC27=1,-1,IF(CC27=2,1,IF(CC27=3,2))))","v":"blank"},null,{"$f":"IF(CE27=0,\"blank\",IF(CE27=1,-1,IF(CE27=2,1,IF(CE27=3,2))))","v":"blank"},null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,0,0,null,0,null,0,null,0]
["The implementation details of the program are not clear.",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,1]
["The implementation details of the program are partially clear."]
["The implementation details of the program are completely clear."]
["Q6. Meaningful outcomes",null,{"$f":"IF(BX31=0,\"blank\",IF(BX31=1,-1,IF(BX31=2,0,IF(BX31=3,1,IF(BX31=4,2,IF(BX31=5,3))))))","v":"blank"},null,{"$f":"IF(BY31=0,\"blank\",IF(BY31=1,-1,IF(BY31=2,0,IF(BY31=3,1,IF(BY31=4,2,IF(BY31=5,3))))))","v":"blank"},null,{"$f":"IF(CA31=0,\"blank\",IF(CA31=1,-1,IF(CA31=2,0,IF(CA31=3,1,IF(CA31=4,2,IF(CA31=5,3))))))","v":"blank"},null,{"$f":"IF(CC31=0,\"blank\",IF(CC31=1,-1,IF(CC31=2,0,IF(CC31=3,1,IF(CC31=4,2,IF(CC31=5,3))))))","v":"blank"},null,{"$f":"IF(CE31=0,\"blank\",IF(CE31=1,-1,IF(CE31=2,0,IF(CE31=3,1,IF(CE31=4,2,IF(CE31=5,3))))))","v":"blank"},null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,0,0,null,0,null,0,null,0]
["It is not clear which outcomes are being measured, e.g., it is not clear whether the study is evaluating geometry skills or algebra skills."]
["The outcomes measured are not at all aligned with the ultimate goal for implementing the intervention, e.g., the study investigates whether an after-school supplemental math program improves geometry skills, despite the fact that the program aims to improve algebra skills."]
["The outcomes measured only capture short-term behavioral changes but not longer-term educational outcomes, e.g., the study only documents whether students are attending an after-school math program, but does not measure whether their math skills are improving."]
["The outcomes measured are aligned with some but not all of the stated goals for implementing the intervention, e.g., the study is measuring algebra skills when the primary goal of the program is to improve both algebra and geometry skills."]
["The outcomes measured are aligned with all of the stated goals for implementing the intervention, e.g., the study is measuring algebra skills when improving algebra skills is the primary goal of the program."]
["Q7. Sample characteristics: Are the characteristics of the sample clear?",null,{"$f":"IF(BX37=0,\"blank\",IF(BX37=1,-1,IF(BX37=2,1,IF(BX37=3,2))))","v":"blank"},null,{"$f":"IF(BY37=0,\"blank\",IF(BY37=1,-1,IF(BY37=2,1,IF(BY37=3,2))))","v":"blank"},null,{"$f":"IF(CA37=0,\"blank\",IF(CA37=1,-1,IF(CA37=2,1,IF(CA37=3,2))))","v":"blank"},null,{"$f":"IF(CC37=0,\"blank\",IF(CC37=1,-1,IF(CC37=2,1,IF(CC37=3,2))))","v":"blank"},null,{"$f":"IF(CE37=0,\"blank\",IF(CE37=1,-1,IF(CE37=2,1,IF(CE37=3,2))))","v":"blank"},null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,0,0,null,0,null,0,null,0]
["The characteristics of the study sample are not clear. "]
["The characteristics of the study sample are moderately clear."]
["The characteristics of the study sample are very clear."]
["Q8. Sample size: Does the size of the sample, i.e., the number of participants in the study, seem adequate? (For researchers: do you think there is enough power to detect an effect if indeed there is one?)",null,{"$f":"IF(BX41=0,\"blank\",IF(BX41=1,-1,IF(BX41=2,0,IF(BX41=3,1,IF(BX41=4,2)))))","v":"blank"},null,{"$f":"IF(BY41=0,\"blank\",IF(BY41=1,-1,IF(BY41=2,0,IF(BY41=3,1,IF(BY41=4,2)))))","v":"blank"},null,{"$f":"IF(CA41=0,\"blank\",IF(CA41=1,-1,IF(CA41=2,0,IF(CA41=3,1,IF(CA41=4,2)))))","v":"blank"},null,{"$f":"IF(CC41=0,\"blank\",IF(CC41=1,-1,IF(CC41=2,0,IF(CC41=3,1,IF(CC41=4,2)))))","v":"blank"},null,{"$f":"IF(CE41=0,\"blank\",IF(CE41=1,-1,IF(CE41=2,0,IF(CE41=3,1,IF(CE41=4,2)))))","v":"blank"},null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,0,0,null,0,null,0,null,0]
["The size of the study sample is unclear."]
["The study does not have an adequate number of participants."]
["The study had a fairly adequate number of participants."]
["The study has a very adequate number of participants."]
["Q9. First determine whether there is a comparison group of any kind.",null,{"$f":"IF(BX46=0,\"blank\",IF(BX46=1,-1,IF(BX46=2,-1,IF(BX46=3,1))))","v":"blank"},null,{"$f":"IF(BY46=0,\"blank\",IF(BY46=1,-1,IF(BY46=2,-1,IF(BY46=3,1))))","v":"blank"},null,{"$f":"IF(CA46=0,\"blank\",IF(CA46=1,-1,IF(CA46=2,-1,IF(CA46=3,1))))","v":"blank"},null,{"$f":"IF(CC46=0,\"blank\",IF(CC46=1,-1,IF(CC46=2,-1,IF(CC46=3,1))))","v":"blank"},null,{"$f":"IF(CE46=0,\"blank\",IF(CE46=1,-1,IF(CE46=2,-1,IF(CE46=3,1))))","v":"blank"},null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,0,0,null,0,null,0,null,0]
["It is not clear whether there is a comparison group."]
["There is no comparison group."]
["Study includes comparison group which does not participate in the program being studied."]
["Q10. Then, identify how that comparison group was selected. (only answer 10a. and 10b.  if the study includes a comparison group)"]
["10a. Clarity of selection process for program v. comparison groups.",null,{"$f":"IF(BX51=0,\"blank\",IF(BX51=1,-1,IF(BX51=2,0)))","v":"blank"},null,{"$f":"IF(BY51=0,\"blank\",IF(BY51=1,-1,IF(BY51=2,0)))","v":"blank"},null,{"$f":"IF(CA51=0,\"blank\",IF(CA51=1,-1,IF(CA51=2,0)))","v":"blank"},null,{"$f":"IF(CC51=0,\"blank\",IF(CC51=1,-1,IF(CC51=2,0)))","v":"blank"},null,{"$f":"IF(CE51=0,\"blank\",IF(CE51=1,-1,IF(CE51=2,0)))","v":"blank"},null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,0,0,null,0,null,0,null,0]
["Unclear how study participants were selected to be in the program or the comparison group."]
["The selection process for program v. comparison group participants is clearly described."]
["10b. Study design",null,{"$f":"IF(BX54=0,\"blank\",IF(BX54=1,1,IF(BX54=2,3,IF(BX54=3,5,IF(BX54=4,6)))))","v":"blank"},null,{"$f":"IF(BY54=0,\"blank\",IF(BY54=1,1,IF(BY54=2,3,IF(BY54=3,5,IF(BY54=4,6)))))","v":"blank"},null,{"$f":"IF(CA54=0,\"blank\",IF(CA54=1,1,IF(CA54=2,3,IF(CA54=3,5,IF(CA54=4,6)))))","v":"blank"},null,{"$f":"IF(CC54=0,\"blank\",IF(CC54=1,1,IF(CC54=2,3,IF(CC54=3,5,IF(CC54=4,6)))))","v":"blank"},null,{"$f":"IF(CE54=0,\"blank\",IF(CE54=1,1,IF(CE54=2,3,IF(CE54=3,5,IF(CE54=4,6)))))","v":"blank"},null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,0,0,null,0,null,0,null,0]
["The study compares outcomes for students/teachers/schools who are receiving the program with outcomes for counterparts who have similar characteristics but are not participating in the program. It may do so either by identifying a comparison group that shares several known characteristics with the program participants, e.g., same grade, gender, SES (statistical matching), or by first matching program participants with non-participants who could have been just as likely to participate in the program, as predicted by known characteristics such as age and gender, and then comparing outcomes for the matched pairs (propensity score matching techniques)."]
["The intervention is provided to students/teachers/schools who are above a cutoff point for eligibility. The study compares participants who are just above the eligibility cutoff, and therefore receive the intervention, with students/teachers/schools who are just below the eligibility cutoff, and therefore do not receive the intervention. This design should ensure the two groups are highly comparable."]
["The study uses a randomized controlled trial (RCT) in which students/teachers/schools are chosen at random to either participate in the program or to serve in a comparison group."]
["The study uses a randomized controlled trial (RCT) plus utilizes propensity score matching or statistical matching techniques."]
["10c. Addressing other possible explanations for the results: If the study does not assign students/teachers/schools at random to participate in the program or in the comparison group, do the authors attempt to account for, or at least discuss the possibility of, other factors besides the treatment explaining the difference in outcomes between treatment and control groups?",null,{"$f":"IF(BX59=0,\"blank\",IF(BX59=1,0,IF(BX59=2,1)))","v":"blank"},null,{"$f":"IF(BY59=0,\"blank\",IF(BY59=1,0,IF(BY59=2,1)))","v":"blank"},null,{"$f":"IF(CA59=0,\"blank\",IF(CA59=1,0,IF(CA59=2,1)))","v":"blank"},null,{"$f":"IF(CC59=0,\"blank\",IF(CC59=1,0,IF(CC59=2,1)))","v":"blank"},null,{"$f":"IF(CE59=0,\"blank\",IF(CE59=1,0,IF(CE59=2,1)))","v":"blank"},null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,0,0,null,0,null,0,null,0]
["The study does not address other possible explanations for the results."]
["The study discusses other possible explanations for the results OR does not need to because participants were assigned at random to treatment or comparison groups and were comparable at the beginning and end of the treatment period (i.e., were comparable at baseline and did not suffer high attrition)."]
["Q11. Measuring outcomes over time",null,{"$f":"IF(BX62=0,\"blank\",IF(BX62=1,0,IF(BX62=2,1,IF(BX62=3,2))))","v":"blank"},null,{"$f":"IF(BY62=0,\"blank\",IF(BY62=1,0,IF(BY62=2,1,IF(BY62=3,2))))","v":"blank"},null,{"$f":"IF(CA62=0,\"blank\",IF(CA62=1,0,IF(CA62=2,1,IF(CA62=3,2))))","v":"blank"},null,{"$f":"IF(CC62=0,\"blank\",IF(CC62=1,0,IF(CC62=2,1,IF(CC62=3,2))))","v":"blank"},null,{"$f":"IF(CE62=0,\"blank\",IF(CE62=1,0,IF(CE62=2,1,IF(CE62=3,2))))","v":"blank"},null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,0,0,null,0,null,0,null,0]
["The study includes only post-test measures."]
["The study includes before and after measures, e.g., a pre-test/survey/observation before the intervention and a post-test/survey/observation after the intervention."]
["The study assesses outcomes multiple times before/ during and after the intervention, which may include a second post-test several months after the intervention has ended."]
["Credibility Index",null,{"$f":"SUMPRODUCT(--(ISNUMBER(C8:C65)),C8:C65)","v":0},null,{"$f":"SUMPRODUCT(--(ISNUMBER(E8:E65)),E8:E65)","v":0},null,{"$f":"SUMPRODUCT(--(ISNUMBER(G8:G65)),G8:G65)","v":0},null,{"$f":"SUMPRODUCT(--(ISNUMBER(I8:I65)),I8:I65)","v":0},null,{"$f":"SUMPRODUCT(--(ISNUMBER(K8:K65)),K8:K65)","v":0}]
["Credibility parameter:",null,{"$f":"IF(C66=0,\"\",IF(C66<10,0.2,IF(C66<20,0.6,IF(C66<=27,1,))))","v":""},null,{"$f":"IF(E66=0,\"\",IF(E66<10,0.2,IF(E66<20,0.6,IF(E66<=27,1,))))","v":""},null,{"$f":"IF(G66=0,\"\",IF(G66<10,0.2,IF(G66<20,0.6,IF(G66<=27,1,))))","v":""},null,{"$f":"IF(I66=0,\"\",IF(I66<10,0.2,IF(I66<20,0.6,IF(I66<=27,1,))))","v":""},null,{"$f":"IF(K66=0,\"\",IF(K66<10,0.2,IF(K66<20,0.6,IF(K66<=27,1,))))","v":""}]
["Credibility rating:",null,{"$f":"IF(C67<=0.2,\"Low Credibility\",IF(C67<=0.6,\"Moderate Credibility\",IF(C67<=1,\"High Credibility\",IF(C67=\"\",\"\"))))","v":""},null,{"$f":"IF(E67<=0.2,\"Low Credibility\",IF(E67<=0.6,\"Moderate Credibility\",IF(E67<=1,\"High Credibility\",IF(E67=\"\",\"\"))))","v":""},null,{"$f":"IF(G67<=0.2,\"Low Credibility\",IF(G67<=0.6,\"Moderate Credibility\",IF(G67<=1,\"High Credibility\",IF(G67=\"\",\"\"))))","v":""},null,{"$f":"IF(I67<=0.2,\"Low Credibility\",IF(I67<=0.6,\"Moderate Credibility\",IF(I67<=1,\"High Credibility\",IF(I67=\"\",\"\"))))","v":""},null,{"$f":"IF(K67<=0.2,\"Low Credibility\",IF(K67<=0.6,\"Moderate Credibility\",IF(K67<=1,\"High Credibility\",IF(K67=\"\",\"\"))))","v":""}]

{"sheet":"4 Background & References","index":5,"n_rows":38,"n_cols":3,"rows_kept":38,"cut":false,"hidden":false,"merged":[[0,0,0,1],[1,0,1,1],[34,0,34,1],[37,0,37,1],[36,0,36,1],[35,0,35,1],[2,0,2,1],[3,0,3,1],[5,0,5,1],[6,0,6,1],[7,0,7,1]]}
["Relevance and Credibility Indices for Studies of Educational Interventions"]
["What are the Relevance and Credibility Indices1?"]
["In order to help you assess whether existing research studies are credible and relevant to your own educational context, our reserach team has developed two indices, the Relevance Index and the Credibility Index. These are essentially rubrics that can help decision-makers evaluate existing studies of educational programs, strategies or interventions to determine how seriously to take the findings and whether they can safely assume the findings might apply in their own context.* \n\nThe Relevance Index helps decision-makers determine how well a study’s findings apply in their own context by assessing the extent to which various characteristics of the study population and context are similar to the decision-maker’s own context.\n\nThe Credibility Index helps decision-makers assess how seriously to take the findings of a study by scoring it on the quality of study design.  "]
["*Note that these indices are designed to be used to assess studies which focus on single investigations rather than for studies which summarize the results of many investigations, such as literature reviews or meta-analyses. While such summaries are useful for providing a general idea of “what works,” it would be hard to assess the relevance of such overviews to your own context."]
[]
["How to use this Worksheet to evaluate a research study:"]
["•Before reading a study you want to consider as evidence, look over the items in each index (Relevance and Credibility) so you know what to look out for. "]
["•To find the information in the study that you will need for completing this worksheet, look for section headings in the study that use terms like “Methods” or “Study Design,” “Measures,” and “Sample” or “Setting.” Search functions in a Word or PDF document can help with this (e.g., Ctrl F)."]
[]
["Key terms:"]
["Comparison group","A comparison group is a group that did not participate in a program and whose results on the outcome of interest can be compared with those of the group that did participate in the program. The key challenge in a good efficacy study is to find a comparison group that closely resembles program participants, meaning both groups should be similar on average across important characteristics, such as student demographics or school characteristics. "]
["Measures","Measures are the items in a research study to which the participants respond2, which are used to assess performance on the outcomes of interest."]
["Outcome","An outcome is a change or impact caused by the program or strategy being evaluated, or it could be a characteristic of the respondents you want to measure.3 An outcome of a reading program for third graders in an elementary school might be to raise reading skills among third grade students."]
["Sample","A sample is a subset of a population, for example, a subset of the students/teachers/schools that make up a target population for an evaluation.  \nA sample is representative if the sample is similar to the target population on all important characteristics. "]
["Statistical power","Statistical power: The probability that the estimate of the program effect will be found statistically significant if a given effect size is determined to have occurred."]
["Sample size","Sample Size: The number of units (e.g., students/teachers/schools) in a sample."]
["Target population","The target population of a study is the complete collection of students/teachers/schools or other units that we want to study or consider.4  For example, the target population of a study on algebra skills of high school students in a particular school district may be all 150,000 students in every high school in the district."]
[]
["Acronyms:"]
["ELL:","English Language Learner"]
["FRPL:","Free and Reduced Price Lunch"]
[]
[]
["Relevance Index","Relevance Rating"]
["Less than 30%","Low Relevance"]
["31 - 69%","Moderate Relevance"]
["70% or higher","High Relevance"]
[]
["Credibility Index","Credibility Rating","Credibility Parameter"]
["Less than 10","Low credibility",0.2]
["10-19","Moderate credibility",0.6]
["20-27","High credibility",1]
[]
[]
["1 The design of these indices was influenced by a number of sources: Digital Promise’s “Evaluating Studies of Ed Tech Products,” NESTA’s “Standards of Evidence,” Alliance for Useful Evidence’s “What Counts as Good Evidence,” David Gough’s “Weight of Evidence: A Framework for the Appraisal of the Quality and Relevance of Evidence,” and REL Central’s Draft “Applicability of Evidence-Based Interventions” Tool."]
["2 http://www.uniteforsight.org/research-methodology/module4"]
["3 https://www.povertyactionlab.org/research-resources/measurement-and-data-collection"]
["4 Lohr. S. L. (2010). Sampling: Design and Analysis. 2nd ed. Boston, MA: Brooks/Cole, Cengage Learning."]

{"sheet":"Reference Table","index":6,"n_rows":35,"n_cols":17,"rows_kept":35,"cut":false,"hidden":true,"merged":[[1,16,2,16],[1,6,2,6],[1,15,2,15]]}
["RI Importance/Own Rating",null,null,null,null,"RI Interpretation",null,null,"RI Decision",null,null,"CI"]
["not at all/unavailable","slightly","moderately","very",null,"Relevance Index","Relevance Rating",null,"Yes","No",null,"Who conducted the study?","Max Points: 2",null,"Credibility Index","Credibility Rating","Credibility Parameter"]
[null,null,null,null,null,"( = Total points for this study/Total possible score x 100)",null,null,null,null,null,4,-1,null,"(Total Points for Parts I, II, III, IV)"]
[null,null,null,null,null,"Less than 30%","Low Relevance",null,null,null,null,"The study was conducted by the program vendor.",0,null,"Less than 10","Low credibility",0.2]
[null,null,null,null,null,"31 - 69%","Moderate Relevance",null,null,null,null,"The study was conducted by an external evaluator hired by the program vendor.",1,null,"10-19","Moderate credibility",0.6]
[null,null,null,null,null,"70% or higher","High Relevance",null,null,null,null,"The study was conducted by an external evaluator acting as an independent third party, i.e., not paid by the vendor.[1] ",2,null,"20-27","High credibility",1]
[]
[null,null,null,null,null,null,null,null,null,null,null,"Who published the study?","Max Points: 2"]
[null,null,null,null,null,null,null,null,null,null,null,"It is not clear where the study was published. ",-1]
[null,null,null,null,null,null,null,null,null,null,null,"The study was published by the vendor.",0]
[null,null,null,null,null,null,null,null,null,null,null,"The study was published by a third party (i.e., other than the vendor) but not a peer-reviewed journal, e.g., a university, research organization, school district or other government agency. Keep in mind some technical reports are also later published in a peer-reviewed journal, so you may wish to check Google Scholar to see if there is a later published version.",1]
[null,null,null,null,null,null,null,null,null,null,null,"The study was published in a peer-reviewed journal.[2] ",2]
[]
[null,null,null,null,null,null,null,null,null,null,null,"Length of time participants are exposed to the educational program/strategy being studied","MAX POINTS 2"]
[null,null,null,null,null,null,null,null,null,null,null,"Length of exposure is not clear from the study. ",-1]
[null,null,null,null,null,null,null,null,null,null,null,"Length of exposure is too short to make a difference.",0]
[null,null,null,null,null,null,null,null,null,null,null,"Length of exposure is too long to reflect likely effect in regular practice.",1]
[null,null,null,null,null,null,null,null,null,null,null,"Length of exposure is about right.",2]
[]
[null,null,null,null,null,null,null,null,null,null,null,"Components of the program: Is it clear what program participants were expected to do compared with business as usual conditions?","MAX POINTS: 2"]
[null,null,null,null,null,null,null,null,null,null,null,"The components of the program/treatment are not clear.",-1]
[null,null,null,null,null,null,null,null,null,null,null,"The components of the program/treatment are partially clear.",1]
[null,null,null,null,null,null,null,null,null,null,null,"The components of the program/treatment are completely clear.",2]
[]
[null,null,null,null,null,null,null,null,null,null,null,"Implementation of the program:  Is it clear how the program was implemented in the study, e.g., how much time participants actually spent engaging in the program activities, how often, whether it was implemented in/out of the classroom, before/ after school hours etc.?","MAX POINTS: 2"]
[null,null,null,null,null,null,null,null,null,null,null,"The implementation details of the program are not clear.",-1]
[null,null,null,null,null,null,null,null,null,null,null,"The implementation details of the program are partially clear.",1]
[null,null,null,null,null,null,null,null,null,null,null,"The implementation details of the program are completely clear.",2]
[]
[null,null,null,null,null,null,null,null,null,null,null,"Meaningful outcomes","MAX POINTS  3"]
[null,null,null,null,null,null,null,null,null,null,null,"It is not clear which outcomes are being measured, e.g., it is not clear whether the study is evaluating geometry skills or algebra skills.",-1]
[null,null,null,null,null,null,null,null,null,null,null,"The outcomes measured are not at all aligned with the ultimate goal for implementing the intervention, e.g., the study investigates whether an after-school supplemental math program improves geometry skills, despite the fact that the program aims to improve algebra skills.",0]
[null,null,null,null,null,null,null,null,null,null,null,"The outcomes measured only capture short-term behavioral changes but not longer-term educational outcomes, e.g., the study only documents whether students are attending an after-school math program, but does not measure whether their math skills are improving.",1]
[null,null,null,null,null,null,null,null,null,null,null,"The outcomes measured are aligned with some but not all of the stated goals for implementing the intervention, e.g., the study is measuring algebra skills when the primary goal of the program is to improve both algebra and geometry skills.",2]
[null,null,null,null,null,null,null,null,null,null,null,"The outcomes measured are aligned with all of the stated goals for implementing the intervention, e.g., the study is measuring algebra skills when improving algebra skills is the primary goal of the program.",3]