SciELO - Scientific Electronic Library Online

 
vol.33 issue4Types of Facets on the Superior Articular Surface of Isan-Thai Dried CalcaneiConnection between Radial and Ulnar Nerves at Humeral Level And Its Clinical Significance: A Cadaveric Case Report author indexsubject indexarticles search
Home Pagealphabetic serial listing  

Services on Demand

Journal

Article

Indicators

Related links

  • On index processCited by Google
  • Have no similar articlesSimilars in SciELO
  • On index processSimilars in Google

Share


International Journal of Morphology

On-line version ISSN 0717-9502

Int. J. Morphol. vol.33 no.4 Temuco Dec. 2015

http://dx.doi.org/10.4067/S0717-95022015000400059 

Int. J. Morphol., 33(4):1553-1558, 2015.

 

Instructions for the Use of MInCiR Scale to Assess Methodological Quality in Prognosis Studies

 

Instrucciones para la Utilización de la Escala MInCiR para Valorar Calidad Metodológica de Estudios de Pronóstico

 

Carlos Manterola*,**,***,****; Ricardo Cartes-Velasquez***,***** & Tamara Otzen***,****,******

* Department of Surgery, Universidad de La Frontera, Temuco, Chile.

** Center for Morphological and Surgical Studies (CEMyQ), Universidad de La Frontera, Temuco, Chile.

*** Ph.D. Program in Medical Sciences, Universidad de La Frontera, Temuco, Chile.

**** Center for Biomedical Research, Universidad Autónoma de Chile, Temuco, Chile.

***** Faculty of Dentistry, Universidad de Concepción, Concepción, Chile.

****** Fellow Research, Universidad Científica del Sur, Lima, Perú.

Correspondence to:


SUMMARY: The assessment of methodological quality (MQ) in biomedical research is an area of dynamic development over recent years globally. Understood as a complex and multidimensional construct, several groups have been proposed tools for its evaluation, between them our group has designed and validated scales to assess MQ in therapy, diagnosis and prognosis studies. However, as with other instruments, it is necessary to specify in detail how the measurement is performed. The aim of this article is to provide a guideline for the standardized application of MInCir MQ scale for prognosis studies. A detailed description of the 11 items and 4 domains of the scale, indicating for each of them how to assess and rate the characteristics in primary prognosis articles is presented. This article provides an application guideline that may help to improve interobserver and intraobserver reliability of MInCir MQ scale for prognosis studies.

KEY WORDS: Investigative Techniques; Methods/epidemiology; Methodological studies; Prognosis; Evaluation Studies as Topic; Epidemiologic Studies; Evidence-Based Medicine.


RESUMEN: La evaluación de la calidad metodológica (CM) en la investigación biomédica es un área de desarrollo dinámico en los últimos años a nivel global. Entendida como un constructo multidimensional y complejo, diversos grupos han propuesto herramientas para su valoración, entre ellos nuestro equipo de trabajo ha diseñado y validado escalas para valorar CM de estudios de terapia, diagnóstico y pronóstico. Sin embargo, como ocurre con otros instrumentos, es necesario especificar detalladamente la forma en que se aplica, de forma tal de poder estandarizar las mediciones. El objetivo de este artículo es explicar la aplicación de la escala MInCir de CM para estudios de pronóstico. Se presenta una descripción pormenorizada de los 11 ítems y 4 dominios que componen la escala, precisando para cada uno de ellos las características a evaluar y puntuar en los artículos primarios de pronóstico. El presente artículo aporta una guía de aplicación que permite otorgar una adecuada confiabilidad intra e inter observador en las mediciones con la escala MInCir Pronóstico.

PALABRAS CLAVE: Técnicas de Investigación; Metodología; Calidad metodológica; Pronóstico; Estudios de evaluación; Estudios epidemiológicos; Medicina basada en evidencia.


 

INTRODUCTION

With the emergence of the paradigm of Evidence-based clinical practice (EBCP) (Manterola, 2009) there has been a strong development of tools designed to evaluating and improving the quality of biomedical research (Hirst & Altman, 2012). However, there is no clear concept about what is needed and how to report the results of health research assessment (Armijo-Olivo et al., 2013), and there are many different approaches that try to cover this vital area of EBPC.

One of the concepts mentioned above is the methodological quality (MQ); which can be understood as a complex and multidimensional construct, which may include items such as: type of design, sample size, methodology, analysis quality, reporting quality, etc., which can be represented as a geometric figure for a better understanding (Manterola et al., 2006).

The MInCir group developed a scale for assessing the MQ of therapy studies (Manterola et al., 2003, 2006, 2009; Moraga et al., 2013); covering surgical and dental specialties (Aravena et al., 2013; Cartes-Velasquez et al., 2014). This has allowed not only to perform bibliometric studies, but also systematic reviews (SR), with a weighted evaluation of the evidence based on the MQ of the primary studies that generated the evidence. This methodology has allowed to include different types of designs in the SR (Manterola et al., 2009), overcoming the limitation imposed by the standard method of meta-analysis, which only considered clinical trials (CT) as primary studies.

However, the reliability of the measurements does not rest only on the psychometric properties of the instrument, but also on the correct application of it by different researchers (Streiner & Norman, 2003); especially when measures a non-physical variable, such as MQ (Manterola et al., 2006, 2009).

MInCir group recently reported the first version of the instructions for the use of therapy MInCir scale (Moraga et al., 2014), in order to standardize their use in the investigations mentioned above. After the developmet of a new scale to assess the MQ of prognosis studies (Zavando, 2011), it is necessary to provide a guidance to standardize its application to ensure its reliability.

The aim of this article is to provide a guideline for the standardized application of MInCir MQ scale for prognosis studies.

INSTRUCTIONS FOR ITS USE

The MInCir scale to assess MQ of prognosis studies consists of 4 domains with 11 items (Table I). The domain 1 evaluates the type of research design;, the domain 2 assigns scores to the size of population studied, including a justification factor (if the study includes a sample size calculation), the domain 3 consists of 5 items that assess the study's methodology, and the domain 4 consist of 4 items that evaluates the statistical analysis and conclusions of the study.

The application of this instrument by the user requires critical evaluation of each of the items to assign the respective score, which sum can give a total score between 7 and 60 points. The cut-off point to define the construct MQ (dichotomous) for this type of studies is 33 points; allowing differentiation as adequate or high MQ when the evaluated article received a score 33 points; and inadequate or low MQ if the score is 32 points (Zavando).

Table. 1. MinCir Scale for prognosis studies.

Below is a detailed guide to each domain and item contained in the scale for a correct interpretation and subsequent implementation:

Domain 1: Research design.

The type of study design must be mentioned in the "material and methods" of the article and must be the same as indicated in the summary. This is what should be evaluated according to the scores indicated by the scale. Thus, a prospective or concurrent cohort and a controlled double-blind and randomized CT will be allocated 15 points in this domain. Moreover, at the lower end of the domain; a case series will be assigned only 3 points.

Unfortunately it is often difficult to clearly identify the study design when the authors do not mention or "make-up" the design, for example:

a) Articles in which authors do not make an appropriate mention or description of the design used in "material and

methods" nor the summary; so it is necessary to determine it from athorough reading of the entire article.

b) In other articles, the authors make a brief description of some design features, such as randomized prospective study instead of randomized CT; or retrospective study instead of retrospective case series. This situation is not ideal, but it is acceptable to score the design.

c) On other occasions, the study design is not consistent with that reported by the authors. In such situations, what is seen most often is an overqualification of the type of design, e.g.: declare a cohort study, when in fact it is a case series (even without follow-up).

d) Lastly, some articles mentioned a design that does not exist; eg "case-control cohort," or "non-concurrent prospective study cohort". in these cases, most of the time, is a series of cases.

It is very unusual that authors develop a complex design and not report the methodology and results according to them appropriately. So, when the design used is not mentioned or mention a design that does not exist, usually it is a series of cases (usually retrospective). On the other hand, if authors report a cohort study, the article results must contain an incidence value, absolute or relative risk with a confidence intervals.

Domain 2: Studied population.

In this domain the score must be assigned according to the number of patients who actually participated in the study, those actually included in the analysis of the reported results. Also it must be considered whether this number of patients was estimated with a suitable statistical procedure at the planning of the study. This estimation will allow applying the "justification factor", that can double the score of the domain. For example a study with 560 patients have at least 7 points, but if the minimum sample size for conducting the study was estimated, then 15 points will be assigned. In the case of a study with 57 patients, with and without justification values with and without justification shall be of 3 and 6 points respectively. Thus, recognizing the "justification factor" is too important, since in many cases the studies are performed including only the available patients without considerations to ensure the statistical power of the results.

Domain 3: Methodology.

This domain evaluates 5 aspects of the methodology: objectives, design, variables, sample size and follow-up.

Item 1. Objectives: They are usually found at the end of the introduction. In some cases and depending on the standards of the journal could be at the beginning of methodology section or a non-IMRD section. There must be consistency between the objectives stated in the body text and the summary, but considering that the latter could be in a slightly abridged version, according to the journal guidelines. In shorter articles, as case series, the objective could only be mentioned in the summary. Following this line of argument, objectives can be assessed at three levels:

1. Clear and concrete objectives: In this case, it is possible to clearly recognize what will be measured, by what means, by whom, where and what will follow. The methodology is based on this objective. Three points are assigned.

2. Vague objectives: When any of the aforementioned elements is absent or improperly drafted. As well as in the MInCir therapy scale (Moraga et al., 2014), it is

possible to recognize in this category objectives like: "The aim of this paper is to report our experience in....", "recent years" "compared to conventional option", etc. In this cases is not possible to clearly recognize the purpose of the study. Two points are assigned.

3. No objectives: These are not visible in any of the previously mentioned places. One point is assigned.

Item 2. Design: It will be assessed in the methodology section, sometimes called "Material and Methods" or "Patients and Methods", as the objective, the design must be also mentioned in the article´s summary or even in the title. In the case of a case report or case series, the design might be mentioned only in the title or in the abstract. Possible options for this item are as follows:

1. Clearly identify the design: It is imperative that the reported design fits to the real design of the study. Is equally acceptable the description of the characteristics of a design, for example: "randomized prospective study", rather than randomized CT; or "longitudinal study, prospective descriptive" rather than prospective series of cases. Three points are assigned.

2. Unknown design: It is not possible to identify the above mentioned design or design does not conform to what is actually executed. For example, the authors report that the design is a cohort study, when in fact it is a case series. One point is assigned.

Item 3. Variables: In this item the proper definition of the outcome, exposure, and confounding variables are evaluated. One point will be assigned for a full definition of each of them.

1. Outcome variables: It is the event of interest, specifically identified in its conceptual and operational definition. Generally it corresponds to a disease or condition, such as: Postoperative morbidity, postoperative pain, disease-free survival at 5 years, etc.

2. Exposure variables: Defined as one or more variable incidents in the occurrence of the event of interest. As in the previous case, the definition must include the operationalization. For example, the existence of comorbidities (diabetes mellitus, dyslipidemia, heart disease, etc.); which was analyzed in a dichotomous (present or absent), or categorized by type of treatment required (medical/surgical/none).

3. Confounding variables: The definition of those that can alter the association between the outcome and exposure; risking a misinterpretation of the actual effect of these. It should also include a conceptual definition and operationalization. For example: Age as a continuous variable or dichotomized into less or more than 45 years.

Item 4. Sample size: Should assess the justification of the number of subjects included in the study, the following options exist:

1. There is a clear calculation or estimation of sample size. Three points are assigned.

2. There is no calculation or estimate of the sample size. It is not possible to find any reference for understanding the inclusion of the number of patients reported. One point is assigned.

Item 5. Follow-up: As in item 3 (variables); in this, three aspects are evaluated independently, assigning one point to each of them, if they are properly reported:

1. The percentage of subjects with follow-up is enunciated: the proportion of patients who completed the follow-up period is mentioned; or the percentage of losses during the follow-up. It should be mentioned the follow-up for the general population and for each of the arms, groups or cohorts compared in the study.

2. Follow-up greater than 80%: This limit should be applied not only to the total population, but each of the arms, groups or cohorts included in the study. For example, "the cohort of diabetic patients had a loss of 11%, and 24% for non-diabetic patients, at the end of the follow-up", in this case zero will be assigned as one of the cohort did not reach the minimum of 80%.

3. Indicate the causes of loss: It clearly describes the reasons for the losses for each of the groups or cohorts; and the strategies implemented for its control. For example: "follow-up, 4 patients changed city, 2 died of accidents not related to the pathology under study, and 3 rejected further participation; the protocol considered at least a monthly communication with each patient in order to maintain the link with the research team and avoid further losses", in this case will be assigned one point.

Domain 4: Analysis and conclusions.

This domain evaluates 5 aspects of the analysis, the results and their interpretation. These relate to: calculation of risk measures and predictive models or association, sufficient information to perform both calculations and conclusions consistent with the objective and methodology.

Item 1. Calculation of risk measures: Risk measures relevant to the stated objective and the type of design is clearly mentioned. Five points are allocated. For example in a case-control study, or in a cross-sectional study, it should mentioned the odds ratio (OR); in the case of cohort studies or CT it should mentioned the absolute risk (AR) and relative risk (RR); and complementary other measures as absolute risk reduction (ARR) and number needed to treat (NNT) or harm (NNH). All these measures are raw or unadjusted. If these are not reported, the item is evaluated with zero points.

Item 2. Information for calculating risk measures: Sufficient information is available to the reader to calculate and confirm the value of the risk measures that are reported or other relevant. Two points are allocated. In general this information should be presented in tables comparing each of the study groups or cohorts according to the outcome variable. If these are not reported, the item is evaluated with zero points.

Item 3. Predictive or association models: The models to assess the overall impact of each exposure and confounding variables over the outcome must be reported. Five points are allocated. The models used can be logistic or linear regression, and survival curves. These models must permit to obtain adjusted risk measures. If these are not reported, the item is evaluated with zero points.

Item 4. Consistency objective-methodology-results: The results reported respond to the objective and the methodology. Three points are allocated. For instance:

1. In the case of studies that focus on rare or long-term diseases, the appropriate design would be a case-control study using the OR as a measure of risk. They must conclude about the impact of exposure variables on the event of interest; in this case there is consistency, and three points are allocated.

2. In other cases there is an incorrect methodology for the stated objective, insufficient follow-up of a cohort, or enrolling patients who do not conform to the objective. In these cases zero points are allocated.

3. It is also possible to see where the conclusion is not consistent with the design. This is frequently seen in case series or case-control studies that raise absolute conclusions, and even recommendations. In these cases zero points are allocated.

DISCUSSION

In the specific case of constructs measured with scales, it is essential to check the validity and reliability by using various strategies (Rojahn et al., 2011; Aravena et al., 2014). However, there are other relevant factors such as the mode of application and the use of the instrument by the evaluator.

Over the exercise of design, construction, validation and application of scales to measure CM by the group MInCir emerges the clear need to explain in detail the way to scores each item of the instrument (Moraga et al., 2014). Despite the score levels for each item appear as self-explanatory, in practice, the assessment of the articles becomes difficult because most articles have a low reporting quality and MQ, often with confusing wording or structure, and misuse of methodological terminology.

Given the above, it is necessary to have instructions as this one. These instructions specify in more detail the application of the scale, including examples that facilitate the identification of specific, but frequent situations. In accordance with the foregoing, these instructions facilitate the calibration of multiple evaluators (Aravena et al., 2014), for carrying out larger bibliometric studies (Manterola et al., 2006) or SR (Concha et al., 2014).

In the case of the scale MInCir therapy, the use of instructions regarding the application of the instrument (Moraga et al., 2014) allowed obtaining reliable measures close to 100% in surgical (Moraga et al., 2013) and dental (Cartes-Velasquez et al., 2014) fields, demonstrating the usefulness of this guide.

Finally we should indicate that when assessing MQ, a very dynamic field (Armijo-Olivo et al.), these guidelines represent the first version. It is expected that these guidelines may be updated periodically, adapting to the dynamics of research in MQ.

REFERENCES

Aravena, T. P.; Cartes-Velásquez, R. & Manterola, D. C. Productividad y calidad metodológica de artículos clínicos en cirugía oral y maxilofacial en Chile. Período 2001-2012. Rev. Chil. Cir., 65(5):382-8, 2013.         [ Links ]

Aravena, P. C.; Moraga, J.; Cartes-Velásquez, R. & Manterola, C. Validez y confiabilidad en investigación odontológica. Int. J. Odontostomat., 8(1):69-75, 2014.         [ Links ]

Armijo-Olivo, S.; Fuentes, J.; Ospina, M.; Saltaji, H. & Hartling, L. Inconsistency in the items included in tools used in general health research and physical therapy to evaluate the methodological quality of randomized controlled trials: a descriptive analysis. BMC Med. Res. Methodol., 13:116, 2013        [ Links ]

Cartes-Velásquez, R.; Manterola, C.; Aravena, P. & Moraga, J. Reliability and validity of MINCIR scale for methodological quality in dental therapy research. Braz. Oral Res., 28:S1806-83242014000100232, 2014.         [ Links ]

Concha, J. A.; Cartes-Velásquez, R. & Delgado, C. M. Single-incision laparoscopic appendectomy versus conventional laparoscopy in adults. A systematic review. Acta Cir. Bras., 29(12):826-31, 2014.         [ Links ]

Hirst, A. & Altman, D. G. Are peer reviewers encouraged to use reporting guidelines? A survey of 116 health research journals. PLoS One, 7(4):e35621, 2012.         [ Links ]

Manterola, D. C.; Pineda, N. V.; Vial, G. M.; Losada, M. H. & Muñoz, N. S. Revisión sistemática de la literatura. Propuesta metodológica para su realización. Rev. Chil. Cir., 55(2):204-8, 2003.         [ Links ]

Manterola, C.; Pineda, V.; Vial, M.; Losada, H. & MINCIR Group. What is the methodologic quality of human therapy studies in ISI surgical publications? Ann. Surg., 244(5):827-32, 2006.         [ Links ]

Manterola, D. C. Medicina basada en la evidencia o medicina basada en pruebas I: generalidades acerca de su aplicación en la práctica clínica cotidiana. Rev. Méd. Clín. Condes, 20(1):125-30, 2009.         [ Links ]

Manterola, C.; Vial, M.; Pineda, V. & Sanhueza, A. Systematic review of literature with different types of designs. Int. J. Morphol., 27(4):1179-86, 2009.         [ Links ]

Moraga, C. J.; Burgos, D. M. E.; Manterola, D. C.; Sanhueza, C. A.; Cartes-Velásquez, V. R.; Urrutia, V. S. & Grupo MINCIR (Metodología e Investigación en Cirugía). Confiabilidad de la escala MINCIR para valorar calidad metodológica de estudios de terapia. Rev. Chil. Cir., 65(3):222-7, 2013.         [ Links ]

Moraga, J.; Manterola, C.; Cartes-Velasquez, R.; Burgos, M. E.; Aravena, P.; Urrutia, S. & Grupo MINCIR. Instructions for

the use of MINCIR scale to assess methodological quality in therapy studies. Int. J. Morphol., 32(1):294-8, 2014.

Rojahn, J.; Rowe, E. W.; Kasdan, S.; Moore, L. & van Ingen, D. J. Psychometric properties of the Aberrant Behavior Checklist, the Anxiety, Depression and Mood Scale, the Assessment of Dual Diagnosis and the Social Performance Survey Schedule in adults with intellectual disabilities. Res. Dev. Disabil., 32(6):2309-20, 2011.         [ Links ]

Streiner, D. L. & Norman, G. R. Health Measurement Scales: A Practical Guide to Their Development and Use. 3rd ed. Oxford, Oxford University Press, 2003.         [ Links ]

Zavando, D. Construcción y validación de una escala de calidad metodológica para estudios de pronóstico. Tesis de Magister en Ciencias Médicas. Temuco, Universidad de La Frontera, 2011.         [ Links ]


Correspondence to:

Dr. Carlos Manterola

Department of Surgery and CEMyQ
Universidad de La Frontera
Manuel Montt 112
Office 408
Temuco
CHILE
Phone: 56-45-2325760
Fax: 56-45-2325761

Email: carlos.manterola@ufrontera.cl

Received: 13-07-2015
Accepted: 22-09-2015

Creative Commons License All the contents of this journal, except where otherwise noted, is licensed under a Creative Commons Attribution License