Cms Critical Thinking Test Nursing


Critical thinking has been identified as a vital element to evidence-based practice (EBP; Profetto-McGrath, 2015; Morténius, Hildingh, & Fridlund, 2016) despite being a complex construct that is difficult to define both from the conceptual and the empirical point of view. The development of critical thinking prepares nurses in achieving the new EBP competencies for practicing nurses (Melnyk, Gallagher-Ford, Long, & Fineout-Overholt, 2014). It is an essential component in nursing practice for providing safe, competent care (Romeo, 2013; Paul, 2014; Edwards, Hawker, Carrier, & Rees, 2015) and is of particular relevance in the current healthcare context, which is both ever-changing and increasingly complex. Furthermore, the need to implement EBP that will serve to help in the design of patient-centered care plans is a factor contributing to increased attention on critical thinking as an educational and professional subject that is indispensable in current nursing (Chang, Chang, Kuo, Yang, & Chou, 2011; Shoulders, Follett, & Eason, 2014). From this, the ability of the nursing professional to improve the quality of care depends in large measure on developing critical thinking skills, especially to improve diagnostic accuracy and to be able to contribute in a favorable manner to more positive results in the health of the patient (Lunney, 2010).

The measurement of the level of critical thinking has been the focus of various studies over the past three decades. A review of several definitions of critical thinking that emerge from these studies is an indication of the conceptual diversity resulting from the disciplinary framework from which they derive (Ennis, 1993; Paul & Elder, 2014). The American Philosophical Association (APA), in its Delphi Report (Facione, 1990), agreed upon a definition of critical thinking as an intellectual process which, in a decided, deliberate, and self-regulated manner, seeks to arrive at a reasonable decision. The report also concluded that components of critical thinking include cognitive abilities and attitudinal disposition.

A definition of critical thinking has emerged from nursing authors. Critical thinking is a cognitive process that represents the capacity to reflect upon reasoning with the aim of minimizing the errors in decision-making(Chao, Liu, Wu, Clark, & Tan, 2013; Shinnick & Woo, 2013; Alfaro-LeFevre, 2016).

Quite a few instruments have been reported in the literature for measuring critical thinking. Some have been used in studies that have yielded data on reliability and validity, although others are less well known, with limited applications and without informed metric properties (Pitt, Powis, Levett-Jones, & Hunter, 2015). The most widely used standardized instruments are those derived from the definition of the APA as a theoretical base. The California Critical Thinking Disposition Inventory (CCTDI; Facione, Facione, & Giancarlo, 1992) was designed to measure the attitudes of critical thinkers in the general adult population. It is complemented by the Critical Thinking Skills Test (CCTST; Facione & Facione, 1992) used to measure critical thinking ability in university students. The Health Science Reasoning Test (HSRT; Facione & Facione, 2006) is an adaptation of the CCTST for students and professionals in the healthcare field. From a review of the literature (Carter, Creedy, & Sidebotham, 2015), it is clear that the reliability of the instruments used to measure critical thinking among nurses has not been systematically examined (Atay & Karabacak, 2012; Naber, Hall, & Schadler, 2014), and they yield results that are inconsistent from one study to another (Zori, Kohn, Gallo, & Friedman, 2013; Gorton & Hayes, 2014; Hunter, Pitt, Croce, & Roche, 2014). The majority of the instruments measure critical thinking ability in the training of nursing professionals but not in their clinical practice, which is a relevant area if we consider that it is an essential competence for the providing of quality care in the setting of today's health care.

Theoretical Framework

Alfaro-LeFevre (2016) put forward the idea of 4-Circle Critical Thinking (CT), a theoretical model that offers a definition of critical thinking applied in the context of clinical practice. The 4-Circle CT Model describes the construct of critical thinking as the integration of four components: (a) personal characteristics (PC), (b) intellectual and cognitive abilities (ICA), (c) interpersonal abilities and self-management (IA), and (d) technical abilities (TA). The first of these, the PCs, are a pattern of intellectual behavior (attitudes, beliefs, and values) that function as an activating element in thinking ability. The second, the ICAs, are knowledge of actions and understanding linked to the nursing process and decision-making. The third, the IAs, are the abilities that allow for therapeutic communication and for obtaining information that is relevant to the patient. And finally, the fourth component, the TAs, is the knowledge and expertise in procedures that are part of the discipline of nursing. Alfaro-LeFevre proposes a series of indicators of critical thinking, the so-called critical thinking indicators (CTIs), for each component. The CTIs are descriptions of behavior that encourage critical thinking in clinical practice. Competence in critical thinking is, then, the result of the combination of attributes in relation to these four dimensions.

The work of Alfaro-LeFevre in this area has had a tremendous conceptual impact on nursing; it is widely referenced by many authors in the field. But no study was found that made the conceptual and structural posits of the model fully operative.

The purpose of the study was to develop and validate an instrument to assess critical thinking ability in nurses working in health care based on the 4-Circle CT Model of Alfaro-LeFevre, which we have called the Nursing Critical Thinking in Clinical Practice Questionnaire (N-CT-4 Practice).


Cross-Sectional Study Conducted in Two Phases

Phase 1: Generation of the items and content validity of the instrument

The items were generated from the 79 CTIs in the 4-Circle CT Model of Alfaro-LeFevre and from an exhaustive review of the literature on critical thinking (Zuriguel Pérez et al., 2014), on the aspect of competence, and on ethical considerations in the nursing profession (International Council of Nurses, 2012). The items were drawn up following the criteria for uniformity of expression recommended by experts (Streiner, Norman, & Cairney, 2014). The instrument was called the Nursing Critical Thinking in Clinical Practice Questionnaire (N-CT-4 Practice), it was first drawn up in Spanish (“Cuestionario del pensamiento crítico enfermero en la práctica clínica”) and was initially made up of 112 items distributed among the four dimensions that make up the theoretical model of reference. With the aim of validating the content, it was referred to a committee of six expert professionals in clinical practice and education who had worked in the area of critical thinking. They were chosen following the selection criteria laid out by Leape, Park, Kahan, and Brook (1992) regarding suitability, heterogeneity, expert knowledge of the subject, and availability.

The determination of content validity was made using the methodology proposed by Lynn (1986) and Polit, Beck, and Owen (2007) on the basis of two calculations: The Item Level Content Validity Index (I-CVI) and the Scale Level Content Validity Index (S-CVI). The experts were enjoined to evaluate the relevance and pertinence of each item on a four-point ordinal scale, from not relevant/not pertinent (1) to very relevant/very pertinent (4). The experts were also asked to make suggestions about how the items might be improved. Acceptable scores for the items were I-CVI ≥ .78, and S-CVI > .80 was considered a high score for validity of content (Lynn, 1986; Polit et al., 2007).

The results of the I-CVI showed that 83% (n = 93) of the items were scored as acceptable. As to the S-CVI, the score was .85, it is evidence accrued in estimating content validity. Items that did not score as acceptable were reviewed, as were the suggestions by the experts for their improvement. Following this review, the I-CVI of 13 items of the 112 items did not meet the cutoff of .78. These 13 items were eliminated from the instrument and 12 items were revised to improve clarity in response to comments from the experts, yielding a final total of 109 items.

Pilot testing and structure of the questionnaire

A pilot test was carried out with a sample group of 18 nurses, whose characteristics were similar to those of the study group, in order to evaluate the comprehensibility and feasibility of the N-CT-4 Practice. The time required to complete the questionnaire was 20–25 minutes. After debriefing of the volunteers, it was decided that no further changes in design or content were in order. The final version of the questionnaire was made up, then, of 109 items covering the four dimensions that make up the 4-Circle CT Model, as follows: personal (39 items); intellectual and cognitive (44 items); interpersonal and self-management (20 items); and technical (6 items). A Likert-like response format was devised with 4 points, running from never or almost never (1) to always or almost always (4), to indicate the frequency with which the professional presented a particular ability in critical thinking in the clinical setting.

Phase 2: Psychometric properties

Psychometric evaluation was then conducted with a sample of 339 nurses.


The participants were nursing professionals selected by convenience criteria from the in-patient medical, surgical, and intensive care units of a 1,100-bed tertiary carehospital in Barcelona, Spain. The study excluded emergency units, operating theatre, and central services or other in which patient are not hospitalized. A list of eligible nurses was obtained (n = 800). The required sample size was estimated to be n = 350, with an α risk of .05 for 5% precision, as recommended by Kline (2016). In order to select the professionals, stratified sampling was carried out with proportions set for the units as follows: medical (44%, n = 154), surgical (41%, n = 143), and critical care (15%, n = 53). All nurses working in these units were invited to participate in the study. Nurses who leave at the time when the study data were being compiled were not included.

In order to evaluate the test–retest reliability, units were also randomly selected and from those, a total of 20 nurses were selected as a follows: medical (44%, n = 9), surgical (41%, n = 8), and critical care (15%, n = 3).

Sociodemographic, professional, and academic information regarding the sample was collected by means of a form made up of 11 questions.

Ethical Considerations

The project was approved by the Clinical Research Ethics Committee of the Vall d'Hebron Hospital (Barcelona). Participants were informed about the authorship and purpose of the research, and were told that all data would remain anonymous and confidential.

Data Collection

Administration of the N-CT-4 Practice and of the form, carried out jointly, took place in March and April 2015. Study instruments were anonymously distributed in unsealed envelopes to enable the nurses to return the completed questionnaires in sealed envelopes. Only the nurses of the randomly selected units were asked to use a 6-digit code where to link to the test–retest instruments, to guarantee the questionnaires (performed with a 14-day gap) belonged to the same person.

Data Analysis

Descriptive statistics were used to summarize the data collected. Analysis of the items included calculation of the average, standard deviation, and corrected item-total correlation.

Internal consistency was calculated using the Cronbach's α coefficient, establishing as acceptable the value of α ≥ .70 (Nunnally & Bernstein, 1994). Test–retest reliability was examined with a subsample of n = 20 selected from the total sample studied. The questionnaire was readministered 2 weeks following the first administration, using the intraclass correlation coefficient (ICC), and considering values ≥.75 as demonstrating excellent reliability (Fleiss, 2011).

Construct validity was assessed by confirmatory factor analysis (CFA) based upon the four dimensions in the theoretical model put forward by Alfaro-LeFevre (2016). The CFA was carried out with structural equation modeling and the estimation of parameters was made using the maximum likelihood model. Model fit was determined with several methods because diverse authors have suggested using a number of indicators to determine the fit of models (Hu & Bentler, 1999; Schreiber, Nora, Stage, Barlow, & King, 2006). The goodness-of-fit of the model was evaluated using the indices and criteria suggested as being acceptable by (Hu & Bentler, 1999): chi-square test (χ2; nonsignificant), the ratio between chi-square and the degree of freedom (χ2/df; <2),the root mean square error of approximation (RMSEA; <.06), the comparative fit index (CFI; >.95), the Tucker–Lewis index (TLI; >.95), and the standardized root mean square residual (SRMR; <.08).

All analysis was carried out using the statistical package R, version 3.3.0.



The questionnaire was completed by n = 339 nurses. Response rate was 96.8%. The majority of the sample were women (87.0%, n = 294) and the average age was 44 years old (SD 11.1, range 22–52 years). Most were working full time (64.6%, n = 219). The nurses with more than 21 years’ experience (45.1%, n = 153) were working in surgery units (43.1%, n = 146). Half had undergone postgraduate education (51.0%, n = 173), although less than half had received training in specific nursing methodologies (33.0%, n = 111).


The total Cronbach's α value for the N-CT-4 Practice was .96, which qualifies as excellent according to (Waltz, Strickland, & Lenz, 2010). It ranged from .78 for the technical dimension to .94 for the intellectual. Most of the items had corrected item-total correlations >.20. Only three items had corrected item-total correlations <.20 (items 3, 5, and 70), but if the items were deleted, it did not increase the total α value. The items in question were “I show my feelings,” “I know how others feel,” and “I treat interventions and actions to prevent or control problems,” respectively (Tables S1–S4, available with the online version of this article). In the analysis of the interitem correlations, no value below .20 or above .80 was identified. These results suggest that none of the 109 items should be eliminated (Kline, 2016).

The ICC for the whole instrument was .77, and for the dimensions ranged from .70 to .84, and were all statistically significant at least at p < .05 level, indicating good stability over a 2-week period.

Construct Validity

The result of the chi-square test was significant (χ2 = 11279.527; p < .0001), indicating that the hypothesis of a perfect model needed to be rejected. However, in light of these values and bearing in mind the problems associated with the use of this test alone, it was felt that other statistical tests were needed to evaluate the theoretical model in question. The adjusted indices based on covariance reported optimal values: RMSEA = .055, SRMR = .65, as did the χ2/df ratio = 1.95, although the incremental measurement indices yielded values below the level of acceptability: CFI = .629, TLI = .621.

All of the values for estimated parameters for the model were significant, in line with what was expected, with p < .05 in all cases except for item number 5, “I know how others feel” (p = .124). None of the variances or correlations yielded values deemed to be inappropriate to the extent that the proposal would be invalidated. Figure 1 offers a graphic representation of the results of the model. The values that appear with the arrows between the circles (latent variables) and the squares (variables) indicate the factor load; the correlations between the circles are represented by means of bidirectional arrows. At the top of the squares are the residual variance values. The first variable associated with each latent variable has a regression value of 1.0 and is represented by a broken arrow.


The need to develop an instrument to assess critical thinking in clinical nurses arose from the observation, in the context of a review of the scientific literature, that most instruments were not specific to the nursing profession or else had been designed to measure critical thinking in student samples for the evaluation of specific educational endeavors.

Critical thinking as an essential skill to support EBP and can contribute positively to patient outcomes. Furthermore, it was felt to be essential to be able to measure critical thinking based on a theoretical model that was complex enough to engage the construct in an effective manner, a relevant consideration, given the complexity of the model.

Along this line, an instrument was developed and validated to assess critical thinking in working nurses, based on the 4-Circle CT Model of Alfaro-LeFevre (2016), which is distinguished by its conceptual clarity. Nevertheless, making the Alfaro-LeFevre model operative is no easy matter, given that the concepts that make up the construct of critical thinking are by nature quite complex. The multidimensional concept of critical thinking has been upheld by most theoreticians in the field (Facione, 1990) who argue that critical thinking is comprised of a series of abilities that must be understood to be interrelated. However, to date, validated instruments that clearly and adequately addressed this multidimensional perspective have not been available, which is why the N-CT-4 Practice represents an important new development.

The results obtained in the present study demonstrate that the N-CT-4 Practice is endowed with good psychometric properties. And the questionnaire was shown to be extremely viable, given that all but two participants filled it out in its entirety.

Regarding the assessment of the internal consistency of the questionnaire, the Cronbach's α coefficient obtained (α = .96) places it in the same line of values as reported for other instruments such as the Critical Thinking Diagnostic (α = .93; Berkow, Virkstis, Stewart, Aronson, & Donohue, 2011) and the CCTDI (α = .90; Facione & Facione, 1992). The ICC would seem to indicate that the questionnaire possesses good stability over time.

From the CFA one may deduce that the initial four-dimensional hypothetical model offers a good fit to the data, although there is room for improvement. It should be borne in mind that the indicators of the fit decrease with the increase in the number of parameters to be considered (Hu & Bentler, 1999), a new, smaller version would therefore improve the results. Nonetheless, given that the goodness-of-fit values were close to those considered appropriate, and that the correlations of both the items and of the dimensions were favorable, the structure of the proposed questionnaire is acceptable. Subsequent studies with broader samples would serve to verify the results obtained in the present study with increased robustness. As to the homogeneity of the items, of the 109 analyzed, all but one, number 5, “I know how others feel,” functioned correctly. This item would also need to be revised in a future version of the N-CT-4 Practice with the aim of bringing it into line with the rest of the questionnaire.

In summary, the N-CT-4 Practice is a research tool that can be used for assessing the levels of critical thinking in nursing practice. Futures studies are needed to investigate the tool's value to measure in the quality of care and patient outcomes.

This study is not without limitations. First, the characteristics of our sample proscribe generalization of the results to other populations of interest. Second, the limitations of a self-administered questionnaire need to be taken into account when interpreting these results. The participants answered freely in accordance with their opinions, but this information was not verified by other means such external observation. Third, the use of CFA fit indices is one of many possible equivalent models, also the design of a specific measuring instrument is new, represent a limitation when analyzing criteria validity. Lastly, the sample analyzed has a small percentage of males is a limitation if one wishes to generalize to other groups with parity between men and women.


The empirical structure of the N-CT-4 Practice is consistent with its theoretical underpinnings; there is evidence that the proposed dimensions behave appropriately for the analysis of critical thinking. Therefore, one may conclude from the results of the study that the N-CT-4 Practice allows for the evaluation of critical thinking on the basis of four interrelated dimensions: The personal dimension, which explores individual patterns of intellectual behavior; the intellectual and cognitive dimension, which examines intellectual abilities related with the discipline of nursing; the interpersonal and self-management dimension, which analyzes interpersonal abilities that allow for the establishing of effective links with the patient, the clinical environment, and other members of the professional team; and finally, the technical dimension, which is concerned with knowledge of the procedures that are part of the nursing profession.

The empirical evidence appears to justify use of the instrument to explore the critical thinking of nurses in the clinical setting. Future research should be designed to increase the metric robustness of the questionnaire by focusing more deeply on those areas pinpointed in the limitations noted above.

Building instruments for psychological evaluation is a complex process. The N-CT-4 Practice offers an initial foray into investigation and assessment of the critical thinking of nurses in the business of providing care to patients. In like manner, the validation of conceptual models is also a painstaking task that calls for empirical studies to provide clinical scientific evidence. The N-CT-4 Practice is a bridge that provides scientific evidence concerning the model of critical thinking advanced by Alfaro-LeFevre.

Future studies should examine the metric properties of the N-CT-4 Practice in relation to other variables and in other samples of interest. Along these lines, it would also be of interest to determine the predictive capacity (sensitivity and specificity) of the N-CT-4 Practice in longitudinal studies. Finally, the present study leaves open the possibility of future studies examining other types of validity (discriminant and convergent). 


  • Critical thinking is vital in developing EBP.
  • The N-CT-4 Practice is valid and reliable and it can be used to measure the critical thinking in the nursing practice.
  • This study makes substantive and methodological contributions that support researchers’ efforts to assess critical thinking in nursing.
  • This instrument can be used to measure nurse's critical thinking competence and consequently design strategies to improve nurse's competence.
  • Further research using this instrument must be conducted in relation to other variables and in other samples of interest.

The Centers for Medicare and Medicaid Services (CMS) recently published a memo that it sent to CLIA inspectors on how to interpret educational requirements. CLIA had always specified the minimum educational requirements for individuals performing laboratory tests.

In recent years there has been quite a  strong lobby from nursing to recognize a nursing degree as a biological science degree, having the requisite credit hours of biology, chemistry etc. Those of us with oversight for point of care testing (POCT) had also been unsure as to whether nurses could perform non-waived tests and maybe even fully manage a POCT program where non-waived tests were utilized.

Now CMS has weighed in definitively by saying that, yes, a nursing degree is a science degree making nurses qualified to perform non-waived tests. A careful reading of the CLIA regulations would suggest that if that is true, then nurses may in fact be allowed to even direct laboratory testing.

In my view, this is an encroachment on our scope of practice that also potentially endangers the patient health.  POCT and some other laboratory tests are seen as merely “recipes” by some inside and outside the profession. But we know that laboratory testing requires critical thinking skills way beyond the memory and motor skills required to add 1 drop of this, 2 drops of that and time for 5 minutes to see if a blue line appears or not.

The CMS guidance to its CLIA inspectors directs them to accept a bachelor’s degree in nursing as a biological science degree qualifying the  holder to perform even high complexity testing, and that an associates in nursing qualifies the holder to perform moderate complexity testing. Yes!

Lest you think this is something that no serious employer will follow, bear in mind that this came after lobbying from nurses who wanted to perform and supervise laboratory testing. They had a definite goal in mind and will certainly take advantage of this new development.

Not wasting much time the Veterans Administration  (VA) has already jumped on the band wagon.

In a recent letter to its members ASCLS said the following regarding the VA decision:

“The Department of Veterans Affairs (VA) has published Proposed Rule that would expand the authority of Advanced Practice Registered Nurses (APRNs) beyond ordering and interpreting lab tests, as they can now, to supervising and performing laboratory testing. If adopted, an APRN could supervise and direct a clinical laboratory.

“In proposed § 17.415(d)(1)(i), a CNP would have full practice authority to provide the following services: Comprehensive histories, physical examinations and other health assessment and screening activities; diagnose, treat, and manage patients with acute and chronic illnesses and diseases; order, perform, supervise, and interpret laboratory and imaging studies; prescribe medication and durable medical equipment and; make appropriate referrals for patients and families; and aid in health promotion, disease prevention, health education, and counseling as well as the diagnosis and management of acute and chronic diseases.”

This is setting a dangerous precedent. Nurses are valuable members of the healthcare team but they have a body of knowledge and a scope of practice different from those of medical laboratory practitioners.

This is not about competition and “staying in ones lane.” It is about a real risk to the public health. I can no more practice nursing from reading a package insert and following schematic workflows than a nurse can (safely and effectively) practice medical lab science.


One thought on “Cms Critical Thinking Test Nursing

Leave a Reply

Your email address will not be published. Required fields are marked *