Canadian Journal of General Internal Medicine
Volume 14, Issue 3, 2019 23
About the Authors
Annabelle Cumyn currently works as Professor in the Department of Medicine at Sherbrooke University.
Sharon E. Card is a General Internist in Saskatoon, SK.
Paul Gibson is an Associate Professor of Medicine and Obstetrics & Gynecology at the University of Calgary.
Corresponding Author
Submitted: October 8, 2018. Accepted: December 20, 2018. Published: August 31, 2019. DOI: 10.22374/cjgim.v14i3.322
Education Research - GIM
Annabelle Cumyn, MD, CM, MHPE, Sharon E. Card, MSc, MD, FRCPC, and Paul Gibson, MD
Obstetric Medicine is an area of expertise within General Internal Medicine (GIM) in Canada.
Essential content for clinical rotations for GIM residents was identified by subject-matter experts
(N=204 items). However, this work did not address the perspective of curriculum stakeholders.
Members of the Canadian GIM Specialty Committee (GIMSC) were surveyed to obtain their
perspective on essential content that GIM residents should acquire in Obstetric Medicine.
GIMSC members (N=14) selected “core content” which reduced the initial content blueprint
by 72%. Some sections of the blueprint were left largely unchanged (e.g., hypertensive disorders
of pregnancy), whereas others were removed entirely (e.g., transplant medicine).
GIMSC were more selective than Canadian Obstetric Internists in choosing the essential
content for GIM residents, with a moderate overall agreement of 78% (kappa coefficient of
0.53). Comparison of perspectives and content mapping may provide useful validity evidence
for further work.
Contexte général
La médecine obstétricale est un domaine dexpertise en médecine interne générale (MGI) au
Canada. Le contenu essentiel des rotations cliniques pour les résidents en GIM a été identifié
par les experts en la matière (N=204 items). Cependant, ce travail na pas pris en compte le
point de vue des parties prenantes du curriculum.
Les membres du Comité canadien de spécialité en GIM (CTSG) ont été sondés afin dobtenir
leur point de vue sur le contenu essentiel que les résidents en GIM devraient acquérir en
médecine obstétrique.
Les membres de lASGDC (N=14) ont choisi le “ contenu de base “, ce qui a réduit le plan de
contenu initial de 72 %. Certaines sections du plan directeur sont demeurées en grande partie
Annabelle Cumyn et al.
Canadian Journal of General Internal Medicine
24 Volume 14, Issue 3, 2019
inchangées (p. ex., troubles hypertensifs de la grossesse), tandis que dautres ont été entièrement
supprimées (p. ex., médecine de transplantation).
LASGDC a été plus sélective que les internistes obstétriciens canadiens dans le choix du contenu
essentiel pour les résidents de lASG, avec un accord global modéré de 78 % (coefficient kappa
de 0,53). La comparaison des points de vue et la cartographie du contenu peuvent fournir des
preuves de validité utiles pour la poursuite des travaux.
Keywords: Obstetric Medicine, curriculum development, validity evidence, competencies and
skills, pregnancy, General Internal Medicine
Obstetric Medicine (OM) is a growing discipline that provides
care for women with medical conditions entering pregnancy or
women who develop medical complications during gestation
and/or the early postpartum period.
General Internal Medicine
(GIM), like other Canadian residency programs, is adapting
to societal needs to ensure that new specialists have achieved
minimal competency in many clinical situations - including
the assessment and management of medical conditions
around pregnancy. Royal College of Physician and Surgeons
(RCPSC) accredited GIM residency programs (PGY 4 and
5) are required to include clinical rotations in OM to ensure
that practicing General Internists have the necessary baseline
competency to evaluate and manage this special population.
A survey of Canadian internists identified different levels of
mastery that should be required of a GIM training program
to facilitate the transition to practice.
” OM was felt to be an
area in which a level of proficiency (rather than expertise)
should be attained. Previous work has already described a gap
between the “importance rating” of an area of practice and the
preparation rating” after residency training in GIM.
In that
Canadian survey, the field of ‘medical problems of pregnancy’
was identified as an area with one of the highest needs for
training improvement in order to support the achievement
of competency.
Competency appears to be based on the acquisition of essential
content, followed by the mobilization of knowledge and deliberation
on practice.
The Royal College of Physicians and Surgeons of
Canada (RCPSC) has committed to competency-based medical
education (CBME) and is transforming post-graduate education
within a unique model called Competence By Design (CBD).
Concurrently, the Canadian Consensus for a Curriculum in OM
(CanCOM) research group was formed in 2010 and includes a
majority of Canadian Obstetric Internists (current membership
of N = 49 physicians). This group of subject-matter experts
validated a content blueprint of N=497 items divided into 21
sections to characterize the emerging field of OM as a whole.
More recently, N=22 members of the research group completed
a survey based on the initial blueprint. The survey asked these
Obstetric Medicine specialists (OMS) to review the blueprint
of N=497 items and select the essential content that should be
the basis of clinical exposure during OM rotations for PGY 4-5
GIM residents in Canada.
In this study, we repeat the survey with members of the
Canadian RCPSC GIMSC to obtain their perspective on essential
content for OM rotations. A second objective is to compare the
GIMSC and OMS perspectives. This comparison will provide
a description of the content for which there is an agreement to
include as essential or exclude as well as areas with discordant
opinions. This map may be useful for the development of
pertinent, valid pedagogical approaches for the teaching of
OM during GIM residency programs. Ultimately, the aim is
to deliver competent GIM specialists to meet the needs of this
unique patient population.
The GIM SC is mandated by the RCPSC to define their discipline-
specific national standards, based on the educational framework
of the RCPSC. At the time of the study, the GIMSC consisted
of 25 members, including 14 GIM Program Directors and 9
appointed representatives (regional, community, examination
and national specialty society) that meet on a regular basis.
We asked members of the Canadian GIMSC to review the
blueprint of N=497 items – divided into 21 sections - and to select
the essential content that they would like post-graduate years
4–5 residents in GIM to be exposed to during their rotation(s)
in OM. The research instrument used was identical to the one
completed by the OMS and was based on prior work developing
and validating a content blueprint for Obstetric Medicine by
Canadian Obstetric Medicine subject-matter experts.
study aimed to compare the essential content selected by the
Original Research
Canadian Journal of General Internal Medicine
Volume 14, Issue 3, 2019 25
OMS and members of the GIMSC in charge of establishing
national competency standards.
Statistical Analysis
Descriptive statistics were used to compare both sets of results.
A cut-off of 80% was set a priori as a measurement of substantial
agreement. A simple Kappa coefficient was used as a measure
of the overall agreement. McNemars test was performed as a
test of marginal homogeneity. The statistical significance was
set at a p value of 0.05 or lower.
Ethics Approval
This study was approved by the research ethics board in Education
and Social Sciences of the Université de Sherbrooke, Quebec,
Fourteen (14) of the twenty-five (25) members of the GIMSC
were present at the time of a planned face-to-face meeting.
All available members agreed to participate in the study and
completed the research instrument. Of the 497 items in the
initial CanCOM content blueprint, 132 items were selected as
core content’ by at least 11/14 of members (a percent agreement
of at least 78.6%). This would result in a 73% reduction of the
initial content blueprint (see Figure 1).
Among the 22 OMS who performed the same task: 204
out of 487 items were selected by at least 18/22 respondents
(a percent agreement of at least 81.8%).
This resulted in a 58%
reduction of the content blueprint (see Figure 1).
The overall agreement between both groups regarding
essential content was moderate: both groups agreed to include
or exclude 380 out of the 487 items, a level of agreement of 78%
with a simple kappa coefficient of 0.53 (0.46–0.61). Conversely,
there were discordant ratings for N=107 items (see Figure 2).
When in disagreement, OMS were significantly more likely
to include items than the GIMSC members (p<0.001).
Within the 21 sections of the content blueprint, six exhibited
low levels of agreement (as detailed in Table 1).
There was a non-significant trend towards a lower level of
agreement for content blueprint sections with a lower number
of items (McNemar’s p=0.11). The remaining 15 sections of the
content blueprint displayed higher levels of agreement (whether
to include or exclude an item) ranging from 70 to 100% (Table 2).
Of interest, regarding the eight sections with an agreement
level inferior or equal to 85%, the OMS group indicated a
persistent preference to include items that the GIMSC would
not have included (see Table 2).
The comparison between the perspectives the two groups
can be mapped (Figure 3).
Each spoke represents one of the 21 sections of the initial
curriculum blueprint.
The smaller blue map shows the essential
content as selected by the GIMSC whereas the larger orange
map displays the content selected for inclusion by the OMS. It
is apparent that some sections (e.g., transplant medicine and
Figure 1. Selection of essential content items by the two groups of raters.
Annabelle Cumyn et al.
Canadian Journal of General Internal Medicine
26 Volume 14, Issue 3, 2019
Table 1. Sections of the Cancom Blueprint** With Low Levels of Agreement
12.0 Respirology 67% Items relating to Asthma were largely excluded by GIMSC and largely included by OMS
2.0 Physiology of pregnancy 67% GIMSC included 5/12 items whereas OMS included 9/12 items
5.0 Acute care and maternal
64% GIMSC included 7/14 items whereas OMS included 4/14 items favouring items related to
3.0 Pharmacology of pregnancy
and lactation
50% GIMSC included 1/8 items whereas OMS included 4/8 items
13.0 Nephrology 41% Certain items excluded by GIMSC were selected by 86% and greater by OMS
1.0 CanMEDS roles 38% GIMSC included 18/35 items whereas the OMS included 8/35
*Agreement in this article refers to the % of items included or excluded by both groups
** The original CanCOM content blueprint can be found at
Figure 2. Description of levels of agreement between the groups of raters.
neoplastic disorders) disappear from the map as they contain
no essential items, whereas other sections (e.g., hypertensive
disorders) remain largely untouched with both maps overlapping
(i.e., 100% agreement).
Members of the GIM Specialty Training Committee reduced
the initial OM content blueprint to essential content for GIM
PGY4-5 residents by removing nearly three-quarters of the
original items. This reduction was significantly greater than the
contraction proposed by OM subject-matter experts.
The overall
level of agreement between both groups regarding which items
to select or exclude was moderate.
Some sections, however,
demonstrated very high levels of agreement – and these were
generally core areas of OM. The final output is a map of N=116
content items that were judged to be “essential” by both groups.
It is interesting to contemplate why subject-matter experts
(OMS) would be more inclusive and thus less likely to reduce
the initial, comprehensive blueprint of an emerging field to a
short list of essential content. One possible explanation is that
Original Research
Canadian Journal of General Internal Medicine
Volume 14, Issue 3, 2019 27
the OMS may find it more difficult to label any content as non-
essential – as the conditions included in the original curriculum
content blueprint are all present in their clinical practice. It is also
possible that GIMSC members had the Competence-by-Design
model in mind during the adjudication process, and were thus
more able to focus on selecting truly essential items. However,
both groups of raters had difficulty in excluding items – as every
item of the 487 items was selected by at least one and usually
2 or more individual raters. Regardless, this study provides
pertinent information – both by highlighting the 78% of the
content for which there was substantial agreement, as well as by
identifying the areas that require further consideration due to
the discordant opinions. For instance, acute care and maternal
resuscitation was a section with a lower overall agreement of
67%. Are the OMS correct in their assessment that resuscitation
should be an essential part of the PGY4-5 GIM training? After
Table 2. Sections of the Cancom Blueprint** With High Levels of Agreement (In Decreasing Order)
4.0 Diagnostic and therapeutic
100% included 2/4 items
6.0 Surgical and anesthetic
100% 6/7 items excluded
14.0 Transplant medicine 100% all 6 items excluded
19.0 Neoplastic disorders 100% all 6 items excluded
7.0 Relevant obstetric and
gynecologic conditions
95% 20/21 items excluded
21.0 Psychiatric illness 95% 20/22 items excluded
8.0 Hypertensive disorders 94% 31/34 items included by both
10.0 Endocrinology 85% Agreement* on 63/74 items; OMS would have included another 11 items related to
hypothyroidism and metabolic syndrome
11.0 Hematology 83% Agreement on 30/36 items; OMS would have included another 6 items related to immune
thrombocytopenic purpura, thrombophilias, and anticoagulation
18.0 Immunologic diseases 81% Agreement on 21/26 items; OMS would have included another 5 items related to systemic
lupus erythematosus
9.0 Cardiology 79% Agreement on 22/28 items; OMS would have included another 6 items related to
congenital heart disease and peripartum cardiomyopathy
16.0 Neurology 79% Agreement on 38/48 items; OMS would have included another 9 items related to
headaches and seizure disorders
15.0 Gastroenterology 75% Agreement on 34/45 items; OMS would have included another 11 items related to
dyspepsia, acute fatty liver of pregnancy, inflammatory bowel disease, and obstetric
20.0 Dermatology 75% Agreement on 6/8 items; OMS would have included another 2 items related to pruritus in
17.0 Infectious disease 70% Agreement on 21/30 items; OMS would have included another 8 items related to sepsis,
pneumonia, and HIV
*Agreement in this article refers to the % of items included or excluded by both groups
** The original CanCOM content blueprint can be found at
Annabelle Cumyn et al.
Canadian Journal of General Internal Medicine
28 Volume 14, Issue 3, 2019
all, resuscitation of the pregnant patient requires adjustments
to BCLS and ACLS protocols that many general practitioners
may not be aware of (such as manual displacement of the uterus,
emergency perimortem Caesarean section within 4 minutes after
cardiac arrest, etc). Of course, such events are rare but, if this
content is judged essential to the development of competency,
then alternate pedagogical strategies such as simulation may need
to be organized. This highlights the importance of developing
curricula from different viewpoints to ensure that the needs
of patients are well served by ensuring that rare but critical
events are included in the education of our graduates. This is
particularly true for a generalist discipline such as GIM where
graduates frequently practice with limited subspecialty back-up.
Another way of reading this data is to see OM content as a
three-tiered pyramid of content expertise (Figure 4).
In this model, one can separate the highly specialized
content that an OM content specialist would acquire within extra
fellowship-level training, from the area of expertise of a GIM
specialist with an interest in the care of women with complex
medical problems, from the more focused competency of every
practicing GIM specialist. If confirmed by further work, this
separation of the content may also set standards with regards
to when a patient requires transfer to very specialized care,
generally in tertiary and quaternary-level health care facilities.
Another important question is how this content blueprint
fits within the transition to CBME. Indeed, proponents of
competency-based curricula remind us that when curricula
rely heavily on lists of content, learning is not well integrated
across the curriculum.
This work is not intended as a step back
towards a focus on process rather than outcomes. Rather, we
consider that a prerequisite to the development of milestones
and entrustable professional activities, as described in the
Figure 3. Mapping of the reduced (essential) content blueprint according to both groups.
Figure 4. Proposed model of a three-level expertise in OM and the number of
corresponding content items.
Original Research
Canadian Journal of General Internal Medicine
Volume 14, Issue 3, 2019 29
Pediatrics Milestones project,
is a solid understanding of the
content domain. In addition, content blueprints have a role to
play in the development of assessment blueprints and to support
the development of novel pedagogical approaches such as the
CanCOM clinical cases. Another way in which these results
may be used in the CBD transition is by supplying data to
better describe learning trajectories within CBME.
while retaining a focus on the outcome of attaining competency,
clinician-educators will need to be able to conclude with some
degree of confidence that graduates have the necessary clinical
exposure, knowledge, and task ability.
This survey includes only the opinions of 14/25 of the 2016
GIMSC members. The results cannot be applied directly to other
countries as Canada is fairly unique in its approach to training
GIM specialists with the extra 2 years of residency training and
the CBD transition.
However, the discipline of OM is developing
across the world and therefore the concept of a baseline, essential
content blueprint could be informative in other settings.
Conclusion and Future Directions
This comparison of opinions regarding essential content for an
OM curriculum within GIM (PGY4-5) residency informs both
by identifying areas of agreement as well as by highlighting the
sections where opinions are more divergent. A substantial part
of the curriculum map displays high levels of agreement, which
may serve to support future work within the CBD framework
such as justifying the list of common or emergent obstetrical
medical presentations that can constitute the target of an
entrustable professional activity. Such evidence may also serve
the development of pedagogical approaches such as simulation
and clinical cases or as validity evidence for assessment.
Future work in this domain could include mixed-methodology
to further explore the 107 discordant content items. Such an
approach would clarify the underlying reasoning and explore
the distinctive motivations between these groups with differing
expertise (content expertise [OMS] versus stakeholders [GIMSC]).
If deemed necessary, recourse to a Delphi methodology could lead
to a further consensus within the discordant items. Alternatively,
this work could be repeated in different settings (countries) or
with residents from a different specialty (for example, residents
specializing in Maternal-Fetal Medicine, high-risk pregnancy).
Another need is addressing patients’ needs wherever their location.
To ensure that all Canadian patients have access to the right
specialist in a timely fashion, the content could be mapped for
those disorders that generalist specialists (GIM) need to be able
to independently manage due to the need to manage close to
patient`s homes (eg emergency conditions such as resuscitation)
versus conditions that require referral from rural to an urban
quaternary centres. Because the complexity of pregnancy with
medical concerns is increasing, regular review and adjustments
of OM curricula will be important.
1. Nelson-Piercy C, Peek MJ, and Swiet MD. Obstetric physicians: are they
needed? The workload of a medical complications in pregnancy clinic. J R
Coll Physicians Lond 1995;30:150–54.
Card SE, Pausjenssen AM, and Ottenbreit RC. Determining specific
competencies for General Internal Medicine residents (PGY 4 and PGY 5).
What are they and are programs currently teaching them? A survey
of practicing Canadian General Internists. BMC Res 2011;Notes 4,
Card SE, Snell L, and O’Brien B. Are Canadian General Internal Medicine
training program graduates well prepared for their future careers? BMC Med
Educ 2006;6:56.
Elstein AS, Shulman LS, and Sprafka SA. Medical problem solving an analysis
of clinical reasoning. Harvard University Press; 1978.
5. Schmidt H, Norman G, and Boshuizen H. A cognitive perspective on medical
expertise: theory and implications. Acad Med 1990;65:611–21.
6. Patel VL and Groen GJ. The general and specific nature of medical expertise:
A critical look. Gen Theory Expert Prospects Limits 1991;93–125.
7. Patel VL, Glaser R, and Arocha JF. Cognition and expertise: acquisition of
medical competence. Clin Invest Med 2000;23:256–60.
8. Norman G, Young M, and Brooks L. Non-analytical models of clinical
reasoning: The role of experience. Med Educ 2007;41:1140–145.
9. Frank JR, et al. Competency-based medical education: theory to practice.
Med Teach 2010;32:638–45.
10. Iobst WF, et al. Competency-based medical education in postgraduate
medical education. Med Teach 2010;32:651–56.
11. Holmboe ES, et al. A call to action: the controversy of and rationale for
competency-based medical education. Med Teach 2017;39:574–81.
12. Frank JR, editor. The CanMEDS 2005 physician competency framework.
Better standards. Better physicians. Better care. Royal College of Physicians
and Surgeons of Canada. Ottawa; 2005.
Cumyn A and Gibson P. Validation of a Canadian curriculum in OM. Obstet
Med Med Pregnancy 2010;3:145–51.
14. Cumyn A and Gibson P. Validation of content of clinical cases in OM for a
shared web-based educational tool. Obstet Med 2018.
15. Cumyn A and Harris IB. A comprehensive process of content validation
of curriculum consensus guidelines for a medical specialty. Med Teach
16. Hallgren KA. Computing inter-rater reliability for observational data: an
overview and tutorial. Tutor Quant Methods Psychol 2012;8:23.
17. Carraccio C. et al. Building a framework of entrustable professional activities,
supported by competencies and milestones, to bridge the educational
continuum. Acad Med 2017;92:324–30.
Gruppen L. et al. Toward a research agenda for competency-based medical
education. Med Teach 2017;39:623–30.
19. Card SE, Clark HD, Elizov M, Kassam N. The Evolution of General Internal
Medicine (GIM) - Google Scholar. Available at:
(Accessed: 15th September 2018)
20. Downing SM. Validity: on the meaningful interpretation of assessment data.
Med Educ 2003;37:830–37.
Annabelle Cumyn et al.