Evaluation of the CNSC’s Research and Support Activities, Internal Audit, Evaluation and Ethics Division

Executive Summary

Introduction

As part of its strategic priority to be a modern, trusted and global regulator, the Canadian Nuclear Safety Commission (CNSC) funds research and support activities to obtain knowledge and information needed to support its regulatory oversight. The Innovation and Research Division (IRD), within the Technical Support Branch (TSB), manages the CNSC’s research and support activities, with a focus on research conducted by external parties.

An evaluation was conducted with the objective of assessing how well research and support activities align with the CNSC’s needs, concentrating on relevance, effectiveness, and efficiency. The evaluation focused on research and support activities of the CNSC, including CNSC activities and outcomes related to the CNSC’s partnership/collaboration with the Federal Nuclear Science and Technology Work Plan (FNST), managed by Atomic Energy of Canada Limited (AECL).

Summary: Research and support activities

Research and support activities consist of those under the Research and Support Program (RSP) stream of the CNSC’s Grants and Contributions Program, as well as other research related activities. Research-related activities include contractual arrangements with external parties, as well as partnerships and/or collaborations with other government departments where there is a shared mandate. The research work provides employees from across the CNSC with access to external independent advice, expertise, experience, information, and other resources via contracts, grants and contributions placed within the research community, and with other parties/stakeholders in the nuclear industry. The research and support activities support the CNSC and Commission members in regulatory decision making.

Results

Relevance: The evaluation found that the RSP is clearly aligned with the CNSC's core responsibility of nuclear regulation and development of a regulatory framework. It was evident that research and support activities are aligned with CNSC priorities and are meeting needs. This alignment with priorities and needs extends to the research conducted through the FNST.
Effectiveness: The evaluation found that there is a need for documentation to demonstrate the achievement of RPP outcomes; however, interview respondents felt that most outcomes are being achieved. The is an opportunity to improve awareness of research and support products across the CNSC based on varying interview responses. Awareness outside the organization could not be fully assessed.
The usefulness of research and support products is occasionally impacted by delays. The extent to which the products are contributing to the regulatory framework is unclear.

Another outcome explored by the evaluation is the development of nuclear expertise in the research community. The evidence was varied, with some interview respondents saying capacity is good and others saying it is not sufficient to meet CNSC’s needs. The availability of funding was noted as an important driver of interest within the research community.
Efficiency: CNSC research activities at CNSC are generally efficiently designed, delivered and resourced. The evaluation found that the program has sufficient oversight and guidance through the Research Advisory Committee (RAC). It was also found that roles and responsibilities are clear, although there are opportunities to increase clarity. Senior management interview respondents indicated that they would like to see increased senior management oversight and direction, including more progress reports, so that the work’s ongoing relevance can be assessed, and more top-down processes to identify research needs.
Processes for project identification, project selection and funding decision making are efficient, appropriate and aligned with best practices, but there are opportunities for improvement, particularly with respect to the timely issuance of agreements. Partnerships (with AECL, for example) are effective.

In terms of performance measurement, IRD carries out reviews and evaluation of all its contracts and contribution agreements, respectively. However, it does not appear that this information is used for reporting against the outcomes in the Performance Information Profile (PIP).

The evaluation also found that senior management interview respondents believe that there is insufficient information for decision making. This includes information available to understand the progress of research projects and activities, as well as the benefits and value of the work. This would enable periodic check-ins and allow decision-makers to assess go or no-go decisions over research projects.

Recommendations

As a result of the evaluation, 4 recommendations were developed to address the identified areas for improvement:

Enhance awareness of the IRD and research projects.
Review the planning of RSP contracts to identify opportunities to improve the efficiency of contract issuance.
Enhance reporting of information available to decision makers.
Improve reporting against specific research outcomes identified in the PIP.

1. Background

As part of its strategic priorities to be a modern, trusted, and global regulator, the CNSC funds approximately $3.6M in research and support activities (FY 2023–24) to obtain knowledge and information needed to support its regulatory oversight. The Innovation and Research Division (IRD), within the Technical Support Branch (TSB), manages the CNSC’s research and support activities, with a focus on research conducted by external parties.

Research and support activities are mission-oriented and not intended to generate new technology, but to validate and inform regulatory decision making by meeting needs defined by various stakeholders across the CNSC.

The CNSC RSP manages an extramural research program that is divided into separate funding categories:

Contribution:
- Federally appropriated conditional transfer payment with specific terms and conditions that must be met by the recipient
- Supports both domestic and international collaborative agreements (e.g., UNENE, CSA, IAEA, and OECD/NEA)
- Limited to non-profit, academic, and government institutions
Grant:
- Federally appropriated transfer payment subject to pre-established eligibility criteria
- Typically used to support outreach
- Limited to non-profit, academic and government institutions
Contract:
- A cost-recoverable written agreement constituting an agreement between a contracting authority and person or firm
- Competitive or sole-source process for research contracts (e.g., consultants, universities)
- Includes all organizations, for profit, non-profit, academic, and government institutions

The purpose of the RSP is to obtain knowledge and information needed to support the CNSC’s regulatory mission. The program provides the CNSC with access to independent advice, expertise, experience, information and other resources via contracts, grants and contributions placed in the private sector, and with other agencies and organizations in Canada and elsewhere.

The program is compiled from project proposals submitted from across the CNSC that are aligned with the CNSC’s research goals, which are presented in appendix D. The users of these activities include senior management, technical specialists, program managers, etc. The delivery of the research and support activities is supported by internal services, from across different branches that include contracting (from Corporate Services Branch), communications (from Regulatory Affairs Branch), and legal (from Legal and Commission Affairs Branch).

The CNSC’s research and support activities consist of those that are under the RSP stream of the CNSC’s Grants and Contributions Program as well as other research-related activities. The RSP provides funding for extramural ^{Footnote 1} research to both private- and public-sector organizations in Canada and internationally. Other CNSC research activities include contractual arrangements with external parties or memoranda of understanding with other government departments where there is a shared mandate.

CNSC research activities also include partnerships and/or collaborations with other government departments to access independent research. These initiatives include:

The Federal Nuclear Science and Technology Work Plan (FNST) managed by Atomic Energy of Canada Limited (AECL): ^{Footnote 2}. The FNST performs nuclear-related science and technology to support core federal roles, responsibilities and priorities. AECL funds the FNST program, investing $76M annually, and is responsible for its management and oversight. The FNST serves the collective interest of federal government entities in the areas of nuclear safety and security, as well as health, energy and the environment, with a workplan that focuses on 4 research themes and activities. The CNSC engages with AECL collectively, as well as with other government entities to develop work programs that meet its needs and priorities and those of the other federal departments and agencies. The FNST governance structure includes an Interdepartmental Steering Committee, Interdepartmental Integration Committee and 4 themed research sub-committees. The CNSC participates in all FNST committees and co-chairs the Theme 2 sub-committee. The FNST governance structure is presented in appendix B.
The Natural Sciences and Engineering Research Council of Canada (NSERC) and CNSC Small Modular Reactor (SMR) Research Grants Initiative is administered under NSERC’s Grants and Scholarships Transfer Payment Program: In Budget 2022, the CNSC was allocated and transferred funds to partner with NSERC to fund academic research related to SMRs, and $15M in funding is available over a 5-year program. The CNSC and NSERC are working together to increase the scientific information available to support evidence-based decision-making and regulatory oversight, and to increase the CNSC’s capacity to regulate SMRs.

Collectively, the research and support activities provide access to relevant independent advice, expertise, experience, information and other resources via contracts, grants and contributions placed within the research community, and with other parties/stakeholders in the nuclear industry. The activities also aim to gain third-party perspectives on nuclear science relevant to the CNSC and to disseminate scientific, technical, and regulatory information with the nuclear industry and the public at large. They support CNSC-led regulatory processes and decision making, including the identification and understanding of potential hazards and the development of tools and procedures to address new, emerging or ongoing regulatory issues. The outcome of these research and support activities results in scientific and technical information that supports the CNSC and Commission members in regulatory decision making. The logic model for research and support activities is presented in appendix A.

The RSP and other research activities are funded through the following mechanisms:

Figure 1 - Research and Support Program funding expenditures (actual)
Funding type	FY 2023–24 ^{Footnote 3} Total $ expended	FY 2023–24 Number of agreements	FY 2022–23 ^{Footnote 4} Total $ expended	FY 2022–23 ^{Footnote 4} Number of agreements	FY 2021–22 ^{Footnote 4} Total $ expended	FY 2021–22 ^{Footnote 4} Number of agreements
Research contracts	$1,426,560	18	$1,123,348	29	$659,250	20
Contribution agreements	$2,069,587	21	$2,116,087	36	$1,809,949	36
Grants	$73,131	10	$69,500	8	$63,000	6
Total funded	$3,569,278	49	$3,308,935	73	$2,532,199	62

The RSP’s actual spending vs. budget is presented in appendix C.

2. Authority

An evaluation of the CNSC’s research activities was included in the CNSC’s Risk-Based Audit and Evaluation Plan for 2023–25, approved by the President.

3. Objective and scope

The objectives of the evaluation are to assess the relevance, effectiveness and efficiency of the CNSC’s research and support activities.

The evaluation focused on research and support activities of the CNSC, including CNSC activities and outcomes related to the FNST. The NSERC-CNSC Small Modular Reactor (SMR) Research Grants Initiative was excluded from the scope of this evaluation, as this program will be scoped into the Internal Audit Evaluation and Ethics Division (IAEED)’s Evaluation of the Small Modular Reactor Readiness Initiative commencing in Q4 of 2024–25.

The evaluation was guided by 3 questions:

To what extent are the research and support activities meeting the needs of the CNSC?

To what extent are the research and support activities achieving their immediate and intermediate outcomes?

To what extent are the design and the delivery of the research and support activities effective and efficient?

4. Methodology

The list of evaluation questions with the associated indicators, methods, and data sources, are contained in the evaluation matrix in appendix E.

The main limitation of the methodology is that only those familiar with research and support activities were asked to participate. This might underestimate the awareness of the activities among CNSC personnel. As well, it was difficult to assess awareness of research and support products outside the organization, as only 2 external partners were interviewed.

5. Findings and Observations

5.1 Relevance – is the RSP meeting the CNSC’s needs?

Summary

CNSC documents, including the Departmental Report and RSP terms and conditions, indicate that the program is clearly aligned with the CNSC's core responsibility of nuclear regulation and the development of a regulatory framework. Project selection tools, along with the RAC, ensure alignment with organizational priorities and needs. Most interview respondents noted that research and support activities are aligned with CNSC priorities and are meeting needs. Senior management interview respondents suggested that alignment with needs could be improved through greater awareness of the process to identify research topics and greater oversight during the conduct of the research. This alignment with priorities and needs extends to the research conducted through the FNST, according to most of the interview respondents who were familiar with this research.

Alignment with priorities of the CNSC

The objectives ^{Footnote 5} of the RSP are to:

acquire independent expertise, advice and information needed to support timely regulatory judgment and decisions
assist in the identification and assessment of operational problems that may give rise to health, safety, security or environmental hazards
assist in the development of capability and tools to be able to address health, safety, security or environmental issues
facilitate the assessment for the technical or scientific basis of licensing decisions and encourage licensees to address these issues
aid in the development of nuclear safety standards

More information about the Research Advisory Committee (RAC):

The RSP reports twice per year to the RAC

The RAC is chaired by the Chief Science Officer and consists of Directors General from the Technical Support Branch and Regulatory Operations Branch, and representatives from the Radiation Protection Division

The role of the RAC is to:

establish the research and innovation priorities in alignment with the strategic priorities of the CNSC
advise on areas of research interest and needs
provide input on and endorsement of the research plan

Given the focus of the program on supporting regulatory decision making, there is clear alignment with the CNSC's core responsibility of nuclear regulation ^{Footnote 6}. One of the CNSC’s priorities is to have a modern approach to nuclear regulation: “The CNSC is committed to a modern approach to nuclear regulation using science- and evidence-based, risk-informed, and technically sound regulatory practices and regulatory framework that consider scientific uncertainties and evolving expectations.” The Class Grants & Contributions terms and conditions state that the expected result of the research and support activities undertaken by the CNSC is a clear and pragmatic regulatory framework.

According to documentation that provides an overview of the RSP, research projects are prioritized each year using specific criteria to rank them as low, medium, or high. This ranking is based on urgency, as well as the link to CNSC priorities and research goals. Readiness to contract, and executability are also prioritization criteria used by IRD. According to interview evidence, one group has implemented a research needs and evaluation form for staff to fill out that includes a section on how research aligns with the CNSC’s mandate and safety requirements. The form has been adopted by IRD and is currently being piloted. In addition, prior to approval of a grant & contribution, the expected research product is assessed to ensure that it aligns with organizational priorities.

The mandate of the RAC is to “ensure that the use of CNSC’s resources is effectively and efficiently prioritized to meet the CNSC’s regulatory mission.” ^{Footnote 7} Discussions about research goals and alignment with priorities and research needs takes place at the RAC. This role and the importance of the RAC to ensure alignment was mentioned by a few interview respondents (including senior management and program personnel). However, one interview respondent (senior management) also mentioned that alignment relies on working level personnel.

Most respondents across all interview types (including all program personnel, most senior management, and most other CNSC personnel) stated that research and support activities at the CNSC are closely aligned with the core responsibilities and strategic priorities of CNSC. Specifically, alignment includes activities that serve the part of the CNSC’s mandate on disseminating objective scientific information, which can include publishing research abstracts.

The IAAED reviewed 14 post project effectiveness reviews and mini-evaluation reviews performed by IRD on completed contracts/renewed contribution agreements to review how the research was used and if there was alignment with the CNSC’s priorities. Upon review of the sampled documentation, it was noted that the research completed was aligned with the priorities of the CNSC. For example, each post project evaluation reviewed met at least 2 of the CNSC’s 4 strategic priorities. This included research results that supported the strategic priorities through providing high quality advice to CNSC, development of nuclear safety standards, regulatory research on new nuclear technology, as well as outreach. This is consistent with most of the interview respondents confirming the alignment between research and support activities and the CNSC’s core responsibilities and strategic priorities. This also supports the role of the RAC in ensuring that research is prioritized to meet the CNSC’s regulatory mission.

IRD will compile and update an annual Research Plan that outlines all of the planned extramural research that will be undertaken over the next 3 years. The research plan is compiled from project proposals submitted by CNSC staff. Each project is linked to one of the CNSC’s research goals (presented in appendix D). The evaluation noted that the 2024–27 Research Plan documented the CNSC's research goals and aligned with CNSC priorities, and it was approved by the Integrated Planning and Resource Management Committee and RAC. According to RAC discussion notes regarding CNSC research priorities, members discussed if the Research Plan allocated funding in right places and if past funding was well spent, among other considerations. RAC members approved the Research Plan, signalling agreement on the purpose of research and support activities. RAC members span across the CNSC areas with interest in research (Regulatory Affairs Branch, Regulatory Operations Branch and Technical Services Branch); therefore, the approval of the Research Plan indicates that the research and support products are aligned with research interests across the CNSC.

In terms of research conducted by other partners, the CNSC Annual Research Plan (2024), cites 10 research goals that are aligned with CNSC priorities, along with the planned research undertaken by its partners, which also aligned with CNSC goals. Moreover, the CNSC is listed as a primary stakeholder in 43 of 90 continuing projects and in 24 of 45 new projects recommended for FNST funding in 2024–25.

When asked about the research conducted by other partners (such as AECL for the FNST program), most program personnel interview respondents see good alignment with CNSC priorities. Conversely, most senior management interview respondents who were familiar enough to comment indicated that the research done through the FNST does not always align with the CNSC's core responsibilities and strategic priorities and that instead, the research is sometimes driven by AECL's internal priorities.

Meeting the needs of the CNSC

The purpose of research and support activities is clearly documented (in the terms and conditions, communications materials developed by IRD). As well, interviewees were clear on the research and support activities undertaken their overall purpose (to contribute to decision making pertaining to the regulatory framework). Most of the budget pertaining to research and support activities was expended in 2023–24 (94%, financial spend vs. budget is presented in Appendix C), suggesting there is a need for research projects. As well, a request was submitted in 2023–24 for an increased budget allocation due to an increase in the number of priority research projects requested.

All program personnel interview respondents indicated that the purpose of the research and support activities undertaken by the division is well understood by those who use the research. There is less understanding among those within the CNSC who are not directly involved. A few program personnel interview respondents suggested there is an opportunity to improve communications (e.g., minimal current presence on the CNSC Hub/intranet, transitioning to SharePoint). It was noted by a program interview respondent that there is an opportunity to improve the understanding among those requesting research of the different mechanisms to secure research resources (e.g., contribution agreements, grants, contracts).

Interview respondents generally supported that research and support activities meet the needs of the CNSC. In particular, all program interview respondents and most other CNSC interview respondents indicated that the organization’s needs are being met through the research undertaken. Program personnel commonly mentioned the RAC and the post-project evaluations as important mechanisms to ensure that research is meeting the needs. Another CNSC interview respondents pointed out that collaborative projects allow for significant contributions with relatively small budget. Another respondent stated the importance of collaborating with students, letting them gain knowledge and potentially join the CNSC in the future. One program respondent mentioned that a more top-down approach to identify research projects could result in more alignment with CNSC needs than the existing bottom-up approach.

The post-project evaluations reviewed by the evaluation team indicated that the research undertaken is meeting the CNSC’s needs. This included research results that supported the CNSC’s needs through providing high quality advice to CNSC, development of nuclear safety standards, regulatory research on new nuclear technology, as well as outreach. This is consistent with the opinions of all program interview respondents and most other CNSC interview respondents.

There was mixed evidence from senior management interview respondents. Only 1 respondent confidently stated that the program is meeting the CNSC’s needs. The other senior management interview respondents said that the extent to which needs are being met is variable and depends on several factors that offer opportunities for improvement. There should be greater awareness of the process to identify research topics. There can be low awareness of available research products, and awareness influences the products’ usage, which leads to meeting needs.

It was noted that there are communication challenges during the conduct of research, including a lack of status reports, strategic oversight, and validation of the ongoing relevance of the research as it is carried out. Suggestions for opportunities provided by program interview respondents echoed those from senior management, including a need for greater outreach and awareness about research opportunities, with a suggestion to target different teams within CNSC who may not be aware that research is a possible tool when doing callouts. Program personnel respondents also mentioned that awareness and accessibility of research products could be improved.

A few other CNSC respondents observed that research activities suffer from limited funding and competition for resources.

In terms of the extent to which research conducted through the FNST is aligned with CNSC needs, among those that could comment, some said that FNST research is less aligned with immediate CNSC needs and can be less valuable to the CNSC. The FNST is a federal program and was established to meet the needs of all participating federal partners, including the CNSC. However, the CNSC has significant involvement in the program: as noted above, the CNSC is listed as a primary stakeholder in several new and ongoing FNST projects. It also sits on 5 of 6 FNST committees.

5.2 Effectiveness – Are outcomes being achieved?

Summary

Most outcomes are being achieved, according to those interviewed for the evaluation. However, there is a lack of documentation to support most of these assertions. Awareness of research and support products appears to be good among those consulted for the evaluation, but awareness could be improved, according to some. Awareness outside the CNSC could not be fully assessed. Internal interview respondents provided several examples of how research and support products are used by the organization: to address uncertainty or lack of clarity in a licensee’s proposal; to help technical specialists better support or challenge the licensee’s position, informing decision making; to validate licensing decisions; to develop regulatory policies and documents; to build and maintain technical knowledge and awareness of industry standards; and to contribute to participation in international forums and academic activities.

The usefulness of research and support products is occasionally impacted by delays. The extent to which the products are contributing to the regulatory framework is unclear. While half of other CNSC interview respondents said they do impact the framework, senior management interview respondents disagreed, citing lack of senior management direction and oversight, lack of awareness, delays, and products with limited impacts.

Another outcome explored by the evaluation is the development of nuclear expertise in the research community. The evidence was varied, with some interview respondents saying capacity is good and other saying it is not sufficient to meet CNSC’s needs. The availability of funding was noted as an important driver of interest within the research community.

Research undertaken by THE FNST is relevant to and leveraged by the CNSC

As noted above, the Research Plan 2024 cites 10 research goals aligned with CNSC priorities and maps FNST research to each goal, demonstrating that FNST research products are aligned with CNSC research interests. Based on FNST project lists, the CNSC is named as a primary stakeholder in 43 of 90 ongoing projects recommended for funding by FNST in FY 2024–25. In the list of recommended new projects, the CNSC is named as the primary stakeholder in 24 of 25 projects, and as a secondary stakeholder in 20 of 45 new projects for 2024–25.

Program interview respondents were confident that FNST work is aligned with CNSC strategic priorities and needs, noting that it has a good track record of taking on work that meets the RSP’s interests. One other CNSC respondent confirmed that FNST research products are used for technical assessments in their division. Additionally, according to program interview respondents, it was mentioned that CNSC gets good results because it actively participates in committees to ensure its interests are met. The CNSC sits on 5 committees, 2 of which it co-chairs: the Research Theme Sub-Committee 2, and the Environmental Stewardship and Waste Management Committee. However, most other CNSC interview respondents were not aware of FNST products and/or some saw limited usefulness in FNST research products (due to limited CNSC input on which products are developed).

Identification and increased understanding of new, emerging or ongoing regulatory issues of interest to the CNSC

The document review confirmed there is a process in place to ensure that research addresses issues of interest to CNSC. This process is outlined above when discussing the alignment of the research and support activities with needs of the CNSC.

Program interview respondents noted that awareness of the research results could be better among others at CNSC, and a few noted there is an opportunity to leverage the upcoming introduction of SharePoint, a tool for content management, document sharing and team collaboration, to increase communication.

When interview respondents were asked how research and support products are used by the department, most interview respondents across the respondent types were able to provide explanations and examples. Generally, research projects are usually initiated to address uncertainty or lack of clarity in a licensee’s proposal. The knowledge gained from the research results helps technical specialists better support or challenge the licensee’s position, informing decision making. Other types of outputs produced and used by the department include standards, computer codes, and models used to validate licensing decisions. Research and support products also help develop regulatory policies and documents.

Some interview respondents mentioned that research and support products are also used to build and maintain technical knowledge and awareness of industry standards. A few interview respondents said that research and support products also contribute to the participation in international forums and academic activities.

A majority of the other CNSC interview respondents reported that the research and support product(s) they requested generally met their needs. Two of those interview respondents pointed out that reports sometimes need additional phases to cover more details.

Most other CNSC interview respondents also noted that research and support products are sufficiently timely to meet decision-making needs. Having said that, a similar proportion also acknowledged that there are delays receiving products, attributed to COVID-19, challenges finding resources to do the research and delays in approval and contracting processes due to backlogs. A few senior management interview respondents said that research products are not sufficiently timely to meet decision-making needs, with one noting recurring missed opportunities for research to make an impact.

According to most program and other CNSC interview respondents, the main driver of use is the alignment of research products with regulatory and operational needs. The next most mentioned influence for whether a product is used was the timeliness of the product’s availability (mentioned by some other CNSC interview respondents). For a few programs and other CNSC interview respondents, the influence comes from the availability of external resources (capacity, time or facilities to conduct research) and the products themselves. Some other CNSC interview respondents said that use is influenced by the accessibility of the process and the level of collaboration and support from IRD. One program respondent also mentioned awareness of existing research products as an influence.

Clear evidence-based regulatory framework that supports nuclear safety

The 3-year Research Plan outlines research goals are aligned with areas outlined in the CNSC Departmental Plan, as well as aligned with this regulation priority (i.e., having a clear evidence-based regulatory framework). There are 54 projects in the current 3-year Research Plan. Of the 54 planned projects, 1 specifically refers to the CNSC’s regulatory documents.

To influence the regulatory framework, CNSC employees must first be aware of the information generated through the research and support activities. Most of the other CNSC interview respondents consider themselves sufficiently aware of the research and support products available through IRD. ^{Footnote 8} One of those interview respondents pointed out a lack of awareness of FNST and broader government research products. Some of the CNSC interview respondents had the impression that most staff are not fully aware of available products. According to these interview respondents, awareness is lacking, in particular, among new staff. In addition, CNSC interview respondents indicated that they were unclear on how to access available products.

When asked about how they become aware of research and support products, some of the CNSC interview respondents become aware through the internal IRD database, although 2 of those interview respondents specified that the database is not user-friendly and hard to search. For one respondent, direct need influences their awareness and for another respondent, annual report on research activities is informative. Communication with IRD was mentioned by 1 respondent as influencing their awareness. A few of the other CNSC interview respondents suggested IRD should increase communication through a newsletter on research activities and presentations to staff.

A few program interview respondents identified opportunities to improve CNSC staff’s awareness of the research and support activities being undertaken. In particular, they mentioned a need for better knowledge transfer, and increased awareness through improved communications to CNSC staff through more presentations and newsletters to inform staff. One senior management respondent suggested that it should be those who requested and are using the research to be promoting the research results within the CNSC.

Only 2 program interview respondents could comment on the extent to which research and support products contribute to the CNSC’s regulatory framework and both interview respondents felt strongly that they did. They noted that this is accomplished through the development of standards through the CSA Group and informing technical documents, as well as outputs from international organizations.

According to half of the other CNSC interview respondents, research and support products contribute to the CNSC’s regulatory framework and decision making; for example, in the development of standards and regulatory documents and in the influence on technical documents. One of the interview respondents indicated that the impact of research and support products on regulatory activities is reviewed through an effectiveness check. Senior management interview respondents, however, did not feel that the regulatory framework has been supported by research and support products, citing lack of senior management direction and oversight, lack of awareness, delays, and products with limited impacts.

External stakeholder awareness of nuclear regulatory information

The primary preoccupation of the research and support activities is to support decision making at the CNSC. Awareness of external stakeholders was not a major focus of the interviews held for the evaluation.

There is data about awareness of CNSC research products through the requests for research products posted on the CNSC website. In 2023, products were requested 32 times by various users, including products produced in each FY (2019 to 2023). The total number of products requested was 15, illustrating that multiple users requested the same paper(s). Users ranged from individuals (where their email could not be associated with an organization), educational institutions, others within the Government of Canada, law firms, nuclear organizations worldwide, etc.

According to the partners interviewed for the evaluation, not surprisingly, awareness of research products depends on the dissemination methods and whether a strategy is developed to communicate the results. For example, the research undertaken via the FNST is confidential and is not shared outside the federal government. However, another partner reported that their research outputs are made available to other researchers through conferences and publications.

The evaluation team performed a review of 14 post-project evaluations performed by IRD on completed contracts/renewed contribution agreements to review how the research was utilized, including the extent to which the research products were disseminated externally. It was noted that of the post-project evaluations reviewed, 11 involved external dissemination, including presentations at technical conferences, other events, publications, etc. The research was utilized through provision of high-quality advice to the CNSC, development of nuclear safety standards, regulatory research on new nuclear technology, as well as outreach.

From the benchmarking conducted for the evaluation, it was also noted that one of the other federal organizations uses the Federal Open Science Repository (FOSR) to document and communicate research outcomes. The FOSR is a web-based platform for scientific articles and publications from Government of Canada science-based departments and agencies. It consolidates, preserves, and provides access to research from the federal government community.

Identification and development of nuclear expertise in the research community

Views were split among program and partner interview respondents about the capacity of nuclear expertise in the research community. About half of the interview respondents said that the research community generally has sufficient capabilities and capacity to respond to requests for research to support the CNSC. However, some mentioned low response rates to requests for proposals due to low awareness among potential researchers or budget constraints. It was also mentioned that the research community is small and divided between doing research for the CNSC and doing research for industry.

One program respondent stated that much of the research is conducted internationally to ensure independent viewpoints.

Most other CNSC interview respondents said that the availability of expertise in the research community drives the community’s capacity to carry out research and support activities for the organization. Some program personnel, about half of other CNSC interview respondents and all partner interview respondents noted that having funding available has a critical influence on the capabilities and capacity of the research community. While 1 program personnel respondent has observed strong interest and expertise from the research community (as evidenced by responses to requests for proposals), another program respondent expressed some concern about declining interest in nuclear disciplines. Some of the other CNSC interview respondents suggested that better allocation of funds and taking a more strategic view is a potential influence for interest within the research community.

It was suggested that the CNSC could influence these capabilities and capacity by enhancing communication through direct outreach to/partnerships with universities and by leveraging existing networks. All partner interview respondents mentioned efforts are being made to improve national coordination of the research community, but that it is difficult and limited by funding and resource constraints.

5.3 Efficiency – Are Design and Delivery Efficient?

Summary

Research activities at the CNSC are generally efficiently designed, delivered and resourced. The RAC provides oversight and guidance, and roles and responsibilities are considered to be clear (although several suggestions were made to increase clarity). Senior management interview respondents would like to see more senior management oversight and direction, including more progress reports to senior management so that the ongoing relevance of the work can be assessed, and more top-down processes to identify research needs.

Processes for project identification, project selection and funding decision-making are efficient, appropriate, and aligned with best practices, according to most consulted for the evaluation. However, there are opportunities for improvement, particularly with respect to the timely issuance of agreements, which was highlighted by interview respondents across respondent types. Partnerships (with AECL, for example) are effective.

In terms of performance measurement, IRD carries out reviews and evaluation of all its contracts and contribution agreements, respectively. However, it does not appear that this information is used for reporting against the outcomes in the Performance Information Profile (PIP) and senior management interview respondents reported that information about research and support activities for decision-making is insufficient.

Three interview respondents noted there is insufficient information available to understand the progress of research projects and activities, as well as the benefits and value of the work. It was noted that additional details and performance check-ins will assist decision makers in determining the value of the research.

Governance

As noted, the RAC is the governance body for research and support activities at CNSC.

All program interview respondents indicated that roles and responsibilities are clearly defined with respect to the delivery of research and support activities at the CNSC. Those roles and responsibilities are described in work instructions internal to IRD.

Most other CNSC interview respondents agreed that roles and responsibilities are clearly defined with respect to the delivery of research and support activities at the CNSC. Two of those interview respondents pointed out that it may not be as clear for newer staff and 1 respondent stated that many people are involved, which causes delays.

As noted above, senior management interview respondents would like to see more senior management oversight and direction, including more progress reports so that the ongoing relevance of the work can be assessed, and more top-down processes to identify research needs. These interview respondents would also like to see more timely development of agreements, and enhanced partnerships to align CNSC work with international priorities.

Some program interview respondents and other CNSC interview respondents noted opportunities to improve the clarity of roles and responsibilities. Program interview respondents suggested there should be a greater distinction of roles (such as between technical specialists and project/program officers). Other CNSC interview respondents indicated a need for clarification of role descriptions and project guidelines, and suggested the development and communication of a process document or flowchart would be beneficial to clarify roles. Some other CNSC interview respondents indicated roles and responsibilities could be clarified by streamlining the processes, in particular contracting, and reducing staff intervention cycles.

Adequacy of Resources

RSP expenditures over the timeframe of the evaluation have steadily increased from $2.5M in FY2021–22, $3.3M in FY2022–23, and most recently $3.6M in FY2023–24, while the total number of agreements issued has varied over the same time frame, changing from 62 to 73 to 49, respectively, each year. IRD has a total of 5 FTEs to conduct the activities associated with delivering research and support activities, including the RSP.

According to program interview respondents, this level of resources is appropriate to deliver the research and support activities for the CNSC, and in fact the program has undertaken some recent hiring.

Processes

There is a lack of updated documentation on processes pertaining to the RSP or other research and support activities. Work instructions exist but are outdated (2016). ^{Footnote 9} Documents pertaining to process are not available in the CNSC Navigator Portal, a management system tool for documentation/policies/ protocols/etc. Additionally, no information about process is available on the CNSC intranet (CNSC Hub).

According to most of those consulted for the evaluation, program processes for project identification, project selection and funding decision making are efficient, appropriate, and aligned with best practices. Most other CNSC interview respondents provided positive feedback about how they communicate their research interest to IRD, as well as other communication with IRD. IRD was noted to be responsive and efficient. Senior management interview respondents provided mixed opinions regarding processes, with 2 saying it is inefficient, cumbersome and long, and 2 describing it as efficient and appropriate.

Best practices being leveraged include using a peer-review process, following guidelines, being agile to respond to specific needs and urgencies, encouraging staff participation in research, and sharing information. A potential best practice, highlighted by 1 program respondent, was the adoption of the AECL project ranking system and visual presentation for funding levels.

The 2 other Government of Canada organizations interviewed as part of this evaluation indicated that their departments strive to follow best practices, but have experienced challenges. Both organizations engage in external research activities and in similar research processes to that used by the CNSC; however, they also engage in different activities that the CNSC may be able to learn from. For example, 1 respondent indicated that they leverage best practices from other funding programs within their department. The same organization also utilizes standardized templates, including newsletter and communication templates, as well as standardized processes and timelines, including presentations to specific committees at designated times throughout the year to seek feedback on research ideas. The second respondent indicated that they also utilize standardized templates and processes, as well as leveraging the expertise from other departments through interdepartmental discussions on research to ensure they are utilizing approaches that build on best practices in other departments. They also staff a departmental science advisor (DSA), who is part of the Departmental Science Advisors Network, that is made up of DSAs from other Government of Canada organizations and chaired by a chief science advisor.

Several other CNSC interview respondents noted opportunities for improvement, including: have more communication about FNST research products and processes; adoption of newer methodologies or technologies; an increase in engagement with universities and research departments; better documentation of workflows; more detailed identification of gaps; a centralized research hub; use of standing offers and supply arrangements, which would speed up contracting; and enhanced clarity on the prioritization process for research proposals.

Feedback about processes was less positive with respect to the timeliness of the delivery of research products, with just under half of other CNSC interview respondents indicating that delivery is usually timely. Senior management interview respondents also voiced concerns with the timeliness of products. This feedback could be driven by perceptions of the timeliness of agreement issuance. In fact, a majority of other CNSC interview respondents mentioned inefficiency and ineffectiveness because of the long delays of the process, especially with the contracting phase. Concerns about the issuance of agreements was echoed by most program interview respondents, noting it is not timely and represents a significant obstacle. Several interview respondents said it can take 6 months to have an agreement issued after IRD has sent the paperwork. This stage was described as being a bottleneck that results in delayed research projects and therefore a delay in the CNSC receiving the results. The delays can also cause budget lapses. Of note, both benchmarked federal organizations use standing offers/supply arrangements to maximize the efficiency of the process.

Relationships with Partners

Working in partnership with other organizations was identified by some interview respondents as a best practice to maximize the efficiency of research and support activities of the department. The RSP has a few partnerships with other organizations. The most significant (in terms of involvement on committees) is the partnership with the FNST at AECL. The CNSC’s role on committees is well documented in terms of reference and generally well understood at the CNSC. The terms of reference for the FNST outline the governance structure to guide and advise on the FNST workplan, roles, responsibilities, and priorities, as well as overall contractual arrangements. The CNSC sits on 5 committees, of which it co-chairs 2: the Research Theme Sub-Committee 2, and Environmental Stewardship and Waste Management committee. The planning guide outlines the FNST’s relationship with participating organizations (including the CNSC) and work priorities and scope by theme area. FNST also provides quarterly updates/progress reports.

The program also has a partnership with the University Network of Excellence in Nuclear Engineering (UNENE). ^{Footnote 10} The program was recently renewed for an additional 3 years, effective August 2024, valued at $390K. The agreement signed with UNENE included terms and conditions, the contribution amount, the project description, and execution details. As a UNENE stakeholder, the CNSC participates on the UNENE Board of Directors as a non-voting member. The CNSC also participates on UNENE’s Research and Education Advisory Committees, which allows the CNSC to influence UNENE’s research so that regulatory priorities for emerging safety issues are addressed in the program.

UNENE publishes an annual report, which outlines work accomplished through the year.

Most program interview respondents noted that the relationships with external partners are documented (through terms of reference or contribution agreements) and are generally working well. One respondent observed that some partnerships are not formally documented but rely on personal relationships and ongoing collaboration.

The 2 partner organizations interviewed for the evaluation noted having very strong and beneficial relationships with CNSC, to address knowledge gaps, increase opportunities for collaboration, and minimize duplication of efforts. One of the benchmarking interview respondents noted CNSC is absent from interdepartmental discussions about research and suggested that there might be opportunities for CNSC to engage that community.

Atomic Energy of Canada Limited (AECL) & Federal Nuclear science and Technology (FNST)

AECL, a federal Crown corporation is responsible for managing nuclear science, technology, and waste in Canada.
The FNST program, led by AECL, maintains Canada’s nuclear science and technology capabilities.
AECL oversees nuclear research and decommissioning activities.
The FNST supports research and innovation focused on safety, health, and environmental protection within the nuclear sector.

University Network of Excellence in Nuclear Engineering (UNENE)

UNENE is a collaborative organization that brings together Canadian universities and nuclear industry partners.
It focuses on advancing nuclear research, providing graduate-level education, and supporting innovation to meet the evolving needs of the nuclear sector.

Performance Measurement

Performance measurement is a systematic approach used to assess the efficiency and effectiveness of projects, programs, and initiatives in an organization to ensure that desired outcomes are achieved. The Government of Canada’s Policy on Results sets out the fundamental requirements for federal departmental accountability for performance information and evaluation, while highlighting the importance of results in management and expenditure decision making, as well as public reporting. As part of this requirement, CNSC has developed an approved Performance Information Profile (PIP) for the Scientific, Regulatory and Public Information Program, which outlines expected outputs and outcomes, indicators, and a strategy for collecting performance data. The Strategic Planning Directorate coordinates the performance measurement at the CNSC. Having evidence to support the regulatory framework is an outcome on the PIP, however, the evaluation was not able to obtain information to assess progress towards this outcome.

According to most program interview respondents, the reporting of research and support activities is mainly focused on outputs, such as financial components (budget utilization and invoice payments) and project execution. Also, research results are communicated through internal channels at the CNSC, and all research results are in the Curie library.

Some program interview respondents stated that performance measurement is challenging due to the long-term nature of research and because there are different parties involved (e.g., invoice payments are tracked by Finance).

Based on available documentation and data, it does not appear that research-related outcomes in the detailed PIP are monitored. In particular, the PIP outcome data is missing for some outcomes. The document provided for the evaluation ^{Footnote 11} does not demonstrate measurement of:

Outcome	Measurement	Indicator
1	The CNSC provides high quality advice to Commission and regulatory programs	3 indicators related to research and support activities, of which all are missing data (completeness of regulatory framework, number of non-peer-reviewed papers/presentations/posters/abstracts by scientific staff, number of articles in peer-reviewed journals by scientific staff)
2	CNSC research needs and associated risks are addressed – (timeliness of posting research abstracts)	5 indicators related to research and support activities, of which 1 is missing data
3	Research provides high quality expertise and advice to staff, the CNSC and regulatory programs	1 indicator, which is missing data (client satisfaction)
4	Increased clarity of regulatory requirements and guidance that supports nuclear safety	1 indicator, which is missing data (completeness of regulatory document framework)

The 2 other Government of Canada organizations indicated that they adopt various approaches to measuring the effectiveness of their research programs. One organization indicated that data is collected and reported on for the following other metric types: internal and external website hits, research dissemination, funding recipient information, etc. The second organization indicated that it is looking to adopt Technology Readiness Levels as a model and adopting a similar approach in the form of Regulatory Readiness Index. Technology Readiness Levels is a project management tool that helps track project development stages. It measures the technical maturity of a technology.

According to the review of documents, all projects undertaken via a contribution agreement are evaluated for renewed agreements. Those evaluations explore the achievement of project outcomes and goals. Over the 3 fiscal years 2021–22, 2022–23 and 2023–24, a total of 41 evaluations were conducted. For contracts, effectiveness reviews are conducted upon completion of the contract. Over the 3 fiscal years 2019–20, 2020–21 and 2021–22, a total of 28 reviews were conducted.

As part of the 14 post-project evaluations reviewed by IAEED on renewed contribution agreements and completed contracts, it was noted that there was alignment with the research completed and the research related outcomes documented. This included research that provided high-quality advice to the CNSC, development of nuclear safety standards, regulatory research on new nuclear technology, as well as outreach. Many of the projects reviewed covered more than one of the outcomes. The post project evaluations indicated that all projects had some component of internal and/or external dissemination through various mechanisms including, publications, presentations, discussions at events/forums, REGDOC revisions, etc. Senior management interview respondents largely reported that information about research and support activities for decision-making is insufficient (a few said the information was satisfactory). While 1 respondent acknowledged that there is a new dashboard, 3 respondents said that briefings and updates on ongoing projects are too general and not sufficiently detailed for decision makers to understand the progress and benefits of research activities, and to make decisions on the continuing or stopping of specific research projects. It was also noted that additional details and performance check-ins will assist decision makers in determining the value of the research.

6. Conclusions and Recommendations

6.1. Conclusions

The evaluation found that the Research and Support Program (RSP) stream of the CNSC’s Grants and Contributions Program as well as other research-related activities are clearly aligned with the CNSC’s core responsibility of nuclear regulation and development of a regulatory framework.

The alignment of research with the department’s priorities and needs is generally ensured through project selection tools in conjunction with the RAC. However, senior management would like to see greater alignment with needs, through increased engagement of senior management in the research identification process, as well as greater oversight during the research execution phase to allow for continuous monitoring. These observations also extend to the research conducted through the FNST.

Overall, most outcomes are being achieved. However, this conclusion is based on input from interviews for the most part; there is a lack of documentation to support these claims. As well, while awareness of the research and support products appears to be good, this was awareness among those interviewed for the evaluation who were identified due to their existing familiarity with the activities. Thus, the level of awareness among others at the CNSC is unclear, although some interview respondents identified this as an area that needs improvement. The awareness of research among external stakeholders was not a major focus of the interviews held for the evaluation and could not be fully assessed. The evaluation identified many examples of how research and support products are used by the CNSC, including: addressing uncertainty or lack of clarity in a licensee’s proposal; helping technical specialists better support or challenge the licensee’s position, informing decision making; validating licensing decisions; developing regulatory policies and documents; building and maintaining technical knowledge and awareness of industry standards; and contributing to the participation in international forums and academic activities. Occasional delays in the delivery of research and support products can impact their usefulness.

In terms of the extent to which the research and support products contribute to the regulatory framework, the evaluation found this to be unclear. While half of other CNSC interview respondents said they feel that the products impact the framework, senior management interview respondents disagreed, citing lack of senior management direction and oversight, lack of awareness, delays, and products with limited impacts.

An additional outcome explored by the evaluation was the development of nuclear expertise within the research community. The evidence in this regard was varied with little agreement among those consulted for the evaluation. The availability of funding was noted as an important driver of interest within the research community.

With respect to efficiency, research activities at the CNSC are generally efficiently designed, delivered and resourced. As well, this is supported by the RAC’s role to oversee and review the CNSC’s research efforts from an integrated perspective. While there were several suggestions to improve the clarity of roles and responsibilities, generally they are considered clear. According to most consulted for the evaluation, processes for project identification, project selection and funding decision making are efficient, appropriate and aligned with best practices.

In addition to the above-noted items related to efficiency, other opportunities for improvement were noted, particularly with respect to the timely issuance of agreements. However, the evaluation found that partnership contracts are (effective (those with AECL, for example). In terms of performance measurement, IRD carries out reviews and evaluation of all its contracts and contribution agreements, respectively; however, it does not appear that this information is used for reporting against the outcomes in the PIP. The evaluation also found that information for decision making is insufficient.

6.2 Recommendations

The successful implementation of the recommendations identified in this report, along with the completion of other initiatives already underway by management, will strengthen the management of research and support activities at the CNSC.

It is recommended that IRD enhance overall CNSC awareness of research programs available from IRD, as well as completed research products, to encourage the utilization of research and support activities within the CNSC.
It is recommended that IRD perform a review over the planning of RSP contracts to help facilitate and improve the efficiency of contract issuance. This may be completed in consultation with Contracting.
It is recommended that IRD work to enhance the level of detail in dashboards, briefings, updates and other material related to ongoing projects to improve the information available to decision makers and to ensure ongoing alignment with needs. This will help decision makers further understand the progress and benefits of specific research activities, as well as assess the continuation of research projects.
It is recommended that IRD improve reporting against specific research outcomes identified in the PIP for the Scientific, Regulatory and Public Information Program.

A management action plan (MAP) to address this evaluation’s 4 recommendations can be found in appendix F.

Appendix A: Logic Model for Research and Support Activities

Appendix B: FNST Governance Structure

Appendix C: Research and Support Funding: Actual Spending vs. Budget

Funding type	FY 2023–24 ^{Footnote 12} Total $ expended	FY 2023–24 ^{Footnote 13} Total $ budget	FY 2022–23 ^{Footnote 14} Total $ expended	FY 2022–23 ^{Footnote 15} Total $ budget	FY 2021–22 ^{Footnote 13} Total $ expended	FY 2021–22 ^{Footnote 13} Total $ budget
Research contracts	$1,426,560	$1,518,294	$1,123,348	$1,718,294	$659,250	$1,560,043
Contribution agreements	$2,069,587	$2,212,841	$2,116,087	$2,112,841	$1,809,949	$2,012,841
Grants	$73,131	$75,000	$69,500	$75,000	$63,000	$75,000
Total	$3,569,278	$3,806,135	$3,308,935	$3,906,135	$2,532,199	$3,647,884

Appendix D: Research Goals FY 2024–25 to FY 2026–27

Research goals

1. Strengthening the CNSC’s licensing, compliance and regulatory framework in preparation for long-term operation / post-refurbishment operation of Canadian nuclear power plants

2. Supporting CNSC staff in their evaluation of pre-licensing and licensing submissions related to advanced reactor technologies

3. Preparing to regulate activities using new technologies (e.g., AI, drones, VR) and explore potential use by the CNSC

4. Enhancing the CNSC’s understanding of the environmental transport and behavior of hazardous/nuclear substances and associated environmental exposures

5. Informing the CNSC’s radiation protection knowledge base to reflect the best available science with respect to the protection of workers and the public

6. Supporting CNSC staff in their evaluation of licensing or other submissions related to waste repositories

7. Supporting Canadian safeguards commitments and influencing international safeguards efforts

8. Strengthening Canadian nuclear forensics capability

9. Supporting the development of standards for nuclear safety and security

10. Supporting capacity building, outreach and safety promotion

Appendix E: Evaluation Matrix

Evaluation Question 1: To what extent is the design appropriate to achieve its expected outcomes?

Indicator	Data source	Method
1.1 Research and support activities are strategically aligned with the core responsibilities and strategic priorities of the CNSC	Departmental plan Senior management Program personnel Other CNSC personnel	Document review Interviews
1.2 Research and support activities meet the needs of the CNSC (including those areas that request research and those that do not)	Senior management Program personnel	Interviews
1.3 The purpose of research and support activities at the CNSC is clear and there is general agreement among internal stakeholders	Program documentation Program personnel	Document review Interviews
1.4 The scope (including budget) for research and support activities is sufficient to respond to requests for research and support decision making	Program planned and actual financial allocation Program personnel	Document review Interviews
1.5 Role of research conducted by partners (i.e., AECL) contributes to the CNSC’s core responsibility and priorities	Program documentation Senior management Program personnel Other CNSC personnel	Document review Interviews

Evaluation question 2: To what extent are the research and support activities achieving their immediate and intermediate outcomes?

Indicator	Data source	Method
Immediate outcome: Research undertaken by the FNST is relevant to and leveraged by the CNSC 1.1 FNST research products are aligned with CNSC research interests 1.2 CNSC personnel use research products developed through the FNST (optional)	Documentation pertaining to meetings with AECL and evidence that FNST research used by the CNSC Program personnel Other CNSC personnel	Document review Interviews
Immediate outcome: Identification and increased understanding of new, emerging or ongoing regulatory issues of interest to the CNSC 1.3 To what extent are CNSC-funded research and support products are aligned with research interests? 1.4 CNSC-funded research and support products are available in a timely manner (adapted from PIP) 1.5 CNSC personnel use research and support products	Documentation (3-year research plan, RAC agendas and minutes, tracking sheet, other) Senior management Program personnel Other CNSC personnel	Document review Interviews
Immediate outcome: Increased interest within the research community of issues related to nuclear safety 1.6 Number and breadth of researchers contributing to research and support products, by research area, institution, over time	Documentation (contribution agreements, grants, contracts) Program personnel CNSC external partners	Document review Interviews
Immediate outcome: Internal and external stakeholders are aware of nuclear regulatory information 1.7 Number of times documents are accessed, by type of user; web page visits 1.8 CNSC personnel and external partners indicate awareness of CNSC research publications and how to access them	Documentation (Statistics Canada Federal Science Survey, International Atomic Energy survey, web stats, tracking documents) Program personnel Other CNSC personnel CNSC external partners	Document review Interviews
Intermediate outcome: The CNSC has clear evidence-based regulatory framework that supports nuclear safety 1.9 Research and support products support the regulatory framework 1.10 Completeness of regulatory document framework (PIP)	Documentation (Regulatory Framework Division tracking system, TSB tracking system, published regulatory documents) Senior management Program personnel Other CNSC personnel	Document review Interviews
Intermediate outcome: Identification and development of nuclear expertise 1.11 Number of peer-reviewed and non-peer-reviewed research and support products developed (adapted from PIP)	Documentation (CNSC Hub pages, TSB tracking system) Senior management Program personnel Other CNSC personnel	Document review Interviews
1.12 Facilitating and hindering factors associated with the achievement of outcomes	Briefing notes Risk/issues logs RAC minutes Other internal documentation Senior management Program personnel Other CNSC personnel CNSC external partners	Document review Interviews

Evaluation question 3: To what extent is the design and delivery of the research and support activities effective and efficient?

Indicator	Data source	Method
3.1 Resources allocated to the activities (FTE, dollars) are adequate to meet the needs of the CNSC and research/support product users (evidence of unmet needs, lapsing, reprofiling, and reasons for variance)	Financial data and reports Program personnel	Document review Interviews
3.2 Examples of efficiencies and opportunities for improved efficiency	Program personnel CNSC external partners Other government departments	Interviews Benchmarking
3.3 Relationships with partners are documented and working well	Program documentation (MOUs, briefing notes, etc.) Program personnel CNSC external partners	Document review Interviews
3.4 Roles and responsibilities are clearly defined for all relevant staff, and the training tools are developed and implemented	Program personnel Other CNSC personnel	Document review Interviews
3.5 Processes pertaining to research and support at CNSC (including project identification, project selection, funding decisions, CA/grant/contract issuance) are clear and operating effectively, efficiently and in a timely manner	Program documentation (terms of reference for RAC, internal process charts and descriptions, etc.) Senior management Program personnel Other CNSC personnel Other government departments	Document review Interviews Benchmarking
3.6 Design leverages known best practices in other government departments	Available information about how other research programs are set up Senior management Program personnel Other CNSC personnel CNSC external partners Other government departments	Benchmarking Document review Interviews
3.7 Expected outcomes are documented, communicated and measured and performance measurement approach supports decision-making and monitoring	Program documentation (PIP, presentations, briefings, reports, departmental reports, etc.) Senior management Program personnel Other government departments	Document review Interviews Benchmarking

Appendix F: Management Action Plan

Observation	Recommendation	Action Owner	Management Action Plan
There is an opportunity to enhance awareness of the RSP, research activities, as well as research and support products across the CNSC.	It is recommended that IRD enhance overall CNSC awareness of research programs available from IRD, as well as completed research products to encourage the utilization of research and support activities within the CNSC.	IRD	Agrees/Disagrees: Agrees Awareness of Research Program IRD is finalizing its SharePoint Communication site. This will include details of support available to staff and act as a repository of research processes and products. (July 2025) In addition, IRD will take active measures to promote awareness via: a) targeted briefings at division meetings; b) an open learning session; and c) communications products developed with the Strategic Communications Directorate. A plan to periodically refresh these on a biannual (2-year) cycle will be created. (November 2025) Improved dissemination of Research products IRD will develop a template for a tailored dissemination plan to be included with each new research project. The idea is to ensure considerations related to dissemination of research results are included at the front end of the project as is appropriate to the scale/aims of the project. (July 2025) Target date: November 2025
There is an opportunity to help facilitate and improve the efficiency of planning contract issuance.	It is recommended that IRD perform a review over the planning of RSP contracts to help facilitate and improve the efficiency of contract issuance. This may be completed in consultation with Contracting.	IRD	Agrees/Disagrees: Agrees IRD will conduct this review with Contract management Services. The review will focus on: The potential for strategic approaches which examine “bundling” or longer-term programs of work to reduce the overall contracting load (fewer but larger projects). Look at alternative contracting vehicles such as supply arrangements. Review whether repatriating the Contribution Agreement process from Contract Management Services to IRD could reduce the workload on contracting and deliver efficiencies in the process. Assess whether ‘front-end- process efficiencies can be gained. Target date: September 2025
There is an opportunity to enhance the level of detail in research and support activity dashboards, briefings, updates and other material related to ongoing projects to improve information available to decision makers.	It is recommended that IRD work to enhance the level of detail in dashboards, briefings, updates, and other material related to ongoing projects to improve the information available to decision makers and ensure ongoing alignment with needs. This will help decision makers further understand the progress and benefits of specific research activities, as well as assess the continuation of research projects.	IRD	Agrees/Disagrees: Agrees IRD will work with the Strategic Planning Directorate to improve dashboards and quarterly reporting measures with particular focus on projects that have lengthy/amended timelines. The ability to support results/outcomes may be limited due to timescale for quarterly reporting. IRD will review, with input from RAC members, the information required to support the RAC and update its terms of reference as required. To demonstrate outcomes/results more effectively, IRD will present the results of its current research utilization reviews and contribution evaluations at RAC and Management Committee at least annually. IRD will also further explore the CNSC’s potential participation in the Departmental Science Advisors Network to further incorporate best practices from other federal science departments. Target date: November 2025
There is an opportunity to improve reporting against specific research outcomes identified in the PIP for the Scientific, Regulatory and Public Information Program.	It is recommended that IRD improve reporting against specific research outcomes identified in the PIP for the Scientific, Regulatory and Public Information Program.	IRD	Agrees/Disagrees: Agrees IRD will work with the Strategic Planning Directorate to improve the research PIP to ensure the PIP continues to be relevant and that meaningful performance information is collected. Target date: September 2025

Appendix G: Acronyms

The following table presents acronyms used in this document.

AECL	Atomic Energy of Canada Limited
CNSC	Canadian Nuclear Safety Commission
DSA	departmental science advisor
FNST	Federal Nuclear Science and Technology Work Plan
FOSR	Federal Open Science Repository
IAEED	Internal Audit, Evaluation and Ethics Division
IRD	Innovation and Research Division
MAP	Management Action Plan
NSERC	Natural Sciences and Engineering Research Council of Canada
PIP	Performance Information Profile
RAC	Research Advisory Committee
RSP	Research and Support Program
SMR	Small Modular Reactor
TSB	Technical Support Branch
UNENE	University Network of Excellence in Nuclear Engineering

Footnotes

Footnote 1

Extramural research is research performed by organizations outside the federal government with federal funds under contract, contribution agreement or grant.

Return to footnote 1 referrer

Footnote 2

Federal Nuclear Science & Technology Work Plan, AECL.

Return to footnote 2 referrer

Footnote 3

Corporate Services Branch, Innovation and Research Division.

Return to footnote 3 referrer

Footnote 4

2021-22, 2022-23 Departmental Results Report, Canadian Nuclear Safety Commission.

Return to footnote 4 referrer

Footnote 5

CNSC Class Grants and Contributions Program terms and conditions.

Return to footnote 5 referrer

Footnote 6

CNSC 2023-24 Departmental Plan.

Return to footnote 6 referrer

Footnote 7

CNSC Research Advisory Committee (RAC) Terms of Reference. Approved October 21, 2015.

Return to footnote 7 referrer

Footnote 8

Recall that other CSNC respondents were identified by the program as being sufficiently aware of the research and support activities to participate in the evaluation.

Return to footnote 8 referrer

Footnote 9

A proposed tracked changes version of the document was developed in 2020.

Return to footnote 9 referrer

Footnote 10

UNENE, a non-profit organization, brings together Canadian universities, nuclear power utilities, and research and regulatory agencies for the support and development of nuclear education, research, and development capabilities at Canadian universities. UNENE also offers a formal education program catered to working professionals in the nuclear industry.

Return to footnote 10 referrer

Footnote 11