Business-led Networks Of Centres Of Excellence (BL-NCE) Evaluation Report 2012

 Report

---

• Executive Summary
• 1.0 Introduction
  • 1.1 Evaluation Context
  • 1.2 Program Overview
  • 1.3 Overview of Funded Networks
  • 1.4 Report Structure
• 2.0 Methodology
  • 2.1 Evaluation Issues
  • 2.2 Evaluation Methodology
  • 2.3 Limitations
• 3.0 Relevance
  • 3.1 Continued need for a network approach to funding of research, development and innovation
  • 3.2 Role for the federal government in supporting the program
  • 3.3 Alignment with federal government priorities
• 4.0 Implementation
  • 4.1 Impact of program design on achievement of program outcomes
• 5.0 Network approach to research, development and innovation
  • 5.1 Enhancement of research, development and innovation in the areas of funded networks
• 6.0 Mobilization and benefits of network research
  • 6.1 Impact on partner organizations
• 7.0 Training of Highly Qualified People
  • 7.1 Impact on training of HQP
• 8.0 Performance (Efficiency and Economy)
  • 8.1 Efficiency and effectiveness of program delivery.
• 9.0 Conclusions and Recommendations
  • 9.1 Conclusions
  • 9.2 Recommendations
• Annex A – Glossary of Acronyms
• Annex B – Bibliography
• Footnotes

---

Executive Summary

Introduction

As announced in Budget 2007, the Government of Canada invested $46 million over four years for the creation of the Business-Led Networks of Centres of Excellence (BL-NCE) program in support of the science and technology (S&T) strategy – Mobilizing Science and Technology to Canada’s Advantage. The goal of the BL-NCE program is to fund large-scale collaborative business-led networks to enhance private sector innovation in order to deliver economic, social, and environmental benefits to Canadians, and to promote an Entrepreneurial Advantage through the translation of knowledge into commercial applications. As such, the program supports innovation in the five priority areas identified in the Budget 2007 and the S&T strategy:

• Environmental science and technologies;
• Natural resources and energy;
• Health and related life sciences and technologies;
• Information and communications technologies; and
• Management, business or finance.

The BL-NCE program introduces a unique and innovative partnership model, where academic and private sector partners are equally engaged and those best positioned to deliver on the research challenge in a sector are funded. It is the only network program of the three funding agencies (the Natural Sciences and Engineering Research Council of Canada, the Social Sciences and Humanities Research Council of Canada and the Canadian Institutes of Health Research) to allow networks to fund private sector partners directly, so that they may do research within their own facilities. The funded networks are headed by not-for-profit consortia, which leverage funding from industry, to fund research which will lead directly and quickly to products and services which generate revenue, and lead to the creation of jobs.

Currently, four BL-NCEs are being funded by the program: Canadian Forest NanoProducts Network (ArboraNano); Green Aviation Research and Development Network (GARDN); Québec Consortium for Drug Discovery (CQDM); and Petroleum Technology Research Centre – Sustainable Technologies for Energy Production Systems (PTRC-STEPS). Funding amounts for these networks range between $8.9 million and $12.4 million and have been awarded for the four year period from 2009 to 2013. The terms and conditions of the BL-NCE program apply for a four-year period ending in June 2012. These currently funded networks are examining nanotechnology-enhanced forestry products, innovative tools for drug discovery, next-generation aviation technologies, and sustainability challenges relating to hydrocarbon production. In order to encourage and promote partnerships with Small and Medium Enterprises (SMEs), $2.8 million of the BL-NCE program’s budget has been specifically allocated for these organizations.

This is the first evaluation of the program. The evaluation examines the program’s ongoing relevance and performance (effectiveness, efficiency and economy) as well as aspects of its implementation to inform program renewal. It covers the time period from program inception (fiscal year 2007-2008) to the end of fiscal year 2010-2011.

Methodology

The methods used by the evaluation to address the evaluation methodology are summarized below.

Overall, the evaluation methodology is strong in providing the basis for reaching conclusions for all issues and questions using multiple lines of evidence. There are limitations with the evaluation methodology; however, they were carefully taken into account when conducting the analyses, and are acknowledged in the interpretation of the findings. The limitations and mitigation measures taken are described below.

Some issues were encountered with the consistency and availability of performance measurement data collected by the program. For example, some networks gathered specific information in support of some of the data reporting requirements while others did not. Due to the timing of the evaluation, the review of program performance data necessarily focused on the first year annual reports which largely reported on information related to the set-up and implementation of the networks. Additionally, the second year annual reports were available late in the evaluation process and it was therefore not feasible to undertake a detailed analysis of these data or incorporate it in the case studies of networks. Consequently, data from the second year annual reports were only incorporated to a limited extent in this report. As a result, the program performance data included in this report may not fully describe the impact of network activities. This limitation was mitigated to some extent by the interviews and surveys with network management, researchers and partners which also collected data on network activities and achievements.

For the survey of BL-NCE partners and researchers, while the response rates were acceptable, the actual number of respondents was small for a quantitative survey. In addition, as a result of the distribution of the population of researchers and partners participating in the program, more than half the BL-NCE partners responding to the survey were from one network whereas more than half the BL-NCE researchers responding were from another network. It is important to note that the response rates to the partner and researcher surveys by network were largely proportional to the distribution of the population of partners and researchers participating in the program by network. The results presented in this report therefore often combined the partners and researchers, in order to provide sufficient sample sizes to statistically conclude on some of the survey data. Care was taken in how survey results were used in drawing conclusions and other mitigating measures were used to ensure that program results were not overly influenced by a single network.

Findings and Conclusions

Relevance

The original rationale for the BL-NCE program remains current. The program is aligned with the federal government priorities set out in Advantage Canada, The Next Phase of Canada’s Economic Action Plan (Budget 2011) and the S&T Strategy. The program is also aligned with departmental strategic outcomes as laid out in the PAA of the three funding agencies. Each of the funded networks is similarly aligned to the priorities set out in the BL-NCE program’s Terms and Conditions: Environmental Science and Technologies, Natural Resources and Energy, and Health and Related Life sciences and Technologies.  The other two priorities identified (Information and communications technologies and Management, business or finance) are not addressed by currently funded networks; however, no fundable networks were identified in these priority areas. The BL-NCE expands the scope of R&D in the industries involved in the funded networks. Two of the four networks would not exist without the BL-NCE program. The other two networks would be limited regionally and in scope without federal support.

There is an ongoing need for a program of the nature of the BL-NCE program. This program helps fill a gap in the innovation spectrum between ‘proof of concept’ and ‘product development’. The BL-NCE program uses a business-led network approach to bring together teams of private and public sector researchers to conduct the collaborative R&D required to address the identified needs of industry. The novelty of the business-led model is that the teams of researchers funded by each network can be university-based, private-sector based, based in a not-for-profit organization, or a combination of the three. The common feature across networks, and the niche of the program, is that the research itself is intended to address industry-specific or business-specific needs by involving the private sector more closely in the design and conduct of the research, thereby better ensuring the take-up and use of the results. The program also helps fill a gap by providing the funding required to undertake this type of research (i.e., applied research to address business-specific needs that is led by the private sector) that would otherwise not be available or that would be insufficient to fully address the identified research needs.

Implementation

Although the program’s experience is limited to only four funded networks, the program design, in particular its business-led approach, is a facilitating factor in ensuring research undertaken addresses the needs of industry in these sectors. However, some of the expected outcomes for the BL-NCE program may have been too ambitious given the four-year timeframe for the program (e.g., address significant research challenges, accelerate commercialization) and the complexities in establishing business-led networks may have been underestimated. The unique characteristics of each network (i.e., administrative capacity, experience of collaborative research, expectations of partners and industry needs) have resulted in a certain degree of flexibility in BL-NCE program implementation. The program’s implementation experienced some difficulties and delays as networks struggled to establish Network Agreements and resolve issues related to intellectual property (IP). As a consequence, research projects did not get underway as quickly as originally proposed in network applications to the program.

The networks have implemented effective models and management practices to achieve outcomes. However, each network has learned key lessons along the way. For example, it is critical to take the necessary time to ensure the right people are involved in the network and supported by a solid governance structure and decision-making processes. The majority representation of industry partners on network Boards and project selection committees help ensure the funded research reflects business needs. Building trust and relationships amongst industry, academia and government partners are key ingredients for long-term success. It is also important to ensure network management has the administrative capacity (i.e., resources and access to specific skill sets) to manage the complexities of the network. A wide range of skill sets on the network boards of directors that include a blend of industry sector, scientific, financial and legal expertise was also important to network implementation and their ongoing performance. Lastly, it is important to identify realistic performance expectations and measures of success that reflect the uniqueness of each network and sectors within which they operate as well as the expected outcomes of the program.

With many of the challenges involved in setting up network governance structures and management practices now behind the program, it is anticipated that the realization of both network and program outcomes should progress more quickly as more research projects are conducted in the remaining years.

Network approach to research, development and innovation

The BL-NCE program has enhanced research, development and innovation in the areas of the four funded networks. The business-led model has encouraged the development of industry-university research partnerships (as evidenced by the 89 projects, involving 378 researchers).

In addition to industry-university partnerships, the business-led model facilitated the development of partnerships between industry sectors, in some cases bringing together sectors that have not traditionally worked together. The Network Agreements and IP arrangements, while requiring a significant up-front investment in time and effort by network management and partners, now facilitate the development of multisectoral, multidisciplinary R&D teams or projects. International collaborations have been established where appropriate.

The level of industry involvement in the development of research priorities, project selection, scientific committees, Board of Directors, and guiding and carrying out research projects ensures that projects are directly relevant to industry’s needs. Network partners are able to fully participate in the decision-making and setting research goals, and are able to influence research planning and agendas.

The networks have developed project portfolios that address the needs of network members. Networks are strengthening links between the research community and industry, and appear to be on track to meet the needs of partners. There is some early evidence of increased visibility of Canadian researchers involved in these networks.

Mobilization and benefits of network research

The business-led network approach (including the development and implementation of a strategic plan, project selection and oversight) is seen as an effective mechanism to promote mobilization of research by industry. All networks have been successful in terms of establishing and building partnerships, helping partners learn to work together and share IP, and building a knowledge base.

Approaches to conducting research vary among the networks with differing amounts of research carried out by universities, businesses and the not-for-profit research organizations. Consequently the strategies for mobilizing research results vary; however, the major mechanisms for mobilization of research identified are networking, IP and non-disclosure agreements, and refereed publications. All networks make use of conferences, workshops and meetings to share research results among network partners, funders and the broader community.

The extent to which network research will have been mobilized by partners and translated into technical applications, products and processes by the end of the four year funding will vary, depending on the sector and the type of research. The commitment of partners and the extent to which a pathway to early commercial applications has also been identified are major factors in the achievement of intended outcomes. While it is generally recognized that it is too early to expect significant achievement of long-term outcomes, in one network, there is early application of research in the development of next generation products.

Training of HQP

While the emphasis varies among networks, all BL-NCE networks contribute to the training of HQP through university-based research. HQP participating in BL-NCE funded projects acquire more technical and professional skills relevant to business than those in the other comparable networks surveyed. They also gain experience relevant to the needs of the industrial stakeholders participating in the network that improves their opportunities for employment after graduation. A total of 83 university-based HQP have participated in research projects funded by the four BL-NCE networks during the first year. This number is expected to increase, as the participation of HQP in some networks to date has been affected by delays in getting university-based research underway. In addition to training of HQP at universities, network research also provides training for the private sector researchers participating in projects through their involvement in the research projects, and through interaction with university researchers and other HQP. In addition, two networks are using a mentorship approach whereby an industry representative provides guidance to network research projects. In the case of one network, the mentorship approach has enabled industry representatives to be directly involved in all phases of projects, helped the researchers stay aligned with the industrial needs, and when the time comes act as a champion to mobilization the research results.

Performance (efficiency and economy)

Efficient and effective means are being used to deliver the BL-NCE program. The evidence shows that the program has been efficient in managing its operational resources in comparison to its grant funds, particularly in comparison to other programs with larger grant funds. The individual networks have also been effective in balancing their administrative expenditures in comparison to research funds; however, some networks have higher administration burdens at this stage given delays in becoming fully operational and getting their research projects approved.

The program has also been effective in exceeding its matching funds requirements based on actual expenditures as well as committed funds. In fact, based on actual expenditures, the projections for partner contributions to expenditures have been exceeded (more than doubled) when all networks are combined. However, funds are not being used at the rate anticipated given delays in network implementation.

Based on committed funds, the combined funded networks have also been effective in exceeding their matching funds requirements. To date, a significant proportion of the non-BL-NCE funds (83%) originate from the private sector (46%) and other public sector organizations (federal and provincial) (37%).

There are few opportunities for improving the efficiency of the program. However, the short timeframe for the program has been a concern of several networks in terms of their ability to maximize their effectiveness (i.e., results), efficiency (i.e., minimized administrative expenses) and economy (i.e., maximized leveraging). Networks were particularly concerned with the lack of relevance of current indicators or measures to their networks. For example, publications were noted as less relevant to business-led networks. More relevant indicators such as improvement to technology readiness were deemed important. It was therefore noted that one key area of improvement is to ensure that the reporting requirements are aligned with business-led networks and are thus less academic in nature.

Recommendations

Recommendation 1: The BL-NCE program is showing early success and the model should therefore be maintained at the federal level.  The BL-NCE program is addressing a continued need for private sector led collaborative research and development and making progress towards the achievement of expected outcomes. It is still too early to firmly conclude that the program will achieve its objectives to increase private sector investments in research in Canada, support the training of skilled researchers, and connect the resulting ideas and talent to businesses seeking to bring innovations to market, particularly given the early stage of the program as well as the limited number of funded networks. However, the findings of the evaluation support the validity and further funding of the program model. The findings also support the involvement of the federal government in funding of the program model as such funding enhances the scope and nature of the funded networks.

Recommendation 2: If renewed or extended, the NCE Secretariat should consider the following to enhance the program’s ongoing relevance and effectiveness. First, allow existing networks to re-apply in future program competitions as there will likely still be an ongoing need for federal government support to these networks to achieve program outcomes. Second, focus on steps to solicit applications for networks in priority areas not funded to date to improve the alignment of the program with priority areas and private-sector needs (i.e., in the two priority areas not yet funded). Third, provide more support for the development of network applications and the implementation of funded networks to help mitigate and/or lessen the challenges that have adversely affected network implementation and operation to date. In terms of support for network implementation, this could include identifying the types of expertise and resources required to implement a business-led network as well as providing additional assistance with the development of network agreements. With respect to the application process, stronger emphasis could be placed on assessing the required expertise and resources in subsequent program competitions by revising the program’s assessment criteria and application requirements.

Recommendation 3: The BL-NCE program’s expected outcomes and performance measurement strategy should be revisited. While the program theory appears appropriate, based on the nature and performance of the four networks funded to date, there is a need to revisit the program logic model, performance measurement strategy and extent to which and the timeframe in which some expected outcomes can be achieved. The evaluation found that there is a need to establish a better linkage between the network level outcomes to program outcomes. Therefore, more work is needed to better demonstrate how the outcomes of individual networks are contributing to program outcomes. This should involve further refining the expected outcomes in the program logic model. This should also involve revisions to the performance measurement strategy as well as assessment of the quality and comprehensiveness of the data collected to improve the relevance, appropriateness and reliability of performance indicators used to measure both network and program performance. Revisions to the performance measurement strategy could be informed by a review of the performance data already collected as well as continued work with the four BL networks.

1.0 Introduction

1.1 Evaluation Context

This report presents the findings from the first evaluation of the Business-Led Networks of Centres of Excellence (BL-NCE) program. The evaluation is based on the program’s Results-based Management and Accountability Framework and Risk Based Audit Framework (RMAF-RBAF) which indicates that a review relevance and effectiveness is to be conducted to inform the potential renewal of funding and continuance of program authorities.¹ The terms and conditions of the BL-NCE program apply for a five-year period ending in June 2012. The current evaluation is intended to meet this requirement to inform the potential renewal process for the program as well as meet the coverage requirements of Treasury Board’s Policy on Evaluation for the program.

As outlined in Section 2.0 of the report, the evaluation addresses issues related to program relevance, implementation and effectiveness (i.e., the extent to which the program is achieving its expected outcomes). The evaluation collected data from the BL-NCE networks, recently funded NCE networks and, where possible, recently funded networks by comparable agency programs as a means of comparison as well as for baseline data for the summative evaluation. The evaluation of the BL-NCE program covers the time period from program inception (fiscal year 2007-2008) to the end of fiscal year 2010-2011.

1.2 Program Overview

Science and Technology (S&T) plays a key role in helping Canadians to address pressing societal challenges. S&T also supports business innovation, enabling economies to improve their long-term productivity and competitiveness and, in so doing, supporting a higher standard of living and quality of life. However, Canadian private sector investment in S&T and new technology, and demand for highly skilled workers is low compared to other Organisation for Economic Co-operation and Development (OECD) countries. This is contributing to weak productivity growth in relation to the United States (US), Canada’s most important trading partner.

Mobilizing Science and Technology to Canada’s Advantage, the Government of Canada’s S&T Strategy, sets out a multi-year policy framework to improve Canada’s long-term competitiveness and quality of life by fostering three inter-related S&T-based advantages. The Strategy encourages an Entrepreneurial Advantage to strengthen private-sector commitment to Research and Development (R&D) and innovation vital to productivity and competitiveness, a Knowledge Advantage to ensure Canadian universities and colleges sustain their world-class research excellence, and a People Advantage so that Canada has access to the highly-skilled researchers and innovators it needs.

Budget 2007 announced a broad range of early actions in support of the S&T Strategy, including three new NCE Secretariat programs to leverage Canada’s strong public sector research base to the benefit of business research and innovation: the BL-NCE program, the Centres of Excellence in Commercialization and Research (CECR) program, and the Industrial R&D Internship (IRDI) program. All three programs are intended to increase private sector investments in research in Canada, support the training of skilled researchers, and connect the resulting ideas and talent to businesses seeking to bring innovations to market. As indicated in Budget 2007, the NCE Secretariat established a Private Sector Advisor Board (PSAB) to ensure that the CECR and BL-NCE programs meet the needs of businesses.

The Government of Canada invested $46 million over four years for the creation of the BL-NCE program; this includes $39,310,473 for grants, $2,837,652 to increase direct involvement of small and medium-sized enterprise (SME) community in network activities (of which $1,417,902 was award in 2010-2011) and $3,851,875 for program administration. The BL-NCE program’s goal is to fund large-scale collaborative networks to support private sector innovation in order to deliver potential economic, social and/or environmental benefits to Canadians and to promote an Entrepreneurial Advantage. As such, the program supports innovation in five priority areas:

• Environmental science and technologies;
• Natural resources and energy;
• Health and related life sciences and technologies;
• Information and communications technologies; and
• Management, business or finance. ²

The overall logic model for the BL-NCE Program is presented in Figure 1-1 as developed for the program’s results-based management and accountability framework (RMAF). The logic model is critical for validating the program theory and, as such the evaluation questions pertaining to the program’s performance (effectiveness) are based on the immediate, intermediate and ultimate outcomes outlined in the logic model (Figure 1-1).

Figure 1-1: BL-NCE Logic Model


Click to view larger image | View long description

1.3 Overview of Funded Networks

There are four networks currently funded by the BL-NCE program for the period from 2008-09 to 2012-13. ³ They are:

• Canadian Forest NanoProducts Network (ArboraNano);
• Green Aviation Research and Development Network (GARDN);
• Québec Consortium for Drug Discovery (CQDM); and
• Petroleum Technology Research Centre – Sustainable Technologies for Energy Production Systems (PTRC-STEPS).

Funding amounts for these networks range between $8.9 million and $12.4 million. Table 1-1 provides a brief overview of the four networks. 

1.4 Report Structure

Section 2.0 of this reports provides an overview of the methodology used to complete this evaluation, how the different lines of evidence address the evaluation issues and questions, and discusses the study limitations. Sections 3.0 to 8.0 present the key evaluation findings and conclusions. Section 9.0 summarizes the conclusions and discusses the ensuing recommendations.

2.0 Methodology

2.1 Evaluation Issues

As a recently implemented program, the BL-NCE program had not been evaluated to date. This evaluation focused on questions related to program’s relevance and performance (effectiveness, economy and efficiency) in order to inform the program renewal process and meet the requirements of Treasury Board’s Policy on Evaluation. In addition, as the BL-NCE program is a new approach for the NCE Secretariat, some questions pertaining to implementation were also examined.

Table 2-1 outlines the evaluation issues and questions. The issues and questions presented in the table were developed in consultation with, and were approved by, the NCE Management and Steering Committees.

2.2 Evaluation Methodology

The evaluation methodology is summarized in Table 2-2. Details follow.

Document review

The document review involved a review of document on the BL-NCE program as a whole, government-wide documents, individual BL-NCE network documents, and literature in general. Documents were reviewed and assessed for their contributions to specific evaluation issues and questions. Key findings from the document review have been incorporated as appropriate throughout this report.  A list of the documents reviewed is included as Annex B.

Network-specific findings from the document review were integrated into individual network case study write-ups.

Data analysis

Data analysis involved analysis of financial and other data on the BL-NCE program as a whole, on individual BL-NCE networks and on comparable networks.  The data was analyzed to help address the program’s relevance, effectiveness, efficiency and economy. Key findings from the document review have been incorporated as appropriate throughout this report.

It is important to note that, during the data analysis tasks, quality issues were identified in the data.  For example, one of the networks (GARDN) had originally identified a large number of researchers, many of which were not actually directly involved in the network.  To the extent feasible, these inconsistencies were corrected to ensure that the data presented throughout this report is as accurate and up-to-date as possible.  Nevertheless, within the scope of this study, it was not possible to fully validate all data.  For example, due to delays in the evaluation reporting phase, second year annual reports were made available in the late stages of this study.  As this was not within the scope of the evaluation, this data could not be validated or corrected with the individual networks as was done with the data from the first year annual reports.

Interviews

A total of 64 individuals were interviewed in person or by telephone. These included program and network stakeholders. The distribution of interviews is outlined in Table 2-2.

To the extent feasible, interviews were conducted in person; however, some were completed by telephone.  Interviews were scheduled at a time that was most convenient to the individual, in the official language of choice.  As interviews were scheduled, individuals were sent the interview guide to help them prepare.  Depending on the type of interview, the interviews took between a half hour and two hours.

Given the small number of interviews conducted, the interviews were distributed as outlined in Table 2-2 for the following reasons:

• Interviews with representatives of the NCE Management Committee:  It was deemed important to consult with representatives of NSERC, CIHR, SSHRC and Industry Canada (IC).  Some of these interviews involved more than one individual to ensure all questions could be fully answered.
• Interview with management and staff of the NCE Secretariat: There are several program and network individuals who could provide valuable input for this study.  However, since a small number of interviews was budgeted, program management and staff were interviewed individually or as a group, given that the information sought was not deemed sensitive.
• Two interviews with members of the PSAB:  Since its creation in 2007 there have been 13 individuals who have been members of PSAB.  Given the small total number of interviews, it was deemed that two would provide sufficient information from PSAB on the relevant issues particularly since the individuals selected have been active members since the Committee’s formation with strong knowledge of the program.
• Interviews with members of the expert panel: There are 36 individuals who have participated as members of the expert panel.  Expert panel chairs were selected for interviews, including those for unsuccessful networks.
• Interviews with unsuccessful applicants: Four organizations completed a full application for BL-NCE program funding but were unsuccessful.  An attempt was made to interview all four but only three were available.
• Interviews with members of various committees and network members⁸:  Members of the various committees and network members were expected to be knowledgeable on the relevance of the program and network from the perspective of the organizations they represent.  Committee members were also expected to be knowledgeable on various aspects of the program’s and network’s implementation and operation having been directly involved in the network’s implementation.  Finally, their input into the program’s and network’s performance was two-fold:  as individuals involved in the performance of the network as a whole as well as from their individual organization’s perspective.  Additionally, the nature of the information obtained from committee members and network members was more qualitative in nature and therefore more suited to an in-depth interview approach.  Weight was therefore given to the number of interviews with committee members and network members.
• Researchers:  Researchers were expected to provide input into the relevance, implementation and performance of the network from the perspective of the research projects they have undertaken.  For this purpose, researchers were suitable candidates for the surveys, as the information sought from them was quantifiable.  Nevertheless, it was expected that there could be some benefit to obtaining qualitative information which may help explain some of the quantitative information gathered through the surveys.  For this purpose, a small number of in-depth interviews with researchers were completed for each network.
• HQP: HQP were expected to provide input into the relevance, implementation and performance of the network from the perspective of the research projects in which they have participated.  Considerations were given for undertaking surveys of HQP participating in the networks.  However, as the BL-NCE networks are still in their infancy, there is a very limited number of HQP that have participated on research projects funded by the BL-NCE networks.  As such, the number of HQP participating in the BL-NCE networks was insufficient to undertake a survey. It was therefore decided to replace the survey of HQP with interviews. While researchers were able to provide some information on the impact of the network on HQP, it was deemed important to supplement this with information obtained directly from two HQP from each of the four networks for a total of eight interviews.

Web-based Surveys

Four web-based surveys, each using a census approach, were administered as part of the evaluation to four groups: BL-NCE network partners; BL-NCE network researchers; partners of comparable NSERC, NCE and CIHR networks; and researchers of comparable NSERC, NCE and CIHR networks. For the purpose of the surveys:

• partners were defined as representatives of organizations who were affiliated with the network as funding partners and/or members of one of the network’s committees and/or member of the network;
• researchers were defined as individuals involved in projects funded by the networks either as the lead researcher or as a member of the project research team; and
• comparative networks were defined as research networks in similar broad research domains, with comparable funding levels, that had been in existence for a similar length of time as the BL-NCE networks (i.e., less than five years).

The list of partners and researchers of BL-NCE networks was compiled from the progress reports submitted by networks to the NCE Secretariat.  It was validated and updated by the individual networks.  The partner population identified for the survey was small for two reasons: first, most networks had a small number of partners involved in their network; and, second, some of the partners were removed from the survey population because they had either been interviewed or eliminated during the interview scheduling process (i.e., refusals, not sufficiently involved in/ aware of the networks to respond to participate in an interview).

The lists of partners and researchers of comparable networks were provided by NSERC, the NCE Secretariat and CIHR based on information available in program databases.

For all surveys, individuals were sent an original email invitation and a follow-up reminder.  Table 2-3 outlines the final sample disposition for each survey.

Case studies

The findings from the network document review, data analysis, interviews and surveys were integrated into individual network case studies which profiled each network and presented network-specific findings for each evaluation issue and question.

2.3 Limitations

Overall, the evaluation methodology is strong in providing the basis for reaching conclusions for all issues and questions using multiple lines of evidence. There are limitations with the evaluation methodology; however, they were carefully taken into account when conducting the analyses, and are acknowledged in the interpretation of the findings. The limitations and mitigation measures taken are described below.

Data quality / Timing of Data:

Some issues were encountered with the consistency and availability of performance measurement data collected by the program.  For example, some networks gathered specific information in support of some of the data reporting requirements (e.g., in-kind and cash contributions by source for each project) while others did not; in cases where the network did not have the detailed information, it was not always possible to obtain this information within the scope and resources available for this study. Additionally, it is primarily only the first year annual reports which were used as the basis for data analysis because the second year annual reports were available late in the evaluation process. While extensive efforts were expended trying to validate this data (as had been done for the first year data), some issues could not be addressed given the timeframe of the evaluation.   Consequently, it was not feasible to undertake a detailed analysis of these data or incorporate it in the case studies of networks and as a result data from the second year annual reports were only incorporated to a limited extent in this report. As a result, the program performance data included in this report may not fully describe the impact of network activities.

Sample sizes for surveys:

For the survey of BL-NCE partners and researchers, while the response rates were acceptable, the actual number of respondents was small for a quantitative survey.  In addition, as a result of the distribution of the population of researchers and partners participating in the program, more than half the BL-NCE partners responding to the survey were from one network whereas more than half the BL-NCE researchers responding were from another network. 

The small sample size for the BL-NCE surveys was a concern for the following reasons: 1) multivariate analysis within the BL-NCE surveys was not often possible; and 2) in-depth analysis to the comparison surveys was also not often feasible.

To mitigate the impact of this, the results presented in this report often combined the partners and researchers, in order to provide sufficient sample sizes to statistically conclude on some of the survey data.  Care was taken in how survey results were used in drawing conclusions and other mitigating measures were used to ensure that program results were not overly influenced by a single network.  These other measures included the following. 1) For the BL-NCE partner survey, statistical comparisons were conducted of the network with the disproportionately high number of partners versus the other three BL-NCE networks; where significant differences were uncovered, these survey results were not used for comparison purposes.  A similar approach and analysis was undertaken for the BL-NCE researcher survey, in which results from the network with the disproportionately high number of researchers were statistically compared to those of the other three networks.  2) When comparing the BL-NCE survey results to those of the comparison survey, only statistically significant differences were highlighted.  Therefore, in some cases, where there were large differences in the observed percentages, they were not reported because they were not statistically significant.  3) In several cases, survey results are presented qualitatively without highlighting specific percentages.

3.0 Relevance

Summary:

As of 2008, “Canadian private sector investment in S&T and new technology, and demand for highly skilled workers is low compared to other OECD countries.”¹⁰ The BL-NCE program is designed to strengthen Canada’s productivity through support for private sector R&D networks focused innovation and. Program documents and interview and survey findings showed support for a business-led network approach to funding of research, development and innovation. While a comparator survey was undertaken and indirect comparators were identified by interviewees and survey respondents, no direct comparator (i.e., business led comparator to the BL-NCE program or individual networks) was identified in the interviews, case studies or document review. Interviewees noted that the federal role in research and innovation is to create an environment across the country that stimulates industrial capacity and to create an entrepreneurial culture within researchers.  Across the four networks, the BL-NCE program has invested in a number of industrial sectors that are significant contributors to the Canadian economy.  Companies in these strategic sectors compete internationally, often against foreign companies that have benefited from their own government’s R&D support programs.

The BL-NCE is well aligned with the 2007 S&T Strategy, Mobilizing Science and Technology to Canada's Advantage. Its five objectives are similarly aligned with the strategic outcomes in the Program Activity Architecture (PAA) of all three granting agencies. The BL-NCE program received meritorious applications that led to funded networks that align with three of the five federal priorities identified in the Terms and Conditions for the BL-NCE program. The original rationale for the program remains current and there is a continued need for the BL-NCE program. The BL-NCE program expands the scope of R&D in the industries involved in the funded networks.

3.1 Continued need for a network approach to funding of research, development and innovation

The BL network approach was identified by interviewees as a useful model to promote research, innovation and training in the private, public and academic sectors.  Program documentation also supports the ongoing need for a network approach to investment in industrially relevant science and technology, particularly in the four funded areas.¹¹ Case studies of the funded networks showed support for a network approach. Collectively, a number of advantages were identified, including:

• Critical mass: Interviewees highlighted the ability of networks to develop a critical mass of people and resources able to produce together what they could not produce separately. Program documentation highlighted the need for a critical mass of research “in a vast country where university and research institutions are dispersed”.¹²
• Shared risks: Program level interviewees noted that the BL networks and the involvement of the federal government allows for shared (distributed) risk across multiple private and public sector organizations. This distribution of risk lessens the net risk borne by any single agency or organization. The shared investment reduces risk to individual firms and allows the networks to fund projects that may have a higher overall risk, but with a commensurate increased opportunity for benefits for the sectors and organizations involved.
• Credibility: The network approach was identified by a small number of interviewees as adding to the credibility of network researchers and the innovation activities of the participating organizations. This enhanced credibility facilitated work with other federally funded networks and researchers.
• Encouragement of industrial collaboration: Interviewees noted that networks contributed to change in business cultures and encouraging organizations to work collaboratively. (Managing the challenges of this cultural shift is discussed in Section 4.1.1.)

Disadvantages of the network approach are discussed in Section 4.1.1 as factors inhibiting the success of the networks and inform the discussion of lessons learned (in the same section).

Two of the four networks were already in place before they were approved for BL-NCE program funding.  Representatives of these two networks indicated that without the BL-NCE program their networks would have been negatively affected.  The other two networks indicated that they likely would not exist.  All networks indicated that they would have been negatively affected in the following ways:

• Geographic scope: the networks would have maintained a regional scope and not expanded to include partners and researchers from other parts of the country;
• Research scope: existing networks would have undertaken more narrow research programs, newly created networks would not have undertaken any research programs;
• Partner commitments: three of the networks indicated that, without federal involvement, some of their partners would not have participated in their network while others would have been less committed; and,
• Research collaborations: the extent of industry-academic-other research collaborations would have been diminished.

These would in turn have negatively affected the overall performance of the networks and diminished their ability to eventually become sustainable. However, representatives from all networks noted that there is a continued need for funding the four networks at this stage as the networks are not yet strong enough to be sustainable without renewed funding, risking a shrinkage or wind down of the funded networks.  Representatives from all networks noted that the networks have not been operational for a sufficient period of time for the original program needs to have been fully been met.

3.1.1 Niche the program occupies in relation to other research network funding programs

BL-NCE documents showed that the niche for the BL-NCE program is to fill the innovation gap between academic and business research that occurs between ‘proof of concept’ and ‘product development’. Interviews, survey data and documents indicate that network projects are indeed ‘mid-stage’ occupying the centre of the R&D spectrum (see Figure 3-1).¹³ By design, the BL-NCE program’s goal is “to fund large-scale collaborative networks to support private sector innovation in order to deliver potential economic, social and/or environmental benefits to Canadians and to promote an Entrepreneurial Advantage” including greater benefit for private sector competitiveness and greater commercial outcomes.¹⁴ The BL networks therefore occupy a unique niche in that the projects are initiated in the ‘mid-stage’ because businesses or the industry have identified these as priorities for their business or industry.

Figure 3-1: Research and Development Spectrum and the BL-NCE Program

The BL-NCE program occupies a unique niche in relation to other NCE funding programs in that the BL networks:

• are private sector led rather than academia;
• are hosted by a not-for-profit consortia of private sector organizations rather than being hosted by universities or traditional not-for-profit organizations;
• are led by the consortia director rather than having a university-based lead;
• have a research agenda set by private sector needs instead of by universities and their stakeholders;
• have shorter funding periods than traditional NCEs (four years not five);
• support costs for research projects with up to 50% BL-NCE funds and matched by private sector in-kind or cash contributions, whereas in the NCE program, the majority of network research costs are paid by program funds and in the case of the CECR program, centre operation and commercialization costs, but not direct research costs, are paid by program funds;
• target research on the five science and technology priority areas identified in Budget 2007 rather than being open to all; and
• have research carried out in university, private sector and / or government labs whereas research for traditional NCE projects is conducted in university labs.¹⁵

Interviewees noted that, as a business-led initiative, the BL-NCE program funds less basic research and more applied research than research completed exclusively in academe. They went on to add that business-led research is of direct relevance to the needs of the industry sectors and firms involved, specifically addressing the gap between academic and business research.  Consequently, most interviewees believed that the BL-NCE program occupies a unique niche in comparison to other research network funding programs, particularly as most could not identify comparable programs. While some comparators were mentioned¹⁶, interviewees also noted that the BL-NCE was different from these other programs.  Some interviewees made the point that, in Canada, government funding focuses on partnerships that include research laboratories and businesses and, as such, most programs were in some way similar. The survey of BL-NCE partners and researchers supported interview findings. Only one in five partners and researchers surveyed indicated that they were aware of similar research networks in Canada and mentioned:

• other BL-NCEs (ArboraNano, GARDN);
• NCEs (AUTO 21, CMC);
• CECR (Montreal Neurological Institute Centre of Excellence in Commercialization and Research (MNI CECR));
• other federal (Sustainable Development Technology Canada);
• provincially funded centres (Centre for Research and Innovation in the Bio-Economy (CRIBE), In Vivo, NanoQuebec);
• industry associations (Consortium for Research and Innovation in Aerospace in Quebec (CRIAQ), Ontario BioAuto Council); and
• others (SAGE¹⁷, General Electric).

3.2 Role for the federal government in supporting the program

The Review of Federal Support to Research and Development notes that one of the key roles of the federal government in fostering innovation is providing appropriate support for business and commercially oriented R&D, whether it be through indirect tax measures, direct assistance to businesses, or funding for public sector or non-profit bodies conducting research of relevance to the private sector.¹⁸ This role is directly aligned with that of the BL-NCE program.

Program level interviewees noted that the federal role in research and innovation is to support an environment across the country that stimulates industrial capacity and to promote an entrepreneurial culture among researchers.  One interviewee explained that, when working internationally, marketing a new technology as ‘Canadian’ is more successful than regional or business-specific marketing. Two interviewees believed that support for research and development in Canada is noticeably lower than in other countries. Others remarked that, if innovation is important to federal policy (as stated in the federal S&T strategy), then programs such as the BL-NCE must be supported.

All network interviewees and most program level interviewees agreed that there is a necessary role for the federal government support of a network approach to research and development. Each of the case studies of the four funded networks identified the ongoing and current need to support BL research and development in each industrial sector involved in the networks.

Through the four funded networks, federal involvement has:

• addressed innovation needs in industries that are significant contributors to the economy;
• allowed for the expansion of networks beyond a regional focus and/or beyond a single sector;
• supported strategic industries; and
• supported industries that compete internationally, often against foreign companies that benefit from their own government’s R&D support programs.

The federal government’s involvement adds credibility to the networks themselves as well as to the importance of the sectors involved (e.g., one industry representative reported that the continued role of the federal government will be important factor to consider given the organization’s competing priorities for R&D funding).

3.3 Alignment with federal government priorities

In Advantage Canada, the federal government identified a productivity and innovation challenge for Canada and reported that the country’s lagging R&D intensity was concentrated in the business sector.¹⁹ The S&T and Innovation Council supported this assessment, noting that “compared to other OECD countries, business R&D in Canada is a comparatively smaller portion of total R&D performed by all sources (i.e., Gross Expenditure on R&D (GERD)).” The Council went on to note that in 2006 “Canada’s business sector performed 55 percent of all R&D, compared to: 77 percent in Japan; 70 percent in the US and Germany; 63 percent in France; and 62 percent in the United Kingdom.”²⁰ In 2009, the Council of Canadian Academies (CCA) reported “Canada’s serious productivity growth problem [was] a business innovation problem.”²¹

As noted in the 2007 S&T Strategy, Mobilizing Science and Technology to Canada's Advantage, “scientific and technological innovations enable modern economies to improve competitiveness and productivity, giving us the means to achieve an even higher standard of living and better quality of life”. The BL-NCE program objectives are of direct relevance to the priorities set out in the S&T Strategy, namely:

• encouraging an Entrepreneurial Advantage to strengthen private-sector commitment to R&D and innovation vital to productivity and competitiveness;
• fostering a Knowledge Advantage to ensure Canadian universities and colleges sustain their world-class research excellence; and
• encouraging a People Advantage so that Canada has access to the highly-skilled researchers and innovators it needs.

Budget 2011 (The Next Phase of Canada’s Economic Action Plan) renews the Government of Canada’s commitment to invest in innovation, education and training. The Plan notes that Budget 2011 makes important progress on, amongst other priorities, improving commercialization and supporting demonstration of new technologies in the marketplace by supporting research links between colleges, universities and business.  This is of direct relevance to the objectives of the BL-NCE program.

Collectively, the funded networks align with three of the strategic outcomes included in the PAA of all three granting councils (NSERC, SSHRC, CIHR), and with federal priorities in the Terms and Conditions for the BL-NCE program namely: environmental science and technologies, natural resources and energy, health and related life sciences and technology, and management, business and finance.²²

Additionally, each agency has a strategic outcome related to the training and support of researchers, and another related to the transfer of knowledge generated through research activities”²³; these are directly linked to the intended outcomes of the BL-NCE program.

The BL-NCE program is aligned with the CCA’s recommended solutions to Canada’s innovation and productivity challenges to:

• sharpen the incentive for innovation-oriented business strategies;
• improve the climate for new ventures so as to better translate opportunities arising from Canada’s university research excellence into viable Canadian-businesses; and
• support areas of particular Canadian strength and opportunity through focused, sector-oriented strategies.”²⁴

Those interviewees who commented agreed that the BL-NCE Program is aligned with the federal priorities, as identified in the S&T Strategy.

4.0 Implementation

Summary:

Although the achievement of program outcomes is limited to date, as most network research projects are still in the early stages, the business-led aspect of the program is seen as a facilitating factor by interviewees and survey respondents as it helps to ensure research undertaken addresses the needs of industry.  A rigorous review process, which involved industry and scientific experts, was used for the selection of the four funded networks.  Overall, interviewees were positive with respect to the BL-NCE program design and network selection process, even those applicants who were not approved for funding.  However, some interviewees felt that expectations for the BL-NCE program may have been set too high given its novelty and four-year timeframe. 

Results from the network case studies provide insights on key factors that are contributing to each network’s success including: leadership of network management; previous experience with industry-university R&D partnerships; willingness of industry partners to make the cultural shift from competition to collaboration; and the significant role played by industry within the networks’ governance structures and decision making processes.  The main obstacles inhibiting implementation and the achievement of outcomes deal with managing cultural change and the learning processes associated with setting up a new program, which resulted in considerable delays for some networks.  Some networks have struggled more than others in establishing themselves, taking more time than anticipated to bring on partners and negotiate acceptable Network and Intellectual Property (IP) Agreements with partners. 

Some best practices have emerged at the network level concerning project selection processes, policies on IP, and having industry experts as mentors on research projects.  Based on observations at the network level, a number of lessons learned have been identified with respect to:  establishing more realistic timeframes for setting up BL networks, ensuring administrative capacity of network management, composition of BODs and the need to develop more appropriate measures of performance for a BL network model.

4.1 Impact of program design on achievement of program outcomes

Overall, interviewees were positive with respect to the program design and the process used to select networks.  The Program Guide provided detailed requirements for governance of the networks (board of directors, funding agreement, network agreement, compliance requirements) and network management (administration, matching fund requirements, stacking provisions, etc.).  Proposals for funded networks were assessed against three criteria:  benefit to Canada, track record and potential of the applicants, and strength of the strategic plan.  The PSAB reviewed Letters of Intent and recommended a short list of BL-NCE applicants to the NCE Steering Committee for advancement to the full application stage.  The NCE Steering Committee appointed Expert Panels to provide detailed evaluations of the applications.  Recommendations were then transmitted to the PSAB, which in turn gave its funding recommendations to the NCE Steering Committee. 
Program level interviewees explained that the program design was based in part on the experience and lessons learned from the Precarn collaborative R&D model²⁵.  Interviewees representing PSAB and Expert Panels indicated that program management was very careful in selecting quality people who together possessed the appropriate breadth of knowledge to fulfill their respective mandates with respect to the selection of networks.  The peer review process was an important element of due diligence for the program, adding credibility to those that were selected.  One expert panelist commented specifically that the question and answer component of the process was very valuable to both applicants and reviewers.  A few suggestions for improvement in the application and selection processes were to:

• consider a more targeted approach to raising awareness about the program to potential applicants;
• provide more guidance to applicants at the call for proposal stage and allow sufficient time for applicants to develop proposals; and
• encourage a stronger linkage with SMEs through the BL-NCE program.

Lessons learned with respect to Program Design

• Balance network readiness with expectations – Some interviewees felt that expectations for the BL-NCE program may have been set too high given the four-year timeframe.  Some networks experienced delays of more than a year as a result of the level of effort required to negotiate acceptable Network and IP Agreements with both university and private sector partners (i.e., to modify the original agreement templates provided by the BL-NCE program).  At this stage, the BL-NCE and network stakeholders, managers and participants are gaining experience in dealing with the challenges and complexities associated with establishing the necessary relationships and trust between industry partners and academia. One lesson learned was that in the future the BL-NCE program should either expect potential applicants to come into the program at a more advanced stage, lower the expectations on results, or lengthen the timeframe for the program.  There was also a suggestion that the BL-NCE program should consider targeting future applicants based on availability of Sector-based Technology Roadmaps²⁶ because they have already gone through extensive planning and consultation processes.
• Clarify expectations of BL-NCE review process – PSAB and Expert Panel members indicated that, although the review process for selecting the networks was very time consuming and labour intensive, it provided an important level of due diligence and credibility for the selected networks.  Applicants were challenged to build a strong business case for their networks and ensure the right partners were at the table.  Based on lessons learned, it was suggested that the NCE Secretariat should provide clear expectations for the time commitment required of PSAB members in the review process and provide more background information on the overall research landscape (e.g., existence of other related research networks) in order to ensure panel members are aware of the broader context.

4.1.1 Models and Management Practices to Achieve Network Outcomes

Facilitating factors at the network level

Based the BL-NCE survey, 17 of 20 partners identified network design, governance structure, leadership, and network project selection process as factors facilitating the performance of the network (either somewhat or significant).  Researchers also identified these factors as facilitators (between 19 and 21 of 41 researchers).

Although each of the funded networks is unique in terms of its industry sector and evolution of the network, in comparing networks a number of factors that facilitated success in one or more networks were identified.

• Capacity of Network Management – The case studies highlight the complexities involved in establishing governance structures and managing relationships.  Interviewees noted that previous experience in working with industry-university R&D partnerships was an asset for network management. The level of resources allocated to administration and management by each network varies. One network has a team of four senior professional senior staff and administrative support working full time managing the network. The other three networks had fewer, and in some cases part-time resources. One network has only 1.5 professional resources, while two networks have a director but share administrative services with other organizations.  Interviewees associated with the network with a larger and full time management team stressed the leadership abilities and skill sets of the network’s management team were a critical success factor for this network.
• Governance structure – In all four networks, the private sector partners play a significant role in strategic decision-making.  Industry partners have majority representation on the Board and project selection committees; thereby ensuring research is driven by industry needs. It was also noted in one case study that individual Board members have extensive experience with university-industry R&D partnerships including other NCEs.  This experience, and pre-existing relationships with university researchers, has helped in the implementation of network projects.  Another network noted that joint meetings of BOD and Scientific Committee improve the Board’s understanding of research needs and plans. It was also observed that the network model provides a neutral ground where private sector competitors can work together on pre-competitive projects in support of their common interests.
• Process for identifying priorities and selecting research projects – Interviewees all agreed that the active role of industry in each network has facilitated the achievement of program outcomes relevant to industry needs.  As one interviewee noted: “university research is usually seen to be ‘curiosity-driven’ and the BL approach allows industry to be the source of research curiosity.”  Years of work by one network in developing a consensus among public and private stakeholders on the priority needs of the sector and building partnerships around the research themes was an important factor in the network’s ability to get off to a quick start.  For another network, a matrix management approach ensures that all research elements and key industry sectors are represented on the Scientific Committee.  Viewing the program from the perspective of the research themes helped to identify crosscutting issues and common research objectives across industrial sectors

Inhibiting factors at the network level

According to interviews and case study analysis, some of the key challenges associated with the implementation of a BL network model include:

• Managing cultural change – All four networks experienced challenges with respect to managing the complexity of the new network model. Implementing the concept of open innovation, wherein industry players collaborate in a research environment and share IP, represented a significant cultural change for firms operating in a highly competitive global marketplace.  For long-term success, taking the time to build trust among partners was seen as critical.  For one network, developing a new research network operating across a number of diverse industry sectors that are not accustomed to working together and with substantially different R&D cultures added even greater complexity.  Observations made in one case study revealed that managing the network required complete transparency.  Aligning universities and SMEs in collaborative efforts was also required. 
• IP issues – Negotiating changes to the draft Network and IP Agreements provided by the BL-NCE that were acceptable to both partners and the BL-NCE Secretariat took considerable time.  As many interviewees noted, the time and effort required to formalize the network was underestimated.  In two networks, some projects took more than a year to get underway and as of March 2011 there were still several Network Member signatures missing from the Network Agreements.  The Network’s revised IP approach is now governed by project-level agreements within a broader Network-level framework (which states that the IP rights reside with the researchers as long as they are a Network Member).  To further manage industry’s IP concerns, one network has limited membership to one company from each industry sector represented; additional industry members from a given sector will be allowed to join with the agreement of the original member.  Network management recognizes that this policy presents a challenge to the recruitment of new Network members, but also attributes their ability to sign several large companies to this approach.  For one network, the complexity associated with the management of IP across industry sectors, with different approaches to R&D management and corporate cultures, was a significant factor in the delays to the Network Agreement.  For the other network, several interviewees commented that the original Network Member Agreement was designed for organizations that are performing research, rather than for those who are funding it.  Discussions with the BL-NCE to develop a revised Network Member Agreement suitable for the network’s unique approach took more than a year.  Problems with the Network Member Agreement negatively affected the willingness of some firms to participate in and contribute to the network.
• Economic downturn – In the case of one network, industry members were unable to meet their original pledges of cash and in-kind support in the first year of the program due to difficult economic conditions in the sector and more broadly. As a result of these challenges, the original research program, which was based on a pooled funding model, was abandoned and a project-based approach was adopted.  Industry members now contribute financial and in-kind resources on a project-by-project basis and the contributions made by the provincial government members are also nominally assigned to specific projects.  The new approach allows companies to invest in those network research projects that are specifically targeted at their needs (e.g., process and product improvements) and requires a lower level of financial commitment. Under this new model Network Members may submit project proposals at any time using a Project Proposal Template.  It should be noted that the network partners and management interviewees felt that the new project-based funding model works well and has increased industry’s level of interest in network research by allowing them to participate in specific projects of direct relevance. 

Best practices at the network level

Some best practices identified by individual networks were:

• Formalized project selection processes – Each network has developed its own process which generally feature Letters of Intent, preliminary review and invitation to submit full proposals, independent peer review process, review and recommendations prepared by the network’s scientific committee with final approval of the industry-led BOD.  One of the networks considered its approach to be a best practice because it utilized an independent peer review process led by a provincial research institute while the network conducted its own project risk assessment.  Use of the provincial research institute to manage the scientific peer review process adds a higher degree of independence. The risk-based assessment performed internally by the network goes beyond scientific review and takes other factors into consideration. This approach is considered a best practice because the results of both the scientific and risk-based assessments are presented to the network’s BOD for the final project selection.
• Business participation in research – In one network, business partners lead each research project, with university researchers contributing their expertise through sub-projects.  This approach ensures that the research is directly aligned with industry needs.  Mobilization of project results is also more direct. (Refer to Section 6 for more details.)
• Mentorship – Two networks utilized a mentorship approach, with an industry representative providing guidance to the research project. In one network, for each of the funded projects, each industry member identifies from within its own global organization a renowned expert to help the investigators stay aligned with the industrial needs and eventually act as a champion when the time comes to enter into a license agreement between the project team and an industry sponsor.  This mentorship approach is considered a best practice by this network because it has allowed each industry partner to be directly involved throughout all phases of projects and has led to better transfer of knowledge.
• Time limit for research agreements – One network stipulates in its Call for Proposals that a research agreement must be signed between the network and all the research entities involved in each project within three months following the final selection announcement, otherwise the project will not proceed.  In addition to clear description of the project, milestones, deliverables, budget, disclosure and publication requirements, the agreement sets out the main terms and conditions of the license option in favour of the network’s industrial sponsors for the use of the results generated by the project (and background IP, if necessary) for research and development purposes. This time limit is viewed to be a best practice because it ensures that selected projects proceed in a timely manner. If an agreement cannot be reached within the three month timeframe, the project will not proceed and funds can be reallocated in a more timely way to other research priorities or research projects.

Lessons learned at the network level

• Time required for start-up – Several program and network level interviewees noted that the timelines for network set-up (e.g., recruiting Network Members, finalizing the Network Agreement) were overly ambitious. In particular, the time required to modify and reach agreement on the Network Agreements and IP policy exceeded initial expectations and led to significant project delays for some (e.g., in two networks it took more than a year to get projects underway).  In one case, this was attributed in part to the multi-sector nature of the network and the involvement of industry partners that had no history of working together.  In another network, management explained that time was required to educate private sector members on government accountability requirements (e.g., terms and conditions of federal funding) and to find the appropriate balance on how the network should operate recognizing the needs and expectations of both its private and public sector funders. Another network observed that, although industry was making a direct contribution to research, there was still a need to ‘play’ according to the rules of academia (e.g., the requirement for independent peer review of proposed projects).  The evaluation lesson learned is that it is important to recognize and anticipate the potential growing pains and learning curve involved in setting up a new type of network model and set expectations accordingly.
• Performance measurement and reporting requirements – Program level interviewees noted that the identification of appropriate measures of success was a major challenge for BL networks.  It was suggested that technology development milestones should be tailored to the different sectors.  The extent to which the BL networks are able increase private sector in-kind and financial contributions will be an important measure of success of the program over the long term.  At the network level, many interviewees noted that measures are not well aligned with BL-NCE objectives and that reporting requirements are burdensome (e.g., several different reporting templates for annual reports requesting extensive information about university-based research). Program-level tools and support are limited; however, network meetings and monthly conference calls were viewed as useful as they provide an opportunity for participants to learn from the experiences of the other networks. The evaluation lesson learned is that traditional metrics used in assessing other government supported university-based R&D network models are not necessarily suited to the BL approach.  More work is required to determine the most appropriate indicators of success, recognizing some measures should be tailored to better reflect the objectives of each network’s research agenda. 
• Administrative capacity – There was variability across the networks with respect to the level of administrative capacity in terms of the number of full time staff resources. In two networks, administrative services are shared with other organizations. One network was adversely affected by the departure of a senior manager, as the replacement was only available on a part time basis for several months.  The evaluation lesson learned is that, while there is no specific evidence of the exact amount that should be set aside for administrative support, a critical mass is essential to ensure that members receive the support required to fully reap the benefits of the network.
• Board of Directors composition – The BOD for some networks are comprised mainly of industry and academic representatives with a specific interest in the network while others include representatives with specific expertise in support of the administration of the network (e.g., legal or financial expertise).  The evaluation lesson learned is that those networks with a blend of sector, research and other expertise had more direct access to advice to help resolve issues (e.g., pertaining to Network Agreements and IP), and to exercise due diligence regarding project applications (e.g., financial capacity).
• Research lead – Different types of research leads were observed. Some networks utilize predominantly university based research with industry involvement; for one, the research is private sector based with university involvement; in others it is a blend of both.  The evaluation lesson learned is that, while there is no right or wrong approach, each approach has impacts in terms of mobilizing research results and length of time expected to commercial benefits. (Refer to Section 6 for more details).

5.0 Network approach to research, development and innovation

Summary:

Study findings show that the BL-NCE model has contributed to industry-relevant research in the areas of the four funded networks.  The BL-NCE program has provided new funding for research, development and innovation and the BL model has had a positive impact on the way in which the networks’ research projects are identified and delivered.  The level of involvement of industry network members in the development of research priorities, the scientific committees and BOD, and in some cases overseeing and carrying out research projects, ensures that projects are directly relevant to industry’s needs.

According to the most recent annual data provided by the networks (2010-11), the total number of network members (Network Agreement signatories) across the four networks is 60; another 50 organizations participate in a partner role, either providing financial or in-kind contributions, and / or participating in research projects.

The BL network model encourages multidisciplinary and multisectoral research teams.  To date, the BL-NCE program has funded 89 projects, involving 378 researchers at 46 different organizations.  Sixty-two percent of network projects are conducted in university labs, 22% are industry-led and 16% are led by a research organization.  The size and scope of projects varies by network: one network has funded 53 one-to-two year projects, each with a relatively small research team; another network has seven large multi-year projects in place that involve a total of 117 researchers.  (More detail is provided below and in Table 1-1.)

The BL-NCE facilitated the development of cross-sectoral networks, bringing together sectors that have not traditionally worked together.  The Network Agreements and IP arrangements, while taking more time to finalize than anticipated in some cases, once in place facilitated the development of multisectoral and multidisciplinary network R&D projects.

While the networks are still in their early days, there is evidence that the program (through the networks) helps to increase the visibility, and enhance the reputation, of Canadian researchers and network firms nationally and internationally.

Overall, the network partners who were interviewed believe that, despite start-up delays at some networks, the BL research projects now underway will meet their needs and have already strengthened links between the research community and industry.

5.1 Enhancement of research, development and innovation in the areas of funded networks

Interviews with network managers and researchers revealed that the BL-NCE program has effectively enhanced research, development and innovation.  They attribute this to the fact that the networks:

• are solution-driven, leading to projects that are more relevant to industry than traditional network programs;
• involve a broader range of expertise than could be achieved through non-network research (e.g., multidisciplinary project teams and, in the case of two networks, multisectoral projects);
• encourage greater collaboration and joint research projects between industry and academia; and
• provide the basis for undertaking larger projects than would otherwise be possible without the networks.

Interviewees indicated that research projects reflect industry’s needs as a result of the approaches used by BL-NCE networks to identify research priorities and select projects.  As described in Section 4, the networks’ BOD must approve all funding decisions, and in the case of three networks the project selection process involves peer review.  As shown in Figure 3-1 (R&D Spectrum and the BL-NCE Program), BL networks focus on proof of concept projects through product development.  The project mix varies by network from mainly pre-commercial research (e.g., the development of improved research platforms for pharmaceutical research) through to near market-ready development projects (e.g., the application of NanoCrystalline Cellulose (NCC) in paper and other products).  One network has developed two project streams: one to accommodate its primary research program, and a second that provides support to smaller, highly innovative projects that may carry more than the usual degree of scientific risk.

Interviewees noted that the Network Agreements and IP arrangements, once established, facilitated the development and implementation of multi-partner projects across sectors and disciplines.  As previously noted, in some cases networks took significantly longer to develop these agreements than originally anticipated.  Network Agreements are now in place and the total number of network members (i.e., signatories to the Network Agreements), as reported by the networks in their annual 2010-11 reports, is 60:  26 private sector organizations, 24 universities, 6 provincial government departments and 4 other organizations.  The number of network members in each network ranges from 4 to 22.  See Table 1-1 for details by network.

In addition to network members there are also partner organizations involved in the networks, either providing funding for network research and / or participating in network projects.  As reported by the networks in their annual 2010-11 reports, a total of 50 partners currently participate in BL-NCE networks: 29 private sector organizations; 7 universities; 6 provincial government organizations; 5 federal government organizations; and 3 other organizations.

While most networks began to fund projects in the first year of the program (one network had projects ready to go as soon as the BL-NCE funding was awarded) the overall rate at which projects were approved was slower than anticipated.  One network’s projects were significantly delayed because industrial partners were unable to provide the funding that had been promised, and building cross-sectoral working relationships took more time than anticipated.

A total of 89 projects received BL-NCE funds in the first two years of the program.  Sixty-two percent of the projects are conducted at universities, 22% are industry-led and 16% are led by a research organization.  In the case of one network, all projects were private sector-led.  The size and scope of projects varies by network: one network funded 53 one-to-two year projects, each with a relatively small research team; another network approved seven large multi-year projects that involve 117 researchers.  (Table 1-1 provides detail by network.)

The researcher survey results illustrate the impact of network funding on project activity.  When asked what would have happened had BL-NCE funding not been available, nine out of ten indicated there would have been a major negative impact on the project, and the others indicated there would have been a minor impact on the project.  When asked what specifically would have happened researchers indicated that the project would not have gone ahead, would have been delayed or the scope would have been reduced.  This is similar to the results from the survey of comparable networks.

Network projects involve 378 researchers at 46 unique network partner or other organizations.  The number of researchers and unique organizations involved in BL-NCE funded projects is shown in Table 5-1.  ‘Other’ organizations include hospitals, governments, and research organizations.  The number of researchers participating in each network’s research projects ranges from 57 to 129.

The level of industry participation in BL-NCE projects varies by network: one network’s projects do not involve any industry researchers, while at another network industry researchers account for 87% of all participating researchers.

There is some early evidence that the BL-NCE networks are enhancing the visibility and reputation (nationally and internationally) of Canadian researchers.  Interviewees felt that it is too early to expect research awards; however, they cited some specific examples of increased awareness of Canadian research including:

• recognition of Canada’s leadership in a network research area (NCC) at the 2010 TAPPI conference held in Finland (TAPPI represents pulp, paper and packaging industries);
• collaboration with France that features joint financing of collaborative research projects between Québec and Alsace;
• technology developed by network member has resulted in a funded project with US government agency; and
• participation of experts from the US and Europe at a recent annual conference.

5.1.1 Facilitation of multidisciplinary, multisectoral and international collaborations to address research challenges

The research, development and innovation challenges addressed by the networks require a wide range of expertise and academic disciplines.  Interview and document review findings showed that the networks have established the necessary research collaborations with relevant researchers, partner organizations, disciplines, institutions and sectors.  The project selection criteria take due consideration of these factors and the project approval process ensures that the collaborations are in place.  However, one of the four networks has been unable to involve a key research organization as the organization is not eligible for BL-NCE funding and there is insufficient funding from other sources to allow for its full participation.

Two networks involve extensive cross-sectoral collaboration.  Of the 14 companies involved in the one network’s project portfolio (network members and partners), half are traditional aerospace companies and the others represent a range of sectors including electronic systems and alternative fuel.  Another network brought together a number of manufacturing sectors (e.g., automotive, aerospace, medical / health, forestry) to develop and apply NCC-enhanced materials.

Interviewees noted that network projects are multidisciplinary and involve multiple organizations by design.  A review of the 89 BL-NCE projects shows that a majority of projects involve at least one company and one university and span a number of technical areas or research disciplines.  For example, one network’s research projects involve biochemists, biologists, cell-biologists, neuroscientists and ophthalmologists.  Another network’s projects involve seven universities from across Canada and researchers from a range of academic disciplines including chemistry, electrical and chemical engineering, materials, composites and mathematics.

In addition, the networks promote multisectoral, multidisciplinary collaborations through network membership and members’ participation in various committees (e.g., BOD, scientific committees, peer review teams).

Two networks have established international collaborations with European-based research organizations to address research challenges.  In accordance with BL-NCE guidelines, organizations and researchers from other countries are not eligible for network funding.  In one network, interviewees reported that international collaborations would not be a great benefit at this time as Canadian researchers have a one to two year lead over their international counterparts. 

Survey results are consistent with interview and document review findings.  The survey results indicate that the BL networks have resulted in multidisciplinary and multisectoral research collaborations as well as in the establishment of the research collaborations needed to address the needs of network organizations.  The survey of comparable networks shows similar results.

In addition, three-quarters of BL-NCE researchers surveyed indicated that they considered the collaborations between the researchers on their research project to be successful to a good or great extent (this is similar to the comparison networks survey results).

The researcher survey results also show that the private sector and universities generally lead at different phases of the innovation spectrum.  In most cases the research is planned, directed and used by the private sector network members, and implemented, operationalized, analyzed and disseminated by university members.  (Note that one network’s research is performed almost entirely by the private sector.)  The comparison survey showed that other networks were university-led in all aspects of the research process except for the use or application of knowledge and technology, where universities were equally as likely as the private sector to lead.

5.1.2 Needs of partner organizations

Interviewees at partner organizations believed that the networks were meeting the needs of their organizations and strengthening links between the research community and industry; however, they also noted that it was too early to expect to see evidence in the form of new products and processes.  These interviewees noted that they were able to fully participate in decision-making, set research goals, and influence research planning and agendas.  No one identified barriers to participation. 

When asked to assess the extent to which the networks have met their organization’s needs, close to one-half of BL-NCE members and partners surveyed indicated that their needs had been met to a great extent; the great majority stated that their needs were met at least to some extent.  The primary ways in which the networks are meeting member needs were enhanced collaborations / networking, access to funds and the type of research funded.  The great majority of partners agreed or strongly agreed that the networks are successfully identifying the members’ interests.

One measure of participation in the networks, and their level of influence, is the extent to which network members are represented on various committees.  The number of Board members per network ranges from 12 to 16, and network signatories account for between 30% and 44% of these positions.  Industry signatories’ participation on the Boards ranges between 17% and 31%. 

As noted above, the networks accommodate a range of research needs (from proof-of-concept through product development) and are able to bring together the appropriate researchers to meet these needs.  The BL-NCE project portfolio (89 projects) includes a wide range of project types from relatively small, one-year projects, to multi-year projects with large research teams (up to 40 researchers on one team).  In one network, all projects are industry-led; in the other three networks the majority of projects are conducted at universities.  Each network’s approach is somewhat different and reflects the needs of its partner organizations. 

6.0 Mobilization and benefits of network research

Summary:

The BL-NCE program has focused the attention of the industry stakeholders participating in the four networks on the use of partnerships with universities and other research organizations for the development of new technology to address major commercial challenges. In the two newly formed networks, the BL-NCEs have built partnerships among firms and universities focused on carrying out R&D aligned with the needs of the industrial sector. The BL-NCE program enabled the pre-existing networks to continue and extend their earlier efforts. The business-led network approach (including the development and implementation of a strategic plan, project selection and oversight) is seen as an effective mechanism to promote mobilization of research by industry. The BL-NCE program has also expanded and broadened the scope of research being carried out. In two networks, mentors provide advice to ensure project alignment with industry needs. Through the networks, partners are becoming more willing to work together and share IP. Two of the networks have been successful in attracting new partners and increased investments. In one network, several original private sector partners are no longer providing funding, and have not been replaced.

Research is carried out in different ways by each of the four networks, with various amounts of research carried out by universities, businesses and not-for-profit research organizations. Mobilization mechanisms vary among the networks. In some cases, knowledge is transferred from universities to firms and in others between firms. The primary methods of knowledge mobilization identified are networking, IP and non-disclosure agreements (NDA)²⁷, refereed publications and patent applications. When the industry end user is the primary participant in research projects, mobilization can occur directly. In other cases, the pathway is less direct. All networks also make use of conferences and / or workshops to share results among stakeholders and to promote the networks to a broader community.

All networks have been successful in terms of establishing and building partnerships, and increasing their partners’ knowledge base and R&D capacity. However, most interviewees and survey respondents agreed that it is too early in the program to expect intermediate and longer term benefits such as new or improved products and processes or increased competitiveness. However, the early application of network research in the development of new products and processes is reported in two networks.

6.1 Impact on partner organizations

The impact on partner organizations of participating in the BL-NCEs varies among the four networks. The following factors have contributed to positive impacts on partner organizations:

• Research driven by industry needs – The BL-NCE program has focused the attention of the industry stakeholders participating in the four networks on the application of network R&D in the development of new technology to address major challenges and their future commercial success. Interviewees reported that the BL network approach (including the development and implementation of a strategic plan, project selection and oversight) is an effective mechanism to promote mobilization of research by industry.
• Culture change – Interviewees also reported that the business-led network approach helped bring the network partners, both research performers and beneficiaries, together to build awareness and improve relevance of funded research. Business has had to learn about the government and university cultures and vice versa. Several interviewees reported that the network approach has increased the level of collaborations and discussions between and among industry partners and universities and helped each understand the other’s perspective. Traditionally many firms have been very protective of their IP and unwilling to even enter into such discussions. For many firms, this is the first time they have participated in a business-led network with this type of approach. Interviewees noted that firms are becoming more willing to share, although this is viewed as a “work in progress”.
• Private sector partners address common objectives – All networks bring together a number of firms in various sectors to work on common precompetitive technological objectives.  By identifying common objectives partner firms are able to leverage their R&D investments with those of other firms. One network firm noted that its participation in the network has raised the profile of R&D at the corporate level and, despite the recent economic downturn, protected its internal R&D budget, which is now seen as an important contribution to developing the next generation of innovative products to help maintain the firm’s competitive position.
• Attraction of new partners – Two of the networks attracted new partners with complementary interests and increased private sector investments. In one network, four additional projects with a total value of $7 million were funded with participation by SMEs, universities and other partners not part of the original proposal. However, in another network, several original industry partners have not continued making financial contributions in the second year.  The primary reason provided by interviewees for reduced participation by industry was that the inability of the provincial research organization to receive BL-NCE funding, unforeseen at the time of the formation of the network, reduced the relevance of the BL-NCE program to industry.
• Increased knowledge base – About three quarters of partners surveyed reported that the networks had increased the knowledge base. In addition, about half reported that the BL networks had positively impacted the R&D of network partners. The results of the comparison group survey were similar.   Most interviewees reported that it is too early to expect significant intermediate or longer term benefits such as the development and adoption of new or improved products and processes or increased competitiveness. However, in one network, the successful application of network research has already resulted in the development of new technologies that are being incorporated in next generation commercial products. 

6.1.1 Mobilization of knowledge and / or technology by partner organizations

Networks have utilized various approaches to fund research, select projects, and carry out research. These various approaches have implications for the type of research carried out and the methods used to mobilize and transfer of knowledge. Factors affecting the mobilization of knowledge and / or technology by partner organizations are:

• Mix of funders and research performers – Each network has a mix of funders, partners, and research performers, depending on the specific circumstances of the sector and the research focus. As well as the BL-NCE program, all networks have funding from other federal and / or provincial government programs. While most private sector funding comes from large firms (cash and in-kind contributions), SMEs participate in and contribute funding in two networks. Research performers vary among the networks, depending on the sources of research capability and expertise. Universities carry out research projects in all networks; in one they are the primary research provider. Network research partners also include federal and provincial government research organizations and not-for-profit research organizations with expertise in the sectors supported by the networks (aerospace, forestry, oil production and pharmaceuticals). Participation by industry in performing network research varies greatly, from no firms in one network, to 15 in another.
• Number of public and private sector researchers – In total, there are 378 researchers participating in the four networks, however the number of researchers participating in each network varies greatly, ranging from 28 to 129. The lowest number is for the network where all research is carried out by universities and a not-for-profit provincial research organization, and the largest number is for the network where research projects are led by industry and embedded in their ongoing development of new and improved products and processes. Most university based research teams are small, consisting of one or two professors and one or two graduate students. There are 196 private sector researchers involved in the research carried out by the four networks. The level of involvement of researchers from industry is dependent on the research capacity of the industrial sector and, consequently the number of industry researchers is highly variable among the networks. The number of industry researchers participating in network research ranges from zero for one network to 112 in another network. The other two networks each involve 42 industry researchers.  Industry research teams vary in size, depending on the size of the project. For the network with the largest number of industry researchers, the large projects have large research teams of over 12 people.

Networks make use of a range of mechanisms to mobilize knowledge that include:

• Formal agreements – The most frequently mentioned mechanisms reported in the survey of BL-NCE partners and researchers were network agreements associated with IP and commercialization and non-disclosure or confidentiality agreements, with almost half of respondents identifying these mechanisms. Based on the results of the comparable networks survey, network agreements associated with IP and commercialization are used to a similar extent in these networks. However, non-disclosure or confidentiality agreements are used much less frequently.  Each BL network has developed procedures to manage IP among project participants. These agreements are expected to facilitate the application of research results by private sector partners. One network also makes use of Material Transfer Agreements as a mechanism to support mobilization of research.
• Publications and articles – The networks also make use of publications and articles to mobilize research.  About a third of the BL partners and researchers surveyed indicated that they have published research findings in refereed publications, some involving joint publications by academic and private sector researchers. For the comparison networks, refereed publications are more important as a mechanism to mobilize research results, with about two thirds of those surveyed reporting the use of refereed publications. At the time of this evaluation, the initial multiyear research projects were still underway within the networks. Consequently, in their first annual BL-NCE report, three of the four networks did not report completing any reports or journal publications. One network which took over funding of an existing program, reported²⁸ a total of 46 publications, including 23 articles published or accepted in refereed journals. The majority of these publications resulted from university-based projects. 
  
  Another approach to facilitate mobilization of research is used by one network, which supports a sector with little industrial research capacity. University-based research in this network results in refereed publications.  To facilitate knowledge mobilization, the network has engaged a technical communications firm to rewrite refereed publications and research reports from universities in a more user-friendly form aligned with the technical needs of their industrial partners.
• Industry mentors – As discussed in Section 4, mentors from industry provide advice to project teams and ensure alignment with industry needs in two of the networks. The mentoring also contributes to improved knowledge transfer.
• Conferences and workshops – Interviewees reported that all networks make use of conferences, workshops and meetings to share research results among stakeholders. A conference held by GARDN in 2011 was attended by an international audience of almost 200 industrial, government and university participants involved in various aspects of aerospace policy, funding and research. The conference, which included presentations on Canadian, European and American initiatives to improve the environmental performance of future aircraft, highlighted the network and its research projects. CQDM has organized annual forums for discussion and exchanges giving voice to different stakeholders mainly from Quebec, but also from the rest of Canada and internationally. In 2009 the forum attracted 150 participants and the following year more than 160 participants including clinical researchers and industry representatives, who emphasized both the need for public-private partnerships and the importance of international collaborations.²⁹

Demonstration of technologies – For one network, the overall strategy is to “develop, demonstrate and commence deployment of technologies”.³⁰ Successful demonstration in real operating conditions is planned as the major strategy to support technology transfer and application of the technology developed in the network by industrial partners. However, this is not expected within the initial four-year funding period. In another network, the strategy is to fund pre-competitive research to develop tools and innovative technologies to facilitate and accelerate the drug discovery process.

6.1.2 Benefits to partner organizations of network activities and the use of network knowledge and / or technology

As discussed in Section 3 (see Figure 3-1), network research is intended to take applications to the advanced development or demonstration stage. Past this point in the chain, individual network partner firms take over and carry out late stage R&D to develop new and improved products and processes aligned with their commercial interests. The industrial research capacity of each sector is a major factor in the ease and extent of knowledge transfer from the network to individual firms. For example, the pharmaceutical industry has a high level of research capacity and can mobilize network research at an earlier stage and more directly than the Canadian oil and gas sector, which has little internal R&D capacity.

Interviewees reported that all networks have been successful in terms of establishing and building partnerships, increasing their knowledge base and building R&D capacity.  This was confirmed in the surveys of network partners and researchers, with a majority of respondents reporting an increase in the R&D and knowledge base of network organizations.  Interviewees noted that it is too early in the program to expect a significant level of intermediate and longer term benefits such as development of new or improved products, services, processes and / or improved productivity and competitiveness.  However, about a quarter of the respondents to the partner and researcher survey reported that these benefits had already occurred.  The comparable networks survey found similar results to those of the BL partners and researchers survey.

Interviewees identified a wide range of future commercial benefits expected to result from the research carried out in the networks. The following are examples from each of the networks;

• Improved printing inks to enhance security and reduce counterfeiting (printing);
• Enhancement of drilling mud characteristics (oil and gas);
• Biosensors to monitor disease progression and drug effects (pharmaceutical);
• Improved antigen expression and purification equipment (pharmaceutical);
• Airframe and engine noise reduction (aerospace);
• Alternative sustainable jet fuel (aerospace)³¹;
• Enhanced heavy oil recovery using solvents (oil); and
• Enhanced bioremediation technology for oil sands (oil).

Interviewees did not report any impacts on government regulations or policies arising from BL-NCE research, although there is potential for impacts on oil and gas production regulations.

Interviewees identified two examples of the early application of research carried out in one network. In one case, the network research has contributed to the development of a next generation flight controller that increases fuel efficiency and reduces GHGs.  In the other case, the research is being incorporated in the design of aircraft landing gear with reduced noise levels. 

Technology transfer to individual companies has been limited and therefore longer term impacts such as increased revenues, cost savings or environmental effects have not yet occurred. However, in one network, a firm reported that being known as a partner in the network has provided an important competitive marketing advantage, as the network is known to be supporting the development of technologies that address the major challenge that the sector must address to remain competitive.

7.0 Training of Highly Qualified People

Summary:

While all networks have an objective of increasing research capacity and mobilizing newly generated knowledge to meet the needs of the sector, the emphasis on the role of training of students on network projects varies among the networks, depending on the particular delivery strategy of the network. Depending on the organizations carrying out the project, network research provides training of HQP at universities and opportunities for developing increased expertise for the private sector researchers participating in projects. In two networks, training of HQP is identified as a major objective, however all networks provide opportunities for training of HQP through university-based research. During the first year, a total of 83 HQP were reported to have worked on network projects. This number is expected to increase, as the participation of HQP in some networks during the first two years of the program has been affected by delays in getting university-based research underway as discussed in Section 4.

By participating in network research projects, students gain expertise and knowledge relevant to the needs of the industrial stakeholders participating in their network. HQP interviewees reported that they participate in network projects in order to gain commercially relevant experience and improve their opportunities for employment in the sector after graduation. Researchers agreed that HQP involved in BL-NCE networks gained more exposure and awareness of industry needs and practices than those involved in other projects. They also had more opportunities to interact with private sector researchers.

7.1 Impact on training of HQP

In two networks, generation and training of HQP is identified as an important outcome.  One of these has established bursaries to encourage students to participate in graduate studies related to network research projects carried out at universities. In the others, there is little or no specific emphasis on training, even though training is identified as an outcome in the BL-NCE logic model (Figure 1-1). However, because all networks are funding university-based research projects, they have each contributed to the training of HQP, as a by-product of the research projects that employ Masters and PhD (Doctor of Philosophy) graduate students and Post-Doctoral Fellows (PDF) as part of the research team. For BL networks, evidence shows that training of HQP at universities and non-university based research personnel is occurring.  Almost two thirds of respondents to the survey of BL-NCE partners and researchers reported that training of both HQP and research personnel has happened. Based on survey results, training of HQP at universities is even more prevalent in comparable networks. However training of non-university based research personnel is much less prevalent in these networks.  This reflects the fact that, for comparable networks, most research projects are carried out at universities. 

Depending on the amount of university-based research conducted by the networks, the level of training of HQP also varies. According to the 2009-2010 reports³² to the BL-NCE, the number of HQP involved in projects in each network at that time ranged between 8 and 34, with a total of 83. Information about the number of HQP who are Canadian or non-Canadian is incomplete, however, for the network with the largest number of students, there were an equal number of Canadian and non-Canadian students. It is expected that the number of HQP participating in BL-NCE research projects will increase over the next two years, as more university-based projects are funded.

HQP interviewees reported that they participated in network projects in order to gain commercially relevant experience and improve their opportunities for employment in the sector after graduation. In addition to the graduate students and PDFs at universities, other organizations in the networks have hired additional young researchers to work on network funded projects. These new employees and other researchers working on network projects are also gaining valuable expertise in carrying out commercially relevant applied research.

7.1.1 Skills and experience relevant to the private or public sector acquired by HQP

The survey of BL-NCE network researchers examined the training opportunities offered to HQP by the network.  The majority of respondents reported that HQP working on BL projects acquire technical and professional skills, conduct research relevant to the private sector, and have access to cutting edge technology and research facilities.  HQP working on BL projects are more likely to acquire technical skills than those working on comparable network projects.  For the other benefits, the results of the BL and comparable networks surveys of researchers are similar.³³

In some networks, students gain experience by participating with lead investigators in conferences, workshops and meetings with industry representatives who provide advice and feedback. Students improve their communication skills by participating in and presenting at conferences and workshops. Students benefit through co-authorships on research publications, which demonstrate the expertise they have gained through their research projects.

A small number of HQP were interviewed as part of the case studies. Students who were interviewed provided comments on the type of research experience that they received through the networks. Students involved in multi-disciplinary research spoke of the benefits of collaborations with other disciplines in broadening their expertise and increasing their number of contacts and employment opportunities. Two students spoke of their appreciation for participation in industrially relevant research projects, with enhanced opportunities for employment in the industrial sector after graduation. One student noted that, for his network project, the level of oversight and expectations of quality are higher than in other university research projects; this student attributed this to industry participation. There is evidence that the networks are attracting foreign students to Canada; for example, one PDF interviewed came to Canada specifically to work on a network research project. Based on the few interviews carried out, the level of interaction of students with researchers and stakeholders varies. In some cases, due to confidentiality agreements and IP issues, the students’ participation in network projects is limited to their specific task.

8.0 Performance (Efficiency and Economy)

Summary:

At the program level, close to $41 million in funding has been approved for the four-year period spanning 2008-2009 to 2011-2012. After two years, the program has funded $31 million of which only $9 million has been expended (or 29%). It is therefore unlikely that the funds will be fully expended at the end of the current funding period. While it may be possible that the funds will be fully committed by March 31, 2013³⁴, it is likely that several projects will not be completed at that time.

Operational expenditures by the NCE Secretariat over the three fiscal years since program start (2008-2009 to 2010-2011) are estimated to total $1.8 million or 5.9% of the grant funds; this is comparable to other programs examined (i.e., Strategic Network Grants (SNG) program and Strategic Grants Program (SGP)).  These other programs represent higher grant funds; operational efficiencies are more feasible for larger programs given a larger critical mass.

At the network level, interviewees noted that the network resources were adequate to achieve expected results; however, the level of dedicated in-house support personnel varies across the four networks (ranging from one part-time support person to five full-time support personnel).  This was noted to impact the level of support that can be provided to members as well as the ability of networks to expand their programming with the current level of staff resources.

Over two years, BL-NCE network partners have contributed more than $38 million through cash or in-kind contributions. Based on the first two years, for every BL-NCE dollar of funding, an additional $1.23 is contributed (cash or in-kind) by partners ($0.76 when public sector funders are excluded).  Overall, the total partner contributions exceed the program requirements for matching funds.

There are limited perceived opportunities for improving the program’s efficiency.  One key area of improvement pertained to simplified reporting requirements to ensure that requirements are relevant to business-led networks rather than academically-driven networks. Another area for improvement related to increasing the funding period from four years to at least five, particularly since this is a very short timeframe to set up networks and show results.

8.1 Efficiency and effectiveness of program delivery

In order to assess the efficiency and effectiveness of program delivery, the program resources (and appropriateness), leveraging, and the perceived effectiveness and efficiency of program delivery were examined.

Efficient use of program resources

Program data indicates that grants funds for the BL-NCE program $42,148,125 for the current four year funding period from FY2008-2009 to FY2011-2012.  As of the end of FY2010-2011, the BL-NCE program had awarded funding totaling $31,013,375: $9,743,875 FY2008-2009; $10,134,750 for FY 2009-2010; and $11,134,750 for FY 2010-2011. 

During that same timeframe, the administrative expenditures of the BL-NCE program are estimated at $1,799,116 representing 5.8% of the grant funds of the grants awarded. A comparison to other programs is provided in Table 8-1.

The table shows that the administrative expenditures for the BL-NCE program are aligned with those of other programs particularly in light of the fact that the BL-NCE has the smallest overall grants fund.  As shown in the table, operational efficiencies are more feasible for larger, established programs (e.g., NCE program) given a larger critical mass and no program start-up costs.

Efficient use of network resources

Network data revealed that the funded networks’ administration expenditures totaled $6,745,215 during the first two complete years³⁵ of the program.  This represents an average 23% of the total network expenditures with the remaining 67% being allocated to research expenditures.  The administrative portion of total expenditures varies extensively across networks from a low of 10.6% to a high of 86.6% of total expenditures.  This is due to the fact that some networks were slower in funding research projects than others.

Of the $31,013,375 funding provided to the BL networks to date, $9,011,035 has been spent by the four networks combined.  This represents only 29% of the grants funds.  Based on other findings presented in other parts of this report, this is at least partially due to the slow start for some networks in:

• Operationalizing the administrative processes for their network (e.g., hiring staff, establishing governance structure);
• Securing partners;
• Finalizing network and IP agreements; and
• Selecting research projects.

All networks indicated that they believed the grants funds would be fully committed by the end of the current funding period, although all projects would not likely be completed.

Interviewees noted that the program resources were adequate to achieve expected results.  However, as noted above, the proportion of funds going to research projects in the first two years varies extensively across networks (see Table 8-2).

The lower / higher administrative expenditures for networks is reflected in the number of staff with ranges from one full-time individual supported by one part-time staff to a full-time personnel of five, as well as in the level of network research expenditures.  The case study findings showed that this has affected the support that could be provided to members.  Interviewees from one network also noted that there was limited room to expand programs with the current level of staff resources.  It should also be noted that the two networks with the highest administration budget show higher levels of research activities (as demonstrated by higher research expenditures).

Efficient use of partner resources

The partners surveyed were asked if they had been involved in the development and / or preparation of the documents required for their organization to participate in the network as well as in reporting on their organization’s participation in the network.  While sample sizes are relatively small, the survey results are summarized in Table 8-3.

Additionally, networks benefit from the work of more than 100 volunteers from a range of public, academic, private and not-for-profit organizations participating in different network committees.

Leveraging

Leveraging is defined as the value of the contributions made by other parties versus the funding provided by the BL-NCE program.  For the purposes of this evaluation, leveraging data was available on the leveraged funds to network expenditures as well as the leveraged funds based on commitments. The amount of funds leveraged by the BL-NCE networks based on expenditures is summarized in Table 8-4. The table shows that the program has actually not only exceeded its matching funds requirements but it has also exceeded its projections. The table also shows that actual leveraging based on expenditures varies significantly across networks.

Table 8-5 summarizes the source of the cash and in-kind contributions (commitments) received by the four funded networks combined.  The table highlights that the program has exceeded its matching fund requirements based on committed funds.  However, as a large proportion (38%) of funds have been contributed by other public sector organizations (federal and provincial), the funds leveraged by industry, universities and others are below the matching funds requirements.

The table also shows that, while the federal government invested $31,013,375 (45%) in the BL-NCE program, industry put only $17,749,427 (26%) relative to the total contribution of the BL-NCE and the partners. While this may be an indication of private sector weakness in increasing its investment for R&D, it could also indicate that some networks are still working to securing private sector support (particularly as the networks are in their infancy).

Perceived effectiveness and efficiency of program delivery

A limited number of interviewees were able to comment on the effectiveness and efficiency of program delivery (as opposed to network delivery).  Interviewee comments are noted in the improvement section which follows (Section 8.1.1).

Based on the survey results, those partners aware of the BL-NCE program were highly satisfied with various aspects of program delivery.  However, partners were least satisfied with the guidelines for the management of IP. These IP concerns support findings highlighted in other sections of this report.

8.1.1 Improvement to program efficiency

Interviewees had few suggestions for improvement to the program’s efficiency.  A large number of network stakeholders commented on possible improvements to the reporting requirements for the program.  Interviewees noted that the requirements were “academic” in nature and therefore less appropriate for business-led networks.  For example, publications were noted as less relevant to business-led networks.  More relevant indicators such as improvement to technology readiness were deemed important.  Additionally, interviewees commented on the extensive details required in the reports (e.g., list of all individuals participating in research projects).  Interviewees also expressed concerns over the limited timeframe (four years) for the program funding and noted that the period was insufficient to ensure the effectiveness (i.e., program results) and efficiency (e.g., sustainability, administrative) of each network. Interviewees also noted that the program’s efficiency had been negatively affected by a lack of staff continuity at the NCE Secretariat.  However, this is an uncontrollable factor that cannot be resolved.

The partner survey results also revealed partner concerns with the length of the program’s funding period (n=2) and also noted the need for more flexibility on eligible researchers or how the research funds can be used (n=2).  All others suggestions were made by only one responding partner.  It is however noteworthy that more than half (53% or 9 out of 17) the partners answering this question could not identify any suggestions for program improvement.

9.0 Conclusions and Recommendations

9.1 Conclusions

9.1.1 Relevance

The original rationale for the BL-NCE program remains current. The program is aligned with the federal government priorities set out in Advantage Canada, The Next Phase of Canada’s Economic Action Plan (Budget 2011) and the S&T Strategy. The program is also aligned with departmental strategic outcomes as laid out in the PAA of the three funding agencies. Each of the funded networks is similarly aligned to the priorities set out in the BL-NCE program’s Terms and Conditions: Environmental Science and Technologies, Natural Resources and Energy, and Health and Related Life sciences and Technologies.  The other two priorities identified (Information and communications technologies and Management, business or finance) are not addressed by currently funded networks; however, no fundable networks were identified in these priority areas. The BL-NCE expands the scope of R&D in the industries involved in the funded networks. Two of the four networks would not exist without the BL-NCE program. The other two networks would be limited regionally and in scope without federal support.

There is an ongoing need for a program of the nature of the BL-NCE program.  This program helps fill a gap in the innovation spectrum between ‘proof of concept’ and ‘product development’.  The BL-NCE program uses a business-led network approach to bring together teams of private and public sector researchers to conduct the collaborative R&D required to address the identified needs of industry.  The novelty of the business-led model is that the teams of researchers funded by each network can be university-based, private-sector based, based in a not-for-profit organization, or a combination of the three.  The common feature across networks, and the niche of the program, is that the research itself is intended to address industry-specific or business-specific needs by involving the private sector more closely in the design and conduct of the research, thereby better ensuring the take-up and use of the results.  The program also helps fill a gap by providing the funding required to undertake this type of research (i.e., applied research to address business-specific needs that is led by the private sector) that would otherwise not be available or that would be insufficient to fully address the identified research needs.

9.1.2 Implementation

Although the program’s experience is limited to only four funded networks, the program design, in particular its business-led approach, is a facilitating factor in ensuring research undertaken addresses the needs of industry in these sectors. However, some of the expected outcomes for the BL-NCE program may have been too ambitious given the four-year timeframe for the program (e.g., address significant research challenges, accelerate commercialization) and the complexities in establishing business-led networks may have been underestimated.  The unique characteristics of each network (i.e., administrative capacity, experience of collaborative research, expectations of partners and industry needs) have resulted in a certain degree of flexibility in BL-NCE program implementation.  The program’s implementation experienced some difficulties and delays as networks struggled to establish Network Agreements and resolve issues related to intellectual property (IP).  As a consequence, research projects did not get underway as quickly as originally proposed in network applications to the program.

The networks have implemented effective models and management practices to achieve outcomes.  However, each network has learned key lessons along the way.  For example, it is critical to take the necessary time to ensure the right people are involved in the network and supported by a solid governance structure and decision-making processes. The majority representation of industry partners on network Boards and project selection committees help ensure the funded research reflects business needs.  Building trust and relationships amongst industry, academia and government partners are key ingredients for long-term success.  It is also important to ensure network management has the administrative capacity (i.e., resources and access to specific skill sets) to manage the complexities of the network.  A wide range of skill sets on the network boards of directors that include a blend of industry sector, scientific, financial and legal expertise was also important to network implementation and their ongoing performance. Lastly, it is important to identify realistic performance expectations and measures of success that reflect the uniqueness of each network and sectors within which they operate as well as the expected outcomes of the program. 

With many of the challenges involved in setting up network governance structures and management practices now behind the program, it is anticipated that the realization of both network and program outcomes should progress more quickly as more research projects are conducted in the remaining years. 

9.1.3 Network approach to research, development and innovation

The BL-NCE program has enhanced research, development and innovation in the areas of the four funded networks.  The business-led model has encouraged the development of industry-university research partnerships (as evidenced by the 89 projects, involving 378 researchers).

In addition to industry-university partnerships, the business-led model facilitated the development of partnerships between industry sectors, in some cases bringing together sectors that have not traditionally worked together.  The Network Agreements and IP arrangements, while requiring a significant up-front investment in time and effort by network management and partners, now facilitate the development of multisectoral, multidisciplinary R&D teams or projects.  International collaborations have been established where appropriate.

The level of industry involvement in the development of research priorities, project selection, scientific committees, Board of Directors, and guiding and carrying out research projects ensures that projects are directly relevant to industry’s needs.  Network partners are able to fully participate in the decision-making and setting research goals, and are able to influence research planning and agendas.

The networks have developed project portfolios that address the needs of network members.  Networks are strengthening links between the research community and industry, and appear to be on track to meet the needs of partners.  There is some early evidence of increased visibility of Canadian researchers involved in these networks.

9.1.4 Mobilization and benefits of network research

The business-led network approach (including the development and implementation of a strategic plan, project selection and oversight) is seen as an effective mechanism to promote mobilization of research by industry. All networks have been successful in terms of establishing and building partnerships, helping partners learn to work together and share IP, and building a knowledge base.

Approaches to conducting research vary among the networks with differing amounts of research carried out by universities, businesses and the not-for-profit research organizations. Consequently the strategies for mobilizing research results vary; however, the major mechanisms for mobilization of research identified are networking, IP and non-disclosure agreements, and refereed publications. All networks make use of conferences, workshops and meetings to share research results among network partners, funders and the broader community.

The extent to which network research will have been mobilized by partners and translated into technical applications, products and processes by the end of the four year funding will vary, depending on the sector and the type of research. The commitment of partners and the extent to which a pathway to early commercial applications has also been identified are major factors in the achievement of intended outcomes. While it is generally recognized that it is too early to expect significant achievement of long-term outcomes, in one network, there is early application of research in the development of next generation products.

9.1.5 Training of HQP

While the emphasis varies among networks, all BL-NCE networks contribute to the training of HQP through university-based research.  HQP participating in BL-NCE funded projects acquire more technical and professional skills relevant to business than those in the other comparable networks surveyed. They also gain experience relevant to the needs of the industrial stakeholders participating in the network that improves their opportunities for employment after graduation. A total of 83 university-based HQP have participated in research projects funded by the four BL-NCE networks during the first year. This number is expected to increase, as the participation of HQP in some networks to date has been affected by delays in getting university-based research underway. In addition to training of HQP at universities, network research also provides training for the private sector researchers participating in projects through their involvement in the research projects, and through interaction with university researchers and other HQP. In addition, two networks are using a mentorship approach whereby an industry representative provides guidance to network research projects. In the case of one network, the mentorship approach has enabled industry representatives to be directly involved in all phases of projects, helped the researchers stay aligned with the industrial needs, and when the time comes act as a champion to mobilization the research results.

9.1.6 Efficiency and Economy

Efficient and effective means are being used to deliver the BL-NCE program.  The evidence shows that the program has been efficient in managing its operational resources in comparison to its grant funds, particularly in comparison to other programs with larger grant funds.  The individual networks have also been effective in balancing their administrative expenditures in comparison to research funds; however, some networks have higher administration burdens at this stage given delays in becoming fully operational and getting their research projects approved.

The program has also been effective in exceeding its matching funds requirements based on actual expenditures as well as committed funds.  In fact, based on actual expenditures, the projections for partner contributions to expenditures have been exceeded (more than doubled) when all networks are combined. However, funds are not being used at the rate anticipated given delays in network implementation.

Based on committed funds, the combined funded networks have also been effective in exceeding their matching funds requirements.  To date, a significant proportion of the non-BL-NCE funds (83%) originate from the private sector (46%) and other public sector organizations (federal and provincial) (37%).

There are few opportunities for improving the efficiency of the program.  However, the short timeframe for the program has been a concern of several networks in terms of their ability to maximize their effectiveness (i.e., results), efficiency (i.e., minimized administrative expenses) and economy (i.e., maximized leveraging).  Networks were particularly concerned with the lack of relevance of current indicators or measures to their networks.  For example, publications were noted as less relevant to business-led networks.  More relevant indicators such as improvement to technology readiness were deemed important.  It was therefore noted that one key area of improvement is to ensure that the reporting requirements are aligned with business-led networks and are thus less academic in nature.

9.2 Recommendations

Recommendation 1: The BL-NCE program is showing early success and the model should therefore be maintained at the federal level.  The BL-NCE program is addressing a continued need for private sector led collaborative research and development and making progress towards the achievement of expected outcomes.  It is still too early to firmly conclude that the program will achieve its objectives to increase private sector investments in research in Canada, support the training of skilled researchers, and connect the resulting ideas and talent to businesses seeking to bring innovations to market, particularly given the early stage of the program as well as the limited number of funded networks. However, the findings of the evaluation support the validity and further funding of the program model. The findings also support the involvement of the federal government in funding of the program model as such funding enhances the scope and nature of the funded networks.

Recommendation 2:  If renewed or extended, the NCE Secretariat should consider the following to enhance the program’s ongoing relevance and effectiveness. First, allow existing networks to re-apply in future program competitions as there will likely still be an ongoing need for federal government support to these networks to achieve program outcomes. Second, focus on steps to solicit applications for networks in priority areas not funded to date to improve the alignment of the program with priority areas and private-sector needs (i.e., in the two priority areas not yet funded). Third, provide more support for the development of network applications and the implementation of funded networks to help mitigate and/or lessen the challenges that have adversely affected network implementation and operation to date. In terms of support for network implementation, this could include identifying the types of expertise and resources required to implement a business-led network as well as providing additional assistance with the development of network agreements. With respect to the application process, stronger emphasis could be placed on assessing the required expertise and resources in subsequent program competitions by revising the program’s assessment criteria and application requirements.

Recommendation 3:  The BL-NCE program’s expected outcomes and performance measurement strategy should be revisited. While the program theory appears appropriate, based on the nature and performance of the four networks funded to date, there is a need to revisit the program logic model, performance measurement strategy and extent to which and the timeframe in which some expected outcomes can be achieved.  The evaluation found that there is a need to establish a better linkage between the network level outcomes to program outcomes.  Therefore, more work is needed to better demonstrate how the outcomes of individual networks are contributing to program outcomes.  This should involve further refining the expected outcomes in the program logic model.  This should also involve revisions to the performance measurement strategy as well as assessment of the quality and comprehensiveness of the data collected to improve the relevance, appropriateness and reliability of performance indicators used to measure both network and program performance.  Revisions to the performance measurement strategy could be informed by a review of the performance data already collected as well as continued work with the four BL networks.

Annex A – Glossary of Acronyms

Annex B – Bibliography

ArboraNano, Annual Report for Fiscal Year 1, July 30, 2010

ArboraNano, ArboraNano Business-Led Network Strategic Plan, September 2009

ArboraNano, ArboraNano, Highlights 2009-2010, October 2010

ArboraNano, Why Canadians are getting Excited about Cellulose, Winter 2010 / Spring 2011

Advisory Committee for the Review of Relevance and Effectiveness of the BL-NCE Program

Bouvier, Michel, CQDM, Progress Scientific Report (12 months), September 2010

Canadian Aviation Environmental Working Group Research Consortium, Full Application, Unknown

Circum Network Inc./R.A. Malatest and Associates Ltd., Evaluation of the Networks of Centres of Excellence Design Report, February 17, 2007

Circum Network Inc./R.A. Malatest and Associates Ltd., Evaluation of the Networks of Centres of Excellence Program, Evaluation, October 19,2007

Circum Network Inc./R.A. Malatest and Associates Ltd., Inter-Agency Evaluation Steering Committee Evaluation of the Networks of Centres of Excellence Partner Questionnaire, Version 3, 2007

Circum Network Inc./R.A. Malatest and Associates Ltd., Inter-Agency Evaluation Steering Committee Evaluation of the Networks of Centres of Excellence Researcher Questionnaire, Final Version, 2007

Circum Network Inc./R.A. Malatest and Associates Ltd., Inter-Agency Evaluation Steering Committee Evaluation of the Networks of Centres of Excellence Student Questionnaire, Version 3, 2007

Circum Network Inc./R.A. Malatest and Associates Ltd., NSERC Evaluation of the Networks of Centres of Excellence Case Study Interview Guide, Version 3, 2007

Circum Network Inc./R.A. Malatest and Associates Ltd., NSERC Evaluation of the Networks of Centres of Excellence Key Informant Interview Guide, Version 3, 2007

Cofsky, Sylvain, GARDN, Fly Green, Making our Dreams a reality, GARDN at a glance, 2011

Côté, Michel, Montréal International and CQDM, International Partnerships for Drug Discovery, January 15th, 2010

CQDM, Activity Report, January 2011

CQDM, Call for Proposals, 2010-2011
CQDM, Complete Application Including book 1, 2 and 3, 2008-2009 Competition - Business-Led NC, 2008

CQDM, Explore Contribution Agreement, Unknown

CQDM, Facilitating Creative Partnerships and Open Innovation in Biopharmaceutical Research, October 6, 2010

CQDM, List of Research Agreements, Unknown

CQDM, Minutes of the Directors Meeting, Apr 22, 2010

CQDM, Minutes of the Directors Meeting, Jul 15, 2010

CQDM, Minutes to CQDM Mentorship Program Meeting, January 21, 2011

CQDM, Minutes to CQDM Strategic Orientation Committee, April 12, 2010

CQDM, Minutes to CQDM Strategic Orientation Committee, July 20, 2010

CQDM, Minutes to CQDM Strategic Orientation Committee, November 27, 2009

CQDM, Minutes to CQDM Strategic Orientation Committee, Ocober 4, 2010

CQDM, Mutual Non-Disclosure and Confidentiality Agreement, Unknown

CQDM, Network and Research Grant Agreement Template, 2011

CQDM, The Power of Ideas, 2010 Annual Report, 2010

CQDM Press Release, CQDM welcomes two additional Pharma members, Boehringer Ingelheim and GlaxoSimth Kline Inc., and launches its EXPLOREProgram to drive innovation, February 16, 2011

Crotogino, Reinhold, 2008 BL-NCE Competition, Export Panel Report for the Canadian Forest NanoProducts Network (ArboraNano) Application, September 9, 2008

Crotogino, Reinhold, ArboraNano Canadian Forest NanoProducts Network, NCE 2009 Annual Meeting, February 2009

Department of Natural Resources, Amendment No. 2 to the Contribution Agreement between Natural Resources Canada and PTRC, 2004

Fehlmann, Max, 2008 BL-NCE Competition, Export Panel Report for the Québec Drug Discovery Consortium (CQDM), September 16, 2008

Fehlmann, Max, An Innovative Response to the Pharmaceutical Industry's Challenges, NCE Annual Meeting, Winnipeg, February 25, 2009

Forest Products Association of Canada, Transforming Canada’s Forest Products Industry:  Summary of Findings From the Future of Bio-pathways Project, February 2010

FPInnovations, Book 1, Business-Led NCE 2008-09 Competition, ArboraNano - the Canadian Forest NanoProducts Network, August 19, 2008

FRSQ Press Release, Unique public-private partnership, June 17, 2008

G1-2009-2010 BL-NCE Annual Qualitative Report

Gajda, Rebecca, Utilizing Collaboration Theory to Evaluate Strategic Alliances, Sage Publications, September 7, 2009, American Journal of Evaluation, [Online]  http://aje.sagepub.com/cgi/content/abstract/25/1/65

GARDN, Annual Report 2009-2010, 2010

GARDN, GARDN announces the projects selected for its 2nd Round of Funding, Unknown

GARDN, Strategic Plan, October 2009

Goss Gilroy Inc., Summative Evaluation of the Standard Research Grants and Research Development Initiatives Programs - Final Case Study Technical Report, September 2010

Goldman, Michel, Presentation to CQDM Forum:  New Challenges for Drug Innovation: The European Perspective, June 8, 2010

Government of Canada, Canada's New Government, Mobilizing Science and Technology to Canada's Advantage, 2007

Government of Canada, Innovation "Road Map", Science, Technology and Innovaiton Council Canada, Unknown

Government of Canada, Mobilizing Science and Technology to Canada's Advantage, 2009

Government of Canada, Network Centres of Excellence Secretariat, Business-Led Networks of Centres of Excellence (BL-NCE) Program Guide, July 2010

Government of Canada, The Canadian Biopharmaceutical Industry Technology Roadmap, [Online]  http://publications.gc.ca/site/eng/297013/publication.html, accessed March 2011

Government of Saskatchewan, Agreement between the Government of Saskatchewan and PTRC, 2009

Jamieson, Pat, PTRC, Conflict of Interest Report, BL-NCE Annual Report 2009-2010, July 26, 2010

Jamieson, Pat, PTRC, Environmental Review Report, BL-NCE Annual Report 2009-2010, July 26, 2010

Jamieson, Pat, PTRC, Statement of the Board of Directors, BL-NCE Annual Report 2009-2010, July 26, 2010

Lawrence, Shawn, Biotechnology Focus, [Online]  www.cqdm.org/assets/upload/news/ArticleBiotechnologyFocusOct2010.pdf, October 2010

Mather, Robert, CAEWG-RC, 2008 BL-NCE Competition Expert Panel Report for CAEWG-RC, September 12, 2008

Morin Lanthier Schulman Govas, AboraNano Financial Statements, March 31, 2010

Natural Resources Canada, PTRC Evaluation Report, October 2002

Natural Resources Canada, Summative Evaluation of the PTRC, January 30, 2007

Networks of Centers of Excellence, Amendment No. 1 to the Funding Agreement between NSERC and SSHRC and CIHR and CQDM, 2011

Networks of Centers of Excellence, Funding Agreement between NSERC and SSHRC and CIHR and CQDM, 2009

Networks of Centres of Excellence, 2009 Impact Report by the Private Sector Advisory Board, 2010

Networks of Centres of Excellence, Appendix C to the BL-NCE Funding Agreement, 2009

Networks of Centres of Excellence, Appendix C to the BL-NCE Funding Agreement, BL-NCE Agreement Among PTRC and Network Members, Unknown

Networks of Centres of Excellence, Appendix C to the BL-NCE Funding Agreement, Unknown

Networks of Centres of Excellence, BL-NCE Funding Agreement Template, Feb 2009

Networks of Centres of Excellence, Delivering Research Excellence and Innovation: The Networks of Centres of Excellence Year in Review 2008-2009, 2009

Networks of Centres of Excellence, Full Application Guide for the 2008-09 Competition, 2008

Networks of Centres of Excellence, NCE Annual Report 2006-2007, The Winning Advantage, 2007

Networks of Centres of Excellence, NCE Annual Report 2006-2007, The Winning Advantage, 2007

Networks of Centres of Excellence, Research That Works: The Networks of Centres of Excellence Year in Review 2007-2008, 2008

Preston, Carolyn K., PTRC, A PTRC Technology Roadmap for Saskatchewan 2008-2050, March 2008

Preston, Carolyn K., PTRC, STEPS BL-NCE

Preston, Carolyn K., PTRC, STEPS Final Application, 2008

Preston, Carolyn K., STEPS, 2008 BL-NCE Competition, Expert Panel for STEPS, September 17, 2008

Preston, Carolyn K., STEPS, Strategic Plan, January 2010

PRTC, Annual Report 209-2010, 2010

R.A. Malatest & Associates, Joint Evaluation Framework for the Business-Led Networks of Centres of Excellence (BL-NCE) Networks of Centres of Excellence (NCE) and Strategic Network Grants (SNG) Programs, August 18, 2010

Raymond Chabot Grant Thornton, Auditor's Report and Financial Statements, March 31, 2010 and 2009

Résaux des Centres d'Excellence gérés par l'Entreprise, Plan Stratégique, Octobre 2009

Science and Technology and Innovation Council, State of the Nation 2008, Canada's Science, Technology and Innovation System, 2008

Science-Metrix, Evaluation of the Collaborative Research and Development Grants Program, Technical Report - Case Studies, March 15, 2010

STEPS, Funding Agreement STEPS, BL-NCE Between PTRC and BP Exploration (Alaska) Inc, April 2009

STEPS, Funding Agreement STEPS, BL-NCE Between PTRC and Canadian Natural Resources Limited, April 2009

STEPS, Funding Agreement STEPS, BL-NCE Between PTRC and Husky Oil Operaitons Limited, April 2009

STEPS, Funding Agreement STEPS, BL-NCE Between PTRC and Nexen Inc, April 2009

STEPS, Funding Agreement STEPS, BL-NCE Between PTRC and Total E&P Canada Ltd., April 2009

Unknown, AboraNano Network Agreement, January 2011

Unknown, Free Form Reports, Section 6.0, Unknown

Unknown, Joint Results-based Management and Accountability Framework and Risk-Based Audit Framework for the Grants Program for Business-led Networks of Centres of Excellence, December 2008

Unknown, List of Contribution Agreements, Unknown

Virtus Group, Auditor's Report, May 19, 2010

Virtus Group, PTRC Audit and Annual Financial Statements, March 31, 2010

---

Footnotes

1. Joint RMAF-RBAF for the BL-NCE (May 2009), p.21.
2. While the areas of information and communication technology and management, business or finance were identified as program priority areas, none of the current networks are directly aligned with these priority areas.
3. The initial funding period was for four years from 2008-09 to 2011-12 but has been extended by one year with no additional funding.
4. Note: CQDM does not track in-kind contributions.
5. With 6 individuals
6. With 4 individuals
7. With 2 individuals
8. It is important to note that some network members are also committee members.
9. Note:  The 39 BL-NCE partners removed include the 23 interviews completed as well as those who were eliminated during the interview scheduling process (e.g., refused, indicated that they were not involved in the network or not sufficiently involved to contribute, etc.).
10. Joint Results‐based Management and Accountability Framework and Risk‐Based Audit Framework for the Grants Program for Business‐led Networks of Centres of Excellence (BL‐NCE Program) December 2008, p.2.
11. Year 1 Annual Reports (2009-2010) for ArboraNano, CQDM, GARDN, and PTRC-STEPS.
12. NCE, BL-NCE Competition. 2008 Review of Full applications Presentation. August 2008, Slide 3.
13. This figure is based on the four funded networks.
14. Joint RBAF-RMAF for the Grants Program for BL‐NCE Program. December 2008.
15. NCE, BL-NCE Competition. 2008 Review of Full applications Presentation. August 2008. Slide 9.
16. The comparative programs mentioned by interviewees were: NSERC’s Strategic Networks (n=2), the National Research Council’s Industrial Research Assistance Program (NRC-IRAP) (n=2), Genome Canada, the other NCEs, Precarn, Paprican, Canadian Water Networks, university research chairs, and provincial programs (Alberta, Prince Edward Island, Québec, and Ontario noted specifically)
17. Survey respondents did not expand on this acronym; however, it may refer to SAGE Labs:  www.horizondiscovery.com/in-vivo-models
18. Public Works and Government Services Canada, Review of Federal Support to Research and Development Expert Panel Consultation Paper, 2011, page 14.
19. Canada. Department of Finance Canada. Advantage Canada, Building a Strong Economy for Canadians. Ottawa. 2006. Available at http://www.fin.gc.ca/ec2006/pdf/plane.pdf. Accessed: 2 June 2011, p. 58 including a reference to OECD, Main S&T Indicators.
20. S&T and Innovation Council, State of the Nation 2008, Canada's Science, Technology and Innovation System, 2008, p.21, including a reference to OECD, Main S&T Indicators, 2008.
21. CCA, Innovation and business strategy [electronic resource]: why Canada falls short / the Expert Panel on Business Innovation in Canada, Ottawa, Canada, June 2009, pp. 11-12.
22. Two other priorities identified in the RMAF-RBAF (Information and communications technologies and Management, business or finance) are not addressed by current funded networks.
23. Joint RBAF-RMAF for the Grants Program for BL‐NCE Program, December 2008, p.3.
24. CCA, Innovation and business strategy [electronic resource]: why Canada falls short / the Expert Panel on Business Innovation in Canada, Ottawa, Canada, June 2009, pp. 11-12.
25. Precarn's distinctive model, proven successful for over 16 years, helps collaborative teams undertake high-risk research on technical challenges associated with new market opportunities. The Precarn Model ensures every project has at least one end user and one academic partner collaborating with the development lead.  Furthermore, the model encourages collaboration among multiple development partners.  This strength drives innovations towards commercial viability; it generates highly qualified entrepreneurial people, and reduces the risk and cost of research.  The Precarn model also drives the development of a network of people and companies all predisposed to collaborative research, all communicating and exchanging ideas, experiences and technology.
26. Technology Roadmaps are industry-led, government-facilitated planning exercises among participants from industry, universities and colleges, and governments, focused on technologies needed by a specific sector. The steps in road mapping are, first, to assess the technology needs for the sector; to identify the promising technologies that could meet the defined needs; and then to plan the best route for the applied research, development and demonstration needed to make the technologies available.
27. A non-disclosure agreement (NDA) is a first step in knowledge transfer. Private sector firms use NDAs to gain access to another organization’s information or technology for the purposes of determining if it is sufficiently relevant to warrant further negotiation to develop a strategic alliance or partnership.
28. STEPS 2009-2010 BL-NCE Report
29. BL-NCE Annual Qualitative Report, 2009-2010. (p.1)
30. STEPS Strategic Plan, pg.6
31. New project with major funding from Sustainable Development and Technology Canada
32. BL-NCE 2009-2010 Reports G3List of Researchers and HQP for all four networks
33. Due to the small sample sizes of the surveys and large margins of error, most of the differences between BL and comparable network responses are not statistically significant.
34. The program has been extended by one year with no additional funding, due to a late start on the first year.
35. While the networks have received three years of grants from the BL-NCE program, the first year funding was awarded in March 2009 and therefore the first complete year was, in fact, the 2009-2010 FY (two years of grants but only one year of operations).
36. Reads: For every BL-NCE dollar, another $1.23 is contributed by all other partners.