Performance Evaluations of Technology Transfer Offices of Major US Research Universities

By analyzing the highlights of the major activities reported by the technology transfer offices (TTOs) of twenty US major universities, the performances of TTO activities are quantitatively assessed and the associated scores are compared with each other. The key performance indicators, which govern the success of the university technology transfer, are specifically selected and examined. Two normalized metrics, overall performance metric (OPM) and patenting control ratio (PCR), which are the representing combined indicators for the TTO performance, are developed and demonstrated. The two metrics are evaluated for each university selected and compared to specifically provide a comprehensive overview of how good is the TTO of a university as compare to those of its peers. Finally, the factors for a successful TTO are described and the major unsolved issues are also discussed.


Introduction
After the passage of the Bayh-Dole Act by United States Congress in 1980, more than two-hundred fi fty U.S. universities have establish a certain type of technology transfer offi ces (TTOs) to cope with the increase amount of technology transfer activities (Stevens, 2004;Anderson, Daim, Lavoie, 2007).The Bayh-Dole Act allows a university to retain title to federally funded inventions and encourages universities to license inventions to industry.The ownership to the intellectual properties (including patent, copyright, and others) provides motivation, especially economic incentives, for a university to promote and emphasize the technology transfer activities.According to the studies by AUTM (Association of University Technology Managers, 2012) and Tseng et al. (Tseng, Raudensky, 2015), the number of patents issued to U.S. universities grew from fewer than 250 in 1980 to 4,700 in 2011, a near 20-fold growth.
Recently, the TTO has become increasingly important, given concerns regarding the university's desire to maximize the returns to its intellectual property, especially the patents it owns.Furthermore, because of the growth of Internet commerce and the advances in digital, information, and biogenetic technologies, a broad range of entrepreneurial companies have been formed and the substances in technology transfer have changed greatly as compared to those in twenty or thirty years ago.For example, the patentability of biological materials and biomedical research tools as well as the technology transfer of digital data and computer software are becoming hot issues in technology transfer only recently (Resnik, 2001;Hoeyer, 2008;Cohen, Lemley, 2001).It should be useful to assess the recent performance of the TTOs in the current technology environment.
Consequently, the purpose of the present paper is to quantitatively assess the developments and performances of the technology transfer activities of twenty major research universities in the fi scal year 2011 (FY 2011).Six leading performance metrics are selected to measure the TTOs' performance.Using these six metrics, an overall performance metric (OPM) is developed to provide a single metric to quantify the accomplishment of the TTOs.The OPM are evaluated for each of the twenty universities, and the resulting scores are compared to each other to demonstrate its simplicity and comprehensiveness.A patenting control ratio (PCR) is also developed for each of the twenty universities to judge the properness of the patenting strategy adopted by its TTO and to gauge whether or not its patenting budget is being well spent.Furthermore, the success factors and the issues related to TTOs are examined and discussed.Finally, the future perspectives of TTO activities are given.

Performance Metrics for Technology Transfer Offi ces (TTOs)
The selections of the performance metrics and the universities for comparison as well as the data collections for the selected universities are presented in this section.

Technology transfer offi ce
In a university, a TTO is an administrative unit to manage all the activities related to technology transfer, where technology transfer refers to a process that transferring intellectual properties rights (mainly patent right) from a university to a for-profi t sector for the purpose of commercialization.Normally, commercialization involves the activities related to patent applying, license pursuing, and startup launching.The TTO revenue counted include income from licensing royalty, reimbursement of patent expense and licensee legal fees, and extraordinary income, which can be nonrecurring items such as sales of equity and payments resolving patent litigation cases.Frequently, upon execution of a license agreement, licensees normally reimburse the licensor or TTO for past patent expenses or initial administration and organization.
Among the 20 universities selected, 11 are from the list of the twenty universities with the largest R&D expenditures in FY2009, reported by US National Science Foundation (NSF), where total 697 U.S. universities are studied (Britt, 2010).The 9 universities which are not on the top 20 list are chosen to allow the present study to have the universities with superior performance in technology transfer or to have a broader geographic representation.For example, the best two TTO revenue performer in Table 1, Northwestern and Columbia Universities are not on the top 20 NSF list and are ranked thirty and twenty-four in the NSF report.Wake Forest and Arizona State Universities and University of Utah, which are not among the top 20 NSF list, are selected for geographic balancing.Furthermore, some of the universities on the top 20 NSF expenditure list, which were not selected, did not post the technology transfer data required and did not fully respond to email requests from the authors.

Metrics data collection
The research expenditure data are obtained directly from the NSF FY2009 report (Britt, 2010), and their associated ranks are the superscripted numbers on the expenditure data listed in Table 1.Note that the effect of research expenditure having 2-3 years of time lag on TTO activities (such as patent filing) is normal.Consequently, the facts that the research expenditure data are based on FY2009 and other TTO performance data are based on FY 2011 are appropriate.As indicated in Table 1, on average, each of these 20 universities spent $713 million in sponsored research, with research expenditure of each university varying from $201 million for Wake Forest University to $1,856 million for Johns Hopkins University.Nonetheless, the twenty universities studied are all RU/VH (very-high research activity) universities categorized by the Carnegie Foundation for the Advancement of Teaching in 2010 (Carnegie Foundation, 2013), given that it is used extensively as the de facto standard by higher education researchers.The typical processes associated with these activities are described in Fig. 1, where the first five steps (green or shadowed blocks) belong to the patenting and the major players associated with each step are also indicated.If any step ends in a red-colored decision shown in the figure, the patenting process is terminated.Suggestions are normally provided for reconsideration.The remaining five steps (the white or nonshadowed blocks) shown in Fig. 1 are related to the licensing process.The process can involve further steps for technology development and startup company launching.

Performance metrics
The performance of university TTOs has been studied by many investigators, and a wide range of metrics has been selected to assess their performance.Trune & Goslin (1998), Rogers, Ying, Joern (2000), Thursby & Thursby (2003), Litan, Mitchell, Reedy (2007) and Roessner, Bond, Okubo, Planting (2009) suggested that license revenue is the most important outcome and performance metric for university technology transfer, while Rogers, Ying, Joern (2000) and Xu, Parry, Song (2011) applied invention disclosures as an antecedent variable to measure the achievement of TTOs, such as the numbers of TTO licenses and startups.Based on the above mentioned studies (Trune, Goslin, 1998;Rogers, Ying, Joern, 2000;Thursby, Thursby, 2003;Litan, Mitchell, Reedy, 2007;Roessner, Bond, Okubo, Planting, 2009;Xu, Parry, Song, 2011) and those by Anderson, Daim, Lavoie, 2007, Abrams, Leung, Stevens, (2009), York & Ahn (2012), Foltz, Barham, Kim, (2000), O 'Shea, Allen, Chevalier, Roche (2005), West (2012) and Kurman (2011), the performance measure for a TTO can be quantified by the following metrics: a) TTO revenue, b) number of invention disclosures, c) number of patent applications, d) number of patents granted, e) number of licenses signed, f) number of start-ups formed, g) research expenditure of university scientists, h) expenditure of patenting activities, i) operation expenditure, j) number of new commercial products, k) employment and productivity growth of startup partners, l) changes in stock prices of industrial partners, etc.The first six metrics are more frequently applied for measuring the performance or accomplishment of a TTO.In the present study, these six metrics are adopted for the quantitative evaluation of TTO performance for twenty major US research universities.It is noteworthy that among these six metrics adopted, TTO revenue is the resulted measure of the entire TTO activities shown in Fig. 1.In fact, the six metrics selected reflects the outcomes of the ten steps of the technology transfer process illustrated in Fig. 1.

Selection of twenty major research universities
Twenty major research universities, which are ranked by their annual TTO revenue, are selected and listed in Table number of licenses signed, number of startup companies launched, and R&D expenditures.All values are obtained either from the TTOs' websites or from direct communication with the TTOs.In fact, all 2011 data have been compared and verified with the data reported by the AUTM FY 2011 Licensing Activity Survey (AUTM, 2012).The number of invention disclosures reported in the table does not include copyright and material (including chemicals, plant-or animalderived material, and software) disclosures.The number of US patent applications does not include US provisional and foreign (non-USA) applications, while the number of US patents issued does not count foreign patents.The number of

Overall Performance Metric and Patenting Control Ratio
Many stakeholders, such as academic researchers, technology transfer offices (TTOs) and private industry, which are involved in technology transfer, have called for a comprehensive way to measure the effectiveness or performance The weighting factor for each startup company launched is 0.0025 because the amount of effort of a TTO spends on a startup is normally similar to that of licensing.The equal weight value assigned for startups can provide some incentive for a TTO because many government programs, including the Act, encourage technology transfer activities directly towards a small-business or startup.One may argue that the value assigned to the startup weighting factor is too high if one believes that a primary reason large numbers of startups are launched, especially in an economic recession with a high unemployment rate, is because starting up a company is an easy option for graduating students to hide their unemployment.Also, during economic recession, industry is less interested in licensing activities (Tseng, Raudensky, 2015).
Patenting activities of the first five steps shown in Fig. 1 are quantified by the numbers of innovation disclosures, patent applications, and patents granted are the main source to provide intellectual property to be transferred and should have an equal weight as that for licensing.TTOs are mainly contributing to the management aspects of these activities, and inventors play the major role in these activities.Thus, each of the three activities is separately assigned a weighting factor of 0.001 for each of the activities performed.The combined weighting value for patenting is 0.003, slightly higher than the single weighting value for licensing.Based on the above discussions and those information presented in Sect. 2 and references (Trune, Goslin, 1998;Rogers, Ying, Joern, 2000;Thursby, Thursby, 2003;Litan, Mitchell, Reedy, 2007;Roessner, Bond, Okubo, Planting, 2009;Xu, Parry, Song, 2011) the values of the six weighting factors assigned should be fair and reliable.

OPM evaluation and comparison
The OPM for the twenty major research universities are calculated and their values varying from 0.42 to 2.17 are summarized in Table 1, where the mean and the standard deviation are 0.98 and 0.45, respectively.Among the 20 universities, MIT and Northwestern, Johns Hopkins, and Columbia University, are excellent performers with scores higher than 1.50, while the scores of Wake Forest and Washington Universities and Arizona State University are less desirable with values less than 0.50.The three of the 20 universities with OPM values between 1.00 and 1.50 should be considered as very good performers.The other 10 universities with scores between 0.50 and 1.00 can be ranked as a good performer or at least having great potentials to become a good performer.
Universities with less desirable scores may imply that their TTO's strategies or approaches for technology transfer are needed to be re-assessed or changed.One has to be careful in using this metric because no information on TTO budget of TTOs (Kurman, 2011).In this section, the performance is evaluated and analyzed for each of the 20 major research universities by a newly developed metric, called the overall performance metric (OPM).This OPM is based on the six leading metrics selected in the precedent section.The rationale and procedures in developing this normalized metric are presented; the scores for each of the 20 selected universities are compared and discussed.A supplementary parameter, called the patenting control ratio (PCR), is also implemented for gauging the effectiveness of the patenting policy and procedure adopted by the universities studied.

Overall performance metric (OPM)
In the preceding section and shown in Fig. 1, it has been discussed that the six performance metrics, not the research expenditure, can be directly representing each steps of the performance of the TTOs activities in different degrees of influence on the relevance to the metric considered.The weighting factor for each of the six metrics is estimated to represent its influence or relevance to the TTO's overall efforts.The higher the relevance or influence of the metric to the TTO efforts, the higher the weighting factor is. Consequently, the OPM can be formulated as: OPM = $M revenue/200 + no. of licenses/400 + no. of startups/400 + no. of patents/1000 + no. of disclosures/1000 + no. of patent applications/1000 (1 where OPM is a combination of the TTO revenue and the numbers of licenses agreed, startups launched, patents issued, disclosures submitted, and patent applications filed, associated with different weighting factors, which are proportionally adjusted to make the average OPM value close to 1.The OPM is developed based on "outcomes" instead of "process".
The value of the weighting factor assigned to each metric is dependent on the level of importance of the metric considered.As discussed earlier, the TTO revenue is the most important outcome of the entirety of TTO activities.It is a major source for covering TTO spending and is often used as the benchmark for gauging a TTO's achievement.
The weighting factor for TTO revenue is 0.005 for every million dollars received.The other five of the six metrics adopted are outcomes of the ten major steps (or activities) of the technology transfer process illustrated earlier in Fig. 1.The number of the licenses signed represents is the outcome of the last five major steps or activities shown in Fig. 1, which are primarily initiated, performed and executed by TTO's staff.In fact, licenses are also providing the basis for producing TTO revenue.A weighting factor of 0.0025 is assigned for each license assigned, which is one-half of that for TTO revenue.
The value of the number of patent applications used in Eq.
(2) should be one to three years before the year of evaluation, i.e. the year of the patents granted, since it normally takes one to three year to go through the application process.In the present analysis, the number of patent applications used in the calculation is one year behind, i.e., based on FY2010 data.The PCRs for the twenty major universities are calculated and their values with the mean and associated standard deviation are summarized in Table 1.As indicated in the table, the mean of the PCR% is 48.1% with a standard deviation of 26.5.If the ratio is higher than the norm, i.e., 48%, it may indicate either the quality of the patent application is generally good or the patenting procedure is well managed by a TTO.However, this ratio should not be too high, because, it means the internal reviewing or screening process is too stringent or too tough, which may cause some good inventions to be mistakenly ignored or rejected; this could also diminish the enthusiasm of the faculty to cooperate with the TTO.
This ratio can also be used to gauge the effectiveness of the patent budget being spent or the properness of the patenting budget.As indicated in Table 1, the PCR varies from 12.9% to 143.1%.Wisconsin and UC Berkeley have scores higher than 68%, while Johns Hopkins, Arizona State, and Colorado have values less than 20%.The former two should request an increase in their patenting budget next year to accommodate more patent applications because their successful rate are so high; the latter three should work harder to improve their patenting process to avoid their patenting budget from cut.The higher the ratio value, the higher the patenting budget should be appropriated in the future.
Certainly, an adequate range of this ratio can be adjusted by each university based on internal benchmarking data and the goals or strategy set for its TTO.It is also understood that patents granted may be the outcome of patent applications filed more than one years before.However, a reasonable number of benchmark universities is selected, and each university studied in the table is relatively large.Therefore, the patent applications data is based on one year before should be appropriate.Certainly, more studies to obtain more data sets and to perform analysis on this ratio should be encouraged.

Success Factors and Issues in University Technology Transfer
In this section, the success factors for university technology transfer are presented; the issues, especially, related to the alignment of institutional interests and faculty participation in technology transfer are also discussed.Some recommendations to alleviate these hurdles are then provided.
or size as well as the university size is incorporated with these metrics evaluated.It could be expected that if more resources are allocated into the TTOs, their scores could be higher.As a result, the performance evaluation should be used to compare a university TTOs from the same type of universities with similar TTO expenditure to eliminate unfair effects on some smaller TTOs.
One should also be careful in comparing the present results with other studies, because the metrics used may be defined differently from those used by other studies.For instance, for licensing revenue, some universities include income based on royalties, upfront licensing fees, and software licenses, while other institutions report equity sales and distributions, maintenance fees, and/or legal settlements.
With patents issued, many universities only count patents granted in the U.S., while others include patents approved from either the U.S. or abroad.The timeliness of reporting may also be a problem, because many universities do not update their performance data every year.The lack of uniform reporting standards and metric definitions can make the comparison less reliable or meaningful and can cause unnecessary confusion to determine what strategies work best for a specific TTO.
In the present study, the collection of metrics data is mainly based on self-reporting or website searching (including AUTM data), although the authors try to be consistent in the data collection, especially through electronic communication.Furthermore, the OPM is focused on the outcome instead of the process of technology transfer, so some decent efforts by TTO may be ignored in the calculation or may not be fully reflected in the metric values adopted.The activities emphasized can be different from university to university.

Patenting control ratio (PCR) and comparison
The PCR is a dimensionless metric and is the number of patents granted normalized by the number of patent applications.It can be expressed as where PCR is in [%] and is a measure of the effectiveness of the patenting process adopted by a TTO.The reason for the two metrics selected to defined the PCR ratio is that these metrics are good indicators for gauging how much work is done for patenting activities by a TTO.Most importantly, the PCR can directly measure the effectiveness of the TTO's patent screening process.

Issues related to university technology transfer
In the issue of the alignment of institutional interests, the licensing revenue share adopted by Stanford mentioned above is the way to align the inventor to enthusiastically work with the TTO and the licensee.Boni and Emerson (2005) found that by reducing its equity percentage in startups and decreasing its licensing percentage from 15% to 5% can improve the institutional alignment issue, especially among the university, inventors and industrial partners.To the factor of the institutional support, the recommendation presented in the preceding section to report the TTO performance metrics and financial data is to provide the essential information to secure the top-down support from the university, especially the administrators and faculty.
Other issues were also arisen when faculty in several universities interviewed by York and Ahn for their study on the success factors [15].One of the major issues is that faculty should be rewarded through tenure and/or promotion for technology transfer activities.It is understood that, normally, an invention disclosure has to be initiated by faculty while the associated incentives or rewards should be appreciated by faculty.Although revenue sharing is significant, the reward towards tenure and promotion is even more attractive and paramount critical.Most university guidelines for tenure and promotion include criteria in three categories: teaching, research, and service [25].However, at most research universities, the academic reward system is only structured to encourage quality scholarship primarily in the form of journal publications, formal contributions to the knowledge base in specific fields; scholarly publications count significantly toward both tenure and promotion as well as salary.As indicated by Boyer [26], some research universities, tenure often depends solely on research publications and research grants.Even articles in refereed teaching journals and teaching grants may not be counted towards tenure at such universities.On the other hand, to demonstrate a strong record on research publications and research grants is not easy, known as "publish or perish" pressure [27].Consequently, the priority to participate in the activities of technology transfer becomes secondary, if the activates are not totally ignored.If faculty can be rewarded towards tenure and/or promotion, the enthusiasm for faculty to join in the technology transfer activities should be greatly improved without any doubt.
Certainly, there are many other issues related to technology transfer occurring in different university campuses.In general, TTOs can provide positive influence on technology transfer through direct communication to major shareholders.For example, by offering a course (can team with a faculty member), TTO staff can educate students and faculty members on the importance of technology transfer to pro-

Success factors
York and Ahn (2012) reviewed the literature studying the factors contributing to the success of university TTOs and identified the following eight major success-factors: a) business strategy and marketing, b) intellectual property protection, c) performance benchmarking, d) revenue generation focus, e) institutional prestige, f) business stakeholder relationships, g) alignment of institutional interests, and h) institutional support.Among them, six factors, including a), b), c), d), and f) are directly related to the TTO activities and have been elaborated in the preceding sections.Only the three factors, e) institutional prestige, g) alignment of institutional interests, and h) institutional support, are further discussed in this section.
By studying the impact of the institutional prestige on technology transfer, Shine, Shane, Di Gregorio (2003) found a relation between institutional prestige and the number of the license agreements made by universities and claimed that prestige a greater predictor of licensing success than previous licensing activities.Their study measured prestige based on graduate school ranking surveyed by US News & World Report and other media.Kumaramangalam (2005) found that higher academic status and collaboration conferred tangible value to entrepreneurial companies.
For example, Stanford University, which is one of the highest ranked universities in US News & World Report, may be used to show the halo effect discussed by Shine, Shane, Di Gregorio (2003).Based on the technology transfer data obtained from its website, Stanford in 2011 had 504 invention disclosures and 123 license agreements.Since 1970 to 2011, Stanford inventions had generated approximately $1.5 Billion in licensing income, approximately $50K in 1970 and $67M in 2011.But only 3 out of 9,000 inventions have been "a big winner" and only 75 have generated over $1 million.The three big winners are $255M from Cohen-Boyer Recombinant DNA, $337M from Google, and $370M from Functional Antibodies.In Stanford, a 15% of the licensing revenue directly goes to TTO to cover its operation cost, while the rest is evenly shared by the inventor, the inventor's department, and the school.In some universities, the incentive to the inventor can be as high as 50% of the royalty sharing after the revenue deducted from the TTO expense.
As compared to the data shown in vide the motivation for students to work with faculty members to synthesize the good ideas.Also, by organizing local seminars to faculty, students, and possible industrial partners, TTO can communicate with the audience directly and can smooth the technology transfer process which enables the TTO to determine what kind additional assistances are worthwhile to the shareholders, especially the local industrial partners.Thus, TTOs can help to create a culture, in which university shareholders value highly in participating in technology transfer.Also, the university as a whole can recognize that the technology transfer of the invention is essential to promote an institution's name recognition and prestige as well as attract and retain top faculty and students, and secure private and public funding.

Concluding Remarks
The background and the major activities of university technology transfer offices (TTOs) are described and discussed to provide the foundation for the selection of six leading metrics for assessing the performance of university TTOs.
The six performance metrics for twenty major research universities are collected and studied.These collected metrics are then used for developing two normalized metrics, overall performance metric (OPM) and patenting control ratio (PCR), for measuring the TTO performance and patenting efficiency, respectively.These normalized metrics enable a university to standardize its performance evaluation process and to establish benchmarks for strategic decision, which help to allocate appropriate resources to TTO and to yield the highest technology transfer performance.
The OPM scores for the 20 universities are compared with each other and the score can be specifically used by a university not only to evaluate its TTO performance (or service) but also, for instance to search alternative approaches adopted by its peer universities to develop an better strategy for resource allocation, personnel configuration, disclosure procedure, or university policy.Of course, the OPM can be more objective and reliable, if it can be normalized by the TTO's expenditures or other resources, such as TTO size or faculty size.Moreover, the weighting factors used to form the OPM should be studied further to increase the objectivity and reliability of the OPM.
The patenting control ratio (PCR) has also been developed and used to gauge the appropriateness of the patenting process set by each of the 20 universities considered.The mean and standard deviation of the PCR for the 20 universities are 48.1% and 26.5%, respectively.If a university's PCR is much higher than this mean value, this implies that the patent screening (or reviewing) process may be too tough, which can make faculty less enthusiastic to cooperate with TTO, especially in preparing invention disclosures.However,

Figure 1 .
Figure 1.Typical processes of technology transfer in university

Table 1 .
(Britt, 2010)ce data for selected major-research universities in FY 2011 *The numbers of the patent applications used in calculation are based on FY2010 data **Superscribed number represents the rank of the R&D expenditures in total 697 US universities in FY 2009 reported by US NSF(Britt, 2010).

Table 1 ,
Stanford's TTO certainly can be one of the top TTO performers.The impact of the institutional prestige on technology transfer is difficult to be quantified and there is a little that can be done on prestige by a TTO.Nevertheless, to provide a good service to faculty and licensees should be always helpful to the university reputation.ifthisratio is too low, it may indicate that the TTO is overbudgeted and the patenting screening process should be more stringent.The success factors and issues related to TTOs are then discussed.Recommendations are provided to help to promote an environment, in which the university community can value highly on technology transfer.As a final remark, innovation in universities and industrial sectors is a key in increasing competitiveness.The technology transfer has driven past US economic gains and will sure to drive future prosperity.The capability and technological assets across the universities and business sectors can work together to ensure that the US is once again the world's most innovative and competitive nation.
ship in his alma mater and was awarded the title Associate Professor and, in 2002, he was bestowed a Full Professorship at the Faculty of Mechanical Engineering.Recently, his attention has been concentrated on both engineering and management research for practical industrial applications.